[ARVADOS] updated: 1.3.0-1465-gb96728249
Git user
git at public.curoverse.com
Thu Aug 8 19:46:55 UTC 2019
Summary of changes:
apps/workbench/test/unit/disabled_api_test.rb | 8 +-
doc/_config.yml | 22 +-
doc/api/crunch-scripts.html.textile.liquid | 2 +
doc/api/methods/humans.html.textile.liquid | 2 +
doc/api/methods/job_tasks.html.textile.liquid | 2 +
doc/api/methods/jobs.html.textile.liquid | 4 +-
.../methods/pipeline_instances.html.textile.liquid | 2 +
.../methods/pipeline_templates.html.textile.liquid | 2 +
doc/api/methods/specimens.html.textile.liquid | 2 +
doc/api/methods/traits.html.textile.liquid | 2 +
.../install-compute-node.html.textile.liquid | 112 --------
.../install-crunch-dispatch.html.textile.liquid | 207 -------------
.../crunch-utility-libraries.html.textile.liquid | 228 ---------------
.../examples/crunch-examples.html.textile.liquid | 102 -------
.../reference/job-pipeline-ref.html.textile.liquid | 14 -
doc/user/topics/arv-run.html.textile.liquid | 163 -----------
.../crunch-tools-overview.html.textile.liquid | 70 -----
doc/user/topics/run-command.html.textile.liquid | 319 ---------------------
...nning-pipeline-command-line.html.textile.liquid | 58 ----
.../topics/tutorial-parallel.html.textile.liquid | 85 ------
.../tutorial-trait-search.html.textile.liquid | 278 ------------------
.../running-external-program.html.textile.liquid | 85 ------
.../tutorial-firstscript.html.textile.liquid | 112 --------
.../tutorial-submit-job.html.textile.liquid | 95 ------
lib/config/generated_config.go | 30 +-
sdk/go/arvadosclient/arvadosclient_test.go | 50 ----
sdk/python/arvados/commands/run.py | 187 +-----------
sdk/python/bin/arv-run | 7 -
sdk/python/setup.py | 1 -
sdk/python/tests/test_arv_run.py | 28 --
sdk/python/tests/test_pipeline_template.py | 62 ----
sdk/python/tests/test_retry_job_helpers.py | 31 --
services/api/test/unit/job_test.rb | 4 +-
33 files changed, 28 insertions(+), 2348 deletions(-)
delete mode 100644 doc/install/install-compute-node.html.textile.liquid
delete mode 100644 doc/install/install-crunch-dispatch.html.textile.liquid
delete mode 100644 doc/sdk/python/crunch-utility-libraries.html.textile.liquid
delete mode 100644 doc/user/examples/crunch-examples.html.textile.liquid
delete mode 100644 doc/user/reference/job-pipeline-ref.html.textile.liquid
delete mode 100644 doc/user/topics/arv-run.html.textile.liquid
delete mode 100644 doc/user/topics/crunch-tools-overview.html.textile.liquid
delete mode 100644 doc/user/topics/run-command.html.textile.liquid
delete mode 100644 doc/user/topics/running-pipeline-command-line.html.textile.liquid
delete mode 100644 doc/user/topics/tutorial-parallel.html.textile.liquid
delete mode 100644 doc/user/topics/tutorial-trait-search.html.textile.liquid
delete mode 100644 doc/user/tutorials/running-external-program.html.textile.liquid
delete mode 100644 doc/user/tutorials/tutorial-firstscript.html.textile.liquid
delete mode 100644 doc/user/tutorials/tutorial-submit-job.html.textile.liquid
delete mode 100755 sdk/python/bin/arv-run
delete mode 100644 sdk/python/tests/test_arv_run.py
delete mode 100644 sdk/python/tests/test_pipeline_template.py
via b96728249db70975be33bb5c5721ec5b8a0eb3ff (commit)
via 448f667a574b50da096051a0d062b9059ab3609f (commit)
from 9f6bfd78b88486d4b3384b4be05c1f5e4f546a9c (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit b96728249db70975be33bb5c5721ec5b8a0eb3ff
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Thu Aug 8 15:46:42 2019 -0400
15133: More test updates. Remove arv-run.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/apps/workbench/test/unit/disabled_api_test.rb b/apps/workbench/test/unit/disabled_api_test.rb
index 9e18a7063..54e7c0819 100644
--- a/apps/workbench/test/unit/disabled_api_test.rb
+++ b/apps/workbench/test/unit/disabled_api_test.rb
@@ -5,13 +5,7 @@
require 'test_helper'
class DisabledApiTest < ActiveSupport::TestCase
- test 'Job.creatable? reflects whether jobs.create API is enabled' do
- use_token(:active) do
- assert(Job.creatable?)
- end
- dd = ArvadosApiClient.new_or_current.discovery.deep_dup
- dd[:resources][:jobs][:methods].delete(:create)
- ArvadosApiClient.any_instance.stubs(:discovery).returns(dd)
+ test 'Job.creatable? is false' do
use_token(:active) do
refute(Job.creatable?)
end
diff --git a/lib/config/generated_config.go b/lib/config/generated_config.go
index 602f30e1d..971b810f5 100644
--- a/lib/config/generated_config.go
+++ b/lib/config/generated_config.go
@@ -576,7 +576,11 @@ Clusters:
AssignNodeHostname: "compute%<slot_number>d"
JobsAPI:
- # Enable the legacy Jobs API. This value must be a string.
+ # Enable the legacy 'jobs' API (crunch v1). This value must be a string.
+ #
+ # Note: this only enables read-only access, creating new
+ # legacy jobs and pipelines is not supported.
+ #
# 'auto' -- (default) enable the Jobs API only if it has been used before
# (i.e., there are job records in the database)
# 'true' -- enable the Jobs API despite lack of existing records.
@@ -589,30 +593,6 @@ Clusters:
# {git_repositories_dir}/arvados/.git
GitInternalDir: /var/lib/arvados/internal.git
- # Docker image to be used when none found in runtime_constraints of a job
- DefaultDockerImage: ""
-
- # none or slurm_immediate
- CrunchJobWrapper: none
-
- # username, or false = do not set uid when running jobs.
- CrunchJobUser: crunch
-
- # The web service must be able to create/write this file, and
- # crunch-job must be able to stat() it.
- CrunchRefreshTrigger: /tmp/crunch_refresh_trigger
-
- # Control job reuse behavior when two completed jobs match the
- # search criteria and have different outputs.
- #
- # If true, in case of a conflict, reuse the earliest job (this is
- # similar to container reuse behavior).
- #
- # If false, in case of a conflict, do not reuse any completed job,
- # but do reuse an already-running job if available (this is the
- # original job reuse behavior, and is still the default).
- ReuseJobIfOutputsDiffer: false
-
CloudVMs:
# Enable the cloud scheduler (experimental).
Enable: false
diff --git a/sdk/go/arvadosclient/arvadosclient_test.go b/sdk/go/arvadosclient/arvadosclient_test.go
index 372f09d14..ecd9e2c6e 100644
--- a/sdk/go/arvadosclient/arvadosclient_test.go
+++ b/sdk/go/arvadosclient/arvadosclient_test.go
@@ -10,7 +10,6 @@ import (
"net/http"
"os"
"testing"
- "time"
"git.curoverse.com/arvados.git/sdk/go/arvadostest"
. "gopkg.in/check.v1"
@@ -114,55 +113,6 @@ func (s *ServerRequiredSuite) TestInvalidResourceType(c *C) {
c.Assert(len(getback), Equals, 0)
}
-func (s *ServerRequiredSuite) TestCreatePipelineTemplate(c *C) {
- arv, err := MakeArvadosClient()
-
- for _, idleConnections := range []bool{
- false,
- true,
- } {
- if idleConnections {
- arv.lastClosedIdlesAt = time.Now().Add(-time.Minute)
- } else {
- arv.lastClosedIdlesAt = time.Now()
- }
-
- getback := make(Dict)
- err = arv.Create("pipeline_templates",
- Dict{"pipeline_template": Dict{
- "name": "tmp",
- "components": Dict{
- "c1": map[string]string{"script": "script1"},
- "c2": map[string]string{"script": "script2"}}}},
- &getback)
- c.Assert(err, Equals, nil)
- c.Assert(getback["name"], Equals, "tmp")
- c.Assert(getback["components"].(map[string]interface{})["c2"].(map[string]interface{})["script"], Equals, "script2")
-
- uuid := getback["uuid"].(string)
-
- getback = make(Dict)
- err = arv.Get("pipeline_templates", uuid, nil, &getback)
- c.Assert(err, Equals, nil)
- c.Assert(getback["name"], Equals, "tmp")
- c.Assert(getback["components"].(map[string]interface{})["c1"].(map[string]interface{})["script"], Equals, "script1")
-
- getback = make(Dict)
- err = arv.Update("pipeline_templates", uuid,
- Dict{
- "pipeline_template": Dict{"name": "tmp2"}},
- &getback)
- c.Assert(err, Equals, nil)
- c.Assert(getback["name"], Equals, "tmp2")
-
- c.Assert(getback["uuid"].(string), Equals, uuid)
- getback = make(Dict)
- err = arv.Delete("pipeline_templates", uuid, nil, &getback)
- c.Assert(err, Equals, nil)
- c.Assert(getback["name"], Equals, "tmp2")
- }
-}
-
func (s *ServerRequiredSuite) TestErrorResponse(c *C) {
arv, _ := MakeArvadosClient()
diff --git a/sdk/python/arvados/commands/run.py b/sdk/python/arvados/commands/run.py
index b17ed2918..f0d518d4e 100644
--- a/sdk/python/arvados/commands/run.py
+++ b/sdk/python/arvados/commands/run.py
@@ -39,34 +39,6 @@ import arvados.config as config
from arvados._version import __version__
-logger = logging.getLogger('arvados.arv-run')
-logger.setLevel(logging.INFO)
-
-arvrun_parser = argparse.ArgumentParser(parents=[arv_cmd.retry_opt])
-arvrun_parser.add_argument('--dry-run', action="store_true",
- help="Print out the pipeline that would be submitted and exit")
-arvrun_parser.add_argument('--local', action="store_true",
- help="Run locally using arv-run-pipeline-instance")
-arvrun_parser.add_argument('--docker-image', type=str,
- help="Docker image to use, otherwise use instance default.")
-arvrun_parser.add_argument('--ignore-rcode', action="store_true",
- help="Commands that return non-zero return codes should not be considered failed.")
-arvrun_parser.add_argument('--no-reuse', action="store_true",
- help="Do not reuse past jobs.")
-arvrun_parser.add_argument('--no-wait', action="store_true",
- help="Do not wait and display logs after submitting command, just exit.")
-arvrun_parser.add_argument('--project-uuid', type=str,
- help="Parent project of the pipeline")
-arvrun_parser.add_argument('--git-dir', type=str, default="",
- help="Git repository passed to arv-crunch-job when using --local")
-arvrun_parser.add_argument('--repository', type=str, default="arvados",
- help="repository field of component, default 'arvados'")
-arvrun_parser.add_argument('--script-version', type=str, default="master",
- help="script_version field of component, default 'master'")
-arvrun_parser.add_argument('--version', action='version',
- version="%s %s" % (sys.argv[0], __version__),
- help='Print version and exit.')
-arvrun_parser.add_argument('args', nargs=argparse.REMAINDER)
class ArvFile(object):
def __init__(self, prefix, fn):
@@ -270,164 +242,7 @@ def uploadfiles(files, api, dry_run=False, num_retries=0,
def main(arguments=None):
- args = arvrun_parser.parse_args(arguments)
-
- if len(args.args) == 0:
- arvrun_parser.print_help()
- return
-
- starting_args = args.args
-
- reading_into = 2
-
- # Parse the command arguments into 'slots'.
- # All words following '>' are output arguments and are collected into slots[0].
- # All words following '<' are input arguments and are collected into slots[1].
- # slots[2..] store the parameters of each command in the pipeline.
- #
- # e.g. arv-run foo arg1 arg2 '|' bar arg3 arg4 '<' input1 input2 input3 '>' output.txt
- # will be parsed into:
- # [['output.txt'],
- # ['input1', 'input2', 'input3'],
- # ['foo', 'arg1', 'arg2'],
- # ['bar', 'arg3', 'arg4']]
- slots = [[], [], []]
- for c in args.args:
- if c.startswith('>'):
- reading_into = 0
- if len(c) > 1:
- slots[reading_into].append(c[1:])
- elif c.startswith('<'):
- reading_into = 1
- if len(c) > 1:
- slots[reading_into].append(c[1:])
- elif c == '|':
- reading_into = len(slots)
- slots.append([])
- else:
- slots[reading_into].append(c)
-
- if slots[0] and len(slots[0]) > 1:
- logger.error("Can only specify a single stdout file (run-command substitutions are permitted)")
- return
-
- if not args.dry_run:
- api = arvados.api('v1')
- if args.project_uuid:
- project = args.project_uuid
- else:
- project = determine_project(os.getcwd(), api.users().current().execute()["uuid"])
-
- # Identify input files. Look at each parameter and test to see if there is
- # a file by that name. This uses 'patterns' to look for within
- # command line arguments, such as --foo=file.txt or -lfile.txt
- patterns = [re.compile("([^=]+=)(.*)"),
- re.compile("(-[A-Za-z])(.+)")]
- for j, command in enumerate(slots[1:]):
- for i, a in enumerate(command):
- if j > 0 and i == 0:
- # j == 0 is stdin, j > 0 is commands
- # always skip program executable (i == 0) in commands
- pass
- elif a.startswith('\\'):
- # if it starts with a \ then don't do any interpretation
- command[i] = a[1:]
- else:
- # See if it looks like a file
- command[i] = statfile('', a)
-
- # If a file named command[i] was found, it would now be an
- # ArvFile or UploadFile. If command[i] is a basestring, that
- # means it doesn't correspond exactly to a file, so do some
- # pattern matching.
- if isinstance(command[i], basestring):
- for p in patterns:
- m = p.match(a)
- if m:
- command[i] = statfile(m.group(1), m.group(2))
- break
-
- files = [c for command in slots[1:] for c in command if isinstance(c, UploadFile)]
- if files:
- uploadfiles(files, api, dry_run=args.dry_run, num_retries=args.retries, project=project)
-
- for i in range(1, len(slots)):
- slots[i] = [("%s%s" % (c.prefix, c.fn)) if isinstance(c, ArvFile) else c for c in slots[i]]
-
- component = {
- "script": "run-command",
- "script_version": args.script_version,
- "repository": args.repository,
- "script_parameters": {
- },
- "runtime_constraints": {}
- }
-
- if args.docker_image:
- component["runtime_constraints"]["docker_image"] = args.docker_image
-
- task_foreach = []
- group_parser = argparse.ArgumentParser()
- group_parser.add_argument('-b', '--batch-size', type=int)
- group_parser.add_argument('args', nargs=argparse.REMAINDER)
-
- for s in range(2, len(slots)):
- for i in range(0, len(slots[s])):
- if slots[s][i] == '--':
- inp = "input%i" % (s-2)
- groupargs = group_parser.parse_args(slots[2][i+1:])
- if groupargs.batch_size:
- component["script_parameters"][inp] = {"value": {"batch":groupargs.args, "size":groupargs.batch_size}}
- slots[s] = slots[s][0:i] + [{"foreach": inp, "command": "$(%s)" % inp}]
- else:
- component["script_parameters"][inp] = groupargs.args
- slots[s] = slots[s][0:i] + ["$(%s)" % inp]
- task_foreach.append(inp)
- break
- if slots[s][i] == '\--':
- slots[s][i] = '--'
-
- if slots[0]:
- component["script_parameters"]["task.stdout"] = slots[0][0]
- if slots[1]:
- task_foreach.append("stdin")
- component["script_parameters"]["stdin"] = slots[1]
- component["script_parameters"]["task.stdin"] = "$(stdin)"
-
- if task_foreach:
- component["script_parameters"]["task.foreach"] = task_foreach
-
- component["script_parameters"]["command"] = slots[2:]
- if args.ignore_rcode:
- component["script_parameters"]["task.ignore_rcode"] = args.ignore_rcode
-
- pipeline = {
- "name": "arv-run " + " | ".join([s[0] for s in slots[2:]]),
- "description": "@" + " ".join(starting_args) + "@",
- "components": {
- "command": component
- },
- "state": "RunningOnClient" if args.local else "RunningOnServer"
- }
-
- if args.dry_run:
- print(json.dumps(pipeline, indent=4))
- else:
- pipeline["owner_uuid"] = project
- pi = api.pipeline_instances().create(body=pipeline, ensure_unique_name=True).execute()
- logger.info("Running pipeline %s", pi["uuid"])
-
- if args.local:
- subprocess.call(["arv-run-pipeline-instance", "--instance", pi["uuid"], "--run-jobs-here"] + (["--no-reuse"] if args.no_reuse else []))
- elif not args.no_wait:
- ws.main(["--pipeline", pi["uuid"]])
-
- pi = api.pipeline_instances().get(uuid=pi["uuid"]).execute()
- logger.info("Pipeline is %s", pi["state"])
- if "output_uuid" in pi["components"]["command"]:
- logger.info("Output is %s", pi["components"]["command"]["output_uuid"])
- else:
- logger.info("No output")
+ raise Exception("Legacy arv-run removed.")
if __name__ == '__main__':
main()
diff --git a/sdk/python/bin/arv-run b/sdk/python/bin/arv-run
deleted file mode 100755
index ebba20182..000000000
--- a/sdk/python/bin/arv-run
+++ /dev/null
@@ -1,7 +0,0 @@
-#!/usr/bin/env python
-# Copyright (C) The Arvados Authors. All rights reserved.
-#
-# SPDX-License-Identifier: Apache-2.0
-
-from arvados.commands.run import main
-main()
diff --git a/sdk/python/setup.py b/sdk/python/setup.py
index 0fc2dde31..5130d187d 100644
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -40,7 +40,6 @@ setup(name='arvados-python-client',
'bin/arv-federation-migrate',
'bin/arv-normalize',
'bin/arv-put',
- 'bin/arv-run',
'bin/arv-ws'
],
data_files=[
diff --git a/sdk/python/tests/test_arv_run.py b/sdk/python/tests/test_arv_run.py
deleted file mode 100644
index 1afc1208b..000000000
--- a/sdk/python/tests/test_arv_run.py
+++ /dev/null
@@ -1,28 +0,0 @@
-# Copyright (C) The Arvados Authors. All rights reserved.
-#
-# SPDX-License-Identifier: Apache-2.0
-
-from __future__ import absolute_import
-import os
-import sys
-import tempfile
-import unittest
-
-import arvados.commands.run as arv_run
-from . import arvados_testutil as tutil
-
-class ArvRunTestCase(unittest.TestCase, tutil.VersionChecker):
- def run_arv_run(self, args):
- sys.argv = ['arv-run'] + args
- return arv_run.main()
-
- def test_unsupported_arg(self):
- with self.assertRaises(SystemExit):
- self.run_arv_run(['-x=unknown'])
-
- def test_version_argument(self):
- with tutil.redirected_streams(
- stdout=tutil.StringIO, stderr=tutil.StringIO) as (out, err):
- with self.assertRaises(SystemExit):
- self.run_arv_run(['--version'])
- self.assertVersionOutput(out, err)
diff --git a/sdk/python/tests/test_pipeline_template.py b/sdk/python/tests/test_pipeline_template.py
deleted file mode 100644
index 88138f3ed..000000000
--- a/sdk/python/tests/test_pipeline_template.py
+++ /dev/null
@@ -1,62 +0,0 @@
-# Copyright (C) The Arvados Authors. All rights reserved.
-#
-# SPDX-License-Identifier: Apache-2.0
-
-from __future__ import absolute_import
-# usage example:
-#
-# ARVADOS_API_TOKEN=abc ARVADOS_API_HOST=arvados.local python -m unittest discover
-
-import unittest
-import arvados
-import apiclient
-from . import run_test_server
-
-class PipelineTemplateTest(run_test_server.TestCaseWithServers):
- MAIN_SERVER = {}
- KEEP_SERVER = {}
-
- def runTest(self):
- run_test_server.authorize_with("admin")
- pt_uuid = arvados.api('v1').pipeline_templates().create(
- body={'name':__file__}
- ).execute()['uuid']
- self.assertEqual(len(pt_uuid), 27,
- 'Unexpected format of pipeline template UUID ("%s")'
- % pt_uuid)
- components = {
- 'x': 'x',
- '-x-': [1,2,{'foo':'bar'}],
- 'Boggis': {'Bunce': '[\'Bean\']'},
- 'SpassBox': True,
- 'spass_box': False,
- 'spass-box': [True, 'Maybe', False]
- }
- update_response = arvados.api('v1').pipeline_templates().update(
- uuid=pt_uuid,
- body={'components':components}
- ).execute()
- self.assertEqual('uuid' in update_response, True,
- 'update() response did not include a uuid')
- self.assertEqual(update_response['uuid'], pt_uuid,
- 'update() response has a different uuid (%s, not %s)'
- % (update_response['uuid'], pt_uuid))
- self.assertEqual(update_response['name'], __file__,
- 'update() response has a different name (%s, not %s)'
- % (update_response['name'], __file__))
- get_response = arvados.api('v1').pipeline_templates().get(
- uuid=pt_uuid
- ).execute()
- self.assertEqual(get_response['components'], components,
- 'components got munged by server (%s -> %s)'
- % (components, update_response['components']))
- delete_response = arvados.api('v1').pipeline_templates().delete(
- uuid=pt_uuid
- ).execute()
- self.assertEqual(delete_response['uuid'], pt_uuid,
- 'delete() response has wrong uuid (%s, not %s)'
- % (delete_response['uuid'], pt_uuid))
- with self.assertRaises(apiclient.errors.HttpError):
- geterror_response = arvados.api('v1').pipeline_templates().get(
- uuid=pt_uuid
- ).execute()
diff --git a/sdk/python/tests/test_retry_job_helpers.py b/sdk/python/tests/test_retry_job_helpers.py
index b9c87a68f..76c62cb0c 100644
--- a/sdk/python/tests/test_retry_job_helpers.py
+++ b/sdk/python/tests/test_retry_job_helpers.py
@@ -76,34 +76,3 @@ class CurrentJobTestCase(ApiClientRetryTestMixin, unittest.TestCase):
def run_method(self):
arvados.current_job()
-
-
-class CurrentTaskTestCase(ApiClientRetryTestMixin, unittest.TestCase):
-
- DEFAULT_EXCEPTION = arvados.errors.ApiError
-
- def setUp(self):
- super(CurrentTaskTestCase, self).setUp()
- os.environ['TASK_UUID'] = 'zzzzz-zzzzz-zzzzzzzzzzzzzzz'
- os.environ['TASK_WORK'] = '.'
-
- def tearDown(self):
- del os.environ['TASK_UUID']
- del os.environ['TASK_WORK']
- arvados._current_task = None
- super(CurrentTaskTestCase, self).tearDown()
-
- def run_method(self):
- arvados.current_task()
-
-
-class TaskSetOutputTestCase(CurrentTaskTestCase, unittest.TestCase):
-
- DEFAULT_EXCEPTION = arvados.errors.ApiError
-
- def tearDown(self):
- super(TaskSetOutputTestCase, self).tearDown()
- run_test_server.reset()
-
- def run_method(self, locator=ApiClientRetryTestMixin.TEST_LOCATOR):
- arvados.task_set_output({'uuid':self.TEST_UUID},s=locator)
diff --git a/services/api/test/unit/job_test.rb b/services/api/test/unit/job_test.rb
index c89c64c8c..5ece9ab49 100644
--- a/services/api/test/unit/job_test.rb
+++ b/services/api/test/unit/job_test.rb
@@ -639,7 +639,7 @@ class JobTest < ActiveSupport::TestCase
test 'enable legacy api configuration option = true' do
Rails.configuration.Containers.JobsAPI.Enable = "true"
check_enable_legacy_jobs_api
- assert_equal({}, Rails.configuration.API.DisabledAPIs)
+ assert_equal(Disable_update_jobs_api_method_list, Rails.configuration.API.DisabledAPIs)
end
test 'enable legacy api configuration option = false' do
@@ -652,7 +652,7 @@ class JobTest < ActiveSupport::TestCase
Rails.configuration.Containers.JobsAPI.Enable = "auto"
assert Job.count > 0
check_enable_legacy_jobs_api
- assert_equal({}, Rails.configuration.API.DisabledAPIs)
+ assert_equal(Disable_update_jobs_api_method_list, Rails.configuration.API.DisabledAPIs)
end
test 'enable legacy api configuration option = auto, no jobs' do
commit 448f667a574b50da096051a0d062b9059ab3609f
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Thu Aug 8 15:07:35 2019 -0400
15133: Delete legacy crunch v1 docs
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/doc/_config.yml b/doc/_config.yml
index 6b572f7fc..0547c8ee9 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -67,18 +67,6 @@ navbar:
- user/copying/agpl-3.0.html
- user/copying/LICENSE-2.0.html
- user/copying/by-sa-3.0.html
- - Obsolete documentation:
- - user/topics/running-pipeline-command-line.html.textile.liquid
- - user/topics/arv-run.html.textile.liquid
- - user/tutorials/running-external-program.html.textile.liquid
- - user/topics/crunch-tools-overview.html.textile.liquid
- - user/tutorials/tutorial-firstscript.html.textile.liquid
- - user/tutorials/tutorial-submit-job.html.textile.liquid
- - user/topics/tutorial-parallel.html.textile.liquid
- - user/topics/run-command.html.textile.liquid
- - user/reference/job-pipeline-ref.html.textile.liquid
- - user/examples/crunch-examples.html.textile.liquid
- - user/topics/tutorial-trait-search.html.textile.liquid
sdk:
- Overview:
- sdk/index.html.textile.liquid
@@ -89,7 +77,6 @@ navbar:
- sdk/python/arvados-fuse.html.textile.liquid
- sdk/python/events.html.textile.liquid
- sdk/python/cookbook.html.textile.liquid
- - sdk/python/crunch-utility-libraries.html.textile.liquid
- CLI:
- sdk/cli/install.html.textile.liquid
- sdk/cli/index.html.textile.liquid
@@ -141,13 +128,13 @@ navbar:
- api/methods/container_requests.html.textile.liquid
- api/methods/containers.html.textile.liquid
- api/methods/workflows.html.textile.liquid
- - Jobs engine (deprecated):
+ - Jobs engine (legacy):
- api/crunch-scripts.html.textile.liquid
- api/methods/jobs.html.textile.liquid
- api/methods/job_tasks.html.textile.liquid
- api/methods/pipeline_instances.html.textile.liquid
- api/methods/pipeline_templates.html.textile.liquid
- - Metadata for bioinformatics:
+ - Metadata for bioinformatics (deprecated):
- api/methods/humans.html.textile.liquid
- api/methods/specimens.html.textile.liquid
- api/methods/traits.html.textile.liquid
@@ -228,8 +215,5 @@ navbar:
- install/crunch2-slurm/install-test.html.textile.liquid
- install/install-nodemanager.html.textile.liquid
- install/install-compute-ping.html.textile.liquid
- - Containers API support on cloud (experimental):
+ - Containers API support on cloud (beta):
- install/install-dispatch-cloud.html.textile.liquid
- - Jobs API support (deprecated):
- - install/install-crunch-dispatch.html.textile.liquid
- - install/install-compute-node.html.textile.liquid
diff --git a/doc/api/crunch-scripts.html.textile.liquid b/doc/api/crunch-scripts.html.textile.liquid
index 3df1db407..86a22ed6c 100644
--- a/doc/api/crunch-scripts.html.textile.liquid
+++ b/doc/api/crunch-scripts.html.textile.liquid
@@ -11,6 +11,8 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+p=. *Legacy. The job APIs are read-only and disabled by default in new installations. Use "container requests":container_requests.html.textile.liquid .*
+
h2. Crunch scripts
A crunch script is responsible for completing a single JobTask. In doing so, it will:
diff --git a/doc/api/methods/humans.html.textile.liquid b/doc/api/methods/humans.html.textile.liquid
index 84cb22c99..e08e941cf 100644
--- a/doc/api/methods/humans.html.textile.liquid
+++ b/doc/api/methods/humans.html.textile.liquid
@@ -11,6 +11,8 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+p=. *Deprecated, likely to be removed in a future version. The recommended way to store metadata is "collection properties":collections.html*
+
API endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/humans@
Object type: @7a9it@
diff --git a/doc/api/methods/job_tasks.html.textile.liquid b/doc/api/methods/job_tasks.html.textile.liquid
index deee3a580..69c3f07e3 100644
--- a/doc/api/methods/job_tasks.html.textile.liquid
+++ b/doc/api/methods/job_tasks.html.textile.liquid
@@ -11,6 +11,8 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+p=. *Legacy. This endpoint is read-only and disabled by default in new installations.*
+
API endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/job_tasks@
Object type: @ot0gb@
diff --git a/doc/api/methods/jobs.html.textile.liquid b/doc/api/methods/jobs.html.textile.liquid
index 2f061866f..8b06c989a 100644
--- a/doc/api/methods/jobs.html.textile.liquid
+++ b/doc/api/methods/jobs.html.textile.liquid
@@ -11,6 +11,8 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+p=. *Legacy. This endpoint is read-only and disabled by default in new installations.*
+
API endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/jobs@
Object type: @8i9sb@
@@ -19,8 +21,6 @@ Example UUID: @zzzzz-8i9sb-0123456789abcde@
h2. Resource
-Deprecated.
-
A job describes a work order to be executed by the Arvados cluster.
Each job has, in addition to the "Common resource fields":{{site.baseurl}}/api/resources.html:
diff --git a/doc/api/methods/pipeline_instances.html.textile.liquid b/doc/api/methods/pipeline_instances.html.textile.liquid
index 09fd4fe48..56c071ef9 100644
--- a/doc/api/methods/pipeline_instances.html.textile.liquid
+++ b/doc/api/methods/pipeline_instances.html.textile.liquid
@@ -11,6 +11,8 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+p=. *Legacy. This endpoint is read-only and disabled by default in new installations.*
+
API endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/pipeline_instances@
Object type: @d1hrv@
diff --git a/doc/api/methods/pipeline_templates.html.textile.liquid b/doc/api/methods/pipeline_templates.html.textile.liquid
index 85df2799e..40297aa05 100644
--- a/doc/api/methods/pipeline_templates.html.textile.liquid
+++ b/doc/api/methods/pipeline_templates.html.textile.liquid
@@ -11,6 +11,8 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+p=. *Legacy. This endpoint is read-only and disabled by default in new installations.*
+
API endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/pipeline_templates@
Object type: @p5p6p@
diff --git a/doc/api/methods/specimens.html.textile.liquid b/doc/api/methods/specimens.html.textile.liquid
index 6ee79ca3d..be3712a20 100644
--- a/doc/api/methods/specimens.html.textile.liquid
+++ b/doc/api/methods/specimens.html.textile.liquid
@@ -10,6 +10,8 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+p=. *Deprecated, likely to be removed in a future version. The recommended way to store metadata is "collection properties":collections.html*
+
API endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/specimens@
Object type: @j58dm@
diff --git a/doc/api/methods/traits.html.textile.liquid b/doc/api/methods/traits.html.textile.liquid
index 34b60cfe7..e48804702 100644
--- a/doc/api/methods/traits.html.textile.liquid
+++ b/doc/api/methods/traits.html.textile.liquid
@@ -11,6 +11,8 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+p=. *Deprecated, likely to be removed in a future version. The recommended way to store metadata is "collection properties":collections.html*
+
API endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/traits@
Object type: @q1cn2@
diff --git a/doc/install/install-compute-node.html.textile.liquid b/doc/install/install-compute-node.html.textile.liquid
deleted file mode 100644
index 02eb2168b..000000000
--- a/doc/install/install-compute-node.html.textile.liquid
+++ /dev/null
@@ -1,112 +0,0 @@
----
-layout: default
-navsection: installguide
-title: Install a compute node
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-h2. Install dependencies
-
-First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
-
-{% include 'note_python_sc' %}
-
-On Red Hat-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">echo 'exclude=python2-llfuse' | sudo tee -a /etc/yum.conf</span>
-~$ <span class="userinput">sudo yum install perl python-virtualenv fuse python-arvados-python-client python-arvados-fuse crunchrunner crunchstat arvados-docker-cleaner iptables ca-certificates</span>
-</code></pre>
-</notextile>
-
-On Debian-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get install perl python-virtualenv fuse python-arvados-python-client python-arvados-fuse crunchrunner crunchstat arvados-docker-cleaner iptables ca-certificates</span>
-</code></pre>
-</notextile>
-
-{% include 'install_compute_docker' %}
-
-h2. Set up SLURM
-
-Install SLURM following "the same process you used to install the Crunch dispatcher":install-crunch-dispatch.html#slurm.
-
-h2. Copy configuration files from the dispatcher (API server)
-
-The @slurm.conf@ and @/etc/munge/munge.key@ files need to be identical across the dispatcher and all compute nodes. Copy the files you created in the "Install the Crunch dispatcher":install-crunch-dispatch.html step to this compute node.
-
-{% include 'install_compute_fuse' %}
-
-{% include 'install_docker_cleaner' %}
-
-h2. Add a Crunch user account
-
-Create a Crunch user account, and add it to the @fuse@ and @docker@ groups so it can use those tools:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo useradd --groups fuse,docker crunch</span>
-</code></pre>
-</notextile>
-
-The crunch user should have the same UID, GID, and home directory across all compute nodes and the dispatcher (API server).
-
-h2. Tell the API server about this compute node
-
-Load your API superuser token on the compute node:
-
-<notextile>
-<pre><code>
-~$ <span class="userinput">HISTIGNORE=$HISTIGNORE:'export ARVADOS_API_TOKEN=*'</span>
-~$ <span class="userinput">export ARVADOS_API_TOKEN=@your-superuser-token@</span>
-~$ <span class="userinput">export ARVADOS_API_HOST=@uuid_prefix.your.domain@</span>
-~$ <span class="userinput">unset ARVADOS_API_HOST_INSECURE</span>
-</code>
-</pre>
-</notextile>
-
-Then execute this script to create a compute node object, and set up a cron job to have the compute node ping the API server every five minutes:
-
-<notextile>
-<pre><code>
-#!/bin/bash
-set -e
-if ! test -f /root/node.json ; then
- python - <<EOF
-import arvados, json, socket
-fqdn = socket.getfqdn()
-hostname, _, domain = fqdn.partition('.')
-node = arvados.api('v1').nodes().create(body={'hostname': hostname, 'domain': domain}).execute()
-with open('/root/node.json', 'w') as node_file:
- json.dump(node, node_file, indent=2)
-EOF
-
- # Make sure /dev/fuse permissions are correct (the device appears after fuse is loaded)
- chmod 1660 /dev/fuse && chgrp fuse /dev/fuse
-fi
-
-UUID=`grep \"uuid\" /root/node.json |cut -f4 -d\"`
-PING_SECRET=`grep \"ping_secret\" /root/node.json |cut -f4 -d\"`
-
-if ! test -f /etc/cron.d/node_ping ; then
- echo "*/5 * * * * root /usr/bin/curl -k -d ping_secret=$PING_SECRET https://$ARVADOS_API_HOST/arvados/v1/nodes/$UUID/ping" > /etc/cron.d/node_ping
-fi
-
-/usr/bin/curl -k -d ping_secret=$PING_SECRET https://$ARVADOS_API_HOST/arvados/v1/nodes/$UUID/ping?ping_secret=$PING_SECRET
-</code>
-</pre>
-</notextile>
-
-And remove your token from the environment:
-
-<notextile>
-<pre><code>
-~$ <span class="userinput">unset ARVADOS_API_TOKEN</span>
-~$ <span class="userinput">unset ARVADOS_API_HOST</span>
-</code>
-</pre>
-</notextile>
diff --git a/doc/install/install-crunch-dispatch.html.textile.liquid b/doc/install/install-crunch-dispatch.html.textile.liquid
deleted file mode 100644
index e0ed14769..000000000
--- a/doc/install/install-crunch-dispatch.html.textile.liquid
+++ /dev/null
@@ -1,207 +0,0 @@
----
-layout: default
-navsection: installguide
-title: Install the Crunch dispatcher
-
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-The dispatcher normally runs on the same host/VM as the API server.
-
-h2. Test the Arvados job queue
-
-Crunch dispatches work from the job queue on the Arvados API server. Before you start installing the Crunch dispatcher, now's a good time to check that the API server and Git server can coordinate to create job records. Run these commands *on your shell server* to create a collection, and a job to calculate the MD5 checksum of every file in it:
-
-<notextile>
-<pre><code>~$ <span class="userinput">echo 'Hello, Crunch!' | arv-put --portable-data-hash -</span>
-…
-d40c7f35d80da669afb9db1896e760ad+49
-~$ <span class="userinput">read -rd $'\000' newjob <<EOF; arv job create --job "$newjob"
-{"script_parameters":{"input":"d40c7f35d80da669afb9db1896e760ad+49"},
- "script_version":"0988acb472849dc0",
- "script":"hash",
- "repository":"arvados"}
-EOF</span>
-</code></pre>
-</notextile>
-
-If you get the error
-
-<pre>
-ArgumentError: Specified script_version does not resolve to a commit
-</pre>
-
-it often means that the API server can't read the specified repository—either because it doesn't exist, or because the user running the API server doesn't have permission to read the repository files. Check the API server's log (@/var/www/arvados-api/current/log/production.log@) for details, and double-check the instructions in the "Git server installation guide":install-arv-git-httpd.html.
-
-If everything goes well, the API server should create a job record, and your @arv@ command will output the JSON for that record. It should have state @Queued@ and script_version @0988acb472849dc08d576ee40493e70bde2132ca at . If the job JSON includes those fields, you can proceed to install the Crunch dispatcher and a compute node. This job will remain queued until you install those services.
-
-h2. Perl SDK dependencies
-
-Install the Perl SDK on the controller.
-
-* See "Perl SDK":{{site.baseurl}}/sdk/perl/index.html page for details.
-
-h2. Python SDK dependencies
-
-Install the Python SDK and CLI tools on controller and all compute nodes.
-
-* See "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html page for details.
-
-h2(#slurm). Set up SLURM
-
-On the API server, install SLURM and munge, and generate a munge key.
-
-On Debian-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo /usr/bin/apt-get install slurm-llnl munge</span>
-~$ <span class="userinput">sudo /usr/sbin/create-munge-key</span>
-</code></pre>
-</notextile>
-
-On Red Hat-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo yum install slurm munge slurm-munge</span>
-</code></pre>
-</notextile>
-
-Now we need to give SLURM a configuration file. On Debian-based systems, this is installed at @/etc/slurm-llnl/slurm.conf at . On Red Hat-based systems, this is installed at @/etc/slurm/slurm.conf at . Here's an example @slurm.conf@:
-
-<notextile>
-<pre>
-ControlMachine=uuid_prefix.your.domain
-SlurmctldPort=6817
-SlurmdPort=6818
-AuthType=auth/munge
-StateSaveLocation=/tmp
-SlurmdSpoolDir=/tmp/slurmd
-SwitchType=switch/none
-MpiDefault=none
-SlurmctldPidFile=/var/run/slurmctld.pid
-SlurmdPidFile=/var/run/slurmd.pid
-ProctrackType=proctrack/pgid
-CacheGroups=0
-ReturnToService=2
-TaskPlugin=task/affinity
-#
-# TIMERS
-SlurmctldTimeout=300
-SlurmdTimeout=300
-InactiveLimit=0
-MinJobAge=300
-KillWait=30
-Waittime=0
-#
-# SCHEDULING
-SchedulerType=sched/backfill
-SchedulerPort=7321
-SelectType=select/linear
-FastSchedule=0
-#
-# LOGGING
-SlurmctldDebug=3
-#SlurmctldLogFile=
-SlurmdDebug=3
-#SlurmdLogFile=
-JobCompType=jobcomp/none
-#JobCompLoc=
-JobAcctGatherType=jobacct_gather/none
-#
-# COMPUTE NODES
-NodeName=DEFAULT
-PartitionName=DEFAULT MaxTime=INFINITE State=UP
-
-NodeName=compute[0-255]
-PartitionName=compute Nodes=compute[0-255] Default=YES Shared=YES
-</pre>
-</notextile>
-
-h3. SLURM configuration essentials
-
-Whenever you change this file, you will need to update the copy _on every compute node_ as well as the controller node, and then run @sudo scontrol reconfigure at .
-
-*@ControlMachine@* should be a DNS name that resolves to the SLURM controller (dispatch/API server). This must resolve correctly on all SLURM worker nodes as well as the controller itself. In general SLURM is very sensitive about all of the nodes being able to communicate with the controller _and one another_, all using the same DNS names.
-
-*@NodeName=compute[0-255]@* establishes that the hostnames of the worker nodes will be compute0, compute1, etc. through compute255.
-* There are several ways to compress sequences of names, like @compute[0-9,80,100-110]@. See the "hostlist" discussion in the @slurm.conf(5)@ and @scontrol(1)@ man pages for more information.
-* It is not necessary for all of the nodes listed here to be alive in order for SLURM to work, although you should make sure the DNS entries exist. It is easiest to define lots of hostnames up front, assigning them to real nodes and updating your DNS records as the nodes appear. This minimizes the frequency of @slurm.conf@ updates and use of @scontrol reconfigure at .
-
-Each hostname in @slurm.conf@ must also resolve correctly on all SLURM worker nodes as well as the controller itself. Furthermore, the hostnames used in the configuration file must match the hostnames reported by @hostname@ or @hostname -s@ on the nodes themselves. This applies to the ControlMachine as well as the worker nodes.
-
-For example:
-* In @slurm.conf@ on control and worker nodes: @ControlMachine=uuid_prefix.your.domain@
-* In @slurm.conf@ on control and worker nodes: @NodeName=compute[0-255]@
-* In @/etc/resolv.conf@ on control and worker nodes: @search uuid_prefix.your.domain@
-* On the control node: @hostname@ reports @uuid_prefix.your.domain@
-* On worker node 123: @hostname@ reports @compute123.uuid_prefix.your.domain@
-
-h3. Automatic hostname assignment
-
-If your worker node bootstrapping script (see "Installing a compute node":install-compute-node.html) does not send the worker's current hostname, the API server will choose an unused hostname from the set given in @application.yml@, which defaults to @compute[0-255]@.
-
-If it is not feasible to give your compute nodes hostnames like compute0, compute1, etc., you can accommodate other naming schemes with a bit of extra configuration.
-
-If you want Arvados to assign names to your nodes with a different consecutive numeric series like @{worker1-0000, worker1-0001, worker1-0002}@, add an entry to @application.yml@; see @/var/www/arvados-api/current/config/application.default.yml@ for details. Example:
-* In @application.yml@: <code>assign_node_hostname: worker1-%<slot_number>04d</code>
-* In @slurm.conf@: <code>NodeName=worker1-[0000-0255]</code>
-
-If your worker hostnames are already assigned by other means, and the full set of names is known in advance, have your worker node bootstrapping script (see "Installing a compute node":install-compute-node.html) send its current hostname, rather than expect Arvados to assign one.
-* In @application.yml@: <code>assign_node_hostname: false</code>
-* In @slurm.conf@: <code>NodeName=alice,bob,clay,darlene</code>
-
-If your worker hostnames are already assigned by other means, but the full set of names is _not_ known in advance, you can use the @slurm.conf@ and @application.yml@ settings in the previous example, but you must also update @slurm.conf@ (both on the controller and on all worker nodes) and run @sudo scontrol reconfigure@ whenever a new node comes online.
-
-h2. Enable SLURM job dispatch
-
-In your API server's @application.yml@ configuration file, add the line @crunch_job_wrapper: :slurm_immediate@ under the appropriate section. (The second colon is not a typo. It denotes a Ruby symbol.)
-
-h2. Crunch user account
-
-Run @sudo adduser crunch at . The crunch user should have the same UID, GID, and home directory on all compute nodes and on the dispatcher (API server).
-
-h2. Run the Crunch dispatcher service
-
-To dispatch Arvados jobs:
-
-* The API server script @crunch-dispatch.rb@ must be running.
-* @crunch-job@ needs the installation path of the Perl SDK in its @PERLLIB at .
-* @crunch-job@ needs the @ARVADOS_API_HOST@ (and, if necessary, @ARVADOS_API_HOST_INSECURE@) environment variable set.
-
-Install runit to monitor the Crunch dispatch daemon. {% include 'install_runit' %}
-
-Install the script below as the run script for the Crunch dispatch service, modifying it as directed by the comments.
-
-<notextile>
-<pre><code>#!/bin/sh
-set -e
-
-rvmexec=""
-## Uncomment this line if you use RVM:
-#rvmexec="/usr/local/rvm/bin/rvm-exec default"
-
-export ARVADOS_API_HOST=<span class="userinput">uuid_prefix.your.domain</span>
-export CRUNCH_DISPATCH_LOCKFILE=/var/lock/crunch-dispatch
-export HOME=$(pwd)
-export RAILS_ENV=production
-
-## Uncomment and edit this line if your compute nodes have cgroup info
-## somewhere other than /sys/fs/cgroup (e.g., "/cgroup" for CentOS 7)
-#export CRUNCH_CGROUP_ROOT="/sys/fs/cgroup"
-
-## Uncomment this line if your cluster uses self-signed SSL certificates:
-#export ARVADOS_API_HOST_INSECURE=yes
-
-# This is the path to docker on your compute nodes. You might need to
-# change it to "docker", "/opt/bin/docker", etc.
-export CRUNCH_JOB_DOCKER_BIN=<span class="userinput">docker.io</span>
-
-fuser -TERM -k $CRUNCH_DISPATCH_LOCKFILE || true
-cd /var/www/arvados-api/current
-exec $rvmexec bundle exec ./script/crunch-dispatch.rb 2>&1
-</code></pre>
-</notextile>
diff --git a/doc/sdk/python/crunch-utility-libraries.html.textile.liquid b/doc/sdk/python/crunch-utility-libraries.html.textile.liquid
deleted file mode 100644
index 3e2631512..000000000
--- a/doc/sdk/python/crunch-utility-libraries.html.textile.liquid
+++ /dev/null
@@ -1,228 +0,0 @@
----
-layout: default
-navsection: sdk
-navmenu: Python
-title: "Crunch utility libraries"
-
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'pipeline_deprecation_notice' %}
-
-Several utility libraries are included with Arvados. They are intended to make it quicker and easier to write your own crunch scripts.
-
-* "Python SDK extras":#pythonsdk
-* "Toolkit wrappers":#toolkit_wrappers
-
-h2(#pythonsdk). Python SDK extras
-
-The Python SDK adds some convenience features that are particularly useful in crunch scripts, in addition to the standard set of API calls.
-
-In a crunch job, the environment variables @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ will be set up so the job has the privileges of the user who submitted the job.
-
-<pre>
-import arvados
-
-my_user = arvados.api().users().current().execute()
-my_uuid = my_user['uuid']
-</pre>
-
-h3. Get the current job and task parameters
-
- at arvados.current_job()@ and @arvados.current_task()@ are convenient ways to retrieve the current Job and Task, using the @JOB_UUID@ and @TASK_UUID@ environment variables provided to each crunch task process.
-
-<pre>
-this_job = arvados.current_job()
-this_task = arvados.current_task()
-this_job_input = this_job['script_parameters']['input']
-this_task_input = this_task['parameters']['input']
-</pre>
-
-h3(#one_task_per_input). Queue a task for each input file
-
-A common pattern for a crunch job is to run one task to scan the input, and one task per input file to do the work.
-
-The @one_task_per_input_file()@ function implements this pattern. Pseudocode:
-
-<pre>
-if this is the job's first (default) task:
- for each file in the 'input' collection:
- queue a new task, with parameters['input'] = file
- exit
-else:
- return
-</pre>
-
-Usage:
-
-<pre>
-import arvados
-arvados.job_setup.one_task_per_input_file(if_sequence=0, and_end_task=True)
-
-# Now do the work on a single file
-my_input = this_task['parameters']['input']
-</pre>
-
-h3. Set the current task's output and success flag
-
-Each task in a crunch job must make an API call to record its output and set its @success@ attribute to True. The object returned by @current_task()@ has a @set_output()@ method to make the process more succinct.
-
-<pre>
-arvados.current_task().set_output(my_output_locator)
-</pre>
-
-h3. arvados_ipc.py
-
-Manage child processes and FIFOs (pipes).
-
-
-This module makes it easier to check the exit status of every child process you start, and close the unused end of each FIFO at the appropriate time.
-
-<pre>
-from arvados_ipc import *
-
-children = {}
-pipes = {}
-
-pipe_setup(pipes, 'hellopipe')
-if 0 == named_fork(children, 'child_a'):
- pipe_closeallbut(pipes, ('hellopipe', 'w'))
- os.write(pipes['hellopipe', 'w'], "Hello, parent.")
- os._exit(0)
-
-pipe_closeallbut(pipes, ('hellopipe', 'r'))
-with os.fdopen(pipes['hellopipe', 'r'], 'rb') as f:
- message = f.read()
- sys.stderr.write("Child says: " + message + "\n")
-
-if not waitpid_and_check_children(children):
- raise Exception("Child process exited non-zero.")
-</pre>
-
-The "crunch scripts" included with Arvados include some more examples of using the arvados_ipc module.
-
-h2(#toolkit_wrappers). Toolkit wrappers
-
-The following *arvados-∗.py* modules provide "extract, build, run" helpers to make it easy to incorporate common analysis tools in your crunch scripts.
-
-h3. arvados_bwa.py
-
-Build and run the "bwa":http://bio-bwa.sourceforge.net/bwa.shtml program.
-
-The module retrieves the bwa source code from Keep, using the job's @bwa_tbz@ parameter.
-
-<pre>
-import arvados_bwa
-arvados_bwa.run('aln', [ref_basename, '-'],
- stdin=open(fastq_filename,'rb'),
- stdout=open(aln_filename,'wb'))
-</pre>
-
-On qr1hi.arvadosapi.com, the source distribution @bwa-0.7.5a.tar.bz2@ is available in the collection @8b6e2c4916133e1d859c9e812861ce13+70 at .
-
-<pre>
-{
- "script_parameters":{
- "bwa_tbz":"8b6e2c4916133e1d859c9e812861ce13+70",
- ...
- },
- ...
-}
-</pre>
-
-h3. arvados_gatk2.py
-
-Extract and run the "Genome Analysis Toolkit":http://www.broadinstitute.org/gatk/ programs.
-
-The module retrieves the binary distribution tarball from Keep, using the job's @gatk_tbz@ parameter.
-
-<pre>
-arvados_gatk2.run(
- args=[
- '-nct', 8,
- '-T', 'BaseRecalibrator',
- '-R', ref_fasta_files[0],
- '-I', input_bam_files[0],
- '-o', recal_file,
- ])
-</pre>
-
-On qr1hi.arvadosapi.com, the binary distribution @GenomeAnalysisTK-2.6-4.tar.bz2@ is available in the collection @5790482512cf6d5d6dfd50b7fd61e1d1+86 at .
-
-The GATK data bundle is available in the collection @d237a90bae3870b3b033aea1e99de4a9+10820 at .
-
-<pre>
-{
- "script_parameters":{
- "gatk_tbz":"7e0a277d6d2353678a11f56bab3b13f2+87",
- "gatk_bundle":"d237a90bae3870b3b033aea1e99de4a9+10820",
- ...
- },
- ...
-}
-</pre>
-
-h3. arvados_samtools.py
-
-Build and run the "samtools":http://samtools.sourceforge.net/samtools.shtml program.
-
-
-The module retrieves the samtools source code from Keep, using the job's @samtools_tgz@ parameter.
-
-<pre>
-import arvados_samtools
-arvados_samtools.run('view', ['-S', '-b', '-'],
- stdin=open(sam_filename,'rb'),
- stdout=open(bam_filename,'wb'))
-</pre>
-
-On qr1hi.arvadosapi.com, the source distribution @samtools-0.1.19.tar.gz@ is available in the collection @c777e23cf13e5d5906abfdc08d84bfdb+74 at .
-
-<pre>
-{
- "script_parameters":{
- "samtools_tgz":"c777e23cf13e5d5906abfdc08d84bfdb+74",
- ...
- },
- ...
-}
-</pre>
-
-
-h3. arvados_picard.py
-
-Build and run the "picard":http://picard.sourceforge.net/command-line-overview.shtml program.
-
-
-The module retrieves the picard binary distribution from Keep, using the job's @picard_zip@ parameter.
-
-<pre>
-import arvados_picard
-arvados_picard.run(
- 'FixMateInformation',
- params={
- 'i': input_bam_path,
- 'o': '/dev/stdout',
- 'quiet': 'true',
- 'so': 'coordinate',
- 'validation_stringency': 'LENIENT',
- 'compression_level': 0
- },
- stdout=open('out.bam','wb'))
-</pre>
-
-On qr1hi.arvadosapi.com, the binary distribution @picard-tools-1.82.zip@ is available in the collection @687f74675c6a0e925dec619cc2bec25f+77 at .
-
-<pre>
-{
- "script_parameters":{
- "picard_zip":"687f74675c6a0e925dec619cc2bec25f+77",
- ...
- },
- ...
-}
-</pre>
diff --git a/doc/user/examples/crunch-examples.html.textile.liquid b/doc/user/examples/crunch-examples.html.textile.liquid
deleted file mode 100644
index c93766a0a..000000000
--- a/doc/user/examples/crunch-examples.html.textile.liquid
+++ /dev/null
@@ -1,102 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Scripts provided by Arvados"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'pipeline_deprecation_notice' %}
-
-Several crunch scripts are included with Arvados in the "/crunch_scripts directory":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/crunch_scripts. They are intended to provide examples and starting points for writing your own scripts.
-
-h4. bwa-aln
-
-Run the bwa aligner on a set of paired-end fastq files, producing a BAM file for each pair. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/bwa-aln
-
-<div class="offset1">
-table(table table-bordered table-condensed).
-|_Parameter_|_Description_|_Example_|
-|bwa_tbz|Collection with the bwa source distribution.|@8b6e2c4916133e1d859c9e812861ce13+70@|
-|samtools_tgz|Collection with the samtools source distribution.|@c777e23cf13e5d5906abfdc08d84bfdb+74@|
-|input|Collection with fastq reads (pairs of *_1.fastq.gz and *_2.fastq.gz).|@d0136bc494c21f79fc1b6a390561e6cb+2778@|
-</div>
-
-h4. bwa-index
-
-Generate an index of a fasta reference genome suitable for use by bwa-aln. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/bwa-index
-
-<div class="offset1">
-table(table table-bordered table-condensed).
-|_Parameter_|_Description_|_Example_|
-|bwa_tbz|Collection with the bwa source distribution.|@8b6e2c4916133e1d859c9e812861ce13+70@|
-|input|Collection with reference data (*.fasta.gz, *.fasta.fai.gz, *.dict.gz).|@c361dbf46ee3397b0958802b346e9b5a+925@|
-</div>
-
-h4. picard-gatk2-prep
-
-Using the FixMateInformation, SortSam, ReorderSam, AddOrReplaceReadGroups, and BuildBamIndex modules from picard, prepare a BAM file for use with the GATK2 tools. Additionally, run picard's CollectAlignmentSummaryMetrics module to produce a @*.casm.tsv@ statistics file for each BAM file. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/picard-gatk2-prep
-
-<div class="offset1">
-table(table table-bordered table-condensed).
-|_Parameter_|_Description_|_Example_|
-|input|Collection containing aligned bam files.||
-|picard_zip|Collection with the picard binary distribution.|@687f74675c6a0e925dec619cc2bec25f+77@|
-|reference|Collection with reference data (*.fasta.gz, *.fasta.fai.gz, *.dict.gz).|@c361dbf46ee3397b0958802b346e9b5a+925@|
-</div>
-
-h4. GATK2-realign
-
-Run GATK's RealignerTargetCreator and IndelRealigner modules on a set of BAM files. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-realign
-
-<div class="offset1">
-table(table table-bordered table-condensed).
-|_Parameter_|_Description_|_Example_|
-|input|Collection containing aligned bam files.||
-|picard_zip|Collection with the picard binary distribution.|@687f74675c6a0e925dec619cc2bec25f+77@|
-|gatk_tbz|Collection with the GATK2 binary distribution.|@7e0a277d6d2353678a11f56bab3b13f2+87@|
-|gatk_bundle|Collection with the GATK data bundle.|@d237a90bae3870b3b033aea1e99de4a9+10820@|
-|known_sites|List of files in the data bundle to use as GATK @-known@ arguments. Optional. |@["dbsnp_137.b37.vcf","Mills_and_1000G_gold_standard.indels.b37.vcf"]@ (this is the default value)|
-|regions|Collection with .bed files indicating sequencing target regions. Optional.||
-|region_padding|Corresponds to GATK @--interval_padding@ argument. Required if a regions parameter is given.|10|
-</div>
-
-h4. GATK2-bqsr
-
-Run GATK's BaseQualityScoreRecalibration module on a set of BAM files. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-bqsr
-
-<div class="offset1">
-table(table table-bordered table-condensed).
-|_Parameter_|_Description_|_Example_|
-|input|Collection containing bam files.||
-|gatk_tbz|Collection with the GATK2 binary distribution.|@7e0a277d6d2353678a11f56bab3b13f2+87@|
-|gatk_bundle|Collection with the GATK data bundle.|@d237a90bae3870b3b033aea1e99de4a9+10820@|
-</div>
-
-h4. GATK2-merge-call
-
-Merge a set of BAM files using picard, and run GATK's UnifiedGenotyper module on the merged set to produce a VCF file. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-merge-call
-
-<div class="offset1">
-table(table table-bordered table-condensed).
-|_Parameter_|_Description_|_Example_|
-|input|Collection containing bam files.||
-|picard_zip|Collection with the picard binary distribution.|@687f74675c6a0e925dec619cc2bec25f+77@|
-|gatk_tbz|Collection with the GATK2 binary distribution.|@7e0a277d6d2353678a11f56bab3b13f2+87@|
-|gatk_bundle|Collection with the GATK data bundle.|@d237a90bae3870b3b033aea1e99de4a9+10820@|
-|regions|Collection with .bed files indicating sequencing target regions. Optional.||
-|region_padding|Corresponds to GATK @--interval_padding@ argument. Required if a regions parameter is given.|10|
-</div>
-
-h4. file-select
-
-Pass through the named files from input to output collection, and ignore the rest. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/file-select
-
-<div class="offset1">
-table(table table-bordered table-condensed).
-|_Parameter_|_Description_|_Example_|
-|names|List of filenames to include in the output.|@["human_g1k_v37.fasta.gz","human_g1k_v37.fasta.fai.gz"]@|
-</div>
diff --git a/doc/user/reference/job-pipeline-ref.html.textile.liquid b/doc/user/reference/job-pipeline-ref.html.textile.liquid
deleted file mode 100644
index f80cec9d4..000000000
--- a/doc/user/reference/job-pipeline-ref.html.textile.liquid
+++ /dev/null
@@ -1,14 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Pipeline template reference"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'pipeline_deprecation_notice' %}
-
-Pipeline template options are described on the "pipeline template schema page.":{{site.baseurl}}/api/methods/pipeline_templates.html
diff --git a/doc/user/topics/arv-run.html.textile.liquid b/doc/user/topics/arv-run.html.textile.liquid
deleted file mode 100644
index 9752ca7d2..000000000
--- a/doc/user/topics/arv-run.html.textile.liquid
+++ /dev/null
@@ -1,163 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Using arv-run"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'crunch1only_begin' %}
-On those sites, the features described here are not yet implemented.
-{% include 'crunch1only_end' %}
-
-The @arv-run@ command enables you create Arvados pipelines at the command line that fan out to multiple concurrent tasks across Arvados compute nodes.
-
-{% include 'tutorial_expectations' %}
-
-h1. Usage
-
-Using @arv-run@ you can write and test command lines interactively, then insert @arv-run@ at the beginning of the command line to run the command on Arvados. For example:
-
-<notextile>
-<pre>
-$ <span class="userinput">cd ~/keep/by_id/3229739b505d2b878b62aed09895a55a+142</span>
-$ <span class="userinput">ls *.fastq</span>
-HWI-ST1027_129_D0THKACXX.1_1.fastq HWI-ST1027_129_D0THKACXX.1_2.fastq
-$ <span class="userinput">grep -H -n ATTGGAGGAAAGATGAGTGAC HWI-ST1027_129_D0THKACXX.1_1.fastq</span>
-HWI-ST1027_129_D0THKACXX.1_1.fastq:14:TCTGGCCCCTGTTGTCTGCATGTAACTTAATACCACAACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCCCAACCTA
-HWI-ST1027_129_D0THKACXX.1_1.fastq:18:AACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCT
-HWI-ST1027_129_D0THKACXX.1_1.fastq:30:ATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCTGTGATACG
-$ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC HWI-ST1027_129_D0THKACXX.1_1.fastq</span>
-Running pipeline qr1hi-d1hrv-mg3bju0u7r6w241
-[...]
- 0 stderr run-command: grep -H -n ATTGGAGGAAAGATGAGTGAC /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq
- 0 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:14:TCTGGCCCCTGTTGTCTGCATGTAACTTAATACCACAACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCCCAACCTA
- 0 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:18:AACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCT
- 0 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:30:ATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCTGTGATACG
- 0 stderr run-command: completed with exit code 0 (success)
-[...]
-</pre>
-</notextile>
-
-A key feature of @arv-run@ is the ability to introspect the command line to determine which arguments are file inputs, and transform those paths so they are usable inside the Arvados container. In the above example, @HWI-ST1027_129_D0THKACXX.1_2.fastq@ is transformed into @/keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq at . @arv-run@ also works together with @arv-mount@ to identify that the file is already part of an Arvados collection. In this case, it will use the existing collection without any upload step. If you specify a file that is only available on the local filesystem, @arv-run@ will upload a new collection.
-
-If you find that @arv-run@ is incorrectly rewriting one of your command line arguments, place a backslash @\@ at the beginning of the affected argument to quote it (suppress rewriting).
-
-h2. Parallel tasks
-
- at arv-run@ will parallelize over files listed on the command line after @-- at .
-
-<notextile>
-<pre>
-$ <span class="userinput">cd ~/keep/by_id/3229739b505d2b878b62aed09895a55a+142</span>
-$ <span class="userinput">ls *.fastq</span>
-HWI-ST1027_129_D0THKACXX.1_1.fastq HWI-ST1027_129_D0THKACXX.1_2.fastq
-$ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC -- *.fastq</span>
-Running pipeline qr1hi-d1hrv-mg3bju0u7r6w241
-[...]
- 0 stderr run-command: parallelizing on input0 with items [u'/keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq', u'/keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_2.fastq']
-[...]
- 1 stderr run-command: grep -H -n ATTGGAGGAAAGATGAGTGAC /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq
- 2 stderr run-command: grep -H -n ATTGGAGGAAAGATGAGTGAC /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_2.fastq
-[...]
- 1 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:14:TCTGGCCCCTGTTGTCTGCATGTAACTTAATACCACAACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCCCAACCTA
- 1 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:18:AACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCT
- 1 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:30:ATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCTGTGATACG
- 1 stderr run-command: completed with exit code 0 (success)
- 2 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_2.fastq:34:CTGGCCCCTGTTGTCTGCATGTAACTTAATACCACAACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAG
- 2 stderr run-command: completed with exit code 0 (success)
-</pre>
-</notextile>
-
-You may specify @--batch-size N@ (or the short form @-bN@) after the @--@ but before listing any files to specify how many files to provide put on the command line for each task. See "Putting it all together" below for an example.
-
-h2. Redirection
-
-You may use standard input (@<@) and standard output (@>@) redirection. This will create a separate task for each file listed in standard input. You are only permitted to supply a single file name for stdout @>@ redirection. If there are multiple tasks with their output sent to the same file, the output will be collated at the end of the pipeline.
-
-(Note: because the syntax is designed to mimic standard shell syntax, it is necessary to quote the metacharacters @<@, @>@ and @|@ as either @\<@, @\>@ and @\|@ or @'<'@, @'>'@ and @'|'@.)
-
-{% include 'arv_run_redirection' %}
-
-You may use "run-command":run-command.html parameter substitution in the output file name to generate different filenames for each task:
-
-<notextile>
-<pre>
-$ <span class="userinput">cd ~/keep/by_id/3229739b505d2b878b62aed09895a55a+142</span>
-$ <span class="userinput">ls *.fastq</span>
-$ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC \< *.fastq \> '$(task.uuid).txt'</span>
-[...]
- 1 stderr run-command: grep -H -n ATTGGAGGAAAGATGAGTGAC < /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq > qr1hi-ot0gb-hmmxf2zubfpmhfk.txt
- 2 stderr run-command: grep -H -n ATTGGAGGAAAGATGAGTGAC < /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_2.fastq > qr1hi-ot0gb-iu2xgy4hkx4mmri.txt
- 1 stderr run-command: completed with exit code 0 (success)
- 1 stderr run-command: the following output files will be saved to keep:
- 1 stderr run-command: 363 ./qr1hi-ot0gb-hmmxf2zubfpmhfk.txt
- 1 stderr run-command: start writing output to keep
- 1 stderr upload wrote 363 total 363
- 2 stderr run-command: completed with exit code 0 (success)
- 2 stderr run-command: the following output files will be saved to keep:
- 2 stderr run-command: 121 ./qr1hi-ot0gb-iu2xgy4hkx4mmri.txt
- 2 stderr run-command: start writing output to keep
- 2 stderr upload wrote 121 total 121
-[...]
-</pre>
-</notextile>
-
-h2. Pipes
-
-Multiple commands may be connected by pipes and execute in the same container:
-
-<notextile>
-<pre>
-$ <span class="userinput">cd ~/keep/by_id/3229739b505d2b878b62aed09895a55a+142</span>
-$ <span class="userinput">ls *.fastq</span>
-$ <span class="userinput">arv-run cat -- *.fastq \| grep -H -n ATTGGAGGAAAGATGAGTGAC \> output.txt</span>
-[...]
- 1 stderr run-command: cat /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq | grep -H -n ATTGGAGGAAAGATGAGTGAC > output.txt
- 2 stderr run-command: cat /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_2.fastq | grep -H -n ATTGGAGGAAAGATGAGTGAC > output.txt
-[...]
-</pre>
-</notextile>
-
-If you need to capture intermediate results of a pipe, use the @tee@ command.
-
-h2. Running a shell script
-
-<notextile>
-<pre>
-$ <span class="userinput">echo 'echo hello world' > hello.sh</span>
-$ <span class="userinput">arv-run /bin/sh hello.sh</span>
-Upload local files: "hello.sh"
-Uploaded to qr1hi-4zz18-23u3hxugbm71qmn
-Running pipeline qr1hi-d1hrv-slcnhq5czo764b1
-[...]
- 0 stderr run-command: /bin/sh /keep/5d3a4131b7d8f233f2a917d8a5c3c2b2+52/hello.sh
- 0 stderr hello world
- 0 stderr run-command: completed with exit code 0 (success)
-[...]
-</pre>
-</notextile>
-
-h2. Additional options
-
-* @--docker-image IMG@ : By default, commands run based in a container created from the @default_docker_image_for_jobs@ setting on the API server. Use this option to specify a different image to use. Note: the Docker image must be uploaded to Arvados using @arv keep docker at .
-* @--dry-run@ : Print out the final Arvados pipeline generated by @arv-run@ without submitting it.
-* @--local@ : By default, the pipeline will be submitted to your configured Arvados instance. Use this option to run the command locally using @arv-run-pipeline-instance --run-jobs-here at .
-* @--ignore-rcode@ : Some commands use non-zero exit codes to indicate nonfatal conditions (e.g., @grep@ returns 1 when no match is found). Set this to indicate that commands that return non-zero return codes should not be considered failed.
-* @--no-wait@ : Do not wait and display logs after submitting command, just exit.
-
-h2. Putting it all together: bwa mem
-
-<notextile>
-<pre>
-$ <span class="userinput">cd ~/keep/by_id/d0136bc494c21f79fc1b6a390561e6cb+2778</span>
-$ <span class="userinput">arv-run --docker-image arvados/jobs-java-bwa-samtools bwa mem ../3514b8e5da0e8d109946bc809b20a78a+5698/human_g1k_v37.fasta -- --batch-size 2 *.fastq.gz \> '$(task.uuid).sam'</span>
- 0 stderr run-command: parallelizing on input0 with items [[u'/keep/d0136bc494c21f79fc1b6a390561e6cb+2778/HWI-ST1027_129_D0THKACXX.1_1.fastq.gz', u'/keep/d0136bc494c21f79fc1b6a390561e6cb+2778/HWI-ST1027_129_D0THKACXX.1_2.fastq.gz'], [u'/keep/d0136bc494c21f79fc1b6a390561e6cb+2778/HWI-ST1027_129_D0THKACXX.2_1.fastq.gz', u'/keep/d0136bc494c21f79fc1b6a390561e6cb+2778/HWI-ST1027_129_D0THKACXX.2_2.fastq.gz']]
-[...]
- 1 stderr run-command: bwa mem /keep/3514b8e5da0e8d109946bc809b20a78a+5698/human_g1k_v37.fasta /keep/d0136bc494c21f79fc1b6a390561e6cb+2778/HWI-ST1027_129_D0THKACXX.1_1.fastq.gz /keep/d0136bc494c21f79fc1b6a390561e6cb+2778/HWI-ST1027_129_D0THKACXX.1_2.fastq.gz > qr1hi-ot0gb-a4bzzyqqz4ubair.sam
- 2 stderr run-command: bwa mem /keep/3514b8e5da0e8d109946bc809b20a78a+5698/human_g1k_v37.fasta /keep/d0136bc494c21f79fc1b6a390561e6cb+2778/HWI-ST1027_129_D0THKACXX.2_1.fastq.gz /keep/d0136bc494c21f79fc1b6a390561e6cb+2778/HWI-ST1027_129_D0THKACXX.2_2.fastq.gz > qr1hi-ot0gb-14j9ncw0ymkxq0v.sam
-</pre>
-</notextile>
diff --git a/doc/user/topics/crunch-tools-overview.html.textile.liquid b/doc/user/topics/crunch-tools-overview.html.textile.liquid
deleted file mode 100644
index c4d01cfc3..000000000
--- a/doc/user/topics/crunch-tools-overview.html.textile.liquid
+++ /dev/null
@@ -1,70 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Tools for writing Crunch pipelines"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'pipeline_deprecation_notice' %}
-
-Arvados includes a number of tools to help you develop pipelines and jobs for Crunch. This overview explains each tool's intended use to help you choose the right one.
-
-h2. Use the "arv-run command-line utility":arv-run.html
-
-arv-run is an interactive command-line tool. You run it as the first command of a traditional Unix shell command line, and it converts that work into an Arvados pipeline. It automatically uploads any required data to Arvados, and dispatches work in parallel when possible. This lets you easily migrate analysis work that you're doing on the command line to Arvados compute nodes.
-
-arv-run is best suited to complement work you already do on the command line. If you write a shell one-liner that generates useful data, you can then call it with arv-run to parallelize it across a larger data set and save the results in Arvados. For example, this run searches multiple FASTQ files in parallel, and saves the results to Keep through shell redirection:
-
-{% include 'arv_run_redirection' %}
-
-arv-run does not generate pipeline templates, or implement higher-level shell constructs like flow control. If you want to make it easy to rerun your pipeline with different data later, or adapt to different inputs, it's best to write your own template.
-
-Refer to the "arv-run documentation":arv-run.html for details.
-
-h2. Write a "pipeline template":{{site.baseurl}}/user/tutorials/running-external-program.html
-
-Pipeline templates describe a set of analysis programs that should be run, and the inputs they require. You can provide a high-level description of how data flows through the pipeline—for example, the outputs of programs A and B are provided as input to program C—and let Crunch take care of the details of starting the individual programs at the right time with the inputs you specified.
-
-Pipeline templates are written in JSON. Once you save a pipeline template in Arvados, you run it by creating a pipeline instance that lists the specific inputs you'd like to use. Arvados Workbench and the @arv pipeline run@ command-line tool both provide high-level interfaces to do this easily. The pipeline's final output(s) will be saved in a project you specify.
-
-See the User Guide topic to learn how to "write and run your own pipelines":{{site.baseurl}}/user/tutorials/running-external-program.html. The rest of this page suggests specific tools to use in your templates.
-
-h3. The "run-command Crunch script":run-command.html
-
-run-command is a Crunch script that is included with Arvados. It builds a command line from its input parameters. It runs that command on files in Collections using the Keep mount provided by Crunch. Output files created by the command are saved in a new collection, which is considered the program's final output. It can run the command in parallel on a list of inputs, and introspect arguments so you can, for example, generate output filenames based on input filenames.
-
-run-command is a great way to use an existing analysis tool inside an Arvados pipeline. You might use one or two tools in a larger pipeline, or convert a simple series of tool invocations into a pipeline to benefit from Arvados' provenance tracking and job reuse. For example, here's a one-step pipeline that uses run-command with bwa to align a single paired-end read FASTQ sample:
-
-<notextile>{% code 'run_command_simple_example' as javascript %}</notextile>
-
-run-command is limited to manipulating the tool's command-line arguments, and can only parallelize on simple lists of inputs. If you need to preprocess input, or dispatch work differently based on those inputs, consider writing your own Crunch script.
-
-Refer to the "run-command reference":run-command.html for details.
-
-h3. Writing "your own Crunch script":{{site.baseurl}}/user/tutorials/tutorial-firstscript.html with the Python SDK
-
-Arvados includes a Python SDK designed to help you write your own Crunch scripts. It provides a native Arvados API client; Collection classes that provide file-like objects to interact with data in Keep; and utility functions to work within Crunch's execution environment. Using the Python SDK, you can efficiently dispatch work with however much sophistication you require.
-
-Writing your own Crunch script is the best way to do analysis in Arvados when an existing tool does not meet your needs. By interacting directly with Arvados objects, you'll have full power to introspect and adapt to your input, introduce minimal overhead, and get very direct error messages in case there's any trouble. As a simple example, here's a Crunch script that checksums each file in a collection in parallel, saving the results in Keep:
-
-<notextile>{% code 'tutorial_hash_script_py' as python %}</notextile>
-
-There's no limit to what you can do with your own Crunch script. The downside is the amount of time and effort you're required to invest to write and debug new code. If you have to do that anyway, writing a Crunch script will give you the most benefit from using Arvados.
-
-Refer to the "User Guide topic on writing Crunch scripts":{{site.baseurl}}/user/tutorials/tutorial-firstscript.html and the "Python SDK reference":{{site.baseurl}}/sdk/python/python.html for details.
-
-h3. Combining run-command and custom Crunch scripts in a pipeline
-
-Just because you need to write some new code to do some work doesn't mean that you have to do all the work in your own Crunch script. You can combine your custom steps with existing tools in a pipeline, passing data between them. For example, maybe there's a third-party tool that does most of the analysis work you need, but you often need to massage the tool's data. You could write your own preprocessing script that creates a new collection to use as the input of a run-command job, or a postprocessing script to create a final output after the tool is done, and tie them all together in a pipeline. Just like Unix pipes, Arvados pipelines let you combine smaller tools to maximize utility.
-
-h3. Using run-command with your legacy scripts
-
-Perhaps you've already written your own analysis program that you want to run inside Arvados. Currently, the easiest way to do that is to copy run-command from the Arvados source code to your own Arvados git repository, along with your internal tool. Then your pipeline can call run-command from your own repository to execute the internal tool alongside it.
-
-This approach has the downside that you'll have to copy and push run-command again any time there's an update you'd like to use. Future Arvados development will make it possible to get code from multiple git repositories, so your job can use the latest run-command in the Arvados source, as well as the latest tool in your own git repository. Follow "Arvados issue #4561":https://arvados.org/issues/4561 for updates.
-
-Alternatively, you can "build a Docker image that includes your program, add it to Arvados":arv-docker.html, then run the Arvados run-command script inside that Docker image.
diff --git a/doc/user/topics/run-command.html.textile.liquid b/doc/user/topics/run-command.html.textile.liquid
deleted file mode 100644
index 6996475c3..000000000
--- a/doc/user/topics/run-command.html.textile.liquid
+++ /dev/null
@@ -1,319 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "run-command reference"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'pipeline_deprecation_notice' %}
-
-The @run-command@ crunch script enables you run command line programs.
-
-{% include 'tutorial_expectations_workstation' %}
-
-h1. Using run-command
-
-The basic @run-command@ process evaluates its inputs and builds a command line, executes the command, and saves the contents of the output directory back to Keep. For large datasets, @run-command@ can schedule concurrent tasks to execute the wrapped program over a range of inputs (see @task.foreach@ below.)
-
- at run-command@ is controlled through the @script_parameters@ section of a pipeline component. @script_parameters@ is a JSON object consisting of key-value pairs. There are three categories of keys that are meaningful to run-command:
-* The @command@ section defining the template to build the command line of task
-* Special processing directives such as @task.foreach@ @task.cwd@ @task.vwd@ @task.stdin@ @task.stdout@
-* User-defined parameters (everything else)
-
-In the following examples, you can use "dry run mode" to determine the command line that @run-command@ will use without actually running the command. For example:
-
-<notextile>
-<pre><code>~$ <span class="userinput">cd $HOME/arvados/crunch_scripts</span>
-~$ <span class="userinput">./run-command --dry-run --script-parameters '{
- "command": ["echo", "hello world"]
-}'</span>
-run-command: echo hello world
-</code></pre>
-</notextile>
-
-h2. Command template
-
-The value of the "command" key is a list. The first parameter of the list is the actual program to invoke, followed by the command arguments. The simplest @run-command@ invocation simply runs a program with static parameters. In this example, run "echo" with the first argument "hello world":
-
-<pre>
-{
- "command": ["echo", "hello world"]
-}
-</pre>
-
-Running this job will print "hello world" to the job log.
-
-By default, the command will start with the current working directory set to the output directory. Anything written to the output directory will be saved to Keep when the command is finished. You can change the default working directory using @task.cwd@ and get the path to the output directory using @$(task.outdir)@ as explained below.
-
-Items in the "command" list may include lists and objects in addition to strings. Lists are flattened to produce the final command line. JSON objects are evaluated as list item functions (see below). For example, the following evaluates to @["echo", "hello", "world"]@:
-
-<pre>
-{
- "command": ["echo", ["hello", "world"]]
-}
-</pre>
-
-Finally, if "command" is a list of lists, it specifies a Unix pipeline where the standard output of the previous command is piped into the standard input of the next command. The following example describes the Unix pipeline @cat foo | grep bar@:
-
-<pre>
-{
- "command": [["cat", "foo"], ["grep", "bar"]]
-}
-</pre>
-
-h2. Parameter substitution
-
-The "command" list can include parameter substitutions. Substitutions are enclosed in "$(...)" and may contain the name of a user-defined parameter. In the following example, the value of "a" is "hello world"; so when "command" is evaluated, it will substitute "hello world" for "$(a)":
-
-<pre>
-{
- "a": "c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2",
- "command": ["echo", "$(file $(a))"]
-}
-</pre>
-
-table(table table-bordered table-condensed).
-|_. Function|_. Action|
-|$(file ...) | Takes a reference to a file within an Arvados collection and evaluates to a file path on the local file system where that file can be accessed by your command. Will raise an error if the file is not accessible.|
-|$(dir ...) | Takes a reference to an Arvados collection or directory within an Arvados collection and evaluates to a directory path on the local file system where that directory can be accessed by your command. The path may include a file name, in which case it will evaluate to the parent directory of the file. Uses Python's os.path.dirname(), so "/foo/bar" will evaluate to "/foo" but "/foo/bar/" will evaluate to "/foo/bar". Will raise an error if the directory is not accessible. |
-|$(basename ...) | Strip leading directory and trailing file extension from the path provided. For example, $(basename /foo/bar.baz.txt) will evaluate to "bar.baz".|
-|$(glob ...) | Take a Unix shell path pattern (supports @*@ @?@ and @[]@) and search the local filesystem, returning the first match found. Use together with $(dir ...) to get a local filesystem path for Arvados collections. For example: $(glob $(dir $(mycollection)/*.bam)) will find the first .bam file in the collection specified by the user parameter "mycollection". If there is more than one match, which one is returned is undefined. Will raise an error if no matches are found.|
-|$(task.tmpdir)|Designated temporary directory. This directory will be discarded when the job completes.|
-|$(task.outdir)|Designated output directory. The contents of this directory will be saved to Keep when the job completes. A symlink to a file in the keep mount will reference existing Keep blocks in your job output collection, with no data copying or duplication.|
-|$(job.srcdir)|Path to the git working directory ($CRUNCH_SRC).|
-|$(node.cores)|Number of CPU cores on the node.|
-|$(job.uuid)|Current job uuid ($JOB_UUID)|
-|$(task.uuid)|Current task uuid ($TASK_UUID)|
-
-h3. Escape sequences
-
-If your command includes a @$()@ sequence that shouldn't be interpreted by run-command—for example, because you're writing shell code that calls a subcommand—you can prevent run-command from interpreting it by placing a backslash in front of the @$@ character. Note that JSON also uses backslash to escape characters, so you'll need to write two backslashes for run-command to see one after parsing the parameter. This example uppercases all alphabetic characters in the "pattern" parameter before using it as a regular expression in grep:
-
-<pre>{"command": ["bash", "-c", "grep \\$(echo '$(pattern)' | tr a-z A-Z) '$(input)'"]}</pre>
-
-You can put a literal backslash in your command by escaping it with another backslash. Ultimately this means that where the primary Unix command includes a single backslash, you'll need to write four backslashes: double the backslashes for run-command escaping, then double them again for JSON escaping.
-
-<pre>{"command": ["grep", "\\\\bword\\\\b", "$(input)"]}</pre>
-
-h2. List context
-
-Where specified by the documentation, parameters may be evaluated in a "list context". That means the value will evaluate to a list instead of a string. Parameter values can be a static list, a path to a file, a path to a directory, or a JSON object describing a list context function.
-
-If the value is a string, it is interpreted as a path. If the path specifies a regular file, that file will be opened as a text file and produce a list with one item for each line in the file (end-of-line characters will be stripped). If the path specifies a directory, produce a list containing all of the entries in the directory. Note that parameter expansion is not performed on list items produced this way.
-
-If the value is a static list, it will evaluate each item and return the expanded list. Each item may be a string (evaluated for parameter substitution), a list (recursively evaluated), or a JSON object (indicating a list function, described below).
-
-If the value is a JSON object, it is evaluated as a list function described below.
-
-h2. List functions
-
-When @run-command@ is evaluating a list (such as "command"), in addition to string parameter substitution, you can use list item functions. In the following functions, you specify the name of a user parameter to act on (@"$(a)"@ in the first example); the value of that user parameter will be evaluated in a list context (as described above) to get the list value. Alternately, you can provide list value directly in line. As an example, the following two fragments yield the same result:
-
-<pre>
-{
- "a": ["alice", "bob"],
- "command": ["echo", {"foreach": "$(a)",
- "var": "a_var",
- "command": ["--something", "$(a_var)"]}]
-}
-</pre>
-
-<pre>
-{
- "command": ["echo", {"foreach": ["alice", "bob"],
- "var": "a_var",
- "command": ["--something", "$(a_var)"]}]
-}
-</pre>
-
-Note: when you provide the list inline with "foreach" or "index", you must include the "var" parameter to specify the substitution variable name to use when evaluating the command fragment.
-
-You can also nest functions. This filters @["alice", "bob", "betty"]@ on the regular expression @"b.*"@ to get the list @["bob", "betty"]@, assigns @a_var@ to each value of the list, then expands @"command"@ to get @["--something", "bob", "--something", "betty"]@.
-
-<pre>
-{
- "command": ["echo", {"foreach": {"filter": ["alice", "bob", "betty"],
- "regex": "b.*"},
- "var": "a_var",
- "command": ["--something", "$(a_var)"]}]
-}
-</pre>
-
-h3. foreach
-
-The @foreach@ list item function (not to be confused with the @task.foreach@ directive) expands a command template for each item in the specified user parameter (the value of the user parameter is evaluated in a list context, as described above). The following example will evaluate "command" to @["echo", "--something", "alice", "--something", "bob"]@:
-
-<pre>
-{
- "a": ["alice", "bob"],
- "command": ["echo", {"foreach": "$(a)",
- "var": "a_var",
- "command": ["--something", "$(a_var)"]}]
-}
-</pre>
-
-h3. index
-
-This function extracts a single item from a list. The value of @index@ is zero-based (i.e. the first item is at index 0, the second item index 1, etc). The following example will evaluate "command" to @["echo", "--something", "bob"]@:
-
-<pre>
-{
- "a": ["alice", "bob"],
- "command": ["echo", {"list": "$(a)",
- "var": "a_var",
- "index": 1,
- "command": ["--something", "$(a_var)"]}]
-}
-</pre>
-
-h3. filter
-
-Filter the list so that it only includes items that match a regular expression. The following example will evaluate to @["echo", "bob"]@
-
-<pre>
-{
- "a": ["alice", "bob"],
- "command": ["echo", {"filter": "$(a)",
- "regex": "b.*"}]
-}
-</pre>
-
-h3. group
-
-Generate a list of lists, where items are grouped on common subexpression match. Items which don't match the regular expression are excluded. In the following example, the subexpression is @(a?)@, resulting in two groups, strings that contain the letter 'a' and strings that do not. The following example evaluates to @["echo", "--group", "alice", "carol", "dave", "--group", "bob", "betty"]@:
-
-<pre>
-{
- "a": ["alice", "bob", "betty", "carol", "dave"],
- "b": {"group": "$(a)",
- "regex": "[^a]*(a?).*"},
- "command": ["echo", {"foreach": "$(b)",
- "var": "b_var",
- "command": ["--group", "$(b_var)"]}]
-}
-</pre>
-
-h3. extract
-
-Generate a list of lists, where items are split by subexpression match. Items which don't match the regular expression are excluded. The following example evaluates to @["echo", "--something", "c", "a", "rol", "--something", "d", "a", "ve"]@:
-
-<pre>
-{
- "a": ["alice", "bob", "carol", "dave"],
- "b": {"extract": "$(a)",
- "regex": "(.+)(a)(.*)"},
- "command": ["echo", {"foreach": "$(b)",
- "var": "b_var",
- "command": ["--something", "$(b_var)"]}]
-}
-</pre>
-
-h3. batch
-
-Generate a list of lists, where items are split into a batch size. If the list does not divide evenly into batch sizes, the last batch will be short. The following example evaluates to @["echo", "--something", "alice", "bob", "--something", "carol", "dave"]@
-
-<pre>
-{
- "a": ["alice", "bob", "carol", "dave"],
- "command": ["echo", {"foreach":{"batch": "$(a)",
- "size": 2},
- "var": "a_var",
- "command": ["--something", "$(a_var)"]}]
-}
-</pre>
-
-h2. Directives
-
-Directives alter the behavior of run-command. All directives are optional.
-
-h3. task.cwd
-
-This directive sets the initial current working directory in which your command will run. If @task.cwd@ is not specified, the default current working directory is @task.outdir at .
-
-h3. task.ignore_rcode
-
-By Unix convention a task which exits with a non-zero return code is considered failed. However, some programs (such as @grep@) return non-zero codes for conditions that should not be considered fatal errors. Set @"task.ignore_rcode": true@ to indicate the task should always be considered a success regardless of the return code.
-
-h3. task.stdin and task.stdout
-
-Provide standard input and standard output redirection.
-
- at task.stdin@ must evaluate to a path to a file to be bound to the standard input stream of the command. When command describes a Unix pipeline, this goes into the first command.
-
- at task.stdout@ specifies the desired file name in the output directory to save the content of standard output. When command describes a Unix pipeline, this captures the output of the last command.
-
-h3. task.env
-
-Set environment variables for the command. Accepts an object mapping environment variables to the desired values. Parameter substitution is performed on values, but not on the environment variable names themselves. Example usage:
-
-<pre>
-{
- "command": ["/bin/sh", "-c", "echo $MY_ENV_VAR"],
- "task.env": {
- "MY_ENV_VAR": "Hello world!"
- }
-}
-</pre>
-
-h3. task.vwd
-
-Background: because Keep collections are read-only, this does not play well with certain tools that expect to be able to write their outputs alongside their inputs (such as tools that generate indexes that are closely associated with the original file.) The run-command's solution to this is the "virtual working directory".
-
- at task.vwd@ specifies a Keep collection with the starting contents of the output directory. @run-command@ will populate @task.outdir@ with directories and symlinks to mirror the contents of the @task.vwd@ collection. Your command will then be able to both access its input files and write its output files from within @task.outdir at . When the command completes, run-command will write the contents of the output directory, which will include the output of your command as well as symlinks to files in starting collection. Note that files from the starting collection remain read-only and cannot be altered, but may be deleted or renamed.
-
-h3. task.foreach
-
-Using @task.foreach@, you can run your command concurrently over large datasets.
-
- at task.foreach@ takes the names of one or more user-defined parameters. The value of these parameters are evaluated in a list context. @run-command@ then generates tasks based on the Cartesian product (i.e. all combinations) of the input lists. The outputs of all tasks are merged to create the final output collection. Note that if two tasks output a file in the same directory with the same name, that file will be concatenated in the final output. In the following example, three tasks will be created for the "grep" command, based on the contents of user parameter "a":
-
-<pre>
-{
- "command": ["echo", "$(a)"],
- "task.foreach": "a",
- "a": ["alice", "bob", "carol"]
-}
-</pre>
-
-This evaluates to the commands:
-<notextile>
-<pre>
-["echo", "alice"]
-["echo", "bob"]
-["echo", "carol"]
-</pre>
-</notextile>
-
-You can also specify multiple parameters:
-
-<pre>
-{
- "a": ["alice", "bob"],
- "b": ["carol", "dave"],
- "task.foreach": ["a", "b"],
- "command": ["echo", "$(a)", "$(b)"]
-}
-</pre>
-
-This evaluates to the commands:
-
-<pre>
-["echo", "alice", "carol"]
-["echo", "alice", "dave"]
-["echo", "bob", "carol"]
-["echo", "bob", "dave"]
-</pre>
-
-h1. Examples
-
-The following is a single task pipeline using @run-command@ to run the bwa alignment tool to align a single paired-end read fastq sample. The input to this pipeline is the reference genome and a collection consisting of two fastq files for the read pair.
-
-<notextile>{% code 'run_command_simple_example' as javascript %}</notextile>
-
-The following is a concurrent task pipeline using @run-command@ to run the bwa alignment tool to align a set of fastq reads over multiple samples. The input to this pipeline is the reference genome and a collection consisting subdirectories for each sample, with each subdirectory containing pairs of fastq files for each set of reads.
-
-<notextile>{% code 'run_command_foreach_example' as javascript %}</notextile>
diff --git a/doc/user/topics/running-pipeline-command-line.html.textile.liquid b/doc/user/topics/running-pipeline-command-line.html.textile.liquid
deleted file mode 100644
index ffa57103b..000000000
--- a/doc/user/topics/running-pipeline-command-line.html.textile.liquid
+++ /dev/null
@@ -1,58 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Running an Arvados pipeline"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'crunch1only_begin' %}
-If the Jobs API is not available, use the "Common Workflow Language":{{site.baseurl}}/user/cwl/cwl-runner.html instead.
-{% include 'crunch1only_end' %}
-
-This tutorial demonstrates how to use the command line to run the same pipeline as described in "running a pipeline using Workbench.":{{site.baseurl}}/user/tutorials/tutorial-workflow-workbench.html
-
-{% include 'tutorial_expectations' %}
-{% include 'tutorial_cluster_name' %}
-
-When you use the command line, you must use Arvados unique identifiers to refer to objects. The identifiers in this example correspond to the following Arvados objects:
-
-* <i class="fa fa-fw fa-gear"></i> "Tutorial align using bwa mem (qr1hi-p5p6p-itzkwxblfermlwv)":{{site.arvados_workbench_host}}/pipeline_templates/qr1hi-p5p6p-itzkwxblfermlwv
-* <i class="fa fa-fw fa-archive"></i> "Tutorial chromosome 19 reference (2463fa9efeb75e099685528b3b9071e0+438)":{{site.arvados_workbench_host}}/collections/2463fa9efeb75e099685528b3b9071e0+438
-* <i class="fa fa-fw fa-archive"></i> "Tutorial sample exome (3229739b505d2b878b62aed09895a55a+142)":{{site.arvados_workbench_host}}/collections/3229739b505d2b878b62aed09895a55a+142
-
-Use @arv pipeline run@ to run the pipeline, supplying the inputs to the bwa-mem component on the command line:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv pipeline run --run-pipeline-here --template qr1hi-p5p6p-itzkwxblfermlwv bwa-mem::reference_collection=2463fa9efeb75e099685528b3b9071e0+438 bwa-mem::sample=3229739b505d2b878b62aed09895a55a+142</span>
-
-2014-07-25 18:05:26 +0000 -- pipeline_instance qr1hi-d1hrv-d14trje19pna7f2
-bwa-mem qr1hi-8i9sb-67n1qvsronmd2z6 queued 2014-07-25T18:05:25Z
-
-2014-07-25 18:05:36 +0000 -- pipeline_instance qr1hi-d1hrv-d14trje19pna7f2
-bwa-mem qr1hi-8i9sb-67n1qvsronmd2z6 {:done=>0, :running=>1, :failed=>0, :todo=>0}
-
-2014-07-25 18:05:46 +0000 -- pipeline_instance qr1hi-d1hrv-d14trje19pna7f2
-bwa-mem qr1hi-8i9sb-67n1qvsronmd2z6 49bae1066f4ebce72e2587a3efa61c7d+88
-</code></pre>
-</notextile>
-
-This instantiates your pipeline and displays periodic status reports in your terminal window. The new pipeline instance will also show up on the Workbench Dashboard.
-
-
- at arv pipeline run@ submits a job for each pipeline component as soon as the component's inputs are known (i.e., any dependencies are satsified). It terminates when there is no work left to do: this means either all components are satisfied and all jobs have completed successfully, _or_ one or more jobs have failed and it is therefore unproductive to submit any further jobs.
-
-The Keep locators of the output of the @bwa-mem@ components are available from the last status report shown above:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv keep ls -s 49bae1066f4ebce72e2587a3efa61c7d+88</span>
- 29226 ./HWI-ST1027_129_D0THKACXX.1_1.sam
-</code></pre>
-</notextile>
-
-h2. Re-using existing jobs and outputs
-
-When satisfying a pipeline component that is not marked as nondeterministic in the pipeline template, @arv pipeline run@ checks for a previously submitted job that satisfies the component's requirements. If such a job is found, @arv pipeline run@ uses the existing job rather than submitting a new one. Usually this is a safe way to conserve time and compute resources. In some cases it's desirable to re-run jobs with identical specifications (e.g., to demonstrate that a job or entire pipeline thought to be repeatable is in fact repeatable). For such cases, job re-use features can be disabled entirely by passing the @--no-reuse@ flag to the @arv pipeline run@ command.
diff --git a/doc/user/topics/tutorial-parallel.html.textile.liquid b/doc/user/topics/tutorial-parallel.html.textile.liquid
deleted file mode 100644
index 7d308dcff..000000000
--- a/doc/user/topics/tutorial-parallel.html.textile.liquid
+++ /dev/null
@@ -1,85 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Concurrent Crunch tasks"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'pipeline_deprecation_notice' %}
-
-In the previous tutorials, we used @arvados.job_setup.one_task_per_input_file()@ to automatically create concurrent jobs by creating a separate task per file. For some types of jobs, you may need to split the work up differently, for example creating tasks to process different segments of a single large file. This tutorial will demonstrate how to create Crunch tasks directly.
-
-Start by entering the @crunch_scripts@ directory of your Git repository:
-
-<notextile>
-<pre><code>~$ <span class="userinput">cd $USER/crunch_scripts</span>
-</code></pre>
-</notextile>
-
-Next, using @nano@ or your favorite Unix text editor, create a new file called @concurrent-hash.py@ in the @crunch_scripts@ directory.
-
-notextile. <pre>~/$USER/crunch_scripts$ <code class="userinput">nano concurrent-hash.py</code></pre>
-
-Add the following code to compute the MD5 hash of each file in a collection:
-
-<notextile> {% code 'concurrent_hash_script_py' as python %} </notextile>
-
-Make the file executable:
-
-notextile. <pre><code>~/$USER/crunch_scripts$ <span class="userinput">chmod +x concurrent-hash.py</span></code></pre>
-
-Add the file to the Git staging area, commit, and push:
-
-<notextile>
-<pre><code>~/$USER/crunch_scripts$ <span class="userinput">git add concurrent-hash.py</span>
-~/$USER/crunch_scripts$ <span class="userinput">git commit -m"concurrent hash"</span>
-~/$USER/crunch_scripts$ <span class="userinput">git push origin master</span>
-</code></pre>
-</notextile>
-
-You should now be able to run your new script using Crunch, with "script" referring to our new "concurrent-hash.py" script. We will use a different input from our previous examples. We will use @887cd41e9c613463eab2f0d885c6dd96+83@ which consists of three files, "alice.txt", "bob.txt" and "carol.txt" (the example collection used previously in "fetching data from Arvados using Keep":{{site.baseurl}}/user/tutorials/tutorial-keep.html#dir).
-
-<notextile>
-<pre><code>~/$USER/crunch_scripts$ <span class="userinput">cat >~/the_job <<EOF
-{
- "script": "concurrent-hash.py",
- "repository": "$USER/$USER",
- "script_version": "master",
- "script_parameters":
- {
- "input": "887cd41e9c613463eab2f0d885c6dd96+83"
- }
-}
-EOF</span>
-~/$USER/crunch_scripts$ <span class="userinput">arv job create --job "$(cat ~/the_job)"</span>
-{
- ...
- "uuid":"qr1hi-xxxxx-xxxxxxxxxxxxxxx"
- ...
-}
-~/$USER/crunch_scripts$ <span class="userinput">arv job get --uuid qr1hi-xxxxx-xxxxxxxxxxxxxxx</span>
-{
- ...
- "output":"e2ccd204bca37c77c0ba59fc470cd0f7+162",
- ...
-}
-</code></pre>
-</notextile>
-
-(Your shell should automatically fill in @$USER@ with your login name. The job JSON that gets saved should have @"repository"@ pointed at your personal Git repository.)
-
-Because the job ran in concurrent, each instance of concurrent-hash creates a separate @md5sum.txt@ as output. Arvados automatically collates theses files into a single collection, which is the output of the job:
-
-<notextile>
-<pre><code>~/$USER/crunch_scripts$ <span class="userinput">arv keep ls e2ccd204bca37c77c0ba59fc470cd0f7+162</span>
-./md5sum.txt
-~/$USER/crunch_scripts$ <span class="userinput">arv-get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt</span>
-0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
-504938460ef369cd275e4ef58994cffe bob.txt
-8f3b36aff310e06f3c5b9e95678ff77a carol.txt
-</code></pre>
-</notextile>
diff --git a/doc/user/topics/tutorial-trait-search.html.textile.liquid b/doc/user/topics/tutorial-trait-search.html.textile.liquid
deleted file mode 100644
index d396802f7..000000000
--- a/doc/user/topics/tutorial-trait-search.html.textile.liquid
+++ /dev/null
@@ -1,278 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Querying the Metadata Database"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'notebox_begin_warning' %}
-The humans, specimens and traits tables are deprecated and will be removed in a future release. The recommended way to store and search on user-defined metadata is using the "properties" field of Arvados resources.
-{% include 'notebox_end' %}
-
-This tutorial introduces the Arvados Metadata Database. The Metadata Database stores information about files in Keep. This example will use the Python SDK to find public WGS (Whole Genome Sequencing) data for people who have reported a certain medical condition.
-
-{% include 'tutorial_expectations' %}
-
-In the tutorial examples, three angle brackets (>>>) will be used to denote code to enter at the interactive Python prompt.
-
-Start by running Python.
-
-<notextile>
-<pre><code>~$ <span class="userinput">python</span>
-Python 2.7.3 (default, Jan 2 2013, 13:56:14)
-[GCC 4.7.2] on linux2
-Type "help", "copyright", "credits" or "license" for more information.
->>>
-</code></pre>
-</notextile>
-
-If everything is set up correctly, you will be able to import the arvados SDK.
-
-notextile. <pre><code>>>> <span class="userinput">import arvados</span></pre></code>
-
-This tutorial will also use the regular expression (re) python module:
-
-<notextile>
-<pre><code>>>> <span class="userinput">import re</span>
-</code></pre>
-</notextile>
-
-h2. Finding traits
-
-notextile. <pre><code>>>> <span class="userinput">all_traits = arvados.api().traits().list(limit=1000).execute()</span></code></pre>
-
-* @arvados.api()@ gets an object that provides access to the Arvados API server
-* @.traits()@ gets an object that provides access to the "traits" resource on the Arvados API server
-* @.list(limit=1000)@ constructs a query to list all elements of the "traits" resource, with a limit of 1000 entries returned
-* @.execute()@ executes the query and returns the result, which we assign to "all_traits"
-
-notextile. <pre><code>>>> <span class="userinput">cancer_traits = filter(lambda t: re.search('cancer', t['name']), all_traits['items'])</span></code></pre>
-
-* @lambda t: re.search('cancer', t['name'])@ is an inline function that takes a parameter @t@ and uses a simple regular expression to test if @t['name']@ contains the substring 'cancer'
-* @all_traits['items']@ is the input sequence of traits
-* @filter@ tests each element @t@ and constructs a new sequence consisting only of the elements that pass the filter
-* @cancer_traits@ gets the result of @filter@
-
-<notextile>
-<pre><code>>>> <span class="userinput">for t in cancer_traits: print(t['uuid'], t['name'])</span>
-...
-qr1hi-q1cn2-8q57g2diohwnzm0 Cervical cancer
-qr1hi-q1cn2-vqp4243janpjbyj Breast cancer
-qr1hi-q1cn2-v6usijujcpwqrn1 Non-melanoma skin cancer
-...
-</code></pre>
-</notextile>
-
-In this tutorial wil will use "Non-melanoma skin cancer" trait with uuid @qr1hi-q1cn2-v6usijujcpwqrn1 at .
-
-notextile. <pre><code>>>> <span class="userinput">non_melanoma_cancer = 'qr1hi-q1cn2-v6usijujcpwqrn1'</code></pre>
-
-h2. Finding humans with the selected trait
-
-We query the "links" resource to find humans that report the selected trait. Links are directional connections between Arvados data items, for example, from a human to their reported traits.
-
-<notextile>
-<pre><code>>>> <span class="userinput">trait_filter = [
- ['link_class', '=', 'human_trait'],
- ['tail_uuid', 'is_a', 'arvados#human'],
- ['head_uuid', '=', non_melanoma_cancer],
- ]
-</code></pre>
-</notextile>
-
-* @['link_class', '=', 'human_trait']@ filters on links that connect phenotype traits to individuals in the database.
-* @['tail_uuid', 'is_a', 'arvados#human']@ filters that the "tail" must be a "human" database object.
-* @['head_uuid', '=', non_melanoma_cancer]@ filters that the "head" of the link must connect to the "trait" database object non_melanoma_cancer .
-
-The query will return links that match all three conditions.
-
-<notextile>
-<pre><code>>>> <span class="userinput">trait_links = arvados.api().links().list(limit=1000, filters=trait_filter).execute()</span>
-</code></pre>
-</notextile>
-
-* @arvados.api()@ gets an object that provides access to the Arvados API server
-* @.links()@ gets an object that provides access to the "links" resource on the Arvados API server
-* @.list(limit=1000, filters=trait_filter)@ constructs a query to elements of the "links" resource that match the criteria discussed above, with a limit of 1000 entries returned
-* @.execute()@ executes the query and returns the result, which we assign to "trait_links"
-
-<notextile>
-<pre><code>>>> <span class="userinput">human_uuids = map(lambda l: l['tail_uuid'], trait_links['items'])</span>
->>> <span class="userinput">human_uuids</span>
-[u'1h9kt-7a9it-c0uqa4kcdh29wdf', u'1h9kt-7a9it-x4tru6mn40hc6ah',
-u'1h9kt-7a9it-yqb8m5s9cpy88i8', u'1h9kt-7a9it-46sm75w200ngwny',
-u'1h9kt-7a9it-gx85a4tdkpzsg3w', u'1h9kt-7a9it-8cvlaa8909lgeo9',
-u'1h9kt-7a9it-as37qum2pq8vizb', u'1h9kt-7a9it-14fph66z2baqxb9',
-u'1h9kt-7a9it-e9zc7i4crmw3v69', u'1h9kt-7a9it-np7f35hlijlxdmt',
-u'1h9kt-7a9it-j9hqyjwbvo9cojn', u'1h9kt-7a9it-lqxdtm1gynmsv13',
-u'1h9kt-7a9it-zkhhxjfg2o22ywq', u'1h9kt-7a9it-nsjoxqd33lzldw9',
-u'1h9kt-7a9it-ytect4smzcgd4kg', u'1h9kt-7a9it-y6tl353b3jc4tos',
-u'1h9kt-7a9it-98f8qave4f8vbs5', u'1h9kt-7a9it-gd72sh15q0p4wq3',
-u'1h9kt-7a9it-zlx25dscak94q9h', u'1h9kt-7a9it-8gronw4rbgmim01',
-u'1h9kt-7a9it-wclfkjcb23tr5es', u'1h9kt-7a9it-rvp2qe7szfz4dy6',
-u'1h9kt-7a9it-50iffhmpzsktwjm', u'1h9kt-7a9it-ul412id5y31a5o8',
-u'1h9kt-7a9it-732kwkfzylmt4ik', u'1h9kt-7a9it-v9zqxegpblsbtai',
-u'1h9kt-7a9it-kmaraqduit1v5wd', u'1h9kt-7a9it-t1nwtlo1hru5vvq',
-u'1h9kt-7a9it-q3w6j9od4ibpoyl', u'1h9kt-7a9it-qz8vzkuuz97ezwv',
-u'1h9kt-7a9it-t1v8sjz6dm9jmjf', u'1h9kt-7a9it-qe8wrbyvuqs5jew']
-</code></pre>
-</notextile>
-
-* @lambda l: l['tail_uuid']@ is an inline function that returns the 'tail_uuid' attribute of 'l'
-* @trait_links['items']@ is the input set from the query
-* @map@ converts each item in a sequence into a different item using the embedded function, in this case to produce a sequence of uuids which refer to humans that have the specified trait.
-
-h2. Find Personal Genome Project identifiers from Arvados UUIDs
-
-<notextile>
-<pre><code>>>> <span class="userinput">human_filters = [
- ["link_class", "=", "identifier"],
- ["head_uuid", "in", human_uuids]
- ]</span>
->>> <span class="userinput">pgpid_links = arvados.api('v1').links().list(limit=1000, filters=human_filters).execute()</span>
->>> <span class="userinput">map(lambda l: l['name'], pgpid_links['items'])</span>
-[u'hu01024B', u'hu11603C', u'hu15402B', u'hu174334', u'hu1BD549', u'hu237A50',
- u'hu34A921', u'hu397733', u'hu414115', u'hu43860C', u'hu474789', u'hu553620',
- u'hu56B3B6', u'hu5917F3', u'hu599905', u'hu5E55F5', u'hu602487', u'hu633787',
- u'hu68F245', u'hu6C3F34', u'hu7260DD', u'hu7A2F1D', u'hu94040B', u'hu9E356F',
- u'huAB8707', u'huB1FD55', u'huB4883B', u'huD09050', u'huD09534', u'huD3A569',
- u'huDF04CC', u'huE2E371']
-</code></pre>
-</notextile>
-
-These PGP IDs let us find public profiles, for example:
-
-* "https://my.pgp-hms.org/profile/huE2E371":https://my.pgp-hms.org/profile/huE2E371
-* "https://my.pgp-hms.org/profile/huDF04CC":https://my.pgp-hms.org/profile/huDF04CC
-* ...
-
-h2. Find genomic data from specific humans
-
-Now we want to find collections in Keep that were provided by these humans. We search the "links" resource for "provenance" links that point to entries in the list of humans with the non-melanoma skin cancer trait:
-
-<notextile>
-<pre><code>>>> <span class="userinput">provenance_links = arvados.api().links().list(limit=1000, filters=[
- ["link_class", "=", "provenance"],
- ["name", "=", "provided"],
- ["tail_uuid", "in", human_uuids]
- ]).execute()
-collection_uuids = map(lambda l: l['head_uuid'], provenance_links['items'])
-
-# build map of human uuid -> PGP ID
-pgpid = {}
-for pgpid_link in pgpid_links['items']:
- pgpid[pgpid_link['head_uuid']] = pgpid_link['name']
-
-# build map of collection uuid -> PGP ID
-for p_link in provenance_links['items']:
- pgpid[p_link['head_uuid']] = pgpid[p_link['tail_uuid']]
-
-# get details (e.g., list of files) of each collection
-collections = arvados.api('v1').collections().list(filters=[
- ["uuid", "in", collection_uuids]
- ]).execute()
-
-# print PGP public profile links with file locators
-for c in collections['items']:
- for f in c['files']:
- print "https://my.pgp-hms.org/profile/%s %s %s%s" % (pgpid[c['uuid']], c['uuid'], ('' if f[0] == '.' else f[0]+'/'), f[1])
-</span>
-https://my.pgp-hms.org/profile/hu43860C a58dca7609fa84c8c38a7e926a97b2fc var-GS00253-DNA_A01_200_37-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/huB1FD55 ea30eb9e46eedf7f05ed6e348c2baf5d var-GS000010320-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/huDF04CC 4ab0df8f22f595d1747a22c476c05873 var-GS000010427-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/hu7A2F1D 756d0ada29b376140f64e7abfe6aa0e7 var-GS000014566-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/hu553620 7ed4e425bb1c7cc18387cbd9388181df var-GS000015272-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/huD09534 542112e210daff30dd3cfea4801a9f2f var-GS000016374-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/hu599905 33a9f3842b01ea3fdf27cc582f5ea2af var-GS000016015-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/hu43860C a58dca7609fa84c8c38a7e926a97b2fc+302 var-GS00253-DNA_A01_200_37-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/huB1FD55 ea30eb9e46eedf7f05ed6e348c2baf5d+291 var-GS000010320-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/huDF04CC 4ab0df8f22f595d1747a22c476c05873+242 var-GS000010427-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/hu7A2F1D 756d0ada29b376140f64e7abfe6aa0e7+242 var-GS000014566-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/hu553620 7ed4e425bb1c7cc18387cbd9388181df+242 var-GS000015272-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/huD09534 542112e210daff30dd3cfea4801a9f2f+242 var-GS000016374-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/hu599905 33a9f3842b01ea3fdf27cc582f5ea2af+242 var-GS000016015-ASM.tsv.bz2
-https://my.pgp-hms.org/profile/hu599905 d6e2e57cd60ba5979006d0b03e45e726+81 Witch_results.zip
-https://my.pgp-hms.org/profile/hu553620 ea4f2d325592a1272f989d141a917fdd+85 Devenwood_results.zip
-https://my.pgp-hms.org/profile/hu7A2F1D 4580f6620bb15b25b18373766e14e4a7+85 Innkeeper_results.zip
-https://my.pgp-hms.org/profile/huD09534 fee37be9440b912eb90f5e779f272416+82 Hallet_results.zip
-</code></pre>
-</notextile>
-
-h3. Search for a variant
-
-Now we will use crunch to issue a 'grep' job to look for variant rs1126809 in each of the "var-" files (these contain variant calls from WGS data).
-
-<notextile>
-<pre><code>>>> <span class="userinput">job = {}
-for c in collections['items']:
- if [] != filter(lambda f: re.search('^var-.*\.tsv\.bz2', f[1]), c['files']):
- job[c['uuid']] = arvados.api('v1').jobs().create(body={
- 'script': 'grep',
- 'script_parameters': {'input': c['uuid'], 'pattern': "rs1126809\\b"},
- 'script_version': 'e7aeb42'
- }).execute()
- print "%s %s" % (pgpid[c['uuid']], job[c['uuid']]['uuid'])
-</span>
-hu43860C qr1hi-8i9sb-wbf3uthbhkcy8ji
-huB1FD55 qr1hi-8i9sb-scklkiy8dc27dab
-huDF04CC qr1hi-8i9sb-pg0w4rfrwfd9srg
-hu7A2F1D qr1hi-8i9sb-n7u0u0rj8b47168
-hu553620 qr1hi-8i9sb-k7gst7vyhg20pt1
-huD09534 qr1hi-8i9sb-4w65pm48123fte5
-hu599905 qr1hi-8i9sb-wmwa5b5r3eghnev
-hu43860C qr1hi-8i9sb-j1mngmakdh8iv9o
-huB1FD55 qr1hi-8i9sb-4j6ehiatcolaoxb
-huDF04CC qr1hi-8i9sb-n6lcmcr3lowqr5u
-hu7A2F1D qr1hi-8i9sb-0hwsdtojfcxjo40
-hu553620 qr1hi-8i9sb-cvvqzqea7jhwb0i
-huD09534 qr1hi-8i9sb-d0y0qtzuwzbrjj0
-hu599905 qr1hi-8i9sb-i9ec9g8d7rt70xg
-</code></pre>
-</notextile>
-
-
-Monitor job progress by refreshing the Jobs page in Workbench, or by using the API:
-
-<notextile>
-<pre><code>>>> <span class="userinput">map(lambda j: arvados.api('v1').jobs().get(uuid=j['uuid']).execute()['success'], job.values())
-[None, True, None, None, None, None, None, None, None, None, None, None, None, None]
-</code></pre>
-</notextile>
-
-Unfinished jobs will appear as None, failed jobs as False, and completed jobs as True.
-
-After the jobs have completed, check output file sizes.
-
-<notextile>
-<pre><code>>>> <span class="userinput">for collection_uuid in job:
- job_uuid = job[collection_uuid]['uuid']
- job_output = arvados.api('v1').jobs().get(uuid=job_uuid).execute()['output']
- output_files = arvados.api('v1').collections().get(uuid=job_output).execute()['files']
- # Test the output size. If greater than zero, that means 'grep' found the variant
- if output_files[0][2] > 0:
- print("%s has variant rs1126809" % (pgpid[collection_uuid]))
- else:
- print("%s does not have variant rs1126809" % (pgpid[collection_uuid]))
-</span>
-hu553620 does not have variant rs1126809
-hu43860C does not have variant rs1126809
-hu599905 has variant rs1126809
-huD09534 has variant rs1126809
-hu553620 does not have variant rs1126809
-huB1FD55 does not have variant rs1126809
-huDF04CC has variant rs1126809
-hu7A2F1D has variant rs1126809
-hu7A2F1D has variant rs1126809
-hu599905 has variant rs1126809
-huDF04CC has variant rs1126809
-huB1FD55 does not have variant rs1126809
-huD09534 has variant rs1126809
-hu43860C does not have variant rs1126809
-</code></pre>
-</notextile>
-
-Thus, of the 14 WGS results available for PGP participants reporting non-melanoma skin cancer, 8 include the rs1126809 variant.
diff --git a/doc/user/tutorials/running-external-program.html.textile.liquid b/doc/user/tutorials/running-external-program.html.textile.liquid
deleted file mode 100644
index a4e58b84b..000000000
--- a/doc/user/tutorials/running-external-program.html.textile.liquid
+++ /dev/null
@@ -1,85 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Writing a pipeline template"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'pipeline_deprecation_notice' %}
-
-This tutorial demonstrates how to construct a two stage pipeline template that uses the "bwa mem":http://bio-bwa.sourceforge.net/ tool to produce a "Sequence Alignment/Map (SAM)":https://samtools.github.io/ file, then uses the "Picard SortSam tool":http://picard.sourceforge.net/command-line-overview.shtml#SortSam to produce a BAM (Binary Alignment/Map) file.
-
-{% include 'tutorial_expectations' %}
-
-Use the following command to create an empty template using @arv create pipeline_template@:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv create pipeline_template</span></code></pre>
-</notextile>
-
-This will open the template record in an interactive text editor (as specified by $EDITOR or $VISUAL, otherwise defaults to @nano@). Now, update the contents of the editor with the following content:
-
-<notextile>{% code 'tutorial_bwa_sortsam_pipeline' as javascript %}</notextile>
-
-* @"name"@ is a human-readable name for the pipeline.
-* @"components"@ is a set of scripts or commands that make up the pipeline. Each component is given an identifier (@"bwa-mem"@ and @"SortSam"@) in this example).
-** Each entry in components @"components"@ is an Arvados job submission. For more information about individual jobs, see the "job resource reference.":{{site.baseurl}}/api/methods/jobs.html
-* @"repository"@, @"script_version"@, and @"script"@ indicate that we intend to use the external @"run-command"@ tool wrapper that is part of the Arvados. These parameters are described in more detail in "Writing a script":tutorial-firstscript.html.
-* @"runtime_constraints"@ describes runtime resource requirements for the component.
-** @"docker_image"@ specifies the "Docker":https://www.docker.com/ runtime environment in which to run the job. The Docker image @"bcosc/arv-base-java"@ supplied here has the Java runtime environment, bwa, and samtools installed.
-** @"arvados_sdk_version"@ specifies a version of the Arvados SDK to load alongside the job's script. The example uses 'master'. If you would like to use a specific version of the sdk, you can find it in the "Arvados Python sdk repository":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*.
-* @"script_parameters"@ describes the component parameters.
-** @"command"@ is the actual command line to invoke the @bwa@ and then @SortSam at . The notation @$()@ denotes macro substitution commands evaluated by the run-command tool wrapper.
-** @"task.stdout"@ indicates that the output of this command should be captured to a file.
-** @$(node.cores)@ evaluates to the number of cores available on the compute node at time the command is run.
-** @$(tmpdir)@ evaluates to the local path for temporary directory the command should use for scratch data.
-** @$(reference_collection)@ evaluates to the script_parameter @"reference_collection"@
-** @$(dir $(...))@ constructs a local path to a directory representing the supplied Arvados collection.
-** @$(file $(...))@ constructs a local path to a given file within the supplied Arvados collection.
-** @$(glob $(...))@ searches the specified path based on a file glob pattern and evalutes to the first result.
-** @$(basename $(...))@ evaluates to the supplied path with leading path portion and trailing filename extensions stripped
-* @"output_of"@ indicates that the @output@ of the @bwa-mem@ component should be used as the @"input"@ script parameter of @SortSam at . Arvados uses these dependencies between components to automatically determine the correct order to run them.
-
-When using @run-command@, the tool should write its output to the current working directory. The output will be automatically uploaded to Keep when the job completes.
-
-See the "run-command reference":{{site.baseurl}}/user/topics/run-command.html for more information about using @run-command at .
-
-*Note:* When trying to get job reproducibility without re-computation, you need to set these parameters to their specific hashes. Using a version such as master in @"arvados_sdk_version"@ will grab the latest version hash, which will allow Arvados to re-compute your job if the sdk gets updated.
-* @"arvados_sdk_version"@ : The latest version can be found on the "Arvados Python sdk repository":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*.
-* @"script_version"@ : The current version of your script in your git repository can be found by using the following command:
-
-<notextile>
-<pre><code>~$ <span class="userinput">git rev-parse HEAD</span></code></pre>
-</notextile>
-
-* @"docker_image"@ : The docker image hash used is found on the "Collection page":https://playground.arvados.org/collections/qr1hi-4zz18-dov6im679g3jr1n as the *Content address*.
-
-h2. Running your pipeline
-
-Your new pipeline template should appear at the top of the Workbench "pipeline templates":{{site.arvados_workbench_host}}/pipeline_templates page. You can run your pipeline "using Workbench":tutorial-workflow-workbench.html or the "command line.":{{site.baseurl}}/user/topics/running-pipeline-command-line.html
-
-Test data is available in the "Arvados Tutorial":{{site.arvados_workbench_host}}/projects/qr1hi-j7d0g-u7zg1qdaowykd8d project:
-
-* Choose <i class="fa fa-fw fa-archive"></i> "Tutorial chromosome 19 reference (2463fa9efeb75e099685528b3b9071e0+438)":{{site.arvados_workbench_host}}/collections/2463fa9efeb75e099685528b3b9071e0+438 for the "reference_collection" parameter
-* Choose <i class="fa fa-fw fa-archive"></i> "Tutorial sample exome (3229739b505d2b878b62aed09895a55a+142)":{{site.arvados_workbench_host}}/collections/3229739b505d2b878b62aed09895a55a+142 for the "sample" parameter
-
-For more information and examples for writing pipelines, see the "pipeline template reference":{{site.baseurl}}/api/methods/pipeline_templates.html
-
-h2. Re-using your pipeline run
-
-Arvados allows users to re-use jobs that have the same inputs in order to save computing time and resources. Users are able to change a job downstream without re-computing earlier jobs. This section shows which version control parameters should be tuned to make sure Arvados will not re-compute your jobs.
-
-Note: Job reuse can only happen if all input collections do not change.
-
-* @"arvados_sdk_version"@ : The arvados_sdk_version parameter is used to download the specific version of the Arvados sdk into the docker image. The latest version can be found in the "Arvados Python sdk repository":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*. Make sure you set this to the same version as the previous run that you are trying to reuse.
-* @"script_version"@ : The script_version is the commit hash of the git branch that the crunch script resides in. This information can be found in your git repository by using the following command:
-
-<notextile>
-<pre><code>~$ <span class="userinput">git rev-parse HEAD</span></code></pre>
-</notextile>
-
-* @"docker_image"@ : This specifies the "Docker":https://www.docker.com/ runtime environment where jobs run their scripts. Docker version control is similar to git, and you can commit and push changes to your images. You must re-use the docker image hash from the previous run to use the same image. It can be found on the "Collection page":https://playground.arvados.org/collections/qr1hi-4zz18-dov6im679g3jr1n as the *Content address* or the *docker_image_locator* in a job's metadata.
diff --git a/doc/user/tutorials/tutorial-firstscript.html.textile.liquid b/doc/user/tutorials/tutorial-firstscript.html.textile.liquid
deleted file mode 100644
index 393769847..000000000
--- a/doc/user/tutorials/tutorial-firstscript.html.textile.liquid
+++ /dev/null
@@ -1,112 +0,0 @@
----
-layout: default
-navsection: userguide
-navmenu: Tutorials
-title: "Writing a Crunch script"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'pipeline_deprecation_notice' %}
-
-This tutorial demonstrates how to write a script using Arvados Python SDK. The Arvados SDK supports access to advanced features not available using the @run-command@ wrapper, such as scheduling concurrent tasks across nodes.
-
-{% include 'tutorial_expectations' %}
-
-This tutorial uses @$USER@ to denote your username. Replace @$USER@ with your user name in all the following examples.
-
-Start by creating a directory called @tutorial@ in your home directory. Next, create a subdirectory called @crunch_scripts@ and change to that directory:
-
-<notextile>
-<pre><code>~$ <span class="userinput">cd $HOME</span>
-~$ <span class="userinput">mkdir -p tutorial/crunch_scripts</span>
-~$ <span class="userinput">cd tutorial/crunch_scripts</span></code></pre>
-</notextile>
-
-Next, using @nano@ or your favorite Unix text editor, create a new file called @hash.py@ in the @crunch_scripts@ directory.
-
-notextile. <pre>~/tutorial/crunch_scripts$ <code class="userinput">nano hash.py</code></pre>
-
-Add the following code to compute the MD5 hash of each file in a collection:
-
-<notextile> {% code 'tutorial_hash_script_py' as python %} </notextile>
-
-Make the file executable:
-
-notextile. <pre><code>~/tutorial/crunch_scripts$ <span class="userinput">chmod +x hash.py</span></code></pre>
-
-Next, create a submission job record. This describes a specific invocation of your script:
-
-<notextile>
-<pre><code>~/tutorial/crunch_scripts$ <span class="userinput">cat >~/the_job <<EOF
-{
- "repository":"",
- "script":"hash.py",
- "script_version":"$HOME/tutorial",
- "script_parameters":{
- "input":"c1bad4b39ca5a924e481008009d94e32+210"
- }
-}
-EOF</span>
-</code></pre>
-</notextile>
-
-You can now run your script on your local workstation or VM using @arv-crunch-job@:
-
-<notextile>
-<pre><code>~/tutorial/crunch_scripts</span>$ <span class="userinput">arv-crunch-job --job "$(cat ~/the_job)"</span>
-2014-08-06_15:16:22 qr1hi-8i9sb-qyrat80ef927lam 14473 check slurm allocation
-2014-08-06_15:16:22 qr1hi-8i9sb-qyrat80ef927lam 14473 node localhost - 1 slots
-2014-08-06_15:16:23 qr1hi-8i9sb-qyrat80ef927lam 14473 start
-2014-08-06_15:16:23 qr1hi-8i9sb-qyrat80ef927lam 14473 script hash.py
-2014-08-06_15:16:23 qr1hi-8i9sb-qyrat80ef927lam 14473 script_version $HOME/tutorial
-2014-08-06_15:16:23 qr1hi-8i9sb-qyrat80ef927lam 14473 script_parameters {"input":"c1bad4b39ca5a924e481008009d94e32+210"}
-2014-08-06_15:16:23 qr1hi-8i9sb-qyrat80ef927lam 14473 runtime_constraints {"max_tasks_per_node":0}
-2014-08-06_15:16:23 qr1hi-8i9sb-qyrat80ef927lam 14473 start level 0
-2014-08-06_15:16:23 qr1hi-8i9sb-qyrat80ef927lam 14473 status: 0 done, 0 running, 1 todo
-2014-08-06_15:16:23 qr1hi-8i9sb-qyrat80ef927lam 14473 0 job_task qr1hi-ot0gb-lptn85mwkrn9pqo
-2014-08-06_15:16:23 qr1hi-8i9sb-qyrat80ef927lam 14473 0 child 14478 started on localhost.1
-2014-08-06_15:16:23 qr1hi-8i9sb-qyrat80ef927lam 14473 status: 0 done, 1 running, 0 todo
-2014-08-06_15:16:24 qr1hi-8i9sb-qyrat80ef927lam 14473 0 stderr crunchstat: Running [stdbuf --output=0 --error=0 /home/$USER/tutorial/crunch_scripts/hash.py]
-2014-08-06_15:16:24 qr1hi-8i9sb-qyrat80ef927lam 14473 0 child 14478 on localhost.1 exit 0 signal 0 success=true
-2014-08-06_15:16:24 qr1hi-8i9sb-qyrat80ef927lam 14473 0 success in 1 seconds
-2014-08-06_15:16:24 qr1hi-8i9sb-qyrat80ef927lam 14473 0 output
-2014-08-06_15:16:25 qr1hi-8i9sb-qyrat80ef927lam 14473 wait for last 0 children to finish
-2014-08-06_15:16:25 qr1hi-8i9sb-qyrat80ef927lam 14473 status: 1 done, 0 running, 1 todo
-2014-08-06_15:16:25 qr1hi-8i9sb-qyrat80ef927lam 14473 start level 1
-2014-08-06_15:16:25 qr1hi-8i9sb-qyrat80ef927lam 14473 status: 1 done, 0 running, 1 todo
-2014-08-06_15:16:25 qr1hi-8i9sb-qyrat80ef927lam 14473 1 job_task qr1hi-ot0gb-e3obm0lv6k6p56a
-2014-08-06_15:16:25 qr1hi-8i9sb-qyrat80ef927lam 14473 1 child 14504 started on localhost.1
-2014-08-06_15:16:25 qr1hi-8i9sb-qyrat80ef927lam 14473 status: 1 done, 1 running, 0 todo
-2014-08-06_15:16:26 qr1hi-8i9sb-qyrat80ef927lam 14473 1 stderr crunchstat: Running [stdbuf --output=0 --error=0 /home/$USER/tutorial/crunch_scripts/hash.py]
-2014-08-06_15:16:35 qr1hi-8i9sb-qyrat80ef927lam 14473 1 child 14504 on localhost.1 exit 0 signal 0 success=true
-2014-08-06_15:16:35 qr1hi-8i9sb-qyrat80ef927lam 14473 1 success in 10 seconds
-2014-08-06_15:16:35 qr1hi-8i9sb-qyrat80ef927lam 14473 1 output 8c20281b9840f624a486e4f1a78a1da8+105+A234be74ceb5ea31db6e11b6be26f3eb76d288ad0 at 54987018
-2014-08-06_15:16:35 qr1hi-8i9sb-qyrat80ef927lam 14473 wait for last 0 children to finish
-2014-08-06_15:16:35 qr1hi-8i9sb-qyrat80ef927lam 14473 status: 2 done, 0 running, 0 todo
-2014-08-06_15:16:35 qr1hi-8i9sb-qyrat80ef927lam 14473 release job allocation
-2014-08-06_15:16:35 qr1hi-8i9sb-qyrat80ef927lam 14473 Freeze not implemented
-2014-08-06_15:16:35 qr1hi-8i9sb-qyrat80ef927lam 14473 collate
-2014-08-06_15:16:36 qr1hi-8i9sb-qyrat80ef927lam 14473 collated output manifest text to send to API server is 105 bytes with access tokens
-2014-08-06_15:16:36 qr1hi-8i9sb-qyrat80ef927lam 14473 output hash c1b44b6dc41ef334cf1136033ca950e6+54
-2014-08-06_15:16:37 qr1hi-8i9sb-qyrat80ef927lam 14473 finish
-2014-08-06_15:16:38 qr1hi-8i9sb-qyrat80ef927lam 14473 log manifest is 7fe8cf1d45d438a3ca3ac4a184b7aff4+83
-</code></pre>
-</notextile>
-
-Although the job runs locally, the output of the job has been saved to Keep, the Arvados file store. The "output hash" line (third from the bottom) provides the portable data hash of the Arvados collection where the script's output has been saved. Copy the output hash and use @arv-ls@ to list the contents of your output collection, and @arv-get@ to download it to the current directory:
-
-<notextile>
-<pre><code>~/tutorial/crunch_scripts$ <span class="userinput">arv-ls c1b44b6dc41ef334cf1136033ca950e6+54</span>
-./md5sum.txt
-~/tutorial/crunch_scripts$ <span class="userinput">arv-get c1b44b6dc41ef334cf1136033ca950e6+54/ .</span>
-0 MiB / 0 MiB 100.0%
-~/tutorial/crunch_scripts$ <span class="userinput">cat md5sum.txt</span>
-44b8ae3fde7a8a88d2f7ebd237625b4f c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2
-</code></pre>
-</notextile>
-
-Running locally is convenient for development and debugging, as it permits a fast iterative development cycle. Your job run is also recorded by Arvados, and will appear in the *Recent jobs and pipelines* panel on the "Workbench Dashboard":{{site.arvados_workbench_host}}. This provides limited provenance, by recording the input parameters, the execution log, and the output. However, running locally does not allow you to scale out to multiple nodes, and does not store the complete system snapshot required to achieve reproducibility; to do that you need to "submit a job to the Arvados cluster":{{site.baseurl}}/user/tutorials/tutorial-submit-job.html.
diff --git a/doc/user/tutorials/tutorial-submit-job.html.textile.liquid b/doc/user/tutorials/tutorial-submit-job.html.textile.liquid
deleted file mode 100644
index ff78aabb6..000000000
--- a/doc/user/tutorials/tutorial-submit-job.html.textile.liquid
+++ /dev/null
@@ -1,95 +0,0 @@
----
-layout: default
-navsection: userguide
-navmenu: Tutorials
-title: "Running on an Arvados cluster"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'pipeline_deprecation_notice' %}
-
-This tutorial demonstrates how to create a pipeline to run your crunch script on an Arvados cluster. Cluster jobs can scale out to multiple nodes, and use @git@ and @docker@ to store the complete system snapshot required to achieve reproducibilty.
-
-{% include 'tutorial_expectations' %}
-
-This tutorial uses @$USER@ to denote your username. Replace @$USER@ with your user name in all the following examples.
-
-Also, this tutorial uses the @tutorial@ arvados repository created in "Adding a new arvados repository":add-new-repository.html as the example repository.
-
-h2. Clone Arvados repository
-
-Please clone the *tutorial* repository using the instructions from "Working with Arvados git repository":git-arvados-guide.html, if you have not yet cloned already.
-
-h2. Creating a Crunch script
-
-Start by entering the @tutorial@ directory created by @git clone at . Next, create a subdirectory called @crunch_scripts@ and change to that directory:
-
-<notextile>
-<pre><code>>~$ <span class="userinput">cd tutorial</span>
-~/tutorial$ <span class="userinput">mkdir crunch_scripts</span>
-~/tutorial$ <span class="userinput">cd crunch_scripts</span></code></pre>
-</notextile>
-
-Next, using @nano@ or your favorite Unix text editor, create a new file called @hash.py@ in the @crunch_scripts@ directory.
-
-notextile. <pre>~/tutorial/crunch_scripts$ <code class="userinput">nano hash.py</code></pre>
-
-Add the following code to compute the MD5 hash of each file in a collection (if you already completed "Writing a Crunch script":tutorial-firstscript.html you can just copy the @hash.py@ file you created previously.)
-
-<notextile> {% code 'tutorial_hash_script_py' as python %} </notextile>
-
-Make the file executable:
-
-notextile. <pre><code>~/tutorial/crunch_scripts$ <span class="userinput">chmod +x hash.py</span></code></pre>
-
-Next, add the file to the staging area. This tells @git@ that the file should be included on the next commit.
-
-notextile. <pre><code>~/tutorial/crunch_scripts$ <span class="userinput">git add hash.py</span></code></pre>
-
-Next, commit your changes. All staged changes are recorded into the local git repository:
-
-<notextile>
-<pre><code>~/tutorial/crunch_scripts$ <span class="userinput">git commit -m "my first script"</span>
-[master (root-commit) 27fd88b] my first script
- 1 file changed, 45 insertions(+)
- create mode 100755 crunch_scripts/hash.py</code></pre>
-</notextile>
-
-Finally, upload your changes to the Arvados server:
-
-<notextile>
-<pre><code>~/tutorial/crunch_scripts$ <span class="userinput">git push origin master</span>
-Counting objects: 4, done.
-Compressing objects: 100% (2/2), done.
-Writing objects: 100% (4/4), 682 bytes, done.
-Total 4 (delta 0), reused 0 (delta 0)
-To git at git.qr1hi.arvadosapi.com:$USER/tutorial.git
- * [new branch] master -> master</code></pre>
-</notextile>
-
-h2. Create a pipeline template
-
-Next, create a new template using @arv create pipeline_template@:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv create pipeline_template</span></code></pre>
-</notextile>
-
-In the editor, enter the following template:
-
-<notextile> {% code 'tutorial_submit_job' as javascript %} </notextile>
-
-* @"repository"@ is the name of a git repository to search for the script version. You can access a list of available git repositories on the Arvados Workbench in the *Repositories* page using the <span class="fa fa-lg fa-user"></span> <span class="caret"></span> top navigation menu icon.
-* @"script_version"@ specifies the version of the script that you wish to run. This can be in the form of an explicit Git revision hash, a tag, or a branch (in which case it will use the HEAD of the specified branch). Arvados logs the script version that was used in the run, enabling you to go back and re-run any past job with the guarantee that the exact same code will be used as was used in the previous run.
-* @"script"@ specifies the filename of the script to run. Crunch expects to find this in the @crunch_scripts/@ subdirectory of the Git repository.
-* @"runtime_constraints"@ describes the runtime environment required to run the job. These are described in the "job record schema":{{site.baseurl}}/api/methods/jobs.html
-
-h2. Running your pipeline
-
-Your new pipeline template should appear at the top of the Workbench "pipeline templates":{{site.arvados_workbench_host}}/pipeline_templates page. You can run your pipeline "using Workbench":tutorial-workflow-workbench.html or the "command line.":{{site.baseurl}}/user/topics/running-pipeline-command-line.html
-
-For more information and examples for writing pipelines, see the "pipeline template reference":{{site.baseurl}}/api/methods/pipeline_templates.html
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list