[ARVADOS] created: 5f1e8bf76f883cb0a779cf2a4eafc9cc34457bcc
Git user
git at public.curoverse.com
Fri Jul 1 10:10:27 EDT 2016
at 5f1e8bf76f883cb0a779cf2a4eafc9cc34457bcc (commit)
commit 5f1e8bf76f883cb0a779cf2a4eafc9cc34457bcc
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Jul 1 10:10:21 2016 -0400
9369: arvados-cwl-runner examples for documentation
diff --git a/doc/user/cwl/cwl-runner.html.textile.liquid b/doc/user/cwl/cwl-runner.html.textile.liquid
index 47e7d5f..a00a6a4 100644
--- a/doc/user/cwl/cwl-runner.html.textile.liquid
+++ b/doc/user/cwl/cwl-runner.html.textile.liquid
@@ -3,3 +3,97 @@ layout: default
navsection: userguide
title: Running a CWL workflow
...
+
+Arvados supports workflows written using the "Common Workflow Language, draft-3":http://commonwl.org/draft-3 .
+
+{% include 'tutorial_expectations' %}
+
+Use @arvados-cwl-runner@ to submit CWL workflows to Arvados. After submitting
+the job, it will wait for the workflow to complete and print out the final
+result to standard output. Note that once submitted, the workflow runs
+entirely on Arvados, so even if you interrupt @arvados-cwl-runner@ or log out,
+the workflow will continue to run.
+
+h2. Submitting workflows
+
+<pre>
+$ git clone https://github.com/curoverse/cwl-tutorial.git
+$ cd cwl-tutorial/bwa-mem
+$ arvados-cwl-runner bwa-mem.cwl bwa-mem-input.yml
+/home/peter/work/scripts/venv/bin/arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Upload local files: "bwa-mem.cwl"
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to 3d0ga-4zz18-h7ljh5u76760ww2
+2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job 3d0ga-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (3d0ga-8i9sb-fm2n3b1w0l6bskg) is Running
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (3d0ga-8i9sb-fm2n3b1w0l6bskg) is Complete
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Overall process status is success
+{
+ "aligned_sam": {
+ "path": "keep:54325254b226664960de07b3b9482349+154/HWI-ST1027_129_D0THKACXX.1_1.sam",
+ "checksum": "sha1$0dc46a3126d0b5d4ce213b5f0e86e2d05a54755a",
+ "class": "File",
+ "size": 30738986
+ }
+}
+</pre>
+
+To submit a workflow and exit immediately, use the @--no-wait@ option. This
+will print out the uuid of the job that was submitted to standard output.
+
+<pre>
+$ arvados-cwl-runner --no-wait bwa-mem.cwl bwa-mem-input.yml
+/home/peter/work/scripts/venv/bin/arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
+2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Upload local files: "bwa-mem.cwl"
+2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Uploaded to 3d0ga-4zz18-eqnfwrow8aysa9q
+2016-06-30 15:07:52 arvados.cwl-runner[12480] INFO: Submitted job 3d0ga-8i9sb-fm2n3b1w0l6bskg
+3d0ga-8i9sb-fm2n3b1w0l6bskg
+</pre>
+
+To run a workflow with local control, use @--local at . This means that the host
+where you run @arvados-cwl-runner@ will be responsible for submitting jobs.
+With @--local@, if you interrupt @arvados-cwl-runner@ or log out, the workflow
+will be terminated.
+
+<pre>
+$ arvados-cwl-runner --local bwa-mem.cwl bwa-mem-input.yml
+/home/peter/work/scripts/venv/bin/arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
+2016-07-01 10:05:19 arvados.cwl-runner[16290] INFO: Pipeline instance 3d0ga-d1hrv-92wcu6ldtio74r4
+2016-07-01 10:05:28 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (3d0ga-8i9sb-2nzzfbuf9zjrj4g) is Queued
+2016-07-01 10:05:29 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (3d0ga-8i9sb-2nzzfbuf9zjrj4g) is Running
+2016-07-01 10:05:45 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (3d0ga-8i9sb-2nzzfbuf9zjrj4g) is Complete
+2016-07-01 10:05:46 arvados.cwl-runner[16290] INFO: Overall process status is success
+{
+ "aligned_sam": {
+ "size": 30738986,
+ "path": "keep:15f56bad0aaa7364819bf14ca2a27c63+88/HWI-ST1027_129_D0THKACXX.1_1.sam",
+ "checksum": "sha1$0dc46a3126d0b5d4ce213b5f0e86e2d05a54755a",
+ "class": "File"
+ }
+}
+(venv)peter at debian:[pts/0]:~/work/arv-cwl-tutorial/bwa-mem [master]
+</pre>
+
+h2. Job reuse
+
+When @arvados-cwl-runner@ will advantage of Arvados job reuse. If you submit a
+workflow and inputs that are identical to one that has run before, it will
+short cut the execution and return the result of the previous job. This also
+applies to individual workflow steps. You can disable this behavior with
+ at --disable-reuse@.
+
+<pre>
+$ arvados-cwl-runner bwa-mem.cwl bwa-mem-input.yml
+/home/peter/work/scripts/venv/bin/arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
+2016-07-01 10:06:48 arvados.arv-run[18370] INFO: Upload local files: "bwa-mem.cwl"
+2016-07-01 10:06:48 arvados.arv-run[18370] INFO: Uploaded to 3d0ga-4zz18-3oc7dfkxqv3q5y7
+2016-07-01 10:06:48 arvados.cwl-runner[18370] INFO: Submitted job 3d0ga-8i9sb-wn38vyacwz0odzn
+2016-07-01 10:06:48 arvados.cwl-runner[18370] INFO: Overall process status is success
+{
+ "aligned_sam": {
+ "path": "keep:54325254b226664960de07b3b9482349+154/HWI-ST1027_129_D0THKACXX.1_1.sam",
+ "checksum": "sha1$0dc46a3126d0b5d4ce213b5f0e86e2d05a54755a",
+ "class": "File",
+ "size": 30738986
+ }
+}
+</pre>
commit e1575a5225a290ad62b452a2389c0a4b9a11a1bc
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jun 30 13:31:46 2016 -0400
9369: add deprecation notices
diff --git a/doc/_config.yml b/doc/_config.yml
index 4115bc9..df1d67d 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -43,13 +43,17 @@ navbar:
- user/tutorials/tutorial-keep-mount.html.textile.liquid
- user/topics/keep.html.textile.liquid
- user/topics/arv-copy.html.textile.liquid
- - Run a pipeline on the command line:
+ - Working on the command line:
+ - user/cwl/cwl-runner.html.textile.liquid
- user/topics/running-pipeline-command-line.html.textile.liquid
- user/topics/arv-run.html.textile.liquid
- - Working with Arvados Repositories:
+ - Working with git repositories:
- user/tutorials/add-new-repository.html.textile.liquid
- user/tutorials/git-arvados-guide.html.textile.liquid
- - Develop a new pipeline:
+ - Develop a CWL workflow:
+ - user/tutorials/intro-crunch.html.textile.liquid
+ - user/cwl/intro-cwl.html.textile.liquid
+ - Develop an Arvados pipeline:
- user/tutorials/intro-crunch.html.textile.liquid
- user/tutorials/running-external-program.html.textile.liquid
- user/topics/crunch-tools-overview.html.textile.liquid
diff --git a/doc/_includes/_pipeline_deprecation_notice.liquid b/doc/_includes/_pipeline_deprecation_notice.liquid
new file mode 100644
index 0000000..2fcb2e7
--- /dev/null
+++ b/doc/_includes/_pipeline_deprecation_notice.liquid
@@ -0,0 +1,3 @@
+{% include 'notebox_begin' %}
+Arvados pipeline templates are deprecated. The recommend way to develop new workflows for Arvados is using the "Common Workflow Language":{{site.baseurl}}/user/cwl/intro-cwl.html
+{% include 'notebox_end' %}
diff --git a/doc/user/cwl/cwl-runner.html.textile.liquid b/doc/user/cwl/cwl-runner.html.textile.liquid
new file mode 100644
index 0000000..47e7d5f
--- /dev/null
+++ b/doc/user/cwl/cwl-runner.html.textile.liquid
@@ -0,0 +1,5 @@
+---
+layout: default
+navsection: userguide
+title: Running a CWL workflow
+...
diff --git a/doc/user/cwl/intro-cwl.html.textile.liquid b/doc/user/cwl/intro-cwl.html.textile.liquid
new file mode 100644
index 0000000..af7ce23
--- /dev/null
+++ b/doc/user/cwl/intro-cwl.html.textile.liquid
@@ -0,0 +1,5 @@
+---
+layout: default
+navsection: userguide
+title: Introduction to Common Workflow Language
+...
diff --git a/doc/user/topics/crunch-tools-overview.html.textile.liquid b/doc/user/topics/crunch-tools-overview.html.textile.liquid
index 994f437..0e19be1 100644
--- a/doc/user/topics/crunch-tools-overview.html.textile.liquid
+++ b/doc/user/topics/crunch-tools-overview.html.textile.liquid
@@ -4,6 +4,8 @@ navsection: userguide
title: "Tools for writing Crunch pipelines"
...
+{% include 'pipeline_deprecation_notice' %}
+
Arvados includes a number of tools to help you develop pipelines and jobs for Crunch. This overview explains each tool's intended use to help you choose the right one.
h2. Use the "arv-run command-line utility":arv-run.html
diff --git a/doc/user/topics/running-pipeline-command-line.html.textile.liquid b/doc/user/topics/running-pipeline-command-line.html.textile.liquid
index 9f10fe4..9a2e12c 100644
--- a/doc/user/topics/running-pipeline-command-line.html.textile.liquid
+++ b/doc/user/topics/running-pipeline-command-line.html.textile.liquid
@@ -1,7 +1,7 @@
---
layout: default
navsection: userguide
-title: "Running a pipeline on the command line"
+title: "Running an Arvados pipeline"
...
This tutorial demonstrates how to use the command line to run the same pipeline as described in "running a pipeline using Workbench.":{{site.baseurl}}/user/tutorials/tutorial-pipeline-workbench.html
diff --git a/doc/user/topics/tutorial-parallel.html.textile.liquid b/doc/user/topics/tutorial-parallel.html.textile.liquid
index 6d0058b..3a7f85c 100644
--- a/doc/user/topics/tutorial-parallel.html.textile.liquid
+++ b/doc/user/topics/tutorial-parallel.html.textile.liquid
@@ -4,6 +4,8 @@ navsection: userguide
title: "Concurrent Crunch tasks"
...
+{% include 'pipeline_deprecation_notice' %}
+
In the previous tutorials, we used @arvados.job_setup.one_task_per_input_file()@ to automatically create concurrent jobs by creating a separate task per file. For some types of jobs, you may need to split the work up differently, for example creating tasks to process different segments of a single large file. This tutorial will demonstrate how to create Crunch tasks directly.
Start by entering the @crunch_scripts@ directory of your Git repository:
diff --git a/doc/user/tutorials/running-external-program.html.textile.liquid b/doc/user/tutorials/running-external-program.html.textile.liquid
index 90dc197..ef4634e 100644
--- a/doc/user/tutorials/running-external-program.html.textile.liquid
+++ b/doc/user/tutorials/running-external-program.html.textile.liquid
@@ -4,6 +4,8 @@ navsection: userguide
title: "Writing a pipeline template"
...
+{% include 'pipeline_deprecation_notice' %}
+
This tutorial demonstrates how to construct a two stage pipeline template that uses the "bwa mem":http://bio-bwa.sourceforge.net/ tool to produce a "Sequence Alignment/Map (SAM)":https://samtools.github.io/ file, then uses the "Picard SortSam tool":http://picard.sourceforge.net/command-line-overview.shtml#SortSam to produce a BAM (Binary Alignment/Map) file.
{% include 'tutorial_expectations' %}
@@ -64,7 +66,7 @@ For more information and examples for writing pipelines, see the "pipeline templ
h2. Re-using your pipeline run
-Arvados allows users to re-use jobs that have the same inputs in order to save computing time and resources. Users are able to change a job downstream without re-computing earlier jobs. This section shows which version control parameters should be tuned to make sure Arvados will not re-compute your jobs.
+Arvados allows users to re-use jobs that have the same inputs in order to save computing time and resources. Users are able to change a job downstream without re-computing earlier jobs. This section shows which version control parameters should be tuned to make sure Arvados will not re-compute your jobs.
Note: Job reuse can only happen if all input collections do not change.
diff --git a/doc/user/tutorials/tutorial-firstscript.html.textile.liquid b/doc/user/tutorials/tutorial-firstscript.html.textile.liquid
index bf73c8c..d4caafe 100644
--- a/doc/user/tutorials/tutorial-firstscript.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-firstscript.html.textile.liquid
@@ -5,6 +5,8 @@ navmenu: Tutorials
title: "Writing a Crunch script"
...
+{% include 'pipeline_deprecation_notice' %}
+
This tutorial demonstrates how to write a script using Arvados Python SDK. The Arvados SDK supports access to advanced features not available using the @run-command@ wrapper, such as scheduling concurrent tasks across nodes.
{% include 'tutorial_expectations' %}
diff --git a/doc/user/tutorials/tutorial-submit-job.html.textile.liquid b/doc/user/tutorials/tutorial-submit-job.html.textile.liquid
index b17f951..47e8dc7 100644
--- a/doc/user/tutorials/tutorial-submit-job.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-submit-job.html.textile.liquid
@@ -5,6 +5,8 @@ navmenu: Tutorials
title: "Running on an Arvados cluster"
...
+{% include 'pipeline_deprecation_notice' %}
+
This tutorial demonstrates how to create a pipeline to run your crunch script on an Arvados cluster. Cluster jobs can scale out to multiple nodes, and use @git@ and @docker@ to store the complete system snapshot required to achieve reproducibilty.
{% include 'tutorial_expectations' %}
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list