[ARVADOS] updated: 3da0fb0eef35b665c9c427fa9acf9b932dd45062
Git user
git at public.curoverse.com
Tue Dec 13 15:46:50 EST 2016
Summary of changes:
doc/_config.yml | 1 +
doc/user/cwl/cwl-run-options.html.textile.liquid | 94 ++++++++++++++++++++++
doc/user/cwl/cwl-runner.html.textile.liquid | 89 +-------------------
.../writing-cwl-workflow.html.textile.liquid | 8 +-
sdk/cwl/arvados_cwl/fsaccess.py | 9 ++-
5 files changed, 108 insertions(+), 93 deletions(-)
create mode 100644 doc/user/cwl/cwl-run-options.html.textile.liquid
via 3da0fb0eef35b665c9c427fa9acf9b932dd45062 (commit)
from 474ed0b55fcf7f8c9d6c195df829ef9b2ecac798 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 3da0fb0eef35b665c9c427fa9acf9b932dd45062
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Tue Dec 13 15:46:40 2016 -0500
10576: Add ability to run pipeline templates containing CWL from command line.
Split out command line documentation from running CWL tutorial.
diff --git a/doc/_config.yml b/doc/_config.yml
index fbe95fa..97f45db 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -45,6 +45,7 @@ navbar:
- user/topics/arv-copy.html.textile.liquid
- Running workflows at the command line:
- user/cwl/cwl-runner.html.textile.liquid
+ - user/cwl/cwl-run-options.html.textile.liquid
- user/topics/running-pipeline-command-line.html.textile.liquid
- user/topics/arv-run.html.textile.liquid
- Working with git repositories:
diff --git a/doc/user/cwl/cwl-run-options.html.textile.liquid b/doc/user/cwl/cwl-run-options.html.textile.liquid
new file mode 100644
index 0000000..fea1e51
--- /dev/null
+++ b/doc/user/cwl/cwl-run-options.html.textile.liquid
@@ -0,0 +1,94 @@
+---
+layout: default
+navsection: userguide
+title: "Using arvados-cwl-runner"
+...
+
+The following command line options are available for @arvados-cwl-runner@:
+
+table(table table-bordered table-condensed).
+|_. Option |_. Description |
+| --basedir BASEDIR| Base directory used to resolve relative references in the input, default to directory of input object file or current directory (if inputs piped/provided on command line).|
+| --version| Print version and exit|
+| --verbose| Default logging|
+| --quiet| Only print warnings and errors.|
+| --debug| Print even more logging|
+| --tool-help| Print command line help for tool|
+| --enable-reuse|Enable job reuse (default)|
+| --disable-reuse|Disable job reuse (always run new jobs).|
+| --project-uuid UUID| Project that will own the workflow jobs, if not provided, will go to home project.|
+| --output-name OUTPUT_NAME|Name to use for collection that stores the final output.|
+| --output-tags OUTPUT_TAGS|Tags for the final output collection separated by commas, e.g., '--output-tags tag0,tag1,tag2'.|
+| --ignore-docker-for-reuse|Ignore Docker image version when deciding whether to reuse past jobs.|
+| --submit| Submit workflow to run on Arvados.|
+| --local| Control workflow from local host (submits jobs to Arvados).|
+| --create-template| (Deprecated) synonym for --create-workflow.|
+| --create-workflow| Create an Arvados workflow (if using the 'containers' API) or pipeline template (if using the 'jobs' API). See --api.|
+| --update-workflow UUID|Update an existing Arvados workflow or pipeline template with the given UUID.|
+| --wait| After submitting workflow runner job, wait for completion.|
+| --no-wait| Submit workflow runner job and exit.|
+| --api WORK_API| Select work submission API, one of 'jobs' or 'containers'. Default is 'jobs' if that API is available, otherwise 'containers'.|
+| --compute-checksum| Compute checksum of contents while collecting outputs|
+| --submit-runner-ram SUBMIT_RUNNER_RAM|RAM (in MiB) required for the workflow runner job (default 1024)|
+| --name NAME| Name to use for workflow execution instance.|
+
+h3. Specify workflow and output names
+
+Use the @--name@ and @--output-name@ options to specify the name of the workflow and name of the output collection.
+
+<notextile>
+<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --name "Example bwa run" --output-name "Example bwa output" bwa-mem.cwl bwa-mem-input.yml</span>
+arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Upload local files: "bwa-mem.cwl"
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to qr1hi-4zz18-h7ljh5u76760ww2
+2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Running
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Complete
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Overall process status is success
+{
+ "aligned_sam": {
+ "path": "keep:54325254b226664960de07b3b9482349+154/HWI-ST1027_129_D0THKACXX.1_1.sam",
+ "checksum": "sha1$0dc46a3126d0b5d4ce213b5f0e86e2d05a54755a",
+ "class": "File",
+ "size": 30738986
+ }
+}
+</code></pre>
+</notextile>
+
+h3. Submit a workflow with no waiting
+
+To submit a workflow and exit immediately, use the @--no-wait@ option. This will submit the workflow to Arvados, print out the UUID of the job that was submitted to standard output, and exit.
+
+<notextile>
+<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --no-wait bwa-mem.cwl bwa-mem-input.yml</span>
+arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
+2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Upload local files: "bwa-mem.cwl"
+2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Uploaded to qr1hi-4zz18-eqnfwrow8aysa9q
+2016-06-30 15:07:52 arvados.cwl-runner[12480] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
+qr1hi-8i9sb-fm2n3b1w0l6bskg
+</code></pre>
+</notextile>
+
+h3. Control a workflow locally
+
+To run a workflow with local control, use @--local at . This means that the host where you run @arvados-cwl-runner@ will be responsible for submitting jobs, however, the jobs themselves will still run on the Arvados cluster. With @--local@, if you interrupt @arvados-cwl-runner@ or log out, the workflow will be terminated.
+
+<notextile>
+<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --local bwa-mem.cwl bwa-mem-input.yml</span>
+arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
+2016-07-01 10:05:19 arvados.cwl-runner[16290] INFO: Pipeline instance qr1hi-d1hrv-92wcu6ldtio74r4
+2016-07-01 10:05:28 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Queued
+2016-07-01 10:05:29 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Running
+2016-07-01 10:05:45 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Complete
+2016-07-01 10:05:46 arvados.cwl-runner[16290] INFO: Overall process status is success
+{
+ "aligned_sam": {
+ "size": 30738986,
+ "path": "keep:15f56bad0aaa7364819bf14ca2a27c63+88/HWI-ST1027_129_D0THKACXX.1_1.sam",
+ "checksum": "sha1$0dc46a3126d0b5d4ce213b5f0e86e2d05a54755a",
+ "class": "File"
+ }
+}
+</code></pre>
+</notextile>
diff --git a/doc/user/cwl/cwl-runner.html.textile.liquid b/doc/user/cwl/cwl-runner.html.textile.liquid
index 77ce286..7c75c1c 100644
--- a/doc/user/cwl/cwl-runner.html.textile.liquid
+++ b/doc/user/cwl/cwl-runner.html.textile.liquid
@@ -108,94 +108,9 @@ h3. Work reuse
Workflows submitted with @arvados-cwl-runner@ will take advantage of Arvados job reuse. If you submit a workflow which is identical to one that has run before, it will short cut the execution and return the result of the previous run. This also applies to individual workflow steps. For example, a two step workflow where the first step has run before will reuse results for first step and only execute the new second step. You can disable this behavior with @--disable-reuse at .
-h2. Command line options
-
-table(table table-bordered table-condensed).
-|_. Option |_. Description |
-| --basedir BASEDIR| Base directory used to resolve relative references in the input, default to directory of input object file or current directory (if inputs piped/provided on command line).|
-| --version| Print version and exit|
-| --verbose| Default logging|
-| --quiet| Only print warnings and errors.|
-| --debug| Print even more logging|
-| --tool-help| Print command line help for tool|
-| --enable-reuse|Enable job reuse (default)|
-| --disable-reuse|Disable job reuse (always run new jobs).|
-| --project-uuid UUID| Project that will own the workflow jobs, if not provided, will go to home project.|
-| --output-name OUTPUT_NAME|Name to use for collection that stores the final output.|
-| --output-tags OUTPUT_TAGS|Tags for the final output collection separated by commas, e.g., '--output-tags tag0,tag1,tag2'.|
-| --ignore-docker-for-reuse|Ignore Docker image version when deciding whether to reuse past jobs.|
-| --submit| Submit workflow to run on Arvados.|
-| --local| Control workflow from local host (submits jobs to Arvados).|
-| --create-template| (Deprecated) synonym for --create-workflow.|
-| --create-workflow| Create an Arvados workflow (if using the 'containers' API) or pipeline template (if using the 'jobs' API). See --api.|
-| --update-workflow UUID|Update an existing Arvados workflow or pipeline template with the given UUID.|
-| --wait| After submitting workflow runner job, wait for completion.|
-| --no-wait| Submit workflow runner job and exit.|
-| --api WORK_API| Select work submission API, one of 'jobs' or 'containers'. Default is 'jobs' if that API is available, otherwise 'containers'.|
-| --compute-checksum| Compute checksum of contents while collecting outputs|
-| --submit-runner-ram SUBMIT_RUNNER_RAM|RAM (in MiB) required for the workflow runner job (default 1024)|
-| --name NAME| Name to use for workflow execution instance.|
-
-h3. Specify workflow and output names
-
-Use the @--name@ and @--output-name@ options to specify the name of the workflow and name of the output collection.
+h3. Command line options
-<notextile>
-<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --name "Example bwa run" --output-name "Example bwa output" bwa-mem.cwl bwa-mem-input.yml</span>
-arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
-2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to qr1hi-4zz18-h7ljh5u76760ww2
-2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Running
-2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Complete
-2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Overall process status is success
-{
- "aligned_sam": {
- "path": "keep:54325254b226664960de07b3b9482349+154/HWI-ST1027_129_D0THKACXX.1_1.sam",
- "checksum": "sha1$0dc46a3126d0b5d4ce213b5f0e86e2d05a54755a",
- "class": "File",
- "size": 30738986
- }
-}
-</code></pre>
-</notextile>
-
-h3. Submit a workflow with no waiting
-
-To submit a workflow and exit immediately, use the @--no-wait@ option. This will submit the workflow to Arvados, print out the UUID of the job that was submitted to standard output, and exit.
-
-<notextile>
-<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --no-wait bwa-mem.cwl bwa-mem-input.yml</span>
-arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
-2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Uploaded to qr1hi-4zz18-eqnfwrow8aysa9q
-2016-06-30 15:07:52 arvados.cwl-runner[12480] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-qr1hi-8i9sb-fm2n3b1w0l6bskg
-</code></pre>
-</notextile>
-
-h3. Control a workflow locally
-
-To run a workflow with local control, use @--local at . This means that the host where you run @arvados-cwl-runner@ will be responsible for submitting jobs, however, the jobs themselves will still run on the Arvados cluster. With @--local@, if you interrupt @arvados-cwl-runner@ or log out, the workflow will be terminated.
-
-<notextile>
-<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --local bwa-mem.cwl bwa-mem-input.yml</span>
-arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
-2016-07-01 10:05:19 arvados.cwl-runner[16290] INFO: Pipeline instance qr1hi-d1hrv-92wcu6ldtio74r4
-2016-07-01 10:05:28 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Queued
-2016-07-01 10:05:29 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Running
-2016-07-01 10:05:45 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Complete
-2016-07-01 10:05:46 arvados.cwl-runner[16290] INFO: Overall process status is success
-{
- "aligned_sam": {
- "size": 30738986,
- "path": "keep:15f56bad0aaa7364819bf14ca2a27c63+88/HWI-ST1027_129_D0THKACXX.1_1.sam",
- "checksum": "sha1$0dc46a3126d0b5d4ce213b5f0e86e2d05a54755a",
- "class": "File"
- }
-}
-</code></pre>
-</notextile>
+See "Using arvados-cwl-runner":{{site.baseurl}}/user/cwl/cwl-run-options.html
h2(#setup). Setting up arvados-cwl-runner
diff --git a/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid b/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid
index 8750053..ab80c97 100644
--- a/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid
+++ b/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid
@@ -36,7 +36,7 @@ arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107,
2016-07-01 14:09:50 arvados.arv-run[3730] INFO: Upload local files: "bwa-mem.cwl"
2016-07-01 14:09:50 arvados.arv-run[3730] INFO: Uploaded to qr1hi-4zz18-0f91qkovk4ml18o
2016-07-01 14:09:50 arvados.cwl-runner[3730] INFO: Created template qr1hi-p5p6p-0deqe6nuuyqns2i
-qr1hi-p5p6p-0deqe6nuuyqns2i
+qr1hi-p5p6p-zuniv58hn8d0qd8
</code></pre>
</notextile>
@@ -45,10 +45,10 @@ h3. Running registered workflows at the command line
You can run a registered workflow at the command line by its UUID:
<notextile>
-<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner 962eh-7fd4e-660fw86nz0w77mk --help</span>
+<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner qr1hi-p5p6p-zuniv58hn8d0qd8 --help</span>
/home/peter/work/scripts/venv/bin/arvados-cwl-runner 0d62edcb9d25bf4dcdb20d8872ea7b438e12fc59 1.0.20161209192028, arvados-python-client 0.1.20161212125425, cwltool 1.0.20161207161158
-Resolved '962eh-7fd4e-660fw86nz0w77mk' to 'arvwf:962eh-7fd4e-660fw86nz0w77mk#main'
-usage: 962eh-7fd4e-660fw86nz0w77mk [-h] [--PL PL] --group_id GROUP_ID
+Resolved 'qr1hi-p5p6p-zuniv58hn8d0qd8' to 'keep:655c6cd07550151b210961ed1d3852cf+57/bwa-mem.cwl'
+usage: qr1hi-p5p6p-zuniv58hn8d0qd8 [-h] [--PL PL] --group_id GROUP_ID
--read_p1 READ_P1 [--read_p2 READ_P2]
[--reference REFERENCE] --sample_id
SAMPLE_ID
diff --git a/sdk/cwl/arvados_cwl/fsaccess.py b/sdk/cwl/arvados_cwl/fsaccess.py
index d1a6463..500ea0f 100644
--- a/sdk/cwl/arvados_cwl/fsaccess.py
+++ b/sdk/cwl/arvados_cwl/fsaccess.py
@@ -184,18 +184,23 @@ class CollectionFetcher(DefaultFetcher):
return super(CollectionFetcher, self).urljoin(base_url, url)
workflow_uuid_pattern = re.compile(r'[a-z0-9]{5}-7fd4e-[a-z0-9]{15}')
+pipeline_template_uuid_pattern = re.compile(r'[a-z0-9]{5}-p5p6p-[a-z0-9]{15}')
def collectionResolver(api_client, document_loader, uri):
if workflow_uuid_pattern.match(uri):
return "arvwf:%s#main" % (uri)
+ if pipeline_template_uuid_pattern.match(uri):
+ pt = api_client.pipeline_templates().get(uuid=uri).execute()
+ return "keep:" + pt["components"].values()[0]["script_parameters"]["cwl:tool"]
+
p = uri.split("/")
if arvados.util.keep_locator_pattern.match(p[0]):
return "keep:%s" % (uri)
if arvados.util.collection_uuid_pattern.match(p[0]):
- return "keep:%s%s" % (self.api_client.collections().
- get(uuid=uri).execute()["portable_data_hash"],
+ return "keep:%s%s" % (api_client.collections().
+ get(uuid=p[0]).execute()["portable_data_hash"],
uri[len(p[0]):])
return cwltool.resolver.tool_resolver(document_loader, uri)
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list