[ARVADOS] updated: 1.3.0-1317-gfe9332723
Git user
git at public.curoverse.com
Wed Jul 17 15:54:44 UTC 2019
Summary of changes:
doc/_config.yml | 1 +
doc/user/cwl/cwl-extensions.html.textile.liquid | 4 +-
doc/user/cwl/cwl-style.html.textile.liquid | 149 ++++++++++-----------
doc/user/cwl/cwl-versions.html.textile.liquid | 32 +++++
.../writing-cwl-workflow.html.textile.liquid | 12 --
5 files changed, 106 insertions(+), 92 deletions(-)
create mode 100644 doc/user/cwl/cwl-versions.html.textile.liquid
via fe93327232956722ca9a5d91ddc3338eaff55f14 (commit)
from b23721a01f61c6d9862ea6cb4d15bd620e06eeef (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit fe93327232956722ca9a5d91ddc3338eaff55f14
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Wed Jul 17 11:53:22 2019 -0400
15028: Reorganize CWL documentation some more
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/doc/_config.yml b/doc/_config.yml
index 6c7d3f2f3..93d1e8cea 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -57,6 +57,7 @@ navbar:
- user/cwl/cwl-style.html.textile.liquid
- user/cwl/federated-workflows.html.textile.liquid
- user/cwl/cwl-extensions.html.textile.liquid
+ - user/cwl/cwl-versions.html.textile.liquid
- user/topics/arv-docker.html.textile.liquid
- Reference:
- user/topics/link-accounts.html.textile.liquid
diff --git a/doc/user/cwl/cwl-extensions.html.textile.liquid b/doc/user/cwl/cwl-extensions.html.textile.liquid
index 847df78f2..d73e83ce0 100644
--- a/doc/user/cwl/cwl-extensions.html.textile.liquid
+++ b/doc/user/cwl/cwl-extensions.html.textile.liquid
@@ -19,7 +19,7 @@ $namespaces:
cwltool: "http://commonwl.org/cwltool#"
{% endcodeblock %}
-For portability, Arvados extensions should go into the @hints@ section of your CWL file, for example:
+For portability, most Arvados extensions should go into the @hints@ section of your CWL file. This makes it possible for your workflows to run other CWL runners that do not recognize Arvados hints. The difference between @hints@ and @requirements@ is that @hints@ are optional features that can be ignored by other runners and still produce the same output, whereas @requirements@ will fail the workflow if they cannot be fulfilled. For example, @arv:IntermediateOutput@ should go in @hints@ as it will have no effect on non-Arvados platforms, however if your workflow explicitly accesses the Arvados API and will fail without it, you should put @arv:APIRequirement@ in @requirements at .
{% codeblock as yaml %}
hints:
@@ -49,8 +49,6 @@ hints:
project_uuid: clsr1-j7d0g-qxc4jcji7n4lafx
{% endcodeblock %}
-The one exception to the @hints@ section this is @arv:APIRequirement@, see note below.
-
h2(#RunInSingleContainer). arv:RunInSingleContainer
Apply this to a workflow step that runs a subworkflow. Indicates that all the steps of the subworkflow should run together in a single container and not be scheduled separately. If you have a sequence of short-running steps (less than 1-2 minutes each) this enables you to avoid scheduling and data transfer overhead by running all the steps together at once. To use this feature, @cwltool@ must be installed in the container image.
diff --git a/doc/user/cwl/cwl-style.html.textile.liquid b/doc/user/cwl/cwl-style.html.textile.liquid
index 4f5e372a6..ee36014cb 100644
--- a/doc/user/cwl/cwl-style.html.textile.liquid
+++ b/doc/user/cwl/cwl-style.html.textile.liquid
@@ -9,85 +9,13 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-h2. Portability
-
-To run on Arvados, a workflow must provide a @DockerRequirement@ in either the @hints@ or @requirements@ section.
-
-Build a reusable library of components. Share tool wrappers and subworkflows between projects. Make use of and contribute to "community maintained workflows and tools":https://github.com/common-workflow-language/workflows and tool registries such as "Dockstore":http://dockstore.org .
-
-Avoid declaring @InlineJavascriptRequirement@ or @ShellCommandRequirement@ unless you specifically need them. Don't include them "just in case" because they change the default behavior and may imply extra overhead.
-
-CommandLineTools wrapping custom scripts should represent the script as an input parameter with the script file as a default value. Use @secondaryFiles@ for scripts that consist of multiple files. For example:
-
-{% codeblock as yaml %}
-cwlVersion: v1.0
-class: CommandLineTool
-baseCommand: python
-inputs:
- script:
- type: File
- inputBinding: {position: 1}
- default:
- class: File
- location: bclfastq.py
- secondaryFiles:
- - class: File
- location: helper1.py
- - class: File
- location: helper2.py
- inputfile:
- type: File
- inputBinding: {position: 2}
-outputs:
- out:
- type: File
- outputBinding:
- glob: "*.fastq"
-{% endcodeblock %}
-
-You can get the designated temporary directory using @$(runtime.tmpdir)@ in your CWL file, or from the @$TMPDIR@ environment variable in your script.
-
-Similarly, you can get the designated output directory using $(runtime.outdir), or from the @HOME@ environment variable in your script.
-
-Avoid specifying resource requirements in CommandLineTool. Prefer to specify them in the workflow. You can provide a default resource requirement in the top level @hints@ section, and individual steps can override it with their own resource requirement.
-
-{% codeblock as yaml %}
-cwlVersion: v1.0
-class: Workflow
-inputs:
- inp: File
-hints:
- ResourceRequirement:
- ramMin: 1000
- coresMin: 1
- tmpdirMin: 45000
-steps:
- step1:
- in: {inp: inp}
- out: [out]
- run: tool1.cwl
- step2:
- in: {inp: step1/inp}
- out: [out]
- run: tool2.cwl
- hints:
- ResourceRequirement:
- ramMin: 2000
- coresMin: 2
- tmpdirMin: 90000
-{% endcodeblock %}
-
-h3. Upgrading to CWL v1.1
-
-CWL v1.1 introduces several features to the standard that were previously available as Arvados extensions. CWL v1.1 syntax is backwards compatible with v1.0, so you can just change @cwlVersion: v1.0@ to @cwlVersion: v1.1@ and update your script to using the standard features. On Arvados, there is only one behavior change between CWL v1.0 and v1.1 to be aware of: for performance reasons, Directory listings are no longer loaded by default. To control loading Directory listings, use "loadListing":https://www.commonwl.org/v1.1/CommandLineTool.html#CommandInputParameter or "LoadListingRequirement":https://www.commonwl.org/v1.1/CommandLineTool.html#LoadListingRequirement (the extension @cwltool:LoadListingRequirement@ is deprecated.)
-
-If a step requires network access, use "NetworkAccess":https://www.commonwl.org/v1.1/CommandLineTool.html#NetworkAccess instead of the Arvados-specific "arv:APIRequirement":cwl-extensions.html#APIRequirement .
+h2(#performance). Performance
-To prevent misbehaving steps from running forever and wasting resources, you can fail the step if it exceeds a certain running time with "ToolTimeLimit":https://www.commonwl.org/v1.1/CommandLineTool.html#ToolTimeLimit instead of the deprecated @cwltool:TimeLimit@ .
+To get the best perfomance from your workflows, be aware of the following Arvados features, behaviors, and best practices:
-To control if an individual step can be reused, use "WorkReuse":https://www.commonwl.org/v1.1/CommandLineTool.html#WorkReuse instead of the deprecated @arv:ReuseRequirement at .
+If you have a sequence of short-running steps (less than 1-2 minutes each), use the Arvados extension "arv:RunInSingleContainer":cwl-extensions.html#RunInSingleContainer to avoid scheduling and data transfer overhead by running all the steps together at once. To use this feature, @cwltool@ must be installed in the container image.
-h2(#performance). Performance
+Avoid declaring @InlineJavascriptRequirement@ or @ShellCommandRequirement@ unless you specifically need them. Don't include them "just in case" because they change the default behavior and may add extra overhead.
When combining a parameter value with a string, such as adding a filename extension, write @$(inputs.file.basename).ext@ instead of @$(inputs.file.basename + 'ext')@. The first form is evaluated as a simple text substitution, the second form (using the @+@ operator) is evaluated as an arbitrary Javascript expression and requires that you declare @InlineJavascriptRequirement at .
@@ -188,4 +116,71 @@ steps:
run: tool3.cwl
{% endcodeblock %}
-If you have a sequence of short-running steps (less than 1-2 minutes each), use the Arvados extension "arv:RunInSingleContainer":cwl-extensions.html#RunInSingleContainer to avoid scheduling and data transfer overhead by running all the steps together at once. To use this feature, @cwltool@ must be installed in the container image.
+
+h2. Portability
+
+To write workflows that are easy to modify and portable across CWL runners (in the event you need to share your workflow with others), there are several best practices to follow:
+
+Workflows should always provide @DockerRequirement@ in the @hints@ or @requirements@ section.
+
+Build a reusable library of components. Share tool wrappers and subworkflows between projects. Make use of and contribute to "community maintained workflows and tools":https://github.com/common-workflow-language/workflows and tool registries such as "Dockstore":http://dockstore.org .
+
+CommandLineTools wrapping custom scripts should represent the script as an input parameter with the script file as a default value. Use @secondaryFiles@ for scripts that consist of multiple files. For example:
+
+{% codeblock as yaml %}
+cwlVersion: v1.0
+class: CommandLineTool
+baseCommand: python
+inputs:
+ script:
+ type: File
+ inputBinding: {position: 1}
+ default:
+ class: File
+ location: bclfastq.py
+ secondaryFiles:
+ - class: File
+ location: helper1.py
+ - class: File
+ location: helper2.py
+ inputfile:
+ type: File
+ inputBinding: {position: 2}
+outputs:
+ out:
+ type: File
+ outputBinding:
+ glob: "*.fastq"
+{% endcodeblock %}
+
+You can get the designated temporary directory using @$(runtime.tmpdir)@ in your CWL file, or from the @$TMPDIR@ environment variable in your script.
+
+Similarly, you can get the designated output directory using $(runtime.outdir), or from the @HOME@ environment variable in your script.
+
+Avoid specifying resource requirements in CommandLineTool. Prefer to specify them in the workflow. You can provide a default resource requirement in the top level @hints@ section, and individual steps can override it with their own resource requirement.
+
+{% codeblock as yaml %}
+cwlVersion: v1.0
+class: Workflow
+inputs:
+ inp: File
+hints:
+ ResourceRequirement:
+ ramMin: 1000
+ coresMin: 1
+ tmpdirMin: 45000
+steps:
+ step1:
+ in: {inp: inp}
+ out: [out]
+ run: tool1.cwl
+ step2:
+ in: {inp: step1/inp}
+ out: [out]
+ run: tool2.cwl
+ hints:
+ ResourceRequirement:
+ ramMin: 2000
+ coresMin: 2
+ tmpdirMin: 90000
+{% endcodeblock %}
diff --git a/doc/user/cwl/cwl-versions.html.textile.liquid b/doc/user/cwl/cwl-versions.html.textile.liquid
new file mode 100644
index 000000000..f20071e02
--- /dev/null
+++ b/doc/user/cwl/cwl-versions.html.textile.liquid
@@ -0,0 +1,32 @@
+---
+layout: default
+navsection: userguide
+title: CWL version and API support
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+h2. Upgrading to CWL v1.1
+
+CWL v1.1 introduces several features to the standard that were previously available as Arvados extensions. CWL v1.1 syntax is backwards compatible with v1.0, so you can just change @cwlVersion: v1.0@ to @cwlVersion: v1.1@ and update your script to using the standard features. On Arvados, there is only one behavior change between CWL v1.0 and v1.1 to be aware of: for performance reasons, Directory listings are no longer loaded by default. To control loading Directory listings, use "loadListing":https://www.commonwl.org/v1.1/CommandLineTool.html#CommandInputParameter or "LoadListingRequirement":https://www.commonwl.org/v1.1/CommandLineTool.html#LoadListingRequirement (the extension @cwltool:LoadListingRequirement@ is deprecated.)
+
+If a step requires network access, use "NetworkAccess":https://www.commonwl.org/v1.1/CommandLineTool.html#NetworkAccess instead of the Arvados-specific "arv:APIRequirement":cwl-extensions.html#APIRequirement .
+
+To prevent misbehaving steps from running forever and wasting resources, you can fail the step if it exceeds a certain running time with "ToolTimeLimit":https://www.commonwl.org/v1.1/CommandLineTool.html#ToolTimeLimit instead of the deprecated @cwltool:TimeLimit@ .
+
+To control if an individual step can be reused, use "WorkReuse":https://www.commonwl.org/v1.1/CommandLineTool.html#WorkReuse instead of the deprecated @arv:ReuseRequirement at .
+
+h2(#migrate). Differences in running CWL on the legacy jobs API vs containers API
+
+Most users can ignore this section.
+
+When migrating your Arvados cluster from using the jobs API (--api=jobs) (sometimes referred to as "crunch v1") to the containers API (--api=containers) ("crunch v2") there are a few differences in behavior:
+
+A tool may fail to find an input file that could be found when run under the jobs API. This is because tools are limited to accessing collections explicitly listed in the input, and further limited to those individual files or subdirectories that are listed. For example, given an explicit file input @/dir/subdir/file1.txt@, a tool will not be allowed to implicitly access a file in the parent directory @/dir/file2.txt at . Use @secondaryFiles@ or a @Directory@ for files that need to be grouped together.
+
+A tool may fail when attempting to rename or delete a file in the output directory. This may happen because files listed in @InitialWorkDirRequirement@ appear in the output directory as normal files (not symlinks) but cannot be moved, renamed or deleted unless marked as "writable" in CWL. These files will be added to the output collection but without any additional copies of the underlying data.
+
+A tool may fail when attempting to access the network. This may happen because, unlike the jobs API, under the containers API network access is disabled by default. Tools which require network access should add @arv:APIRequirement: {}@ to the @requirements@ section.
diff --git a/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid b/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid
index a69f0e6f3..dd537c46a 100644
--- a/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid
+++ b/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid
@@ -207,15 +207,3 @@ arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107,
}
</code></pre>
</notextile>
-
-h2(#migrate). Differences in running CWL on the legacy jobs API vs containers API
-
-Most users can ignore this section.
-
-When migrating your Arvados cluster from using the jobs API (--api=jobs) (sometimes referred to as "crunch v1") to the containers API (--api=containers) ("crunch v2") there are a few differences in behavior:
-
-A tool may fail to find an input file that could be found when run under the jobs API. This is because tools are limited to accessing collections explicitly listed in the input, and further limited to those individual files or subdirectories that are listed. For example, given an explicit file input @/dir/subdir/file1.txt@, a tool will not be allowed to implicitly access a file in the parent directory @/dir/file2.txt at . Use @secondaryFiles@ or a @Directory@ for files that need to be grouped together.
-
-A tool may fail when attempting to rename or delete a file in the output directory. This may happen because files listed in @InitialWorkDirRequirement@ appear in the output directory as normal files (not symlinks) but cannot be moved, renamed or deleted unless marked as "writable" in CWL. These files will be added to the output collection but without any additional copies of the underlying data.
-
-A tool may fail when attempting to access the network. This may happen because, unlike the jobs API, under the containers API network access is disabled by default. Tools which require network access should add @arv:APIRequirement: {}@ to the @requirements@ section.
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list