[ARVADOS] created: 1.2.0-217-g97bb4392b

Wed Oct 17 15:54:40 EDT 2018

at  97bb4392bfcfbeb78cdc12caebe6e6a0a7decf62 (commit)


commit 97bb4392bfcfbeb78cdc12caebe6e6a0a7decf62
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date:   Wed Oct 17 15:54:00 2018 -0400

    14366: Add "upgrade to containers api" to "upgrading" section
    
    Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>

diff --git a/doc/_config.yml b/doc/_config.yml
index 4ab858550..e591c44f5 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -152,6 +152,7 @@ navbar:
     - Upgrading and migrations:
       - admin/upgrading.html.textile.liquid
       - install/migrate-docker19.html.textile.liquid
+      - admin/upgrade-crunch2.html.textile.liquid
     - Users and Groups:
       - install/cheat_sheet.html.textile.liquid
       - admin/activation.html.textile.liquid
diff --git a/doc/admin/upgrade-crunch2.html.textile.liquid b/doc/admin/upgrade-crunch2.html.textile.liquid
new file mode 100644
index 000000000..19463581f
--- /dev/null
+++ b/doc/admin/upgrade-crunch2.html.textile.liquid
@@ -0,0 +1,53 @@
+---
+layout: default
+navsection: admin
+title: Upgrading to Containers API
+...
+
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+The "containers" API is the recommended way to submit compute work to Arvados.  It supersedes the "jobs" API, which is deprecated.
+
+h2. Benefits over the "jobs" API
+
+* Simpler and more robust execution with fewer points of failure
+* Automatic retry for containers that fail to run to completion due to infrastructure errors
+* Scales to thousands of simultaneous containers
+* Able to support alternate schedulers/dispatchers in addition to slurm
+* Improved logging, different streams logs/metrics stored in different files in the log collection
+* Records more upfront detail about the compute node, and additional metrics (such as available disk space over the course of the container run)
+* Better behavior when deciding whether to reuse past work -- pick the oldest container that matches the criteria
+* Can reuse running containers between workflows, cancelling a workflow will not cancel containers that are shared with other workflows
+* Supports setting time-to-live on intermediate output collections for automatic cleanup
+* Supports "secret" inputs, suitable for passwords or access tokens, which are hidden from the API responses and logs, and forgotten after use
+* Does not require "git" for dispatching work
+
+h2. Differences from the "jobs" API
+
+Containers cannot reuse jobs (but can reuse other containers)
+
+Uses the service "crunch-dispatch-slurm":{{site.baseurl}}/install/crunch2-slurm/install-dispatch.html instead of @crunch-dispatch.rb@
+
+Non-CWL Arvados "pipeline templates" are not supported with containers.  Pipeline templates should be rewritten in CWL and registered as "Workflows".
+
+The containers APIs is incompatible with the jobs API, code which integrates with the "jobs" API must be updated to work with containers
+
+Containers have network access disabled by default
+
+The keep mount only exposes collections which are explicitly listed as inputs
+
+h2. Migrating to "containers" API
+
+Run your workflows using @arvados-cwl-runner --api=containers@ (only necessary if both the jobs and containers APIs are enabled, if the jobs API is disabled, it will use the containers API automatically)
+
+Register your workflows so they can be run from workbench using @arvados-cwl-runner --api=containers --create-workflow@
+
+Read "Migrating running CWL on jobs API to containers API":{{site.baseurl}}/user/cwl/cwl-style.html#migrate
+
+Use @arv:APIRequirement: {}@ in the @requirements@ section of your CWL file to enable network access for the container (see "Arvados CWL Extensions":{{site.baseurl}}/user/cwl/cwl-extensions.html)
+
+For examples on how to manage container requests with the Python SDK, see "Python cookbook":{{site.baseurl}}/sdk/python/cookbook.html

commit 6e35e04712ca8046f27220d8db2d39ea217129bc
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date:   Wed Oct 17 15:43:15 2018 -0400

    14366: Improve wording of "upgrading to containers API" section
    
    Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>

diff --git a/doc/user/cwl/cwl-style.html.textile.liquid b/doc/user/cwl/cwl-style.html.textile.liquid
index 07cb4aa90..fe53f4a45 100644
--- a/doc/user/cwl/cwl-style.html.textile.liquid
+++ b/doc/user/cwl/cwl-style.html.textile.liquid
@@ -178,7 +178,8 @@ steps:
 
 h2(#migrate). Migrating running CWL on jobs API to containers API
 
-* When migrating from jobs API (--api=jobs) (sometimes referred to as "crunch v1") to the containers API (--api=containers) ("crunch v2") there are a few differences in behavior:
-** The tool is limited to accessing only collections which are explicitly listed in the input, and further limited to only the subdirectories of collections listed in input.  For example, given an explicit file input @/dir/subdir/file1.txt@, a tool will not be able to implicitly access the file @/dir/file2.txt at .  Use @secondaryFiles@ or a @Directory@ input to describe trees of files.
-** Files listed in @InitialWorkDirRequirement@ appear in the output directory as normal files (not symlinks) but cannot be moved, renamed or deleted.  These files will be added to the output collection but without any additional copies of the underlying data.
-** Tools are disallowed network access by default.  Tools which require network access must include @arv:APIRequirement: {}@ in their @requirements@ section.
+When migrating from jobs API (--api=jobs) (sometimes referred to as "crunch v1") to the containers API (--api=containers) ("crunch v2") there are a few differences in behavior:
+
+* A tool may fail to find an input file that could be found when run under the jobs API.  This is because tools are limited to accessing collections explicitly listed in the input, and further limited to those individual files or subdirectories that are listed.  For example, given an explicit file input @/dir/subdir/file1.txt@, a tool will not be allowed to implicitly access a file in the parent directory @/dir/file2.txt at .  Use @secondaryFiles@ or a @Directory@ for files that need to be grouped together.
+* A tool may fail when attempting to rename or delete a file in the output directory.  This may happen because files listed in @InitialWorkDirRequirement@ appear in the output directory as normal files (not symlinks) but cannot be moved, renamed or deleted unless marked as "writable" in CWL.  These files will be added to the output collection but without any additional copies of the underlying data.
+* A tool may fail when attempting to access the network.  This may happen because, unlike the jobs API, under the containers API network access is disabled by default.  Tools which require network access should add @arv:APIRequirement: {}@ to the @requirements@ section.

-----------------------------------------------------------------------


hooks/post-receive
-- 


