[ARVADOS] updated: c40265a873d73f03a9ca077f18fe305d883fb4a5
Git user
git at public.curoverse.com
Thu Nov 17 13:11:39 EST 2016
Summary of changes:
.gitignore | 3 +
.../_container_runtime_constraints.liquid | 1 -
.../_container_scheduling_parameters.liquid | 7 ++
doc/api/execution.html.textile.liquid | 2 +-
.../methods/container_requests.html.textile.liquid | 7 ++
doc/api/methods/containers.html.textile.liquid | 5 ++
doc/api/tokens.html.textile.liquid | 2 +-
sdk/cwl/arvados_cwl/__init__.py | 23 ++++--
sdk/cwl/arvados_cwl/arvcontainer.py | 3 +
sdk/cwl/arvados_cwl/arvjob.py | 6 +-
sdk/cwl/arvados_cwl/crunch_script.py | 7 +-
sdk/cwl/arvados_cwl/runner.py | 3 +-
sdk/cwl/tests/test_make_output.py | 9 ++-
sdk/cwl/tests/test_submit.py | 88 ++++++++++++++++++++++
.../arvados/v1/container_requests_controller.rb | 1 +
.../arvados/v1/containers_controller.rb | 1 +
services/api/app/models/container_request.rb | 2 +-
.../v1/container_requests_controller_test.rb | 22 ++++++
tools/arvbox/lib/arvbox/docker/Dockerfile.base | 48 ++++++++----
tools/arvbox/lib/arvbox/docker/common.sh | 6 +-
.../lib/arvbox/docker/service/ready/run-service | 4 +-
21 files changed, 215 insertions(+), 35 deletions(-)
create mode 100644 doc/_includes/_container_scheduling_parameters.liquid
create mode 100644 services/api/test/functional/arvados/v1/container_requests_controller_test.rb
via c40265a873d73f03a9ca077f18fe305d883fb4a5 (commit)
via d53262e8e07785a9d2566966ee41ab8949ef1962 (commit)
via ce924ad9cdbab7ede0df7409d43b2660c1329979 (commit)
via 2d8150ed20d97491930d99a52d923872bca73939 (commit)
via 47f987da576d7dac80e5a03ad6613b5eb1f58660 (commit)
via ad825d1c73655d3f62fd485a1bc32fc3e76531f6 (commit)
via a4762b52738e12c39d93c14501a2f62463b05d07 (commit)
via 5fe0d5e7496fad7bd1c4bab0e5ca5f348c0eec63 (commit)
via 4bccbaed84c6b398f4cb4dbc7a9bc345e79d6550 (commit)
via b6bc9bbe45a68a07b1ea8139e1f4d698873739a7 (commit)
via a9a677e1655c461e742e46cc3c239f8605f4fc6b (commit)
via ddce4f6de6f0a77b45e9f3358eb6f0c1f1870fba (commit)
via e7a865275a832420b9d63c0ab3ebf87eaca57d26 (commit)
via d8db04f0165e57ab8021f2c1e4ff7061e494ceba (commit)
via d2f9e7809bd1f63638600c7fc8189182c0f327c0 (commit)
via 2251688e66191ff1169080f50868bf57e463659c (commit)
via c14246b9a21d038fc6fa850f4032659a98397784 (commit)
via 5d2ef6f7a2a8f93ec411c420287f30af92294520 (commit)
via 780d334ec4b2d47379d0330ace77b3821c880842 (commit)
via b21e623902cb32b6d5a2fcf2c6ac9d92d472cb58 (commit)
via 90c48f84391d6b4d6b8ed366d5a42d24bf6c696f (commit)
via 5cd2757a79e7be7ed00156a69191893c3bb7e1c6 (commit)
via 4225d058e0bc4380203fe5959e7e54febc91e83b (commit)
via 1e6d7656fbfd1f954571157fc7e7e4f75ea5911e (commit)
via 79786a56410ef381499fb0bfdc5a18407ab33082 (commit)
via 5fc627d22b47723289251e1e1d9dc45c0e1bd49e (commit)
from 8668135855b400e7f6047ecacd9cfa27fed723f7 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit c40265a873d73f03a9ca077f18fe305d883fb4a5
Author: radhika <radhika at curoverse.com>
Date: Thu Nov 17 13:11:08 2016 -0500
10293: update documentation with scheduling_parameters, log_uuid and output_uuid.
diff --git a/doc/_includes/_container_runtime_constraints.liquid b/doc/_includes/_container_runtime_constraints.liquid
index 849db42..d505bfd 100644
--- a/doc/_includes/_container_runtime_constraints.liquid
+++ b/doc/_includes/_container_runtime_constraints.liquid
@@ -8,4 +8,3 @@ table(table table-bordered table-condensed).
|vcpus|integer|Number of cores to be used to run this process.|Optional. However, a ContainerRequest that is in "Committed" state must provide this.|
|keep_cache_ram|integer|Number of keep cache bytes to be used to run this process.|Optional.|
|API|boolean|When set, ARVADOS_API_HOST and ARVADOS_API_TOKEN will be set, and container will have networking enabled to access the Arvados API server.|Optional.|
-|partition|array of strings|Specify the names of one or more compute partitions that may run this container. If not provided, the system chooses where to run the container.|Optional.|
diff --git a/doc/_includes/_container_scheduling_parameters.liquid b/doc/_includes/_container_scheduling_parameters.liquid
new file mode 100644
index 0000000..ee2ca07
--- /dev/null
+++ b/doc/_includes/_container_scheduling_parameters.liquid
@@ -0,0 +1,7 @@
+Scheduling parameters
+
+Parameters to be passed to the container scheduler (e.g., SLURM) when running a container.
+
+table(table table-bordered table-condensed).
+|_. Key|_. Type|_. Description|_. Notes|
+|partitions|array of strings|The names of one or more compute partitions that may run this container. If not provided, the system will choose where to run the container.|Optional.|
diff --git a/doc/api/methods/container_requests.html.textile.liquid b/doc/api/methods/container_requests.html.textile.liquid
index 304226d..1d7a244 100644
--- a/doc/api/methods/container_requests.html.textile.liquid
+++ b/doc/api/methods/container_requests.html.textile.liquid
@@ -35,6 +35,9 @@ table(table table-bordered table-condensed).
"vcpus":2,
"API":true
}</code></pre>See "Runtime constraints":#runtime_constraints for more details.|
+|scheduling_parameters|hash|Parameters to be passed to the container scheduler when running this container.|e.g.,<pre><code>{
+"partitions":["fastcpu","vfastcpu"]
+}</code></pre>See "Scheduling parameters":#scheduling_parameters for more details.|
|container_image|string|Portable data hash of a collection containing the docker image to run the container.|Required.|
|environment|hash|Environment variables and values that should be set in the container environment (@docker run --env@). This augments and (when conflicts exist) overrides environment variables given in the image's Dockerfile.||
|cwd|string|Initial working directory, given as an absolute path (in the container) or a path relative to the WORKDIR given in the image's Dockerfile.|Required.|
@@ -43,12 +46,16 @@ table(table table-bordered table-condensed).
|priority|integer|Higher value means spend more resources on this container_request, i.e., go ahead of other queued containers, bring up more nodes etc.|Priority 0 means a container should not be run on behalf of this request. Clients are expected to submit container requests with zero priority in order to prevew the container that will be used to satisfy it. Priority can be null if and only if state!="Committed".|
|expires_at|datetime|After this time, priority is considered to be zero.|Not yet implemented.|
|use_existing|boolean|If possible, use an existing (non-failed) container to satisfy the request instead of creating a new one.|Default is true|
+|log_uuid|string|Log collection containing log messages provided by the scheduler and crunch processes.|Null if the container has not yet completed.|
+|output_uuid|string|Output collection created when the container finished successfully.|Null if the container has failed or not yet completed.|
|filters|string|Additional constraints for satisfying the container_request, given in the same form as the filters parameter accepted by the container_requests.list API.|
h2(#mount_types). {% include 'mount_types' %}
h2(#runtime_constraints). {% include 'container_runtime_constraints' %}
+h2(#scheduling_parameters). {% include 'container_scheduling_parameters' %}
+
h2(#container_reuse). Container reuse
When a container request is "Committed", the system will try to find and reuse any preexisting Container with the same exact command, cwd, environment, output_path, container_image, mounts, and runtime_constraints as this container request. The serialized fields environment, mounts and runtime_constraints are sorted to facilitate comparison.
diff --git a/doc/api/methods/containers.html.textile.liquid b/doc/api/methods/containers.html.textile.liquid
index 221141c..7eed8b0 100644
--- a/doc/api/methods/containers.html.textile.liquid
+++ b/doc/api/methods/containers.html.textile.liquid
@@ -36,6 +36,9 @@ Generally this will contain additional keys that are not present in any correspo
"vcpus":2,
"API":true
}</code></pre>See "Runtime constraints":#runtime_constraints for more details.|
+|scheduling_parameters|hash|Parameters to be passed to the container scheduler when running this container.|e.g.,<pre><code>{
+"partitions":["fastcpu","vfastcpu"]
+}</code></pre>See "Scheduling parameters":#scheduling_parameters for more details.|
|output|string|Portable data hash of the output collection.|Null if the container is not yet finished.|
|container_image|string|Portable data hash of a collection containing the docker image used to run the container.||
|progress|number|A number between 0.0 and 1.0 describing the fraction of work done.||
@@ -58,6 +61,8 @@ h2(#mount_types). {% include 'mount_types' %}
h2(#runtime_constraints). {% include 'container_runtime_constraints' %}
+h2(#scheduling_parameters). {% include 'container_scheduling_parameters' %}
+
h2. Methods
See "Common resource methods":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update at .
commit d53262e8e07785a9d2566966ee41ab8949ef1962
Merge: a4762b5 ce924ad
Author: radhika <radhika at curoverse.com>
Date: Wed Nov 16 17:04:23 2016 -0500
Merge branch 'master' into 10293-cwl-cr-output
commit a4762b52738e12c39d93c14501a2f62463b05d07
Merge: 5fe0d5e 4bccbae
Author: radhika <radhika at curoverse.com>
Date: Wed Nov 16 15:58:45 2016 -0500
Merge branch '10293-container-request-output-uuid' into 10293-cwl-cr-output
commit 5fe0d5e7496fad7bd1c4bab0e5ca5f348c0eec63
Merge: a9a677e b6bc9bb
Author: radhika <radhika at curoverse.com>
Date: Wed Nov 16 15:58:35 2016 -0500
Merge branch 'master' into 10293-cwl-cr-output
commit a9a677e1655c461e742e46cc3c239f8605f4fc6b
Merge: 8668135 d2f9e78
Author: radhika <radhika at curoverse.com>
Date: Wed Nov 16 15:33:47 2016 -0500
Merge branch 'master' into 10293-cwl-cr-output
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list