[ARVADOS] updated: f00a2304c3668e74c6ab53e27793166b02b709d2
Git user
git at public.curoverse.com
Thu Sep 8 20:49:26 EDT 2016
Summary of changes:
.../_container_runtime_constraints.liquid | 9 ++++
doc/_includes/_mount_types.liquid | 58 ++++++++++++++++++++++
.../methods/container_requests.html.textile.liquid | 4 +-
doc/api/schema/Container.html.textile.liquid | 26 +++++++---
.../schema/ContainerRequest.html.textile.liquid | 18 ++++---
5 files changed, 101 insertions(+), 14 deletions(-)
create mode 100644 doc/_includes/_container_runtime_constraints.liquid
create mode 100644 doc/_includes/_mount_types.liquid
via f00a2304c3668e74c6ab53e27793166b02b709d2 (commit)
from 769189c406bb42761f4d7bfe35fd2cc36edcef00 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit f00a2304c3668e74c6ab53e27793166b02b709d2
Author: radhika <radhika at curoverse.com>
Date: Thu Sep 8 20:48:30 2016 -0400
9835: more details such as mount_types added.
diff --git a/doc/_includes/_container_runtime_constraints.liquid b/doc/_includes/_container_runtime_constraints.liquid
new file mode 100644
index 0000000..5c86e8f
--- /dev/null
+++ b/doc/_includes/_container_runtime_constraints.liquid
@@ -0,0 +1,9 @@
+Runtime constraints
+
+Runtime constraints restrict the container's access to compute resources and the outside world (in addition to its explicitly stated inputs and output).
+
+table(table table-bordered table-condensed).
+|_. Key|_. Type|_. Description|_. Notes|
+|ram|integer|Number of ram bytes to be used to run this process.|Required when in "Committed" state.|
+|vcpus|integer|Number of cores to be used to run this process.|Required when in "Committed" state.|
+|API|boolean|When set, use Crunch2 to run this process.|Optional.|
diff --git a/doc/_includes/_mount_types.liquid b/doc/_includes/_mount_types.liquid
new file mode 100644
index 0000000..33765c4
--- /dev/null
+++ b/doc/_includes/_mount_types.liquid
@@ -0,0 +1,58 @@
+Mount types
+
+The "mounts" hash is the primary mechanism for adding data to the container at runtime (beyond what is already in the container image).
+
+Each value of the "mounts" hash is itself a hash, whose "kind" key determines the handler used to attach data to the container.
+
+table(table table-bordered table-condensed).
+|_. Mount type|_.Kind|_. Expected keys|_. Description|_. Examples|_. Discussion|
+|Arvados data collection|@collection@|@"portable_data_hash"@ _or_ @"uuid"@ _may_ be provided. If not provided, a new collection will be created. This is useful when @"writable":true@ and the container's @output_path@ is (or is a subdirectory of) this mount target.
+@"writable"@ may be provided with a @true@ or @false@ to indicate the path must (or must not) be writable. If not specified, the system can choose.
+@"path"@ may be provided, and defaults to @"/"@.|At container startup, the target path will have the same directory structure as the given path within the collection. Even if the files/directories are writable in the container, modifications will _not_ be saved back to the original collections when the container ends.|<pre><code>{
+ "kind":"collection",
+ "uuid":"...",
+ "path":"/foo.txt"
+}
+{
+ "kind":"collection",
+ "uuid":"..."
+}</code></pre>||
+|Git tree|@git_tree@|One of { @"git-url"@, @"repository_name"@, @"uuid"@ } must be provided.
+One of { @"commit"@, @"revisions"@ } must be provided.
+"path" may be provided. The default path is "/".|At container startup, the target path will have the source tree indicated by the given revision. The @.git@ metadata directory _will not_ be available: typically the system will use @git-archive@ rather than @git-checkout@ to prepare the target directory.
+- If a value is given for @"revisions"@, it will be resolved to a set of commits (as desribed in the "ranges" section of git-revisions(1)) and the container request will be satisfiable by any commit in that set.
+- If a value is given for @"commit"@, it will be resolved to a single commit, and the tree resulting from that commit will be used.
+- @"path"@ can be used to select a subdirectory or a single file from the tree indicated by the selected commit.
+- Multiple commits can resolve to the same tree: for example, the file/directory given in @"path"@ might not have changed between commits A and B.
+- The resolved mount (found in the Container record) will have only the "kind" key and a "blob" or "tree" key indicating the 40-character hash of the git tree/blob used.|<pre><code>{
+ "kind":"git_tree",
+ "uuid":"zzzzz-s0uqq-xxxxxxxxxxxxxxx",
+ "commit":"master"
+}
+{
+ "kind":"git_tree",
+ "uuid":"zzzzz-s0uqq-xxxxxxxxxxxxxxx",
+ "commit_range":"bugfix^..master",
+ "path":"/crunch_scripts/grep"
+}</code></pre>||
+|Temporary directory|@tmp@|@"capacity"@: capacity (in bytes) of the storage device.
+@"device_type"@ (optional, default "network"): one of @{"ram", "ssd", "disk", "network"}@ indicating the acceptable level of performance.|At container startup, the target path will be empty. When the container finishes, the content will be discarded. This will be backed by a storage mechanism no slower than the specified type.|<pre><code>{
+ "kind":"tmp",
+ "capacity":100000000000
+}
+{
+ "kind":"tmp",
+ "capacity":1000000000,
+ "device_type":"ram"
+}</code></pre>||
+|Keep|@keep@|Expose all readable collections via arv-mount.|Requires suitable runtime constraints.|<pre><code>{
+ "kind":"keep"
+}</code></pre>||
+|Mounted file or directory|@file@|@"path"@: absolute path (inside the container) of a file or directory that is (or is inside) another mount target.|Can be used for "stdin" and "stdout" targets.|<pre><code>{
+ "kind":"file",
+ "path":"/mounted_tmp/a.out"
+}</code></pre>||
+|JSON document|@json@|A JSON-encoded string, array, or object.||<pre>{
+ "kind":"json",
+ "content":{"foo":"bar"}
+}</pre>||
diff --git a/doc/api/methods/container_requests.html.textile.liquid b/doc/api/methods/container_requests.html.textile.liquid
index 491698e..88a53a2 100644
--- a/doc/api/methods/container_requests.html.textile.liquid
+++ b/doc/api/methods/container_requests.html.textile.liquid
@@ -69,4 +69,6 @@ table(table table-bordered table-condensed).
{background:#ccffcc}.|uuid|string|The UUID of the ContainerRequest in question.|path||
|container_request|object||query||
-Note: To cancel a ContainerRequest, set it's priority to zero.
+{% include 'notebox_begin' %}
+Setting the priority of a committed container_request to 0 may cancel a running container assigned for it.
+{% include 'notebox_end' %}
diff --git a/doc/api/schema/Container.html.textile.liquid b/doc/api/schema/Container.html.textile.liquid
index e690b82..d29d420 100644
--- a/doc/api/schema/Container.html.textile.liquid
+++ b/doc/api/schema/Container.html.textile.liquid
@@ -7,8 +7,8 @@ title: Container
...
A Container:
-* Captures information about the environment in which a Crunch2 process was run. For example, git trees, data collections, and docker images are stored as content addresses. This makes it possible to reason about the difference between two processes, and to replay a process at a different time and place.
-* Clients can read Container records, but only the system can create or modify them.
+* Precisely describes the environment in which a Crunch2 process should run. For example, git trees, data collections, and docker images are stored as content addresses. This makes it possible to reason about the difference between two processes, and to replay a process at a different time and place.
+* Container records are created by the system to fulfill container requests.
h2. Methods
@@ -20,22 +20,22 @@ Each Container offers the following attributes, in addition to the usual "attrib
table(table table-bordered table-condensed).
|_. Attribute|_. Type|_. Description|_. Notes|
-|state|string|The allowed states are "Queued", "Locked", "Running", "Cancelled" and "Complete".|See <A href="https://dev.arvados.org/projects/arvados/wiki/Containers_API#Container-states">Container states</A> for additional details.|
+|state|string|The allowed states are "Queued", "Locked", "Running", "Cancelled" and "Complete".|See "Container states":#container_states for more details.|
|started_at|datetime|When this container started running.|Null if container has not yet started.|
|finished_at|datetime|When this container finished.|Null if container has not yet finished.|
-|log|string|Portable data hash of the log collection.|Null if the container is not yet finished.|
+|log|string|Portable data hash of the collection containing logs from a completed container run.|Null if the container is not yet finished.|
|environment|hash|Environment variables and values that should be set in the container environment (@docker run --env@). This augments and (when conflicts exist) overrides environment variables given in the image's Dockerfile.|Must be equal to a ContainerRequest's environment in order to satisfy the ContainerRequest.|
|cwd|string|Initial working directory.|Must be equal to a ContainerRequest's cwd in order to satisfy the ContainerRequest|
|command|array of strings|Command to execute.| Must be equal to a ContainerRequest's command in order to satisfy the ContainerRequest.|
|output_path|string|Path to a directory or file inside the container that should be preserved as this container's output when it finishes.|Must be equal to a ContainerRequest's output_path in order to satisfy the ContainerRequest.|
-|mounts|hash|Must contain the same keys as the ContainerRequest being satisfied. Each value must be within the range of values described in the ContainerRequest at the time the Container is assigned to the ContainerRequest.|See <A href="https://dev.arvados.org/projects/arvados/wiki/Containers_API#Mount-types">Mount types</A> for additional details.|
+|mounts|hash|Must contain the same keys as the ContainerRequest being satisfied. Each value must be within the range of values described in the ContainerRequest at the time the Container is assigned to the ContainerRequest.|See "Mount types":#mount_types for more details.|
|runtime_constraints|hash|Compute resources, and access to the outside world, that are / were available to the container.
Generally this will contain additional keys that are not present in any corresponding ContainerRequests: for example, even if no ContainerRequests specified constraints on the number of CPU cores, the number of cores actually used will be recorded here.|e.g.,
<pre><code>{
"ram":12000000000,
"vcpus":2,
"API":true
-}</code></pre>|
+}</code></pre>See "Runtime constraints":#runtime_constraints for more details.|
|output|string|Portable data hash of the output collection.|Null if the container is not yet finished.|
|container_image|string|Portable data hash of a collection containing the docker image used to run the container.||
|progress|number|A number between 0.0 and 1.0 describing the fraction of work done.||
@@ -43,3 +43,17 @@ Generally this will contain additional keys that are not present in any correspo
|exit_code|integer|Process exit code.|Null if state!="Complete"|
|auth_uuid|string|UUID of a token to be passed into the container itself, used to access Keep-backed mounts, etc.|Null if state∉{"Locked","Running"}|
|locked_by_uuid|string|UUID of a token, indicating which dispatch process changed state to Locked. If null, any token can be used to lock. If not null, only the indicated token can modify this container.|Null if state∉{"Locked","Running"}|
+
+h2(#container_states). Container states
+
+table(table table-bordered table-condensed).
+|_. State|_. Sgnificance|_. Allowed next|
+|Queued|Waiting for a dispatcher to lock it and try to run the container.|Locked, Cancelled|
+|Locked|A dispatcher has "taken" the container and is allocating resources for it. The container has not started yet.|Queued, Running, Cancelled|
+|Running|Resources have been allocated and the contained process has been started (or is about to start). Crunch-run _must_ set state to Running _before_ there is any possibility that user code will run in the container.|Complete, Cancelled|
+|Complete|Container was running, and the contained process/command has exited.|-|
+|Cancelled|The container did not run long enough to produce an exit code. This includes cases where the container didn't even start, cases where the container was interrupted/killed before it exited by itself (e.g., priority changed to 0), and cases where some problem prevented the system from capturing the contained process's exit status (exit code and output).|-|
+
+h2(#mount_types). {% include 'mount_types' %}
+
+h2(#runtime_constraints). {% include 'container_runtime_constraints' %}
diff --git a/doc/api/schema/ContainerRequest.html.textile.liquid b/doc/api/schema/ContainerRequest.html.textile.liquid
index 15a0c1d..06dbc61 100644
--- a/doc/api/schema/ContainerRequest.html.textile.liquid
+++ b/doc/api/schema/ContainerRequest.html.textile.liquid
@@ -19,28 +19,32 @@ h2. Resource
Each ContainerRequest offers the following attributes, in addition to the usual "attributes of Arvados resources":{{site.baseurl}}/api/resources.html:
+All attributes are optional, unless otherwise marked as required.
+
table(table table-bordered table-condensed).
|_. Attribute|_. Type|_. Description|_. Notes|
|name|string|The name of the container_request.||
|description|string|The description of the container_request.||
|properties|hash|Client-defined structured data that does not affect how the container is run.||
|state|string|The allowed states are "Uncommitted", "Committed", and "Final".|Once a request is Committed, the only attributes that can be modified are priority, container_uuid, and container_count_max. A request in the "Final" state cannot have any of its functional parts modified (i.e., only name, description, and properties fields can be modified).|
-|requesting_container_uuid|string|Set by the system to the uuid of a container corresponding to the user's token, if one exists.|When the referenced container ends, the container request is automatically completed.|
+|requesting_container_uuid|string|The uuid of the parent container that created this container_request, if any. Represents a process tree.|The priority of this container_request is inherited from the parent container, if the parent container is cancelled, this container_request will be cancelled as well.|
|container_uuid|string|The container that satisfies this container_request.||
|container_count_max|integer|Maximum number of containers to start, i.e., the maximum number of "attempts" to be made.||
-|mounts|hash|Objects to attach to the container's filesystem and stdin/stdout.|See <A href="https://dev.arvados.org/projects/arvados/wiki/Containers_API#Mount-types">Mount types</A> for additional details.|
-|runtime_constraints|hash|Restrict the container's access to compute resources and the outside world. Required and must specify ram and vcpus when the container_request is in Committed state.|e.g.,
-<pre><code>{
+|mounts|hash|Objects to attach to the container's filesystem and stdin/stdout.|See "Mount types":#mount_types for more details.|
+|runtime_constraints|hash|Restrict the container's access to compute resources and the outside world.|Required when in "Committed" state. e.g.,<pre><code>{
"ram":12000000000,
"vcpus":2,
"API":true
-}</code></pre>|
+}</code></pre>See "Runtime constraints":#runtime_constraints for more details.|
|container_image|string|Portable data hash of a collection containing the docker image to run the container.|Required.|
|environment|hash|Environment variables and values that should be set in the container environment (@docker run --env@). This augments and (when conflicts exist) overrides environment variables given in the image's Dockerfile.||
|cwd|string|Initial working directory, given as an absolute path (in the container) or a path relative to the WORKDIR given in the image's Dockerfile.|Required.|
-|command|array of strings|Command to execute in the container.|Required.
-e.g., @["echo","hello"]@|
+|command|array of strings|Command to execute in the container.|Required. e.g., @["echo","hello"]@|
|output_path|string|Path to a directory or file inside the container that should be preserved as container's output when it finishes. This path must be, or be inside, one of the mount targets. For best performance, point output_path to a writable collection mount.|Required.|
|priority|integer|Higher value means spend more resources on this container_request, i.e., go ahead of other queued containers, bring up more nodes etc.|Priority 0 means a container should not be run on behalf of this request. Clients are expected to submit ContainerRequests with zero priority in order to prevew the container that will be used to satisfy it. Priority can be null if and only if state!="Committed".|
|expires_at|datetime|After this time, priority is considered to be zero.|Not yet implemented.|
|filters|string|Additional constraints for satisfying the container_request, given in the same form as the filters parameter accepted by the container_requests.list API.||
+
+h2(#mount_types). {% include 'mount_types' %}
+
+h2(#runtime_constraints). {% include 'container_runtime_constraints' %}
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list