[ARVADOS] updated: 8ac5537956b1a5c4eaf5a3740eb5b915337bcb96
git at public.curoverse.com
git at public.curoverse.com
Fri Oct 3 13:45:12 EDT 2014
Summary of changes:
crunch_scripts/crunchutil/subst.py | 17 +-
crunch_scripts/run-command | 43 ++---
doc/user/topics/run-command.html.textile.liquid | 207 ++++++++++++++++++++++--
3 files changed, 230 insertions(+), 37 deletions(-)
via 8ac5537956b1a5c4eaf5a3740eb5b915337bcb96 (commit)
via 6f2855585712ef7222f739a5afcef0aa736e01ac (commit)
via 81e7eb42156a21e17d81c4a71d1dfee4f0fbd52f (commit)
from 0eb4abfca9127ad1a7a8153840f876e512c624cb (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 8ac5537956b1a5c4eaf5a3740eb5b915337bcb96
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Oct 3 13:43:56 2014 -0400
4042: First draft document complete, need to verify that examples all work.
diff --git a/doc/user/topics/run-command.html.textile.liquid b/doc/user/topics/run-command.html.textile.liquid
index 85eefa8..47b058f 100644
--- a/doc/user/topics/run-command.html.textile.liquid
+++ b/doc/user/topics/run-command.html.textile.liquid
@@ -6,33 +6,214 @@ title: "run-command reference"
The @run-command@ crunch script enables you run command line programs.
-h2. Basic usage
+h1. Using run-command
-Run-command is controlled through the script_parameters section of a pipeline component. The script_parameters of run-command section consists of three parts:
-* User-defined parameters
-* Special processing directives @task.foreach@ @task.cwd@ @task.vwd@ @task.stdin@ @task.stdout@
-* The @command@ section defines the template to actually build the command line of task
+The basic run-command process evaluates its inputs and builds a command line, executes the command, and saves the contents of the output directory back to Keep. For large datasets, run-command can schedule concurrent tasks to execute the wrapped program over a range of inputs (see @task.foreach@ below.)
-h3. Parameter substitution
+Run-command is controlled through the script_parameters section of a pipeline component. Script_parameters is a JSON object consisting of key-value pairs. There are three categories of keys that are meaningful to run-command:
+* The @command@ section defining the template to build the command line of task
+* Special processing directives such as @task.foreach@ @task.cwd@ @task.vwd@ @task.stdin@ @task.stdout@
+* User-defined parameters (everything else)
-Any entry in the "script_parameters" object that is not a directive or reserved parameter is a user-defined parameter. User-defined parameters can be used in parameter substitution and in directives. String substitution is indicated by wrapping the expression with @$(...)@. In the following example, using @$(varA)@ for the value of "varB" denotes that it should be replaced with the value of the parameter "varA", which is "ABC".
+h2. Command template
+
+The value of the "command" key is a list. The first parameter of the list is the actual program to invoke, followed by the commmand arguments. The simplest run-command invocation simply runs a program with static parameters. In this example, run "echo" with the first argument "hello":
+
+<pre>
+ "script_parameters": {
+ "command": ["echo", "hello world"]
+ }
+</pre>
+
+Running this job will print "hello world" to the job log.
+
+By default, the command will start with the current working directory set to the output directory. Anything written to the output directory will be saved to Keep when the command is finished. You can change the default working directory using the @task.cwd@ and get the path to the output directory using @$(task.outdir)@ as explained below.
+
+Items in the "command" list may include lists and objects in addition to strings. Lists are flattened to produce the final command line. JSON objects are evaluted as list item functions (see below). For example, the following evalutes to ["echo", "hello", "world"]:
+
+<pre>
+ "script_parameters": {
+ "command": ["echo", ["hello", "world"]]
+ }
+</pre>
+
+h2. Parameter substitution
+
+The "command" list can include parameter substitutions. Substitutions are enclosed in "$(...)" and may contain the name of a user-defined parameter. In the following example, the value of "a" is "hello world"; so when "command" is evaluated, it will substitute "hello world" for "$(a)":
+
+<pre>
+"script_parameters": {
+ "command": ["echo", "$(a)"],
+ "a": "hello world"
+}
+</pre>
+
+h2. Special parameters
+
+In addition to user-defined parameters, there are special parameters supplied by run-command that provide some information about the runtime environment.
+
+table(table table-bordered table-condensed).
+|_. Parameter |_. Value |
+|$(node.cores) |Number of cores on the current node|
+|$(task.tmpdir) |Path to the temporary directory for this task |
+|$(task.outdir) |Path to the task's designated output directory. This |
+|$(task.uuid) |The current task's unique identifier |
+|$(job.srcdir) |The directory containing the source code for the run-command- script |
+|$(job.uuid) |The current job's unique identifier |
+
+h2. Substitution functions
+
+Substitutions can also make use of functions. Functions take a single parameter and substitution is performed recursively from the inside out. In the following example, the parameter $(a) is evaluated first, then the $(file ...) function applied to get a local filesystem path, to produce a command like @["echo", "/path/to/keep/mount/c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2"]@:
+
+<pre>
+"script_parameters": {
+ "command": ["echo", "$(file $(a))"],
+ "a": "c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2"
+}
+</pre>
+
+table(table table-bordered table-condensed).
+|_. Function|_. Action|
+|$(file ...) | Takes a reference to a file within an Arvados collection and evaluates to a file path on the local file system where that file can be accessed by your command. Will raise an error if the file is not accessable.|
+|$(dir ...) | Takes a reference to an Arvados collection or directory within an Arvados collection and evaluates to a directory path on the local file system where that directory can be accessed by your command. The path may include a file name, in which case it will evaluate to the parent directory of the file. Uses Python's os.path.dirname(), so "/foo/bar" will evaluate to "/foo" but "/foo/bar/" will evaluate to "/foo/bar". Will raise an error if the directory is not accessable. |
+|$(basename ...) | Strip leading directory and trailing file extension from the path provided. For example, $(basename /foo/bar.baz.txt) will evaluate to "bar.baz".|
+|$(glob ...) | Take a unix shell path pattern (supports @*@ @?@ and @[]@) and search the local filesystem, returning the first match found. Use together with $(dir ...) to get a local filesystem path for Arvados collections. For example: $(glob $(dir $(mycollection)/*.bam)) will find the first .bam file in the collection specified by the user parameter "mycollection". If there is more than one match, which one is returned is undefined. Will raise an error if no matches are found.|
+
+h2. List context
+
+When a parameter is evaluted in a list context, that means its value should evaluate to a list instead of a string. Parameter values can be a static list (as demonstrated above), a path to a file, a path to a directory, or a JSON object describing a list context function.
+
+If the value is a static list, it will evaluate the list items for parameter substition and list functions.
+
+If the value is a string, it is interpreted as a path. If the path specifies a regular file, that file will be opened as a text file and produce a list with one item for each line in the file (end-of-line characters will be stripped). If the path specifies a directory, produce a list containing all of the entries in the directory. Note that parameter expansion is not performed lists produced this way.
+
+If the value is a JSON object, it is evaluated as a list function described below.
+
+h2. List functions
+
+When run-command is evaluating a list (such as "command"), in addition to string parameter substitution, you can use list items functions.
+
+h3. foreach
+
+The @foreach@ list item function (not to be confused with the @task.foreach@ directive) expands a command template for each item in the specified user parameter (the value of the user parameter is evaluated in a list context, as described below). The following example will evaluate "command" to @["echo", "--something", "alice", "--something", "bob"]@:
+
+<pre>
+"script_parameters": {
+ "command": ["echo", {"foreach": "a", "command": ["--something", "$(a)"]}],
+ "a": ["alice", "bob"]
+}
+</pre>
+
+h3. index
+
+The "index" list item function extracts a single item from a list. The "index" is zero-based (i.e. the first item is at index 0, the second item index 1, etc). The following example will evaluate "command" to ["echo", "--something", "bob"]:
+
+<pre>
+"script_parameters": {
+ "command": ["echo", {"list": "a", index: 1, "command": ["--something", "$(a)"]}],
+ "a": ["alice", "bob"]
+}
+</pre>
+
+h3. filter
+
+Filter the list so that it only includes items that match a regular expression. The following example will evaluate to @["echo", "bob"]@
+
+<pre>
+"script_parameters": {
+ "command": ["echo", {"filter": "a", regex: "b.*"]}],
+ "a": ["alice", "bob"]
+}
+</pre>
+
+h3. group
+
+Generate a list of lists, where items are grouped on common subexpression match. Items which don't match the regular expression are excluded. The following example evaluates to @["echo", "--group", "alice", "carol", "dave", "--group", "bob"]@:
+
+<pre>
+"script_parameters": {
+ "command": ["echo", {"foreach": {"group": "a", regex: ".*(a?).*"]}, "command":["--group", {"foreach": "a", "command":"$(a)"}]],
+ "a": ["alice", "bob", "carol", "dave"]
+}
+</pre>
+
+h3. extract
+
+Generate a list of lists, where items are split by subexpression match. Items which don't match the regular expression are excluded. The following example evaluates to @["echo", "c", "a", "rol", "d", "a", "ve"]@:
+
+<pre>
+"script_parameters": {
+ "command": ["echo", {"foreach": {"extract": "a", regex: "(.+)(a)(.*)"]}, "command":[{"foreach": "a", "command":"$(a)"}]],
+ "a": ["alice", "bob", "carol", "dave"]
+}
+</pre>
+
+h2. Directives
+
+Directives alter the behavior of run-command. All directives are optional.
+
+h3. task.cwd
+
+This directive sets the initial current working directory that your command will run in. If @task.cwd@ is not specified, the default current working directory is @task.outdir at .
+
+h3. task.stdin and task.stdout
+
+Provide standard input and standard output redirection.
+
+ at task.stdin@ must evalute to a path to a file to be bound to the commands's standard input stream.
+
+ at task.stdout@ specifies the desired file name in the output directory to save the content of standard output.
+
+h3. task.vwd
+
+Background: because Keep collections are read-only, this does not play well with certain tools that expect to be able to write their outputs alongside their inputs (such as tools that generate indexes that are closely associated with the original file.) The run-command's solution to this is the "virtual working directory".
+
+ at task.vwd@ specifies a Keep collection with the starting contents of the directory. Run-command will then populate @task.outdir@ with directories and symlinks to mirror the contents of the @task.vwd@ collection. Your command will then be able to both access its input files and write it output files in @task.outdir at . When the command completes, the output collection will merge the output of your command with the contents of the starting collection. Note that files in the starting collection remain read-only and cannot be altered or deleted.
+
+h3. task.foreach
+
+Using @task.foreach@, you can run your command concurrently over large datasets.
+
+ at task.foreach@ takes the names of one or more user-defined parameters. The value of these parameters are evaluated in a list context. Run-command then generates tasks based on the cartesian product (i.e. all combinations) of the input lists. The outputs of all tasks are merged to create the final output collection. Note that if two tasks output a file in the same directory with the same name, that file will be concatinated in the final output. In the following example, three tasks will be created for the "grep" command, based on the contents of user parameter "a":
<pre>
"script_parameters": {
- "varA": "ABC",
- "varB": "$(varA)"
+ "command": ["echo", "$(a)"],
+ "task.foreach": "a",
+ "a": ["alice", "bob", "carol"]
}
</pre>
+This evaluates to the commands:
+<notextile>
+<pre>
+["echo", "alice"]
+["echo", "bob"]
+["echo", "carol"]
+</pre>
+</notextile>
+You can also specify multiple parameters:
-h3. Directives
+<pre>
+"script_parameters": {
+ "command": ["echo", "$(a)", "$(b)"],
+ "task.foreach": ["a", "b"],
+ "a": ["alice", "bob"],
+ "b": ["carol", "dave"]
+}
+</pre>
-h3. Command template
+This evaluates to the commands:
-The command section
+<pre>
+["echo", "alice", "carol"]
+["echo", "alice", "dave"]
+["echo", "bob", "carol"]
+["echo", "bob", "dave"]
+</pre>
-h2. Examples
+h1. Examples
<notextile>{% code 'run_command_simple_example' as javascript %}</notextile>
commit 6f2855585712ef7222f739a5afcef0aa736e01ac
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Oct 3 13:43:39 2014 -0400
4042: Functions defined in get_items() now moved into expand_items()
diff --git a/crunch_scripts/run-command b/crunch_scripts/run-command
index 279a0e6..d620c7e 100755
--- a/crunch_scripts/run-command
+++ b/crunch_scripts/run-command
@@ -97,27 +97,6 @@ def expand_item(p, c):
params = copy.copy(p)
params[var] = items[int(c["index"])]
return expand_list(params, c["command"])
- elif isinstance(c, list):
- return expand_list(p, c)
- elif isinstance(c, basestring):
- return [subst.do_substitution(p, c)]
-
- return []
-
-def expand_list(p, l):
- if isinstance(l, basestring):
- return expand_item(p, l)
- else:
- return [exp for arg in l for exp in expand_item(p, arg)]
-
-def add_to_group(gr, match):
- m = ('^_^').join(match.groups())
- if m not in gr:
- gr[m] = []
- gr[m].append(match.group(0))
-
-def get_items(p, value):
- if isinstance(value, dict):
if "regex" in value:
pattern = re.compile(value["regex"])
if "filter" in value:
@@ -139,6 +118,28 @@ def get_items(p, value):
if p:
r.append(p.groups())
return r
+ elif isinstance(c, list):
+ return expand_list(p, c)
+ elif isinstance(c, basestring):
+ return [subst.do_substitution(p, c)]
+
+ return []
+
+def expand_list(p, l):
+ if isinstance(l, basestring):
+ return expand_item(p, l)
+ else:
+ return [exp for arg in l for exp in expand_item(p, arg)]
+
+def add_to_group(gr, match):
+ m = ('^_^').join(match.groups())
+ if m not in gr:
+ gr[m] = []
+ gr[m].append(match.group(0))
+
+def get_items(p, value):
+ if isinstance(value, dict):
+ return expand_list(p, value)
if isinstance(value, list):
return expand_list(p, value)
commit 81e7eb42156a21e17d81c4a71d1dfee4f0fbd52f
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Oct 3 13:42:47 2014 -0400
4042: Subst will raise exceptions if files/directories don't exist.
diff --git a/crunch_scripts/crunchutil/subst.py b/crunch_scripts/crunchutil/subst.py
index ff34863..b7336fa 100644
--- a/crunch_scripts/crunchutil/subst.py
+++ b/crunch_scripts/crunchutil/subst.py
@@ -1,5 +1,6 @@
import os
import glob
+import stat
class SubstitutionError(Exception):
pass
@@ -35,13 +36,23 @@ def search(c):
return None
def sub_file(v):
- return os.path.join(os.environ['TASK_KEEPMOUNT'], v)
+ path = os.path.join(os.environ['TASK_KEEPMOUNT'], v)
+ st = os.stat(path)
+ if st and stat.S_ISREG(st.st_mode):
+ return path
+ else:
+ raise SubstitutionError("$(file {}) is not accessable or is not a regular file".format(path))
def sub_dir(v):
d = os.path.dirname(v)
if d == '':
d = v
- return os.path.join(os.environ['TASK_KEEPMOUNT'], d)
+ path = os.path.join(os.environ['TASK_KEEPMOUNT'], d)
+ st = os.stat(path)
+ if st and stat.S_ISDIR(st.st_mode):
+ return path
+ else:
+ raise SubstitutionError("$(dir {}) is not accessable or is not a directory".format(path))
def sub_basename(v):
return os.path.splitext(os.path.basename(v))[0]
@@ -49,7 +60,7 @@ def sub_basename(v):
def sub_glob(v):
l = glob.glob(v)
if len(l) == 0:
- raise SubstitutionError("$(glob): No match on '%s'" % v)
+ raise SubstitutionError("$(glob {}) no match fonud".format(v))
else:
return l[0]
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list