[ARVADOS] updated: d071c34ca20aa86a5a053abcffb7414dbd8f4933

git at public.curoverse.com git at public.curoverse.com
Thu Oct 16 16:32:39 EDT 2014


Summary of changes:
 crunch_scripts/run-command                      |  5 ++-
 doc/user/topics/arv-run.html.textile.liquid     | 60 +++++++++++++++++++------
 doc/user/topics/run-command.html.textile.liquid |  4 ++
 sdk/python/arvados/commands/run.py              | 25 ++++++++---
 sdk/python/arvados/commands/ws.py               | 22 ++++++---
 sdk/python/arvados/events.py                    | 48 +++++++++++++++-----
 sdk/python/setup.py                             |  2 +-
 sdk/python/tests/test_websockets.py             | 55 +++++++++++------------
 services/keepstore/perms.go                     | 27 +++++------
 services/keepstore/perms_test.go                | 31 ++++++++++---
 10 files changed, 193 insertions(+), 86 deletions(-)

       via  d071c34ca20aa86a5a053abcffb7414dbd8f4933 (commit)
       via  6ef24c81cd9d84e5022b776f63328b8ea352c4d1 (commit)
       via  ac4f4ab5c6d677096cc335af7cfc0d9b10043b93 (commit)
       via  25708a232b618757d54d36e3fe545302cff9ca42 (commit)
       via  0dab3a40d540b872ee689420d3fcfcebb760d15b (commit)
       via  fa541df794b486efa220bd83bb119e87cca8e2de (commit)
       via  04b8def3069732ac8038c39f1541cdc11cf265a0 (commit)
       via  d05825ffb039ddddfff00cf92837d0ca049ee364 (commit)
       via  66b6518c70a67a94317feaa47e555808bd13b015 (commit)
       via  f02109541ae4f462c4d1838d78ad06cf318098ce (commit)
       via  23796191a7efdae462ee7509b4641be4d63d584d (commit)
       via  245c6abec4bf824ad8c8683e59ab6ee08c836cf7 (commit)
       via  8297039dfdf78e712855a1a223c5ea15094244ba (commit)
       via  723f0c8477dbd1d6a5e90a29dd3003c9b00d04aa (commit)
       via  b19a6058168a290fe789b2228c13935edc6e5546 (commit)
       via  1d0d4b16be0e9478dd6f98014a30610c5e579afa (commit)
       via  556fb8540d73e3af07a5bea07e3c831fe477b958 (commit)
       via  5e2eeeaabfdd2673ee58cc0da54e943fd2841d04 (commit)
       via  f39807a3c5dd32520bebd4fcd5254f214351eb4f (commit)
      from  76ad4467c9b5f223383520f04157b1721a62aa5f (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.


commit d071c34ca20aa86a5a053abcffb7414dbd8f4933
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Thu Oct 16 16:32:30 2014 -0400

    3609: Further documentation improvements, --local now runs pipeline runner instead of just arv-crunch-job, added --no-reuse and --no-wait

diff --git a/crunch_scripts/run-command b/crunch_scripts/run-command
index e21089e..6c27a94 100755
--- a/crunch_scripts/run-command
+++ b/crunch_scripts/run-command
@@ -352,7 +352,10 @@ try:
     while len(pids) > 0:
         (pid, status) = os.wait()
         pids.discard(pid)
-        rcode[pid] = (status >> 8)
+        if not taskp.get("task.ignore_rcode"):
+            rcode[pid] = (status >> 8)
+        else:
+            rcode[pid] = 0
 
     if sig.sig is not None:
         logger.critical("terminating on signal %s" % sig.sig)
diff --git a/doc/user/topics/arv-run.html.textile.liquid b/doc/user/topics/arv-run.html.textile.liquid
index dcf03ed..0d7d8c1 100644
--- a/doc/user/topics/arv-run.html.textile.liquid
+++ b/doc/user/topics/arv-run.html.textile.liquid
@@ -44,6 +44,18 @@ h2. Parallel tasks
 HWI-ST1027_129_D0THKACXX.1_1.fastq  HWI-ST1027_129_D0THKACXX.1_2.fastq
 $ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC -- *.fastq</span>
 Running pipeline qr1hi-d1hrv-mg3bju0u7r6w241
+[...]
+Thu Oct 16 19:27:42 2014 qr1hi-8i9sb-r0n6w78aq0knsoj 2331 0 stderr run-command: parallelizing on input0 with items [u'/keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq', u'/keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_2.fastq']
+[...]
+Thu Oct 16 19:27:45 2014 qr1hi-8i9sb-r0n6w78aq0knsoj 2331 1 stderr run-command: grep -H -n ATTGGAGGAAAGATGAGTGAC /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq
+Thu Oct 16 19:27:46 2014 qr1hi-8i9sb-r0n6w78aq0knsoj 2331 2 stderr run-command: grep -H -n ATTGGAGGAAAGATGAGTGAC /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_2.fastq
+[...]
+Thu Oct 16 19:27:46 2014 qr1hi-8i9sb-r0n6w78aq0knsoj 2331 1 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:14:TCTGGCCCCTGTTGTCTGCATGTAACTTAATACCACAACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCCCAACCTA
+Thu Oct 16 19:27:46 2014 qr1hi-8i9sb-r0n6w78aq0knsoj 2331 1 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:18:AACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCT
+Thu Oct 16 19:27:46 2014 qr1hi-8i9sb-r0n6w78aq0knsoj 2331 1 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:30:ATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCTGTGATACG
+Thu Oct 16 19:27:47 2014 qr1hi-8i9sb-r0n6w78aq0knsoj 2331 1 stderr run-command: completed with exit code 0 (success)
+Thu Oct 16 19:27:47 2014 qr1hi-8i9sb-r0n6w78aq0knsoj 2331 2 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_2.fastq:34:CTGGCCCCTGTTGTCTGCATGTAACTTAATACCACAACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAG
+Thu Oct 16 19:27:47 2014 qr1hi-8i9sb-r0n6w78aq0knsoj 2331 2 stderr run-command: completed with exit code 0 (success)
 </pre>
 </notextile>
 
diff --git a/doc/user/topics/run-command.html.textile.liquid b/doc/user/topics/run-command.html.textile.liquid
index c025008..7d92876 100644
--- a/doc/user/topics/run-command.html.textile.liquid
+++ b/doc/user/topics/run-command.html.textile.liquid
@@ -180,6 +180,10 @@ h3. task.cwd
 
 This directive sets the initial current working directory that your command will run in.  If @task.cwd@ is not specified, the default current working directory is @task.outdir at .
 
+h3. task.ignore_rcode
+
+By Unix convention a task which exits with a non-zero return code is considered failed.  However, some programs (such as @grep@) return non-zero codes for conditions that should not be considered fatal errors.  Set @"task.ignore_rcode": true@ to indicate the task should always succeed.
+
 h3. task.stdin and task.stdout
 
 Provide standard input and standard output redirection.
diff --git a/sdk/python/arvados/commands/run.py b/sdk/python/arvados/commands/run.py
index 2375f58..7b0cb35 100644
--- a/sdk/python/arvados/commands/run.py
+++ b/sdk/python/arvados/commands/run.py
@@ -18,6 +18,9 @@ arvrun_parser = argparse.ArgumentParser()
 arvrun_parser.add_argument('--dry-run', action="store_true", help="Print out the pipeline that would be submitted and exit")
 arvrun_parser.add_argument('--local', action="store_true", help="Run locally using arv-crunch-job")
 arvrun_parser.add_argument('--docker-image', type=str, default="arvados/jobs", help="Docker image to use, default arvados/jobs")
+arvrun_parser.add_argument('--ignore-rcode', action="store_true", help="Set this to indicate commands that return non-zero return codes should not be considered failed.")
+arvrun_parser.add_argument('--no-reuse', action="store_true", help="Do not reuse past jobs.")
+arvrun_parser.add_argument('--no-wait', action="store_true", help="Do not wait and display logs after submitting command, just exit.")
 arvrun_parser.add_argument('--git-dir', type=str, default="", help="Git repository passed to arv-crunch-job when using --local")
 arvrun_parser.add_argument('--repository', type=str, default="arvados", help="repository field of component, default 'arvados'")
 arvrun_parser.add_argument('--script-version', type=str, default="master", help="script_version field of component, default 'master'")
@@ -196,6 +199,8 @@ def main(arguments=None):
         component["script_parameters"]["task.foreach"] = task_foreach
 
     component["script_parameters"]["command"] = slots[2:]
+    if args.ignore_rcode:
+        component["script_parameters"]["task.ignore_rcode"] = args.ignore_rcode
 
     pipeline = {
         "name": " | ".join([s[0] for s in slots[2:]]),
@@ -203,18 +208,27 @@ def main(arguments=None):
         "components": {
             "command": component
         },
-        "state":"RunningOnServer"
+        "state": "RunningOnClient" if args.local else "RunningOnServer"
     }
 
     if args.dry_run:
         print(json.dumps(pipeline, indent=4))
-    elif args.local:
-        subprocess.call(["arv-crunch-job", "--job", json.dumps(component), "--git-dir", args.git_dir])
     else:
         api = arvados.api('v1')
         pi = api.pipeline_instances().create(body=pipeline).execute()
         print "Running pipeline %s" % pi["uuid"]
-        ws.main(["--pipeline", pi["uuid"]])
+
+        if args.local:
+            subprocess.call(["arv-run-pipeline-instance", "--instance", pi["uuid"], "--run-jobs-here"] + (["--no-reuse"] if args.no_reuse else []))
+        elif not args.no_wait:
+            ws.main(["--pipeline", pi["uuid"]])
+
+        pi = api.pipeline_instances().get(uuid=pi["uuid"]).execute()
+        print "Pipeline is %s" % pi["state"]
+        if "output_uuid" in pi["components"]["command"]:
+            print "Output is %s" % pi["components"]["command"]["output_uuid"]
+        else:
+            print "No output"
 
 if __name__ == '__main__':
     main()
diff --git a/sdk/python/arvados/commands/ws.py b/sdk/python/arvados/commands/ws.py
index 674daad..04e3f64 100644
--- a/sdk/python/arvados/commands/ws.py
+++ b/sdk/python/arvados/commands/ws.py
@@ -71,6 +71,14 @@ def main(arguments=None):
             elif ev["event_type"] in ("create", "update"):
                 if ev["object_kind"] == "arvados#pipelineInstance":
                     update_subscribed_components(ev["properties"]["new_attributes"]["components"])
+
+                if ev["object_kind"] == "arvados#pipelineInstance" and args.pipeline:
+                    if ev["properties"]["new_attributes"]["state"] in ("Complete", "Failed", "Paused"):
+                        ws.close()
+
+                if ev["object_kind"] == "arvados#job" and args.job:
+                    if ev["properties"]["new_attributes"]["state"] in ("Complete", "Failed", "Cancelled"):
+                        ws.close()
         elif 'status' in ev and ev['status'] == 200:
             pass
         else:
@@ -82,9 +90,9 @@ def main(arguments=None):
             if args.pipeline:
                 c = api.pipeline_instances().get(uuid=args.pipeline).execute()
                 update_subscribed_components(c["components"])
-
-            while True:
-                signal.pause()
+                if c["state"] in ("Complete", "Failed", "Paused"):
+                    ws.close()
+            ws.run_forever()
     except KeyboardInterrupt:
         pass
     except Exception as e:
diff --git a/sdk/python/arvados/events.py b/sdk/python/arvados/events.py
index e6038fc..d1abc0f 100644
--- a/sdk/python/arvados/events.py
+++ b/sdk/python/arvados/events.py
@@ -78,9 +78,19 @@ class PollClient(threading.Thread):
             self.id = max_id
             self.stop.wait(self.poll_time)
 
+    def run_forever(self):
+        self.stop.wait()
+
     def close(self):
         self.stop.set()
-        self.join()
+        try:
+            self.join()
+        except RuntimeError:
+            # "join() raises a RuntimeError if an attempt is made to join the
+            # current thread as that would cause a deadlock. It is also an
+            # error to join() a thread before it has been started and attempts
+            # to do so raises the same exception."
+            pass
 
     def subscribe(self, filters):
         self.on_event({'status': 200})

commit 6ef24c81cd9d84e5022b776f63328b8ea352c4d1
Merge: fa541df ac4f4ab
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Thu Oct 16 15:35:53 2014 -0400

    Merge branch 'master' into 3609-arv-run


commit fa541df794b486efa220bd83bb119e87cca8e2de
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Thu Oct 16 15:21:03 2014 -0400

    3609: Set "name" to abbreviated form, put full command line in the description field.  Continuing work on documentation.

diff --git a/doc/user/topics/arv-run.html.textile.liquid b/doc/user/topics/arv-run.html.textile.liquid
index 5613fdb..dcf03ed 100644
--- a/doc/user/topics/arv-run.html.textile.liquid
+++ b/doc/user/topics/arv-run.html.textile.liquid
@@ -35,19 +35,19 @@ Thu Oct 16 17:30:42 2014 qr1hi-8i9sb-8wdaabnughiolpy 13541 0 stderr run-command:
 
 A key feature of @arv-run@ is the ability to introspect the command line to determine which arguments are file inputs, and transform those paths so they are usable inside the Arvados container.  In the above example, @HWI-ST1027_129_D0THKACXX.1_2.fastq@ is transformed into @/keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq at .  In the above example, @arv-run@ works together with @arv-mount@ to identify that the file is already part of an Arvados collection.  In this case, it will use the existing collection without any upload step.  If you specify a file that is only available on the local filesystem, @arv-run@ will upload a new collection and use that.
 
- at arv-run@ will parallelize on the files listed on the command line after @-- at .  You may specify @--batch-size N@ after the @--@ but before listing any files to specify how many files to provide put on the command line for each task.  The syntax is designed to mimic standard shell syntax, so it is usually necessary to quote the metacharacters < > and | as either \< \> and \| or '<' '>' and '|'.
+h2. Parallel tasks
+
+ at arv-run@ will parallelize over files listed on the command line after @-- at .
 
 <notextile>
 <pre>
-$ <span class="userinput">cd ~/keep/by_id/3229739b505d2b878b62aed09895a55a+142</span>
-$ <span class="userinput">ls *.fastq</span>
 HWI-ST1027_129_D0THKACXX.1_1.fastq  HWI-ST1027_129_D0THKACXX.1_2.fastq
-$ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC -- *.fastq \> output.txt</span>
+$ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC -- *.fastq</span>
 Running pipeline qr1hi-d1hrv-mg3bju0u7r6w241
 </pre>
 </notextile>
 
-You may use stdin @<@ redirection on multiple files.  This will create a separate task for each input file:
+You may use also stdin @<@ redirection on multiple files.  This will create a separate task for each input file.  Because the syntax is designed to mimic standard shell syntax, it is necessary to quote the metacharacters @<@, @>@ and @|@ as either @\<@, @\>@ and @\|@ or @'<'@, @'>'@ and @'|'@.
 
 <notextile>
 <pre>
@@ -65,6 +65,8 @@ Use @arv-run --dry-run@ to print out the final Arvados pipeline generated by @ar
 
 By default, the pipeline will be submitted to your configured Arvado instance.  Use @arv-run --local@ to run the command locally using "arv-crunch-job".
 
+You may specify @--batch-size N@ after the @--@ but before listing any files to specify how many files to provide put on the command line for each task.
+
 h1. Examples
 
 Run one @grep@ task per file, with each input files piped from stdin.  Redirect the output to output.txt.
diff --git a/sdk/python/arvados/commands/run.py b/sdk/python/arvados/commands/run.py
index 06c4bf8..2375f58 100644
--- a/sdk/python/arvados/commands/run.py
+++ b/sdk/python/arvados/commands/run.py
@@ -198,7 +198,8 @@ def main(arguments=None):
     component["script_parameters"]["command"] = slots[2:]
 
     pipeline = {
-        "name": " ".join(starting_args),
+        "name": " | ".join([s[0] for s in slots[2:]]),
+        "description": "@" + " ".join(starting_args) + "@",
         "components": {
             "command": component
         },

commit 04b8def3069732ac8038c39f1541cdc11cf265a0
Merge: 66b6518 d05825f
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Thu Oct 16 15:05:57 2014 -0400

    Merge branch '3609-arv-ws' into 3609-arv-run


commit 66b6518c70a67a94317feaa47e555808bd13b015
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Thu Oct 16 13:44:55 2014 -0400

    Better documentation WIP

diff --git a/doc/user/topics/arv-run.html.textile.liquid b/doc/user/topics/arv-run.html.textile.liquid
index 186c732..5613fdb 100644
--- a/doc/user/topics/arv-run.html.textile.liquid
+++ b/doc/user/topics/arv-run.html.textile.liquid
@@ -10,16 +10,37 @@ The @arv-run@ command enables you create Arvados pipelines at the command line t
 
 h1. Usage
 
- at arv-run@ takes a command or command pipeline, along with stdin and stdout redirection, and creates an Arvados pipeline to run the command.  The syntax is designed to mimic standard shell syntax, so it is usually necessary to quote the metacharacters < > and | as either \< \> and \| or '<' '>' and '|'.
+Using @arv-run@ you can write and test command lines interactively, then insert @arv-run@ at the beginning of the command line to run the command on Arvados.  For example:
 
- at arv-run@ introspects the command line to determine which arguments are file inputs.  If you specify a file that is only available on the local filesystem, it will be first uploaded to Arvados, and then the command line will be rewritten to refer to the newly uploaded file.  @arv-run@ also works together with @arv-mount@ to identify if a file specified on the command line is part of an Arvados collection.  If so, the command line will be rewritten to refer to the file within the collection without any upload necessary.
+<notextile>
+<pre>
+$ <span class="userinput">cd ~/keep/by_id/3229739b505d2b878b62aed09895a55a+142</span>
+$ <span class="userinput">ls *.fastq</span>
+HWI-ST1027_129_D0THKACXX.1_1.fastq  HWI-ST1027_129_D0THKACXX.1_2.fastq
+$ <span class="userinput">grep -H -n ATTGGAGGAAAGATGAGTGAC HWI-ST1027_129_D0THKACXX.1_1.fastq</span>
+HWI-ST1027_129_D0THKACXX.1_1.fastq:14:TCTGGCCCCTGTTGTCTGCATGTAACTTAATACCACAACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCCCAACCTA
+HWI-ST1027_129_D0THKACXX.1_1.fastq:18:AACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCT
+HWI-ST1027_129_D0THKACXX.1_1.fastq:30:ATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCTGTGATACG
+$ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC HWI-ST1027_129_D0THKACXX.1_1.fastq</span>
+Running pipeline qr1hi-d1hrv-mg3bju0u7r6w241
+[...]
+Thu Oct 16 17:30:41 2014 qr1hi-8i9sb-8wdaabnughiolpy 13541 0 stderr run-command: grep -H -n ATTGGAGGAAAGATGAGTGAC /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq
+Thu Oct 16 17:30:41 2014 qr1hi-8i9sb-8wdaabnughiolpy 13541 0 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:14:TCTGGCCCCTGTTGTCTGCATGTAACTTAATACCACAACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCCCAACCTA
+Thu Oct 16 17:30:41 2014 qr1hi-8i9sb-8wdaabnughiolpy 13541 0 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:18:AACCAGGCATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCT
+Thu Oct 16 17:30:41 2014 qr1hi-8i9sb-8wdaabnughiolpy 13541 0 stderr /keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq:30:ATAGGGGAAAGATTGGAGGAAAGATGAGTGACAGCATCAACTTCTCTCACAACCTAGGCCAGTAAGTAGTGCTTGTGCTCATCTCCTTGGCTGTGATACG
+Thu Oct 16 17:30:42 2014 qr1hi-8i9sb-8wdaabnughiolpy 13541 0 stderr run-command: completed with exit code 0 (success)
+[...]
+</pre>
+</notextile>
+
+A key feature of @arv-run@ is the ability to introspect the command line to determine which arguments are file inputs, and transform those paths so they are usable inside the Arvados container.  In the above example, @HWI-ST1027_129_D0THKACXX.1_2.fastq@ is transformed into @/keep/3229739b505d2b878b62aed09895a55a+142/HWI-ST1027_129_D0THKACXX.1_1.fastq at .  In the above example, @arv-run@ works together with @arv-mount@ to identify that the file is already part of an Arvados collection.  In this case, it will use the existing collection without any upload step.  If you specify a file that is only available on the local filesystem, @arv-run@ will upload a new collection and use that.
 
- at arv-run@ will parallelize on the files listed on the command line after @-- at .  You may specify @--batch-size N@ after the @--@ but before listing any files to specify how many files to provide put on the command line for each task (see below for example).
+ at arv-run@ will parallelize on the files listed on the command line after @-- at .  You may specify @--batch-size N@ after the @--@ but before listing any files to specify how many files to provide put on the command line for each task.  The syntax is designed to mimic standard shell syntax, so it is usually necessary to quote the metacharacters < > and | as either \< \> and \| or '<' '>' and '|'.
 
 <notextile>
 <pre>
 $ <span class="userinput">cd ~/keep/by_id/3229739b505d2b878b62aed09895a55a+142</span>
-$ <span class="userinput">ls</span>
+$ <span class="userinput">ls *.fastq</span>
 HWI-ST1027_129_D0THKACXX.1_1.fastq  HWI-ST1027_129_D0THKACXX.1_2.fastq
 $ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC -- *.fastq \> output.txt</span>
 Running pipeline qr1hi-d1hrv-mg3bju0u7r6w241

commit f02109541ae4f462c4d1838d78ad06cf318098ce
Merge: 245c6ab 2379619
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Thu Oct 16 13:21:01 2014 -0400

    Merge branch '3609-arv-ws' into 3609-arv-run


commit 245c6abec4bf824ad8c8683e59ab6ee08c836cf7
Merge: 556fb85 723f0c8
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Thu Oct 16 13:18:27 2014 -0400

    Merge branch '3609-arv-ws' into 3609-arv-run


commit 556fb8540d73e3af07a5bea07e3c831fe477b958
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Thu Oct 16 11:28:40 2014 -0400

    Making arv-run documentation more readable.

diff --git a/doc/user/topics/arv-run.html.textile.liquid b/doc/user/topics/arv-run.html.textile.liquid
index b406e6b..186c732 100644
--- a/doc/user/topics/arv-run.html.textile.liquid
+++ b/doc/user/topics/arv-run.html.textile.liquid
@@ -8,27 +8,31 @@ The @arv-run@ command enables you create Arvados pipelines at the command line t
 
 {% include 'tutorial_expectations' %}
 
-h1. Quick introduction
+h1. Usage
+
+ at arv-run@ takes a command or command pipeline, along with stdin and stdout redirection, and creates an Arvados pipeline to run the command.  The syntax is designed to mimic standard shell syntax, so it is usually necessary to quote the metacharacters < > and | as either \< \> and \| or '<' '>' and '|'.
 
-Run one @grep@ task per file, and redirect the output to output.txt
+ at arv-run@ introspects the command line to determine which arguments are file inputs.  If you specify a file that is only available on the local filesystem, it will be first uploaded to Arvados, and then the command line will be rewritten to refer to the newly uploaded file.  @arv-run@ also works together with @arv-mount@ to identify if a file specified on the command line is part of an Arvados collection.  If so, the command line will be rewritten to refer to the file within the collection without any upload necessary.
+
+ at arv-run@ will parallelize on the files listed on the command line after @-- at .  You may specify @--batch-size N@ after the @--@ but before listing any files to specify how many files to provide put on the command line for each task (see below for example).
 
 <notextile>
 <pre>
 $ <span class="userinput">cd ~/keep/by_id/3229739b505d2b878b62aed09895a55a+142</span>
+$ <span class="userinput">ls</span>
+HWI-ST1027_129_D0THKACXX.1_1.fastq  HWI-ST1027_129_D0THKACXX.1_2.fastq
 $ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC -- *.fastq \> output.txt</span>
 Running pipeline qr1hi-d1hrv-mg3bju0u7r6w241
 </pre>
 </notextile>
 
-h1. Usage
-
- at arv-run@ takes a command or command pipeline, along with stdin and stdout redirection, and creates an Arvados pipeline to run the command.  The syntax is designed to mimic standard shell syntax, so it is usually necessary to quote the metacharacters < > and | as either \< \> and \| or '<' '>' and '|'.
+You may use stdin @<@ redirection on multiple files.  This will create a separate task for each input file:
 
- at arv-run@ introspects the command line to determine which arguments are file inputs.  If you specify a file that is only available on the local filesystem, it will be first uploaded to Arvados, and then the command line will be rewritten to refer to the newly uploaded file.  @arv-run@ also works together with @arv-mount@ to identify if a file specified on the command line is part of an Arvados collection.  If so, the command line will be rewritten to refer to the file within the collection without any upload necessary.
-
- at arv-run@ will parallelize on the files listed on the command line after @-- at .  You may specify @--batch-size N@ after the @--@ but before listing any files to specify how many files to provide put on the command line for each task (see below for example).
-
-You may use stdin @<@ redirection on multiple files.  This will create a separate task for each input file.
+<notextile>
+<pre>
+$ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC \< *.fastq \> output.txt</span>
+</pre>
+</notextile>
 
 You are only permitted to supply a single file name for stdout @>@ redirection.  If there are multiple tasks, their output will be collated at the end of the pipeline.  Alternately, you may use "run-command":run-command.html parameter substitution in the file name to generate different filenames for each task.
 
@@ -44,11 +48,6 @@ h1. Examples
 
 Run one @grep@ task per file, with each input files piped from stdin.  Redirect the output to output.txt.
 
-<notextile>
-<pre>
-$ <span class="userinput">arv-run grep -H -n ATTGGAGGAAAGATGAGTGAC \< *.fastq \> output.txt</span>
-</pre>
-</notextile>
 
 Run @cat | grep@ once per file.  Redirect the output to output.txt.
 

-----------------------------------------------------------------------


hooks/post-receive
-- 




More information about the arvados-commits mailing list