[ARVADOS] updated: 927d11b99091dc4493446681209bec514c92f1bf
git at public.curoverse.com
git at public.curoverse.com
Tue Mar 11 16:13:38 EDT 2014
Summary of changes:
doc/_includes/_webring.liquid | 29 +++++++++
doc/_layouts/default.html.liquid | 3 +-
.../check-environment.html.textile.liquid | 2 -
.../getting_started/workbench.html.textile.liquid | 2 -
doc/user/index.html.textile.liquid | 1 -
doc/user/topics/keep.html.textile.liquid | 19 +++++-
...nning-pipeline-command-line.html.textile.liquid | 61 +++++++++++++++-----
...rial-gatk-variantfiltration.html.textile.liquid | 2 +-
doc/user/topics/tutorial-job1.html.textile.liquid | 2 +-
.../topics/tutorial-parallel.html.textile.liquid | 2 +-
.../running-external-program.html.textile.liquid | 10 +--
.../tutorial-firstscript.html.textile.liquid | 11 ++--
.../tutorials/tutorial-keep.html.textile.liquid | 4 -
.../tutorial-new-pipeline.html.textile.liquid | 19 ++++---
14 files changed, 115 insertions(+), 52 deletions(-)
create mode 100644 doc/_includes/_webring.liquid
via 927d11b99091dc4493446681209bec514c92f1bf (commit)
via 8c90084189d5ddb42359688aabd57f77ae867000 (commit)
via 4d3ece2c9b2b840a2899c03be215bf222fd52b13 (commit)
via 0191262856619bfaaa62629cc2f0b0ef15b5b9a2 (commit)
via 024bc6551b9d06a5fa9a8aa7c71ce1895e8a0ede (commit)
via 9a3fd18942055b0ec78f85aad3df4bbd2cc4f548 (commit)
via aa961ea35ffb01f88777d8e86336627e88da725d (commit)
via e86868d96caea7af41140e9d3e26f40b144a0ff6 (commit)
via 9ab861b00881ea7a5737d89b90ad17ceb264c8da (commit)
from fdbef7b63085492ddbc9201726508d63d0c8bf8f (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 927d11b99091dc4493446681209bec514c92f1bf
Author: Brett Smith <brett at curoverse.com>
Date: Tue Mar 11 16:12:11 2014 -0400
doc: Note warning for get one collection block.
Added per discussion with Tom. We're also interested in fixing the
warning, but in the meantime, this is worth documenting.
diff --git a/doc/user/topics/keep.html.textile.liquid b/doc/user/topics/keep.html.textile.liquid
index b8d787d..dae133a 100644
--- a/doc/user/topics/keep.html.textile.liquid
+++ b/doc/user/topics/keep.html.textile.liquid
@@ -23,6 +23,16 @@ Let's use @arv keep get@ to download the first datablock:
notextile. <pre><code>~$ <span class="userinput">cd /scratch/<b>you</b></span>
/scratch/<b>you</b>$ <span class="userinput">arv keep get 204e43b8a1185621ca55a94839582e6f+67108864 > block1</span></code></pre>
+{% include 'notebox_begin' %}
+
+When you run this command, you may get this API warning:
+
+notextile. <pre><code>WARNING:root:API lookup failed for collection 204e43b8a1185621ca55a94839582e6f+67108864 (<class 'apiclient.errors.HttpError'>: <HttpError 404 when requesting https://qr1hi.arvadosapi.com/arvados/v1/collections/204e43b8a1185621ca55a94839582e6f%2B67108864?alt=json returned "Not Found">)</code></pre>
+
+This happens because @arv keep get@ tries to find a collection with this identifier. When that fails, it emits this warning, then looks for a datablock instead, which succeeds.
+
+{% include 'notebox_end' %}
+
Let's look at the size and compute the md5 hash of @block1@:
<notextile>
commit 8c90084189d5ddb42359688aabd57f77ae867000
Author: Brett Smith <brett at curoverse.com>
Date: Tue Mar 11 16:08:19 2014 -0400
doc: Improve "How Keep Works" flow.
This addresses the fact that this page was broken off from the
original Keep tutorial page.
diff --git a/doc/user/topics/keep.html.textile.liquid b/doc/user/topics/keep.html.textile.liquid
index 4d169af..b8d787d 100644
--- a/doc/user/topics/keep.html.textile.liquid
+++ b/doc/user/topics/keep.html.textile.liquid
@@ -8,10 +8,10 @@ In Keep, information is stored in *data blocks*. Data blocks are normally betwe
In order to reassemble the file, Keep stores a *collection* data block which lists in sequence the data blocks that make up the original file. A collection data block may store the information for multiple files, including a directory structure.
-In this example we will use @c1bad4b39ca5a924e481008009d94e32+210@ which we added to keep in the previous section. First let us examine the contents of this collection using @arv keep get@:
+In this example we will use @c1bad4b39ca5a924e481008009d94e32+210@ which we added to Keep in "the first Keep tutorial":{{ site.baseurl }}/users/tutorial/tutorial-keep.html. First let us examine the contents of this collection using @arv keep get@:
<notextile>
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep get c1bad4b39ca5a924e481008009d94e32+210</span>
+<pre><code>~$ <span class="userinput">arv keep get c1bad4b39ca5a924e481008009d94e32+210</span>
. 204e43b8a1185621ca55a94839582e6f+67108864 b9677abbac956bd3e86b1deb28dfac03+67108864 fc15aff2a762b13f521baf042140acec+67108864 323d2a3ce20370c4ca1d3462a344f8fd+25885655 0:227212247:var-GS000016015-ASM.tsv.bz2
</code></pre>
</notextile>
@@ -20,7 +20,8 @@ The command @arv keep get@ fetches the contents of the locator @c1bad4b39ca5a924
Let's use @arv keep get@ to download the first datablock:
-notextile. <pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep get 204e43b8a1185621ca55a94839582e6f+67108864 > block1</span></code></pre>
+notextile. <pre><code>~$ <span class="userinput">cd /scratch/<b>you</b></span>
+/scratch/<b>you</b>$ <span class="userinput">arv keep get 204e43b8a1185621ca55a94839582e6f+67108864 > block1</span></code></pre>
Let's look at the size and compute the md5 hash of @block1@:
@@ -32,6 +33,6 @@ Let's look at the size and compute the md5 hash of @block1@:
</code></pre>
</notextile>
-Notice that the block identifer <code>204e43b8a1185621ca55a94839582e6f+67108864</code> of:
+Notice that the block identifer <code>204e43b8a1185621ca55a94839582e6f+67108864</code> consists of:
* the md5 hash @204e43b8a1185621ca55a94839582e6f@ which matches the md5 hash of @block1@
* a size hint @67108864@ which matches the size of @block1@
commit 4d3ece2c9b2b840a2899c03be215bf222fd52b13
Author: Brett Smith <brett at curoverse.com>
Date: Tue Mar 11 15:28:05 2014 -0400
doc: Make CLI pipeline run up-to-date.
This refreshes Keep hashes and the output format of `arv pipeline run`.
diff --git a/doc/user/topics/running-pipeline-command-line.html.textile.liquid b/doc/user/topics/running-pipeline-command-line.html.textile.liquid
index cda0540..1b8550f 100644
--- a/doc/user/topics/running-pipeline-command-line.html.textile.liquid
+++ b/doc/user/topics/running-pipeline-command-line.html.textile.liquid
@@ -40,11 +40,13 @@ You can run this pipeline from the command line using @arv pipeline run@, fillin
2013-12-16 14:08:40 +0000 -- pipeline_instance qr1hi-d1hrv-vxzkp38nlde9yyr
do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 queued 2013-12-16T14:08:40Z
filter - -
+
2013-12-16 14:08:51 +0000 -- pipeline_instance qr1hi-d1hrv-vxzkp38nlde9yyr
-do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 e2ccd204bca37c77c0ba59fc470cd0f7+162
+do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 8e1b6acdd3f2f1da722538127c5c6202+56
filter qr1hi-8i9sb-w5k40fztqgg9i2x queued 2013-12-16T14:08:50Z
+
2013-12-16 14:09:01 +0000 -- pipeline_instance qr1hi-d1hrv-vxzkp38nlde9yyr
-do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 e2ccd204bca37c77c0ba59fc470cd0f7+162
+do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 8e1b6acdd3f2f1da722538127c5c6202+56
filter qr1hi-8i9sb-w5k40fztqgg9i2x 735ac35adf430126cf836547731f3af6+56
</code></pre>
</notextile>
@@ -56,11 +58,11 @@ Arvados adds each pipeline component to the job queue as its dependencies are sa
The Keep locators of the output of each of @"do_hash"@ and @"filter"@ component are available from the output log shown above. The output is also available on the Workbench by navigating to %(rarr)→% Compute %(rarr)→% Pipeline instances %(rarr)→% pipeline uuid under the *id* column %(rarr)→% components.
<notextile>
-<pre><code>$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt</span>
+<pre><code>~$ <span class="userinput">arv keep get 8e1b6acdd3f2f1da722538127c5c6202+56/md5sum.txt</span>
0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
504938460ef369cd275e4ef58994cffe bob.txt
8f3b36aff310e06f3c5b9e95678ff77a carol.txt
-$ <span class="userinput">arv keep get 735ac35adf430126cf836547731f3af6+56/0-filter.txt</span>
+~$ <span class="userinput">arv keep get 735ac35adf430126cf836547731f3af6+56/0-filter.txt</span>
0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
</code></pre>
</notextile>
@@ -69,7 +71,7 @@ Indeed, the filter has picked out just the "alice" file as having a hash that st
h3. Running a pipeline with different parameters
-Notice that the pipeline definition explicitly specifies the Keep locator for the input:
+Notice that the pipeline template explicitly specifies the Keep locator for the input:
<notextile>
<pre><code>...
@@ -85,32 +87,32 @@ Notice that the pipeline definition explicitly specifies the Keep locator for th
You can specify values for pipeline component script_parameters like this:
<notextile>
-<pre><code>$ <span class="userinput">arv pipeline run --template qr1hi-d1hrv-vxzkp38nlde9yyr do_hash::input=33a9f3842b01ea3fdf27cc582f5ea2af+242</span>
+<pre><code>~$ <span class="userinput">arv pipeline run --template qr1hi-p5p6p-xxxxxxxxxxxxxxx do_hash::input=c1bad4b39ca5a924e481008009d94e32+210</span>
2013-12-17 20:31:24 +0000 -- pipeline_instance qr1hi-d1hrv-tlkq20687akys8e
do_hash qr1hi-8i9sb-rffhuay4jryl2n2 queued 2013-12-17T20:31:24Z
filter - -
+
2013-12-17 20:31:34 +0000 -- pipeline_instance qr1hi-d1hrv-tlkq20687akys8e
do_hash qr1hi-8i9sb-rffhuay4jryl2n2 {:done=>1, :running=>1, :failed=>0, :todo=>0}
filter - -
-2013-12-17 20:31:44 +0000 -- pipeline_instance qr1hi-d1hrv-tlkq20687akys8e
-do_hash qr1hi-8i9sb-rffhuay4jryl2n2 {:done=>1, :running=>1, :failed=>0, :todo=>0}
-filter - -
+
2013-12-17 20:31:55 +0000 -- pipeline_instance qr1hi-d1hrv-tlkq20687akys8e
do_hash qr1hi-8i9sb-rffhuay4jryl2n2 880b55fb4470b148a447ff38cacdd952+54
filter qr1hi-8i9sb-j347g1sqovdh0op queued 2013-12-17T20:31:55Z
+
2013-12-17 20:32:05 +0000 -- pipeline_instance qr1hi-d1hrv-tlkq20687akys8e
do_hash qr1hi-8i9sb-rffhuay4jryl2n2 880b55fb4470b148a447ff38cacdd952+54
-filter qr1hi-8i9sb-j347g1sqovdh0op fb728f0ffe152058fa64b9aeed344cb5+54
+filter qr1hi-8i9sb-j347g1sqovdh0op 490cd451c8108824b8a17e3723e1f236+19
</code></pre>
</notextile>
Now check the output:
<notextile>
-<pre><code>$ <span class="userinput">arv keep get 880b55fb4470b148a447ff38cacdd952+54/md5sum.txt</span>
+<pre><code>~$ <span class="userinput">arv keep get 880b55fb4470b148a447ff38cacdd952+54/md5sum.txt</span>
44b8ae3fde7a8a88d2f7ebd237625b4f var-GS000016015-ASM.tsv.bz2
-$ <span class="userinput">arv keep get fb728f0ffe152058fa64b9aeed344cb5+54/0-filter.txt</span>
-
+~$ <span class="userinput">arv keep get 490cd451c8108824b8a17e3723e1f236+19/0-filter.txt</span>
+~$
</code></pre>
</notextile>
commit 0191262856619bfaaa62629cc2f0b0ef15b5b9a2
Author: Brett Smith <brett at curoverse.com>
Date: Tue Mar 11 15:26:52 2014 -0400
doc: Improve flow of CLI pipeline run.
The previous version assumed that we already had a static pipeline
template. That's no longer true under the current flow, so instruct
the user to make one at the start of the tutorial.
diff --git a/doc/user/topics/running-pipeline-command-line.html.textile.liquid b/doc/user/topics/running-pipeline-command-line.html.textile.liquid
index 3f85077..cda0540 100644
--- a/doc/user/topics/running-pipeline-command-line.html.textile.liquid
+++ b/doc/user/topics/running-pipeline-command-line.html.textile.liquid
@@ -4,10 +4,39 @@ navsection: userguide
title: "Running a pipeline on the command line"
...
-It is possible run pipelines on the command line using @arv pipeline run@ using the UUID that you received from @arv pipeline create@:
+In "Writing a pipeline":{{ site.baseurl }}/user/tutorials/tutorial-firstscript.html, we learned how to create a pipeline template on the command-line. Let's create one that doesn't require any user input to start:
<notextile>
-<pre><code>$ <span class="userinput">arv pipeline run --template qr1hi-p5p6p-xxxxxxxxxxxxxxx</span>
+<pre><code>~$ <span class="userinput">cat >the_pipeline <<EOF
+{
+ "name":"Filter md5 hash values",
+ "components":{
+ "do_hash":{
+ "script":"hash.py",
+ "script_parameters":{
+ "input": "887cd41e9c613463eab2f0d885c6dd96+83"
+ },
+ "script_version":"<b>you</b>:master"
+ },
+ "filter":{
+ "script":"0-filter.py",
+ "script_parameters":{
+ "input":{
+ "output_of":"do_hash"
+ }
+ },
+ "script_version":"<b>you</b>:master"
+ }
+ }
+}
+EOF</span>
+~$ <span class="userinput">arv pipeline_template create --pipeline-template "$(cat the_pipeline)"</span></code></pre>
+</notextile>
+
+You can run this pipeline from the command line using @arv pipeline run@, filling in the UUID that you received from @arv pipeline_template create@:
+
+<notextile>
+<pre><code>~$ <span class="userinput">arv pipeline run --template qr1hi-p5p6p-xxxxxxxxxxxxxxx</span>
2013-12-16 14:08:40 +0000 -- pipeline_instance qr1hi-d1hrv-vxzkp38nlde9yyr
do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 queued 2013-12-16T14:08:40Z
filter - -
commit 024bc6551b9d06a5fa9a8aa7c71ce1895e8a0ede
Author: Brett Smith <brett at curoverse.com>
Date: Tue Mar 11 14:11:22 2014 -0400
doc: Consistent style in outside program tutorial.
diff --git a/doc/user/tutorials/running-external-program.html.textile.liquid b/doc/user/tutorials/running-external-program.html.textile.liquid
index be257f8..b555d77 100644
--- a/doc/user/tutorials/running-external-program.html.textile.liquid
+++ b/doc/user/tutorials/running-external-program.html.textile.liquid
@@ -41,7 +41,7 @@ Next, add the file to @git@ staging, commit and push:
You should now be able to run your new script using Crunch, with "script" referring to our new "run-md5sum.py" script.
<notextile>
-<pre><code>$ <span class="userinput">cat >the_pipeline <<EOF
+<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">cat >~/the_pipeline <<EOF
{
"name":"Run external md5sum program",
"components":{
@@ -53,16 +53,12 @@ You should now be able to run your new script using Crunch, with "script" referr
"dataclass": "Collection"
}
},
- "script_version":"you:master"
+ "script_version":"<b>you</b>:master"
}
}
}
EOF
-</span></code></pre>
-</notextile>
-
-<notextile>
-<pre><code>$ <span class="userinput">arv pipeline_template create --pipeline-template "$(cat the_pipeline)"</span>
+</span>~/<b>you</b>/crunch_scripts$ <span class="userinput">arv pipeline_template create --pipeline-template "$(cat ~/the_pipeline)"</span>
</code></pre>
</notextile>
commit 9a3fd18942055b0ec78f85aad3df4bbd2cc4f548
Author: Brett Smith <brett at curoverse.com>
Date: Tue Mar 11 14:04:39 2014 -0400
doc: Consistent style in new pipeline tutorial.
diff --git a/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid b/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid
index fe849a5..b09e624 100644
--- a/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid
@@ -8,18 +8,21 @@ A pipeline in Arvados is a collection of crunch scripts, in which the output fro
*This tutorial assumes that you are "logged into an Arvados VM instance":{{site.baseurl}}/user/getting_started/ssh-access.html#login, and have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html*
+This tutorial uses *@you@* to denote your username. Replace *@you@* with your user name in all the following examples.
+
h2. Create a new script
-Our second script will filter the output of @hash.py@ and only include hashes that start with 0. Create a new script in @crunch_scripts/@ called @0-filter.py@:
+Our second script will filter the output of @hash.py@ and only include hashes that start with 0. Create a new script in <notextile><code>~/<b>you</b>/crunch_scripts/</code></notextile> called @0-filter.py@:
<notextile> {% code '0_filter_py' as python %} </notextile>
Now add it to git:
<notextile>
-<pre><code>$ <span class="userinput">git add 0-filter.py</span>
-$ <span class="userinput">git commit -m"zero filter"</span>
-$ <span class="userinput">git push origin master</span>
+<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">chmod +x 0-filter.py</span>
+~/<b>you</b>/crunch_scripts$ <span class="userinput">git add 0-filter.py</span>
+~/<b>you</b>/crunch_scripts$ <span class="userinput">git commit -m"zero filter"</span>
+~/<b>you</b>/crunch_scripts$ <span class="userinput">git push origin master</span>
</code></pre>
</notextile>
@@ -28,7 +31,7 @@ h2. Create a pipeline template
Next, create a file that contains the pipeline definition:
<notextile>
-<pre><code>$ <span class="userinput">cat >the_pipeline <<EOF
+<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">cat >~/the_pipeline <<EOF
{
"name":"Filter md5 hash values",
"components":{
@@ -40,7 +43,7 @@ Next, create a file that contains the pipeline definition:
"dataclass": "Collection"
}
},
- "script_version":"you:master"
+ "script_version":"<b>you</b>:master"
},
"filter":{
"script":"0-filter.py",
@@ -49,7 +52,7 @@ Next, create a file that contains the pipeline definition:
"output_of":"do_hash"
}
},
- "script_version":"you:master"
+ "script_version":"<b>you</b>:master"
}
}
}
@@ -62,7 +65,7 @@ EOF
Now, use @arv pipeline_template create@ tell Arvados about your pipeline template:
<notextile>
-<pre><code>$ <span class="userinput">arv pipeline_template create --pipeline-template "$(cat the_pipeline)"</span>
+<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">arv pipeline_template create --pipeline-template "$(cat ~/the_pipeline)"</span>
</code></pre>
</notextile>
commit aa961ea35ffb01f88777d8e86336627e88da725d
Author: Brett Smith <brett at curoverse.com>
Date: Tue Mar 11 13:50:08 2014 -0400
doc: Small bugfixes for first pipeline template.
diff --git a/doc/user/tutorials/tutorial-firstscript.html.textile.liquid b/doc/user/tutorials/tutorial-firstscript.html.textile.liquid
index 97a7547..245e890 100644
--- a/doc/user/tutorials/tutorial-firstscript.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-firstscript.html.textile.liquid
@@ -98,7 +98,8 @@ h2. Create a pipeline template
Next, create a file that contains the pipeline definition:
<notextile>
-<pre><code>$ <span class="userinput">cat >the_pipeline <<EOF
+<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">cd ~</span>
+~$ <span class="userinput">cat >the_pipeline <<EOF
{
"name":"My first pipeline",
"components":{
@@ -120,7 +121,7 @@ EOF
* @cat@ is a standard Unix utility that simply copies standard input to standard output
* @<<EOF@ tells the shell to direct the following lines into the standard input for @cat@ up until it sees the line @EOF@
-* @>the_job@ redirects standard output to a file called @the_job@
+* @>the_pipeline@ redirects standard output to a file called @the_pipeline@
* @"name"@ is a human-readable name for the pipeline
* @"components"@ is a set of scripts that make up the pipeline
* The component is listed with a human-readable name (@"do_hash"@ in this example)
@@ -131,7 +132,7 @@ EOF
Now, use @arv pipeline_template create@ tell Arvados about your pipeline template:
<notextile>
-<pre><code>$ <span class="userinput">arv pipeline_template create --pipeline-template "$(cat the_pipeline)"</span>
+<pre><code>~$ <span class="userinput">arv pipeline_template create --pipeline-template "$(cat the_pipeline)"</span>
</code></pre>
</notextile>
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list