[ARVADOS] updated: 69fab19d5c5fdb685a4b6651ac2077a467a6ca9a
git at public.curoverse.com
git at public.curoverse.com
Fri Jan 17 16:20:56 EST 2014
Summary of changes:
doc/user/getting_started/ssh-access.textile | 15 ++++-----
doc/user/tutorials/tutorial-firstscript.textile | 18 +++++++----
.../tutorial-gatk-variantfiltration.textile | 8 ++--
doc/user/tutorials/tutorial-job-debug.textile | 4 +-
doc/user/tutorials/tutorial-job1.textile | 31 ++++++++++---------
doc/user/tutorials/tutorial-keep.textile | 23 ++++++++++----
doc/user/tutorials/tutorial-new-pipeline.textile | 10 +++---
doc/user/tutorials/tutorial-parallel.textile | 6 ++--
doc/user/tutorials/tutorial-trait-search.textile | 22 +++++++-------
9 files changed, 75 insertions(+), 62 deletions(-)
via 69fab19d5c5fdb685a4b6651ac2077a467a6ca9a (commit)
via 7d6aad89ecb3005d5a9daedce265077ae1f19fcb (commit)
from 807c64b1fe731cff16e887876221dc2949e620d6 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 69fab19d5c5fdb685a4b6651ac2077a467a6ca9a
Merge: 7d6aad8 807c64b
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Jan 17 16:21:24 2014 -0500
Merge branch 'master' of git.clinicalfuture.com:arvados
commit 7d6aad89ecb3005d5a9daedce265077ae1f19fcb
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Jan 17 16:20:58 2014 -0500
refs #1912
* ssh-access tweaked to emphasize need for user to use their own user name and not 'you' and changed example ssh config to use *.qr1hi to *.arvados
* tutorial-firstscript suggests using nano to edit files instead of leaving the user hanging
* Fixed inconsistent keep locators for manifests caused by removal of +K at qr1hi from locators
* Removed +K at qr1hi from all locators used in examples
diff --git a/doc/user/getting_started/ssh-access.textile b/doc/user/getting_started/ssh-access.textile
index 430e80f..1e9b022 100644
--- a/doc/user/getting_started/ssh-access.textile
+++ b/doc/user/getting_started/ssh-access.textile
@@ -149,7 +149,7 @@ Paste the public key that you copied to the clipboard in the previous section in
h1(#login). Using ssh to log into an Arvados VM
-To determine the name and login information of the VM instance that you will connect to, go to the Arvados workbench and click on "Virtual machines" under *Configuration* or use the menu item _Access %(rarr)→% VMs_ to see a list of virtual machines that you have access to. The *hostname* columns lists the name of each available VM. The *logins* column will have a value in the form of @["you"]@. Ignore the square brackets and quotes to get your login name. In this guide the hostname will be _shell_ and the login will be _you_. Replace these with your hostname and login as appropriate.
+To determine the name and login information of the VM instance that you will connect to, go to the Arvados workbench and click on "Virtual machines" under *Configuration* or use the menu item _Access %(rarr)→% VMs_ to see a list of virtual machines that you have access to. The *hostname* columns lists the name of each available VM. The *logins* column will have a value in the form of @["you"]@. Skip the square brackets and quotes to get your login name. In this guide the hostname will be _shell_ and the login will be _you_. Replace these with your hostname and login as appropriate.
This section consists of two sets of instructions, depending on whether you will be logging in using a "Unix":#unixvm (Linux, OS X, Cygwin) or "Windows":#windowsvm client.
@@ -157,9 +157,9 @@ h2(#unixvm). Logging in using command line ssh (Unix)
h3. Connecting to the VM
-Use the following command to connect to the "shell" VM instance as "you". Replace <code>you at shell</code> at the end of the following command with your login and hostname:
+Use the following command to connect to the "shell" VM instance as "you". Replace *<code>you at shell</code>* at the end of the following command with your *login* and *hostname* from Workbench:
-notextile. <pre><code>$ <span class="userinput">ssh -o "ProxyCommand ssh -a -x -p2222 turnout at switchyard.{{ site.arvados_api_host }} shell" -A -x you at shell</span></code></pre>
+notextile. <pre><code>$ <span class="userinput">ssh -o "ProxyCommand ssh -a -x -p2222 turnout at switchyard.{{ site.arvados_api_host }} shell" -A -x <b>you at shell</b></span></code></pre>
There are several things going on here:
@@ -173,7 +173,7 @@ The VMs typically have addresses that are not globally routable, so you cannot l
* @shell@ is the name of the VM that we want to connect to. This is sent to the switchyard server as if it were an ssh command, and the switchyard server connects to the VM on our behalf.
* After the ProxyCommand section, the @-x@ must be repeated because it applies to the connection to VM instead of the switchyard.
* @-A@ specifies that we want to forward access to @ssh-agent@ to the VM.
-* Finally, <code>you at shell</code> specifies your username and repeats the hostname of the VM. The username can be found in the *logins* column in the VMs Workbench page, discussed above.
+* Finally, *<code>you at shell</code>* specifies your username and repeats the hostname of the VM. The username can be found in the *logins* column in the VMs Workbench page, discussed above.
You should now be able to log into the Arvados VM and "check your environment.":check-environment.html
@@ -182,17 +182,16 @@ h3. Configuration (optional)
Since the above command line is cumbersome, it can be greatly simplfied by adding the following section your @~/.ssh/config@ file:
<notextile>
-<pre><code class="userinput">Host *.qr1hi
+<pre><code class="userinput">Host *.arvados
ProxyCommand ssh -a -x -p2222 turnout at switchyard.{{ site.arvados_api_host }} $SSH_PROXY_FLAGS %h
- User you
ForwardAgent yes
ForwardX11 no
</code></pre>
</notextile>
-This will recognize any host ending in ".qr1hi" and automatically apply the proxy, user and forwarding settings discussed previously. For example:
+This will recognize any host ending in ".arvados" and automatically apply the proxy, user and forwarding settings from the configuration file, allowing you to log in with a much simpler command:
-notextile. <pre><code>$ <span class="userinput">ssh shell.qr1hi</span></code></pre>
+notextile. <pre><code>$ <span class="userinput">ssh <b>you at shell</b>.arvados</span></code></pre>
h2(#windowsvm). Logging in using PuTTY (Windows)
diff --git a/doc/user/tutorials/tutorial-firstscript.textile b/doc/user/tutorials/tutorial-firstscript.textile
index dbc3631..4917cf9 100644
--- a/doc/user/tutorials/tutorial-firstscript.textile
+++ b/doc/user/tutorials/tutorial-firstscript.textile
@@ -41,12 +41,12 @@ For more information about using @git@, try
notextile. <pre><code>$ <span class="userinput">man gittutorial</span></code></pre>
-or "click here to search Google for git tutorials":http://google.com/#q=git+tutorial
+or <b>"click here to search Google for git tutorials":http://google.com/#q=git+tutorial</b>
{% include notebox-end.html %}
h2. Creating a Crunch script
-Start by entering the @you@ directory, creating a subdirectory called @crunch_scripts@ and changing to that directory:
+Start by entering the @you@ directory created by @git clone at . Next create a subdirectory called @crunch_scripts@ and change to that directory:
<notextile>
<pre><code>$ <span class="userinput">cd you</span>
@@ -54,7 +54,11 @@ $ <span class="userinput">mkdir crunch_scripts</span>
$ <span class="userinput">cd crunch_scripts</span></code></pre>
</notextile>
-Next, using your favorite text editor, create a new file called @hash.py@ in the @crunch_scripts@ directory. Add the following code to compute the md5 hash of each file in a collection:
+Next, using @nano@ or your favorite Unix text editor, create a new file called @hash.py@ in the @crunch_scripts@ directory.
+
+notextile. <pre>$ <code class="userinput">nano hash.py</code></pre>
+
+Add the following code to compute the md5 hash of each file in a collection:
<pre><code class="userinput">{% include tutorial_hash_script.py %}</code></pre>
@@ -87,7 +91,7 @@ To git at git.qr1hi.arvadosapi.com:you.git
* [new branch] master -> master</code></pre>
</notextile>
-You should now be able to run your script using Crunch, similar to how we did it in the "first tutorial.":tutorial-job1.html The field @"script_version"@ should be @you:master@ to tell Crunch to run the script that you just uploaded.
+You should now be able to run your script using Crunch, similar to how we did it in the "first tutorial.":tutorial-job1.html The field @"script_version"@ should be @you:master@ to tell Crunch to run the script at the head of the "master" git branch, which you just uploaded.
<notextile>
<pre><code>$ <span class="userinput">cat >the_job <<EOF
@@ -96,7 +100,7 @@ You should now be able to run your script using Crunch, similar to how we did it
"script_version": "you:master",
"script_parameters":
{
- "input": "33a9f3842b01ea3fdf27cc582f5ea2af"
+ "input": "c1bad4b39ca5a924e481008009d94e32+210"
}
}
EOF</span>
@@ -109,10 +113,10 @@ $ <span class="userinput">arv -h job create --job "$(cat the_job)"</span>
$ <span class="userinput">arv -h job get --uuid qr1hi-xxxxx-xxxxxxxxxxxxxxx</span>
{
...
- "output":"880b55fb4470b148a447ff38cacdd952+54+K at qr1hi",
+ "output":"880b55fb4470b148a447ff38cacdd952+54",
...
}
-$ <span class="userinput">arv keep get 880b55fb4470b148a447ff38cacdd952+54+K at qr1hi/md5sum.txt</span>
+$ <span class="userinput">arv keep get 880b55fb4470b148a447ff38cacdd952+54/md5sum.txt</span>
44b8ae3fde7a8a88d2f7ebd237625b4f var-GS000016015-ASM.tsv.bz2
</code></pre>
</notextile>
diff --git a/doc/user/tutorials/tutorial-gatk-variantfiltration.textile b/doc/user/tutorials/tutorial-gatk-variantfiltration.textile
index 77306d0..8163bd3 100644
--- a/doc/user/tutorials/tutorial-gatk-variantfiltration.textile
+++ b/doc/user/tutorials/tutorial-gatk-variantfiltration.textile
@@ -189,23 +189,23 @@ Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 1 child 4984 started o
Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 status: 1 done, 1 running, 0 todo
Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 1 child 4984 on compute13.1 exit 0 signal 0 success=true
Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 1 success in 110 seconds
-Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 1 output bedd6ff56b3ae9f90d873b1fcb72f9a3+91+K at qr1hi
+Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 1 output bedd6ff56b3ae9f90d873b1fcb72f9a3+91
Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 wait for last 0 children to finish
Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 status: 2 done, 0 running, 0 todo
Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 release job allocation
Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 Freeze not implemented
Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 collate
Tue Dec 17 19:04:10 2013 salloc: Job allocation 1251 has been revoked.
-Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 output bedd6ff56b3ae9f90d873b1fcb72f9a3+91+K at qr1hi
+Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 output bedd6ff56b3ae9f90d873b1fcb72f9a3+91
Tue Dec 17 19:04:11 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 finish
-Tue Dec 17 19:04:12 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 log manifest is 1e77aaceee2df499e14dc5dde5c3d328+91+K at qr1hi
+Tue Dec 17 19:04:12 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 log manifest is 1e77aaceee2df499e14dc5dde5c3d328+91
</code></pre>
</notextile>
Once the job completes, the output can be found in hu34D5B9-exome-filtered.vcf:
<notextile>
-$ <span class="userinput">arv keep ls bedd6ff56b3ae9f90d873b1fcb72f9a3+91+K at qr1hi</span>
+$ <span class="userinput">arv keep ls bedd6ff56b3ae9f90d873b1fcb72f9a3+91</span>
hu34D5B9-exome-filtered.vcf
</code></pre>
</notextile>
diff --git a/doc/user/tutorials/tutorial-job-debug.textile b/doc/user/tutorials/tutorial-job-debug.textile
index 9ea454c..24364f0 100644
--- a/doc/user/tutorials/tutorial-job-debug.textile
+++ b/doc/user/tutorials/tutorial-job-debug.textile
@@ -64,7 +64,7 @@ $ <span class="userinput">arv-crunch-job --job "$(cat the_job)"</span>
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827 Freeze not implemented
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827 collate
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827 output d41d8cd98f00b204e9800998ecf8427e+0
-2013-12-12_21:36:44 qr1hi-8i9sb-okzukfzkpbrnhst 29827 meta key is c00bfbd58e6f58ce3bebdd47f745a70f+1857+K at qr1hi
+2013-12-12_21:36:44 qr1hi-8i9sb-okzukfzkpbrnhst 29827 meta key is c00bfbd58e6f58ce3bebdd47f745a70f+1857
</code></pre>
</notextile>
@@ -127,7 +127,7 @@ $ <span class="userinput">arv-crunch-job --job "$(cat the_job)"</span>
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578 collate
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578 output 576c44d762ba241b0a674aa43152b52a+53
2013-12-12_21:57:03 qr1hi-8i9sb-79260ykfew5trzl 31578 finish
-2013-12-12_21:57:04 qr1hi-8i9sb-79260ykfew5trzl 31578 meta key is 9f937693334d0c9234ccc1f808ee7117+1761+K at qr1hi
+2013-12-12_21:57:04 qr1hi-8i9sb-79260ykfew5trzl 31578 meta key is 9f937693334d0c9234ccc1f808ee7117+1761
</code></pre>
</notextile>
diff --git a/doc/user/tutorials/tutorial-job1.textile b/doc/user/tutorials/tutorial-job1.textile
index 253c4bd..982eb75 100644
--- a/doc/user/tutorials/tutorial-job1.textile
+++ b/doc/user/tutorials/tutorial-job1.textile
@@ -35,7 +35,7 @@ Crunch jobs are described using JSON objects. For example:
"script_version": "arvados:master",
"script_parameters":
{
- "input": "33a9f3842b01ea3fdf27cc582f5ea2af"
+ "input": "c1bad4b39ca5a924e481008009d94e32+210"
}
}
EOF
@@ -68,7 +68,7 @@ Use @arv job create@ to actually submit the job. It should print out a JSON obj
"priority":null,
"script":"hash",
"script_parameters":{
- "input":"33a9f3842b01ea3fdf27cc582f5ea2af"
+ "input":"c1bad4b39ca5a924e481008009d94e32+210"
},
"script_version":"d9cd657b733d578ac0d2167dd75967aa4f22e0ac",
"cancelled_at":null,
@@ -84,7 +84,7 @@ Use @arv job create@ to actually submit the job. It should print out a JSON obj
"runtime_constraints":{},
"tasks_summary":{},
"dependencies":[
- "33a9f3842b01ea3fdf27cc582f5ea2af"
+ "c1bad4b39ca5a924e481008009d94e32+210"
],
"log_stream_href":"https://qr1hi.arvadosapi.com/arvados/v1/jobs/qr1hi-8i9sb-1pm1t02dezhupss/log_tail_follow"
}
@@ -129,7 +129,7 @@ You can access the job output under the *output* column of the _Compute %(rarr)&
"priority":null,
"script":"hash",
"script_parameters":{
- "input":"33a9f3842b01ea3fdf27cc582f5ea2af"
+ "input":"c1bad4b39ca5a924e481008009d94e32+210"
},
"script_version":"d9cd657b733d578ac0d2167dd75967aa4f22e0ac",
"cancelled_at":null,
@@ -137,11 +137,11 @@ You can access the job output under the *output* column of the _Compute %(rarr)&
"cancelled_by_user_uuid":null,
"started_at":"2013-12-16T20:44:36Z",
"finished_at":"2013-12-16T20:44:53Z",
- "output":"880b55fb4470b148a447ff38cacdd952+54+K at qr1hi",
+ "output":"880b55fb4470b148a447ff38cacdd952+54",
"success":true,
"running":false,
"is_locked_by_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
- "log":"2afdc6c8b67372ffd22d8ce89d35411f+91+K at qr1hi",
+ "log":"2afdc6c8b67372ffd22d8ce89d35411f+91",
"runtime_constraints":{},
"tasks_summary":{
"done":2,
@@ -150,19 +150,19 @@ You can access the job output under the *output* column of the _Compute %(rarr)&
"todo":0
},
"dependencies":[
- "33a9f3842b01ea3fdf27cc582f5ea2af"
+ "c1bad4b39ca5a924e481008009d94e32+210"
],
"log_stream_href":null
}
</code></pre>
</notextile>
-* @"output"@ is the unique identifier for this specific job's output. This is a Keep collection. Because the output of Arvados jobs should be deterministic, the known expected output is <code>880b55fb4470b148a447ff38cacdd952+54+K at qr1hi</code>.
+* @"output"@ is the unique identifier for this specific job's output. This is a Keep collection. Because the output of Arvados jobs should be deterministic, the known expected output is <code>880b55fb4470b148a447ff38cacdd952+54</code>.
Now you can list the files in the collection:
<notextile>
-<pre><code>$ <span class="userinput">arv keep get 880b55fb4470b148a447ff38cacdd952+54+K at qr1hi</span>
+<pre><code>$ <span class="userinput">arv keep get 880b55fb4470b148a447ff38cacdd952+54</span>
. 78b268d1e03d87f8270bdee9d5d427c5+61 0:61:md5sum.txt
</code></pre>
</notextile>
@@ -170,7 +170,7 @@ Now you can list the files in the collection:
This collection consists of the @md5sum.txt@ file. Use @arv keep get@ to show the contents of the @md5sum.txt@ file:
<notextile>
-<pre><code>$ <span class="userinput">arv keep get 880b55fb4470b148a447ff38cacdd952+54+K at qr1hi/md5sum.txt</span>
+<pre><code>$ <span class="userinput">arv keep get 880b55fb4470b148a447ff38cacdd952+54/md5sum.txt</span>
44b8ae3fde7a8a88d2f7ebd237625b4f var-GS000016015-ASM.tsv.bz2
</code></pre>
</notextile>
@@ -182,15 +182,15 @@ h2. The job log
When the job completes, you can access the job log. The keep identifier listed in the @"log"@ field from @arv job get@ specifies a collection. You can list the files in the collection:
<notextile>
-<pre><code>$ <span class="userinput">arv keep ls 2afdc6c8b67372ffd22d8ce89d35411f+91+K at qr1hi</span>
-qr1hi-8i9sb-1pm1t02dezhupss.log.txt
+<pre><code>$ <span class="userinput">arv keep ls xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+91</span>
+qr1hi-8i9sb-xxxxxxxxxxxxxxx.log.txt
</code></pre>
</notextile>
The log collection consists of one log file named with the job id. You can access it using @arv keep get@:
<notextile>
-<pre><code>$ <span class="userinput">arv keep get 2afdc6c8b67372ffd22d8ce89d35411f+91+K at qr1hi/qr1hi-8i9sb-1pm1t02dezhupss.log.txt</span>
+<pre><code>$ <span class="userinput">arv keep get xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+91/qr1hi-8i9sb-xxxxxxxxxxxxxxx.log.txt</span>
2013-12-16_20:44:35 qr1hi-8i9sb-1pm1t02dezhupss 7575 check slurm allocation
2013-12-16_20:44:35 qr1hi-8i9sb-1pm1t02dezhupss 7575 node compute13 - 8 slots
2013-12-16_20:44:36 qr1hi-8i9sb-1pm1t02dezhupss 7575 start
@@ -199,12 +199,13 @@ The log collection consists of one log file named with the job id. You can acce
2013-12-16_20:44:37 qr1hi-8i9sb-1pm1t02dezhupss 7575 Install exited 0
2013-12-16_20:44:37 qr1hi-8i9sb-1pm1t02dezhupss 7575 script hash
2013-12-16_20:44:37 qr1hi-8i9sb-1pm1t02dezhupss 7575 script_version d9cd657b733d578ac0d2167dd75967aa4f22e0ac
-2013-12-16_20:44:37 qr1hi-8i9sb-1pm1t02dezhupss 7575 script_parameters {"input":"33a9f3842b01ea3fdf27cc582f5ea2af"}
+2013-12-16_20:44:37 qr1hi-8i9sb-1pm1t02dezhupss 7575 script_parameters {"input":"c1bad4b39ca5a924e481008009d94e32+210"}
2013-12-16_20:44:37 qr1hi-8i9sb-1pm1t02dezhupss 7575 runtime_constraints {"max_tasks_per_node":0}
2013-12-16_20:44:37 qr1hi-8i9sb-1pm1t02dezhupss 7575 start level 0
2013-12-16_20:44:37 qr1hi-8i9sb-1pm1t02dezhupss 7575 status: 0 done, 0 running, 1 todo
2013-12-16_20:44:38 qr1hi-8i9sb-1pm1t02dezhupss 7575 0 job_task qr1hi-ot0gb-23c1k3kwrf8da62
2013-12-16_20:44:38 qr1hi-8i9sb-1pm1t02dezhupss 7575 0 child 7681 started on compute13.1
+
2013-12-16_20:44:38 qr1hi-8i9sb-1pm1t02dezhupss 7575 status: 0 done, 1 running, 0 todo
2013-12-16_20:44:39 qr1hi-8i9sb-1pm1t02dezhupss 7575 0 child 7681 on compute13.1 exit 0 signal 0 success=true
2013-12-16_20:44:39 qr1hi-8i9sb-1pm1t02dezhupss 7575 0 success in 1 seconds
@@ -224,7 +225,7 @@ The log collection consists of one log file named with the job id. You can acce
2013-12-16_20:44:52 qr1hi-8i9sb-1pm1t02dezhupss 7575 release job allocation
2013-12-16_20:44:52 qr1hi-8i9sb-1pm1t02dezhupss 7575 Freeze not implemented
2013-12-16_20:44:52 qr1hi-8i9sb-1pm1t02dezhupss 7575 collate
-2013-12-16_20:44:53 qr1hi-8i9sb-1pm1t02dezhupss 7575 output 880b55fb4470b148a447ff38cacdd952+54+K at qr1hi
+2013-12-16_20:44:53 qr1hi-8i9sb-1pm1t02dezhupss 7575 output 880b55fb4470b148a447ff38cacdd952+54
2013-12-16_20:44:53 qr1hi-8i9sb-1pm1t02dezhupss 7575 finish
</code></pre>
</notextile>
diff --git a/doc/user/tutorials/tutorial-keep.textile b/doc/user/tutorials/tutorial-keep.textile
index 8255440..fab979e 100644
--- a/doc/user/tutorials/tutorial-keep.textile
+++ b/doc/user/tutorials/tutorial-keep.textile
@@ -21,29 +21,38 @@ The Arvados distributed file system is called *Keep*. Keep is a content-address
h1. Putting Data into Keep
-We will start with downloading a freely available VCF exome from the "Personal Genome Project (PGP)":http://www.personalgenomes.org subject "hu599905":https://my.personalgenomes.org/profile/hu599905 and add it to Keep. From an Arvados VM instance:
+We will start with downloading a freely available VCF file from the "Personal Genome Project (PGP)":http://www.personalgenomes.org subject "hu599905":https://my.personalgenomes.org/profile/hu599905 to a staging directory on the VM, and then add it to Keep.
+
+First, log into the Arvados VM instance and set up the staging area:
+
+notextile. <pre><code>$ <span class="userinput">mkdir /scratch/$USER</span></code></pre>
+
+Next, download the file:
<notextile>
-<pre><code>$ <span class="userinput">curl -o var-GS000016015-ASM.tsv.bz2 'https://warehouse.personalgenomes.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2'</span>
+<pre><code>$ <span class="userinput">mkdir /scratch/$USER</span>
+$ <span class="userinput">cd /scratch/$USER</span>
+$ <span class="userinput">curl -o var-GS000016015-ASM.tsv.bz2 'https://warehouse.personalgenomes.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2'</span>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 216M 100 216M 0 0 10.0M 0 0:00:21 0:00:21 --:--:-- 9361k
</code></pre>
</notextile>
-Alternately, if you have your own data, for example @MyExome.vcf@, you can use @rsync@ on your local computer to copy it to the shell VM:
+Alternately, if you have your own data, for example @MyData.vcf@, you can use @scp@ or @rsync@ to copy from your local workstation to the shell VM (run this on your local workstation):
-notextile. <pre><code>$ <span class="userinput">rsync MyExome.vcf shell.qr1hi:MyExome.vcf</span></code></pre>
+notextile. <pre><code>$ <span class="userinput">scp MyData.vcf <b>you at shell</b>.arvados>:/scratch/MyData.vcf</span></code></pre>
Now use @arv keep put@ to add your VCF data to Keep:
<notextile>
-<pre><code>$ <span class="userinput">arv keep put var-GS000016015-ASM.tsv.bz2</span>
-33a9f3842b01ea3fdf27cc582f5ea2af
+<pre><code>$ <span class="userinput">cd /scratch/$USER</span>
+$ <span class="userinput">arv keep put var-GS000016015-ASM.tsv.bz2</span>
+c1bad4b39ca5a924e481008009d94e32+210
</code></pre>
</notextile>
-The output value @33a9f3842b01ea3fdf27cc582f5ea2af@ is the Keep locator. This enables you to access the file you just uploaded, and is explained in the next section.
+The output value @c1bad4b39ca5a924e481008009d94e32+210@ is the Keep locator. This enables you to access the file you just uploaded, and is explained in the next section.
h2. Putting a directory
diff --git a/doc/user/tutorials/tutorial-new-pipeline.textile b/doc/user/tutorials/tutorial-new-pipeline.textile
index 5277bfc..7a9ecfb 100644
--- a/doc/user/tutorials/tutorial-new-pipeline.textile
+++ b/doc/user/tutorials/tutorial-new-pipeline.textile
@@ -84,10 +84,10 @@ Run the pipeline using @arv pipeline run@, using the UUID that you received from
do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 queued 2013-12-16T14:08:40Z
filter - -
2013-12-16 14:08:51 +0000 -- pipeline_instance qr1hi-d1hrv-vxzkp38nlde9yyr
-do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 e2ccd204bca37c77c0ba59fc470cd0f7+162+K at qr1hi
+do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 e2ccd204bca37c77c0ba59fc470cd0f7+162
filter qr1hi-8i9sb-w5k40fztqgg9i2x queued 2013-12-16T14:08:50Z
2013-12-16 14:09:01 +0000 -- pipeline_instance qr1hi-d1hrv-vxzkp38nlde9yyr
-do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 e2ccd204bca37c77c0ba59fc470cd0f7+162+K at qr1hi
+do_hash qr1hi-8i9sb-hoyc2u964ecv1s6 e2ccd204bca37c77c0ba59fc470cd0f7+162
filter qr1hi-8i9sb-w5k40fztqgg9i2x 735ac35adf430126cf836547731f3af6+56
</code></pre>
</notextile>
@@ -99,7 +99,7 @@ Arvados adds each pipeline component to the job queue as its dependencies are sa
The Keep locators of the output of each of @"do_hash"@ and @"filter"@ component are available from the output log shown above. The output is also available on the Workbench by navigating to %(rarr)→% Compute %(rarr)→% Pipeline instances %(rarr)→% pipeline uuid under the *id* column %(rarr)→% components.
<notextile>
-<pre><code>$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162+K at qr1hi/md5sum.txt</span>
+<pre><code>$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt</span>
0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
504938460ef369cd275e4ef58994cffe bob.txt
8f3b36aff310e06f3c5b9e95678ff77a carol.txt
@@ -139,10 +139,10 @@ filter - -
do_hash qr1hi-8i9sb-rffhuay4jryl2n2 {:done=>1, :running=>1, :failed=>0, :todo=>0}
filter - -
2013-12-17 20:31:55 +0000 -- pipeline_instance qr1hi-d1hrv-tlkq20687akys8e
-do_hash qr1hi-8i9sb-rffhuay4jryl2n2 880b55fb4470b148a447ff38cacdd952+54+K at qr1hi
+do_hash qr1hi-8i9sb-rffhuay4jryl2n2 880b55fb4470b148a447ff38cacdd952+54
filter qr1hi-8i9sb-j347g1sqovdh0op queued 2013-12-17T20:31:55Z
2013-12-17 20:32:05 +0000 -- pipeline_instance qr1hi-d1hrv-tlkq20687akys8e
-do_hash qr1hi-8i9sb-rffhuay4jryl2n2 880b55fb4470b148a447ff38cacdd952+54+K at qr1hi
+do_hash qr1hi-8i9sb-rffhuay4jryl2n2 880b55fb4470b148a447ff38cacdd952+54
filter qr1hi-8i9sb-j347g1sqovdh0op fb728f0ffe152058fa64b9aeed344cb5+54
</code></pre>
</notextile>
diff --git a/doc/user/tutorials/tutorial-parallel.textile b/doc/user/tutorials/tutorial-parallel.textile
index 149da77..b76d1a5 100644
--- a/doc/user/tutorials/tutorial-parallel.textile
+++ b/doc/user/tutorials/tutorial-parallel.textile
@@ -56,7 +56,7 @@ $ <span class="userinput">arv -h job create --job "$(cat the_job)"</span>
$ <span class="userinput">arv -h job get --uuid qr1hi-xxxxx-xxxxxxxxxxxxxxx</span>
{
...
- "output":"e2ccd204bca37c77c0ba59fc470cd0f7+162+K at qr1hi",
+ "output":"e2ccd204bca37c77c0ba59fc470cd0f7+162",
...
}
</code></pre>
@@ -65,11 +65,11 @@ $ <span class="userinput">arv -h job get --uuid qr1hi-xxxxx-xxxxxxxxxxxxxxx</spa
Because the job ran in parallel, each instance of parallel-hash creates a separate @md5sum.txt@ as output. Arvados automatically collates theses files into a single collection, which is the output of the job:
<notextile>
-<pre><code>$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162+K at qr1hi</span>
+<pre><code>$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162</span>
md5sum.txt
md5sum.txt
md5sum.txt
-$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162+K at qr1hi/md5sum.txt</span>
+$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt</span>
0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
504938460ef369cd275e4ef58994cffe bob.txt
8f3b36aff310e06f3c5b9e95678ff77a carol.txt
diff --git a/doc/user/tutorials/tutorial-trait-search.textile b/doc/user/tutorials/tutorial-trait-search.textile
index 0381d28..9ea4cb6 100644
--- a/doc/user/tutorials/tutorial-trait-search.textile
+++ b/doc/user/tutorials/tutorial-trait-search.textile
@@ -183,17 +183,17 @@ https://my.personalgenomes.org/profile/hu7A2F1D 756d0ada29b376140f64e7abfe6aa0e7
https://my.personalgenomes.org/profile/hu553620 7ed4e425bb1c7cc18387cbd9388181df var-GS000015272-ASM.tsv.bz2
https://my.personalgenomes.org/profile/huD09534 542112e210daff30dd3cfea4801a9f2f var-GS000016374-ASM.tsv.bz2
https://my.personalgenomes.org/profile/hu599905 33a9f3842b01ea3fdf27cc582f5ea2af var-GS000016015-ASM.tsv.bz2
-https://my.personalgenomes.org/profile/hu43860C a58dca7609fa84c8c38a7e926a97b2fc+302+K at qr1hi var-GS00253-DNA_A01_200_37-ASM.tsv.bz2
-https://my.personalgenomes.org/profile/huB1FD55 ea30eb9e46eedf7f05ed6e348c2baf5d+291+K at qr1hi var-GS000010320-ASM.tsv.bz2
-https://my.personalgenomes.org/profile/huDF04CC 4ab0df8f22f595d1747a22c476c05873+242+K at qr1hi var-GS000010427-ASM.tsv.bz2
-https://my.personalgenomes.org/profile/hu7A2F1D 756d0ada29b376140f64e7abfe6aa0e7+242+K at qr1hi var-GS000014566-ASM.tsv.bz2
-https://my.personalgenomes.org/profile/hu553620 7ed4e425bb1c7cc18387cbd9388181df+242+K at qr1hi var-GS000015272-ASM.tsv.bz2
-https://my.personalgenomes.org/profile/huD09534 542112e210daff30dd3cfea4801a9f2f+242+K at qr1hi var-GS000016374-ASM.tsv.bz2
-https://my.personalgenomes.org/profile/hu599905 33a9f3842b01ea3fdf27cc582f5ea2af+242+K at qr1hi var-GS000016015-ASM.tsv.bz2
-https://my.personalgenomes.org/profile/hu599905 d6e2e57cd60ba5979006d0b03e45e726+81+K at qr1hi Witch_results.zip
-https://my.personalgenomes.org/profile/hu553620 ea4f2d325592a1272f989d141a917fdd+85+K at qr1hi Devenwood_results.zip
-https://my.personalgenomes.org/profile/hu7A2F1D 4580f6620bb15b25b18373766e14e4a7+85+K at qr1hi Innkeeper_results.zip
-https://my.personalgenomes.org/profile/huD09534 fee37be9440b912eb90f5e779f272416+82+K at qr1hi Hallet_results.zip
+https://my.personalgenomes.org/profile/hu43860C a58dca7609fa84c8c38a7e926a97b2fc+302 var-GS00253-DNA_A01_200_37-ASM.tsv.bz2
+https://my.personalgenomes.org/profile/huB1FD55 ea30eb9e46eedf7f05ed6e348c2baf5d+291 var-GS000010320-ASM.tsv.bz2
+https://my.personalgenomes.org/profile/huDF04CC 4ab0df8f22f595d1747a22c476c05873+242 var-GS000010427-ASM.tsv.bz2
+https://my.personalgenomes.org/profile/hu7A2F1D 756d0ada29b376140f64e7abfe6aa0e7+242 var-GS000014566-ASM.tsv.bz2
+https://my.personalgenomes.org/profile/hu553620 7ed4e425bb1c7cc18387cbd9388181df+242 var-GS000015272-ASM.tsv.bz2
+https://my.personalgenomes.org/profile/huD09534 542112e210daff30dd3cfea4801a9f2f+242 var-GS000016374-ASM.tsv.bz2
+https://my.personalgenomes.org/profile/hu599905 33a9f3842b01ea3fdf27cc582f5ea2af+242 var-GS000016015-ASM.tsv.bz2
+https://my.personalgenomes.org/profile/hu599905 d6e2e57cd60ba5979006d0b03e45e726+81 Witch_results.zip
+https://my.personalgenomes.org/profile/hu553620 ea4f2d325592a1272f989d141a917fdd+85 Devenwood_results.zip
+https://my.personalgenomes.org/profile/hu7A2F1D 4580f6620bb15b25b18373766e14e4a7+85 Innkeeper_results.zip
+https://my.personalgenomes.org/profile/huD09534 fee37be9440b912eb90f5e779f272416+82 Hallet_results.zip
</code></pre>
</notextile>
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list