[ARVADOS] created: ca493dcca8463dc5976b31de0b0dfed3c4d26d9b
git at public.curoverse.com
git at public.curoverse.com
Sat Nov 22 17:05:33 EST 2014
at ca493dcca8463dc5976b31de0b0dfed3c4d26d9b (commit)
commit ca493dcca8463dc5976b31de0b0dfed3c4d26d9b
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Nov 22 17:04:27 2014 -0500
Refactor the 'Uploading data' page.
refs #4336
diff --git a/doc/user/tutorials/tutorial-keep.html.textile.liquid b/doc/user/tutorials/tutorial-keep.html.textile.liquid
index 85d7dcc..ada6d1f 100644
--- a/doc/user/tutorials/tutorial-keep.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep.html.textile.liquid
@@ -4,53 +4,53 @@ navsection: userguide
title: "Uploading data"
...
-This tutorial describes how to to upload new Arvados data collections using the command line tool @arv-put at . This example uses a freely available TSV file containing variant annotations from "Personal Genome Project (PGP)":http://www.pgp-hms.org participant "hu599905.":https://my.pgp-hms.org/profile/hu599905
+This tutorial describes how to to upload new Arvados data collections using the command line tool @arv keep put at .
notextile. <div class="spaced-out">
-# Begin by installing the "Arvados Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html on the system from which you will upload the data (such as your workstation, or a server containing data from your sequencer). This will install the Arvados file upload tool, @arv-put at . Alternately, you can log into an Arvados VM (instructions for "Unix":{{site.baseurl}}/user/getting_started/ssh-access-unix.html#login or "Windows":{{site.baseurl}}/user/getting_started/ssh-access-windows.html#login).
-# On the system from which you will upload data, configure the environment with the Arvados instance host name and authentication token as decribed in "Getting an API token.":{{site.baseurl}}/user/reference/api-tokens.html (If you are logged into an Arvados VM, you can skip this step.)
-# Download the following example file. (If you are uploading your own data, you can skip this step.)
-<notextile>
-<pre><code>~$ <span class="userinput">curl -o var-GS000016015-ASM.tsv.bz2 'https://warehouse.pgp-hms.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2'</span>
- % Total % Received % Xferd Average Speed Time Time Time Current
- Dload Upload Total Spent Left Speed
-100 216M 100 216M 0 0 10.0M 0 0:00:21 0:00:21 --:--:-- 9361k
-</code></pre>
-</notextile>
-# Now upload the file to Keep using @arv-put@:
+{% include 'tutorial_expectations' %}
+
+h3. Upload
+
+To upload a file to Keep using @arv keep put@:
<notextile>
-<pre><code>~$ <span class="userinput">arv-put var-GS000016015-ASM.tsv.bz2</span>
+<pre><code>~$ <span class="userinput">arv keep put var-GS000016015-ASM.tsv.bz2</span>
216M / 216M 100.0%
Collection saved as ...
qr1hi-4zz18-xxxxxxxxxxxxxxx
</code></pre>
</notextile>
-* The output value @qr1hi-4zz18-xxxxxxxxxxxxxxx@ is the uuid of the Arvados collection created.
-
-Now visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu, select your *Home* project. Your newly uploaded collection should appear near the top of the *Data collections* tab. The collection locator printed by @arv-put@ will appear under the *name* column.
-
-To move the collection to a different project, check the box at the left of the collection row. Pull down the *Selection...*<span class="caret"></span> menu near the top of the page tab, and select *Move selected*. This will open a dialog box where you can select a destination project for the collection. Click a project, then finally the <span class="btn btn-sm btn-primary">Move</span> button.
-
-!{{ site.baseurl }}/images/workbench-move-selected.png!
-
-Click on the *<i class="fa fa-fw fa-archive"></i> Show* button next to the collection's listing on a project page to go to the Workbench page for your collection. On this page, you can see the collection's contents, download individual files, and set sharing options.
-
-notextile. </div>
+The output value @qr1hi-4zz18-xxxxxxxxxxxxxxx@ is the uuid of the Arvados collection created.
-h2(#dir). Putting a directory
+The file used in this example is a freely available TSV file containing variant annotations from "Personal Genome Project (PGP)":http://www.pgp-hms.org participant "hu599905.":https://my.pgp-hms.org/profile/hu599905), downloadable "here":https://warehouse.pgp-hms.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2.
-If you give @arv-put@ a directory, it will recursively upload the entire directory:
+<notextile><a name="dir"></a></notextile>It is also possible to upload an entire directory with @arv keep put@:
<notextile>
<pre><code>~$ <span class="userinput">mkdir tmp</span>
~$ <span class="userinput">echo "hello alice" > tmp/alice.txt</span>
~$ <span class="userinput">echo "hello bob" > tmp/bob.txt</span>
~$ <span class="userinput">echo "hello carol" > tmp/carol.txt</span>
-~$ <span class="userinput">arv-put tmp</span>
+~$ <span class="userinput">arv keep put tmp</span>
0M / 0M 100.0%
Collection saved as ...
qr1hi-4zz18-yyyyyyyyyyyyyyy
</code></pre>
</notextile>
+
+In both examples, the @arv keep put@ command created a collection. The first collection contains the single uploaded file. The second collection contains the entire uploaded directory.
+
+ at arv keep put@ accepts quite a few optional command line arguments, which are described "on the arv subcommands":{{site.baseurl}}/sdk/cli/subcommands.html#arv-keep-put page.
+
+h3. Locate your collection in Workbench
+
+Visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu, select your *Home* project. Your newly uploaded collection should appear near the top of the *Data collections* tab. The collection locator printed by @arv keep put@ will appear under the *name* column.
+
+To move the collection to a different project, check the box at the left of the collection row. Pull down the *Selection...*<span class="caret"></span> menu near the top of the page tab, and select *Move selected*. This will open a dialog box where you can select a destination project for the collection. Click a project, then finally the <span class="btn btn-sm btn-primary">Move</span> button.
+
+!{{ site.baseurl }}/images/workbench-move-selected.png!
+
+Click on the *<i class="fa fa-fw fa-archive"></i> Show* button next to the collection's listing on a project page to go to the Workbench page for your collection. On this page, you can see the collection's contents, download individual files, and set sharing options.
+
+notextile. </div>
commit 7a503f8bceb9969d41116ed04f1b9edf53c169c2
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Nov 22 17:03:57 2014 -0500
Add basic documentation for arv keep put and arv keep get.
refs #4336
diff --git a/doc/sdk/cli/subcommands.html.textile.liquid b/doc/sdk/cli/subcommands.html.textile.liquid
index ea74eb7..dddfa30 100644
--- a/doc/sdk/cli/subcommands.html.textile.liquid
+++ b/doc/sdk/cli/subcommands.html.textile.liquid
@@ -161,6 +161,133 @@ Available methods: ls, get, put, less, check, docker
</pre>
</notextile>
+h3(#arv-keep-put). arv keep put
+
+<notextile>
+<pre>
+$ <code class="userinput">arv keep put --help</code>
+usage: arv-put [-h] [--max-manifest-depth N | --normalize]
+ [--as-stream | --stream | --as-manifest | --in-manifest | --manifest | --as-raw | --raw]
+ [--use-filename FILENAME] [--filename FILENAME]
+ [--portable-data-hash] [--project-uuid UUID] [--name NAME]
+ [--progress | --no-progress | --batch-progress]
+ [--resume | --no-resume] [--retries RETRIES]
+ [path [path ...]]
+
+Copy data from the local filesystem to Keep.
+
+positional arguments:
+ path Local file or directory. Default: read from standard
+ input.
+
+optional arguments:
+ -h, --help show this help message and exit
+ --max-manifest-depth N
+ Maximum depth of directory tree to represent in the
+ manifest structure. A directory structure deeper than
+ this will be represented as a single stream in the
+ manifest. If N=0, the manifest will contain a single
+ stream. Default: -1 (unlimited), i.e., exactly one
+ manifest stream per filesystem directory that contains
+ files.
+ --normalize Normalize the manifest by re-ordering files and
+ streams after writing data.
+ --as-stream Synonym for --stream.
+ --stream Store the file content and display the resulting
+ manifest on stdout. Do not write the manifest to Keep
+ or save a Collection object in Arvados.
+ --as-manifest Synonym for --manifest.
+ --in-manifest Synonym for --manifest.
+ --manifest Store the file data and resulting manifest in Keep,
+ save a Collection object in Arvados, and display the
+ manifest locator (Collection uuid) on stdout. This is
+ the default behavior.
+ --as-raw Synonym for --raw.
+ --raw Store the file content and display the data block
+ locators on stdout, separated by commas, with a
+ trailing newline. Do not store a manifest.
+ --use-filename FILENAME
+ Synonym for --filename.
+ --filename FILENAME Use the given filename in the manifest, instead of the
+ name of the local file. This is useful when "-" or
+ "/dev/stdin" is given as an input file. It can be used
+ only if there is exactly one path given and it is not
+ a directory. Implies --manifest.
+ --portable-data-hash Print the portable data hash instead of the Arvados
+ UUID for the collection created by the upload.
+ --project-uuid UUID Store the collection in the specified project, instead
+ of your Home project.
+ --name NAME Save the collection with the specified name.
+ --progress Display human-readable progress on stderr (bytes and,
+ if possible, percentage of total data size). This is
+ the default behavior when stderr is a tty.
+ --no-progress Do not display human-readable progress on stderr, even
+ if stderr is a tty.
+ --batch-progress Display machine-readable progress on stderr (bytes
+ and, if known, total data size).
+ --resume Continue interrupted uploads from cached state
+ (default).
+ --no-resume Do not continue interrupted uploads from cached state.
+ --retries RETRIES Maximum number of times to retry server requests that
+ encounter temporary failures (e.g., server down).
+ Default 3.
+</pre>
+</notextile>
+
+h3(#arv-keep-get). arv keep get
+
+<notextile>
+<pre>
+$ <code class="userinput">arv keep get --help</code>
+usage: arv-get [-h] [--retries RETRIES]
+ [--progress | --no-progress | --batch-progress]
+ [--hash HASH | --md5sum] [-n] [-r] [-f | --skip-existing]
+ locator [destination]
+
+Copy data from Keep to a local file or pipe.
+
+positional arguments:
+ locator Collection locator, optionally with a file path or
+ prefix.
+ destination Local file or directory where the data is to be written.
+ Default: /dev/stdout.
+
+optional arguments:
+ -h, --help show this help message and exit
+ --retries RETRIES Maximum number of times to retry server requests that
+ encounter temporary failures (e.g., server down). Default
+ 3.
+ --progress Display human-readable progress on stderr (bytes and, if
+ possible, percentage of total data size). This is the
+ default behavior when it is not expected to interfere
+ with the output: specifically, stderr is a tty _and_
+ either stdout is not a tty, or output is being written to
+ named files rather than stdout.
+ --no-progress Do not display human-readable progress on stderr.
+ --batch-progress Display machine-readable progress on stderr (bytes and,
+ if known, total data size).
+ --hash HASH Display the hash of each file as it is read from Keep,
+ using the given hash algorithm. Supported algorithms
+ include md5, sha1, sha224, sha256, sha384, and sha512.
+ --md5sum Display the MD5 hash of each file as it is read from
+ Keep.
+ -n Do not write any data -- just read from Keep, and report
+ md5sums if requested.
+ -r Retrieve all files in the specified collection/prefix.
+ This is the default behavior if the "locator" argument
+ ends with a forward slash.
+ -f Overwrite existing files while writing. The default
+ behavior is to refuse to write *anything* if any of the
+ output files already exist. As a special case, -f is not
+ needed to write to /dev/stdout.
+ --skip-existing Skip files that already exist. The default behavior is to
+ refuse to write *anything* if any files exist that would
+ have to be overwritten. This option causes even devices,
+ sockets, and fifos to be skipped.
+</pre>
+</notextile>
+
+
h3(#arv-pipeline-run). arv pipeline run
@arv pipeline run@ can be used to start a pipeline run from the command line.
commit 34609757e88667be13d2e8294a2cfbd4c5cccfac
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Nov 22 17:03:08 2014 -0500
Link directly to the CLI SDK install instructions.
refs #4336
diff --git a/doc/install/install-shell-server.html.textile.liquid b/doc/install/install-shell-server.html.textile.liquid
index 537f1a4..25ddf7b 100644
--- a/doc/install/install-shell-server.html.textile.liquid
+++ b/doc/install/install-shell-server.html.textile.liquid
@@ -14,4 +14,4 @@ Please follow the "API token guide":{{site.baseurl}}/user/reference/api-tokens.h
h2. Install the SDKs
-Install the "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html and the "Command line SDK":{{site.baseurl}}/sdk/cli/index.html
+Install the "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html and the "Command line SDK":{{site.baseurl}}/sdk/cli/install.html
commit efbf7dad04f239fd83ed477a3b9a1cf5b5b979aa
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Nov 22 16:57:43 2014 -0500
Make the 'check-environment' blurb stand out more from the regular text.
refs #4336
diff --git a/doc/_includes/_tutorial_expectations.liquid b/doc/_includes/_tutorial_expectations.liquid
index 6a8b5b5..a371d24 100644
--- a/doc/_includes/_tutorial_expectations.liquid
+++ b/doc/_includes/_tutorial_expectations.liquid
@@ -1 +1,3 @@
-*This tutorial assumes either that you are logged into an Arvados VM instance (instructions for "Unix":{{site.baseurl}}/user/getting_started/ssh-access-unix.html#login or "Windows":{{site.baseurl}}/user/getting_started/ssh-access-windows.html#login) or you have installed the Arvados "Command line SDK":{{site.baseurl}}/sdk/cli/index.html and "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html on your workstation and have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html*
+{% include 'notebox_begin' %}
+This tutorial assumes either that you are logged into an Arvados VM instance (instructions for "Unix":{{site.baseurl}}/user/getting_started/ssh-access-unix.html#login or "Windows":{{site.baseurl}}/user/getting_started/ssh-access-windows.html#login) or you have installed the Arvados "Command line SDK":{{site.baseurl}}/sdk/cli/install.html and "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html on your workstation and have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html
+{% include 'notebox_end' %}
diff --git a/doc/css/bootstrap.css b/doc/css/bootstrap.css
index 14cc1f4..16b635c 100644
--- a/doc/css/bootstrap.css
+++ b/doc/css/bootstrap.css
@@ -4473,7 +4473,7 @@ a.thumbnail.active {
}
.alert-info {
color: #31708f;
- background-color: #d9edf7;
+ background-color: #edf6fa;
border-color: #bce8f1;
}
.alert-info hr {
@@ -4481,6 +4481,10 @@ a.thumbnail.active {
}
.alert-info .alert-link {
color: #245269;
+ font-weight: bold;
+}
+.alert-info a {
+ font-weight: bold;
}
.alert-warning {
color: #8a6d3b;
diff --git a/doc/user/getting_started/check-environment.html.textile.liquid b/doc/user/getting_started/check-environment.html.textile.liquid
index 82b3fa2..46156b7 100644
--- a/doc/user/getting_started/check-environment.html.textile.liquid
+++ b/doc/user/getting_started/check-environment.html.textile.liquid
@@ -4,7 +4,7 @@ navsection: userguide
title: "Checking your environment"
...
-First, log into an Arvados VM instance (instructions for "Unix":{{site.baseurl}}/user/getting_started/ssh-access-unix.html#login or "Windows":{{site.baseurl}}/user/getting_started/ssh-access-windows.html#login) or install the Arvados "Command line SDK":{{site.baseurl}}/sdk/cli/index.html and "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html on your workstation.
+First, log into an Arvados VM instance (instructions for "Unix":{{site.baseurl}}/user/getting_started/ssh-access-unix.html#login or "Windows":{{site.baseurl}}/user/getting_started/ssh-access-windows.html#login) or install the Arvados "Command line SDK":{{site.baseurl}}/sdk/cli/install.html and "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html on your workstation.
Check that you are able to access the Arvados API server using @arv user current at . If it is able to access the API server, it will print out information about your account:
commit a34aaf6d03786c865d327cf16537ddf3ff221b3a
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Nov 22 15:17:57 2014 -0500
Wording improvements on the sdk cli installation page.
refs #4336
diff --git a/doc/sdk/cli/install.html.textile.liquid b/doc/sdk/cli/install.html.textile.liquid
index 18e1f70..df55077 100644
--- a/doc/sdk/cli/install.html.textile.liquid
+++ b/doc/sdk/cli/install.html.textile.liquid
@@ -6,9 +6,7 @@ title: "Installation"
...
-If you are logged in to an Arvados VM, the @arv@ tool should be installed.
-
-To use @arv@ elsewhere, you can either install the @arvados-cli@ gem via RubyGems or build and install the package from source.
+To use the @arv@ command, you can either install the @arvados-cli@ gem via RubyGems or build and install the package from source.
h4. Prerequisites: Ruby >= 2.1.0 and curl libraries
commit b9d52229719c49e13aa8dfaae69da88a23fbfe44
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Nov 22 15:12:43 2014 -0500
Fix typo.
refs #4336
diff --git a/doc/user/tutorials/tutorial-keep.html.textile.liquid b/doc/user/tutorials/tutorial-keep.html.textile.liquid
index dc67339..85d7dcc 100644
--- a/doc/user/tutorials/tutorial-keep.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep.html.textile.liquid
@@ -9,7 +9,7 @@ This tutorial describes how to to upload new Arvados data collections using the
notextile. <div class="spaced-out">
# Begin by installing the "Arvados Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html on the system from which you will upload the data (such as your workstation, or a server containing data from your sequencer). This will install the Arvados file upload tool, @arv-put at . Alternately, you can log into an Arvados VM (instructions for "Unix":{{site.baseurl}}/user/getting_started/ssh-access-unix.html#login or "Windows":{{site.baseurl}}/user/getting_started/ssh-access-windows.html#login).
-# On system from which you will upload data, configure the environment with the Arvados instance host name and authentication token as decribed in "Getting an API token.":{{site.baseurl}}/user/reference/api-tokens.html (If you are logged into an Arvados VM, you can skip this step.)
+# On the system from which you will upload data, configure the environment with the Arvados instance host name and authentication token as decribed in "Getting an API token.":{{site.baseurl}}/user/reference/api-tokens.html (If you are logged into an Arvados VM, you can skip this step.)
# Download the following example file. (If you are uploading your own data, you can skip this step.)
<notextile>
<pre><code>~$ <span class="userinput">curl -o var-GS000016015-ASM.tsv.bz2 'https://warehouse.pgp-hms.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2'</span>
commit 4c7615263f17507405895ceb906f7ecae0658d1d
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Nov 22 15:08:22 2014 -0500
Add third installation option for the python SDK: our debian/ubuntu packages.
refs #4336
diff --git a/doc/sdk/python/sdk-python.html.textile.liquid b/doc/sdk/python/sdk-python.html.textile.liquid
index 89b77c9..ead804e 100644
--- a/doc/sdk/python/sdk-python.html.textile.liquid
+++ b/doc/sdk/python/sdk-python.html.textile.liquid
@@ -37,7 +37,26 @@ $ <code class="userinput">sudo pip install --pre arvados-python-client</code>
</pre>
</notextile>
-h4. Option 2: build and install from source
+h4. Option 2: install from distribution packages (Debian/Ubuntu only)
+
+First add @http://apt.arvados.org@ to your list of apt repositories:
+
+<notextile>
+<pre>
+$ <code class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</code>
+</pre>
+</notextile>
+
+Then install the package:
+
+<notextile>
+<pre>
+$ <code class="userinput">sudo apt-get update</code>
+$ <code class="userinput">sudo apt-get install python-arvados-python-client</code>
+</pre>
+</notextile>
+
+h4. Option 3: build and install from source
<notextile>
<pre>
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list