[ARVADOS] updated: 1.3.0-1088-g5da148c4b
Git user
git at public.curoverse.com
Fri Jun 14 17:36:26 UTC 2019
Summary of changes:
.../crunch2-slurm/install-test.html.textile.liquid | 2 +-
doc/sdk/cli/subcommands.html.textile.liquid | 14 +++++++++++++-
doc/sdk/python/sdk-python.html.textile.liquid | 8 +++++++-
doc/user/topics/keep.html.textile.liquid | 12 ++++++------
.../tutorial-gatk-variantfiltration.html.textile.liquid | 2 +-
doc/user/topics/tutorial-job1.html.textile.liquid | 8 ++++----
doc/user/topics/tutorial-parallel.html.textile.liquid | 2 +-
doc/user/tutorials/tutorial-keep.html.textile.liquid | 16 ++++++++--------
sdk/python/arvados_version.py | 2 +-
9 files changed, 42 insertions(+), 24 deletions(-)
via 5da148c4b88713680d19ef15e238e7b7a932e6f6 (commit)
via c36be066880763f596b1181c7fde1bb13151e126 (commit)
via 5ff4dac7a9ee28eb35a796b80421a6d3d1e5ecd0 (commit)
via b017da127fc3a1747e50f81b3f5d2fea62ba6c94 (commit)
from d33dab9c7baafe56cdcd0e1266f03750531720b4 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 5da148c4b88713680d19ef15e238e7b7a932e6f6
Merge: d33dab9c7 c36be0668
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Fri Jun 14 13:35:51 2019 -0400
Merge branch '15368-python-deps' closes #15368
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
commit c36be066880763f596b1181c7fde1bb13151e126
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Fri Jun 14 13:10:57 2019 -0400
15368: Don't require git to install arvados-python-client
It was catching CalledProcessError but if git is not installed the
error is actually OSError.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/sdk/python/arvados_version.py b/sdk/python/arvados_version.py
index 2e6484cab..692146465 100644
--- a/sdk/python/arvados_version.py
+++ b/sdk/python/arvados_version.py
@@ -34,7 +34,7 @@ def get_version(setup_dir, module):
else:
try:
save_version(setup_dir, module, git_latest_tag() + git_timestamp_tag())
- except subprocess.CalledProcessError:
+ except (subprocess.CalledProcessError, OSError):
pass
return read_version(setup_dir, module)
commit 5ff4dac7a9ee28eb35a796b80421a6d3d1e5ecd0
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Fri Jun 14 13:07:17 2019 -0400
15368: Change doc references from "arv keep put/get" to "arv-put/get"
Since the "arv" frontend just adds overhead and requires installing
both ruby and python SDK instead of just the python sdk.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/doc/install/crunch2-slurm/install-test.html.textile.liquid b/doc/install/crunch2-slurm/install-test.html.textile.liquid
index ca509bb7d..03a5d18b4 100644
--- a/doc/install/crunch2-slurm/install-test.html.textile.liquid
+++ b/doc/install/crunch2-slurm/install-test.html.textile.liquid
@@ -106,7 +106,7 @@ You can use standard Keep tools to view the container's output and logs from the
./crunch-run.txt
./stderr.txt
./stdout.txt
-~$ <span class="userinput">arv keep get <b>a01df2f7e5bc1c2ad59c60a837e90dc6+166</b>/stdout.txt</span>
+~$ <span class="userinput">arv-get <b>a01df2f7e5bc1c2ad59c60a837e90dc6+166</b>/stdout.txt</span>
2016-08-05T13:53:06.201011Z Hello, Crunch!
</code></pre>
</notextile>
diff --git a/doc/sdk/cli/subcommands.html.textile.liquid b/doc/sdk/cli/subcommands.html.textile.liquid
index f9652ef38..b23ec46fa 100644
--- a/doc/sdk/cli/subcommands.html.textile.liquid
+++ b/doc/sdk/cli/subcommands.html.textile.liquid
@@ -140,6 +140,8 @@ arv tag remove --all
h3(#arv-ws). arv ws
+This is a frontend to @arv-ws at .
+
@arv ws@ provides access to the websockets event stream.
<notextile>
@@ -168,7 +170,7 @@ optional arguments:
h3(#arv-keep). arv keep
- at arv keep@ provides access to the Keep storage service.
+ at arv keep@ commands for accessing the Keep storage service.
<notextile>
<pre>
@@ -182,6 +184,8 @@ Available methods: ls, get, put, docker
h3(#arv-keep-ls). arv keep ls
+This is a frontend to @arv-ls at .
+
<notextile>
<pre>
$ <code class="userinput">arv keep ls --help</code>
@@ -203,6 +207,8 @@ optional arguments:
h3(#arv-keep-get). arv keep get
+This is a frontend to @arv-get at .
+
<notextile>
<pre>
$ <code class="userinput">arv keep get --help</code>
@@ -256,6 +262,8 @@ optional arguments:
h3(#arv-keep-put). arv keep put
+This is a frontend to @arv-put at .
+
<notextile>
<pre>
$ <code class="userinput">arv keep put --help</code>
@@ -335,6 +343,8 @@ optional arguments:
h3(#arv-pipeline-run). arv pipeline run
+WARNING: this uses the obsolete "job" API. Don't use this. You should use @arvados-cwl-runner@ instead.
+
@arv pipeline run@ can be used to start a pipeline run from the command line.
The User Guide has a page with a bit more information on "using arv pipeline run":{{site.baseurl}}/user/topics/running-pipeline-command-line.html.
@@ -381,6 +391,8 @@ Options:
h3(#arv-run). arv run
+WARNING: this uses the obsolete "job" API. Don't use this. You should use @arvados-cwl-runner@ instead.
+
The @arv-run@ command creates Arvados pipelines at the command line that fan out to multiple concurrent tasks across Arvados compute nodes.
The User Guide has a page on "using arv-run":{{site.baseurl}}/user/topics/arv-run.html.
diff --git a/doc/user/topics/keep.html.textile.liquid b/doc/user/topics/keep.html.textile.liquid
index dd504398e..68b6a87d0 100644
--- a/doc/user/topics/keep.html.textile.liquid
+++ b/doc/user/topics/keep.html.textile.liquid
@@ -19,20 +19,20 @@ In Keep, information is stored in *data blocks*. Data blocks are normally betwe
In order to reassemble the file, Keep stores a *collection* data block which lists in sequence the data blocks that make up the original file. A collection data block may store the information for multiple files, including a directory structure.
-In this example we will use @c1bad4b39ca5a924e481008009d94e32+210@, which we added to Keep in "how to upload data":{{ site.baseurl }}/user/tutorials/tutorial-keep.html. First let us examine the contents of this collection using @arv keep get@:
+In this example we will use @c1bad4b39ca5a924e481008009d94e32+210@, which we added to Keep in "how to upload data":{{ site.baseurl }}/user/tutorials/tutorial-keep.html. First let us examine the contents of this collection using @arv-get@:
<notextile>
-<pre><code>~$ <span class="userinput">arv keep get c1bad4b39ca5a924e481008009d94e32+210</span>
+<pre><code>~$ <span class="userinput">arv-get c1bad4b39ca5a924e481008009d94e32+210</span>
. 204e43b8a1185621ca55a94839582e6f+67108864 b9677abbac956bd3e86b1deb28dfac03+67108864 fc15aff2a762b13f521baf042140acec+67108864 323d2a3ce20370c4ca1d3462a344f8fd+25885655 0:227212247:var-GS000016015-ASM.tsv.bz2
</code></pre>
</notextile>
-The command @arv keep get@ fetches the contents of the collection @c1bad4b39ca5a924e481008009d94e32+210 at . In this example, this collection includes a single file @var-GS000016015-ASM.tsv.bz2@ which is 227212247 bytes long, and is stored using four sequential data blocks, @204e43b8a1185621ca55a94839582e6f+67108864@, @b9677abbac956bd3e86b1deb28dfac03+67108864@, @fc15aff2a762b13f521baf042140acec+67108864@, and @323d2a3ce20370c4ca1d3462a344f8fd+25885655 at .
+The command @arv-get@ fetches the contents of the collection @c1bad4b39ca5a924e481008009d94e32+210 at . In this example, this collection includes a single file @var-GS000016015-ASM.tsv.bz2@ which is 227212247 bytes long, and is stored using four sequential data blocks, @204e43b8a1185621ca55a94839582e6f+67108864@, @b9677abbac956bd3e86b1deb28dfac03+67108864@, @fc15aff2a762b13f521baf042140acec+67108864@, and @323d2a3ce20370c4ca1d3462a344f8fd+25885655 at .
-Let's use @arv keep get@ to download the first data block:
+Let's use @arv-get@ to download the first data block:
notextile. <pre><code>~$ <span class="userinput">cd /scratch/<b>you</b></span>
-/scratch/<b>you</b>$ <span class="userinput">arv keep get 204e43b8a1185621ca55a94839582e6f+67108864 > block1</span></code></pre>
+/scratch/<b>you</b>$ <span class="userinput">arv-get 204e43b8a1185621ca55a94839582e6f+67108864 > block1</span></code></pre>
{% include 'notebox_begin' %}
@@ -40,7 +40,7 @@ When you run this command, you may get this API warning:
notextile. <pre><code>WARNING:root:API lookup failed for collection 204e43b8a1185621ca55a94839582e6f+67108864 (<class 'apiclient.errors.HttpError'>: <HttpError 404 when requesting https://qr1hi.arvadosapi.com/arvados/v1/collections/204e43b8a1185621ca55a94839582e6f%2B67108864?alt=json returned "Not Found">)</code></pre>
-This happens because @arv keep get@ tries to find a collection with this identifier. When that fails, it emits this warning, then looks for a datablock instead, which succeeds.
+This happens because @arv-get@ tries to find a collection with this identifier. When that fails, it emits this warning, then looks for a datablock instead, which succeeds.
{% include 'notebox_end' %}
diff --git a/doc/user/topics/tutorial-gatk-variantfiltration.html.textile.liquid b/doc/user/topics/tutorial-gatk-variantfiltration.html.textile.liquid
index 752488e15..544ccbd35 100644
--- a/doc/user/topics/tutorial-gatk-variantfiltration.html.textile.liquid
+++ b/doc/user/topics/tutorial-gatk-variantfiltration.html.textile.liquid
@@ -18,7 +18,7 @@ h2. Installing GATK
Download the GATK binary tarball[1] -- e.g., @GenomeAnalysisTK-2.6-4.tar.bz2@ -- and "copy it to your Arvados VM":{{site.baseurl}}/user/tutorials/tutorial-keep.html.
<notextile>
-<pre><code>~$ <span class="userinput">arv keep put GenomeAnalysisTK-2.6-4.tar.bz2</span>
+<pre><code>~$ <span class="userinput">arv-put GenomeAnalysisTK-2.6-4.tar.bz2</span>
c905c8d8443a9c44274d98b7c6cfaa32+94
</code></pre>
</notextile>
diff --git a/doc/user/topics/tutorial-job1.html.textile.liquid b/doc/user/topics/tutorial-job1.html.textile.liquid
index 34c452a90..f7a206010 100644
--- a/doc/user/topics/tutorial-job1.html.textile.liquid
+++ b/doc/user/topics/tutorial-job1.html.textile.liquid
@@ -149,10 +149,10 @@ Now you can list the files in the collection:
</code></pre>
</notextile>
-This collection consists of the @md5sum.txt@ file. Use @arv keep get@ to show the contents of the @md5sum.txt@ file:
+This collection consists of the @md5sum.txt@ file. Use @arv-get@ to show the contents of the @md5sum.txt@ file:
<notextile>
-<pre><code>~$ <span class="userinput">arv keep get dd755dbc8d49a67f4fe7dc843e4f10a6+54/md5sum.txt</span>
+<pre><code>~$ <span class="userinput">arv-get dd755dbc8d49a67f4fe7dc843e4f10a6+54/md5sum.txt</span>
44b8ae3fde7a8a88d2f7ebd237625b4f ./var-GS000016015-ASM.tsv.bz2
</code></pre>
</notextile>
@@ -171,10 +171,10 @@ On the command line, the Keep identifier listed in the @"log"@ field from @arv j
</code></pre>
</notextile>
-The log collection consists of one log file named with the job's UUID. You can access it using @arv keep get@:
+The log collection consists of one log file named with the job's UUID. You can access it using @arv-get@:
<notextile>
-<pre><code>~$ <span class="userinput">arv keep get xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+91/qr1hi-8i9sb-xxxxxxxxxxxxxxx.log.txt</span>
+<pre><code>~$ <span class="userinput">arv-get xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+91/qr1hi-8i9sb-xxxxxxxxxxxxxxx.log.txt</span>
2013-12-16_20:44:35 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 check slurm allocation
2013-12-16_20:44:35 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 node compute13 - 8 slots
2013-12-16_20:44:36 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 start
diff --git a/doc/user/topics/tutorial-parallel.html.textile.liquid b/doc/user/topics/tutorial-parallel.html.textile.liquid
index 4ff402ea7..7d308dcff 100644
--- a/doc/user/topics/tutorial-parallel.html.textile.liquid
+++ b/doc/user/topics/tutorial-parallel.html.textile.liquid
@@ -77,7 +77,7 @@ Because the job ran in concurrent, each instance of concurrent-hash creates a se
<notextile>
<pre><code>~/$USER/crunch_scripts$ <span class="userinput">arv keep ls e2ccd204bca37c77c0ba59fc470cd0f7+162</span>
./md5sum.txt
-~/$USER/crunch_scripts$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt</span>
+~/$USER/crunch_scripts$ <span class="userinput">arv-get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt</span>
0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
504938460ef369cd275e4ef58994cffe bob.txt
8f3b36aff310e06f3c5b9e95678ff77a carol.txt
diff --git a/doc/user/tutorials/tutorial-keep.html.textile.liquid b/doc/user/tutorials/tutorial-keep.html.textile.liquid
index 53cdfe4a0..ec7086db9 100644
--- a/doc/user/tutorials/tutorial-keep.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep.html.textile.liquid
@@ -9,7 +9,7 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Arvados Data collections can be uploaded using either the @arv keep put@ command line tool or using Workbench.
+Arvados Data collections can be uploaded using either the @arv-put@ command line tool or using Workbench.
# "*Upload using command line tool*":#upload-using-command
# "*Upload using Workbench*":#upload-using-workbench
@@ -20,9 +20,9 @@ h2(#upload-using-command). Upload using command line tool
{% include 'tutorial_expectations' %}
-To upload a file to Keep using @arv keep put@:
+To upload a file to Keep using @arv-put@:
<notextile>
-<pre><code>~$ <span class="userinput">arv keep put var-GS000016015-ASM.tsv.bz2</span>
+<pre><code>~$ <span class="userinput">arv-put var-GS000016015-ASM.tsv.bz2</span>
216M / 216M 100.0%
Collection saved as ...
qr1hi-4zz18-xxxxxxxxxxxxxxx
@@ -34,27 +34,27 @@ The output value @qr1hi-4zz18-xxxxxxxxxxxxxxx@ is the uuid of the Arvados collec
Note: The file used in this example is a freely available TSV file containing variant annotations from the "Personal Genome Project (PGP)":http://www.pgp-hms.org participant "hu599905":https://my.pgp-hms.org/profile/hu599905), downloadable "here":https://warehouse.pgp-hms.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2. Alternatively, you can replace @var-GS000016015-ASM.tsv.bz2@ with the name of any file you have locally, or you could get the TSV file by "downloading it from Keep.":{{site.baseurl}}/user/tutorials/tutorial-keep-get.html
-<notextile><a name="dir"></a></notextile>It is also possible to upload an entire directory with @arv keep put@:
+<notextile><a name="dir"></a></notextile>It is also possible to upload an entire directory with @arv-put@:
<notextile>
<pre><code>~$ <span class="userinput">mkdir tmp</span>
~$ <span class="userinput">echo "hello alice" > tmp/alice.txt</span>
~$ <span class="userinput">echo "hello bob" > tmp/bob.txt</span>
~$ <span class="userinput">echo "hello carol" > tmp/carol.txt</span>
-~$ <span class="userinput">arv keep put tmp</span>
+~$ <span class="userinput">arv-put tmp</span>
0M / 0M 100.0%
Collection saved as ...
qr1hi-4zz18-yyyyyyyyyyyyyyy
</code></pre>
</notextile>
-In both examples, the @arv keep put@ command created a collection. The first collection contains the single uploaded file. The second collection contains the entire uploaded directory.
+In both examples, the @arv-put@ command created a collection. The first collection contains the single uploaded file. The second collection contains the entire uploaded directory.
- at arv keep put@ accepts quite a few optional command line arguments, which are described on the "arv subcommands":{{site.baseurl}}/sdk/cli/subcommands.html#arv-keep-put page.
+ at arv-put@ accepts quite a few optional command line arguments, which are described on the "arv subcommands":{{site.baseurl}}/sdk/cli/subcommands.html#arv-keep-put page.
h3. Locate your collection in Workbench
-Visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu, select your *Home* project. Your newly uploaded collection should appear near the top of the *Data collections* tab. The collection name printed by @arv keep put@ will appear under the *name* column.
+Visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu, select your *Home* project. Your newly uploaded collection should appear near the top of the *Data collections* tab. The collection name printed by @arv-put@ will appear under the *name* column.
To move the collection to a different project, check the box at the left of the collection row. Pull down the *Selection...*<span class="caret"></span> menu near the top of the page tab, and select *Move selected...* button. This will open a dialog box where you can select a destination project for the collection. Click a project, then finally the <span class="btn btn-sm btn-primary">Move</span> button.
commit b017da127fc3a1747e50f81b3f5d2fea62ba6c94
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Fri Jun 14 13:06:08 2019 -0400
15368: Add note about debian dependencies for installing from pip
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/doc/sdk/python/sdk-python.html.textile.liquid b/doc/sdk/python/sdk-python.html.textile.liquid
index 9c0ec475b..c8b2b67b1 100644
--- a/doc/sdk/python/sdk-python.html.textile.liquid
+++ b/doc/sdk/python/sdk-python.html.textile.liquid
@@ -24,7 +24,13 @@ h3. Option 1: Install with pip
This installation method is recommended to make the SDK available for use in your own Python programs. It can coexist with the system-wide installation method from a distribution package (option 2, below).
-Run @pip install arvados-python-client@ in an appropriate installation environment, such as a virtualenv.
+Run @pip install arvados-python-client@ in an appropriate installation environment, such as a @virtualenv at .
+
+The SDK uses @pycurl@ which depends on the @libcurl@ C library. To build the module you may have to install additional packages. On Debian 9 this is:
+
+<pre>
+$ apt-get install git build-essential python3-dev libcurl4-openssl-dev libssl1.0-dev
+</pre>
If your version of @pip@ is 1.4 or newer, the @pip install@ command might give an error: "Could not find a version that satisfies the requirement arvados-python-client". If this happens, try @pip install --pre arvados-python-client at .
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list