[ARVADOS] updated: 1.1.4-164-gda87145
Git user
git at public.curoverse.com
Thu Apr 26 14:14:15 EDT 2018
Summary of changes:
doc/Rakefile | 2 +-
doc/_config.yml | 26 +-
.../{Arvados arch.odg => Arvados_arch.odg} | Bin
.../crunch-utility-libraries.html.textile.liquid | 4 +-
.../tutorial-trait-search.html.textile.liquid | 12 +-
sdk/R/README.Rmd | 342 +++++++++++----------
6 files changed, 198 insertions(+), 188 deletions(-)
rename doc/architecture/{Arvados arch.odg => Arvados_arch.odg} (100%)
via da8714584a1649fb128e0f13718c2507c518a210 (commit)
from 400828b30aa206b172e0d0f7d42f965423829202 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit da8714584a1649fb128e0f13718c2507c518a210
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Thu Apr 26 14:13:27 2018 -0400
13282: doc updates
* Improve presentation of R SDK
* Move obsolete crunch1 documentation to "Obsolete" section.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/doc/Rakefile b/doc/Rakefile
index 855b829..079f7da 100644
--- a/doc/Rakefile
+++ b/doc/Rakefile
@@ -54,7 +54,7 @@ navmenu: R
title: "R SDK Overview"
...
-#{rd.read.gsub("```", "\n~~~\n").gsub(/^[ ]+/, "")}
+#{rd.read.gsub(/^```$/, "~~~").gsub(/^```(\w)$/, "~~~\\1")}
EOF
)
end
diff --git a/doc/_config.yml b/doc/_config.yml
index 50a9f7a..4766e3f 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -40,8 +40,6 @@ navbar:
- Running workflows at the command line:
- user/cwl/cwl-runner.html.textile.liquid
- user/cwl/cwl-run-options.html.textile.liquid
- # - user/topics/running-pipeline-command-line.html.textile.liquid
- # - user/topics/arv-run.html.textile.liquid
- Working with git repositories:
- user/tutorials/add-new-repository.html.textile.liquid
- user/tutorials/git-arvados-guide.html.textile.liquid
@@ -51,23 +49,25 @@ navbar:
- user/cwl/cwl-style.html.textile.liquid
- user/cwl/cwl-extensions.html.textile.liquid
- user/topics/arv-docker.html.textile.liquid
- # - user/tutorials/running-external-program.html.textile.liquid
- # - user/topics/crunch-tools-overview.html.textile.liquid
- # - user/tutorials/tutorial-firstscript.html.textile.liquid
- # - user/tutorials/tutorial-submit-job.html.textile.liquid
- # - user/topics/tutorial-parallel.html.textile.liquid
- Reference:
- user/reference/cookbook.html.textile.liquid
- # - user/topics/run-command.html.textile.liquid
- # - user/reference/job-pipeline-ref.html.textile.liquid
- # - user/examples/crunch-examples.html.textile.liquid
- #- Query the metadata database:
- # - user/topics/tutorial-trait-search.html.textile.liquid
- Arvados License:
- user/copying/copying.html.textile.liquid
- user/copying/agpl-3.0.html
- user/copying/LICENSE-2.0.html
- user/copying/by-sa-3.0.html
+ - Obsolete documentation:
+ - user/topics/running-pipeline-command-line.html.textile.liquid
+ - user/topics/arv-run.html.textile.liquid
+ - user/tutorials/running-external-program.html.textile.liquid
+ - user/topics/crunch-tools-overview.html.textile.liquid
+ - user/tutorials/tutorial-firstscript.html.textile.liquid
+ - user/tutorials/tutorial-submit-job.html.textile.liquid
+ - user/topics/tutorial-parallel.html.textile.liquid
+ - user/topics/run-command.html.textile.liquid
+ - user/reference/job-pipeline-ref.html.textile.liquid
+ - user/examples/crunch-examples.html.textile.liquid
+ - user/topics/tutorial-trait-search.html.textile.liquid
sdk:
- Overview:
- sdk/index.html.textile.liquid
@@ -75,10 +75,10 @@ navbar:
- sdk/python/sdk-python.html.textile.liquid
- sdk/python/example.html.textile.liquid
- sdk/python/python.html.textile.liquid
- # - sdk/python/crunch-utility-libraries.html.textile.liquid
- sdk/python/arvados-fuse.html.textile.liquid
- sdk/python/events.html.textile.liquid
- sdk/python/cookbook.html.textile.liquid
+ - sdk/python/crunch-utility-libraries.html.textile.liquid
- CLI:
- sdk/cli/install.html.textile.liquid
- sdk/cli/index.html.textile.liquid
diff --git a/doc/architecture/Arvados arch.odg b/doc/architecture/Arvados_arch.odg
similarity index 100%
rename from doc/architecture/Arvados arch.odg
rename to doc/architecture/Arvados_arch.odg
diff --git a/doc/sdk/python/crunch-utility-libraries.html.textile.liquid b/doc/sdk/python/crunch-utility-libraries.html.textile.liquid
index 64019bb..3e26315 100644
--- a/doc/sdk/python/crunch-utility-libraries.html.textile.liquid
+++ b/doc/sdk/python/crunch-utility-libraries.html.textile.liquid
@@ -11,6 +11,8 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+{% include 'pipeline_deprecation_notice' %}
+
Several utility libraries are included with Arvados. They are intended to make it quicker and easier to write your own crunch scripts.
* "Python SDK extras":#pythonsdk
@@ -224,5 +226,3 @@ On qr1hi.arvadosapi.com, the binary distribution @picard-tools-1.82.zip@ is avai
...
}
</pre>
-
-
diff --git a/doc/user/topics/tutorial-trait-search.html.textile.liquid b/doc/user/topics/tutorial-trait-search.html.textile.liquid
index 7015d41..d396802 100644
--- a/doc/user/topics/tutorial-trait-search.html.textile.liquid
+++ b/doc/user/topics/tutorial-trait-search.html.textile.liquid
@@ -9,23 +9,27 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+{% include 'notebox_begin_warning' %}
+The humans, specimens and traits tables are deprecated and will be removed in a future release. The recommended way to store and search on user-defined metadata is using the "properties" field of Arvados resources.
+{% include 'notebox_end' %}
+
This tutorial introduces the Arvados Metadata Database. The Metadata Database stores information about files in Keep. This example will use the Python SDK to find public WGS (Whole Genome Sequencing) data for people who have reported a certain medical condition.
{% include 'tutorial_expectations' %}
In the tutorial examples, three angle brackets (>>>) will be used to denote code to enter at the interactive Python prompt.
-Start by running Python.
+Start by running Python.
<notextile>
<pre><code>~$ <span class="userinput">python</span>
-Python 2.7.3 (default, Jan 2 2013, 13:56:14)
+Python 2.7.3 (default, Jan 2 2013, 13:56:14)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
</code></pre>
</notextile>
-
+
If everything is set up correctly, you will be able to import the arvados SDK.
notextile. <pre><code>>>> <span class="userinput">import arvados</span></pre></code>
@@ -248,7 +252,7 @@ After the jobs have completed, check output file sizes.
job_uuid = job[collection_uuid]['uuid']
job_output = arvados.api('v1').jobs().get(uuid=job_uuid).execute()['output']
output_files = arvados.api('v1').collections().get(uuid=job_output).execute()['files']
- # Test the output size. If greater than zero, that means 'grep' found the variant
+ # Test the output size. If greater than zero, that means 'grep' found the variant
if output_files[0][2] > 0:
print("%s has variant rs1126809" % (pgpid[collection_uuid]))
else:
diff --git a/sdk/R/README.Rmd b/sdk/R/README.Rmd
index 8a64e93..cfa0ce5 100644
--- a/sdk/R/README.Rmd
+++ b/sdk/R/README.Rmd
@@ -5,312 +5,318 @@ The API is not final and feedback is solicited from users on ways in which it co
### Installation
-```install.packages("ArvadosR", repos=c("http://r.arvados.org", getOption("repos")["CRAN"]), dependencies=TRUE)```
+```{r include=FALSE}
+knitr::opts_chunk$set(eval=FALSE)
+```
+
+```{r}
+install.packages("ArvadosR", repos=c("http://r.arvados.org", getOption("repos")["CRAN"]), dependencies=TRUE)
+```
Note: on Linux, you may have to install supporting packages.
On Centos 7, this is:
-```yum install libxml2-devel openssl-devel curl-devel```
+```{bash}
+yum install libxml2-devel openssl-devel curl-devel
+```
On Debian, this is:
-```apt-get install build-essential libxml2-dev libssl-dev libcurl4-gnutls-dev```
+```{bash}
+apt-get install build-essential libxml2-dev libssl-dev libcurl4-gnutls-dev
+```
### Usage
#### Initializing API
-```{r include=FALSE}
-knitr::opts_chunk$set(eval = FALSE)
-```
-
* Load Library and Initialize API:
- ```{r}
- library('ArvadosR')
- # use environment variables ARVADOS_API_TOKEN and ARVADOS_API_HOST
- arv <- Arvados$new()
+```{r}
+library('ArvadosR')
+# use environment variables ARVADOS_API_TOKEN and ARVADOS_API_HOST
+arv <- Arvados$new()
- # provide them explicitly
- arv <- Arvados$new("your Arvados token", "example.arvadosapi.com")
- ```
+# provide them explicitly
+arv <- Arvados$new("your Arvados token", "example.arvadosapi.com")
+```
- Optionally, add numRetries parameter to specify number of times to retry failed service requests.
- Default is 0.
+Optionally, add numRetries parameter to specify number of times to retry failed service requests.
+Default is 0.
- ```{r}
- arv <- Arvados$new("your Arvados token", "example.arvadosapi.com", numRetries = 3)
- ```
+```{r}
+arv <- Arvados$new("your Arvados token", "example.arvadosapi.com", numRetries = 3)
+```
- This parameter can be set at any time using setNumRetries
+This parameter can be set at any time using setNumRetries
- ```{r}
- arv$setNumRetries(5)
- ```
+```{r}
+arv$setNumRetries(5)
+```
#### Working with collections
* Get a collection:
- ```{r}
- collection <- arv$collections.get("uuid")
- ```
+```{r}
+collection <- arv$collections.get("uuid")
+```
* List collections:
- ```{r}
- # offset of 0 and default limit of 100
- collectionList <- arv$collections.list(list(list("name", "like", "Test%")))
+```{r}
+# offset of 0 and default limit of 100
+collectionList <- arv$collections.list(list(list("name", "like", "Test%")))
- collectionList <- arv$collections.list(list(list("name", "like", "Test%")), limit = 10, offset = 2)
- ```
+collectionList <- arv$collections.list(list(list("name", "like", "Test%")), limit = 10, offset = 2)
+```
- ```{r}
- # count of total number of items (may be more than returned due to paging)
- collectionList$items_available
+```{r}
+# count of total number of items (may be more than returned due to paging)
+collectionList$items_available
- # items which match the filter criteria
- collectionList$items
- ```
+# items which match the filter criteria
+collectionList$items
+```
* List all collections even if the number of items is greater than maximum API limit:
- ```{r}
- collectionList <- listAll(arv$collections.list, list(list("name", "like", "Test%")))
- ```
+```{r}
+collectionList <- listAll(arv$collections.list, list(list("name", "like", "Test%")))
+```
* Delete a collection:
- ```{r}
- deletedCollection <- arv$collections.delete("uuid")
- ```
+```{r}
+deletedCollection <- arv$collections.delete("uuid")
+```
* Update a collection's metadata:
- ```{r}
- updatedCollection <- arv$collections.update(list(name = "New name", description = "New description"), "uuid")
- ```
+```{r}
+updatedCollection <- arv$collections.update(list(name = "New name", description = "New description"), "uuid")
+```
* Create collection:
- ```{r}
- newCollection <- arv$collections.create(list(name = "Example", description = "This is a test collection"))
- ```
+```{r}
+newCollection <- arv$collections.create(list(name = "Example", description = "This is a test collection"))
+```
#### Manipulating collection content
* Create collection object:
- ```{r}
- collection <- Collection$new(arv, "uuid")
- ```
+```{r}
+collection <- Collection$new(arv, "uuid")
+```
* Get list of files:
- ```{r}
- files <- collection$getFileListing()
- ```
+```{r}
+files <- collection$getFileListing()
+```
* Get ArvadosFile or Subcollection from internal tree-like structure:
- ```{r}
- arvadosFile <- collection$get("location/to/my/file.cpp")
- ```
+```{r}
+arvadosFile <- collection$get("location/to/my/file.cpp")
+```
or
- ```{r}
- arvadosSubcollection <- collection$get("location/to/my/directory/")
- ```
+```{r}
+arvadosSubcollection <- collection$get("location/to/my/directory/")
+```
* Read a table:
- ```{r}
- arvadosFile <- collection$get("myinput.txt")
- arvConnection <- arvadosFile$connection("r")
- mytable <- read.table(arvConnection)
- ```
+```{r}
+arvadosFile <- collection$get("myinput.txt")
+arvConnection <- arvadosFile$connection("r")
+mytable <- read.table(arvConnection)
+```
* Write a table:
- ```{r}
- arvadosFile <- collection$create("myoutput.txt")
- arvConnection <- arvadosFile$connection("w")
- write.table(mytable, arvConnection)
- arvadosFile$flush()
- ```
+```{r}
+arvadosFile <- collection$create("myoutput.txt")
+arvConnection <- arvadosFile$connection("w")
+write.table(mytable, arvConnection)
+arvadosFile$flush()
+```
* Write to existing file (override current content of the file):
- ```{r}
- arvadosFile <- collection$get("location/to/my/file.cpp")
- arvadosFile$write("This is new file content")
- ```
+```{r}
+arvadosFile <- collection$get("location/to/my/file.cpp")
+arvadosFile$write("This is new file content")
+```
* Read whole file or just a portion of it:
- ```{r}
- fileContent <- arvadosFile$read()
- fileContent <- arvadosFile$read("text")
- fileContent <- arvadosFile$read("raw", offset = 1024, length = 512)
- ```
+```{r}
+fileContent <- arvadosFile$read()
+fileContent <- arvadosFile$read("text")
+fileContent <- arvadosFile$read("raw", offset = 1024, length = 512)
+```
* Get ArvadosFile or Subcollection size:
- ```{r}
- size <- arvadosFile$getSizeInBytes()
- ```
+```{r}
+size <- arvadosFile$getSizeInBytes()
+```
or
- ```{r}
- size <- arvadosSubcollection$getSizeInBytes()
- ```
+```{r}
+size <- arvadosSubcollection$getSizeInBytes()
+```
* Create new file in a collection:
- ```{r}
- collection$create(fileNames, optionalRelativePath)
- ```
+```{r}
+collection$create(fileNames, optionalRelativePath)
+```
Example:
- ```{r}
- mainFile <- collection$create("main.cpp", "cpp/src/")
- fileList <- collection$create(c("main.cpp", lib.dll), "cpp/src/")
- ```
+```{r}
+mainFile <- collection$create("main.cpp", "cpp/src/")
+fileList <- collection$create(c("main.cpp", lib.dll), "cpp/src/")
+```
* Add existing ArvadosFile or Subcollection to a collection:
- ```{r}
- folder <- Subcollection$new("src")
- file <- ArvadosFile$new("main.cpp")
- folder$add(file)
- ```
+```{r}
+folder <- Subcollection$new("src")
+file <- ArvadosFile$new("main.cpp")
+folder$add(file)
+```
- ```{r}
- collection$add(folder, "cpp")
- ```
+```{r}
+collection$add(folder, "cpp")
+```
- This examples will add file "main.cpp" in "./cpp/src/" folder if folder exists.
- If subcollection contains more files or folders they will be added recursively.
+This examples will add file "main.cpp" in "./cpp/src/" folder if folder exists.
+If subcollection contains more files or folders they will be added recursively.
* Delete file from a collection:
- ```{r}
- collection$remove("location/to/my/file.cpp")
- ```
+```{r}
+collection$remove("location/to/my/file.cpp")
+```
- You can remove both Subcollection and ArvadosFile.
- If subcollection contains more files or folders they will be removed recursively.
+You can remove both Subcollection and ArvadosFile.
+If subcollection contains more files or folders they will be removed recursively.
- You can also remove multiple files at once:
+You can also remove multiple files at once:
- ```{r}
- collection$remove(c("path/to/my/file.cpp", "path/to/other/file.cpp"))
- ```
+```{r}
+collection$remove(c("path/to/my/file.cpp", "path/to/other/file.cpp"))
+```
* Delete file or folder from a Subcollection:
- ```{r}
- subcollection <- collection$get("mySubcollection/")
- subcollection$remove("fileInsideSubcollection.exe")
- subcollection$remove("folderInsideSubcollection/")
- ```
+```{r}
+subcollection <- collection$get("mySubcollection/")
+subcollection$remove("fileInsideSubcollection.exe")
+subcollection$remove("folderInsideSubcollection/")
+```
* Move file or folder inside collection:
- Directley from collection
+Directley from collection
- ```{r}
- collection$move("folder/file.cpp", "file.cpp")
- ```
+```{r}
+collection$move("folder/file.cpp", "file.cpp")
+```
- Or from file
+Or from file
- ```{r}
- file <- collection$get("location/to/my/file.cpp")
- file$move("newDestination/file.cpp")
- ```
+```{r}
+file <- collection$get("location/to/my/file.cpp")
+file$move("newDestination/file.cpp")
+```
- Or from subcollection
+Or from subcollection
- ```{r}
- subcollection <- collection$get("location/to/folder")
- subcollection$move("newDestination/folder")
- ```
+```{r}
+subcollection <- collection$get("location/to/folder")
+subcollection$move("newDestination/folder")
+```
- Make sure to include new file name in destination.
- In second example file$move("newDestination/") will not work.
+Make sure to include new file name in destination.
+In second example file$move("newDestination/") will not work.
#### Working with Aravdos projects
* Get a project:
- ```{r}
- project <- arv$projects.get("uuid")
- ```
+```{r}
+project <- arv$projects.get("uuid")
+```
* List projects:
- ```{r}
- # list subprojects of a project
- projects <- arv$projects.list(list(list("owner_uuid", "=", "aaaaa-j7d0g-ccccccccccccccc")))
+```{r}
+list subprojects of a project
+projects <- arv$projects.list(list(list("owner_uuid", "=", "aaaaa-j7d0g-ccccccccccccccc")))
- # list projects which have names beginning with Example
- examples <- arv$projects.list(list(list("name","like","Example%")))
- ```
+list projects which have names beginning with Example
+examples <- arv$projects.list(list(list("name","like","Example%")))
+```
* List all projects even if the number of items is greater than maximum API limit:
- ```{r}
- projects <- listAll(arv$projects.list, list(list("name","like","Example%")))
- ```
+```{r}
+projects <- listAll(arv$projects.list, list(list("name","like","Example%")))
+```
* Delete a project:
- ```{r}
- deletedProject <- arv$projects.delete("uuid")
- ```
+```{r}
+deletedProject <- arv$projects.delete("uuid")
+```
* Update project:
- ```{r}
- updatedProject <- arv$projects.update(list(name = "new_name", description = "new description"), "uuid")
- ```
+```{r}
+updatedProject <- arv$projects.update(list(name = "new_name", description = "new description"), "uuid")
+```
* Create project:
- ```{r}
- newProject <- arv$projects.update(list(name = "project_name", description = "project description"))
- ```
+```{r}
+newProject <- arv$projects.update(list(name = "project_name", description = "project description"))
+```
#### Help
* View help page of Arvados classes by puting ? before class name:
- ```{r}
- ?Arvados
- ?Collection
- ?Subcollection
- ?ArvadosFile
- ```
+```{r}
+?Arvados
+?Collection
+?Subcollection
+?ArvadosFile
+```
* View help page of any method defined in Arvados class by puting ? before method name:
- ```{r}
- ?collections.update
- ?jobs.get
- ```
+```{r}
+?collections.update
+?jobs.get
+```
### Building the ArvadosR package
- ```
- cd arvados/sdk && R CMD build R
- ```
+```{bash}
+cd arvados/sdk && R CMD build R
+```
This will create a tarball of the ArvadosR package in the current directory.
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list