[ARVADOS] updated: e59d21d3f47cbee83a6dc389584bd7b17bec270c
Git user
git at public.curoverse.com
Tue Nov 1 16:04:26 EDT 2016
Summary of changes:
doc/api/execution.html.textile.liquid | 2 +-
doc/api/index.html.textile.liquid | 2 ++
doc/api/methods/collections.html.textile.liquid | 28 +++++++---------
doc/api/storage.html.textile.liquid | 44 +++++++++++++++++++++++++
doc/sdk/go/example.html.textile.liquid | 2 ++
doc/sdk/go/index.html.textile.liquid | 2 ++
doc/sdk/index.html.textile.liquid | 2 ++
doc/user/cwl/cwl-runner.html.textile.liquid | 1 +
8 files changed, 65 insertions(+), 18 deletions(-)
create mode 100644 doc/api/storage.html.textile.liquid
via e59d21d3f47cbee83a6dc389584bd7b17bec270c (commit)
from 051efbd3d843baa690b334e57fd09fad6a908bb9 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit e59d21d3f47cbee83a6dc389584bd7b17bec270c
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Tue Nov 1 16:04:23 2016 -0400
10346: Start documenting how to access Keep.
diff --git a/doc/api/execution.html.textile.liquid b/doc/api/execution.html.textile.liquid
index ade3f5c..f057842 100644
--- a/doc/api/execution.html.textile.liquid
+++ b/doc/api/execution.html.textile.liquid
@@ -4,7 +4,7 @@ navsection: api
title: Computing with Crunch
...
-Crunch is the name for the Arvados system for managing computation. It provides an abstract API to various clouds and HPC resource allocation and scheduling systems.
+Crunch is the name for the Arvados system for managing computation. It provides an abstract API to various clouds and HPC resource allocation and scheduling systems, and integrates closely with Keep storage and the Arvados permission system.
h2. Container API
diff --git a/doc/api/index.html.textile.liquid b/doc/api/index.html.textile.liquid
index 726255b..9eff786 100644
--- a/doc/api/index.html.textile.liquid
+++ b/doc/api/index.html.textile.liquid
@@ -8,3 +8,5 @@ title: API Reference
This reference describes the semantics of Arvados resources and how to programatically access Arvados via its REST API. Each resource listed in this section is exposed on the Arvados API server under the @/arvados/v1/@ path prefix, for example, @https://{{ site.arvados_api_host }}/arvados/v1/collections at .
The API server publishes a machine-readable description of its endpoints and some additional site configuration values via a JSON-formatted discovery document. This is available at @/discovery/v1/apis/arvados/v1/rest@, for example @https://{{ site.arvados_api_host }}/discovery/v1/apis/arvados/v1/rest at . Some Arvados SDKs use the discovery document to generate language bindings.
+
+Many Arvados Workbench pages, under the the *Advanced* tab, provide examples of API and SDK use for accessing the current resource .
diff --git a/doc/api/methods/collections.html.textile.liquid b/doc/api/methods/collections.html.textile.liquid
index 563e20e..4379f72 100644
--- a/doc/api/methods/collections.html.textile.liquid
+++ b/doc/api/methods/collections.html.textile.liquid
@@ -14,22 +14,6 @@ UUID type: @4zz18@
Required arguments are displayed in %{background:#ccffcc}green%.
-h2. Methods
-
-See "collections":{{site.baseurl}}/api/methods/collections.html
-
-h3. Conditions of creating a Collection
-
-The @uuid@ and @manifest_text@ attributes must be provided when creating a Collection. The cryptographic digest of the supplied @manifest_text@ must match the supplied @uuid at .
-
-h3. Side effects of creating a Collection
-
-Referenced data can be protected from garbage collection. See the section about "resources" links on the "Links":Link.html page.
-
-Data can be shared with other users via the Arvados permission model.
-
-Clients can request checks of data integrity and storage redundancy.
-
h2. Resource
Each collection has, in addition to the usual "attributes of Arvados resources":{{site.baseurl}}/api/resources.html:
@@ -38,12 +22,22 @@ table(table table-bordered table-condensed).
|_. Attribute|_. Type|_. Description|_. Example|
|name|string|||
|description|text|||
-|portable_data_hash|string|||
+|portable_data_hash|string|The MD5 sum of the stripped manifest text.||
|manifest_text|text|||
|replication_desired|number|Minimum storage replication level desired for each data block referenced by this collection. A value of @null@ signifies that the site default replication level (typically 2) is desired.|@2@|
|replication_confirmed|number|Replication level most recently confirmed by the storage system. This field is null when a collection is first created, and is reset to null when the manifest_text changes in a way that introduces a new data block. An integer value indicates the replication level of the _least replicated_ data block in the collection.|@2@, null|
|replication_confirmed_at|datetime|When replication_confirmed was confirmed. If replication_confirmed is null, this field is also null.||
+h3. Conditions of creating a Collection
+
+The @portable_data_hash@ and @manifest_text@ attributes must be provided when creating a Collection. The cryptographic digest of the supplied @manifest_text@ must match the supplied @portable_data_hash at .
+
+h3. Side effects of creating a Collection
+
+Referenced blocks are protected from garbage collection in Keep.
+
+Data can be shared with other users via the Arvados permission model.
+
h2. Methods
h3. create
diff --git a/doc/api/storage.html.textile.liquid b/doc/api/storage.html.textile.liquid
new file mode 100644
index 0000000..90f9f5e
--- /dev/null
+++ b/doc/api/storage.html.textile.liquid
@@ -0,0 +1,44 @@
+---
+layout: default
+navsection: api
+title: Storage in Keep
+...
+
+h2. Storing data
+
+Storing data in Keep follows this process:
+
+# The client fetches a list of keep servers (or proxies) using the @accessible@ method on "keep_services":{{site.baseurl}}/api/methods/keep_services.html
+# Data is split into 64 MiB blocks and the MD5 hash is computed for each block.
+# The client uploads each block to one or more Keep servers, based on the number of desired replicas. The priority order is determined using rendezvous hashing, described below.
+# The Keep server returns a block locator (the MD5 sum of the block) and a "signed token" which the client can use as proof of knowledge for the block.
+# The client constructs a @manifest@ which lists the blocks by MD5 hash and how to reassemble them into the original files.
+# The client creates a "collection":{{site.baseurl}}/api/methods/collections.html and provides the @manifest_text@
+# The API server accepts the collection after validating the signed tokens (proof of knowledge) for each block.
+
+h2. Fetching data
+
+# The client requests a @collection@ object including @manifest_text@
+# The server adds "token signatures" which provide proof of access for each block
+# The client fetches a list of keep servers (or proxies) using the @accessible@ method on "keep_services":{{site.baseurl}}/api/methods/keep_services.html
+# For each data block, the client chooses the highest priority server using rendezvous hashing, described below.
+# The client sends the data block request to the keep server, including the token signature (proof of access).
+# The server provides the block data after validating the token signature for the block.
+
+h2. Keep server API
+
+The Keep server is accessed via a simple HTTP REST API.
+
+*GET /blockidentifier+size+A at token*
+
+Fetch the data block, if the token is valid.
+
+*PUT /blockidentifier*
+
+Returns the md5 sum of the data along with the signed token.
+
+h2. Rendezvous hashing
+
+
+
+h2. Manifest format
diff --git a/doc/sdk/go/example.html.textile.liquid b/doc/sdk/go/example.html.textile.liquid
index d8fccae..02c238a 100644
--- a/doc/sdk/go/example.html.textile.liquid
+++ b/doc/sdk/go/example.html.textile.liquid
@@ -5,6 +5,8 @@ navmenu: Python
title: Examples
...
+See "Arvados GoDoc":https://godoc.org/git.curoverse.com/arvados.git/sdk/go for detailed documentation.
+
h2. Initialize SDK
<pre>
diff --git a/doc/sdk/go/index.html.textile.liquid b/doc/sdk/go/index.html.textile.liquid
index f18320a..81f4f99 100644
--- a/doc/sdk/go/index.html.textile.liquid
+++ b/doc/sdk/go/index.html.textile.liquid
@@ -7,6 +7,8 @@ title: "Installation"
The Go ("Golang":http://golang.org) SDK provides a generic set of wrappers so you can make API calls easily.
+See "Arvados GoDoc":https://godoc.org/git.curoverse.com/arvados.git/sdk/go for detailed documentation.
+
h3. Installation
Use @go get git.curoverse.com/arvados.git/sdk/go/arvadosclient at . The go tools will fetch the relevant code and dependencies for you.
diff --git a/doc/sdk/index.html.textile.liquid b/doc/sdk/index.html.textile.liquid
index 990f09c..7633228 100644
--- a/doc/sdk/index.html.textile.liquid
+++ b/doc/sdk/index.html.textile.liquid
@@ -12,3 +12,5 @@ This section documents language bindings for the "Arvados API":{{site.baseurl}}/
* "Perl SDK":{{site.baseurl}}/sdk/perl/index.html
* "Ruby SDK":{{site.baseurl}}/sdk/ruby/index.html
* "Java SDK":{{site.baseurl}}/sdk/java/index.html
+
+Many Arvados Workbench pages, under the the *Advanced* tab, provide examples of API and SDK use for accessing the current resource .
diff --git a/doc/user/cwl/cwl-runner.html.textile.liquid b/doc/user/cwl/cwl-runner.html.textile.liquid
index c00f475..6dac43a 100644
--- a/doc/user/cwl/cwl-runner.html.textile.liquid
+++ b/doc/user/cwl/cwl-runner.html.textile.liquid
@@ -15,6 +15,7 @@ The @arvados-cwl-runner@ client is installed by default on Arvados shell nodes.
<notextile>
<pre><code>~$ <span class="userinput">virtualenv ~/venv</span>
~$ <span class="userinput">. ~/venv/bin/activate</span>
+~$ <span class="userinput">pip install -U setuptools</span>
~$ <span class="userinput">pip install arvados-cwl-runner</span>
</code></pre>
</notextile>
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list