[ARVADOS] updated: 1.3.0-3054-g8a00acc17
Git user
git at public.arvados.org
Thu Sep 3 14:13:51 UTC 2020
Summary of changes:
doc/admin/keep-recovering-data.html.textile.liquid | 91 ++++++++++
doc/architecture/keep-clients.html.textile.liquid | 39 +++++
.../manifest-format.html.textile.liquid | 184 +++++++++++++++++++++
3 files changed, 314 insertions(+)
create mode 100644 doc/admin/keep-recovering-data.html.textile.liquid
create mode 100644 doc/architecture/keep-clients.html.textile.liquid
create mode 100644 doc/architecture/manifest-format.html.textile.liquid
via 8a00acc17403e7836b88b1e9e66b4ff47d5505f2 (commit)
from f7f5dfb456c1ddb6db63cf6e6a79e5d468d20a20 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 8a00acc17403e7836b88b1e9e66b4ff47d5505f2
Author: Ward Vandewege <ward at curii.com>
Date: Thu Sep 3 10:13:37 2020 -0400
16421: documentation: check in all new files.
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/admin/keep-recovering-data.html.textile.liquid b/doc/admin/keep-recovering-data.html.textile.liquid
new file mode 100644
index 000000000..e0949fd55
--- /dev/null
+++ b/doc/admin/keep-recovering-data.html.textile.liquid
@@ -0,0 +1,91 @@
+---
+layout: default
+navsection: admin
+title: "Recovering data"
+...
+
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+h2(#recover_collection). Recovering collections
+
+To recover a collection the manifest is required. Arvados has a built-in audit log, which consists of a row added to the "logs" table in the PostgreSQL database each time an Arvados object is created, modified, or deleted. Collection manifests are included, unless they are listed in the @UnloggedAttributes@ configuration parameter. The audit log is retained for up to @MaxAge at .
+
+In some cases, it is possible to recover files that have been lost by modifying or deleting a collection.
+
+Possibility of recovery depends on many factors, including:
+* Whether the collection manifest is still available, e.g., in an audit log entry
+* Whether the data blocks are also referenced by other collections
+* Whether the data blocks have been unreferenced long enough to be marked for deletion/trash by keep-balance
+* Blob signature TTL, trash lifetime, trash check interval, and other config settings
+
+To attempt recovery of a previous version of a deleted/modified collection, use the @arvados-server recover-collection@ command. It should be run on one of your server nodes where the @arvados-server@ package is installed and the @/etc/arvados/config.yml@ file is up to date.
+
+Specify the collection you want to recover by passing either the UUID of an audit log entry, or a file containing the manifest.
+
+If recovery is successful, the @recover-collection@ program saves the recovered data a new collection belonging to the system user, and prints the new collection's UUID on stdout.
+
+<pre>
+# arvados-server recover-collection 9tee4-57u5n-nb5awmk1pahac2t
+INFO[2020-06-05T19:52:29.557761245Z] loaded log entry logged_event_time="2020-06-05 16:48:01.438791 +0000 UTC" logged_event_type=update old_collection_uuid=9tee4-4zz18-1ex26g95epmgw5w src=9tee4-57u5n-nb5awmk1pahac2t
+INFO[2020-06-05T19:52:29.642145127Z] recovery succeeded UUID=9tee4-4zz18-5trfp4k4xxg97f1 src=9tee4-57u5n-nb5awmk1pahac2t
+9tee4-4zz18-5trfp4k4xxg97f1
+INFO[2020-06-05T19:52:29.644699436Z] exiting
+</pre>
+
+In this example, the original data has been restored and saved in a new collection with UUID @9tee4-4zz18-5trfp4k4xxg97f1 at .
+
+For more options, run @arvados-server recover-collection -help at .
+
+h2(#untrashing_lost_blocks). Untrashing lost blocks
+
+In some cases it is possible to recover data blocks that were trashed erroneously by @keep-balance@ (e.g. due to an install/config error).
+
+If you suspect blocks have been trashed erroneously, you should immediately:
+
+* On all keepstore servers: set @BlobTrashCheckInterval@ to a long time like 2400h
+* On all keepstore servers: restart keepstore
+* Stop the keep-balance service
+
+When you think you have corrected the underlying problem, you should:
+
+* Set @BlobMissingReport@ to a suitable value (perhaps "/tmp/keep-balance-lost-blocks.txt").
+* Start @keep-balance@
+* After @keep-balance@ completes its first sweep, inspect /tmp/keep-balance-lost-blocks.txt. If it's not empty, you can request all keepstores to untrash any blocks that are still recoverable with a script like this:
+
+<notextile>
+<pre><code>
+#!/bin/bash
+set -e
+
+# see Client.AuthToken in /etc/arvados/keep-balance/keep-balance.yml
+token=xxxxxxx-your-system-auth-token-xxxxxxx
+
+# all keep server hostnames
+hosts=(keep0 keep1 keep2 keep3 keep4 keep5)
+
+while read hash pdhs; do
+ echo "${hash}"
+ for h in ${hosts[@]}; do
+ if curl -fgs -H "Authorization: Bearer $token" -X PUT "http://${h}:25107/untrash/$hash"; then
+ echo "${hash} ok ${host}"
+ fi
+ done
+done < /tmp/keep-balance-lost-blocks.txt
+</code>
+</pre>
+</notextile>
+
+Obviously this could be improved upon with increased parallelism for large scale tasks, if needed. Any blocks which were successfully untrashed can be removed from the list of blocks and collections which need to be recovered.
+
+h2(#regenerating_lost_blocks). Regenerating lost blocks
+
+For blocks which were trashed long enough ago that they've been deleted, it may be possible to regenerate them by rerunning the workflows which generated them. To do this, the process is:
+
+* Delete the affected collections so that job reuse doesn't attempt to reuse them (it's likely that if one block is missing, they all are, so they're unlikely to contain any useful data)
+* Resubmit any container requests for which you want the output collections regenerated
+
+The Arvados repository contains a tool that can be used to generate a report to help with this task at "arvados/tools/keep-xref/keep-xref.py":https://github.com/arvados/arvados/blob/master/tools/keep-xref/keep-xref.py
diff --git a/doc/architecture/keep-clients.html.textile.liquid b/doc/architecture/keep-clients.html.textile.liquid
new file mode 100644
index 000000000..31e549f0c
--- /dev/null
+++ b/doc/architecture/keep-clients.html.textile.liquid
@@ -0,0 +1,39 @@
+---
+layout: default
+navsection: architecture
+title: Keep clients
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+Keep clients are applications such as @arv-get@, @arv-put@ and @arv-mount@ which store and retrieve data from Keep. In doing so, these programs interact with both the API server (which stores file metadata in the form of @collection@ objects) and individual @keepstore@ servers (which store the actual data blocks).
+
+!(full-width){{site.baseurl}}/images/Keep_reading_writing_block.svg!
+
+h2. Storing a file
+
+# The client discovers keep servers (or proxies) using the @accessible@ method on "keep_services":{{site.baseurl}}/api/methods/keep_services.html
+# Data is split into 64 MiB blocks and the MD5 hash is computed for each block.
+# The client uploads each block to one or more Keep servers, based on the number of desired replicas. The priority order is determined using rendezvous hashing, described below.
+# The Keep server returns a block locator (the MD5 sum of the block) and a "signed token" which the client can use as proof of knowledge for the block.
+# The client constructs a @manifest@ which lists the blocks by MD5 hash and how to reassemble them into the original files.
+# The client creates a "collection":{{site.baseurl}}/api/methods/collections.html and provides the @manifest_text@
+# The API server accepts the collection after validating the signed tokens (proof of knowledge) for each block.
+
+h2. Fetching a file
+
+# The client requests a @collection@ object including @manifest_text@ from the APIs server
+# The server adds "token signatures" to the @manifest_text@ and returns it to the client.
+# The client discovers keep servers (or proxies) using the @accessible@ method on "keep_services":{{site.baseurl}}/api/methods/keep_services.html
+# For each data block, the client chooses the highest priority server using rendezvous hashing, described below.
+# The client sends the data block request to the keep server, along with the token signature from the API which proves to Keep servers that the client is permitted to read a given block.
+# The server provides the block data after validating the token signature for the block (if the server does not have the block, it returns a 404 and the client tries the next highest priority server)
+
+h2(#rendezvous). Rendezvous hashing
+!(full-width){{site.baseurl}}/images/Keep_rendezvous_hashing.svg!
+
+Each @keep_service@ resource has an assigned uuid. To determine priority assignments of blocks to servers, for each keep service compute the MD5 sum of the string concatenation of the block locator (hex-coded hash part only) and service uuid, then sort this list in descending order. Blocks are preferentially placed on servers with the highest weight.
+
diff --git a/doc/architecture/manifest-format.html.textile.liquid b/doc/architecture/manifest-format.html.textile.liquid
new file mode 100644
index 000000000..1780768bc
--- /dev/null
+++ b/doc/architecture/manifest-format.html.textile.liquid
@@ -0,0 +1,184 @@
+---
+layout: default
+navsection: architecture
+title: Manifest format
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+Each collection record has a @manifest_text@ field, which describes how to reassemble keep blocks into files. Each block identifier in the manifest has an added signature which is used to confirm permission to read the block. To read a block from a keepstore server, the client must provide the block identifier, the signature, and the same API token used to retrieve the collection record.
+
+!(full-width){{site.baseurl}}/images/Keep_manifests.svg!
+
+h2. Manifest v1
+
+A manifest is utf-8 encoded text, consisting of zero or more newline-terminated streams.
+
+<pre>
+manifest ::= stream*
+stream ::= stream-name (" " locator)+ (" " file-segment)+ "\n"
+stream-name ::= "." ("/" path-component)*
+path-component ::= <printable ASCII - (whitespace, "/")>+
+file-segment ::= position ":" size ":" filename
+position ::= [0-9]+
+size ::= [0-9]+
+filename ::= path-component ("/" path-component)*
+</pre>
+
+Notes:
+
+* The first token is the stream name, consisting of one or more path components, delimited by @"/"@.
+** The first path component is always @"."@.
+** No path component is empty.
+** No path component following the first one can be "." or "..".
+** The stream name never begins or ends with @"/"@.
+* The next N tokens are "keep locators":#locator
+** These describe the "data stream". By logically concatenating the blocks in the order that they appear, we can refer to "positions" in the data stream.
+* File tokens come after the sequence of keep locators.
+** A file token has three parts, delimited by @":"@: position, size, filename.
+** Position and size are given in decimal
+** The position is the position in the data stream
+** The size is the count of bytes following the position in the data stream. A file size may cross multiple blocks in the data stream.
+** Filename may contain @"/"@ characters, but must not start or end with @"/"@, and must not contain @"//"@.
+** Filename components (delimited by @"/"@) must not be @"."@ or @".."@.
+** There may be multiple file tokens.
+
+It is legal to have multiple file tokens in the manifest (possible across different streams) with the same combined path name @stream name + "/" + filename at . This must be interpreted as a concatenation of file content, in the order that the file tokens appear in the manifest.
+
+Spaces are represented by the escape sequence @\040 at . Spaces in stream names and filenames must be translated when reading and writing manifests. A manifest may not contain TAB characters, nor other ASCII whitespace characters or control codes other than the spaces or newlines used as delimiters specified above. A manifest always ends with a newline -- except the empty (zero-length) string, which is a valid manifest.
+
+h3. Normalized manifest v1
+
+A normalized manifest is a manifest that meets the following additional restrictions:
+
+* Streams are in alphanumeric order.
+* Each stream name is unique within the manifest.
+* Files within a stream are listed in alphanumeric order.
+* Blocks within a stream are ordered based on order of file tokens of the stream. A given block is listed at most once in a stream.
+* Filename must not contain @"/"@ (the stream name represents the path prefix)
+
+h3. Example manifests
+
+A manifest with four files in two directories:
+
+<pre>
+. 930625b054ce894ac40596c3f5a0d947+33 0:0:a 0:0:b 0:33:output.txt
+./c d41d8cd98f00b204e9800998ecf8427e+0 0:0:d
+</pre>
+
+The same manifest with permission signatures on each block:
+
+<pre>
+. 930625b054ce894ac40596c3f5a0d947+33+A1f27a35dd9af37191d63ad8eb8985624451e7b79 at 5835c8bc 0:0:a 0:0:b 0:33:output.txt
+./c d41d8cd98f00b204e9800998ecf8427e+0+A27117dcd30c013a6e85d6d74c9a50179a1446efa at 5835c8bc 0:0:d
+</pre>
+
+A manifest containing a file consisting of multiple blocks and a space in the file name:
+
+<pre>
+. c449ed86671e4a34a8b8b9430850beba+67108864 09fcfea01c3a141b89dd0dcfa1b7768e+22534144 0:89643008:Docker\040image.tar
+</pre>
+h2(#locator). Keep locator format
+
+BNF notation for a valid Keep locator string (with hints). For example: *d41d8cd98f00b204e9800998ecf8427e+0+Z+Ada39a3ee5e6b4b0d3255bfef95601890afd80709 at 53bed294*
+
+<pre>
+locator ::= sized-digest hint*
+sized-digest ::= digest size-hint
+digest ::= <32 lowercase hexadecimal digits>
+size-hint ::= "+" [0-9]+
+hint ::= "+" hint-type hint-content
+hint-type ::= [A-Z]+
+hint-content ::= [A-Za-z0-9 at _-]*
+sign-hint ::= "+A" <40 lowercase hexadecimal digits> "@" sign-timestamp
+remote-sign-hint ::= "+R" [A-Za-z0-9]{5} "-" <40 lowercase hexadecimal digits> "@" sign-timestamp
+sign-timestamp ::= <8 lowercase hexadecimal digits>
+</pre>
+
+h3. Regular expression to validate locator
+
+<pre>
+/^([0-9a-f]{32})\+([0-9]+)(\+[A-Z][-A-Za-z0-9 at _]*)*$/
+</pre>
+
+h3. Valid locators
+
+table(table table-bordered table-condensed).
+|@d41d8cd98f00b204e9800998ecf8427e+0@|
+|@d41d8cd98f00b204e9800998ecf8427e+0+Z@|
+|<code>d41d8cd98f00b204e9800998ecf8427e+0+Z+Ada39a3ee5e6b4b0d3255bfef95601890afd80709 at 53bed294</code>|
+|<code>930625b054ce894ac40596c3f5a0d947+33+Rzzzzz-1f27a35dd9af37191d63ad8eb8985624451e7b79 at 5835c8bc</code>|
+
+h3. Invalid locators
+
+table(table table-bordered table-condensed).
+||Why|
+|@d41d8cd98f00b204e9800998ecf8427e@|No size hint|
+|@d41d8cd98f00b204e9800998ecf8427e+Z+0@|Other hint before size hint|
+|@d41d8cd98f00b204e9800998ecf8427e+0+0@|Multiple size hints|
+|@d41d8cd98f00b204e9800998ecf8427e+0+z@|Hint does not start with uppercase letter|
+|@d41d8cd98f00b204e9800998ecf8427e+0+Zfoo*bar@|Hint contains invalid character @*@|
+
+h3. Token signatures
+
+A token signature (sign-hint) provides proof-of-access for a data block. It is computed by taking a SHA1 HMAC of the blob signing token (a shared secret between the API server and keep servers), block digest, current API token, expiration timestamp, and blob signature TTL.
+
+When communicating with the @keepstore@ to fetch a block, or the API server to create or update a collection, the service computes the expected token signature for each block and compares it to the token signature that was presented by the client. Keep clients receive valid block signatures when uploading a block to a keep store (getting back a signed token as proof of knowledge) or, from the API server, getting the manifest text of a collection on which the user has read permission.
+
+Security of a token signature is derived from the following characteristics:
+
+# Valid signatures can only be generated by entities that know the shared secret (the "blob signing token")
+# A signature can only be used by an entity that also know the API token that was used to generate it.
+# It expires after a set date (the expiration time, based on the "blob signature time-to-live (TTL)")
+
+h3(#federationsignatures). Federation and signatures
+
+When a collection record is returned through a federation request, the keep blocks listed in the manifest may not be available on the local cluster, and the keep block signatures returned by the remote cluster are not valid for the local cluster. To solve this, @arvados-controller@ rewrites the signatures in the manifest to "remote cluster" signatures.
+
+A local signature comes after the block identifier and block size, and starts with @+A@:
+
+<code>930625b054ce894ac40596c3f5a0d947+33+A1f27a35dd9af37191d63ad8eb8985624451e7b79 at 5835c8bc</code>
+
+A remote cluster signature starts with @+R@, then the cluster id of the cluster it originated from (@zzzzz@ in this example), a dash, and then the original signature:
+
+<code>930625b054ce894ac40596c3f5a0d947+33+Rzzzzz-1f27a35dd9af37191d63ad8eb8985624451e7b79 at 5835c8bc</code>
+
+When the client provides a remote-signed block locator to keepstore, the keepstore proxies the request to the remote cluster.
+
+# keepstore determines the cluster id to contact from the first part of the @+R@ signature
+# creates a salted token using the API token and cluster id
+# contacts the "accessible" endpoint on the remote cluster to determine the remote cluster's keepstore or keepproxy hosts
+# converts the remote signature @+R@ back to a local signature @+A@
+# contacts the remote keepstore or keepproxy host and requests the block using the local signature
+# returns the block contents back to the client
+
+h3(#example). Example
+
+This example uses @c1bad4b39ca5a924e481008009d94e32+210@, which is the content hash of a @collection@ that was added to Keep in "how to upload data":{{ site.baseurl }}/user/tutorials/tutorial-keep.html. Get the collection manifest using @arv-get@:
+
+<notextile>
+<pre><code>~$ <span class="userinput">arv-get c1bad4b39ca5a924e481008009d94e32+210</span>
+. 204e43b8a1185621ca55a94839582e6f+67108864+Aasignatureforthisblockaaaaaaaaaaaaaaaaaa at 5f612ee6 b9677abbac956bd3e86b1deb28dfac03+67108864+Aasignatureforthisblockbbbbbbbbbbbbbbbbbb at 5f612ee6 fc15aff2a762b13f521baf042140acec+67108864+Aasignatureforthisblockcccccccccccccccccc at 5f612ee6 323d2a3ce20370c4ca1d3462a344f8fd+25885655+Aasignatureforthisblockdddddddddddddddddd at 5f612ee6 0:227212247:var-GS000016015-ASM.tsv.bz2
+</code></pre>
+</notextile>
+
+This collection includes a single file @var-GS000016015-ASM.tsv.bz2@ which is 227212247 bytes long. It is stored using four sequential data blocks with hashes @204e43b8a1185621ca55a94839582e6f+67108864@, @b9677abbac956bd3e86b1deb28dfac03+67108864@, @fc15aff2a762b13f521baf042140acec+67108864@, and @323d2a3ce20370c4ca1d3462a344f8fd+25885655 at . Each of the block hashes is followed by the rest of their "locator":#locator.
+
+Use @arv-get@ to download the first data block:
+
+notextile. <pre><code>~$ <span class="userinput">arv-get 204e43b8a1185621ca55a94839582e6f+67108864+Aasignatureforthisblockaaaaaaaaaaaaaaaaaa at 5f612ee6 > block1</span></code></pre>
+
+Inspect the size and compute the MD5 hash of @block1@:
+
+<notextile>
+<pre><code>~$ <span class="userinput">ls -l block1</span>
+-rw-r--r-- 1 you group 67108864 Dec 9 20:14 block1
+~$ <span class="userinput">md5sum block1</span>
+204e43b8a1185621ca55a94839582e6f block1
+</code></pre>
+</notextile>
+
+As expected, the md5sum of the contents of the block matches the @digest@ part of the "locator":#locator, and the size of the contents matches the @size-hint at .
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list