[ARVADOS] updated: 1.3.0-3055-gdafd68e63
Git user
git at public.arvados.org
Fri Sep 4 15:27:02 UTC 2020
Summary of changes:
doc/_config.yml | 6 ++--
doc/admin/keep-recovering-data.html.textile.liquid | 24 +++++++++++++--
.../keep-data-lifecycle.html.textile.liquid | 4 ++-
doc/architecture/storage.html.textile.liquid | 2 +-
...l-keep-collection-lifecycle.html.textile.liquid | 34 +++++++---------------
5 files changed, 38 insertions(+), 32 deletions(-)
via dafd68e639a416fcae216c639c6e38beab53b214 (commit)
from 8a00acc17403e7836b88b1e9e66b4ff47d5505f2 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit dafd68e639a416fcae216c639c6e38beab53b214
Author: Ward Vandewege <ward at curii.com>
Date: Fri Sep 4 11:26:49 2020 -0400
16421: improvements based on review comments.
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/_config.yml b/doc/_config.yml
index 25321b1ab..a85576e45 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -143,12 +143,12 @@ navbar:
architecture:
- Topics:
- architecture/index.html.textile.liquid
- - Keep (storage):
+ - Storage in Keep:
- architecture/storage.html.textile.liquid
+ - architecture/keep-clients.html.textile.liquid
- architecture/keep-data-lifecycle.html.textile.liquid
- architecture/manifest-format.html.textile.liquid
- - architecture/keep-clients.html.textile.liquid
- - Crunch (computation):
+ - Computation with Crunch:
- api/execution.html.textile.liquid
- Other:
- api/permission-model.html.textile.liquid
diff --git a/doc/admin/keep-recovering-data.html.textile.liquid b/doc/admin/keep-recovering-data.html.textile.liquid
index e0949fd55..3a9f51e56 100644
--- a/doc/admin/keep-recovering-data.html.textile.liquid
+++ b/doc/admin/keep-recovering-data.html.textile.liquid
@@ -10,9 +10,27 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+Arvados has several features to prevent accidental loss or deletion of data, but accidents can happen. This page lays out the options to recover deleted or overwritten collections.
+
+For more detail on the data lifecycle in Arvados, see the "Data lifecycle":{{ site.baseurl }}/architecture/keep-data-lifecycle.html page.
+
+h2(#check_the_trash). Check the trash
+
+When a collection is deleted, it is moved to the trash. It will remain there for the duration of @Collections.DefaultTrashLifetime@, and it can be untrashed via workbench or with the cli tools, as described in "Recovering trashed collections":{{ site.baseurl }}/user/tutorials/tutorial-keep-collection-lifecycle.html#trash-recovery.
+
+h2(#check_other_collections). Check for other collections with the same PDH
+
+Multiple collections may share a _portable data hash_, i.e. have the same contents. If another collection exists with the same portable data hash, recovering data is not necessary, everything is still stored in Keep. A new copy of the collection can be made to make the data available in the correct project and with the correct permissions.
+
+h2(#check_collection_versioning). Consider collection versioning
+
+Arvados supports collection versioning. If it has been "enabled":{{ site.baseurl }}/admin/collection-versioning.html on your cluster, the deleted collection may be recoverable from an older version. See "Using collection versioning":{{ site.baseurl }}/user/topics/collection-versioning.html for details.
+
h2(#recover_collection). Recovering collections
-To recover a collection the manifest is required. Arvados has a built-in audit log, which consists of a row added to the "logs" table in the PostgreSQL database each time an Arvados object is created, modified, or deleted. Collection manifests are included, unless they are listed in the @UnloggedAttributes@ configuration parameter. The audit log is retained for up to @MaxAge at .
+When all the above options fail, it may still be possible to recover a collection that has been deleted.
+
+To recover a collection the manifest is required. Arvados has a built-in audit log, which consists of a row added to the "logs" table in the PostgreSQL database each time an Arvados object is created, modified, or deleted. Collection manifests are included, unless they are listed in the @AuditLogs.UnloggedAttributes@ configuration parameter. The audit log is retained for up to @AuditLogs.MaxAge at .
In some cases, it is possible to recover files that have been lost by modifying or deleting a collection.
@@ -52,7 +70,7 @@ If you suspect blocks have been trashed erroneously, you should immediately:
When you think you have corrected the underlying problem, you should:
-* Set @BlobMissingReport@ to a suitable value (perhaps "/tmp/keep-balance-lost-blocks.txt").
+* Set @Collections.BlobMissingReport@ to a suitable value (perhaps "/tmp/keep-balance-lost-blocks.txt").
* Start @keep-balance@
* After @keep-balance@ completes its first sweep, inspect /tmp/keep-balance-lost-blocks.txt. If it's not empty, you can request all keepstores to untrash any blocks that are still recoverable with a script like this:
@@ -79,7 +97,7 @@ done < /tmp/keep-balance-lost-blocks.txt
</pre>
</notextile>
-Obviously this could be improved upon with increased parallelism for large scale tasks, if needed. Any blocks which were successfully untrashed can be removed from the list of blocks and collections which need to be recovered.
+Any blocks which were successfully untrashed can be removed from the list of blocks and collections which need to be recovered.
h2(#regenerating_lost_blocks). Regenerating lost blocks
diff --git a/doc/architecture/keep-data-lifecycle.html.textile.liquid b/doc/architecture/keep-data-lifecycle.html.textile.liquid
index c86e63c47..a2e31fb0e 100644
--- a/doc/architecture/keep-data-lifecycle.html.textile.liquid
+++ b/doc/architecture/keep-data-lifecycle.html.textile.liquid
@@ -24,7 +24,9 @@ The nominal state is *persisted* which means the data can be can be accessed nor
A collection is *expiring* when it has a *trash_at* time in the future. An expiring collection can be accessed as normal, but is scheduled to be trashed automatically at the *trash_at* time.
-A collection is *trashed* when it has a *trash_at* time in the past. The *is_trashed* attribute will also be "true". The delete operation immediately puts the collection in the trash by setting the *trash_at* time to "now". Once trashed, the collection is no longer readable through normal data access APIs. The collection will have *delete_at* set to some time in the future. The trashed collection is recoverable until the delete_at time passes, at which point the collection is permanently deleted.
+A collection is *trashed* when it has a *trash_at* time in the past. The *is_trashed* attribute will also be "true". The delete operation immediately puts the collection in the trash by setting the *trash_at* time to "now", and *delete_at* defaults to "now" + @Collections.DefaultTrashLifetime at . Once trashed, the collection is no longer readable through normal data access APIs. The collection will have *delete_at* set to some time in the future. The trashed collection is recoverable until the *delete_at* time passes, at which point the collection is permanently deleted.
+
+See "Recovering trashed collections":{{ site.baseurl }}/user/tutorials/tutorial-keep-collection-lifecycle.html#trash-recovery for instructions to recover trashed collections.
h3(#collection_attributes). Collection lifecycle attributes
diff --git a/doc/architecture/storage.html.textile.liquid b/doc/architecture/storage.html.textile.liquid
index 18423c68c..3218363f4 100644
--- a/doc/architecture/storage.html.textile.liquid
+++ b/doc/architecture/storage.html.textile.liquid
@@ -1,7 +1,7 @@
---
layout: default
navsection: architecture
-title: Keep - Content-Addressable Storage
+title: Introduction to Keep
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
diff --git a/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid b/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
index c4b9e31ec..9ddec04f5 100644
--- a/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
@@ -1,7 +1,7 @@
---
layout: default
navsection: userguide
-title: "Keep collection lifecycle"
+title: "Trashing and untrashing data"
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
@@ -9,23 +9,18 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-During it's lifetime, a keep collection can be in various states. These states are *persisted*, *expiring*, *trashed* and *permanently deleted*.
+Collections have a sophisticated data lifecycle, which is documented in the architecture guide at "Collection lifecycle":{{ site.baseurl }}/architecture/keep-data-lifecycle.html#collection_lifecycle.
-The nominal state is *persisted* which means the data can be can be accessed normally and will be retained indefinitely.
+Arvados supports trashing (deletion) of collections. For a period of time after a collection is trashed, it can be "untrashed". After that period, the collection is permanently deleted, though there may still be ways to recover the data, see "Recovering data":{{ site.baseurl }}/admin/keep-recovering-data.html in the admin guide for more details.
-A collection is *expiring* when it has a *trash_at* time in the future. An expiring collection can be accessed as normal, but is scheduled to be trashed automatically at the *trash_at* time.
-
-A collection is *trashed* when it has a *trash_at* time in the past. The *is_trashed* attribute will also be "true". The delete operation immediately puts the collection in the trash by setting the *trash_at* time to "now". Once trashed, the collection is no longer readable through normal data access APIs. The collection will have *delete_at* set to some time in the future. The trashed collection is recoverable until the delete_at time passes, at which point the collection is permanently deleted.
-
-# "*Deleting / trashing collections*":#delete-collection
+# "*Trashing (deleting) collections*":#delete-collection
# "*Recovering trashed collections*":#trash-recovery
-# "*Collection lifecycle attributes*":#collection_attributes
{% include 'tutorial_expectations' %}
-h2(#delete-collection). Deleting / trashing collections
+h2(#delete-collection). Trashing (deleting) collections
-A collection can be deleted using either the arv command line tool or the workbench.
+A collection can be trashed using workbench or the arv command line tool.
h3. Trashing a collection using workbench
@@ -39,15 +34,15 @@ arv collection delete --uuid=zzzzz-4zz18-xxxxxxxxxxxxxxx
h2(#trash-recovery). Recovering trashed collections
-A collection can be un-trashed / recovered using either the arv command line tool or the workbench.
+A collection can be untrashed / recovered using workbench or the arv command line tool.
-h3. Un-trashing a collection using workbench
+h3. Untrashing a collection using workbench
To untrash a collection using workbench, go to trash page on workbench by clicking on the "Trash" icon in the top navigation in workbench and use the recycle icon or selection dropdown option.
!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/trash-button-topnav.png!
-h3. Un-trashing a collection using arv command line tool
+h3. Untrashing a collection using arv command line tool
You can list the trashed collections using the list command.
@@ -61,13 +56,4 @@ You can then untrash a particular collection using arv using it's uuid.
arv collection untrash --uuid=zzzzz-4zz18-xxxxxxxxxxxxxxx
</pre>
-h2(#collection_attributes). Collection lifecycle attributes
-
-As listed above the attributes that are used to manage a collection lifecycle are it's *is_trashed*, *trash_at*, and *delete_at*. The table below lists the values of these attributes and how they influence the state of a collection and it's accessibility.
-
-table(table table-bordered table-condensed).
-|_. collection state|_. is_trashed|_. trash_at|_. delete_at|_. get|_. list|_. list?include_trash=true|_. can be modified|
-|persisted collection|false |null |null |yes |yes |yes |yes |
-|expiring collection|false |future |future |yes |yes |yes |yes |
-|trashed collection|true |past |future |no |no |yes |only is_trashed, trash_at and delete_at attribtues|
-|deleted collection|true|past |past |no |no |no |no |
+The architecture section has a more detailed description of the "data lifecycle":{{ site.baseurl }}/architecture/keep-data-lifecycle.html in Keep.
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list