[arvados] updated: 2.7.0-5310-g357858970a
git repository hosting
git at public.arvados.org
Tue Nov 7 15:15:40 UTC 2023
Summary of changes:
.../app/controllers/application_controller.rb | 2 +-
doc/_config.yml | 1 +
doc/admin/keep-faster-gc-s3.html.textile.liquid | 30 +++++++++++++
.../install-dispatch-cloud.html.textile.liquid | 2 +-
lib/config/config.default.yml | 4 ++
sdk/ruby/lib/arvados.rb | 10 ++---
sdk/ruby/test/sdk_fixtures.rb | 2 +-
sdk/ruby/test/test_keep_manifest.rb | 2 -
sdk/ruby/test/test_request_id.rb | 4 +-
services/api/Gemfile | 12 +++--
services/api/Gemfile.lock | 21 +++++----
services/api/app/models/authorized_key.rb | 11 +++--
services/api/app/models/container.rb | 20 ++++-----
services/api/app/models/container_request.rb | 20 ++++-----
services/api/app/models/node.rb | 9 ++--
services/api/app/models/pipeline_instance.rb | 9 ++--
.../config/initializers/request_id_middleware.rb | 4 +-
.../functional/arvados/v1/users_controller_test.rb | 2 +-
.../api_client_authorizations_scopes_test.rb | 51 ++++++++++++----------
services/api/test/integration/cross_origin_test.rb | 4 +-
services/api/test/test_helper.rb | 12 ++---
tools/arvbox/lib/arvbox/docker/Dockerfile.base | 6 +--
.../multi_host/aws/pillars/arvados.sls | 5 +++
tools/salt-install/installer.sh | 11 ++++-
24 files changed, 149 insertions(+), 105 deletions(-)
create mode 100644 doc/admin/keep-faster-gc-s3.html.textile.liquid
via 357858970aef82a2a87b5105d133a52196987381 (commit)
via c306aaa029d8e65a4399f27fec982e4e2f72ba8d (commit)
via 9f42b105d76cffd91e2d91a6b5d25ff4418f192c (commit)
via 56a2e5c346622637639e2c93cabba14cf853bb11 (commit)
via b6c0e611996b07c10f65c6fb3c204e7ca5a1b29b (commit)
via 9c78326f370f1875e41422e2d2c8a3c5a86c5bf8 (commit)
via 894e1a3d70e9ec1b3e8619d1822410d665fabab4 (commit)
via 525e81bdbe18ea369f5a61a713e743608bab2e42 (commit)
via 5fec43173f9744cbf49e422468a71cd8a9b5d9d3 (commit)
via 3ed6fab9c3ec7ca31133e4d4c750d546727b1847 (commit)
via ed1d6f12ddd0bd0f46ee47bf560cada40cd00835 (commit)
via c51e59e03bc721de2837db7958415766bd7b46c8 (commit)
via 14c8fb3d5ae381e6bc0609530209ee7f7a584fe3 (commit)
via 365e79dc8fdeeed324c3e6f4f20ce200f4d28c3c (commit)
via 88e18b7b9826b05e8485a6a99593ecda850969d7 (commit)
via 7435f58364e90b8bf75db21602592ff0e7559b48 (commit)
via 9f40073eab9f4368bfb215c27edf4fcf52ef80b8 (commit)
via 788884922ce53e8884ff55f0fb11cb967629b72f (commit)
via be13306c210c458dfd74f9a1aab8377534f8760f (commit)
via 5b15a8ec15e1afb4eb5e61af59bf237dd346ff8e (commit)
via ebbb591d0d68e10934e3c8132b24b12a70daaf99 (commit)
from 235f2d60f025823a72927d7ce56f4214c92cc875 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 357858970aef82a2a87b5105d133a52196987381
Author: Tom Clegg <tom at curii.com>
Date: Tue Nov 7 10:13:33 2023 -0500
21126: Add admin instructions re S3 garbage collection speedup.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at curii.com>
diff --git a/doc/_config.yml b/doc/_config.yml
index cbb082aefb..04ff5caa9b 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -195,6 +195,7 @@ navbar:
- admin/storage-classes.html.textile.liquid
- admin/keep-recovering-data.html.textile.liquid
- admin/keep-measuring-deduplication.html.textile.liquid
+ - admin/keep-faster-gc-s3.html.textile.liquid
- Cloud:
- admin/spot-instances.html.textile.liquid
- admin/cloudtest.html.textile.liquid
diff --git a/doc/admin/keep-faster-gc-s3.html.textile.liquid b/doc/admin/keep-faster-gc-s3.html.textile.liquid
new file mode 100644
index 0000000000..872a772d2d
--- /dev/null
+++ b/doc/admin/keep-faster-gc-s3.html.textile.liquid
@@ -0,0 +1,30 @@
+---
+layout: default
+navsection: admin
+title: "Faster garbage collection in S3"
+...
+
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+When there is a large number of unneeded blocks stored in an S3 bucket, particularly when using @PrefixLength: 0@, the speed of garbage collection can be severely limited by AWS API rate limits and Arvados's multi-step trash/delete process.
+
+The multi-step trash/delete process can be short-circuited by setting @BlobTrashLifetime@ to zero and enabling @UnsafeDelete@ on S3-backed volumes. However, on an actively used cluster such a configuration *can result in data loss* in the rare case where a given block is trashed and then rewritten soon afterward, and S3 processes the write and delete requests in the opposite order.
+
+The following steps can be used to temporarily disable writes on an S3 bucket to enable faster garbage collection without data loss or service interruption. Note that garbage collection on other S3 volumes will be temporarily disabled during this procedure.
+# Create a new S3 bucket and configure it as an additional volume (this step may be skipped if the configuration already has enough writable volumes that clients will still be able to write blocks while the target volume is read-only). We recommend using @PrefixLength: 3@ for the new volume because this results in a much higher rate limit for I/O and garbage collection operations compared to the default @PrefixLength: 0 at . If the target volume configuration specifies @StorageClasses@, use the same values for the new volume.
+# Shut down the @keep-balance@ service.
+# Update the target volume configuration with @ReadOnly: true@, @AllowTrashWhileReadOnly: true@, and @UnsafeDelete: true at .
+# In the @Collections@ configuration section, set @BlobTrashLifetime: 0 at . Note this will effectively disable garbage collection on other S3-backed volumes, and garbage collection will delete blocks outright (bypassing the recoverable trash phase) on any non-S3 volumes.
+# Restart all @keepstore@ services with the updated configuration.
+# Run @keep-balance@ with command line options @-commit-pulls=false -commit-trash=true at . For example, use @sudo systemctl edit keep-balance@ to override the default command line: <notextile><pre><code>[Service]
+ExecStart=
+ExecStart=/usr/bin/keep-balance -commit-pulls=false -commit-trash=true</code></pre></notextile> ...and then start the service using @sudo systemctl start keep-balance at . Note that if you do not use @-commit-pulls=false@ here, keep-balance will also start copying data from the target volume to other volumes, which can slow down garbage collection progress.
+# Garbage collection should now proceed faster on the target volume. Monitor progress by watching @keep-balance@ logs and metrics. When garbage collection is complete, keep-balance logs will show an empty changeset: <notextile><pre><code>zzzzz-bi6l4-0123456789abcdef (keep0.zzzzz.arvadosapi.com:25107, disk): ChangeSet{Pulls:0, Trashes:0}</code></pre></notextile>
+# Revert @BlobTrashLifetime@ to its previous value (or the default value, @336h@) and set @UnsafeDelete: false@ on the target volume.
+# Optionally, revert to @ReadOnly: false@ and @AllowTrashWhileReadOnly: false@ on the target volume and/or set @ReadOnly: true@ and @AllowTrashWhileReadOnly: true@ on the new volume, depending on where you intend to store new data in future. If the target volume has @PrefixLength: 0@ and the new volume has @PrefixLength: 3@, you can skip this step: new data will be stored on the new volume, some existing data will be moved automatically to other volumes, and some will be left on the target volume as long as it's needed.
+# Revert the @-commit-pulls=false@ change to the @keep-balance@ startup script, and restart @keep-balance at .
+# Restart all @keepstore@ services with the updated configuration.
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list