[ARVADOS] updated: 1.1.4-396-g679d5507d
Git user
git at public.curoverse.com
Thu Jun 14 15:30:23 EDT 2018
Summary of changes:
doc/_config.yml | 1 +
doc/admin/storage-classes.html.textile.liquid | 67 +++++++++++++++++++
...onfigure-azure-blob-storage.html.textile.liquid | 8 +--
.../configure-fs-storage.html.textile.liquid | 56 ++++++++++++++++
...configure-s3-object-storage.html.textile.liquid | 27 ++++++--
doc/install/install-keepstore.html.textile.liquid | 78 +++++-----------------
6 files changed, 164 insertions(+), 73 deletions(-)
create mode 100644 doc/admin/storage-classes.html.textile.liquid
create mode 100644 doc/install/configure-fs-storage.html.textile.liquid
via 679d5507d6c390fd03e5c71300001f2696b1f5b1 (commit)
via 4d82174d21dd5cdf0668a10ca016e72338fd97e6 (commit)
via 1a37db7b0a70b5672cd0f7c86b19b85ba824378f (commit)
from 296ac256c898c145f7e4566d86d5f09f4e966b43 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 679d5507d6c390fd03e5c71300001f2696b1f5b1
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Thu Jun 14 15:24:46 2018 -0400
13579: Admin documentation about storage classes
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/doc/_config.yml b/doc/_config.yml
index 49b9a3b4a..17073347c 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -168,6 +168,7 @@ navbar:
- install/install-shell-server.html.textile.liquid
- install/create-standard-objects.html.textile.liquid
- install/install-keepstore.html.textile.liquid
+ - install/configure-fs-storage.html.textile.liquid
- install/configure-s3-object-storage.html.textile.liquid
- install/configure-azure-blob-storage.html.textile.liquid
- install/install-keepproxy.html.textile.liquid
diff --git a/doc/admin/storage-classes.html.textile.liquid b/doc/admin/storage-classes.html.textile.liquid
index eed202d36..b518502c7 100644
--- a/doc/admin/storage-classes.html.textile.liquid
+++ b/doc/admin/storage-classes.html.textile.liquid
@@ -10,4 +10,58 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Storage classes (sometimes also called "storage tiers") allow you to control which volumes should be used to store particular collection data blocks. This can be used to implement data storage policies such as moving data to archival storage.
+Storage classes (alternately known as "storage tiers") allow you to control which volumes should be used to store particular collection data blocks. This can be used to implement data storage policies such as moving data to archival storage.
+
+h3. Configuring storage classes
+
+The storage classes for each volume are set in the per-volume "keepstore configuration":{{site.baseurl}}/install/install-keepstore.html
+
+<pre>
+Volumes:
+ - ... Volume configuration ...
+ #
+ # If no storage classes are specified, will use [default]
+ #
+ StorageClasses: null
+
+ - ... Volume configuration ...
+ #
+ # Specify this volume is in the "archival" storage class.
+ #
+ StorageClasses: [archival]
+
+</pre>
+
+Names of storage classes are internal to the cluster and decided by the administrator. Aside from "default", Arvados currently does not define any standard storage class names.
+
+h3. Using storage classes
+
+Desired storage classes are specified by setting the "storage_classes_desired" field of a Collection. For example, at the command line:
+
+<pre>
+$ arv collection update --uuid zzzzz-4zz18-dhhm0ay8k8cqkvg --collection '{"storage_classes_desired": ["archival"]}'
+</pre>
+
+By setting "storage_classes_desired" to "archival", the blocks that make up the collection will be preferentially moved to keepstore volumes which are configured with the "archival" storage class.
+
+You may also specify the desired storage class when using @arv-put@:
+
+<pre>
+$ arv-put --storage-classes=hot myfile.txt
+</pre>
+
+Collection blocks will be in the "default" storage class if not otherwise specified.
+
+A collection may have only specify one desired storage class.
+
+A user with write access to a collection may set any storage class.
+
+h3. Storage management notes
+
+The "keep-balance":{{site.baseurl}}/install/install-keep-balance.html service is responsible for deciding which blocks should be placed on which keepstore volumes. As part of the rebalancing behavior, it will determine where a block should go in order to satisfy the desired storage classes, and issue pull requests to copy the block from its original volume to the desired volume. The block will subsequently be moved to trash on the original volume.
+
+If a block appears in multiple collections with different storage classes, the block will be stored in separate volumes for each storage class, even if that results in overreplication, unless there is a volume which has all the desired storage classes.
+
+If a collection has a desired storage class which is not available in any keepstore volume, the collection's blocks will remain in place, and an error will appear in the @keep-balance@ logs.
+
+This feature does not provide a hard guarantee on where data will be stored. Data may be written to default storage and moved to the desired storage class later. If controlling data locality is a hard requirement (such as legal restrictions on the location of data) we recommend setting up multiple Arvados clusters.
diff --git a/doc/install/configure-azure-blob-storage.html.textile.liquid b/doc/install/configure-azure-blob-storage.html.textile.liquid
index d2c4e709d..41f86f9ba 100644
--- a/doc/install/configure-azure-blob-storage.html.textile.liquid
+++ b/doc/install/configure-azure-blob-storage.html.textile.liquid
@@ -76,8 +76,8 @@ Volumes:
# environment. See below for an example of an alternate cloud environment.
StorageBaseURL: ""
- # Storage classes to associate with this volume. See "Configuring
- # storage classes" in the "Admin" section of doc.arvados.org.
+ # Storage classes to associate with this volume. See "Storage
+ # classes" in the "Admin" section of doc.arvados.org.
StorageClasses: null
- # Example configuration to use Azure China.
diff --git a/doc/install/configure-fs-storage.html.textile.liquid b/doc/install/configure-fs-storage.html.textile.liquid
index 730eed665..ddd54c3f0 100644
--- a/doc/install/configure-fs-storage.html.textile.liquid
+++ b/doc/install/configure-fs-storage.html.textile.liquid
@@ -42,8 +42,8 @@ Volumes:
# don't want this.
Serialize: false
- # Storage classes to associate with this volume. See "Configuring
- # storage classes" in the "Admin" section of doc.arvados.org.
+ # Storage classes to associate with this volume. See "Storage
+ # classes" in the "Admin" section of doc.arvados.org.
StorageClasses: null
# Example of a second volume section
diff --git a/doc/install/configure-s3-object-storage.html.textile.liquid b/doc/install/configure-s3-object-storage.html.textile.liquid
index b4d9e929c..88172fa9f 100644
--- a/doc/install/configure-s3-object-storage.html.textile.liquid
+++ b/doc/install/configure-s3-object-storage.html.textile.liquid
@@ -71,8 +71,8 @@ Volumes:
# This is used to inform replication decisions at the Keep layer.
S3Replication: 2
- # Storage classes to associate with this volume. See "Configuring
- # storage classes" in the "Admin" section of doc.arvados.org.
+ # Storage classes to associate with this volume. See
+ # "Storage classes" in the "Admin" section of doc.arvados.org.
StorageClasses: null
# Enable deletion (garbage collection) even when TrashLifetime is
@@ -88,7 +88,7 @@ h3. Example config for Google cloud storage
See previous section for documentation of configuration fields.
-</pre>
+<pre>
Volumes:
- # Example configuration using alternate storage provider
# Configuration for Google cloud storage
diff --git a/doc/install/install-keepstore.html.textile.liquid b/doc/install/install-keepstore.html.textile.liquid
index 402d15cb5..05a6a0d53 100644
--- a/doc/install/install-keepstore.html.textile.liquid
+++ b/doc/install/install-keepstore.html.textile.liquid
@@ -166,7 +166,7 @@ When a block is newly written, it is protected from deletion for the duration in
If keep-balance instructs keepstore to trash a block which is older than @BlobSignatureTTL@, and @EnableDelete@ is true, the block will be moved to "trash". A block which is in the trash is no longer accessible by read requests, but has not yet been permanently deleted. Blocks which are in the trash may be recovered using the "untrash" API endpoint. Blocks are permanently deleted after they have been in the trash for the duration in @TrashLifetime at .
-Keep-balance is also responsible for balancing the distribution of blocks across keepstore servers by asking servers to pull blocks from other servers (as determined by their storage class and "rendezvous hashing order":{{site.baseurl}}/api/storage.html). Pulling a block makes a copy. If a block is overreplicated (i.e. there are excess copies) after pulling, it will be subsequently trashed on the original server.
+Keep-balance is also responsible for balancing the distribution of blocks across keepstore servers by asking servers to pull blocks from other servers (as determined by their "storage class":{{site.baseurl}}/admin/storage-classes.html and "rendezvous hashing order":{{site.baseurl}}/api/storage.html). Pulling a block makes a copy. If a block is overreplicated (i.e. there are excess copies) after pulling, it will be subsequently trashed on the original server.
h3. Configure storage volumes
commit 4d82174d21dd5cdf0668a10ca016e72338fd97e6
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Fri Jun 8 14:29:35 2018 -0400
docs wip
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/doc/admin/storage-classes.html.textile.liquid b/doc/admin/storage-classes.html.textile.liquid
new file mode 100644
index 000000000..eed202d36
--- /dev/null
+++ b/doc/admin/storage-classes.html.textile.liquid
@@ -0,0 +1,13 @@
+---
+layout: default
+navsection: admin
+title: Storage classes
+...
+
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+Storage classes (sometimes also called "storage tiers") allow you to control which volumes should be used to store particular collection data blocks. This can be used to implement data storage policies such as moving data to archival storage.
commit 1a37db7b0a70b5672cd0f7c86b19b85ba824378f
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Thu Jun 14 12:22:09 2018 -0400
13579: Update from comments
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/doc/install/configure-azure-blob-storage.html.textile.liquid b/doc/install/configure-azure-blob-storage.html.textile.liquid
index e34373726..d2c4e709d 100644
--- a/doc/install/configure-azure-blob-storage.html.textile.liquid
+++ b/doc/install/configure-azure-blob-storage.html.textile.liquid
@@ -9,7 +9,7 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-As an alternative to local and network-attached POSIX filesystems, Keepstore can store data in an Azure Storage container.
+Keepstore can store data in one or more Azure Storage containers.
h2. Create a storage container
@@ -64,7 +64,7 @@ Volumes:
ReadOnly: false
# Amount of time to wait for a response before failing the request
- RequestTimeout: 10m0s
+ RequestTimeout: 2m0s
# The storage account name, used for authentication
StorageAccountName: exampleStorageAccountName
diff --git a/doc/install/configure-fs-storage.html.textile.liquid b/doc/install/configure-fs-storage.html.textile.liquid
new file mode 100644
index 000000000..730eed665
--- /dev/null
+++ b/doc/install/configure-fs-storage.html.textile.liquid
@@ -0,0 +1,56 @@
+---
+layout: default
+navsection: installguide
+title: Filesystem storage
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+Keepstore can store data in local and network-attached POSIX filesystems.
+
+h2. Setting up filesystem mounts
+
+Volumes are configured in the @Volumes@ section of the configuration file. You may provide multiple volumes for a single keepstore process to manage multiple disks. Keepstore distributes blocks among volumes in round-robin fashion.
+
+<pre>
+Volumes:
+- # The volume type, indicates this is a filesystem directory.
+ Type: Directory
+
+ # The directory that will be used as the backing store.
+ Root: /mnt/local-disk
+
+ # How much replication is performed by the underlying filesystem.
+ # (for example, a network filesystem may provide its own replication).
+ # This is used to inform replication decisions at the Keep layer.
+ DirectoryReplication: 1
+
+ # If true, do not accept write or trash operations, only reads.
+ ReadOnly: false
+
+ # When true, read and write operations (for whole 64MiB blocks) on
+ # an individual volume will queued and issued serially. When
+ # false, read and write operations will be issued concurrently.
+ #
+ # May improve throughput if you experience contention when there are
+ # multiple requests to the same volume.
+ #
+ # When using SSDs, RAID, or a parallel network filesystem, you probably
+ # don't want this.
+ Serialize: false
+
+ # Storage classes to associate with this volume. See "Configuring
+ # storage classes" in the "Admin" section of doc.arvados.org.
+ StorageClasses: null
+
+ # Example of a second volume section
+- DirectoryReplication: 2
+ ReadOnly: false
+ Root: /mnt/network-disk
+ Serialize: false
+ StorageClasses: null
+ Type: Directory
+</pre>
diff --git a/doc/install/configure-s3-object-storage.html.textile.liquid b/doc/install/configure-s3-object-storage.html.textile.liquid
index 33195f3ea..b4d9e929c 100644
--- a/doc/install/configure-s3-object-storage.html.textile.liquid
+++ b/doc/install/configure-s3-object-storage.html.textile.liquid
@@ -9,7 +9,7 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-As an alternative to local and network-attached POSIX filesystems, Keepstore can store data in object storage compatible with the S3 API, such as Amazon S3, Google Cloud Storage, or Ceph RADOS.
+Keepstore can store data in object storage compatible with the S3 API, such as Amazon S3, Google Cloud Storage, or Ceph RADOS.
h2. Configure keepstore
@@ -23,14 +23,18 @@ EOF</span>
</code></pre>
</notextile>
-Next, edit the @Volumes@ section of the @keepstore.yml@ config file:
+Next, edit the @Volumes@ section of the @keepstore.yml@ config file.
+
+h3. Example config for Amazon S3
<pre>
Volumes:
- # The volume type, this indicates object storage compatible with the S3 API
Type: S3
- # Storage provider (blank uses Amazon S3 by default)
+ # Storage provider. If blank, uses Amazon S3 by default.
+ # See below for example alternate configuration for Google cloud
+ # storage.
Endpoint: ""
# The bucket to use for the backing store.
@@ -61,7 +65,7 @@ Volumes:
# Maximum time to wait for a complete response from the backend before
# failing the request.
- ReadTimeout: 5m0s
+ ReadTimeout: 2m0s
# How much replication is performed by the underlying bucket.
# This is used to inform replication decisions at the Keep layer.
@@ -76,7 +80,16 @@ Volumes:
# that can cause data loss. Do not enable this unless you know what
# you are doing.
UnsafeDelete: false
+</pre>
+
+Start (or restart) keepstore, and check its log file to confirm it is using the new configuration.
+h3. Example config for Google cloud storage
+
+See previous section for documentation of configuration fields.
+
+</pre>
+Volumes:
- # Example configuration using alternate storage provider
# Configuration for Google cloud storage
Endpoint: https://storage.googleapis.com
@@ -90,7 +103,7 @@ Volumes:
LocationConstraint: false
RaceWindow: 24h0m0s
ReadOnly: false
- ReadTimeout: 5m0s
+ ReadTimeout: 2m0s
S3Replication: 2
StorageClasses: null
UnsafeDelete: false
diff --git a/doc/install/install-keepstore.html.textile.liquid b/doc/install/install-keepstore.html.textile.liquid
index 61553f73e..402d15cb5 100644
--- a/doc/install/install-keepstore.html.textile.liquid
+++ b/doc/install/install-keepstore.html.textile.liquid
@@ -116,8 +116,9 @@ MaxRequests: 0
PIDFile: ""
# Maximum number of concurrent pull operations. Default is 1, i.e.,
-# pull lists are processed serially.
-PullWorkers: 0
+# pull lists are processed serially. A pull operation copies a block
+# from another keepstore server.
+PullWorkers: 1
# Honor read requests only if a valid signature is provided. This
# should be true, except for development use and when migrating from
@@ -150,79 +151,32 @@ TrashCheckInterval: 24h0m0s
# recovered using an /untrash request.
TrashLifetime: 336h0m0s
-# Maximum number of concurrent trash operations. Default is 1, i.e.,
-# trash lists are processed serially.
+# Maximum number of concurrent trash operations (moving a block to the
+# trash, or permanently deleting it) . Default is 1, i.e., trash lists
+# are processed serially. If individual trash operations have high
+# latency (eg some cloud platforms) you should increase this.
TrashWorkers: 1
</pre>
h3. Notes on storage management
-On its own, a keepstore server never deletes data. The "keep-balance":install-keep-balance.html service service determines which blocks are candidates for deletion and instructs the keepstore to move those blocks to the trash.
+On its own, a keepstore server never deletes data. The "keep-balance":install-keep-balance.html service determines which blocks are candidates for deletion and instructs the keepstore to move those blocks to the trash.
When a block is newly written, it is protected from deletion for the duration in @BlobSignatureTTL at . During this time, it cannot be trashed.
-If keep-balance instructs keepstore to trash a block which is older than @BlobSignatureTTL@, and @EnableDelete@ is true, the block will be moved to "trash". A block which is in the trash has been moved out of the way and is no longer accessible by read requests, but has not yet been permanently deleted. Blocks which are in the trash may be recovered using the "untrash" API endpoint. Blocks are permanently deleted after they have been in the trash for the duration in @TrashLifetime at .
+If keep-balance instructs keepstore to trash a block which is older than @BlobSignatureTTL@, and @EnableDelete@ is true, the block will be moved to "trash". A block which is in the trash is no longer accessible by read requests, but has not yet been permanently deleted. Blocks which are in the trash may be recovered using the "untrash" API endpoint. Blocks are permanently deleted after they have been in the trash for the duration in @TrashLifetime at .
-h3. Configure storage volumes
-
-Available storage volume types include cloud object storage and POSIX filesystems.
-
-If you are using S3-compatible object storage (including Amazon S3, Google Cloud Storage, and Ceph RADOS), follow the setup instructions "S3 Object Storage":configure-s3-object-storage.html page instead and then "Run keepstore as a supervised service.":#keepstoreservice
-
-If you are using Azure Blob Storage, follow the setup instructions "Azure Blob Storage":configure-azure-blob-storage.html and then proceed to "Run keepstore as a supervised service.":#keepstoreservice
+Keep-balance is also responsible for balancing the distribution of blocks across keepstore servers by asking servers to pull blocks from other servers (as determined by their storage class and "rendezvous hashing order":{{site.baseurl}}/api/storage.html). Pulling a block makes a copy. If a block is overreplicated (i.e. there are excess copies) after pulling, it will be subsequently trashed on the original server.
-To use a POSIX filesystem, including both local filesystems (ext4, xfs) and network file system such as GPFS or Lustre, continue reading this section.
-
-h4. Setting up filesystem mounts
+h3. Configure storage volumes
-Volumes are configured in the @Volumes@ section of the configuration
-file. You may provide multiple volumes for a single keepstore process
-to manage multiple disks. Keepstore distributes blocks among volumes
-in round-robin fashion.
+Available storage volume types include POSIX filesystems and cloud object storage.
-<pre>
-Volumes:
-- # The volume type, indicates this is a filesystem directory.
- Type: Directory
-
- # The actual directory that will be used as the backing store.
- Root: /mnt/local-disk
-
- # How much replication is performed by the underlying filesystem.
- # (for example, a network filesystem may provide its own replication).
- # This is used to inform replication decisions at the Keep layer.
- DirectoryReplication: 1
-
- # If true, do not accept write or trash operations, only reads.
- ReadOnly: false
-
- # When true, read and write operations (for whole 64MiB blocks) on
- # an individual volume will queued and issued sequentially. When
- # false, read and write operations will be issued concurrently as
- # they come in.
- #
- # When using spinning disks where storage partitions map 1:1 to
- # physical disks that are dedicated to Keepstore, enabling this may
- # reduce contention and improve throughput by minimizing seeks.
- #
- # When using SSDs, RAID, or a parallel network filesystem, you probably
- # don't want this.
- Serialize: true
-
- # Storage classes to associate with this volume. See "Configuring
- # storage classes" in the "Admin" section of doc.arvados.org.
- StorageClasses: null
-
- # Example of a second volume section
-- DirectoryReplication: 2
- ReadOnly: false
- Root: /mnt/network-disk
- Serialize: false
- StorageClasses: null
- Type: Directory
-</pre>
+* To use a POSIX filesystem, including both local filesystems (ext4, xfs) and network file system such as GPFS or Lustre, follow the setup instructions on "Filesystem storage":configure-fs-storage.html
+* If you are using S3-compatible object storage (including Amazon S3, Google Cloud Storage, and Ceph RADOS), follow the setup instructions on "S3 Object Storage":configure-s3-object-storage.html
+* If you are using Azure Blob Storage, follow the setup instructions on "Azure Blob Storage":configure-azure-blob-storage.html
-h3(#keepstoreservice). Run keepstore as a supervised service
+h3. Run keepstore as a supervised service
Install runit to supervise the keepstore daemon. {% include 'install_runit' %}
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list