[ARVADOS] updated: 1.3.0-1601-g08d2f48c3
Git user
git at public.curoverse.com
Wed Sep 11 18:43:35 UTC 2019
Summary of changes:
doc/_includes/_assign_volume_uuid.liquid | 2 +-
...onfigure-azure-blob-storage.html.textile.liquid | 149 +++++++++--------
...configure-s3-object-storage.html.textile.liquid | 176 +++++++++------------
3 files changed, 160 insertions(+), 167 deletions(-)
via 08d2f48c3cf3eac9d71261172677c54f03669fe5 (commit)
via 04dd8d57e5ef90dd68876ffe5c72003645c6e3a2 (commit)
from c9a3f26fb35a6903178c3dbbc4dc3decb0ba2cf0 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 08d2f48c3cf3eac9d71261172677c54f03669fe5
Author: Tom Clegg <tclegg at veritasgenetics.com>
Date: Wed Sep 11 14:42:42 2019 -0400
13647: Update Azure volume docs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg at veritasgenetics.com>
diff --git a/doc/_includes/_assign_volume_uuid.liquid b/doc/_includes/_assign_volume_uuid.liquid
index 999d7d664..cdd0f1a6d 100644
--- a/doc/_includes/_assign_volume_uuid.liquid
+++ b/doc/_includes/_assign_volume_uuid.liquid
@@ -4,4 +4,4 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Note that each volume has a UUID, like "zzzzz-nyw5e-0123456789abcde". You assign these manually: replace "zzzzz" with your cluster ID, and replace "0123456789abcde" with an arbitrary string of alphanumerics. Once assigned, they should not be changed.
+Note that each volume has a UUID, like @zzzzz-nyw5e-0123456789abcde at . You assign these manually: replace @zzzzz@ with your cluster ID, and replace @0123456789abcde@ with an arbitrary string of 15 alphanumerics. Once assigned, UUIDs should not be changed.
diff --git a/doc/install/configure-azure-blob-storage.html.textile.liquid b/doc/install/configure-azure-blob-storage.html.textile.liquid
index 8a0e7bfa0..174cb6a93 100644
--- a/doc/install/configure-azure-blob-storage.html.textile.liquid
+++ b/doc/install/configure-azure-blob-storage.html.textile.liquid
@@ -17,9 +17,9 @@ Before starting the configuration of individual keepstore servers is good to hav
Another decision is how many VMs should be running keepstore. For example there could be 8 VMs with one core each or one machine with 8 cores. Or anything in between. Assuming is the same cost for Cloud resources, there is always the benefit of distributing the risk of faulty VMs. The recommendation is to start with 2 VMs and expand in pairs. Having a minimum of 2 cores each. The total amount of VMs will be a function of the budget and the pipeline traffic to avoid saturation during periods of high usage. Standard D v3 family is a balanced choice, making Standard_D2_v3 the 2-core option
-There are many options for storage accounts. You can read details from Azure on their documentation https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction. The type of storage and access tier will be a function of the budget and desired responsiveness. A balanced option is to have General-purpose Standard Storage account and use Blob storage, hot access tiers.
+There are many options for storage accounts. You can read details from Azure on their documentation "https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction":https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction. The type of storage and access tier will be a function of the budget and desired responsiveness. A balanced option is to have General-purpose Standard Storage account and use Blob storage, hot access tiers.
-Keepstore can be configure to reflect the level of underlaying redundancy the storage will have. This is call data replication option. For example LRS (Locally Redundant Storage) saves 3 copies of the data. There desired redundancy can be chosen at the keepstore layer or at the Storage Accunt layer. The decision where the redundancy will be done and the type of Storage Account data replication (LRS, ZRS, GRS and RA-GRS) has trade-offs. Please read more on https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy and decide what is best for your needs.
+Keepstore can be configure to reflect the level of underlaying redundancy the storage will have. This is call data replication option. For example LRS (Locally Redundant Storage) saves 3 copies of the data. There desired redundancy can be chosen at the keepstore layer or at the Storage Accunt layer. The decision where the redundancy will be done and the type of Storage Account data replication (LRS, ZRS, GRS and RA-GRS) has trade-offs. Please read more on "https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy":https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy and decide what is best for your needs.
h2. Create a storage container
@@ -46,68 +46,83 @@ Note that Keepstore services may be configued to use multiple Azure Storage acco
h2. Configure keepstore
-Copy the primary storage account key to a file where it will be accessible to keepstore at startup time.
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo sh -c 'cat >/etc/arvados/keepstore/azure_storage_account_key.txt <<EOF'
-zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz==
-EOF</span>
-~$ <span class="userinput">sudo chmod 0400 /etc/arvados/keepstore/azure_storage_account_key.txt</span>
-</code></pre>
-</notextile>
-
-Next, edit the @Volumes@ section of the @keepstore.yml@ config file:
-
-<pre>
-Volumes:
-- # The volume type, this indicates Azure blob storage
- Type: Azure
-
- # How much replication is performed by the underlying container.
- # This is used to inform replication decisions at the Keep layer.
- AzureReplication: 3
-
- # The storage container to use for the backing store.
- ContainerName: exampleContainerName
-
- # If true, do not accept write or trash operations, only reads.
- ReadOnly: false
-
- # Amount of time to wait for a response before failing the request
- RequestTimeout: 2m0s
-
- # The storage account name, used for authentication
- StorageAccountName: exampleStorageAccountName
-
- # The storage account secret key, used for authentication
- StorageAccountKeyFile: /etc/arvados/keepstore/azure_storage_account_key.txt
-
- # The cloud environment to use. If blank, use the default cloud
- # environment. See below for an example of an alternate cloud environment.
- StorageBaseURL: ""
-
- # Storage classes to associate with this volume. See "Storage
- # classes" in the "Admin" section of doc.arvados.org.
- StorageClasses: null
-
-- # Example configuration to use Azure China.
- #
- # The alternate cloud environment to use.
- # Note that cloud environments are different from regions. A
- # cloud environment is an entirely separate instance of Azure with
- # separate accounts, requiring separate credentials.
- #
- StorageBaseURL: core.chinacloudapi.cn
- StorageAccountKeyFile: /etc/arvados/keepstore/azure_cn_storage_account_key.txt
- StorageAccountName: cn-account-name
- ContainerName: exampleChinaContainerName
-
- # The rest are the same as above
- Type: Azure
- AzureReplication: 3
- ReadOnly: false
- RequestTimeout: 10m0s
- StorageClasses: null
-</pre>
-
-Start (or restart) keepstore, and check its log file to confirm it is using the new configuration.
+Volumes are configured in the @Volumes@ section of the cluster configuration file.
+
+{% include 'assign_volume_uuid' %}
+
+<notextile><pre><code>Clusters:
+ <span class="userinput">uuid_prefix</span>:
+ Volumes:
+ <span class="userinput">uuid_prefix</span>-nyw5e-<span class="userinput">000000000000000</span>:
+ AccessViaHosts:
+ # This section determines which keepstore servers access the
+ # volume. In this example, keep0 has read/write access, and
+ # keep1 has read-only access.
+ #
+ # If the AccessViaHosts section is empty, all keepstore
+ # servers will have read/write access to the volume.
+ "http://keep0.zzzzz.example.com:25107/": {}
+ "http://keep1.zzzzz.example.com:25107/": {ReadOnly: true}
+
+ Driver: Azure
+ DriverParameters:
+ # Storage account name and secret key, used for
+ # authentication.
+ StorageAccountName: exampleStorageAccountName
+ StorageAccountKey: zzzzzzzzzzzzzzzzzzzzzzzzzz
+
+ # The cloud environment to use,
+ # e.g. "core.chinacloudapi.cn". Defaults to
+ # "core.windows.net" if blank or omitted.
+ StorageBaseURL: ""
+
+ # Storage container name.
+ ContainerName: exampleContainerName
+
+ # Time to wait for an upstream response before failing the
+ # request.
+ RequestTimeout: 10m
+
+ # Time to wait before retrying a failed "list blobs" Azure
+ # API call.
+ ListBlobsRetryDelay: 10s
+
+ # Maximum attempts at a "list blobs" Azure API call before
+ # giving up.
+ ListBlobsMaxAttempts: 12
+
+ # If non-zero, use multiple concurrent requests (each
+ # requesting MaxGetBytes bytes) when retrieving data. If
+ # zero or omitted, get the entire blob with one request.
+ #
+ # Normally this is zero but if you find that 4 small
+ # requests complete faster than a single large request, for
+ # example, you might set this to 16777216 (64 MiB รท 4).
+ MaxGetBytes: 0
+
+ # Time to wait for an unexpectedly empty blob to become
+ # non-empty. Azure's create-and-write operation is not
+ # atomic. The default value typically allows concurrent GET
+ # and PUT requests to succeed despite the race window.
+ WriteRaceInterval: 15s
+
+ # Time to wait between GET attempts while waiting for
+ # WriteRaceInterval to expire.
+ WriteRacePollTime: 1s
+
+ # How much replication is provided by the underlying storage
+ # container. This is used to inform replication decisions at
+ # the Keep layer.
+ Replication: 3
+
+ # If true, do not accept write or trash operations, even if
+ # AccessViaHosts.*.ReadOnly is false.
+ #
+ # If false or omitted, enable write access (subject to
+ # AccessViaHosts.*.ReadOnly, where applicable).
+ ReadOnly: false
+
+ # Storage classes to associate with this volume. See "Storage
+ # classes" in the "Admin" section of doc.arvados.org.
+ StorageClasses: null
+</code></pre></notextile>
commit 04dd8d57e5ef90dd68876ffe5c72003645c6e3a2
Author: Tom Clegg <tclegg at veritasgenetics.com>
Date: Wed Sep 11 11:53:01 2019 -0400
13647: Update S3 volume config docs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg at veritasgenetics.com>
diff --git a/doc/install/configure-s3-object-storage.html.textile.liquid b/doc/install/configure-s3-object-storage.html.textile.liquid
index 88172fa9f..9becc9a58 100644
--- a/doc/install/configure-s3-object-storage.html.textile.liquid
+++ b/doc/install/configure-s3-object-storage.html.textile.liquid
@@ -11,102 +11,80 @@ SPDX-License-Identifier: CC-BY-SA-3.0
Keepstore can store data in object storage compatible with the S3 API, such as Amazon S3, Google Cloud Storage, or Ceph RADOS.
-h2. Configure keepstore
-
-Copy the "access key" and "secret key" to files where they will be accessible to keepstore at startup time.
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo sh -c 'cat >/etc/arvados/keepstore/aws_s3_access_key.txt <<EOF'
-zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz==
-EOF</span>
-~$ <span class="userinput">sudo chmod 0400 /etc/arvados/keepstore/aws_s3_access_key.txt</span>
-</code></pre>
-</notextile>
-
-Next, edit the @Volumes@ section of the @keepstore.yml@ config file.
-
-h3. Example config for Amazon S3
-
-<pre>
-Volumes:
-- # The volume type, this indicates object storage compatible with the S3 API
- Type: S3
-
- # Storage provider. If blank, uses Amazon S3 by default.
- # See below for example alternate configuration for Google cloud
- # storage.
- Endpoint: ""
-
- # The bucket to use for the backing store.
- Bucket: example-bucket-name
-
- # The region where the bucket is located.
- Region: us-east-1
-
- # The credentials to use to access the bucket.
- AccessKeyFile: /etc/arvados/keepstore/aws_s3_access_key.txt
- SecretKeyFile: /etc/arvados/keepstore/aws_s3_secret_key.txt
-
- # Maximum time to wait making the initial connection to the backend before
- # failing the request.
- ConnectTimeout: 1m0s
-
- # Page size for s3 "list bucket contents" requests
- IndexPageSize: 1000
-
- # True if the region requires a LocationConstraint declaration
- LocationConstraint: false
-
- # Maximum eventual consistency latency
- RaceWindow: 24h0m0s
-
- # If true, do not accept write or trash operations, only reads.
- ReadOnly: false
-
- # Maximum time to wait for a complete response from the backend before
- # failing the request.
- ReadTimeout: 2m0s
-
- # How much replication is performed by the underlying bucket.
- # This is used to inform replication decisions at the Keep layer.
- S3Replication: 2
-
- # Storage classes to associate with this volume. See
- # "Storage classes" in the "Admin" section of doc.arvados.org.
- StorageClasses: null
-
- # Enable deletion (garbage collection) even when TrashLifetime is
- # zero. WARNING: eventual consistency may result in race conditions
- # that can cause data loss. Do not enable this unless you know what
- # you are doing.
- UnsafeDelete: false
-</pre>
-
-Start (or restart) keepstore, and check its log file to confirm it is using the new configuration.
-
-h3. Example config for Google cloud storage
-
-See previous section for documentation of configuration fields.
-
-<pre>
-Volumes:
-- # Example configuration using alternate storage provider
- # Configuration for Google cloud storage
- Endpoint: https://storage.googleapis.com
- Region: ""
-
- AccessKeyFile: /etc/arvados/keepstore/gce_s3_access_key.txt
- SecretKeyFile: /etc/arvados/keepstore/gce_s3_secret_key.txt
- Bucket: example-bucket-name
- ConnectTimeout: 1m0s
- IndexPageSize: 1000
- LocationConstraint: false
- RaceWindow: 24h0m0s
- ReadOnly: false
- ReadTimeout: 2m0s
- S3Replication: 2
- StorageClasses: null
- UnsafeDelete: false
-</pre>
-
-Start (or restart) keepstore, and check its log file to confirm it is using the new configuration.
+Volumes are configured in the @Volumes@ section of the cluster configuration file.
+
+{% include 'assign_volume_uuid' %}
+
+<notextile><pre><code>Clusters:
+ <span class="userinput">uuid_prefix</span>:
+ Volumes:
+ <span class="userinput">uuid_prefix</span>-nyw5e-<span class="userinput">000000000000000</span>:
+ AccessViaHosts:
+ # This section determines which keepstore servers access the
+ # volume. In this example, keep0 has read/write access, and
+ # keep1 has read-only access.
+ #
+ # If the AccessViaHosts section is empty, all keepstore
+ # servers will have read/write access to the volume.
+ "http://keep0.zzzzz.example.com:25107/": {}
+ "http://keep1.zzzzz.example.com:25107/": {ReadOnly: true}
+
+ Driver: S3
+ DriverParameters:
+ # The credentials to use to access the bucket.
+ AccessKey: aaaaa
+ SecretKey: aaaaa
+
+ # Storage provider endpoint. For Amazon S3, use "" or
+ # omit. For Google Cloud Storage, use
+ # "https://storage.googleapis.com".
+ Endpoint: ""
+
+ # Storage provider region. For Google Cloud Storage, use ""
+ # or omit.
+ Region: us-east-1a
+
+ # Change to true if the region requires a LocationConstraint
+ # declaration.
+ LocationConstraint: false
+
+ # Bucket name.
+ Bucket: example-bucket-name
+
+ # Requested page size for "list bucket contents" requests.
+ IndexPageSize: 1000
+
+ # Maximum time to wait while making the initial connection
+ # to the backend before failing the request.
+ ConnectTimeout: 1m
+
+ # Maximum time to wait for a complete response from the
+ # backend before failing the request.
+ ReadTimeout: 2m
+
+ # Maximum eventual consistency latency
+ RaceWindow: 24h
+
+ # Enable deletion (garbage collection) even when the
+ # configured BlobTrashLifetime is zero. WARNING: eventual
+ # consistency may result in race conditions that can cause
+ # data loss. Do not enable this unless you understand and
+ # accept the risk.
+ UnsafeDelete: false
+
+ # How much replication is provided by the underlying bucket.
+ # This is used to inform replication decisions at the Keep
+ # layer.
+ Replication: 2
+
+ # If true, do not accept write or trash operations, even if
+ # AccessViaHosts.*.ReadOnly is false.
+ #
+ # If false or omitted, enable write access (subject to
+ # AccessViaHosts.*.ReadOnly, where applicable).
+ ReadOnly: false
+
+ # Storage classes to associate with this volume. See "Storage
+ # classes" in the "Admin" section of doc.arvados.org.
+ StorageClasses: null
+</code></pre></notextile>
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list