[ARVADOS] updated: 2.1.0-1963-g1d2f12862

Git user git at public.arvados.org
Mon Feb 21 14:32:42 UTC 2022


Summary of changes:
 .../install-compute-node.html.textile.liquid       | 59 +++++++++++++---------
 tools/compute-images/build.sh                      | 44 ++++++++--------
 2 files changed, 55 insertions(+), 48 deletions(-)

       via  1d2f128625a110a42a7162e70509ea089461ec69 (commit)
       via  485956a33887efb4ff3c7b2714a2a195e28c7070 (commit)
      from  e32eba7538014c707c2516cfd86c55230bba3325 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.


commit 1d2f128625a110a42a7162e70509ea089461ec69
Author: Ward Vandewege <ward at curii.com>
Date:   Mon Feb 21 09:32:08 2022 -0500

    18772: document the AWS EBS autoscaler support. Refactor the "Build a
           cloud compute node image" documentation page to improve the flow.
    
    Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>

diff --git a/doc/install/crunch2-cloud/install-compute-node.html.textile.liquid b/doc/install/crunch2-cloud/install-compute-node.html.textile.liquid
index 89771514e..3bfb818a8 100644
--- a/doc/install/crunch2-cloud/install-compute-node.html.textile.liquid
+++ b/doc/install/crunch2-cloud/install-compute-node.html.textile.liquid
@@ -17,8 +17,11 @@ SPDX-License-Identifier: CC-BY-SA-3.0
 # "Create an SSH keypair":#sshkeypair
 # "Compute image requirements":#requirements
 # "The build script":#building
+# "DNS resolution":#dns-resolution
+# "NVIDIA GPU support":#nvidia
 # "Singularity mksquashfs configuration":#singularity_mksquashfs_configuration
 # "Build an AWS image":#aws
+## "Autoscaling compute node scratch space":#aws-ebs-autoscaler
 # "Build an Azure image":#azure
 
 h2(#introduction). Introduction
@@ -56,12 +59,6 @@ foktmqOY8MyctzFgXBpGTxPliGjqo8OkrOyQP2g+FL7v+Km31Xs61P8=
 </code></pre>
 </notextile>
 
-{% assign show_docker_warning = true %}
-
-{% include 'singularity_mksquashfs_configuration' %}
-
-The desired amount of memory to make available for @mksquashfs@ can be configured in an argument to "the build script":#building. It defaults to @256M at .
-
 h2(#requirements). Compute image requirements
 
 Arvados comes with a build script to automate the creation of a suitable compute node image (see "The build script":#building below). It is provided as a convenience. It is also possible to create a compute node image via other means. These are the requirements:
@@ -101,6 +98,8 @@ Options:
       VPC id for AWS, otherwise packer will pick the default one
   --aws-subnet-id
       Subnet id for AWS otherwise packer will pick the default one for the VPC
+  --aws-ebs-autoscale (default: false)
+      Install the AWS EBS autoscaler daemon.
   --gcp-project-id (default: false, required if building for GCP)
       GCP project id
   --gcp-account-file (default: false, required if building for GCP)
@@ -131,10 +130,29 @@ Options:
       Output debug information
 </code></pre></notextile>
 
-h2(#building). NVIDIA GPU support
+h2(#dns-resolution). DNS resolution
+
+Compute nodes must be able to resolve the hostnames of the API server and any keepstore servers to your internal IP addresses. You can do this by running an internal DNS resolver. The IP address of the resolver should be passed as the value for the @--resolver@ argument to "the build script":#building.
+
+Alternatively, the services could be hardcoded into an @/etc/hosts@ file. For example:
+
+<notextile><pre><code>10.20.30.40     <span class="userinput">ClusterID.example.com</span>
+10.20.30.41     <span class="userinput">keep1.ClusterID.example.com</span>
+10.20.30.42     <span class="userinput">keep2.ClusterID.example.com</span>
+</code></pre></notextile>
+
+Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.
+
+h2(#nvidia). NVIDIA GPU support
 
 If you plan on using instance types with NVIDIA GPUs, add @--nvidia-gpu-support@ to the build command line.  Arvados uses the same compute image for both GPU and non-GPU instance types.  The GPU tooling is ignored when using the image with a non-GPU instance type.
 
+{% assign show_docker_warning = true %}
+
+{% include 'singularity_mksquashfs_configuration' %}
+
+The desired amount of memory to make available for @mksquashfs@ can be configured in an argument to "the build script":#building. It defaults to @256M at .
+
 h2(#aws). Build an AWS image
 
 <notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-aws.json \
@@ -155,17 +173,19 @@ For @ClusterID@, fill in your cluster ID. The @VPC@ and @Subnet@ should be confi
 
 @ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
 
-Compute nodes must be able to resolve the hostnames of the API server and any keepstore servers to your internal IP addresses. You can do this by running an internal DNS resolver. The IP address of the resolver should replace the string @ResolverIP@ in the command above.
+h3(#aws-ebs-autoscaler). Autoscaling compute node scratch space
 
-Alternatively, the services could be hardcoded into an @/etc/hosts@ file. For example:
+If you want to add the AWS EBS autoscaler daemon in your images, add the @--aws-ebs-autoscale@ flag to the "the build script":#building. Doing so will make the compute image scratch space scale automatically as needed. The @/InstanceTypes@ list should be modified so that all @AddedScratch@ lines are removed, and the @IncludedScratch@ value should be set to a (fictional) high number. This way, the scratch space requirements will be met by all the defined instance type. For example:
 
-<notextile><pre><code>10.20.30.40     <span class="userinput">ClusterID.example.com</span>
-10.20.30.41     <span class="userinput">keep1.ClusterID.example.com</span>
-10.20.30.42     <span class="userinput">keep2.ClusterID.example.com</span>
+<notextile><pre><code>
+c5large:
+        ProviderType: c5.large
+        VCPUs: 2
+        RAM: 4GiB
+        IncludedScratch: 16TB
+        Price: 0.085
 </code></pre></notextile>
 
-Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.
-
 h2(#azure). Build an Azure image
 
 <notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-azure.json \
@@ -195,14 +215,3 @@ These secrets can be generated from the Azure portal, or with the cli using a co
 </code></pre></notextile>
 
 @ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
-
-Compute nodes must be able to resolve the hostnames of the API server and any keepstore servers to your internal IP addresses. You can do this by running an internal DNS resolver. The IP address of the resolver should replace the string @ResolverIP@ in the command above.
-
-Alternatively, the services could be hardcoded into an @/etc/hosts@ file. For example:
-
-<notextile><pre><code>10.20.30.40     <span class="userinput">ClusterID.example.com</span>
-10.20.30.41     <span class="userinput">keep1.ClusterID.example.com</span>
-10.20.30.42     <span class="userinput">keep2.ClusterID.example.com</span>
-</code></pre></notextile>
-
-Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.

commit 485956a33887efb4ff3c7b2714a2a195e28c7070
Author: Ward Vandewege <ward at curii.com>
Date:   Mon Feb 21 09:08:30 2022 -0500

    18772: consistently use bash comparison operators in build.sh
    
    Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>

diff --git a/tools/compute-images/build.sh b/tools/compute-images/build.sh
index ea7676db1..769b9a5b5 100755
--- a/tools/compute-images/build.sh
+++ b/tools/compute-images/build.sh
@@ -193,7 +193,7 @@ while [ $# -gt 0 ]; do
 done
 
 
-if [[ "$JSON_FILE" == "" ]] || [[ ! -f "$JSON_FILE" ]]; then
+if [[ -z "$JSON_FILE" ]] || [[ ! -f "$JSON_FILE" ]]; then
   echo >&2 "$helpmessage"
   echo >&2
   echo >&2 "ERROR: packer json file not found"
@@ -209,7 +209,7 @@ if [[ -z "$ARVADOS_CLUSTER_ID" ]]; then
   exit 1
 fi
 
-if [[ "$PUBLIC_KEY_FILE" == "" ]] || [[ ! -f "$PUBLIC_KEY_FILE" ]]; then
+if [[ -z "$PUBLIC_KEY_FILE" ]] || [[ ! -f "$PUBLIC_KEY_FILE" ]]; then
   echo >&2 "$helpmessage"
   echo >&2
   echo >&2 "ERROR: public key file file not found"
@@ -228,66 +228,64 @@ fi
 
 EXTRA2=""
 
-if [[ "$AWS_SOURCE_AMI" != "" ]]; then
+if [[ -n "$AWS_SOURCE_AMI" ]]; then
   EXTRA2+=" -var aws_source_ami=$AWS_SOURCE_AMI"
 fi
-if [[ "$AWS_PROFILE" != "" ]]; then
+if [[ -n "$AWS_PROFILE" ]]; then
   EXTRA2+=" -var aws_profile=$AWS_PROFILE"
 fi
-if [[ "$AWS_VPC_ID" != "" ]]; then
+if [[ -n "$AWS_VPC_ID" ]]; then
   EXTRA2+=" -var vpc_id=$AWS_VPC_ID -var associate_public_ip_address=true "
 fi
-if [[ "$AWS_SUBNET_ID" != "" ]]; then
+if [[ -n "$AWS_SUBNET_ID" ]]; then
   EXTRA2+=" -var subnet_id=$AWS_SUBNET_ID -var associate_public_ip_address=true "
 fi
-if [[ "$AWS_DEFAULT_REGION" != "" ]]; then
+if [[ -n "$AWS_DEFAULT_REGION" ]]; then
   EXTRA2+=" -var aws_default_region=$AWS_DEFAULT_REGION"
 fi
-if [[ "$AWS_EBS_AUTOSCALE" != "" ]]; then
+if [[ -n "$AWS_EBS_AUTOSCALE" ]]; then
   EXTRA2+=" -var aws_ebs_autoscale=$AWS_EBS_AUTOSCALE"
 fi
-if [[ "$GCP_PROJECT_ID" != "" ]]; then
+if [[ -n "$GCP_PROJECT_ID" ]]; then
   EXTRA2+=" -var project_id=$GCP_PROJECT_ID"
 fi
-if [[ "$GCP_ACCOUNT_FILE" != "" ]]; then
+if [[ -n "$GCP_ACCOUNT_FILE" ]]; then
   EXTRA2+=" -var account_file=$GCP_ACCOUNT_FILE"
 fi
-if [[ "$GCP_ZONE" != "" ]]; then
+if [[ -n "$GCP_ZONE" ]]; then
   EXTRA2+=" -var zone=$GCP_ZONE"
 fi
-if [[ "$AZURE_RESOURCE_GROUP" != "" ]]; then
+if [[ -n "$AZURE_RESOURCE_GROUP" ]]; then
   EXTRA2+=" -var resource_group=$AZURE_RESOURCE_GROUP"
 fi
-if [[ "$AZURE_LOCATION" != "" ]]; then
+if [[ -n "$AZURE_LOCATION" ]]; then
   EXTRA2+=" -var location=$AZURE_LOCATION"
 fi
-if [[ "$AZURE_SKU" != "" ]]; then
+if [[ -n "$AZURE_SKU" ]]; then
   EXTRA2+=" -var image_sku=$AZURE_SKU"
 fi
-if [[ "$AZURE_CLOUD_ENVIRONMENT" != "" ]]; then
+if [[ -n "$AZURE_CLOUD_ENVIRONMENT" ]]; then
   EXTRA2+=" -var cloud_environment_name=$AZURE_CLOUD_ENVIRONMENT"
 fi
-if [[ "$SSH_USER" != "" ]]; then
+if [[ -n "$SSH_USER" ]]; then
   EXTRA2+=" -var ssh_user=$SSH_USER"
 fi
-if [[ "$RESOLVER" != "" ]]; then
+if [[ -n "$RESOLVER" ]]; then
   EXTRA2+=" -var resolver=$RESOLVER"
 fi
-if [[ "$REPOSUFFIX" != "" ]]; then
+if [[ -n "$REPOSUFFIX" ]]; then
   EXTRA2+=" -var reposuffix=$REPOSUFFIX"
 fi
-if [[ "$PUBLIC_KEY_FILE" != "" ]]; then
+if [[ -n "$PUBLIC_KEY_FILE" ]]; then
   EXTRA2+=" -var public_key_file=$PUBLIC_KEY_FILE"
 fi
-if [[ "$MKSQUASHFS_MEM" != "" ]]; then
+if [[ -n "$MKSQUASHFS_MEM" ]]; then
   EXTRA2+=" -var mksquashfs_mem=$MKSQUASHFS_MEM"
 fi
-if [[ "$NVIDIA_GPU_SUPPORT" != "" ]]; then
+if [[ -n "$NVIDIA_GPU_SUPPORT" ]]; then
   EXTRA2+=" -var nvidia_gpu_support=$NVIDIA_GPU_SUPPORT"
 fi
 
-
-
 echo
 packer version
 echo

-----------------------------------------------------------------------


hooks/post-receive
-- 




More information about the arvados-commits mailing list