[ARVADOS] updated: 2.3.2-42-gbf81b8485
Git user
git at public.arvados.org
Tue Mar 1 02:39:57 UTC 2022
Summary of changes:
doc/_includes/_install_custom_certificates.liquid | 16 ++--
.../_multi_host_install_custom_certificates.liquid | 28 ++++++
doc/install/salt-multi-host.html.textile.liquid | 2 +-
doc/install/salt-single-host.html.textile.liquid | 106 +++++++++++++++------
...ocal.params.example.single_host_single_hostname | 22 +----
tools/salt-install/provision.sh | 11 ++-
6 files changed, 128 insertions(+), 57 deletions(-)
create mode 100644 doc/_includes/_multi_host_install_custom_certificates.liquid
via bf81b84851b7fe2271b46655d79e73d76311c37a (commit)
from 3a26616cec20d3cf9d3ac0d3f3930bf217ec7b64 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit bf81b84851b7fe2271b46655d79e73d76311c37a
Author: Ward Vandewege <ward at curii.com>
Date: Mon Feb 28 21:38:19 2022 -0500
18785: update documentation, fix run-tests.sh bug in single host/single
hostname + LE mode, change default workbench port to 443 in
single host.
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/_includes/_install_custom_certificates.liquid b/doc/_includes/_install_custom_certificates.liquid
index a6d809d15..80878c049 100644
--- a/doc/_includes/_install_custom_certificates.liquid
+++ b/doc/_includes/_install_custom_certificates.liquid
@@ -4,17 +4,19 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-If you plan to use custom certificates, please set the variable <i>SSL_MODE=bring-your-own</i> and copy your certificates to the directory specified with the variable @CUSTOM_CERTS_DIR@ (usually "./certs") in the remote directory where you copied the @provision.sh@ script. From this dir, the provision script will install the certificates required for the role you're installing.
+Copy your certificates to the directory specified with the variable @CUSTOM_CERTS_DIR@ in the remote directory where you copied the @provision.sh@ script. The provision script will find the certificates there.
+
+For a @single hostname@ setup, the certificate and its key need to be copied to a file named after @HOSTNAME_EXT at .
+
+For example, if @HOSTNAME_EXT@ is defined as @my-arvados.example.net@, the script will look for
-When using custom certificates in a single-host / single-hostname setup, the certificate and its key need to be copied to a file named after <i>${HOSTNAME_EXT}</i>. Ie., for "HOSTNAME_EXT='my-arvados.example.net', the script will lookup for
<notextile>
-<pre><code>
-# ${CUSTOM_CERTS_DIR}/my-arvados.example.net.crt
-# ${CUSTOM_CERTS_DIR}/my-arvados.example.net.key
+<pre><code>${CUSTOM_CERTS_DIR}/my-arvados.example.net.crt
+${CUSTOM_CERTS_DIR}/my-arvados.example.net.key
</code></pre>
</notextile>
-For a setup with multiple hostnames, the script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>download / collections</i>):
+For a @multiple hostnames@ setup, the script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>download / collections</i>):
* "controller"
* "websocket"
@@ -25,7 +27,7 @@ For a setup with multiple hostnames, the script expects cert/key files with thes
* "collections" # Part of keepweb
* "keepproxy"
-Ie., for 'keepproxy', the script will look for
+E.g. for 'keepproxy', the script will look for
<notextile>
<pre><code>${CUSTOM_CERTS_DIR}/keepproxy.crt
diff --git a/doc/_includes/_multi_host_install_custom_certificates.liquid b/doc/_includes/_multi_host_install_custom_certificates.liquid
new file mode 100644
index 000000000..b831aadcf
--- /dev/null
+++ b/doc/_includes/_multi_host_install_custom_certificates.liquid
@@ -0,0 +1,28 @@
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+Copy your certificates to the directory specified with the variable @CUSTOM_CERTS_DIR@ in the remote directory where you copied the @provision.sh@ script. The provision script will find the certificates there.
+
+The script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>download / collections</i>):
+
+* "controller"
+* "websocket"
+* "workbench"
+* "workbench2"
+* "webshell"
+* "download" # Part of keepweb
+* "collections" # Part of keepweb
+* "keepproxy"
+
+E.g. for 'keepproxy', the script will look for
+
+<notextile>
+<pre><code>${CUSTOM_CERTS_DIR}/keepproxy.crt
+${CUSTOM_CERTS_DIR}/keepproxy.key
+</code></pre>
+</notextile>
+
+Make sure that all the FQDNs that you will use for the public-facing applications (API/controller, Workbench, Keepproxy/Keepweb) are reachable.
diff --git a/doc/install/salt-multi-host.html.textile.liquid b/doc/install/salt-multi-host.html.textile.liquid
index f8723a0ce..6ef274a03 100644
--- a/doc/install/salt-multi-host.html.textile.liquid
+++ b/doc/install/salt-multi-host.html.textile.liquid
@@ -98,7 +98,7 @@ Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_IN
The <i>multi_host</i> example includes Let's Encrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53.
-{% include 'install_custom_certificates' %}
+{% include 'multi_host_install_custom_certificates' %}
If you want to use valid certificates provided by Let's Encrypt, set the variable <i>SSL_MODE=lets-encrypt</i> and make sure that all the FQDNs that you will use for the public-facing applications (API/controller, Workbench, Keepproxy/Keepweb) are reachable.
diff --git a/doc/install/salt-single-host.html.textile.liquid b/doc/install/salt-single-host.html.textile.liquid
index 2691332fd..6a0d17a31 100644
--- a/doc/install/salt-single-host.html.textile.liquid
+++ b/doc/install/salt-single-host.html.textile.liquid
@@ -15,7 +15,11 @@ SPDX-License-Identifier: CC-BY-SA-3.0
# "Choose the desired configuration":#choose_configuration
## "Single host / single hostname":#single_host_single_hostname
## "Single host / multiple hostnames (Alternative configuration)":#single_host_multiple_hostnames
-## "Further customization of the installation (modifying the salt pillars and states)":#further_customization
+# "Choose the SSL configuration (SSL_MODE)":#certificates
+## "Using self-signed certificates":#self-signed
+## "Using Let's Encrypt certificates":#lets-encrypt
+## "Using your own certificates":#bring-your-own
+# "Further customization of the installation (modifying the salt pillars and states)":#further_customization
# "Run the provision.sh script":#run_provision_script
# "Final configuration steps":#final_steps
## "Install the CA root certificate (required in both alternatives)":#ca_root_certificate
@@ -38,21 +42,20 @@ It is possible to start with the single host installation method and modify the
h2(#prerequisites). Prerequisites and planning
-Arvados requires SSL for (almost) all network traffic. This installation method supports the following options for the required SSL certificate(s): @self-signed@ and @bring your own certificates at .
-
-Prerequisites
+Prerequisites:
* git
* a dedicated (virtual) machine for your Arvados server with at least 2 cores and 8 GiB of RAM, running a "supported Arvados distribution":{{site.baseurl}}/install/install-manual-prerequisites.html#supportedlinux
-* ports 9443-9445, 11002, 14202, 18002, 35101 need to be reachable from your client (configurable, see below)
* at least one DNS hostname that resolves to the IP address of your Arvados server
-* one or more SSL certificates matching the hostname(s) in use (only when using @bring your own certificates@)
+* ports 443, 9443-9445, 11002, 14202, 18002, 35101 need to be reachable from your client (configurable, see below)
+* port 80 needs to be reachable from everywhere on the internet (only when using "Let's Encrypt":#lets-encrypt)
+* one or more SSL certificates matching the hostname(s) in use (only when using "bring your own certificate(s)":#bring-your-own)
h2(#single_host). Single host install using the provision.sh script
{% include 'branchname' %}
-This procedure will install all the main Arvados components to get you up and running in a single host. The whole installation procedure takes somewhere between 15 to 60 minutes, depending on the host resources and its network bandwidth. As a reference, on a virtual machine with 1 core and 1 GB RAM, it takes ~25 minutes to do the initial install.
+This procedure will install all the main Arvados components to get you up and running in a single host.
This is a package-based installation method, however the installation script is currently distributed in source form via @git@:
@@ -63,15 +66,13 @@ cd arvados/tools/salt-install
</code></pre>
</notextile>
-The @provision.sh@ script will help you deploy Arvados by preparing your environment to be able to run the installer, then running it. The actual installer is located at "arvados-formula":https://git.arvados.org/arvados-formula.git/tree/refs/heads/{{ branchname }} and will be cloned during the running of the @provision.sh@ script. The installer is built using "Saltstack":https://saltproject.io/ and @provision.sh@ performs the install using master-less mode.
+The @provision.sh@ script will help you deploy Arvados by preparing your environment to be able to run the installer, then running it. The actual installer is located in the "arvados-formula git repository":https://git.arvados.org/arvados-formula.git/tree/refs/heads/{{ branchname }} and will be cloned during the running of the @provision.sh@ script. The installer is built using "Saltstack":https://saltproject.io/ and @provision.sh@ performs the install using master-less mode.
After setting up a few variables in a config file (next step), you'll be ready to run it and get Arvados deployed.
h2(#choose_configuration). Choose the desired configuration
-For documentation's sake, we will use the cluster name <i>arva2</i> and the domain <i>arv.local</i>. If you don't change them as required in the next steps, installation won't proceed.
-
-Arvados' single host installation can be done in two fashions:
+Arvados' single host installation can be done in two ways:
* Using a single hostname, assigning <i>a different port (other than 443) for each user-facing service</i>: This choice is easier to setup, but the user will need to know the port/s for the different services she wants to connect to.
* Using multiple hostnames on the same IP: this setup involves a few extra steps but each service will have a meaningful hostname so it will make easier to access them later.
@@ -87,10 +88,6 @@ cp -r config_examples/single_host/single_hostname local_config_dir
Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_PORT, *_TOKEN</b> and <b>*KEY</b> variables.
-The <i>single_host</i> examples use self-signed SSL certificates by default, which are deployed using the same mechanism used to deploy custom certificates.
-
-When setting (SSL_MODE=lets-encrypt), please note: When using AWS, EC2 instances can have a default hostname that ends with `amazonaws.com`. Let's Encrypt has a blacklist of domain names for which it will not issue certificates, and that blacklist includes the `amazonaws.com` domain. In order to use Let's Encrypt certificates on AWS EC2, you will need to bring your own domain name and point a hostname in that domain at your EC2 instance.
-
h3(#single_host_multiple_hostnames). Single host / multiple hostnames (Alternative configuration)
<notextile>
<pre><code>cp local.params.example.single_host_multiple_hostnames local.params
@@ -100,6 +97,61 @@ cp -r config_examples/single_host/multiple_hostnames local_config_dir
Edit the variables in the <i>local.params</i> file.
+h2(#certificates). Choose the SSL configuration (SSL_MODE)
+
+Arvados requires SSL certificates to work correctly. This installer supports these options:
+
+* @self-signed@: let the installer create self-signed certificate(s)
+* @lets-encrypt@: automatically obtain and install SSL certificates for your hostname(s)
+* @bring-your-own@: supply your own certificate(s) in the `certs` directory
+
+h3(#self-signed). Using self-signed certificates
+
+In the default configuration, this installer uses self-signed certificate(s):
+
+<notextile>
+<pre><code>SSL_MODE="self-signed"
+</code></pre>
+</notextile>
+
+When connecting to the Arvados web interface for the first time, you will need to accept the self-signed certificate as trusted to bypass the browser warnings.
+
+h3(#lets-encrypt). Using Let's Encrypt certificates
+
+To automatically get (a) valid certificate(s) via Let's Encrypt, change the configuration like this:
+
+<notextile>
+<pre><code>SSL_MODE="lets-encrypt"
+</code></pre>
+</notextile>
+
+It is important that the DNS hostnames defined in the configuration resolve to the Arvados instance(s), so that Let's Encrypt can validate the domainname ownership and issue the certificate(s).
+
+When using AWS, EC2 instances can have a default hostname that ends with `amazonaws.com`. Let's Encrypt has a blacklist of domain names for which it will not issue certificates, and that blacklist includes the `amazonaws.com` domain, which means the default hostname can not be used to get a certificate from Let's Encrypt.
+
+For a @single hostname@ setup, the hostname must be defined in @HOSTNAME_EXT@ and resolve to the IP address of your Arvados instance.
+
+For a @multiple hostnames@ setup, the hostnames are created by combining the values of @CLUSTER@ and @DOMAIN@ from the configuration with a prefix. These hostnames must resolve to the IP address of your Arvados instance:
+
+* @CLUSTER at .@DOMAIN@
+* ws. at CLUSTER@. at DOMAIN@
+* workbench. at CLUSTER@. at DOMAIN@
+* workbench2. at CLUSTER@. at DOMAIN@
+* webshell. at CLUSTER@. at DOMAIN@
+* download. at CLUSTER@. at DOMAIN@
+* collections. at CLUSTER@. at DOMAIN@
+* keep. at CLUSTER@. at DOMAIN@
+
+h3(#bring-your-own). Using your own certificates
+
+To supply your own certificates, change the configuration like this:
+
+<notextile>
+<pre><code>SSL_MODE="bring-your-own"
+CUSTOM_CERTS_DIR="${SCRIPT_DIR}/certs"
+</code></pre>
+</notextile>
+
{% include 'install_custom_certificates' %}
h3(#further_customization). Further customization of the installation (modifying the salt pillars and states)
@@ -126,7 +178,7 @@ ssh user at host sudo ./provision.sh -c /path/to/your/local.params.file
</code></pre>
</notextile>
-and wait for it to finish.
+and wait for it to finish. The script will need 5 to 10 minutes to install and configure everything.
If everything goes OK, you'll get some final lines stating something like:
@@ -173,11 +225,11 @@ To access your Arvados instance using command line clients (such as arv-get and
</code></pre>
</notextile>
-h3(#single_host_multiple_hostnames_dns_configuration). DNS configuration (single host / multiple hostnames)
+h3(#single_host_multiple_hostnames_dns_configuration). Local DNS configuration (multiple hostnames only)
-When using multiple hostnames, after the setup is done, you need to set up your DNS to be able to access the cluster.
+When using multiple hostnames, you need to set up your DNS to be able to access the cluster.
-If you don't have access to the domain's DNS to add the required entries, the simplest way to do it is to edit your @/etc/hosts@ file (as root):
+If you don't have access to the domain's DNS to add the required entries, the simplest way to do it is to edit your @/etc/hosts@ file (as root). Change @CLUSTER@, @DOMAIN@ and @HOST_IP@ to your local values:
<notextile>
<pre><code>export CLUSTER="arva2"
@@ -191,21 +243,17 @@ echo "${HOST_IP} api keep keep0 collections download ws workbench workbench2 ${C
h2(#initial_user). Initial user and login
-At this point you should be able to log into the Arvados cluster. The initial URL will be:
-
-* https://workbench.arva2.arv.local
+At this point you should be able to log on to your new Arvados cluster.
-or, in general, the url format will be:
+For a @single hostname@ setup, the workbench URL will be
-* https://workbench.@<cluster>.<domain>@
+* https://@HOSTNAME_EXT@
-By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.
+For a @multiple hostnames@ setup, the workbench URL will be
-Assuming you didn't change these values in the @local.params@ file, the initial credentials are:
+* https://workbench.@CLUSTER@.@DOMAIN@
-* User: 'admin'
-* Password: 'password'
-* Email: 'admin at arva2.arv.local'
+By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster. The username, password and e-mail address for the initial user are configured in the @local.params@ file.
h2(#test_install). Test the installed cluster running a simple workflow
diff --git a/tools/salt-install/local.params.example.single_host_single_hostname b/tools/salt-install/local.params.example.single_host_single_hostname
index d01c73d2c..48354a0d9 100644
--- a/tools/salt-install/local.params.example.single_host_single_hostname
+++ b/tools/salt-install/local.params.example.single_host_single_hostname
@@ -29,7 +29,7 @@ KEEP_EXT_SSL_PORT=35101
KEEPWEB_EXT_SSL_PORT=11002
WEBSHELL_EXT_SSL_PORT=14202
WEBSOCKET_EXT_SSL_PORT=18002
-WORKBENCH1_EXT_SSL_PORT=9444
+WORKBENCH1_EXT_SSL_PORT=443
WORKBENCH2_EXT_SSL_PORT=9445
INITIAL_USER="admin"
@@ -51,31 +51,19 @@ DATABASE_PASSWORD=please_set_this_to_some_secure_value
# Arvados requires SSL certificates to work correctly. This installer supports these options:
# * self-signed: let the installer create self-signed certificate(s)
# * bring-your-own: supply your own certificate(s) in the `certs` directory
+# * lets-encrypt: automatically obtain and install SSL certificates for your hostname(s)
#
# See https://doc.arvados.org/intall/salt-single-host.html#certificates for more information.
SSL_MODE="self-signed"
-# If you want to use letsencrypt, set SSL_MODE="lets-encrypt"
-# A single certificate for the external hostname of the host will be retrieved, using
-# "standalone" mode of LE.
-
-# If you going to provide your own certificate for Arvados, the provision script can
-# help you deploy it. In order to do that, you need to set `SSL_MODE=bring-your-own` above,
-# and copy the required certificate under the directory specified in the next line.
-# The cert will be copied from this directory by the provision script.
-# Please set it to the FULL PATH to the certs dir if you're going to use a different dir
-# Default is "${SCRIPT_DIR}/certs", where the variable "SCRIPT_DIR" has the path to the
-# directory where the "provision.sh" script was copied in the destination host.
+# Only used when SSL_MODE is set to "bring-your-own".
+# See https://doc.arvados.org/intall/salt-single-host.html#bring-your-own for more information.
# CUSTOM_CERTS_DIR="${SCRIPT_DIR}/certs"
-# The script expects cert/key files with the filename matcing ${HOSTNAME_EXT} above
-# Ie., for "HOSTNAME_EXT='my-arvados.example.net', the script will lookup for
-# ${CUSTOM_CERTS_DIR}/my-arvados.example.net.crt
-# ${CUSTOM_CERTS_DIR}/my-arvados.example.net.key
-# The certs will be copied from this directory by the provision script.
# The directory to check for the config files (pillars, states) you want to use.
# There are a few examples under 'config_examples'.
# CONFIG_DIR="local_config_dir"
+
# Extra states to apply. If you use your own subdir, change this value accordingly
# EXTRA_STATES_DIR="${CONFIG_DIR}/states"
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index f1ca3c9ec..037c22275 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -413,7 +413,7 @@ fi
mkdir -p ${T_DIR}
# Replace cluster and domain name in the test files
for f in $(ls "${SOURCE_TESTS_DIR}"/*); do
- sed "s#__CLUSTER__#${CLUSTER}#g;
+ FILTERS="s#__CLUSTER__#${CLUSTER}#g;
s#__CONTROLLER_EXT_SSL_PORT__#${CONTROLLER_EXT_SSL_PORT}#g;
s#__DOMAIN__#${DOMAIN}#g;
s#__IP_INT__#${IP_INT}#g;
@@ -421,8 +421,13 @@ for f in $(ls "${SOURCE_TESTS_DIR}"/*); do
s#__INITIAL_USER_PASSWORD__#${INITIAL_USER_PASSWORD}#g
s#__INITIAL_USER__#${INITIAL_USER}#g;
s#__DATABASE_PASSWORD__#${DATABASE_PASSWORD}#g;
- s#__SYSTEM_ROOT_TOKEN__#${SYSTEM_ROOT_TOKEN}#g" \
- "${f}" > ${T_DIR}/$(basename "${f}")
+ s#__SYSTEM_ROOT_TOKEN__#${SYSTEM_ROOT_TOKEN}#g"
+ if [ "$USE_SINGLE_HOSTNAME" = "yes" ]; then
+ FILTERS="s#__CLUSTER__.__DOMAIN__#${HOSTNAME_EXT}#g;
+ $FILTERS"
+ fi
+ sed "$FILTERS" \
+ "${f}" > ${T_DIR}/$(basename "${f}")
done
chmod 755 ${T_DIR}/run-test.sh
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list