[ARVADOS] created: 2.1.0-265-gd8241cba0
Git user
git at public.arvados.org
Tue Jan 26 12:00:53 UTC 2021
at d8241cba03d70a7b6e02613a4ce0e9b72e0ef9cb (commit)
commit d8241cba03d70a7b6e02613a4ce0e9b72e0ef9cb
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Jan 26 08:58:07 2021 -0300
feat(provision): refactor to add other setup examples
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/README.md b/tools/salt-install/config_examples/single_host/multiple_hostnames/README.md
new file mode 100644
index 000000000..17ca89a9f
--- /dev/null
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/README.md
@@ -0,0 +1,20 @@
+Single host with multiple hostnames
+===================================
+
+These files let you setup Arvados on a single host using different hostnames
+for each of its components nginx's virtualhosts.
+
+The hostnames are composed after the variables "CLUSTER" and "DOMAIN" set in
+the `local.params` file.
+
+The virtual hosts' hostnames that will be used are:
+
+* CLUSTER.DOMAIN
+* collections.CLUSTER.DOMAIN
+* download.CLUSTER.DOMAIN
+* keep.CLUSTER.DOMAIN
+* keep0.CLUSTER.DOMAIN
+* webshell.CLUSTER.DOMAIN
+* workbench.CLUSTER.DOMAIN
+* workbench2.CLUSTER.DOMAIN
+* ws.CLUSTER.DOMAIN
diff --git a/tools/salt-install/single_host/arvados.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
similarity index 100%
copy from tools/salt-install/single_host/arvados.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
diff --git a/tools/salt-install/single_host/docker.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/docker.sls
similarity index 100%
copy from tools/salt-install/single_host/docker.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/docker.sls
diff --git a/tools/salt-install/single_host/locale.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/locale.sls
similarity index 100%
copy from tools/salt-install/single_host/locale.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/locale.sls
diff --git a/tools/salt-install/single_host/nginx_api_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_api_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_api_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_api_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_controller_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_controller_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_controller_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepproxy_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_keepproxy_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepproxy_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepweb_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_keepweb_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepweb_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_passenger.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_passenger.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_passenger.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_passenger.sls
diff --git a/tools/salt-install/single_host/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_webshell_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_webshell_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_webshell_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_websocket_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_websocket_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_websocket_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench2_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_workbench2_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench2_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_workbench_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench_configuration.sls
diff --git a/tools/salt-install/single_host/postgresql.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/postgresql.sls
similarity index 100%
copy from tools/salt-install/single_host/postgresql.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/postgresql.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/README.md b/tools/salt-install/config_examples/single_host/single_hostname/README.md
new file mode 100644
index 000000000..9c7ab96c3
--- /dev/null
+++ b/tools/salt-install/config_examples/single_host/single_hostname/README.md
@@ -0,0 +1,23 @@
+Single host with a single hostname
+==================================
+
+These files let you setup Arvados on a single host using a single hostname
+for all of its components nginx's virtualhosts.
+
+The hostname MUST be given in the `local.params` file. The script won't try
+to guess it because, depending on the network architecture where you're
+installing Arvados, things might not work as expected.
+
+The services will be available on the same hostname but different ports,
+which can be given on the `local.params` file or will default to the following
+values:
+
+* CLUSTER.DOMAIN
+* collections
+* download
+* keep
+* keep0
+* webshell
+* workbench
+* workbench2
+* ws
diff --git a/tools/salt-install/single_host/arvados.sls b/tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
similarity index 83%
rename from tools/salt-install/single_host/arvados.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
index a06244270..9a228faa4 100644
--- a/tools/salt-install/single_host/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
@@ -107,7 +107,7 @@ arvados:
# <cluster>-nyw5e-<volume>
__CLUSTER__-nyw5e-000000000000000:
AccessViaHosts:
- http://keep0.__CLUSTER__.__DOMAIN__:25107:
+ http://__HOSTNAME__:25107:
ReadOnly: false
Replication: 2
Driver: Directory
@@ -122,38 +122,38 @@ arvados:
Services:
Controller:
- ExternalURL: https://__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: https://__HOSTNAME__:__CONTROLLER_EXT_SSL_PORT__
InternalURLs:
http://controller.internal:8003: {}
DispatchCloud:
InternalURLs:
- http://__CLUSTER__.__DOMAIN__:9006: {}
+ http://__HOSTNAME__:9006: {}
Keepbalance:
InternalURLs:
- http://__CLUSTER__.__DOMAIN__:9005: {}
+ http://__HOSTNAME__:9005: {}
Keepproxy:
- ExternalURL: https://keep.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: https://__HOSTNAME__:__KEEP_EXT_SSL_PORT__
InternalURLs:
http://keep.internal:25100: {}
Keepstore:
InternalURLs:
- http://keep0.__CLUSTER__.__DOMAIN__:25107: {}
+ http://__HOSTNAME__:25107: {}
RailsAPI:
InternalURLs:
http://api.internal:8004: {}
WebDAV:
- ExternalURL: https://collections.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: https://__HOSTNAME__:__COLLECTIONS_EXT_SSL_PORT__
InternalURLs:
http://collections.internal:9002: {}
WebDAVDownload:
- ExternalURL: https://download.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: https://__HOSTNAME__:__DOWNLOAD_EXT_SSL_PORT__
WebShell:
- ExternalURL: https://webshell.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: https://__HOSTNAME__:__WEBSHELL_EXT_SSL_PORT__
Websocket:
- ExternalURL: wss://ws.__CLUSTER__.__DOMAIN__/websocket
+ ExternalURL: wss://__HOSTNAME__:__WEBSOCKET_EXT_SSL_PORT__/websocket
InternalURLs:
http://ws.internal:8005: {}
Workbench1:
- ExternalURL: https://workbench.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: https://__HOSTNAME__:__WORKBENCH1_EXT__SSL_PORT__
Workbench2:
- ExternalURL: https://workbench2.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: https://__HOSTNAME__:__WORKBENCH2_EXT__SSL_PORT__
diff --git a/tools/salt-install/single_host/docker.sls b/tools/salt-install/config_examples/single_host/single_hostname/docker.sls
similarity index 100%
rename from tools/salt-install/single_host/docker.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/docker.sls
diff --git a/tools/salt-install/single_host/locale.sls b/tools/salt-install/config_examples/single_host/single_hostname/locale.sls
similarity index 100%
rename from tools/salt-install/single_host/locale.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/locale.sls
diff --git a/tools/salt-install/single_host/nginx_api_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls
similarity index 100%
rename from tools/salt-install/single_host/nginx_api_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
similarity index 92%
rename from tools/salt-install/single_host/nginx_controller_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
index 00c3b3a13..2eb33b835 100644
--- a/tools/salt-install/single_host/nginx_controller_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
@@ -25,7 +25,7 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- 80 default
- location /.well-known:
@@ -38,9 +38,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://controller_upstream'
diff --git a/tools/salt-install/single_host/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
similarity index 90%
rename from tools/salt-install/single_host/nginx_keepproxy_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
index 6554f79a7..b26de2710 100644
--- a/tools/salt-install/single_host/nginx_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
@@ -21,9 +21,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: keep.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __KEEP_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -34,9 +34,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: keep.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __KEEP_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://keepproxy_upstream'
diff --git a/tools/salt-install/single_host/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
similarity index 77%
rename from tools/salt-install/single_host/nginx_keepweb_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
index cc871b9da..98a3cdf94 100644
--- a/tools/salt-install/single_host/nginx_keepweb_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
@@ -15,15 +15,15 @@ nginx:
servers:
managed:
- ### DEFAULT
+ ### COLLECTIONS / DOWNLOAD
arvados_collections_download_default:
enabled: true
overwrite: true
config:
- server:
- - server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __KEEPWEB_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -35,9 +35,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __KEEPWEB_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://collections_downloads_upstream'
@@ -53,5 +53,5 @@ nginx:
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- include: 'snippets/arvados-snakeoil.conf'
- - access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
- - error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
+ - access_log: /var/log/nginx/keepweb.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/keepweb.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/single_host/nginx_passenger.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_passenger.sls
similarity index 100%
rename from tools/salt-install/single_host/nginx_passenger.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_passenger.sls
diff --git a/tools/salt-install/single_host/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
similarity index 92%
rename from tools/salt-install/single_host/nginx_webshell_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
index a0756b7ce..dac606123 100644
--- a/tools/salt-install/single_host/nginx_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
@@ -22,9 +22,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: webshell.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WEBSHELL_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -35,11 +35,11 @@ nginx:
overwrite: true
config:
- server:
- - server_name: webshell.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WEBSHELL_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- - location /shell.__CLUSTER__.__DOMAIN__:
+ - location /__HOSTNAME__:
- proxy_pass: 'http://webshell_upstream'
- proxy_read_timeout: 90
- proxy_connect_timeout: 90
diff --git a/tools/salt-install/single_host/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
similarity index 90%
rename from tools/salt-install/single_host/nginx_websocket_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
index ebe03f733..827524cbe 100644
--- a/tools/salt-install/single_host/nginx_websocket_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
@@ -21,9 +21,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: ws.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WEBSOCKET_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -34,9 +34,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: ws.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WEBSOCKET_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://websocket_upstream'
diff --git a/tools/salt-install/single_host/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
similarity index 80%
rename from tools/salt-install/single_host/nginx_workbench2_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
index 8930be408..7f90cbc82 100644
--- a/tools/salt-install/single_host/nginx_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
@@ -19,9 +19,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench2.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WORKBENCH2_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -32,9 +32,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench2.__CLUSTER__.__DOMAIN__
+ - server_name: workbench2.__HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WORKBENCH2_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- root: /var/www/arvados-workbench2/workbench2
@@ -42,7 +42,7 @@ nginx:
- 'if (-f $document_root/maintenance.html)':
- return: 503
- location /config.json:
- - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__"}' ~ "'" }}
+ - return: {{ "200 '" ~ '{"API_HOST":"__HOSTNAME__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
- include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/single_host/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
similarity index 91%
rename from tools/salt-install/single_host/nginx_workbench_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
index be571ca77..0cbd3e14a 100644
--- a/tools/salt-install/single_host/nginx_workbench_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
@@ -28,9 +28,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WORKBENCH_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -41,9 +41,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench.__CLUSTER__.__DOMAIN__
+ - server_name: workbench.__HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WORKBENCH1_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://workbench_upstream'
diff --git a/tools/salt-install/single_host/postgresql.sls b/tools/salt-install/config_examples/single_host/single_hostname/postgresql.sls
similarity index 100%
rename from tools/salt-install/single_host/postgresql.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/postgresql.sls
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example
new file mode 100644
index 000000000..a88301b2a
--- /dev/null
+++ b/tools/salt-install/local.params.example
@@ -0,0 +1,64 @@
+##########################################################
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: CC-BY-SA-3.0
+
+# These are the basic parameters to configure the installation
+
+# The 5 letters name you want to give your cluster
+CLUSTER="fixme"
+
+# The domainname you want tou give to your cluster's hosts
+DOMAIN="some.domain"
+
+# When setting the cluster in a single host, you can use a single hostname
+# to access all the instances. When using virtualization (ie AWS), this should be
+# the EXTERNAL hostname for the instance.
+# If empty, the INTERNAL HOST IP will be used
+HOSTNAME=""
+CONTROLLER_EXT_SSL_PORT=8000
+KEEP_EXT_SSL_PORT=25101
+# Both for collections and downloads
+KEEPWEB_EXT_SSL_PORT=9002
+WEBSHELL_EXT_SSL_PORT=4202
+WEBSOCKET_EXT_SSL_PORT=8002
+WORKBENCH1_EXT_SSL_PORT=443
+WORKBENCH2_EXT_SSL_PORT=3001
+
+INITIAL_USER="admin"
+
+# If not specified, the initial user email will be composed as
+# INITIAL_USER at CLUSTER.DOMAIN
+INITIAL_USER_EMAIL="admin at fixme.localdomain"
+INITIAL_USER_PASSWORD="password"
+
+# The example config files you want to use. There are a few examples
+# under 'config_examples'
+CONFIG_DIR="config_examples/single_host/single_hostname"
+
+# Which release of Arvados repo you want to use
+RELEASE="production"
+# Which version of Arvados you want to install. Defaults to 'latest'
+# in the desired repo
+VERSION="latest"
+
+# Host SSL port where you want to point your browser to access Arvados
+# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
+# You can point it to another port if desired
+# In Vagrant, make sure it matches what you set in the Vagrantfile
+HOST_SSL_PORT=443
+
+# This is an arvados-formula setting.
+# If branch is set, the script will switch to it before running salt
+# Usually not needed, only used for testing
+BRANCH="master"
+
+##########################################################
+# Usually there's no need to modify things below this line
+
+# Formulas versions
+ARVADOS_TAG="v1.1.4"
+POSTGRES_TAG="v0.41.3"
+NGINX_TAG="v2.4.0"
+DOCKER_TAG="v1.0.0"
+LOCALE_TAG="v0.3.4"
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index b97d71965..261586a63 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -15,15 +15,6 @@ set -o pipefail
# capture the directory that the script is running from
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-CONFIG_DIR="single_host"
-RELEASE="production"
-VERSION="latest"
-ARVADOS_TAG="v1.1.4"
-POSTGRES_TAG="v0.41.3"
-NGINX_TAG="v2.4.0"
-DOCKER_TAG="v1.0.0"
-LOCALE_TAG="v0.3.4"
-
usage() {
echo >&2
echo >&2 "Usage: ${0} [-h] [-h]"
@@ -113,6 +104,15 @@ LOG_LEVEL="info"
HOST_SSL_PORT=443
TESTS_DIR="tests"
+CONFIG_DIR="config_examples/single_host/multiple_hostnames"
+RELEASE="production"
+VERSION="latest"
+ARVADOS_TAG="v1.1.4"
+POSTGRES_TAG="v0.41.3"
+NGINX_TAG="v2.4.0"
+DOCKER_TAG="v1.0.0"
+LOCALE_TAG="v0.3.4"
+
arguments ${@}
if [ -s ${CONFIG} ]; then
@@ -156,9 +156,7 @@ pillar_roots:
- ${P_DIR}
EOFSM
-mkdir -p ${S_DIR}
-mkdir -p ${F_DIR}
-mkdir -p ${P_DIR}
+mkdir -p ${S_DIR} ${F_DIR} ${P_DIR}
# States
cat > ${S_DIR}/top.sls << EOFTSLS
commit 102277e608eca622b0ea568845744921e48fe2e7
Author: Javier Bértoli <jbertoli at curii.com>
Date: Mon Jan 25 07:10:01 2021 -0300
feat(provision): manage setup params from local file
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/Vagrantfile b/tools/salt-install/Vagrantfile
index 6966ea834..60f57ca66 100644
--- a/tools/salt-install/Vagrantfile
+++ b/tools/salt-install/Vagrantfile
@@ -34,6 +34,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
path: "provision.sh",
args: [
# "--debug",
+ "--config /vagrant/local.params",
"--test",
"--vagrant",
"--ssl-port=8443"
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 79712d3f9..b97d71965 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -24,43 +24,37 @@ NGINX_TAG="v2.4.0"
DOCKER_TAG="v1.0.0"
LOCALE_TAG="v0.3.4"
-if [ -s ${SCRIPT_DIR}/local.params ]; then
- source ${SCRIPT_DIR}/local.params
-else
- echo >&2 "Please create a '${SCRIPT_DIR}/local.params' file with initial values, as described in FIXME_URL_TO_DESCR"
- exit 1
-fi
-
usage() {
echo >&2
echo >&2 "Usage: ${0} [-h] [-h]"
echo >&2
echo >&2 "${0} options:"
- echo >&2 " -d, --debug Run salt installation in debug mode"
- echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
- echo >&2 " -t, --test Test installation running a CWL workflow"
- echo >&2 " -r, --roles List of Arvados roles to apply to the host, comma separated"
- echo >&2 " Possible values are:"
- echo >&2 " api"
- echo >&2 " controller"
- echo >&2 " keepstore"
- echo >&2 " websocket"
- echo >&2 " keepweb"
- echo >&2 " workbench2"
- echo >&2 " keepproxy"
- echo >&2 " shell"
- echo >&2 " workbench"
- echo >&2 " dispatcher"
- echo >&2 " Defaults to applying them all"
- echo >&2 " -h, --help Display this help and exit"
- echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
+ echo >&2 " -d, --debug Run salt installation in debug mode"
+ echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
+ echo >&2 " -c <local.params>, --config <local.params> Path to the local.params config file"
+ echo >&2 " -t, --test Test installation running a CWL workflow"
+ echo >&2 " -r, --roles List of Arvados roles to apply to the host, comma separated"
+ echo >&2 " Possible values are:"
+ echo >&2 " api"
+ echo >&2 " controller"
+ echo >&2 " keepstore"
+ echo >&2 " websocket"
+ echo >&2 " keepweb"
+ echo >&2 " workbench2"
+ echo >&2 " keepproxy"
+ echo >&2 " shell"
+ echo >&2 " workbench"
+ echo >&2 " dispatcher"
+ echo >&2 " Defaults to applying them all"
+ echo >&2 " -h, --help Display this help and exit"
+ echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
echo >&2
}
arguments() {
# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
- TEMP=$(getopt -o dhp:r:tv \
- --long debug,help,ssl-port:,roles:,test,vagrant \
+ TEMP=$(getopt -o c:dhp:r:tv \
+ --long config:,debug,help,ssl-port:,roles:,test,vagrant \
-n "${0}" -- "${@}")
if [ ${?} != 0 ] ; then echo "GNU getopt missing? Use -h for help"; exit 1 ; fi
@@ -69,6 +63,10 @@ arguments() {
while [ ${#} -ge 1 ]; do
case ${1} in
+ -c | --config)
+ CONFIG=${2}
+ shift 2
+ ;;
-d | --debug)
LOG_LEVEL="debug"
shift
@@ -110,12 +108,20 @@ arguments() {
done
}
+CONFIG="${SCRIPT_DIR}/local.params"
LOG_LEVEL="info"
HOST_SSL_PORT=443
TESTS_DIR="tests"
arguments ${@}
+if [ -s ${CONFIG} ]; then
+ source ${CONFIG}
+else
+ echo >&2 "Please create a '${CONFIG}' file with initial values, as described in FIXME_URL_TO_DESCR"
+ exit 1
+fi
+
# Salt's dir
## states
S_DIR="/srv/salt"
commit 677877d5327f559730ca2d8483f48da16880a9b5
Author: Javier Bértoli <jbertoli at curii.com>
Date: Fri Jan 22 12:41:12 2021 -0300
feat(provision): allow to install individual roles
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/README.md b/tools/salt-install/README.md
index 10d08b414..58800dd6f 100644
--- a/tools/salt-install/README.md
+++ b/tools/salt-install/README.md
@@ -6,15 +6,21 @@
##### About
-This directory holds a small script to install Arvados on a single node, using the
+This directory holds a small script to help you get Arvados up and running, using the
[Saltstack arvados-formula](https://github.com/saltstack-formulas/arvados-formula)
in master-less mode.
-The fastest way to get it running is to modify the first lines in the `provision.sh`
-script to suit your needs, copy it in the host where you want to install Arvados
-and run it as root.
+There are a few preset examples that you can use:
-There's an example `Vagrantfile` also, to install it in a vagrant box if you want
+* `single_host`: Install all the Arvados components in a single host. Suitable for testing
+ or demo-ing, but not recommended for production use.
+* `multi_host/aws`: Let's you install different Arvados components in different hosts on AWS.
+
+The fastest way to get it running is to copy the `local.params.example` file to `local.params`,
+edit and modify the file to suit your needs, copy this file along with the `provision.sh` script
+into the host where you want to install Arvados and run the `provision.sh` script as root.
+
+There's an example `Vagrantfile` also, to install Arvados in a vagrant box if you want
to try it locally.
For more information, please read https://doc.arvados.org/main/install/salt-single-host.html
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 31266c1b8..79712d3f9 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -10,55 +10,26 @@
#
# vagrant up
-##########################################################
-# This section are the basic parameters to configure the installation
-
-# The 5 letters name you want to give your cluster
-CLUSTER="arva2"
-DOMAIN="arv.local"
-
-INITIAL_USER="admin"
+set -o pipefail
-# If not specified, the initial user email will be composed as
-# INITIAL_USER at CLUSTER.DOMAIN
-INITIAL_USER_EMAIL="${INITIAL_USER}@${CLUSTER}.${DOMAIN}"
-INITIAL_USER_PASSWORD="password"
+# capture the directory that the script is running from
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-# The example config you want to use. Currently, only "single_host" is
-# available
CONFIG_DIR="single_host"
-
-# Which release of Arvados repo you want to use
RELEASE="production"
-# Which version of Arvados you want to install. Defaults to 'latest'
-# in the desired repo
VERSION="latest"
-
-# Host SSL port where you want to point your browser to access Arvados
-# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
-# You can point it to another port if desired
-# In Vagrant, make sure it matches what you set in the Vagrantfile
-# HOST_SSL_PORT=443
-
-# This is a arvados-formula setting.
-# If branch is set, the script will switch to it before running salt
-# Usually not needed, only used for testing
-# BRANCH="master"
-
-##########################################################
-# Usually there's no need to modify things below this line
-
-# Formulas versions
ARVADOS_TAG="v1.1.4"
POSTGRES_TAG="v0.41.3"
NGINX_TAG="v2.4.0"
DOCKER_TAG="v1.0.0"
LOCALE_TAG="v0.3.4"
-set -o pipefail
-
-# capture the directory that the script is running from
-SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+if [ -s ${SCRIPT_DIR}/local.params ]; then
+ source ${SCRIPT_DIR}/local.params
+else
+ echo >&2 "Please create a '${SCRIPT_DIR}/local.params' file with initial values, as described in FIXME_URL_TO_DESCR"
+ exit 1
+fi
usage() {
echo >&2
@@ -68,6 +39,19 @@ usage() {
echo >&2 " -d, --debug Run salt installation in debug mode"
echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
echo >&2 " -t, --test Test installation running a CWL workflow"
+ echo >&2 " -r, --roles List of Arvados roles to apply to the host, comma separated"
+ echo >&2 " Possible values are:"
+ echo >&2 " api"
+ echo >&2 " controller"
+ echo >&2 " keepstore"
+ echo >&2 " websocket"
+ echo >&2 " keepweb"
+ echo >&2 " workbench2"
+ echo >&2 " keepproxy"
+ echo >&2 " shell"
+ echo >&2 " workbench"
+ echo >&2 " dispatcher"
+ echo >&2 " Defaults to applying them all"
echo >&2 " -h, --help Display this help and exit"
echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
echo >&2
@@ -75,8 +59,8 @@ usage() {
arguments() {
# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
- TEMP=$(getopt -o dhp:tv \
- --long debug,help,ssl-port:,test,vagrant \
+ TEMP=$(getopt -o dhp:r:tv \
+ --long debug,help,ssl-port:,roles:,test,vagrant \
-n "${0}" -- "${@}")
if [ ${?} != 0 ] ; then echo "GNU getopt missing? Use -h for help"; exit 1 ; fi
@@ -89,6 +73,23 @@ arguments() {
LOG_LEVEL="debug"
shift
;;
+ -p | --ssl-port)
+ HOST_SSL_PORT=${2}
+ shift 2
+ ;;
+ -r | --roles)
+ for i in ${2//,/ }
+ do
+ # Verify the role exists
+ if [[ ! "api,controller,keepstore,websocket,keepweb,workbench2,keepproxy,shell,workbench,dispatcher" == *"$i"* ]]; then
+ echo "The role '${i}' is not a valid role"
+ usage
+ exit 1
+ fi
+ ROLES="${ROLES} ${i}"
+ done
+ shift 2
+ ;;
-t | --test)
TEST="yes"
shift
@@ -97,10 +98,6 @@ arguments() {
VAGRANT="yes"
shift
;;
- -p | --ssl-port)
- HOST_SSL_PORT=${2}
- shift 2
- ;;
--)
shift
break
@@ -167,9 +164,17 @@ base:
- nginx.passenger
- postgres
- docker
- - arvados
EOFTSLS
+# If we want specific roles for a node, just add those states
+if [ -z "${ROLES}" ]; then
+ echo ' - arvados' >> ${S_DIR}/top.sls
+else
+ for R in ${ROLES}; do
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ done
+fi
+
# Pillars
cat > ${P_DIR}/top.sls << EOFPSLS
base:
@@ -191,7 +196,7 @@ EOFPSLS
# Get the formula and dependencies
cd ${F_DIR} || exit 1
-git clone --branch "${ARVADOS_TAG}" https://github.com/saltstack-formulas/arvados-formula.git
+git clone --branch "${ARVADOS_TAG}" https://github.com/arvados/arvados-formula.git
git clone --branch "${DOCKER_TAG}" https://github.com/saltstack-formulas/docker-formula.git
git clone --branch "${LOCALE_TAG}" https://github.com/saltstack-formulas/locale-formula.git
git clone --branch "${NGINX_TAG}" https://github.com/saltstack-formulas/nginx-formula.git
commit 2e6894963f87d0b3f955de8c75acfac8451fd6b2
Author: Javier Bértoli <jbertoli at curii.com>
Date: Fri Jan 15 09:36:07 2021 -0300
Documentation: Explain that the canonical salt installer is hosted in Arvados' github account
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/doc/install/salt-multi-host.html.textile.liquid b/doc/install/salt-multi-host.html.textile.liquid
index 4ba153faf..50de6e439 100644
--- a/doc/install/salt-multi-host.html.textile.liquid
+++ b/doc/install/salt-multi-host.html.textile.liquid
@@ -40,25 +40,25 @@ The formulas we use are:
* "docker":https://github.com/saltstack-formulas/docker-formula.git
* "locale":https://github.com/saltstack-formulas/locale-formula.git
-There are example Salt pillar files for each of those formulas in the "arvados-formula's test/salt/pillar/examples":https://github.com/saltstack-formulas/arvados-formula/tree/master/test/salt/pillar/examples directory. As they are, they allow you to get all the main Arvados components up and running.
+There are example Salt pillar files for each of those formulas in the "arvados-formula's test/salt/pillar/examples":https://github.com/arvados/arvados-formula/tree/master/test/salt/pillar/examples directory. As they are, they allow you to get all the main Arvados components up and running.
h2(#saltstack). Install Arvados using Saltstack
This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
-The Arvados formula we maintain is located in the Saltstack's community repository of formulas:
+The Arvados formula we maintain is located in Arvados' Github account and should be considered the canonical place to download its most up-to-date version:
-* "arvados-formula":https://github.com/saltstack-formulas/arvados-formula.git
+* "arvados-formula":https://github.com/arvados/arvados-formula.git
-The @development@ version lives in our own repository
+As the Saltstack's community keeps a "repository of formulas":https://github.com/saltstack-formulas/ in Github, we also provide
-* "arvados-formula development":https://github.com/arvados/arvados-formula.git
+* "a copy of the formula":https://github.com/saltstack-formulas/arvados-formula.git
-This last one might break from time to time, as we try and add new features. Use with caution.
+there, and do our best effort to keep it in sync with ours.
-As much as possible, we try to keep it up to date, with example pillars to help you deploy Arvados.
+A @development@ branch exists which uses Arvados' development repositories. This last one might break from time to time, as we try and add new features. As much as possible, we try to keep it up to date, with example pillars to help you deploy Arvados. Use with caution.
-For those familiar with Saltstack, the process to get it deployed is similar to any other formula:
+For those familiar with Saltstack, the process to get Arvados deployed is similar to any other formula:
1. Fork/copy the formula to your Salt master host.
2. Edit the Arvados, nginx, postgres, locale and docker pillars to match your desired configuration.
diff --git a/doc/install/salt.html.textile.liquid b/doc/install/salt.html.textile.liquid
index 8f5ecc8c6..2b7aa6602 100644
--- a/doc/install/salt.html.textile.liquid
+++ b/doc/install/salt.html.textile.liquid
@@ -14,7 +14,7 @@ SPDX-License-Identifier: CC-BY-SA-3.0
h2(#introduction). Introduction
-To ease the installation of the various Arvados components, we have developed a "Saltstack":https://www.saltstack.com/ 's "arvados-formula":https://github.com/saltstack-formulas/arvados-formula which can help you get an Arvados cluster up and running.
+To ease the installation of the various Arvados components, we have developed a "Saltstack":https://www.saltstack.com/ 's "arvados-formula":https://github.com/arvados/arvados-formula which can help you get an Arvados cluster up and running.
Saltstack is a Python-based, open-source software for event-driven IT automation, remote task execution, and configuration management. It can be used in a master/minion setup or master-less.
@@ -24,6 +24,6 @@ h2(#installmethod). Choose an installation method
The salt formulas can be used in different ways. Choose one of these three options to install Arvados:
-* "Use Vagrant to install Arvados in a virtual machine":salt-vagrant.html
* "Arvados on a single host":salt-single-host.html
+* "Use Vagrant to install Arvados in a virtual machine":salt-vagrant.html
* "Arvados across multiple hosts":salt-multi-host.html
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list