[ARVADOS] updated: 2.1.0-467-g1b9d2df0d
Git user
git at public.arvados.org
Tue Feb 16 14:22:18 UTC 2021
Summary of changes:
cmd/arvados-client/cmd.go | 2 +
cmd/arvados-client/container_gateway.go | 208 +++++++++++
cmd/arvados-client/container_gateway_test.go | 82 +++++
doc/Rakefile | 31 ++
doc/_config.yml | 1 +
doc/admin/federation.html.textile.liquid | 2 +-
doc/admin/upgrading.html.textile.liquid | 9 +
doc/api/methods/containers.html.textile.liquid | 2 +
.../methods/pipeline_instances.html.textile.liquid | 2 +-
doc/architecture/federation.html.textile.liquid | 4 +-
doc/install/install-api-server.html.textile.liquid | 10 +-
...nstall-manual-prerequisites.html.textile.liquid | 8 +-
doc/install/setup-login.html.textile.liquid | 2 +-
.../cwl/arvados-vscode-training.html.md.liquid | 215 +++++++++++
doc/user/cwl/images/AddNew.png | Bin 0 -> 1060 bytes
doc/user/cwl/images/Explorer.png | Bin 0 -> 1213 bytes
doc/user/cwl/images/Extensions.png | Bin 0 -> 1256 bytes
doc/user/cwl/images/RemoteExplorer.png | Bin 0 -> 1474 bytes
doc/user/cwl/images/SSHTargets.png | Bin 0 -> 3244 bytes
.../writing-cwl-workflow.html.textile.liquid | 2 +-
go.mod | 2 +
go.sum | 4 +
lib/boot/nginx.go | 6 +-
lib/boot/passenger.go | 6 +-
lib/boot/postgresql.go | 1 +
lib/boot/supervisor.go | 22 +-
lib/config/config.default.yml | 49 ++-
lib/config/export.go | 409 +++++++++++----------
lib/config/generated_config.go | 49 ++-
lib/controller/federation/conn.go | 4 +
lib/controller/handler.go | 2 +
lib/controller/localdb/container_gateway.go | 174 +++++++++
lib/controller/localdb/container_gateway_test.go | 172 +++++++++
lib/controller/localdb/login.go | 2 +
lib/controller/localdb/login_oidc.go | 23 +-
lib/controller/localdb/login_oidc_test.go | 22 +-
lib/controller/router/router.go | 7 +
lib/controller/rpc/conn.go | 79 ++++
lib/costanalyzer/costanalyzer.go | 11 +-
lib/costanalyzer/costanalyzer_test.go | 4 +-
lib/crunchrun/container_gateway.go | 335 +++++++++++++++++
lib/crunchrun/crunchrun.go | 18 +-
lib/dispatchcloud/dispatcher_test.go | 7 +-
lib/dispatchcloud/node_size.go | 1 +
lib/dispatchcloud/node_size_test.go | 8 +-
lib/dispatchcloud/test/stub_driver.go | 3 +-
lib/dispatchcloud/worker/pool.go | 16 +-
lib/dispatchcloud/worker/pool_test.go | 5 +-
lib/dispatchcloud/worker/runner.go | 12 +-
lib/dispatchcloud/worker/worker_test.go | 2 +
lib/selfsigned/cert.go | 76 ++++
lib/selfsigned/cert_test.go | 26 ++
sdk/go/arvados/api.go | 18 +
sdk/go/arvados/client.go | 13 +
sdk/go/arvados/config.go | 28 +-
sdk/go/arvados/container.go | 46 +--
sdk/go/arvados/container_gateway.go | 74 ++++
sdk/go/arvadostest/api.go | 4 +
sdk/python/tests/run_test_server.py | 4 +
services/api/app/models/container.rb | 11 +-
...108033940_add_gateway_address_to_containers.rb} | 4 +-
...dd_interactive_session_started_to_containers.rb | 9 +
services/api/db/structure.sql | 8 +-
services/api/test/unit/container_test.rb | 2 +
services/keepstore/volume.go | 2 +-
tools/salt-install/Vagrantfile | 15 +-
....params.example.single_host_multiple_hostnames} | 8 +-
...cal.params.example.single_host_single_hostname} | 24 +-
tools/salt-install/provision.sh | 6 +
69 files changed, 2088 insertions(+), 325 deletions(-)
create mode 100644 cmd/arvados-client/container_gateway.go
create mode 100644 cmd/arvados-client/container_gateway_test.go
create mode 100644 doc/user/cwl/arvados-vscode-training.html.md.liquid
create mode 100644 doc/user/cwl/images/AddNew.png
create mode 100644 doc/user/cwl/images/Explorer.png
create mode 100644 doc/user/cwl/images/Extensions.png
create mode 100644 doc/user/cwl/images/RemoteExplorer.png
create mode 100644 doc/user/cwl/images/SSHTargets.png
create mode 100644 lib/controller/localdb/container_gateway.go
create mode 100644 lib/controller/localdb/container_gateway_test.go
create mode 100644 lib/crunchrun/container_gateway.go
create mode 100644 lib/selfsigned/cert.go
create mode 100644 lib/selfsigned/cert_test.go
create mode 100644 sdk/go/arvados/container_gateway.go
copy services/api/db/migrate/{20130523060112_add_created_by_job_task_to_job_tasks.rb => 20210108033940_add_gateway_address_to_containers.rb} (50%)
create mode 100644 services/api/db/migrate/20210126183521_add_interactive_session_started_to_containers.rb
copy tools/salt-install/{local.params.example => local.params.example.single_host_multiple_hostnames} (92%)
rename tools/salt-install/{local.params.example => local.params.example.single_host_single_hostname} (84%)
discards dda765028860f60b35682151a35cbdd86205295a (commit)
discards 86d1e1b0dd81a40ad2a404b8d0ba7ce4f768cb40 (commit)
discards 9d52d145339f979c3ff57e5145c2c6d4ab036ad3 (commit)
discards f44a0323c6a8cb88f72c6a2440185596ff579cf9 (commit)
discards 8579c7a81e432a4c6fdb0e29947a3093783e63d9 (commit)
discards d0218fb8eb1b5d87a33435a4bf09f5ea1e22af25 (commit)
discards a396085e56ad9bec41468c745a4212367323dc18 (commit)
discards d6ec1985ae59b64ca055a50f2a84f54682b27987 (commit)
discards ffd332a1849c7fef8aecfb27c442d67a34e29294 (commit)
discards c590159bf245425b96187358b53498999fa64051 (commit)
discards 6c058acc4ea63b29ef049715cc5bb104ca4e7bd7 (commit)
discards 33160dea02e7552732fe23cbcc3e061b1a5245bb (commit)
discards 2b9350438e027719deecf85bfdf9fc9ae4ef177d (commit)
discards b57db2bcf9ba4ef5f74b9b1e4c0a0788f6439658 (commit)
discards e15b1cd2d4c29381d0ccbdc33cb48e5034dab2f9 (commit)
via 1b9d2df0d5ddd92635385890d8ca60daab111170 (commit)
via 60a59df7e6359bd3c83b79554c0943900f83bc4d (commit)
via 963b5ff60a6f27b5834113de5fb0747f8ab52142 (commit)
via 7a35170a489debb2a42753d15c4d7935a601c6e1 (commit)
via 67151930898579a5e7af6c772571a43fcffa4a5d (commit)
via 9aa498ee5e857045ef26ae6ba9e68c450effddf7 (commit)
via de02a06e369216376ff642bbc52e79214dc37957 (commit)
via efd412f4a22839eb9a7d7ce52f59cfff1b77e4b0 (commit)
via d10dcb434de0bdb16c047da9e45c19ba2ef9b1c4 (commit)
via 8195246d1e71c8e1801f85c95e33402775099e7d (commit)
via 55d74d65d3dfc4cd1aafff3ac32fb3179f66cab6 (commit)
via f023eb5138f8886820f33901b46b67ba9a0d24a2 (commit)
via a8ccabb165c144229ed47128843ea123778565af (commit)
via c0bfa951ebd8e56c4feb0443a1b414ffe8eb63c1 (commit)
via 9074ec89ae2bd38072cd1296be788168be9e3618 (commit)
via b3852e1fd52fa0471555004f14e8e9448480c935 (commit)
via 491ff06465da8b5dc3678dfb44622a824e583488 (commit)
via ccb5293993df4d535b3ca1e3224a5a146d8f90c2 (commit)
via b5884b515a7fe6255761020cdd39d450db6d603b (commit)
via a999ea55a6fdfabeca12c8d8db24214698ae2908 (commit)
via 8685bdc41012f1623cc02b573e27439fdf314799 (commit)
via 5dc622e37805f511b04eb66557441b28fba13b80 (commit)
via 4dc6c2c37fba1c59e3e57ad47a1ee55d0593ed04 (commit)
via 20bd61710abe56bb63df4b7a906d04c7c27f765a (commit)
via 046cc7b98a96f76cbc1e150bd0468f4238f8dfe9 (commit)
via 8b1aca5c3415bfee3b4bc242596e1ee68ddef354 (commit)
via 5ccc1a94d1621e319990a4362c131d5ab776c478 (commit)
via ea194f11512bbe272fd9a2767c43870b25216de7 (commit)
via 3639e67a0d23f327be3032fcdf57dff555e7a927 (commit)
via 279dd72bcd41f38e4bcd9ee87d7e8803903f4bca (commit)
via 92935cb2cea3a5bae6da115b0119a8da5952b7e4 (commit)
via 94a62edb315ec297a02fb6c9a2016bcaa17fac9b (commit)
via 42bf31f017a009585eaac2fe44a83b2596b3e5c8 (commit)
via 9135120ba24f505aa47655b6f3f6dca3450215d5 (commit)
via ae44eb7e4d96e44ca3dc0481cef7cef354a23292 (commit)
via f30c8ed35e3e1ad7cb3cb51fc6d83f56a04ae8de (commit)
via 69260a600b0cf8878292169c8537951e94de9863 (commit)
via 0810be42fe3bc0215dd9b6edaad0a9e4ef6802a6 (commit)
via fdaa80f645f6eb32cb770beefd0a1ba0c864bf84 (commit)
via e85a787d224545accc7ee1e8cdda0145dd9ac806 (commit)
via 5e99f2c5cb423b08b18ba8b7ba9c25035b9c689e (commit)
via 0bcd1ca37a225f37dd52081070c91aa2ba68d49a (commit)
via dfa749b3eacff1da33fe470a2a4af3e291defdcf (commit)
via 7079cf10005d72b4866379250baa60840721ace5 (commit)
via f81aae076a78fec5dbafb30e410c596aa2b96b92 (commit)
via d106841c5b3e10f40c903923ea3f6121985d5502 (commit)
via 984703527b7205372d28a9c27e8356986d3ba278 (commit)
via 60a72f91316ab1d878420adceb98e585c0e3b72f (commit)
via 38a0f5e77f2190487d03d6538337d3b7055fd1e8 (commit)
via 41a052d1faf57249eeb86674256372225ff9b7ed (commit)
via ba6b58b83c0c3a7a5f31b551e430a7b983fb998d (commit)
via 6a64c9760267bd33f6058eac9c303205c1c21be8 (commit)
via 77c8223f5ddd64cff2b08d0857749644c474946f (commit)
via 0945afa5523fb45f827750e4d1700df4ff222295 (commit)
via 28e5a3c88c43be042332ec5d03b65a00509da978 (commit)
via dba001d209d9658cdf0adec182376137c5d65244 (commit)
via e6cac792d173f4b4073420aabbb1a506ae966c94 (commit)
via 87694b32da9e3c8a92f7eedb5c71299d199023fb (commit)
via 82a9595a6a35c648fff62c2497d858f1ab062578 (commit)
via 0d4be3c57b4e6f5be9699cc9fe445c2d5af0174e (commit)
via b36492a0a569b9116ccf156430c901f4002d8814 (commit)
via 3c972761eada11023194a48b619d451d3f3c1854 (commit)
via 4046a93e9f5b47771b10cd48c4357a452ed6dc6a (commit)
via 88bb9b9fc4392f4a3514ae59e3ffd454d3ce90a8 (commit)
via 7975d01ef884809f05f380b0e8f275ef0418808b (commit)
via dad86790c5f3edbcf38702542ba46cb7a9e7f42a (commit)
via 4f45c40368659d6dbf357cecfc37622c1722090a (commit)
via 35b422b074dd06dfea090a65b6072b09600302ff (commit)
via 452fbd0bf14678f1ceb8e60a8864693e062b89b6 (commit)
via 8c283c6262809df6f9db1aa176a1a8a5e95a717b (commit)
via 879bd007e5133c3124cb91a0fb660dfda9cf4ace (commit)
via a80122e544ddf72fe31d02398d914089b300d1c9 (commit)
via f2c6e467c4cfd079b00070ac4f50b551b6ee3bdf (commit)
via 8439ad9df4a6d9b28bbed985bf87599f2d1b3820 (commit)
via 5d05dd3ef4822dfb3b476777ba1aacaafed8278d (commit)
via bdd9b68e133bb3b3ef7914893ad15f311ce2de24 (commit)
via 0a6c5d8a8758df0395e40b3cf1d125e59d6c88dd (commit)
via 5c34cfad641b76ffaa43be8234af1da9d5736ec2 (commit)
via 03141ccc3015a04f2f171b5a532a9dfd57b8bcd6 (commit)
via a2c539196d3b76b466039dfd00e7595ec9dd6abc (commit)
via 4cc07d6ab4ae16bd6bc69b2c8022414d66ee4113 (commit)
via 6be5fe90dc9a5452b432176bfa83cba83070f8cf (commit)
via fed1d9f5e7a442abb6b2e86114e8c3d05cff5329 (commit)
via 2c3418d59cd27806322e07519ef9483c2cb433c2 (commit)
via 0a39317d5ec49ad439833df0a70965394cafb6e8 (commit)
via 56e130608f8977d20b21c54f6ab8973d71e045a0 (commit)
This update added new revisions after undoing existing revisions. That is
to say, the old revision is not a strict subset of the new revision. This
situation occurs when you --force push a change and generate a repository
containing something like this:
* -- * -- B -- O -- O -- O (dda765028860f60b35682151a35cbdd86205295a)
\
N -- N -- N (1b9d2df0d5ddd92635385890d8ca60daab111170)
When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 1b9d2df0d5ddd92635385890d8ca60daab111170
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 16 11:21:20 2021 -0300
fix(provision): force user to properly set cluster & domain parameters
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/Vagrantfile b/tools/salt-install/Vagrantfile
index 666c6c48f..0b75cac84 100644
--- a/tools/salt-install/Vagrantfile
+++ b/tools/salt-install/Vagrantfile
@@ -31,14 +31,15 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# WEBSOCKET
arv.vm.network "forwarded_port", guest: 8002, host: 8002
arv.vm.provision "shell",
- inline: "sed 's#fixme#harpo#g;
+ inline: "sed 's#cluster_fixme_or_this_wont_work#harpo#g;
+ s#domain_fixme_or_this_wont_work#local#g;
s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=8443#g' \
- /vagrant/local.params.example > /vagrant/local.params.single_host_multiple_hostnames"
+ /vagrant/local.params.example.single_host_multiple_hostnames > /tmp/local.params.single_host_multiple_hostnames"
arv.vm.provision "shell",
path: "provision.sh",
args: [
# "--debug",
- "--config /vagrant/local.params.single_host_multiple_hostnames",
+ "--config /tmp/local.params.single_host_multiple_hostnames",
"--test",
"--vagrant"
].join(" ")
@@ -63,9 +64,9 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
## arv.vm.network "forwarded_port", guest: 14202, host: 14202
## arv.vm.network "forwarded_port", guest: 18002, host: 18002
## arv.vm.provision "shell",
- ## inline: "sed 's#HOSTNAME_EXT=\"\"#HOSTNAME_EXT=\"zeppo.local.cluster\"#g;
- ## s#CLUSTER=\"fixme\"#CLUSTER=\"zeppo\"#g;
- ## s#DOMAIN=\"some.domain\"#DOMAIN=\"local.cluster\"#g;
+ ## inline: "sed 's#HOSTNAME_EXT=\"\"#HOSTNAME_EXT=\"zeppo.local\"#g;
+ ## s#cluster_fixme_or_this_wont_work#harpo#g;
+ ## s#domain_fixme_or_this_wont_work#local#g;
## s#CONFIG_DIR=\"config_examples/single_host/multiple_hostnames\"#CONFIG_DIR=\"config_examples/single_host/single_hostname\"#g;
## s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=9443#g;
## s#KEEP_EXT_SSL_PORT=25101#KEEP_EXT_SSL_PORT=35101#g;
@@ -74,7 +75,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
## s#WEBSOCKET_EXT_SSL_PORT=8002#WEBSOCKET_EXT_SSL_PORT=18002#g;
## s#WORKBENCH1_EXT_SSL_PORT=443#WORKBENCH1_EXT_SSL_PORT=9444#g;
## s#WORKBENCH2_EXT_SSL_PORT=3001#WORKBENCH2_EXT_SSL_PORT=9445#g;' \
- ## /vagrant/local.params.example > /vagrant/local.params.single_host_single_hostname"
+ ## /vagrant/local.params.example.single_host_single_hostnames > /tmp/local.params.single_host_single_hostnames"
## arv.vm.provision "shell",
## path: "provision.sh",
## args: [
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example.single_host_multiple_hostnames
similarity index 92%
copy from tools/salt-install/local.params.example
copy to tools/salt-install/local.params.example.single_host_multiple_hostnames
index 88d6a75d6..b1ada8525 100644
--- a/tools/salt-install/local.params.example
+++ b/tools/salt-install/local.params.example.single_host_multiple_hostnames
@@ -5,11 +5,11 @@
# These are the basic parameters to configure the installation
-# The 5 letters name you want to give your cluster
-CLUSTER="fixme"
+# The FIVE ALPHANUMERIC CHARACTERS name you want to give your cluster
+CLUSTER="cluster_fixme_or_this_wont_work"
# The domainname you want tou give to your cluster's hosts
-DOMAIN="some.domain"
+DOMAIN="domain_fixme_or_this_wont_work"
# When setting the cluster in a single host, you can use a single hostname
# to access all the instances. When using virtualization (ie AWS), this should be
@@ -36,7 +36,7 @@ INITIAL_USER="admin"
# If not specified, the initial user email will be composed as
# INITIAL_USER at CLUSTER.DOMAIN
-INITIAL_USER_EMAIL="admin at fixme.localdomain"
+INITIAL_USER_EMAIL="admincluster_ at cluster_fixme_or_this_wont_work.domain_fixme_or_this_wont_work"
INITIAL_USER_PASSWORD="password"
# YOU SHOULD CHANGE THESE TO SOME RANDOM STRINGS
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example.single_host_single_hostname
similarity index 84%
rename from tools/salt-install/local.params.example
rename to tools/salt-install/local.params.example.single_host_single_hostname
index 88d6a75d6..31645e3d4 100644
--- a/tools/salt-install/local.params.example
+++ b/tools/salt-install/local.params.example.single_host_single_hostname
@@ -5,11 +5,11 @@
# These are the basic parameters to configure the installation
-# The 5 letters name you want to give your cluster
-CLUSTER="fixme"
+# The FIVE ALPHANUMERIC CHARACTERS name you want to give your cluster
+CLUSTER="cluster_fixme_or_this_wont_work"
# The domainname you want tou give to your cluster's hosts
-DOMAIN="some.domain"
+DOMAIN="domain_fixme_or_this_wont_work"
# When setting the cluster in a single host, you can use a single hostname
# to access all the instances. When using virtualization (ie AWS), this should be
@@ -23,20 +23,20 @@ HOSTNAME_INT="127.0.1.1"
# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
# You can point it to another port if desired
# In Vagrant, make sure it matches what you set in the Vagrantfile (8443)
-CONTROLLER_EXT_SSL_PORT=443
-KEEP_EXT_SSL_PORT=25101
+CONTROLLER_EXT_SSL_PORT=9443
+KEEP_EXT_SSL_PORT=35101
# Both for collections and downloads
-KEEPWEB_EXT_SSL_PORT=9002
-WEBSHELL_EXT_SSL_PORT=4202
-WEBSOCKET_EXT_SSL_PORT=8002
-WORKBENCH1_EXT_SSL_PORT=443
-WORKBENCH2_EXT_SSL_PORT=3001
+KEEPWEB_EXT_SSL_PORT=11002
+WEBSHELL_EXT_SSL_PORT=14202
+WEBSOCKET_EXT_SSL_PORT=18002
+WORKBENCH1_EXT_SSL_PORT=9444
+WORKBENCH2_EXT_SSL_PORT=9445
INITIAL_USER="admin"
# If not specified, the initial user email will be composed as
# INITIAL_USER at CLUSTER.DOMAIN
-INITIAL_USER_EMAIL="admin at fixme.localdomain"
+INITIAL_USER_EMAIL="admincluster_ at cluster_fixme_or_this_wont_work.domain_fixme_or_this_wont_work"
INITIAL_USER_PASSWORD="password"
# YOU SHOULD CHANGE THESE TO SOME RANDOM STRINGS
@@ -50,7 +50,7 @@ WORKBENCH_SECRET_KEY=workbenchsecretkeymushaveatleast32characters
# There are a few examples under 'config_examples'. If you don't change this
# variable, the single_host, multiple_hostnames config will be used
# CONFIG_DIR="config_examples/single_host/single_hostname"
-CONFIG_DIR="config_examples/single_host/multiple_hostnames"
+CONFIG_DIR="config_examples/single_host/single_hostname"
# Extra states to pply. iIf you use your own subdir, change this value accordingly
EXTRA_STATES_DIR="${F_DIR}/arvados-formula/test/salt/states/examples/single_host"
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 9b19854d0..6f0b84cbd 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -151,6 +151,12 @@ else
exit 1
fi
+if grep -q 'fixme_or_this_wont_work' ${CONFIG_FILE} ; then
+ echo >&2 "The config file ${CONFIG_FILE} has some parameters that need to be modified."
+ echo >&2 "Please, fix them and re-run the provision script."
+ exit 1
+fi
+
if ! grep -E '^[[:alnum:]]{5}$' <<<${CLUSTER} ; then
echo >&2 "ERROR: <CLUSTER> must be exactly 5 alphanumeric characters long"
echo >&2 "Fix the cluster name in the 'local.params' file and re-run the provision script"
commit 60a59df7e6359bd3c83b79554c0943900f83bc4d
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 15:06:14 2021 -0300
refactor(provision): naming consistency
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index f74f001cb..9b19854d0 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -107,8 +107,12 @@ TESTS_DIR="tests"
CLUSTER=""
DOMAIN=""
+
+# Hostnames/IPs used for single-host deploys
HOSTNAME_EXT=""
HOSTNAME_INT="127.0.1.1"
+
+# Initial user setup
INITIAL_USER=""
INITIAL_USER_EMAIL=""
INITIAL_USER_PASSWORD=""
@@ -196,10 +200,10 @@ fi
if [ "x${VAGRANT}" = "xyes" ]; then
SOURCE_PILLARS_DIR="/vagrant/${CONFIG_DIR}/pillars"
- TESTS_DIR="/vagrant/${TESTS_DIR}"
+ SOURCE_TESTS_DIR="/vagrant/${TESTS_DIR}"
else
SOURCE_PILLARS_DIR="${SCRIPT_DIR}/${CONFIG_DIR}/pillars"
- TESTS_DIR="${SCRIPT_DIR}/${TESTS_DIR}"
+ SOURCE_TESTS_DIR="${SCRIPT_DIR}/${TESTS_DIR}"
fi
SOURCE_STATES_DIR="${EXTRA_STATES_DIR}"
@@ -233,7 +237,7 @@ done
mkdir -p /tmp/cluster_tests
# Replace cluster and domain name in the test files
-for f in "${TESTS_DIR}"/*; do
+for f in "${SOURCE_TESTS_DIR}"/*; do
sed "s/__CLUSTER__/${CLUSTER}/g;
s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
s/__DOMAIN__/${DOMAIN}/g;
commit 963b5ff60a6f27b5834113de5fb0747f8ab52142
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 12:01:17 2021 -0300
fix(provision): add missing roles sections
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index f91d67d69..f74f001cb 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -346,7 +346,7 @@ else
grep -q "nginx_passenger" ${P_DIR}/top.sls || echo " - nginx_passenger" >> ${P_DIR}/top.sls
grep -q "nginx_${R}_configuration" ${P_DIR}/top.sls || echo " - nginx_${R}_configuration" >> ${P_DIR}/top.sls
;;
- "workbench" | "workbench2" | "keepweb" | "keepproxy")
+ "controller" | "websocket" | "workbench" | "workbench2" | "keepweb" | "keepproxy")
# States
grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
@@ -369,6 +369,12 @@ else
# Pillars
# ATM, no specific pillar needed
;;
+ "keepstore")
+ # States
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ # Pillars
+ # ATM, no specific pillar needed
+ ;;
*)
echo "Unknown role ${R}"
exit 1
commit 7a35170a489debb2a42753d15c4d7935a601c6e1
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 11:53:08 2021 -0300
fix(provision): make the salt-call binary check compatible with sh
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 08d17090c..f91d67d69 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -156,7 +156,7 @@ fi
apt-get update
apt-get install -y curl git jq
-if [ which salt-call ]; then
+if which salt-call; then
echo "Salt already installed"
else
curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
commit 67151930898579a5e7af6c772571a43fcffa4a5d
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 11:31:24 2021 -0300
fix(provision): add only specific pillars for each role
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index d30ff88a4..08d17090c 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -277,6 +277,9 @@ if [ -d "${SOURCE_STATES_DIR}" ]; then
fi
# Now, we build the SALT states/pillars trees
+# As we need to separate both states and pillars in case we want specific
+# roles, we iterate on both at the same time
+
# States
cat > ${S_DIR}/top.sls << EOFTSLS
base:
@@ -284,7 +287,16 @@ base:
- locale
EOFTSLS
-if [ -d "${SOURCE_STATES_DIR}" ]; then
+# Pillars
+cat > ${P_DIR}/top.sls << EOFPSLS
+base:
+ '*':
+ - locale
+ - arvados
+EOFPSLS
+
+# States, extra states
+if [ -d "${F_DIR}"/extra/extra ]; then
for f in "${F_DIR}"/extra/extra/*.sls; do
echo " - extra.$(basename ${f} | sed 's/.sls$//g')" >> ${S_DIR}/top.sls
done
@@ -293,58 +305,78 @@ fi
# If we want specific roles for a node, just add the desired states
# and its dependencies
if [ -z "${ROLES}" ]; then
- echo ' - nginx.passenger' >> ${S_DIR}/top.sls
- echo ' - postgres' >> ${S_DIR}/top.sls
- echo ' - docker' >> ${S_DIR}/top.sls
- echo ' - arvados' >> ${S_DIR}/top.sls
+ # States
+ echo " - nginx.passenger" >> ${S_DIR}/top.sls
+ echo " - postgres" >> ${S_DIR}/top.sls
+ echo " - docker" >> ${S_DIR}/top.sls
+ echo " - arvados" >> ${S_DIR}/top.sls
+
+ # Pillars
+ echo " - docker" >> ${P_DIR}/top.sls
+ echo " - nginx_api_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_controller_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_keepproxy_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_keepweb_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_passenger" >> ${P_DIR}/top.sls
+ echo " - nginx_websocket_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_webshell_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_workbench2_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_workbench_configuration" >> ${P_DIR}/top.sls
+ echo " - postgresql" >> ${P_DIR}/top.sls
else
# If we add individual roles, make sure we add the repo first
echo " - arvados.repo" >> ${S_DIR}/top.sls
for R in ${ROLES}; do
case "${R}" in
"database")
+ # States
echo " - postgres" >> ${S_DIR}/top.sls
+ # Pillars
+ echo ' - postgresql' >> ${P_DIR}/top.sls
;;
"api")
+ # States
# FIXME: https://dev.arvados.org/issues/17352
grep -q "postgres.client" ${S_DIR}/top.sls || echo " - postgres.client" >> ${S_DIR}/top.sls
grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ # Pillars
+ grep -q "docker" ${P_DIR}/top.sls || echo " - docker" >> ${P_DIR}/top.sls
+ grep -q "postgresql" ${P_DIR}/top.sls || echo " - postgresql" >> ${P_DIR}/top.sls
+ grep -q "nginx_passenger" ${P_DIR}/top.sls || echo " - nginx_passenger" >> ${P_DIR}/top.sls
+ grep -q "nginx_${R}_configuration" ${P_DIR}/top.sls || echo " - nginx_${R}_configuration" >> ${P_DIR}/top.sls
;;
"workbench" | "workbench2" | "keepweb" | "keepproxy")
+ # States
grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ # Pillars
+ grep -q "nginx_passenger" ${P_DIR}/top.sls || echo " - nginx_passenger" >> ${P_DIR}/top.sls
+ grep -q "nginx_${R}_configuration" ${P_DIR}/top.sls || echo " - nginx_${R}_configuration" >> ${P_DIR}/top.sls
;;
- "shell" | "dispatcher")
- grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ "shell")
+ # States
+ grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ # Pillars
+ grep -q "" ${P_DIR}/top.sls || echo " - docker" >> ${P_DIR}/top.sls
+ grep -q "nginx_webshell_configuration" ${P_DIR}/top.sls || echo " - nginx_webshell_configuration" >> ${P_DIR}/top.sls
+ ;;
+ "dispatcher")
+ # States
+ grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ # Pillars
+ # ATM, no specific pillar needed
;;
*)
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ echo "Unknown role ${R}"
+ exit 1
;;
esac
done
fi
-# Pillars
-cat > ${P_DIR}/top.sls << EOFPSLS
-base:
- '*':
- - arvados
- - docker
- - locale
- - nginx_api_configuration
- - nginx_controller_configuration
- - nginx_keepproxy_configuration
- - nginx_keepweb_configuration
- - nginx_passenger
- - nginx_websocket_configuration
- - nginx_webshell_configuration
- - nginx_workbench2_configuration
- - nginx_workbench_configuration
- - postgresql
-EOFPSLS
-
# FIXME! #16992 Temporary fix for psql call in arvados-api-server
if [ -e /root/.psqlrc ]; then
if ! ( grep 'pset pager off' /root/.psqlrc ); then
commit 9aa498ee5e857045ef26ae6ba9e68c450effddf7
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 09:35:19 2021 -0300
fix(provision): remove deprecated parameter
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
index 710c4da86..de9cd9648 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
@@ -80,7 +80,6 @@ arvados:
tokens:
system_root: __SYSTEM_ROOT_TOKEN__
management: __MANAGEMENT_TOKEN__
- rails_secret: __RAILS_SECRET_TOKEN__
anonymous_user: __ANONYMOUS_USER_TOKEN__
### KEYS
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
index f3d2bcb9e..31d3a0d50 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
@@ -80,7 +80,6 @@ arvados:
tokens:
system_root: __SYSTEM_ROOT_TOKEN__
management: __MANAGEMENT_TOKEN__
- rails_secret: __RAILS_SECRET_TOKEN__
anonymous_user: __ANONYMOUS_USER_TOKEN__
### KEYS
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example
index d02ba95c3..88d6a75d6 100644
--- a/tools/salt-install/local.params.example
+++ b/tools/salt-install/local.params.example
@@ -43,7 +43,6 @@ INITIAL_USER_PASSWORD="password"
BLOB_SIGNING_KEY=blobsigningkeymushaveatleast32characters
MANAGEMENT_TOKEN=managementtokenmushaveatleast32characters
SYSTEM_ROOT_TOKEN=systemroottokenmushaveatleast32characters
-RAILS_SECRET_TOKEN=railssecrettokenmushaveatleast32characters
ANONYMOUS_USER_TOKEN=anonymoususertokenmushaveatleast32characters
WORKBENCH_SECRET_KEY=workbenchsecretkeymushaveatleast32characters
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 1d9ae0fe9..d30ff88a4 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -220,7 +220,6 @@ for f in "${SOURCE_PILLARS_DIR}"/*; do
s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
- s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
s/__RELEASE__/${RELEASE}/g;
s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
s/__VERSION__/${VERSION}/g;
@@ -265,7 +264,6 @@ if [ -d "${SOURCE_STATES_DIR}" ]; then
s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
- s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
s/__RELEASE__/${RELEASE}/g;
s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
s/__VERSION__/${VERSION}/g;
commit de02a06e369216376ff642bbc52e79214dc37957
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 08:05:26 2021 -0300
fix(provision): check salt-call is installed
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index f5b986d1f..1d9ae0fe9 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -156,12 +156,12 @@ fi
apt-get update
apt-get install -y curl git jq
-dpkg -l |grep salt-minion
-if [ ${?} -eq 0 ]; then
+if [ which salt-call ]; then
echo "Salt already installed"
else
curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
sh /tmp/bootstrap_salt.sh -XdfP -x python3
+ /bin/systemctl stop salt-minion.service
/bin/systemctl disable salt-minion.service
fi
commit efd412f4a22839eb9a7d7ce52f59cfff1b77e4b0
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 07:49:50 2021 -0300
fix(provision): add missing postgres dependency for arvados-api-server
refs #17246 & #17352
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 486f43057..f5b986d1f 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -307,11 +307,17 @@ else
"database")
echo " - postgres" >> ${S_DIR}/top.sls
;;
- "api","workbench","workbench2","keepweb","keepproxy")
+ "api")
+ # FIXME: https://dev.arvados.org/issues/17352
+ grep -q "postgres.client" ${S_DIR}/top.sls || echo " - postgres.client" >> ${S_DIR}/top.sls
grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
echo " - arvados.${R}" >> ${S_DIR}/top.sls
;;
- "shell","dispatcher")
+ "workbench" | "workbench2" | "keepweb" | "keepproxy")
+ grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ "shell" | "dispatcher")
grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
echo " - arvados.${R}" >> ${S_DIR}/top.sls
;;
commit d10dcb434de0bdb16c047da9e45c19ba2ef9b1c4
Author: Javier Bértoli <jbertoli at curii.com>
Date: Thu Feb 4 14:35:09 2021 -0300
fix(provision): refactor single host architectures
Allow to use a single-host/single-hostname or single-host/multiple-hostnames setup
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/Vagrantfile b/tools/salt-install/Vagrantfile
index 60f57ca66..666c6c48f 100644
--- a/tools/salt-install/Vagrantfile
+++ b/tools/salt-install/Vagrantfile
@@ -11,9 +11,10 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.ssh.insert_key = false
config.ssh.forward_x11 = true
- config.vm.define "arvados" do |arv|
+ # A single_host multiple_hostnames example
+ config.vm.define "arvados-sh-mn" do |arv|
arv.vm.box = "bento/debian-10"
- arv.vm.hostname = "vagrant.local"
+ arv.vm.hostname = "harpo.local"
# CPU/RAM
config.vm.provider :virtualbox do |v|
v.memory = 2048
@@ -21,23 +22,66 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
end
# Networking
+ # WEBUI PORT
arv.vm.network "forwarded_port", guest: 8443, host: 8443
- arv.vm.network "forwarded_port", guest: 25100, host: 25100
+ # KEEPPROXY
+ arv.vm.network "forwarded_port", guest: 25101, host: 25101
+ # KEEPWEB
arv.vm.network "forwarded_port", guest: 9002, host: 9002
- arv.vm.network "forwarded_port", guest: 9000, host: 9000
- arv.vm.network "forwarded_port", guest: 8900, host: 8900
+ # WEBSOCKET
arv.vm.network "forwarded_port", guest: 8002, host: 8002
- arv.vm.network "forwarded_port", guest: 8001, host: 8001
- arv.vm.network "forwarded_port", guest: 8000, host: 8000
- arv.vm.network "forwarded_port", guest: 3001, host: 3001
+ arv.vm.provision "shell",
+ inline: "sed 's#fixme#harpo#g;
+ s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=8443#g' \
+ /vagrant/local.params.example > /vagrant/local.params.single_host_multiple_hostnames"
arv.vm.provision "shell",
path: "provision.sh",
args: [
# "--debug",
- "--config /vagrant/local.params",
+ "--config /vagrant/local.params.single_host_multiple_hostnames",
"--test",
- "--vagrant",
- "--ssl-port=8443"
+ "--vagrant"
].join(" ")
end
+
+ ## # A single_host single_hostname example
+ ## config.vm.define "arvados-sh-sn" do |arv|
+ ## arv.vm.box = "bento/debian-10"
+ ## arv.vm.hostname = "zeppo.local"
+ ## # CPU/RAM
+ ## config.vm.provider :virtualbox do |v|
+ ## v.memory = 2048
+ ## v.cpus = 2
+ ## end
+
+ ## # Networking
+ ## arv.vm.network "forwarded_port", guest: 9443, host: 9443
+ ## arv.vm.network "forwarded_port", guest: 9444, host: 9444
+ ## arv.vm.network "forwarded_port", guest: 9445, host: 9445
+ ## arv.vm.network "forwarded_port", guest: 35101, host: 35101
+ ## arv.vm.network "forwarded_port", guest: 10002, host: 10002
+ ## arv.vm.network "forwarded_port", guest: 14202, host: 14202
+ ## arv.vm.network "forwarded_port", guest: 18002, host: 18002
+ ## arv.vm.provision "shell",
+ ## inline: "sed 's#HOSTNAME_EXT=\"\"#HOSTNAME_EXT=\"zeppo.local.cluster\"#g;
+ ## s#CLUSTER=\"fixme\"#CLUSTER=\"zeppo\"#g;
+ ## s#DOMAIN=\"some.domain\"#DOMAIN=\"local.cluster\"#g;
+ ## s#CONFIG_DIR=\"config_examples/single_host/multiple_hostnames\"#CONFIG_DIR=\"config_examples/single_host/single_hostname\"#g;
+ ## s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=9443#g;
+ ## s#KEEP_EXT_SSL_PORT=25101#KEEP_EXT_SSL_PORT=35101#g;
+ ## s#KEEPWEB_EXT_SSL_PORT=9002#KEEPWEB_EXT_SSL_PORT=11002#g;
+ ## s#WEBSHELL_EXT_SSL_PORT=4202#WEBSHELL_EXT_SSL_PORT=14202#g;
+ ## s#WEBSOCKET_EXT_SSL_PORT=8002#WEBSOCKET_EXT_SSL_PORT=18002#g;
+ ## s#WORKBENCH1_EXT_SSL_PORT=443#WORKBENCH1_EXT_SSL_PORT=9444#g;
+ ## s#WORKBENCH2_EXT_SSL_PORT=3001#WORKBENCH2_EXT_SSL_PORT=9445#g;' \
+ ## /vagrant/local.params.example > /vagrant/local.params.single_host_single_hostname"
+ ## arv.vm.provision "shell",
+ ## path: "provision.sh",
+ ## args: [
+ ## # "--debug",
+ ## "--config /vagrant/local.params.single_host_single_hostname",
+ ## "--test",
+ ## "--vagrant"
+ ## ].join(" ")
+ ## end
end
diff --git a/tools/salt-install/Vagrantfile.single_host_single_hostname.example b/tools/salt-install/Vagrantfile.single_host_single_hostname.example
new file mode 100644
index 000000000..666c6c48f
--- /dev/null
+++ b/tools/salt-install/Vagrantfile.single_host_single_hostname.example
@@ -0,0 +1,87 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+# Vagrantfile API/syntax version. Don"t touch unless you know what you"re doing!
+VAGRANTFILE_API_VERSION = "2".freeze
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+ config.ssh.insert_key = false
+ config.ssh.forward_x11 = true
+
+ # A single_host multiple_hostnames example
+ config.vm.define "arvados-sh-mn" do |arv|
+ arv.vm.box = "bento/debian-10"
+ arv.vm.hostname = "harpo.local"
+ # CPU/RAM
+ config.vm.provider :virtualbox do |v|
+ v.memory = 2048
+ v.cpus = 2
+ end
+
+ # Networking
+ # WEBUI PORT
+ arv.vm.network "forwarded_port", guest: 8443, host: 8443
+ # KEEPPROXY
+ arv.vm.network "forwarded_port", guest: 25101, host: 25101
+ # KEEPWEB
+ arv.vm.network "forwarded_port", guest: 9002, host: 9002
+ # WEBSOCKET
+ arv.vm.network "forwarded_port", guest: 8002, host: 8002
+ arv.vm.provision "shell",
+ inline: "sed 's#fixme#harpo#g;
+ s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=8443#g' \
+ /vagrant/local.params.example > /vagrant/local.params.single_host_multiple_hostnames"
+ arv.vm.provision "shell",
+ path: "provision.sh",
+ args: [
+ # "--debug",
+ "--config /vagrant/local.params.single_host_multiple_hostnames",
+ "--test",
+ "--vagrant"
+ ].join(" ")
+ end
+
+ ## # A single_host single_hostname example
+ ## config.vm.define "arvados-sh-sn" do |arv|
+ ## arv.vm.box = "bento/debian-10"
+ ## arv.vm.hostname = "zeppo.local"
+ ## # CPU/RAM
+ ## config.vm.provider :virtualbox do |v|
+ ## v.memory = 2048
+ ## v.cpus = 2
+ ## end
+
+ ## # Networking
+ ## arv.vm.network "forwarded_port", guest: 9443, host: 9443
+ ## arv.vm.network "forwarded_port", guest: 9444, host: 9444
+ ## arv.vm.network "forwarded_port", guest: 9445, host: 9445
+ ## arv.vm.network "forwarded_port", guest: 35101, host: 35101
+ ## arv.vm.network "forwarded_port", guest: 10002, host: 10002
+ ## arv.vm.network "forwarded_port", guest: 14202, host: 14202
+ ## arv.vm.network "forwarded_port", guest: 18002, host: 18002
+ ## arv.vm.provision "shell",
+ ## inline: "sed 's#HOSTNAME_EXT=\"\"#HOSTNAME_EXT=\"zeppo.local.cluster\"#g;
+ ## s#CLUSTER=\"fixme\"#CLUSTER=\"zeppo\"#g;
+ ## s#DOMAIN=\"some.domain\"#DOMAIN=\"local.cluster\"#g;
+ ## s#CONFIG_DIR=\"config_examples/single_host/multiple_hostnames\"#CONFIG_DIR=\"config_examples/single_host/single_hostname\"#g;
+ ## s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=9443#g;
+ ## s#KEEP_EXT_SSL_PORT=25101#KEEP_EXT_SSL_PORT=35101#g;
+ ## s#KEEPWEB_EXT_SSL_PORT=9002#KEEPWEB_EXT_SSL_PORT=11002#g;
+ ## s#WEBSHELL_EXT_SSL_PORT=4202#WEBSHELL_EXT_SSL_PORT=14202#g;
+ ## s#WEBSOCKET_EXT_SSL_PORT=8002#WEBSOCKET_EXT_SSL_PORT=18002#g;
+ ## s#WORKBENCH1_EXT_SSL_PORT=443#WORKBENCH1_EXT_SSL_PORT=9444#g;
+ ## s#WORKBENCH2_EXT_SSL_PORT=3001#WORKBENCH2_EXT_SSL_PORT=9445#g;' \
+ ## /vagrant/local.params.example > /vagrant/local.params.single_host_single_hostname"
+ ## arv.vm.provision "shell",
+ ## path: "provision.sh",
+ ## args: [
+ ## # "--debug",
+ ## "--config /vagrant/local.params.single_host_single_hostname",
+ ## "--test",
+ ## "--vagrant"
+ ## ].join(" ")
+ ## end
+end
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
index 6c6dec26f..710c4da86 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
@@ -118,7 +118,7 @@ arvados:
Services:
Controller:
- ExternalURL: 'https://__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
InternalURLs:
'http://controller.internal:8003': {}
DispatchCloud:
@@ -128,7 +128,7 @@ arvados:
InternalURLs:
'http://__CLUSTER__.__DOMAIN__:9005': {}
Keepproxy:
- ExternalURL: 'https://keep.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://keep.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
InternalURLs:
'http://keep.internal:25100': {}
Keepstore:
@@ -138,18 +138,18 @@ arvados:
InternalURLs:
'http://api.internal:8004': {}
WebDAV:
- ExternalURL: 'https://collections.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://collections.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
InternalURLs:
'http://collections.internal:9002': {}
WebDAVDownload:
- ExternalURL: 'https://download.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://download.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
WebShell:
- ExternalURL: 'https://webshell.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://webshell.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
Websocket:
ExternalURL: 'wss://ws.__CLUSTER__.__DOMAIN__/websocket'
InternalURLs:
'http://ws.internal:8005': {}
Workbench1:
- ExternalURL: 'https://workbench.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://workbench.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
Workbench2:
- ExternalURL: 'https://workbench2.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://workbench2.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls
index 00c3b3a13..3adf0580a 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls
@@ -40,7 +40,7 @@ nginx:
- server:
- server_name: __CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://controller_upstream'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls
index 6554f79a7..2d8922df9 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls
@@ -36,7 +36,7 @@ nginx:
- server:
- server_name: keep.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://keepproxy_upstream'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls
index cc871b9da..d180a3bad 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls
@@ -37,7 +37,7 @@ nginx:
- server:
- server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://collections_downloads_upstream'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls
index a0756b7ce..e75f04434 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls
@@ -37,7 +37,7 @@ nginx:
- server:
- server_name: webshell.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /shell.__CLUSTER__.__DOMAIN__:
- proxy_pass: 'http://webshell_upstream'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls
index ebe03f733..3a354ac29 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls
@@ -36,7 +36,7 @@ nginx:
- server:
- server_name: ws.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://websocket_upstream'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls
index 8930be408..8fdd55399 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls
@@ -34,7 +34,7 @@ nginx:
- server:
- server_name: workbench2.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- root: /var/www/arvados-workbench2/workbench2
@@ -42,7 +42,7 @@ nginx:
- 'if (-f $document_root/maintenance.html)':
- return: 503
- location /config.json:
- - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__"}' ~ "'" }}
+ - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
- include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls
index be571ca77..649af10b6 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls
@@ -43,7 +43,7 @@ nginx:
- server:
- server_name: workbench.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://workbench_upstream'
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
index f0e7a19a4..1b21aaaeb 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
@@ -22,11 +22,11 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__EXT__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __WEBSHELL_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- - location /__HOSTNAME__:
+ - location /__HOSTNAME_EXT__:
- proxy_pass: 'http://webshell_upstream'
- proxy_read_timeout: 90
- proxy_connect_timeout: 90
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
index f783e523f..462443c1f 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
@@ -28,7 +28,7 @@ nginx:
- 'if (-f $document_root/maintenance.html)':
- return: 503
- location /config.json:
- - return: {{ "200 '" ~ '{"API_HOST":"__HOSTNAME__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
+ - return: {{ "200 '" ~ '{"API_HOST":"__HOSTNAME_EXT__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
- include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example
index bd9b1c411..d02ba95c3 100644
--- a/tools/salt-install/local.params.example
+++ b/tools/salt-install/local.params.example
@@ -16,9 +16,14 @@ DOMAIN="some.domain"
# the EXTERNAL/PUBLIC hostname for the instance.
# If empty, the INTERNAL HOST IP will be used
HOSTNAME_EXT=""
-# The internal hostname for the host
+# The internal hostname for the host. In the example files, only used in the
+# single_host/single_hostname example
HOSTNAME_INT="127.0.1.1"
-CONTROLLER_EXT_SSL_PORT=8000
+# Host SSL port where you want to point your browser to access Arvados
+# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
+# You can point it to another port if desired
+# In Vagrant, make sure it matches what you set in the Vagrantfile (8443)
+CONTROLLER_EXT_SSL_PORT=443
KEEP_EXT_SSL_PORT=25101
# Both for collections and downloads
KEEPWEB_EXT_SSL_PORT=9002
@@ -42,9 +47,16 @@ RAILS_SECRET_TOKEN=railssecrettokenmushaveatleast32characters
ANONYMOUS_USER_TOKEN=anonymoususertokenmushaveatleast32characters
WORKBENCH_SECRET_KEY=workbenchsecretkeymushaveatleast32characters
-# The example config files you want to use. There are a few examples
-# under 'config_examples'
-CONFIG_DIR="config_examples/single_host/single_hostname"
+# The directory to check for the config files (pillars, states) you want to use.
+# There are a few examples under 'config_examples'. If you don't change this
+# variable, the single_host, multiple_hostnames config will be used
+# CONFIG_DIR="config_examples/single_host/single_hostname"
+CONFIG_DIR="config_examples/single_host/multiple_hostnames"
+# Extra states to pply. iIf you use your own subdir, change this value accordingly
+EXTRA_STATES_DIR="${F_DIR}/arvados-formula/test/salt/states/examples/single_host"
+
+# When using the single_host/single_hostname example, change to this one
+# EXTRA_STATES_DIR="${CONFIG_DIR}/states"
# Which release of Arvados repo you want to use
RELEASE="production"
@@ -52,12 +64,6 @@ RELEASE="production"
# in the desired repo
VERSION="latest"
-# Host SSL port where you want to point your browser to access Arvados
-# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
-# You can point it to another port if desired
-# In Vagrant, make sure it matches what you set in the Vagrantfile
-HOST_SSL_PORT=443
-
# This is an arvados-formula setting.
# If branch is set, the script will switch to it before running salt
# Usually not needed, only used for testing
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index a7605e1d9..486f43057 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/bin/bash -x
# Copyright (C) The Arvados Authors. All rights reserved.
#
@@ -55,7 +55,7 @@ arguments() {
while [ ${#} -ge 1 ]; do
case ${1} in
-c | --config)
- CONFIG=${2}
+ CONFIG_FILE=${2}
shift 2
;;
-d | --debug)
@@ -63,7 +63,7 @@ arguments() {
shift
;;
-p | --ssl-port)
- HOST_SSL_PORT=${2}
+ CONTROLLER_EXT_SSL_PORT=${2}
shift 2
;;
-r | --roles)
@@ -102,7 +102,7 @@ arguments() {
CONFIG="${SCRIPT_DIR}/local.params"
CONFIG_DIR="config_examples/single_host/multiple_hostnames"
LOG_LEVEL="info"
-HOST_SSL_PORT=443
+CONTROLLER_EXT_SSL_PORT=443
TESTS_DIR="tests"
CLUSTER=""
@@ -130,12 +130,20 @@ NGINX_TAG="v2.4.0"
DOCKER_TAG="v1.0.0"
LOCALE_TAG="v0.3.4"
+# Salt's dir
+## states
+S_DIR="/srv/salt"
+## formulas
+F_DIR="/srv/formulas"
+##pillars
+P_DIR="/srv/pillars"
+
arguments ${@}
-if [ -s ${CONFIG} ]; then
- source ${CONFIG}
+if [ -s ${CONFIG_FILE} ]; then
+ source ${CONFIG_FILE}
else
- echo >&2 "Please create a '${CONFIG}' file with initial values, as described in FIXME_URL_TO_DESCR"
+ echo >&2 "Please create a '${CONFIG_FILE}' file with initial values, as described in FIXME_URL_TO_DESCR"
exit 1
fi
@@ -145,14 +153,6 @@ if ! grep -E '^[[:alnum:]]{5}$' <<<${CLUSTER} ; then
exit 1
fi
-# Salt's dir
-## states
-S_DIR="/srv/salt"
-## formulas
-F_DIR="/srv/formulas"
-##pillars
-P_DIR="/srv/pillars"
-
apt-get update
apt-get install -y curl git jq
@@ -161,7 +161,7 @@ if [ ${?} -eq 0 ]; then
echo "Salt already installed"
else
curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
- sh /tmp/bootstrap_salt.sh -XUdfP -x python3
+ sh /tmp/bootstrap_salt.sh -XdfP -x python3
/bin/systemctl disable salt-minion.service
fi
@@ -172,7 +172,6 @@ file_roots:
base:
- ${S_DIR}
- ${F_DIR}/*
- - ${F_DIR}/*/test/salt/states/examples
pillar_roots:
base:
@@ -181,64 +180,6 @@ EOFSM
mkdir -p ${S_DIR} ${F_DIR} ${P_DIR}
-# States
-cat > ${S_DIR}/top.sls << EOFTSLS
-base:
- '*':
- # - single_host.host_entries
- # - single_host.snakeoil_certs
- - locale
-EOFTSLS
-
-# If we want specific roles for a node, just add the desired states
-# and its dependencies
-if [ -z "${ROLES}" ]; then
- echo ' - nginx.passenger' >> ${S_DIR}/top.sls
- echo ' - postgres' >> ${S_DIR}/top.sls
- echo ' - docker' >> ${S_DIR}/top.sls
- echo ' - arvados' >> ${S_DIR}/top.sls
-else
- # If we add individual roles, make sure we add the repo first
- echo " - arvados.repo" >> ${S_DIR}/top.sls
- for R in ${ROLES}; do
- case "${R}" in
- "database")
- echo " - postgres" >> ${S_DIR}/top.sls
- ::
- "api","workbench","workbench2","keepweb","keepproxy")
- grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
- ;;
- "shell","dispatcher")
- grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
- ;;
- *)
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
- ::
- esac
- done
-fi
-
-# Pillars
-cat > ${P_DIR}/top.sls << EOFPSLS
-base:
- '*':
- - arvados
- - docker
- - locale
- - nginx_api_configuration
- - nginx_controller_configuration
- - nginx_keepproxy_configuration
- - nginx_keepweb_configuration
- - nginx_passenger
- - nginx_websocket_configuration
- - nginx_webshell_configuration
- - nginx_workbench2_configuration
- - nginx_workbench_configuration
- - postgresql
-EOFPSLS
-
# Get the formula and dependencies
cd ${F_DIR} || exit 1
git clone --branch "${ARVADOS_TAG}" https://github.com/arvados/arvados-formula.git
@@ -255,39 +196,39 @@ fi
if [ "x${VAGRANT}" = "xyes" ]; then
SOURCE_PILLARS_DIR="/vagrant/${CONFIG_DIR}/pillars"
- SOURCE_STATES_DIR="/vagrant/${CONFIG_DIR}/states"
TESTS_DIR="/vagrant/${TESTS_DIR}"
else
SOURCE_PILLARS_DIR="${SCRIPT_DIR}/${CONFIG_DIR}/pillars"
- SOURCE_STATES_DIR="${SCRIPT_DIR}/${CONFIG_DIR}/states"
TESTS_DIR="${SCRIPT_DIR}/${TESTS_DIR}"
fi
-# Replace cluster and domain name in the example pillars
+SOURCE_STATES_DIR="${EXTRA_STATES_DIR}"
+
+# Replace variables (cluster, domain, etc) in the pillars, states and tests
+# to ease deployment for newcomers
for f in "${SOURCE_PILLARS_DIR}"/*; do
- sed "s/__CLUSTER__/${CLUSTER}/g;
- s/__DOMAIN__/${DOMAIN}/g;
- s/__RELEASE__/${RELEASE}/g;
+ sed "s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
+ s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
- s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
- s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
- s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
- s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
- s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__CLUSTER__/${CLUSTER}/g;
+ s/__DOMAIN__/${DOMAIN}/g;
s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
- s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
- s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
- s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
- s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
+ s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
- s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
- s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
- s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
- s/__VERSION__/${VERSION}/g" \
+ s/__RELEASE__/${RELEASE}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
+ s/__VERSION__/${VERSION}/g;
+ s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
+ s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
+ s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
+ s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g" \
"${f}" > "${P_DIR}"/$(basename "${f}")
done
@@ -295,48 +236,111 @@ mkdir -p /tmp/cluster_tests
# Replace cluster and domain name in the test files
for f in "${TESTS_DIR}"/*; do
sed "s/__CLUSTER__/${CLUSTER}/g;
+ s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
s/__DOMAIN__/${DOMAIN}/g;
s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
- s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
- s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
- s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
- s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
- s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g" \
+ s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g" \
"${f}" > "/tmp/cluster_tests"/$(basename "${f}")
done
chmod 755 /tmp/cluster_tests/run-test.sh
# Replace helper state files that differ from the formula's examples
-if -d "${SOURCE_STATES_DIR}"; then
+if [ -d "${SOURCE_STATES_DIR}" ]; then
+ mkdir -p "${F_DIR}"/extra/extra
+
for f in "${SOURCE_STATES_DIR}"/*; do
- sed "s/__CLUSTER__/${CLUSTER}/g;
- s/__DOMAIN__/${DOMAIN}/g;
- s/__RELEASE__/${RELEASE}/g;
+ sed "s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
+ s/__CLUSTER__/${CLUSTER}/g;
+ s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
- s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
- s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
- s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
- s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
- s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__DOMAIN__/${DOMAIN}/g;
s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
- s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
- s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
- s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
- s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
+ s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
- s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
- s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
- s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
- s/__VERSION__/${VERSION}/g" \
- "${f}" > "${F_DIR}/arvados-formula/test/salt/states/examples/single_host"/$(basename "${f}")
+ s/__RELEASE__/${RELEASE}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
+ s/__VERSION__/${VERSION}/g;
+ s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
+ s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
+ s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
+ s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g" \
+ "${f}" > "${F_DIR}/extra/extra"/$(basename "${f}")
+ done
+fi
+
+# Now, we build the SALT states/pillars trees
+# States
+cat > ${S_DIR}/top.sls << EOFTSLS
+base:
+ '*':
+ - locale
+EOFTSLS
+
+if [ -d "${SOURCE_STATES_DIR}" ]; then
+ for f in "${F_DIR}"/extra/extra/*.sls; do
+ echo " - extra.$(basename ${f} | sed 's/.sls$//g')" >> ${S_DIR}/top.sls
+ done
+fi
+
+# If we want specific roles for a node, just add the desired states
+# and its dependencies
+if [ -z "${ROLES}" ]; then
+ echo ' - nginx.passenger' >> ${S_DIR}/top.sls
+ echo ' - postgres' >> ${S_DIR}/top.sls
+ echo ' - docker' >> ${S_DIR}/top.sls
+ echo ' - arvados' >> ${S_DIR}/top.sls
+else
+ # If we add individual roles, make sure we add the repo first
+ echo " - arvados.repo" >> ${S_DIR}/top.sls
+ for R in ${ROLES}; do
+ case "${R}" in
+ "database")
+ echo " - postgres" >> ${S_DIR}/top.sls
+ ;;
+ "api","workbench","workbench2","keepweb","keepproxy")
+ grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ "shell","dispatcher")
+ grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ *)
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ esac
done
fi
+# Pillars
+cat > ${P_DIR}/top.sls << EOFPSLS
+base:
+ '*':
+ - arvados
+ - docker
+ - locale
+ - nginx_api_configuration
+ - nginx_controller_configuration
+ - nginx_keepproxy_configuration
+ - nginx_keepweb_configuration
+ - nginx_passenger
+ - nginx_websocket_configuration
+ - nginx_webshell_configuration
+ - nginx_workbench2_configuration
+ - nginx_workbench_configuration
+ - postgresql
+EOFPSLS
+
# FIXME! #16992 Temporary fix for psql call in arvados-api-server
if [ -e /root/.psqlrc ]; then
if ! ( grep 'pset pager off' /root/.psqlrc ); then
@@ -369,12 +373,12 @@ fi
echo "Copying the Arvados CA certificate to the installer dir, so you can import it"
# If running in a vagrant VM, also add default user to docker group
if [ "x${VAGRANT}" = "xyes" ]; then
- cp /etc/ssl/certs/arvados-snakeoil-ca.pem /vagrant
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem /vagrant/${CLUSTER}.${DOMAIN}-arvados-snakeoil-ca.pem
echo "Adding the vagrant user to the docker group"
usermod -a -G docker vagrant
else
- cp /etc/ssl/certs/arvados-snakeoil-ca.pem ${SCRIPT_DIR}
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem ${SCRIPT_DIR}/${CLUSTER}.${DOMAIN}-arvados-snakeoil-ca.pem
fi
# Test that the installation finished correctly
diff --git a/tools/salt-install/tests/run-test.sh b/tools/salt-install/tests/run-test.sh
index 16ee2851e..6bc8422f8 100755
--- a/tools/salt-install/tests/run-test.sh
+++ b/tools/salt-install/tests/run-test.sh
@@ -4,7 +4,7 @@
# SPDX-License-Identifier: Apache-2.0
export ARVADOS_API_TOKEN=__SYSTEM_ROOT_TOKEN__
-export ARVADOS_API_HOST=__HOSTNAME_INT__:__CONTROLLER_EXT_SSL_PORT__
+export ARVADOS_API_HOST=__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__
export ARVADOS_API_HOST_INSECURE=true
set -o pipefail
commit 8195246d1e71c8e1801f85c95e33402775099e7d
Author: Javier Bértoli <jbertoli at curii.com>
Date: Wed Feb 3 16:57:53 2021 -0300
feat(provision): refactor to manage different infrastructure configurations
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index ce1588c19..a7605e1d9 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -70,7 +70,7 @@ arguments() {
for i in ${2//,/ }
do
# Verify the role exists
- if [[ ! "api,controller,keepstore,websocket,keepweb,workbench2,keepproxy,shell,workbench,dispatcher" == *"$i"* ]]; then
+ if [[ ! "database,api,controller,keepstore,websocket,keepweb,workbench2,keepproxy,shell,workbench,dispatcher" == *"$i"* ]]; then
echo "The role '${i}' is not a valid role"
usage
exit 1
@@ -185,20 +185,38 @@ mkdir -p ${S_DIR} ${F_DIR} ${P_DIR}
cat > ${S_DIR}/top.sls << EOFTSLS
base:
'*':
- - single_host.host_entries
- - single_host.snakeoil_certs
+ # - single_host.host_entries
+ # - single_host.snakeoil_certs
- locale
- - nginx.passenger
- - postgres
- - docker
EOFTSLS
-# If we want specific roles for a node, just add those states
+# If we want specific roles for a node, just add the desired states
+# and its dependencies
if [ -z "${ROLES}" ]; then
+ echo ' - nginx.passenger' >> ${S_DIR}/top.sls
+ echo ' - postgres' >> ${S_DIR}/top.sls
+ echo ' - docker' >> ${S_DIR}/top.sls
echo ' - arvados' >> ${S_DIR}/top.sls
else
+ # If we add individual roles, make sure we add the repo first
+ echo " - arvados.repo" >> ${S_DIR}/top.sls
for R in ${ROLES}; do
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ case "${R}" in
+ "database")
+ echo " - postgres" >> ${S_DIR}/top.sls
+ ::
+ "api","workbench","workbench2","keepweb","keepproxy")
+ grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ "shell","dispatcher")
+ grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ *)
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ::
+ esac
done
fi
@@ -285,37 +303,39 @@ for f in "${TESTS_DIR}"/*; do
s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g" \
- ${f} > /tmp/cluster_tests/$(basename ${f})
+ "${f}" > "/tmp/cluster_tests"/$(basename "${f}")
done
chmod 755 /tmp/cluster_tests/run-test.sh
# Replace helper state files that differ from the formula's examples
-for f in "${SOURCE_STATES_DIR}"/*; do
- sed "s/__CLUSTER__/${CLUSTER}/g;
- s/__DOMAIN__/${DOMAIN}/g;
- s/__RELEASE__/${RELEASE}/g;
- s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
- s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
- s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
- s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
- s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
- s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
- s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
- s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
- s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
- s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
- s/__INITIAL_USER__/${INITIAL_USER}/g;
- s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
- s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
- s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
- s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
- s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
- s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
- s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
- s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
- s/__VERSION__/${VERSION}/g" \
- "${f}" > "${F_DIR}"/arvados-formula/test/salt/states/examples/single_host/$(basename "${f}")
-done
+if -d "${SOURCE_STATES_DIR}"; then
+ for f in "${SOURCE_STATES_DIR}"/*; do
+ sed "s/__CLUSTER__/${CLUSTER}/g;
+ s/__DOMAIN__/${DOMAIN}/g;
+ s/__RELEASE__/${RELEASE}/g;
+ s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
+ s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
+ s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
+ s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
+ s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
+ s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
+ s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
+ s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
+ s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
+ s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
+ s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
+ s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
+ s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
+ s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
+ s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
+ s/__VERSION__/${VERSION}/g" \
+ "${f}" > "${F_DIR}/arvados-formula/test/salt/states/examples/single_host"/$(basename "${f}")
+ done
+fi
# FIXME! #16992 Temporary fix for psql call in arvados-api-server
if [ -e /root/.psqlrc ]; then
commit 55d74d65d3dfc4cd1aafff3ac32fb3179f66cab6
Author: Javier Bértoli <jbertoli at curii.com>
Date: Wed Jan 27 13:07:14 2021 -0300
feat(provision): check the cluster name provided is exactly 5 chars long
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index facb2e88e..ce1588c19 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -139,6 +139,12 @@ else
exit 1
fi
+if ! grep -E '^[[:alnum:]]{5}$' <<<${CLUSTER} ; then
+ echo >&2 "ERROR: <CLUSTER> must be exactly 5 alphanumeric characters long"
+ echo >&2 "Fix the cluster name in the 'local.params' file and re-run the provision script"
+ exit 1
+fi
+
# Salt's dir
## states
S_DIR="/srv/salt"
commit f023eb5138f8886820f33901b46b67ba9a0d24a2
Author: Javier Bértoli <jbertoli at curii.com>
Date: Wed Jan 27 09:54:49 2021 -0300
feat(provision): refactor to add other setup examples
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
similarity index 90%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
index 4aa4735d8..6c6dec26f 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
@@ -78,19 +78,15 @@ arvados:
### TOKENS
tokens:
- system_root: changemesystemroottoken
- management: changememanagementtoken
- rails_secret: changemerailssecrettoken
- anonymous_user: changemeanonymoususertoken
+ system_root: __SYSTEM_ROOT_TOKEN__
+ management: __MANAGEMENT_TOKEN__
+ rails_secret: __RAILS_SECRET_TOKEN__
+ anonymous_user: __ANONYMOUS_USER_TOKEN__
### KEYS
secrets:
- blob_signing_key: changemeblobsigningkey
- workbench_secret_key: changemeworkbenchsecretkey
- dispatcher_access_key: changemedispatcheraccesskey
- dispatcher_secret_key: changeme_dispatchersecretkey
- keep_access_key: changemekeepaccesskey
- keep_secret_key: changemekeepsecretkey
+ blob_signing_key: __BLOB_SIGNING_KEY__
+ workbench_secret_key: __WORKBENCH_SECRET_KEY__
Login:
Test:
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/docker.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/docker.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/docker.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/docker.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/locale.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/locale.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/locale.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/locale.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_api_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_api_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_api_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_api_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_controller_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepproxy_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepweb_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_passenger.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_passenger.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_passenger.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_passenger.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_webshell_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_websocket_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench2_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/postgresql.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/postgresql.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/postgresql.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/postgresql.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/arvados.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
similarity index 68%
rename from tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
index e5e458665..f3d2bcb9e 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
@@ -78,19 +78,15 @@ arvados:
### TOKENS
tokens:
- system_root: changemesystemroottoken
- management: changememanagementtoken
- rails_secret: changemerailssecrettoken
- anonymous_user: changemeanonymoususertoken
+ system_root: __SYSTEM_ROOT_TOKEN__
+ management: __MANAGEMENT_TOKEN__
+ rails_secret: __RAILS_SECRET_TOKEN__
+ anonymous_user: __ANONYMOUS_USER_TOKEN__
### KEYS
secrets:
- blob_signing_key: changemeblobsigningkey
- workbench_secret_key: changemeworkbenchsecretkey
- dispatcher_access_key: changemedispatcheraccesskey
- dispatcher_secret_key: changeme_dispatchersecretkey
- keep_access_key: changemekeepaccesskey
- keep_secret_key: changemekeepsecretkey
+ blob_signing_key: __BLOB_SIGNING_KEY__
+ workbench_secret_key: __WORKBENCH_SECRET_KEY__
Login:
Test:
@@ -107,7 +103,7 @@ arvados:
# <cluster>-nyw5e-<volume>
__CLUSTER__-nyw5e-000000000000000:
AccessViaHosts:
- 'http://__HOSTNAME__:25107':
+ 'http://__HOSTNAME_INT__:25107':
ReadOnly: false
Replication: 2
Driver: Directory
@@ -122,38 +118,32 @@ arvados:
Services:
Controller:
- ExternalURL: 'https://__HOSTNAME__:__CONTROLLER_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__CONTROLLER_EXT_SSL_PORT__'
InternalURLs:
- 'http://controller.internal:8003': {}
- DispatchCloud:
- InternalURLs:
- 'http://__HOSTNAME__:9006': {}
- Keepbalance:
- InternalURLs:
- 'http://__HOSTNAME__:9005': {}
+ 'http://__HOSTNAME_INT__:8003': {}
Keepproxy:
- ExternalURL: 'https://__HOSTNAME__:__KEEP_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__KEEP_EXT_SSL_PORT__'
InternalURLs:
- 'http://keep.internal:25100': {}
+ 'http://__HOSTNAME_INT__:25100': {}
Keepstore:
InternalURLs:
- 'http://keep0.internal:25107': {}
+ 'http://__HOSTNAME_INT__:25107': {}
RailsAPI:
InternalURLs:
- 'http://api.internal:8004': {}
+ 'http://__HOSTNAME_INT__:8004': {}
WebDAV:
- ExternalURL: 'https://__HOSTNAME__:__KEEPWEB_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__KEEPWEB_EXT_SSL_PORT__'
InternalURLs:
- 'http://collections.internal:9002': {}
+ 'http://__HOSTNAME_INT__:9003': {}
WebDAVDownload:
- ExternalURL: 'https://__HOSTNAME__:__KEEPWEB_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__KEEPWEB_EXT_SSL_PORT__'
WebShell:
- ExternalURL: 'https://__HOSTNAME__:__WEBSHELL_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__WEBSHELL_EXT_SSL_PORT__'
Websocket:
- ExternalURL: 'wss://__HOSTNAME__:__WEBSOCKET_EXT_SSL_PORT__/websocket'
+ ExternalURL: 'wss://__HOSTNAME_EXT__:__WEBSOCKET_EXT_SSL_PORT__/websocket'
InternalURLs:
- 'http://ws.internal:8005': {}
+ 'http://__HOSTNAME_INT__:8005': {}
Workbench1:
- ExternalURL: 'https://__HOSTNAME__:__WORKBENCH1_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__WORKBENCH1_EXT_SSL_PORT__'
Workbench2:
- ExternalURL: 'https://__HOSTNAME__:__WORKBENCH2_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__WORKBENCH2_EXT_SSL_PORT__'
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/docker.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/docker.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/single_hostname/docker.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/docker.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/locale.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/locale.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/single_hostname/locale.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/locale.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_api_configuration.sls
similarity index 93%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_api_configuration.sls
index b2f12c773..18f09af50 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_api_configuration.sls
@@ -18,7 +18,7 @@ nginx:
overwrite: true
config:
- server:
- - listen: 'api.internal:8004'
+ - listen: '__HOSTNAME_INT__:8004'
- server_name: api
- root: /var/www/arvados-api/current/public
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_controller_configuration.sls
similarity index 87%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_controller_configuration.sls
index 2eb33b835..b7b75ab9c 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_controller_configuration.sls
@@ -14,7 +14,7 @@ nginx:
default: 1
'127.0.0.0/8': 0
upstream controller_upstream:
- - server: 'controller.internal:8003 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:8003 fail_timeout=10s'
### SITES
servers:
@@ -25,9 +25,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: _
- listen:
- - 80 default
+ - 80 default_server
- location /.well-known:
- root: /var/www
- location /:
@@ -38,9 +38,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl default_server
- index: index.html index.htm
- location /:
- proxy_pass: 'http://controller_upstream'
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepproxy_configuration.sls
similarity index 73%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepproxy_configuration.sls
index b26de2710..81d72aac7 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepproxy_configuration.sls
@@ -11,30 +11,16 @@ nginx:
### STREAMS
http:
upstream keepproxy_upstream:
- - server: 'keep.internal:25100 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:25100 fail_timeout=10s'
servers:
managed:
- ### DEFAULT
- arvados_keepproxy_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __KEEP_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
arvados_keepproxy_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __KEEP_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepweb_configuration.sls
similarity index 72%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepweb_configuration.sls
index 98a3cdf94..fcb56c994 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepweb_configuration.sls
@@ -11,31 +11,17 @@ nginx:
### STREAMS
http:
upstream collections_downloads_upstream:
- - server: 'collections.internal:9002 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:9003 fail_timeout=10s'
servers:
managed:
- ### COLLECTIONS / DOWNLOAD
- arvados_collections_download_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __KEEPWEB_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
### COLLECTIONS / DOWNLOAD
arvados_collections_download_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __KEEPWEB_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_passenger.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_passenger.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_passenger.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_passenger.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
similarity index 84%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
index dac606123..f0e7a19a4 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
@@ -12,30 +12,17 @@ nginx:
### STREAMS
http:
upstream webshell_upstream:
- - server: 'shell.internal:4200 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:4200 fail_timeout=10s'
### SITES
servers:
managed:
- arvados_webshell_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __WEBSHELL_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
arvados_webshell_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: __HOSTNAME__EXT__
- listen:
- __WEBSHELL_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_websocket_configuration.sls
similarity index 74%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_websocket_configuration.sls
index 827524cbe..7c4ff7835 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_websocket_configuration.sls
@@ -11,30 +11,16 @@ nginx:
### STREAMS
http:
upstream websocket_upstream:
- - server: 'ws.internal:8005 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:8005 fail_timeout=10s'
servers:
managed:
- ### DEFAULT
- arvados_websocket_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __WEBSOCKET_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
arvados_websocket_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __WEBSOCKET_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
similarity index 70%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
index 7f90cbc82..f783e523f 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
@@ -13,26 +13,12 @@ nginx:
### SITES
servers:
managed:
- ### DEFAULT
- arvados_workbench2_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __WORKBENCH2_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
arvados_workbench2_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: workbench2.__HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __WORKBENCH2_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench_configuration.sls
similarity index 76%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench_configuration.sls
index 0cbd3e14a..9ed6e3b87 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench_configuration.sls
@@ -17,31 +17,17 @@ nginx:
### STREAMS
http:
upstream workbench_upstream:
- - server: 'workbench.internal:9000 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:9000 fail_timeout=10s'
### SITES
servers:
managed:
- ### DEFAULT
- arvados_workbench_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __WORKBENCH_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
arvados_workbench_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: workbench.__HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __WORKBENCH1_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
@@ -63,7 +49,7 @@ nginx:
overwrite: true
config:
- server:
- - listen: 'workbench.internal:9000'
+ - listen: '__HOSTNAME_INT__:9000'
- server_name: workbench
- root: /var/www/arvados-workbench/current/public
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/postgresql.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/postgresql.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/single_hostname/postgresql.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/postgresql.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/states/host_entries.sls b/tools/salt-install/config_examples/single_host/single_hostname/states/host_entries.sls
new file mode 100644
index 000000000..7e3957c57
--- /dev/null
+++ b/tools/salt-install/config_examples/single_host/single_hostname/states/host_entries.sls
@@ -0,0 +1,32 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+{%- set curr_tpldir = tpldir %}
+{%- set tpldir = 'arvados' %}
+{%- from "arvados/map.jinja" import arvados with context %}
+{%- set tpldir = curr_tpldir %}
+
+arvados_test_salt_states_examples_single_host_etc_hosts_host_present:
+ host.present:
+ - ip: 127.0.0.2
+ - names:
+ - {{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ # FIXME! This just works for our testings.
+ # Won't work if the cluster name != host name
+ {%- for entry in [
+ 'api',
+ 'collections',
+ 'controller',
+ 'download',
+ 'keep',
+ 'keepweb',
+ 'keep0',
+ 'shell',
+ 'workbench',
+ 'workbench2',
+ 'ws',
+ ]
+ %}
+ - {{ entry }}
+ {%- endfor %}
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/states/snakeoil_certs.sls b/tools/salt-install/config_examples/single_host/single_hostname/states/snakeoil_certs.sls
new file mode 100644
index 000000000..375cc84eb
--- /dev/null
+++ b/tools/salt-install/config_examples/single_host/single_hostname/states/snakeoil_certs.sls
@@ -0,0 +1,156 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+{%- set curr_tpldir = tpldir %}
+{%- set tpldir = 'arvados' %}
+{%- from "arvados/map.jinja" import arvados with context %}
+{%- set tpldir = curr_tpldir %}
+
+include:
+ - nginx.service
+
+{%- set arvados_ca_cert_file = '/etc/ssl/certs/arvados-snakeoil-ca.pem' %}
+{%- set arvados_ca_key_file = '/etc/ssl/private/arvados-snakeoil-ca.key' %}
+{%- set arvados_cert_file = '/etc/ssl/certs/arvados-snakeoil-cert.pem' %}
+{%- set arvados_csr_file = '/etc/ssl/private/arvados-snakeoil-cert.csr' %}
+{%- set arvados_key_file = '/etc/ssl/private/arvados-snakeoil-cert.key' %}
+
+{%- if grains.get('os_family') == 'Debian' %}
+ {%- set arvados_ca_cert_dest = '/usr/local/share/ca-certificates/arvados-snakeoil-ca.crt' %}
+ {%- set update_ca_cert = '/usr/sbin/update-ca-certificates' %}
+ {%- set openssl_conf = '/etc/ssl/openssl.cnf' %}
+{%- else %}
+ {%- set arvados_ca_cert_dest = '/etc/pki/ca-trust/source/anchors/arvados-snakeoil-ca.pem' %}
+ {%- set update_ca_cert = '/usr/bin/update-ca-trust' %}
+ {%- set openssl_conf = '/etc/pki/tls/openssl.cnf' %}
+{%- endif %}
+
+arvados_test_salt_states_examples_single_host_snakeoil_certs_dependencies_pkg_installed:
+ pkg.installed:
+ - pkgs:
+ - openssl
+ - ca-certificates
+
+arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_ca_cmd_run:
+ # Taken from https://github.com/arvados/arvados/blob/master/tools/arvbox/lib/arvbox/docker/service/certificate/run
+ cmd.run:
+ - name: |
+ # These dirs are not to CentOS-ish, but this is a helper script
+ # and they should be enough
+ mkdir -p /etc/ssl/certs/ /etc/ssl/private/ && \
+ openssl req \
+ -new \
+ -nodes \
+ -sha256 \
+ -x509 \
+ -subj "/C=CC/ST=Some State/O=Arvados Formula/OU=arvados-formula/CN=snakeoil-ca-{{ arvados.cluster.name }}.{{ arvados.cluster.domain }}" \
+ -extensions x509_ext \
+ -config <(cat {{ openssl_conf }} \
+ <(printf "\n[x509_ext]\nbasicConstraints=critical,CA:true,pathlen:0\nkeyUsage=critical,keyCertSign,cRLSign")) \
+ -out {{ arvados_ca_cert_file }} \
+ -keyout {{ arvados_ca_key_file }} \
+ -days 365 && \
+ cp {{ arvados_ca_cert_file }} {{ arvados_ca_cert_dest }} && \
+ {{ update_ca_cert }}
+ - unless:
+ - test -f {{ arvados_ca_cert_file }}
+ - openssl verify -CAfile {{ arvados_ca_cert_file }} {{ arvados_ca_cert_file }}
+ - require:
+ - pkg: arvados_test_salt_states_examples_single_host_snakeoil_certs_dependencies_pkg_installed
+
+arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_cert_cmd_run:
+ cmd.run:
+ - name: |
+ cat > /tmp/openssl.cnf <<-CNF
+ [req]
+ default_bits = 2048
+ prompt = no
+ default_md = sha256
+ req_extensions = rext
+ distinguished_name = dn
+ [dn]
+ C = CC
+ ST = Some State
+ L = Some Location
+ O = Arvados Formula
+ OU = arvados-formula
+ CN = {{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ emailAddress = admin@{{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ [rext]
+ subjectAltName = @alt_names
+ [alt_names]
+ {%- for entry in grains.get('ipv4') %}
+ IP.{{ loop.index }} = {{ entry }}
+ {%- endfor %}
+ {%- for entry in [
+ 'keep',
+ 'collections',
+ 'download',
+ 'keepweb',
+ 'ws',
+ 'workbench',
+ 'workbench2',
+ ]
+ %}
+ DNS.{{ loop.index }} = {{ entry }}
+ {%- endfor %}
+ DNS.8 = {{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ DNS.9 = '__HOSTNAME_EXT__'
+ DNS.10 = '__HOSTNAME_INT__'
+ CNF
+
+ # The req
+ openssl req \
+ -config /tmp/openssl.cnf \
+ -new \
+ -nodes \
+ -sha256 \
+ -out {{ arvados_csr_file }} \
+ -keyout {{ arvados_key_file }} > /tmp/snake_oil_certs.output 2>&1 && \
+ # The cert
+ openssl x509 \
+ -req \
+ -days 365 \
+ -in {{ arvados_csr_file }} \
+ -out {{ arvados_cert_file }} \
+ -extfile /tmp/openssl.cnf \
+ -extensions rext \
+ -CA {{ arvados_ca_cert_file }} \
+ -CAkey {{ arvados_ca_key_file }} \
+ -set_serial $(date +%s) && \
+ chmod 0644 {{ arvados_cert_file }} && \
+ chmod 0640 {{ arvados_key_file }}
+ - unless:
+ - test -f {{ arvados_key_file }}
+ - openssl verify -CAfile {{ arvados_ca_cert_file }} {{ arvados_cert_file }}
+ - require:
+ - pkg: arvados_test_salt_states_examples_single_host_snakeoil_certs_dependencies_pkg_installed
+ - cmd: arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_ca_cmd_run
+
+{%- if grains.get('os_family') == 'Debian' %}
+arvados_test_salt_states_examples_single_host_snakeoil_certs_ssl_cert_pkg_installed:
+ pkg.installed:
+ - name: ssl-cert
+ - require_in:
+ - sls: postgres
+
+arvados_test_salt_states_examples_single_host_snakeoil_certs_certs_permissions_cmd_run:
+ cmd.run:
+ - name: |
+ chown root:ssl-cert {{ arvados_key_file }}
+ - require:
+ - cmd: arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_cert_cmd_run
+ - pkg: arvados_test_salt_states_examples_single_host_snakeoil_certs_ssl_cert_pkg_installed
+{%- endif %}
+
+arvados_test_salt_states_examples_single_host_snakeoil_certs_nginx_snakeoil_file_managed:
+ file.managed:
+ - name: /etc/nginx/snippets/arvados-snakeoil.conf
+ - contents: |
+ ssl_certificate {{ arvados_cert_file }};
+ ssl_certificate_key {{ arvados_key_file }};
+ - watch_in:
+ - service: nginx_service
+
+
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example
index a88301b2a..bd9b1c411 100644
--- a/tools/salt-install/local.params.example
+++ b/tools/salt-install/local.params.example
@@ -13,9 +13,11 @@ DOMAIN="some.domain"
# When setting the cluster in a single host, you can use a single hostname
# to access all the instances. When using virtualization (ie AWS), this should be
-# the EXTERNAL hostname for the instance.
+# the EXTERNAL/PUBLIC hostname for the instance.
# If empty, the INTERNAL HOST IP will be used
-HOSTNAME=""
+HOSTNAME_EXT=""
+# The internal hostname for the host
+HOSTNAME_INT="127.0.1.1"
CONTROLLER_EXT_SSL_PORT=8000
KEEP_EXT_SSL_PORT=25101
# Both for collections and downloads
@@ -32,6 +34,14 @@ INITIAL_USER="admin"
INITIAL_USER_EMAIL="admin at fixme.localdomain"
INITIAL_USER_PASSWORD="password"
+# YOU SHOULD CHANGE THESE TO SOME RANDOM STRINGS
+BLOB_SIGNING_KEY=blobsigningkeymushaveatleast32characters
+MANAGEMENT_TOKEN=managementtokenmushaveatleast32characters
+SYSTEM_ROOT_TOKEN=systemroottokenmushaveatleast32characters
+RAILS_SECRET_TOKEN=railssecrettokenmushaveatleast32characters
+ANONYMOUS_USER_TOKEN=anonymoususertokenmushaveatleast32characters
+WORKBENCH_SECRET_KEY=workbenchsecretkeymushaveatleast32characters
+
# The example config files you want to use. There are a few examples
# under 'config_examples'
CONFIG_DIR="config_examples/single_host/single_hostname"
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index f3df4109a..facb2e88e 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -107,7 +107,8 @@ TESTS_DIR="tests"
CLUSTER=""
DOMAIN=""
-HOSTNAME=""
+HOSTNAME_EXT=""
+HOSTNAME_INT="127.0.1.1"
INITIAL_USER=""
INITIAL_USER_EMAIL=""
INITIAL_USER_PASSWORD=""
@@ -229,14 +230,16 @@ if [ "x${BRANCH}" != "x" ]; then
fi
if [ "x${VAGRANT}" = "xyes" ]; then
- SOURCE_PILLARS_DIR="/vagrant/${CONFIG_DIR}"
+ SOURCE_PILLARS_DIR="/vagrant/${CONFIG_DIR}/pillars"
+ SOURCE_STATES_DIR="/vagrant/${CONFIG_DIR}/states"
TESTS_DIR="/vagrant/${TESTS_DIR}"
else
- SOURCE_PILLARS_DIR="${SCRIPT_DIR}/${CONFIG_DIR}"
+ SOURCE_PILLARS_DIR="${SCRIPT_DIR}/${CONFIG_DIR}/pillars"
+ SOURCE_STATES_DIR="${SCRIPT_DIR}/${CONFIG_DIR}/states"
TESTS_DIR="${SCRIPT_DIR}/${TESTS_DIR}"
fi
-# Replace cluster and domain name in the example pillars and test files
+# Replace cluster and domain name in the example pillars
for f in "${SOURCE_PILLARS_DIR}"/*; do
sed "s/__CLUSTER__/${CLUSTER}/g;
s/__DOMAIN__/${DOMAIN}/g;
@@ -244,25 +247,35 @@ for f in "${SOURCE_PILLARS_DIR}"/*; do
s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
- s/__WORKBENCH1_EXT__SSL_PORT__/${WORKBENCH1_EXT__SSL_PORT}/g;
- s/__WORKBENCH2_EXT__SSL_PORT__/${WORKBENCH2_EXT__SSL_PORT}/g;
+ s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
+ s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
- s/__HOSTNAME__/${HOSTNAME}/g;
+ s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
+ s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
+ s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
+ s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
+ s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
+ s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
+ s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
s/__VERSION__/${VERSION}/g" \
"${f}" > "${P_DIR}"/$(basename "${f}")
done
mkdir -p /tmp/cluster_tests
-# Replace cluster and domain name in the example pillars and test files
+# Replace cluster and domain name in the test files
for f in "${TESTS_DIR}"/*; do
sed "s/__CLUSTER__/${CLUSTER}/g;
s/__DOMAIN__/${DOMAIN}/g;
+ s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
+ s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g" \
@@ -270,6 +283,34 @@ for f in "${TESTS_DIR}"/*; do
done
chmod 755 /tmp/cluster_tests/run-test.sh
+# Replace helper state files that differ from the formula's examples
+for f in "${SOURCE_STATES_DIR}"/*; do
+ sed "s/__CLUSTER__/${CLUSTER}/g;
+ s/__DOMAIN__/${DOMAIN}/g;
+ s/__RELEASE__/${RELEASE}/g;
+ s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
+ s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
+ s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
+ s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
+ s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
+ s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
+ s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
+ s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
+ s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
+ s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
+ s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
+ s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
+ s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
+ s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
+ s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
+ s/__VERSION__/${VERSION}/g" \
+ "${f}" > "${F_DIR}"/arvados-formula/test/salt/states/examples/single_host/$(basename "${f}")
+done
+
# FIXME! #16992 Temporary fix for psql call in arvados-api-server
if [ -e /root/.psqlrc ]; then
if ! ( grep 'pset pager off' /root/.psqlrc ); then
diff --git a/tools/salt-install/tests/run-test.sh b/tools/salt-install/tests/run-test.sh
index 8d9de6fdf..16ee2851e 100755
--- a/tools/salt-install/tests/run-test.sh
+++ b/tools/salt-install/tests/run-test.sh
@@ -3,8 +3,8 @@
#
# SPDX-License-Identifier: Apache-2.0
-export ARVADOS_API_TOKEN=changemesystemroottoken
-export ARVADOS_API_HOST=__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+export ARVADOS_API_TOKEN=__SYSTEM_ROOT_TOKEN__
+export ARVADOS_API_HOST=__HOSTNAME_INT__:__CONTROLLER_EXT_SSL_PORT__
export ARVADOS_API_HOST_INSECURE=true
set -o pipefail
commit a8ccabb165c144229ed47128843ea123778565af
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Jan 26 08:58:07 2021 -0300
feat(provision): refactor to add other setup examples
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/README.md b/tools/salt-install/config_examples/single_host/multiple_hostnames/README.md
new file mode 100644
index 000000000..17ca89a9f
--- /dev/null
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/README.md
@@ -0,0 +1,20 @@
+Single host with multiple hostnames
+===================================
+
+These files let you setup Arvados on a single host using different hostnames
+for each of its components nginx's virtualhosts.
+
+The hostnames are composed after the variables "CLUSTER" and "DOMAIN" set in
+the `local.params` file.
+
+The virtual hosts' hostnames that will be used are:
+
+* CLUSTER.DOMAIN
+* collections.CLUSTER.DOMAIN
+* download.CLUSTER.DOMAIN
+* keep.CLUSTER.DOMAIN
+* keep0.CLUSTER.DOMAIN
+* webshell.CLUSTER.DOMAIN
+* workbench.CLUSTER.DOMAIN
+* workbench2.CLUSTER.DOMAIN
+* ws.CLUSTER.DOMAIN
diff --git a/tools/salt-install/single_host/arvados.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
similarity index 78%
copy from tools/salt-install/single_host/arvados.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
index a06244270..4aa4735d8 100644
--- a/tools/salt-install/single_host/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
@@ -107,7 +107,7 @@ arvados:
# <cluster>-nyw5e-<volume>
__CLUSTER__-nyw5e-000000000000000:
AccessViaHosts:
- http://keep0.__CLUSTER__.__DOMAIN__:25107:
+ 'http://keep0.__CLUSTER__.__DOMAIN__:25107':
ReadOnly: false
Replication: 2
Driver: Directory
@@ -122,38 +122,38 @@ arvados:
Services:
Controller:
- ExternalURL: https://__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
InternalURLs:
- http://controller.internal:8003: {}
+ 'http://controller.internal:8003': {}
DispatchCloud:
InternalURLs:
- http://__CLUSTER__.__DOMAIN__:9006: {}
+ 'http://__CLUSTER__.__DOMAIN__:9006': {}
Keepbalance:
InternalURLs:
- http://__CLUSTER__.__DOMAIN__:9005: {}
+ 'http://__CLUSTER__.__DOMAIN__:9005': {}
Keepproxy:
- ExternalURL: https://keep.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://keep.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
InternalURLs:
- http://keep.internal:25100: {}
+ 'http://keep.internal:25100': {}
Keepstore:
InternalURLs:
- http://keep0.__CLUSTER__.__DOMAIN__:25107: {}
+ 'http://keep0.__CLUSTER__.__DOMAIN__:25107': {}
RailsAPI:
InternalURLs:
- http://api.internal:8004: {}
+ 'http://api.internal:8004': {}
WebDAV:
- ExternalURL: https://collections.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://collections.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
InternalURLs:
- http://collections.internal:9002: {}
+ 'http://collections.internal:9002': {}
WebDAVDownload:
- ExternalURL: https://download.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://download.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
WebShell:
- ExternalURL: https://webshell.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://webshell.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
Websocket:
- ExternalURL: wss://ws.__CLUSTER__.__DOMAIN__/websocket
+ ExternalURL: 'wss://ws.__CLUSTER__.__DOMAIN__/websocket'
InternalURLs:
- http://ws.internal:8005: {}
+ 'http://ws.internal:8005': {}
Workbench1:
- ExternalURL: https://workbench.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://workbench.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
Workbench2:
- ExternalURL: https://workbench2.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://workbench2.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
diff --git a/tools/salt-install/single_host/docker.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/docker.sls
similarity index 100%
copy from tools/salt-install/single_host/docker.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/docker.sls
diff --git a/tools/salt-install/single_host/locale.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/locale.sls
similarity index 100%
copy from tools/salt-install/single_host/locale.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/locale.sls
diff --git a/tools/salt-install/single_host/nginx_api_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_api_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_api_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_api_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_controller_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_controller_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_controller_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepproxy_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_keepproxy_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepproxy_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepweb_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_keepweb_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepweb_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_passenger.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_passenger.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_passenger.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_passenger.sls
diff --git a/tools/salt-install/single_host/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_webshell_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_webshell_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_webshell_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_websocket_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_websocket_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_websocket_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench2_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_workbench2_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench2_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_workbench_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench_configuration.sls
diff --git a/tools/salt-install/single_host/postgresql.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/postgresql.sls
similarity index 100%
copy from tools/salt-install/single_host/postgresql.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/postgresql.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/README.md b/tools/salt-install/config_examples/single_host/single_hostname/README.md
new file mode 100644
index 000000000..9c7ab96c3
--- /dev/null
+++ b/tools/salt-install/config_examples/single_host/single_hostname/README.md
@@ -0,0 +1,23 @@
+Single host with a single hostname
+==================================
+
+These files let you setup Arvados on a single host using a single hostname
+for all of its components nginx's virtualhosts.
+
+The hostname MUST be given in the `local.params` file. The script won't try
+to guess it because, depending on the network architecture where you're
+installing Arvados, things might not work as expected.
+
+The services will be available on the same hostname but different ports,
+which can be given on the `local.params` file or will default to the following
+values:
+
+* CLUSTER.DOMAIN
+* collections
+* download
+* keep
+* keep0
+* webshell
+* workbench
+* workbench2
+* ws
diff --git a/tools/salt-install/single_host/arvados.sls b/tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
similarity index 78%
rename from tools/salt-install/single_host/arvados.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
index a06244270..e5e458665 100644
--- a/tools/salt-install/single_host/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
@@ -107,7 +107,7 @@ arvados:
# <cluster>-nyw5e-<volume>
__CLUSTER__-nyw5e-000000000000000:
AccessViaHosts:
- http://keep0.__CLUSTER__.__DOMAIN__:25107:
+ 'http://__HOSTNAME__:25107':
ReadOnly: false
Replication: 2
Driver: Directory
@@ -122,38 +122,38 @@ arvados:
Services:
Controller:
- ExternalURL: https://__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__CONTROLLER_EXT_SSL_PORT__'
InternalURLs:
- http://controller.internal:8003: {}
+ 'http://controller.internal:8003': {}
DispatchCloud:
InternalURLs:
- http://__CLUSTER__.__DOMAIN__:9006: {}
+ 'http://__HOSTNAME__:9006': {}
Keepbalance:
InternalURLs:
- http://__CLUSTER__.__DOMAIN__:9005: {}
+ 'http://__HOSTNAME__:9005': {}
Keepproxy:
- ExternalURL: https://keep.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__KEEP_EXT_SSL_PORT__'
InternalURLs:
- http://keep.internal:25100: {}
+ 'http://keep.internal:25100': {}
Keepstore:
InternalURLs:
- http://keep0.__CLUSTER__.__DOMAIN__:25107: {}
+ 'http://keep0.internal:25107': {}
RailsAPI:
InternalURLs:
- http://api.internal:8004: {}
+ 'http://api.internal:8004': {}
WebDAV:
- ExternalURL: https://collections.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__KEEPWEB_EXT_SSL_PORT__'
InternalURLs:
- http://collections.internal:9002: {}
+ 'http://collections.internal:9002': {}
WebDAVDownload:
- ExternalURL: https://download.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__KEEPWEB_EXT_SSL_PORT__'
WebShell:
- ExternalURL: https://webshell.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__WEBSHELL_EXT_SSL_PORT__'
Websocket:
- ExternalURL: wss://ws.__CLUSTER__.__DOMAIN__/websocket
+ ExternalURL: 'wss://__HOSTNAME__:__WEBSOCKET_EXT_SSL_PORT__/websocket'
InternalURLs:
- http://ws.internal:8005: {}
+ 'http://ws.internal:8005': {}
Workbench1:
- ExternalURL: https://workbench.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__WORKBENCH1_EXT_SSL_PORT__'
Workbench2:
- ExternalURL: https://workbench2.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__WORKBENCH2_EXT_SSL_PORT__'
diff --git a/tools/salt-install/single_host/docker.sls b/tools/salt-install/config_examples/single_host/single_hostname/docker.sls
similarity index 100%
rename from tools/salt-install/single_host/docker.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/docker.sls
diff --git a/tools/salt-install/single_host/locale.sls b/tools/salt-install/config_examples/single_host/single_hostname/locale.sls
similarity index 100%
rename from tools/salt-install/single_host/locale.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/locale.sls
diff --git a/tools/salt-install/single_host/nginx_api_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls
similarity index 100%
rename from tools/salt-install/single_host/nginx_api_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
similarity index 92%
rename from tools/salt-install/single_host/nginx_controller_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
index 00c3b3a13..2eb33b835 100644
--- a/tools/salt-install/single_host/nginx_controller_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
@@ -25,7 +25,7 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- 80 default
- location /.well-known:
@@ -38,9 +38,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://controller_upstream'
diff --git a/tools/salt-install/single_host/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
similarity index 90%
rename from tools/salt-install/single_host/nginx_keepproxy_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
index 6554f79a7..b26de2710 100644
--- a/tools/salt-install/single_host/nginx_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
@@ -21,9 +21,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: keep.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __KEEP_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -34,9 +34,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: keep.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __KEEP_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://keepproxy_upstream'
diff --git a/tools/salt-install/single_host/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
similarity index 77%
rename from tools/salt-install/single_host/nginx_keepweb_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
index cc871b9da..98a3cdf94 100644
--- a/tools/salt-install/single_host/nginx_keepweb_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
@@ -15,15 +15,15 @@ nginx:
servers:
managed:
- ### DEFAULT
+ ### COLLECTIONS / DOWNLOAD
arvados_collections_download_default:
enabled: true
overwrite: true
config:
- server:
- - server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __KEEPWEB_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -35,9 +35,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __KEEPWEB_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://collections_downloads_upstream'
@@ -53,5 +53,5 @@ nginx:
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- include: 'snippets/arvados-snakeoil.conf'
- - access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
- - error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
+ - access_log: /var/log/nginx/keepweb.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/keepweb.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/single_host/nginx_passenger.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_passenger.sls
similarity index 100%
rename from tools/salt-install/single_host/nginx_passenger.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_passenger.sls
diff --git a/tools/salt-install/single_host/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
similarity index 92%
rename from tools/salt-install/single_host/nginx_webshell_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
index a0756b7ce..dac606123 100644
--- a/tools/salt-install/single_host/nginx_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
@@ -22,9 +22,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: webshell.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WEBSHELL_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -35,11 +35,11 @@ nginx:
overwrite: true
config:
- server:
- - server_name: webshell.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WEBSHELL_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- - location /shell.__CLUSTER__.__DOMAIN__:
+ - location /__HOSTNAME__:
- proxy_pass: 'http://webshell_upstream'
- proxy_read_timeout: 90
- proxy_connect_timeout: 90
diff --git a/tools/salt-install/single_host/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
similarity index 90%
rename from tools/salt-install/single_host/nginx_websocket_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
index ebe03f733..827524cbe 100644
--- a/tools/salt-install/single_host/nginx_websocket_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
@@ -21,9 +21,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: ws.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WEBSOCKET_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -34,9 +34,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: ws.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WEBSOCKET_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://websocket_upstream'
diff --git a/tools/salt-install/single_host/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
similarity index 80%
rename from tools/salt-install/single_host/nginx_workbench2_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
index 8930be408..7f90cbc82 100644
--- a/tools/salt-install/single_host/nginx_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
@@ -19,9 +19,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench2.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WORKBENCH2_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -32,9 +32,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench2.__CLUSTER__.__DOMAIN__
+ - server_name: workbench2.__HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WORKBENCH2_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- root: /var/www/arvados-workbench2/workbench2
@@ -42,7 +42,7 @@ nginx:
- 'if (-f $document_root/maintenance.html)':
- return: 503
- location /config.json:
- - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__"}' ~ "'" }}
+ - return: {{ "200 '" ~ '{"API_HOST":"__HOSTNAME__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
- include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/single_host/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
similarity index 91%
rename from tools/salt-install/single_host/nginx_workbench_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
index be571ca77..0cbd3e14a 100644
--- a/tools/salt-install/single_host/nginx_workbench_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
@@ -28,9 +28,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WORKBENCH_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -41,9 +41,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench.__CLUSTER__.__DOMAIN__
+ - server_name: workbench.__HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WORKBENCH1_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://workbench_upstream'
diff --git a/tools/salt-install/single_host/postgresql.sls b/tools/salt-install/config_examples/single_host/single_hostname/postgresql.sls
similarity index 100%
rename from tools/salt-install/single_host/postgresql.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/postgresql.sls
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example
new file mode 100644
index 000000000..a88301b2a
--- /dev/null
+++ b/tools/salt-install/local.params.example
@@ -0,0 +1,64 @@
+##########################################################
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: CC-BY-SA-3.0
+
+# These are the basic parameters to configure the installation
+
+# The 5 letters name you want to give your cluster
+CLUSTER="fixme"
+
+# The domainname you want tou give to your cluster's hosts
+DOMAIN="some.domain"
+
+# When setting the cluster in a single host, you can use a single hostname
+# to access all the instances. When using virtualization (ie AWS), this should be
+# the EXTERNAL hostname for the instance.
+# If empty, the INTERNAL HOST IP will be used
+HOSTNAME=""
+CONTROLLER_EXT_SSL_PORT=8000
+KEEP_EXT_SSL_PORT=25101
+# Both for collections and downloads
+KEEPWEB_EXT_SSL_PORT=9002
+WEBSHELL_EXT_SSL_PORT=4202
+WEBSOCKET_EXT_SSL_PORT=8002
+WORKBENCH1_EXT_SSL_PORT=443
+WORKBENCH2_EXT_SSL_PORT=3001
+
+INITIAL_USER="admin"
+
+# If not specified, the initial user email will be composed as
+# INITIAL_USER at CLUSTER.DOMAIN
+INITIAL_USER_EMAIL="admin at fixme.localdomain"
+INITIAL_USER_PASSWORD="password"
+
+# The example config files you want to use. There are a few examples
+# under 'config_examples'
+CONFIG_DIR="config_examples/single_host/single_hostname"
+
+# Which release of Arvados repo you want to use
+RELEASE="production"
+# Which version of Arvados you want to install. Defaults to 'latest'
+# in the desired repo
+VERSION="latest"
+
+# Host SSL port where you want to point your browser to access Arvados
+# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
+# You can point it to another port if desired
+# In Vagrant, make sure it matches what you set in the Vagrantfile
+HOST_SSL_PORT=443
+
+# This is an arvados-formula setting.
+# If branch is set, the script will switch to it before running salt
+# Usually not needed, only used for testing
+BRANCH="master"
+
+##########################################################
+# Usually there's no need to modify things below this line
+
+# Formulas versions
+ARVADOS_TAG="v1.1.4"
+POSTGRES_TAG="v0.41.3"
+NGINX_TAG="v2.4.0"
+DOCKER_TAG="v1.0.0"
+LOCALE_TAG="v0.3.4"
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index b97d71965..f3df4109a 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -15,15 +15,6 @@ set -o pipefail
# capture the directory that the script is running from
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-CONFIG_DIR="single_host"
-RELEASE="production"
-VERSION="latest"
-ARVADOS_TAG="v1.1.4"
-POSTGRES_TAG="v0.41.3"
-NGINX_TAG="v2.4.0"
-DOCKER_TAG="v1.0.0"
-LOCALE_TAG="v0.3.4"
-
usage() {
echo >&2
echo >&2 "Usage: ${0} [-h] [-h]"
@@ -109,10 +100,35 @@ arguments() {
}
CONFIG="${SCRIPT_DIR}/local.params"
+CONFIG_DIR="config_examples/single_host/multiple_hostnames"
LOG_LEVEL="info"
HOST_SSL_PORT=443
TESTS_DIR="tests"
+CLUSTER=""
+DOMAIN=""
+HOSTNAME=""
+INITIAL_USER=""
+INITIAL_USER_EMAIL=""
+INITIAL_USER_PASSWORD=""
+
+CONTROLLER_EXT_SSL_PORT=8000
+KEEP_EXT_SSL_PORT=25101
+# Both for collections and downloads
+KEEPWEB_EXT_SSL_PORT=9002
+WEBSHELL_EXT_SSL_PORT=4202
+WEBSOCKET_EXT_SSL_PORT=8002
+WORKBENCH1_EXT_SSL_PORT=443
+WORKBENCH2_EXT_SSL_PORT=3001
+
+RELEASE="production"
+VERSION="latest"
+ARVADOS_TAG="v1.1.4"
+POSTGRES_TAG="v0.41.3"
+NGINX_TAG="v2.4.0"
+DOCKER_TAG="v1.0.0"
+LOCALE_TAG="v0.3.4"
+
arguments ${@}
if [ -s ${CONFIG} ]; then
@@ -156,9 +172,7 @@ pillar_roots:
- ${P_DIR}
EOFSM
-mkdir -p ${S_DIR}
-mkdir -p ${F_DIR}
-mkdir -p ${P_DIR}
+mkdir -p ${S_DIR} ${F_DIR} ${P_DIR}
# States
cat > ${S_DIR}/top.sls << EOFTSLS
@@ -227,8 +241,15 @@ for f in "${SOURCE_PILLARS_DIR}"/*; do
sed "s/__CLUSTER__/${CLUSTER}/g;
s/__DOMAIN__/${DOMAIN}/g;
s/__RELEASE__/${RELEASE}/g;
+ s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
+ s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
+ s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
+ s/__WORKBENCH1_EXT__SSL_PORT__/${WORKBENCH1_EXT__SSL_PORT}/g;
+ s/__WORKBENCH2_EXT__SSL_PORT__/${WORKBENCH2_EXT__SSL_PORT}/g;
+ s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__HOSTNAME__/${HOSTNAME}/g;
+ s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
- s/__GUEST_SSL_PORT__/${GUEST_SSL_PORT}/g;
s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
commit c0bfa951ebd8e56c4feb0443a1b414ffe8eb63c1
Author: Javier Bértoli <jbertoli at curii.com>
Date: Mon Jan 25 07:10:01 2021 -0300
feat(provision): manage setup params from local file
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/Vagrantfile b/tools/salt-install/Vagrantfile
index 6966ea834..60f57ca66 100644
--- a/tools/salt-install/Vagrantfile
+++ b/tools/salt-install/Vagrantfile
@@ -34,6 +34,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
path: "provision.sh",
args: [
# "--debug",
+ "--config /vagrant/local.params",
"--test",
"--vagrant",
"--ssl-port=8443"
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 79712d3f9..b97d71965 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -24,43 +24,37 @@ NGINX_TAG="v2.4.0"
DOCKER_TAG="v1.0.0"
LOCALE_TAG="v0.3.4"
-if [ -s ${SCRIPT_DIR}/local.params ]; then
- source ${SCRIPT_DIR}/local.params
-else
- echo >&2 "Please create a '${SCRIPT_DIR}/local.params' file with initial values, as described in FIXME_URL_TO_DESCR"
- exit 1
-fi
-
usage() {
echo >&2
echo >&2 "Usage: ${0} [-h] [-h]"
echo >&2
echo >&2 "${0} options:"
- echo >&2 " -d, --debug Run salt installation in debug mode"
- echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
- echo >&2 " -t, --test Test installation running a CWL workflow"
- echo >&2 " -r, --roles List of Arvados roles to apply to the host, comma separated"
- echo >&2 " Possible values are:"
- echo >&2 " api"
- echo >&2 " controller"
- echo >&2 " keepstore"
- echo >&2 " websocket"
- echo >&2 " keepweb"
- echo >&2 " workbench2"
- echo >&2 " keepproxy"
- echo >&2 " shell"
- echo >&2 " workbench"
- echo >&2 " dispatcher"
- echo >&2 " Defaults to applying them all"
- echo >&2 " -h, --help Display this help and exit"
- echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
+ echo >&2 " -d, --debug Run salt installation in debug mode"
+ echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
+ echo >&2 " -c <local.params>, --config <local.params> Path to the local.params config file"
+ echo >&2 " -t, --test Test installation running a CWL workflow"
+ echo >&2 " -r, --roles List of Arvados roles to apply to the host, comma separated"
+ echo >&2 " Possible values are:"
+ echo >&2 " api"
+ echo >&2 " controller"
+ echo >&2 " keepstore"
+ echo >&2 " websocket"
+ echo >&2 " keepweb"
+ echo >&2 " workbench2"
+ echo >&2 " keepproxy"
+ echo >&2 " shell"
+ echo >&2 " workbench"
+ echo >&2 " dispatcher"
+ echo >&2 " Defaults to applying them all"
+ echo >&2 " -h, --help Display this help and exit"
+ echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
echo >&2
}
arguments() {
# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
- TEMP=$(getopt -o dhp:r:tv \
- --long debug,help,ssl-port:,roles:,test,vagrant \
+ TEMP=$(getopt -o c:dhp:r:tv \
+ --long config:,debug,help,ssl-port:,roles:,test,vagrant \
-n "${0}" -- "${@}")
if [ ${?} != 0 ] ; then echo "GNU getopt missing? Use -h for help"; exit 1 ; fi
@@ -69,6 +63,10 @@ arguments() {
while [ ${#} -ge 1 ]; do
case ${1} in
+ -c | --config)
+ CONFIG=${2}
+ shift 2
+ ;;
-d | --debug)
LOG_LEVEL="debug"
shift
@@ -110,12 +108,20 @@ arguments() {
done
}
+CONFIG="${SCRIPT_DIR}/local.params"
LOG_LEVEL="info"
HOST_SSL_PORT=443
TESTS_DIR="tests"
arguments ${@}
+if [ -s ${CONFIG} ]; then
+ source ${CONFIG}
+else
+ echo >&2 "Please create a '${CONFIG}' file with initial values, as described in FIXME_URL_TO_DESCR"
+ exit 1
+fi
+
# Salt's dir
## states
S_DIR="/srv/salt"
commit 9074ec89ae2bd38072cd1296be788168be9e3618
Author: Javier Bértoli <jbertoli at curii.com>
Date: Fri Jan 22 12:41:12 2021 -0300
feat(provision): allow to install individual roles
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/README.md b/tools/salt-install/README.md
index 10d08b414..58800dd6f 100644
--- a/tools/salt-install/README.md
+++ b/tools/salt-install/README.md
@@ -6,15 +6,21 @@
##### About
-This directory holds a small script to install Arvados on a single node, using the
+This directory holds a small script to help you get Arvados up and running, using the
[Saltstack arvados-formula](https://github.com/saltstack-formulas/arvados-formula)
in master-less mode.
-The fastest way to get it running is to modify the first lines in the `provision.sh`
-script to suit your needs, copy it in the host where you want to install Arvados
-and run it as root.
+There are a few preset examples that you can use:
-There's an example `Vagrantfile` also, to install it in a vagrant box if you want
+* `single_host`: Install all the Arvados components in a single host. Suitable for testing
+ or demo-ing, but not recommended for production use.
+* `multi_host/aws`: Let's you install different Arvados components in different hosts on AWS.
+
+The fastest way to get it running is to copy the `local.params.example` file to `local.params`,
+edit and modify the file to suit your needs, copy this file along with the `provision.sh` script
+into the host where you want to install Arvados and run the `provision.sh` script as root.
+
+There's an example `Vagrantfile` also, to install Arvados in a vagrant box if you want
to try it locally.
For more information, please read https://doc.arvados.org/main/install/salt-single-host.html
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 31266c1b8..79712d3f9 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -10,55 +10,26 @@
#
# vagrant up
-##########################################################
-# This section are the basic parameters to configure the installation
-
-# The 5 letters name you want to give your cluster
-CLUSTER="arva2"
-DOMAIN="arv.local"
-
-INITIAL_USER="admin"
+set -o pipefail
-# If not specified, the initial user email will be composed as
-# INITIAL_USER at CLUSTER.DOMAIN
-INITIAL_USER_EMAIL="${INITIAL_USER}@${CLUSTER}.${DOMAIN}"
-INITIAL_USER_PASSWORD="password"
+# capture the directory that the script is running from
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-# The example config you want to use. Currently, only "single_host" is
-# available
CONFIG_DIR="single_host"
-
-# Which release of Arvados repo you want to use
RELEASE="production"
-# Which version of Arvados you want to install. Defaults to 'latest'
-# in the desired repo
VERSION="latest"
-
-# Host SSL port where you want to point your browser to access Arvados
-# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
-# You can point it to another port if desired
-# In Vagrant, make sure it matches what you set in the Vagrantfile
-# HOST_SSL_PORT=443
-
-# This is a arvados-formula setting.
-# If branch is set, the script will switch to it before running salt
-# Usually not needed, only used for testing
-# BRANCH="master"
-
-##########################################################
-# Usually there's no need to modify things below this line
-
-# Formulas versions
ARVADOS_TAG="v1.1.4"
POSTGRES_TAG="v0.41.3"
NGINX_TAG="v2.4.0"
DOCKER_TAG="v1.0.0"
LOCALE_TAG="v0.3.4"
-set -o pipefail
-
-# capture the directory that the script is running from
-SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+if [ -s ${SCRIPT_DIR}/local.params ]; then
+ source ${SCRIPT_DIR}/local.params
+else
+ echo >&2 "Please create a '${SCRIPT_DIR}/local.params' file with initial values, as described in FIXME_URL_TO_DESCR"
+ exit 1
+fi
usage() {
echo >&2
@@ -68,6 +39,19 @@ usage() {
echo >&2 " -d, --debug Run salt installation in debug mode"
echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
echo >&2 " -t, --test Test installation running a CWL workflow"
+ echo >&2 " -r, --roles List of Arvados roles to apply to the host, comma separated"
+ echo >&2 " Possible values are:"
+ echo >&2 " api"
+ echo >&2 " controller"
+ echo >&2 " keepstore"
+ echo >&2 " websocket"
+ echo >&2 " keepweb"
+ echo >&2 " workbench2"
+ echo >&2 " keepproxy"
+ echo >&2 " shell"
+ echo >&2 " workbench"
+ echo >&2 " dispatcher"
+ echo >&2 " Defaults to applying them all"
echo >&2 " -h, --help Display this help and exit"
echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
echo >&2
@@ -75,8 +59,8 @@ usage() {
arguments() {
# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
- TEMP=$(getopt -o dhp:tv \
- --long debug,help,ssl-port:,test,vagrant \
+ TEMP=$(getopt -o dhp:r:tv \
+ --long debug,help,ssl-port:,roles:,test,vagrant \
-n "${0}" -- "${@}")
if [ ${?} != 0 ] ; then echo "GNU getopt missing? Use -h for help"; exit 1 ; fi
@@ -89,6 +73,23 @@ arguments() {
LOG_LEVEL="debug"
shift
;;
+ -p | --ssl-port)
+ HOST_SSL_PORT=${2}
+ shift 2
+ ;;
+ -r | --roles)
+ for i in ${2//,/ }
+ do
+ # Verify the role exists
+ if [[ ! "api,controller,keepstore,websocket,keepweb,workbench2,keepproxy,shell,workbench,dispatcher" == *"$i"* ]]; then
+ echo "The role '${i}' is not a valid role"
+ usage
+ exit 1
+ fi
+ ROLES="${ROLES} ${i}"
+ done
+ shift 2
+ ;;
-t | --test)
TEST="yes"
shift
@@ -97,10 +98,6 @@ arguments() {
VAGRANT="yes"
shift
;;
- -p | --ssl-port)
- HOST_SSL_PORT=${2}
- shift 2
- ;;
--)
shift
break
@@ -167,9 +164,17 @@ base:
- nginx.passenger
- postgres
- docker
- - arvados
EOFTSLS
+# If we want specific roles for a node, just add those states
+if [ -z "${ROLES}" ]; then
+ echo ' - arvados' >> ${S_DIR}/top.sls
+else
+ for R in ${ROLES}; do
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ done
+fi
+
# Pillars
cat > ${P_DIR}/top.sls << EOFPSLS
base:
@@ -191,7 +196,7 @@ EOFPSLS
# Get the formula and dependencies
cd ${F_DIR} || exit 1
-git clone --branch "${ARVADOS_TAG}" https://github.com/saltstack-formulas/arvados-formula.git
+git clone --branch "${ARVADOS_TAG}" https://github.com/arvados/arvados-formula.git
git clone --branch "${DOCKER_TAG}" https://github.com/saltstack-formulas/docker-formula.git
git clone --branch "${LOCALE_TAG}" https://github.com/saltstack-formulas/locale-formula.git
git clone --branch "${NGINX_TAG}" https://github.com/saltstack-formulas/nginx-formula.git
commit b3852e1fd52fa0471555004f14e8e9448480c935
Author: Javier Bértoli <jbertoli at curii.com>
Date: Fri Jan 15 09:36:07 2021 -0300
Documentation: Explain that the canonical salt installer is hosted in Arvados' github account
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/doc/install/salt-multi-host.html.textile.liquid b/doc/install/salt-multi-host.html.textile.liquid
index 4ba153faf..50de6e439 100644
--- a/doc/install/salt-multi-host.html.textile.liquid
+++ b/doc/install/salt-multi-host.html.textile.liquid
@@ -40,25 +40,25 @@ The formulas we use are:
* "docker":https://github.com/saltstack-formulas/docker-formula.git
* "locale":https://github.com/saltstack-formulas/locale-formula.git
-There are example Salt pillar files for each of those formulas in the "arvados-formula's test/salt/pillar/examples":https://github.com/saltstack-formulas/arvados-formula/tree/master/test/salt/pillar/examples directory. As they are, they allow you to get all the main Arvados components up and running.
+There are example Salt pillar files for each of those formulas in the "arvados-formula's test/salt/pillar/examples":https://github.com/arvados/arvados-formula/tree/master/test/salt/pillar/examples directory. As they are, they allow you to get all the main Arvados components up and running.
h2(#saltstack). Install Arvados using Saltstack
This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
-The Arvados formula we maintain is located in the Saltstack's community repository of formulas:
+The Arvados formula we maintain is located in Arvados' Github account and should be considered the canonical place to download its most up-to-date version:
-* "arvados-formula":https://github.com/saltstack-formulas/arvados-formula.git
+* "arvados-formula":https://github.com/arvados/arvados-formula.git
-The @development@ version lives in our own repository
+As the Saltstack's community keeps a "repository of formulas":https://github.com/saltstack-formulas/ in Github, we also provide
-* "arvados-formula development":https://github.com/arvados/arvados-formula.git
+* "a copy of the formula":https://github.com/saltstack-formulas/arvados-formula.git
-This last one might break from time to time, as we try and add new features. Use with caution.
+there, and do our best effort to keep it in sync with ours.
-As much as possible, we try to keep it up to date, with example pillars to help you deploy Arvados.
+A @development@ branch exists which uses Arvados' development repositories. This last one might break from time to time, as we try and add new features. As much as possible, we try to keep it up to date, with example pillars to help you deploy Arvados. Use with caution.
-For those familiar with Saltstack, the process to get it deployed is similar to any other formula:
+For those familiar with Saltstack, the process to get Arvados deployed is similar to any other formula:
1. Fork/copy the formula to your Salt master host.
2. Edit the Arvados, nginx, postgres, locale and docker pillars to match your desired configuration.
diff --git a/doc/install/salt.html.textile.liquid b/doc/install/salt.html.textile.liquid
index 8f5ecc8c6..2b7aa6602 100644
--- a/doc/install/salt.html.textile.liquid
+++ b/doc/install/salt.html.textile.liquid
@@ -14,7 +14,7 @@ SPDX-License-Identifier: CC-BY-SA-3.0
h2(#introduction). Introduction
-To ease the installation of the various Arvados components, we have developed a "Saltstack":https://www.saltstack.com/ 's "arvados-formula":https://github.com/saltstack-formulas/arvados-formula which can help you get an Arvados cluster up and running.
+To ease the installation of the various Arvados components, we have developed a "Saltstack":https://www.saltstack.com/ 's "arvados-formula":https://github.com/arvados/arvados-formula which can help you get an Arvados cluster up and running.
Saltstack is a Python-based, open-source software for event-driven IT automation, remote task execution, and configuration management. It can be used in a master/minion setup or master-less.
@@ -24,6 +24,6 @@ h2(#installmethod). Choose an installation method
The salt formulas can be used in different ways. Choose one of these three options to install Arvados:
-* "Use Vagrant to install Arvados in a virtual machine":salt-vagrant.html
* "Arvados on a single host":salt-single-host.html
+* "Use Vagrant to install Arvados in a virtual machine":salt-vagrant.html
* "Arvados across multiple hosts":salt-multi-host.html
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list