[ARVADOS] updated: 2.1.0-396-gdda765028
Git user
git at public.arvados.org
Thu Feb 11 14:40:48 UTC 2021
Summary of changes:
.../container_requests_controller_test.rb | 1 -
build/package-build-dockerfiles/Makefile | 4 +-
build/package-build-dockerfiles/centos7/Dockerfile | 4 +-
.../package-build-dockerfiles/debian10/Dockerfile | 4 +-
.../ubuntu1604/Dockerfile | 4 +-
.../ubuntu1804/Dockerfile | 4 +-
.../ubuntu2004/Dockerfile | 4 +-
build/run-library.sh | 4 +
build/run-tests.sh | 4 +-
cmd/arvados-package/build.go | 155 ++++++++++
cmd/arvados-package/build_test.go | 68 +++++
cmd/arvados-package/cmd.go | 143 +++++++++
cmd/arvados-package/fpm.go | 127 ++++++++
cmd/arvados-package/install.go | 134 ++++++++
cmd/arvados-server/cmd.go | 1 +
doc/Rakefile | 9 +
doc/_config.yml | 7 +-
doc/_includes/_install_compute_docker.liquid | 7 +-
doc/admin/activation.html.textile.liquid | 1 -
doc/admin/federation.html.textile.liquid | 4 +-
doc/admin/troubleshooting.html.textile.liquid | 1 -
doc/admin/user-management-cli.html.textile.liquid | 8 +-
doc/admin/user-management.html.textile.liquid | 4 +-
doc/api/keep-web-urls.html.textile.liquid | 10 +
.../methods/container_requests.html.textile.liquid | 2 +-
doc/api/methods/users.html.textile.liquid | 6 +-
doc/api/permission-model.html.textile.liquid | 4 +-
doc/api/tokens.html.textile.liquid | 2 +-
.../keep-components-overview.html.textile.liquid | 61 ++++
doc/install/automatic.html.textile.liquid | 47 +++
doc/install/cheat_sheet.html.textile.liquid | 1 -
doc/install/client.html.textile.liquid | 14 -
.../install-dispatch-cloud.html.textile.liquid | 4 +
....liquid => configure-slurm.html.textile.liquid} | 26 +-
.../install-prerequisites.html.textile.liquid | 10 -
doc/install/install-keep-web.html.textile.liquid | 13 +-
.../install-manual-overview.html.textile.liquid | 20 --
...nstall-manual-prerequisites.html.textile.liquid | 2 +-
doc/install/pre-built-docker.html.textile.liquid | 75 -----
doc/user/composer/c1.png | Bin 17420 -> 0 bytes
doc/user/composer/c10.png | Bin 33020 -> 0 bytes
doc/user/composer/c11.png | Bin 26144 -> 0 bytes
doc/user/composer/c12.png | Bin 175462 -> 0 bytes
doc/user/composer/c13.png | Bin 11645 -> 0 bytes
doc/user/composer/c14.png | Bin 13116 -> 0 bytes
doc/user/composer/c15.png | Bin 50066 -> 0 bytes
doc/user/composer/c16.png | Bin 32472 -> 0 bytes
doc/user/composer/c17.png | Bin 34259 -> 0 bytes
doc/user/composer/c18.png | Bin 59780 -> 0 bytes
doc/user/composer/c19.png | Bin 53580 -> 0 bytes
doc/user/composer/c2.png | Bin 41693 -> 0 bytes
doc/user/composer/c20.png | Bin 23625 -> 0 bytes
doc/user/composer/c21.png | Bin 11379 -> 0 bytes
doc/user/composer/c22.png | Bin 13713 -> 0 bytes
doc/user/composer/c23.png | Bin 12253 -> 0 bytes
doc/user/composer/c24.png | Bin 19931 -> 0 bytes
doc/user/composer/c2b.png | Bin 14285 -> 0 bytes
doc/user/composer/c2c.png | Bin 9123 -> 0 bytes
doc/user/composer/c3.png | Bin 38471 -> 0 bytes
doc/user/composer/c4.png | Bin 8811 -> 0 bytes
doc/user/composer/c5.png | Bin 24020 -> 0 bytes
doc/user/composer/c6.png | Bin 41978 -> 0 bytes
doc/user/composer/c7.png | Bin 26904 -> 0 bytes
doc/user/composer/c8.png | Bin 44358 -> 0 bytes
doc/user/composer/c9.png | Bin 18920 -> 0 bytes
doc/user/composer/composer.html.textile.liquid | 119 --------
.../getting_started/setup-cli.html.textile.liquid | 20 ++
.../tutorials/intro-crunch.html.textile.liquid | 28 --
lib/boot/cert.go | 22 +-
lib/boot/cmd.go | 6 +
lib/boot/helpers.go | 144 +++++++++
lib/boot/nginx.go | 45 ++-
lib/boot/passenger.go | 59 +++-
lib/boot/postgresql.go | 31 +-
lib/boot/seed.go | 7 +-
lib/boot/service.go | 10 +-
lib/boot/supervisor.go | 204 +++++++++----
lib/cloud/ec2/ec2.go | 21 +-
lib/config/config.default.yml | 12 +-
lib/config/generated_config.go | 12 +-
lib/controller/cmd.go | 1 +
lib/controller/federation/conn.go | 62 ++++
lib/controller/federation/generate.go | 2 +-
lib/controller/federation/generated.go | 41 +++
lib/controller/federation_test.go | 137 +++++----
lib/controller/handler.go | 2 +
lib/controller/handler_test.go | 2 +
lib/controller/integration_test.go | 338 ++++++++++++---------
lib/controller/localdb/conn.go | 9 +-
lib/controller/localdb/login.go | 31 +-
lib/controller/localdb/login_ldap.go | 6 +-
lib/controller/localdb/login_ldap_test.go | 4 +-
lib/controller/localdb/login_oidc.go | 20 +-
lib/controller/localdb/login_oidc_test.go | 60 ++++
lib/controller/localdb/login_pam.go | 6 +-
lib/controller/localdb/login_pam_test.go | 4 +-
lib/controller/localdb/login_testuser.go | 6 +-
lib/controller/localdb/login_testuser_test.go | 4 +-
lib/controller/router/response.go | 76 +++--
lib/controller/router/router.go | 35 +++
lib/controller/rpc/conn.go | 63 +++-
lib/crunchrun/crunchrun.go | 39 ++-
lib/crunchrun/crunchrun_test.go | 6 +-
lib/dispatchcloud/dispatcher.go | 6 +
lib/install/deps.go | 291 +++++++++++++++---
lib/install/init.go | 267 ++++++++++++++++
sdk/cwl/arvados_cwl/arvcontainer.py | 11 +
sdk/cwl/arvados_cwl/executor.py | 7 +-
sdk/cwl/arvados_cwl/pathmapper.py | 9 +-
sdk/cwl/arvados_cwl/runner.py | 3 +-
.../cases/cat.cwl => 17267-broken-schemas.cwl} | 13 +-
sdk/cwl/tests/arvados-tests.yml | 10 +
sdk/go/arvados/api.go | 10 +
sdk/go/arvados/container.go | 5 +-
sdk/go/arvadostest/api.go | 20 ++
sdk/go/health/aggregator.go | 3 +
sdk/python/tests/nginx.conf | 70 ++---
.../app/controllers/user_sessions_controller.rb | 35 ++-
services/api/app/models/arvados_model.rb | 34 +++
services/api/app/models/container.rb | 10 +-
services/api/app/models/container_request.rb | 25 +-
services/api/test/fixtures/container_requests.yml | 47 ++-
.../v1/container_requests_controller_test.rb | 25 +-
.../functional/user_sessions_controller_test.rb | 25 ++
services/api/test/unit/container_request_test.rb | 47 +--
services/api/test/unit/container_test.rb | 14 +-
services/keep-balance/main.go | 8 +-
services/keep-web/s3_test.go | 1 +
tools/compute-images/arvados-images-aws.json | 3 +-
tools/compute-images/arvados-images-azure.json | 3 +-
tools/compute-images/build.sh | 13 +-
tools/keep-block-check/keep-block-check.go | 2 +-
tools/keep-block-check/keep-block-check_test.go | 10 +-
tools/salt-install/Vagrantfile | 66 +++-
...Vagrantfile.single_host_single_hostname.example | 87 ++++++
.../multiple_hostnames/pillars/arvados.sls | 15 +-
.../pillars/nginx_controller_configuration.sls | 2 +-
.../pillars/nginx_keepproxy_configuration.sls | 2 +-
.../pillars/nginx_keepweb_configuration.sls | 2 +-
.../pillars/nginx_webshell_configuration.sls | 2 +-
.../pillars/nginx_websocket_configuration.sls | 2 +-
.../pillars/nginx_workbench2_configuration.sls | 4 +-
.../pillars/nginx_workbench_configuration.sls | 2 +-
.../single_hostname/pillars/arvados.sls | 1 -
.../pillars/nginx_webshell_configuration.sls | 4 +-
.../pillars/nginx_workbench2_configuration.sls | 2 +-
tools/salt-install/local.params.example | 29 +-
tools/salt-install/provision.sh | 302 +++++++++++-------
tools/salt-install/tests/run-test.sh | 2 +-
tools/sync-groups/federation_test.go | 211 +++++++++++++
150 files changed, 3336 insertions(+), 1119 deletions(-)
create mode 100644 cmd/arvados-package/build.go
create mode 100644 cmd/arvados-package/build_test.go
create mode 100644 cmd/arvados-package/cmd.go
create mode 100644 cmd/arvados-package/fpm.go
create mode 100644 cmd/arvados-package/install.go
delete mode 120000 doc/admin/activation.html.textile.liquid
delete mode 120000 doc/admin/troubleshooting.html.textile.liquid
create mode 100644 doc/architecture/keep-components-overview.html.textile.liquid
create mode 100644 doc/install/automatic.html.textile.liquid
delete mode 120000 doc/install/cheat_sheet.html.textile.liquid
delete mode 100644 doc/install/client.html.textile.liquid
rename doc/install/crunch2-slurm/{install-slurm.html.textile.liquid => configure-slurm.html.textile.liquid} (88%)
delete mode 100644 doc/install/crunch2-slurm/install-prerequisites.html.textile.liquid
delete mode 100644 doc/install/install-manual-overview.html.textile.liquid
delete mode 100644 doc/install/pre-built-docker.html.textile.liquid
delete mode 100644 doc/user/composer/c1.png
delete mode 100644 doc/user/composer/c10.png
delete mode 100644 doc/user/composer/c11.png
delete mode 100644 doc/user/composer/c12.png
delete mode 100644 doc/user/composer/c13.png
delete mode 100644 doc/user/composer/c14.png
delete mode 100644 doc/user/composer/c15.png
delete mode 100644 doc/user/composer/c16.png
delete mode 100644 doc/user/composer/c17.png
delete mode 100644 doc/user/composer/c18.png
delete mode 100644 doc/user/composer/c19.png
delete mode 100644 doc/user/composer/c2.png
delete mode 100644 doc/user/composer/c20.png
delete mode 100644 doc/user/composer/c21.png
delete mode 100644 doc/user/composer/c22.png
delete mode 100644 doc/user/composer/c23.png
delete mode 100644 doc/user/composer/c24.png
delete mode 100644 doc/user/composer/c2b.png
delete mode 100644 doc/user/composer/c2c.png
delete mode 100644 doc/user/composer/c3.png
delete mode 100644 doc/user/composer/c4.png
delete mode 100644 doc/user/composer/c5.png
delete mode 100644 doc/user/composer/c6.png
delete mode 100644 doc/user/composer/c7.png
delete mode 100644 doc/user/composer/c8.png
delete mode 100644 doc/user/composer/c9.png
delete mode 100644 doc/user/composer/composer.html.textile.liquid
create mode 100644 doc/user/getting_started/setup-cli.html.textile.liquid
delete mode 100644 doc/user/tutorials/intro-crunch.html.textile.liquid
create mode 100644 lib/boot/helpers.go
create mode 100644 lib/install/init.go
copy sdk/cwl/tests/{federation/cases/cat.cwl => 17267-broken-schemas.cwl} (56%)
create mode 100644 tools/salt-install/Vagrantfile.single_host_single_hostname.example
create mode 100644 tools/sync-groups/federation_test.go
discards 45b514e1d644f9cb6d6b7ff0acb8372c8ac4d9f8 (commit)
discards 3aa47e60ebd104793c52c1b4e5145f73b51571e3 (commit)
discards 2a662b839254ff1ba35641834e4546c013aba2bd (commit)
discards 102277e608eca622b0ea568845744921e48fe2e7 (commit)
discards 677877d5327f559730ca2d8483f48da16880a9b5 (commit)
discards 2e6894963f87d0b3f955de8c75acfac8451fd6b2 (commit)
via dda765028860f60b35682151a35cbdd86205295a (commit)
via 86d1e1b0dd81a40ad2a404b8d0ba7ce4f768cb40 (commit)
via 9d52d145339f979c3ff57e5145c2c6d4ab036ad3 (commit)
via f44a0323c6a8cb88f72c6a2440185596ff579cf9 (commit)
via 8579c7a81e432a4c6fdb0e29947a3093783e63d9 (commit)
via d0218fb8eb1b5d87a33435a4bf09f5ea1e22af25 (commit)
via a396085e56ad9bec41468c745a4212367323dc18 (commit)
via d6ec1985ae59b64ca055a50f2a84f54682b27987 (commit)
via ffd332a1849c7fef8aecfb27c442d67a34e29294 (commit)
via c590159bf245425b96187358b53498999fa64051 (commit)
via 6c058acc4ea63b29ef049715cc5bb104ca4e7bd7 (commit)
via 33160dea02e7552732fe23cbcc3e061b1a5245bb (commit)
via 2b9350438e027719deecf85bfdf9fc9ae4ef177d (commit)
via b57db2bcf9ba4ef5f74b9b1e4c0a0788f6439658 (commit)
via e15b1cd2d4c29381d0ccbdc33cb48e5034dab2f9 (commit)
via 0ba33e2045cf58071f8e60d31eb9e00274a12e9e (commit)
via b27a423ff3ece7cf87e13bbcc30a3ada02a35d70 (commit)
via 08c80b2ba6a8fc090d2e998dfcf2f280c8508019 (commit)
via c797b629ddb0cf2808123a1b86d31b0153f1aca0 (commit)
via 22fcc610c13c817c415cff26f2c94c17f271e4d3 (commit)
via 81d57969ee70d6118937b4f4f61f6e1cd235de44 (commit)
via 78762462bbea3d51223bdf0ed01668ac1bbbcedb (commit)
via 40e21ed3a79d7da89b9805a700ff3844c7d9a610 (commit)
via f6519a2ffd957ed31d8a8c92da2f1d4ffed71b04 (commit)
via b5c47e5ca546f893caa6189671966d98af6665ab (commit)
via eb0dd856470fe91ba4592ee24c05a5fed11af217 (commit)
via d515c8eb95b39065f74ea6d0b368d8ca3705fd4a (commit)
via 64f732dbb2c0fac4cd108f142f28d6d2efeae324 (commit)
via 020482b349bf285ed726dbf05521b686be7daa2b (commit)
via e28e5002e502a5c238e6c0ae5daede811af9aa55 (commit)
via 225adb2aa5ae0b3af3b799c9001a9edc65c7d3d0 (commit)
via a795a8801fecdcbe5418c5d631b5b1ba1b22f1d7 (commit)
via 331ebba87c6b1fa5d90a842ad0beee5d0f67155c (commit)
via ae3d77edc33d0c9e11af1a59478feaf7251d8ca1 (commit)
via 3e7ab60f94a6076383bf4aec41b6271310ab8823 (commit)
via 8646fae34fff9aa379a34388e90106f31d5206a4 (commit)
via a847c48edba67805cff0104386845be799d74073 (commit)
via 3d1a90b959e3035de4f3be1c511a721291507b5f (commit)
via c81ce0ff89ad019d982bdccb376a823894eea733 (commit)
via 3cb8be1e32a1b2e477501c38cefa4c48bbbb8c29 (commit)
via 2135f16dd8edc4c9b3cded88eb907dd24cc65293 (commit)
via e60380ca96dab6348b9b6ca40e5d09867b94ab85 (commit)
via c72e7388b50feaf79651719d6e3a7fbe8239c334 (commit)
via 7885ae2c39bd2e7a38943fe9a56463fb4349a5ac (commit)
via 5a1a3d656c6d2d92edaed19c735a9b1e1eb39f71 (commit)
via 4865911a605128adb454b3280b1cf9dcd38f499e (commit)
via 6470f7ce527c2cc44e25de402bd0418c821d13a8 (commit)
via fc3531079e960359afb1039a5fd24a059a813baa (commit)
via 7631343e1bfafddca31b1ababbaa63c8a9aea1bb (commit)
via 969441a091ce3aa1eb7a9525d3ab85f24fbd8fdd (commit)
via f0d3eae5fc05aaad38a2998627c59637e3ef606c (commit)
via bec7d0354140620311407073f08573b5d1c037d5 (commit)
via 75efbc85be494c802f3c6822875cb64b9a34de6b (commit)
via 714cc9a889786a062641f3627d557667cc71c337 (commit)
via 01e15db1f4a331508117bc841256acec8ca361de (commit)
via 5e846bd0291b1eba00b294373fde76bf95a9e091 (commit)
via bff33bf988ea57ef226d108c574a68053be287ce (commit)
via 7c99875efea3ec5415427063cf79ddc44c58f55f (commit)
via 8cbdc1b04e64e2d5e394e86aba699b361bcdc24a (commit)
via 80dbda890bf58bda79654cf4cebdfbc2b07d6b1f (commit)
via 18def2a271e02fd64749fe650034f50d1b659e45 (commit)
via 591922d85a4cbb6fc890bde7b5f4848f4c92468a (commit)
via 4248a4ebd23813e9bdcd68547ae03ff7d6082463 (commit)
via fbc95892b4b8cce3cba9ae024c252bd31146c714 (commit)
via a6fd7801f084f280cbf668f8a1f6bf9d9bbd0def (commit)
via 8cd08f2ce640e0b1967db489d29e3761ac63f0d7 (commit)
via 41dec55a4b007a79c58f77d36ad0940b5354b453 (commit)
via 4c6c49190b5a8949120d822e053657f64146df70 (commit)
via 3576206ef265d0040bcc93899b9885f16b5919e6 (commit)
via bd535934937eb0863bd8eaec0b62ab81bc8e4700 (commit)
via 5727f64521ea7222422dd48e48793a0fe10253f4 (commit)
via 882bfd2383da2f9d3cbd2b5f258e727e385a4fbb (commit)
via 3ec51dcf456b2afe02857089895a261653abddd4 (commit)
via 8bed34a9d78506caff52f550d5e47224207ac73c (commit)
via dec7dc55c641f70c5966afc2661deff25dc2cf6d (commit)
via 11e02a3a1148f5f47de17b0fb0d33cb4042820b7 (commit)
via 4c30d75e647f42318fd0069613b3ed4f82c70ea0 (commit)
via efac197a128851bd5e894267b3b7a75268182f94 (commit)
via 8889791d095df8578745e0a1b3c2f4a721f54123 (commit)
via 56816829f5708cb4b3915c1f2a9387fcf42653ca (commit)
via b01ff2414daaf5fd8ff7f0e78ed49e63d431ccd3 (commit)
via a4a17913e299851f0b5979189d36997a62ff43e1 (commit)
via edc2687671a2fec74304c6e00092034d50c8f667 (commit)
via d20342c56db7e148f793a29fd79619ce2e3f0243 (commit)
via 4085665f87c82939961e55bf004ca47c03ca30ac (commit)
via 1297d71125f5a694996147406edf38bdc913396c (commit)
via 29a8298caf922a2651fbd51baf7b97b58ae25840 (commit)
via 41305b5ac71cc9a306dc654c42c11ffcc4258a47 (commit)
via f64f557db0bfe6f33d434853a94ee5cff7e69a5d (commit)
via c271137b5f704f3df510b0b0dd8eceba25e006a8 (commit)
via 7ab4905be555e3e5f65c3020ccafe3cc88f9e839 (commit)
via 59e8b47bee1c9699cbb2d16369481bd688da6e3d (commit)
via b979f80113c4f8a4c13c96b4679c28788c099333 (commit)
via cc45e3eb3609af1fd3b186e3aa6e806bcf1e92ae (commit)
via 57c3fbfb0f75dc38be717d52f29abe5bcefea1bb (commit)
via 6cd0e19a277812f302f46693d824cc17d17ed6b1 (commit)
via d71d4273d1f3d0b6381efafa649b81c6b4107cf1 (commit)
via 2cfb41d14010e26d97df93c4cf8ad00f0ac01701 (commit)
via 985ece13dd7147f153c7a59c5b3665570c5943c8 (commit)
via e1756a76e8c7fed496f513909bd2c295e24cf8ee (commit)
via 9c139cc4e2f519e2deb3617918a9199b76c33f10 (commit)
via ae339cf6802741ea52fbe5cd8716306dcf99b021 (commit)
via 684b81a42cb9290bde284723603d6e4dbfd7ad35 (commit)
via d77f9e6f8a14e14f477dfbf944914c78d0805e9f (commit)
via 0137f9e3cd33fe676a862afd727563e2602e0984 (commit)
via 6c1fd146c0b22b374f00ef94c961df8e1e1913a7 (commit)
via 706c36ff5644bb675af36c15c5f30ef0adde9102 (commit)
via 3aaefcb3c76ff470b475d950398d01255e87712a (commit)
via c59af50bc2f7a366cd12a8dd6fc7d7e3b1c32480 (commit)
via f1bc1872a9ead4f3cb1c23c79507870c1b0f7b53 (commit)
via 3feaf72ad2a9a7b740bb3976f351f5020129eda4 (commit)
via 209403e32bdd6164404240af2959de86a3c46165 (commit)
via 54849feb0cf30bd3d78f14f1a41e6bc464fb80d6 (commit)
via d5c034b4e90d3fba74e5b0a879d8af424023d7b1 (commit)
via 20649f2d57f8e5c8bddfb672c164b7a96f75efab (commit)
via 821c7273388a56ad5ae9b89a0cf2f351388e8b61 (commit)
via da7a2e35f1c53a64a360c66db0c260a9ab2190f2 (commit)
via 0b6013ac5520b7508474705d654f1384994d10e5 (commit)
via ab5199f71c7eaf8bf8fe2b4477353cf432faf1a7 (commit)
via f8d13408e99839f52260f889a5089126761eecb1 (commit)
via e8d1a643cdbc3a5f4c0e5c745da58d9f7e1248d8 (commit)
via 426103b2d2f071ab0d57b3f9aaea58a3f8455c4f (commit)
via bb1054c0e6a9bed14e593004f380111b2e948047 (commit)
via f2da5f54256377e604c76b74805381b2acf38f50 (commit)
via f81d411d2becc23fe640366620d9568149d89bd6 (commit)
via 24987125b33655306c048ab64f0ffdfb2f6a6c5b (commit)
via 1c2f9259deeedaad103b80fda7f440b8e60caa3f (commit)
via 75d050ab135619fcefecbfa32aaad4dab33e7588 (commit)
via 8891660b68c10dd9d019f1caf8b7bc0b1eb493d8 (commit)
via 398ddf58f05416e0ce62ac8faef018a44bf8cc4c (commit)
via 7ac1ea4e9bc1b399368a95649647a55c8093d649 (commit)
via 60801a7c485c50e3c2f518f9b3ad6f86c8783fca (commit)
via ee7d60f02525760a2480157011659e7b2210f6b8 (commit)
via 12441486d7fb3b51d2fea9d9b1da0d8e4f3ef3e0 (commit)
via b4d6c92791088a0b6a39a71b2b8b6ddc4b13af41 (commit)
This update added new revisions after undoing existing revisions. That is
to say, the old revision is not a strict subset of the new revision. This
situation occurs when you --force push a change and generate a repository
containing something like this:
* -- * -- B -- O -- O -- O (45b514e1d644f9cb6d6b7ff0acb8372c8ac4d9f8)
\
N -- N -- N (dda765028860f60b35682151a35cbdd86205295a)
When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit dda765028860f60b35682151a35cbdd86205295a
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 15:06:14 2021 -0300
refactor(provision): naming consistency
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index f74f001cb..9b19854d0 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -107,8 +107,12 @@ TESTS_DIR="tests"
CLUSTER=""
DOMAIN=""
+
+# Hostnames/IPs used for single-host deploys
HOSTNAME_EXT=""
HOSTNAME_INT="127.0.1.1"
+
+# Initial user setup
INITIAL_USER=""
INITIAL_USER_EMAIL=""
INITIAL_USER_PASSWORD=""
@@ -196,10 +200,10 @@ fi
if [ "x${VAGRANT}" = "xyes" ]; then
SOURCE_PILLARS_DIR="/vagrant/${CONFIG_DIR}/pillars"
- TESTS_DIR="/vagrant/${TESTS_DIR}"
+ SOURCE_TESTS_DIR="/vagrant/${TESTS_DIR}"
else
SOURCE_PILLARS_DIR="${SCRIPT_DIR}/${CONFIG_DIR}/pillars"
- TESTS_DIR="${SCRIPT_DIR}/${TESTS_DIR}"
+ SOURCE_TESTS_DIR="${SCRIPT_DIR}/${TESTS_DIR}"
fi
SOURCE_STATES_DIR="${EXTRA_STATES_DIR}"
@@ -233,7 +237,7 @@ done
mkdir -p /tmp/cluster_tests
# Replace cluster and domain name in the test files
-for f in "${TESTS_DIR}"/*; do
+for f in "${SOURCE_TESTS_DIR}"/*; do
sed "s/__CLUSTER__/${CLUSTER}/g;
s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
s/__DOMAIN__/${DOMAIN}/g;
commit 86d1e1b0dd81a40ad2a404b8d0ba7ce4f768cb40
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 12:01:17 2021 -0300
fix(provision): add missing roles sections
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index f91d67d69..f74f001cb 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -346,7 +346,7 @@ else
grep -q "nginx_passenger" ${P_DIR}/top.sls || echo " - nginx_passenger" >> ${P_DIR}/top.sls
grep -q "nginx_${R}_configuration" ${P_DIR}/top.sls || echo " - nginx_${R}_configuration" >> ${P_DIR}/top.sls
;;
- "workbench" | "workbench2" | "keepweb" | "keepproxy")
+ "controller" | "websocket" | "workbench" | "workbench2" | "keepweb" | "keepproxy")
# States
grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
@@ -369,6 +369,12 @@ else
# Pillars
# ATM, no specific pillar needed
;;
+ "keepstore")
+ # States
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ # Pillars
+ # ATM, no specific pillar needed
+ ;;
*)
echo "Unknown role ${R}"
exit 1
commit 9d52d145339f979c3ff57e5145c2c6d4ab036ad3
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 11:53:08 2021 -0300
fix(provision): make the salt-call binary check compatible with sh
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 08d17090c..f91d67d69 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -156,7 +156,7 @@ fi
apt-get update
apt-get install -y curl git jq
-if [ which salt-call ]; then
+if which salt-call; then
echo "Salt already installed"
else
curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
commit f44a0323c6a8cb88f72c6a2440185596ff579cf9
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 11:31:24 2021 -0300
fix(provision): add only specific pillars for each role
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index d30ff88a4..08d17090c 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -277,6 +277,9 @@ if [ -d "${SOURCE_STATES_DIR}" ]; then
fi
# Now, we build the SALT states/pillars trees
+# As we need to separate both states and pillars in case we want specific
+# roles, we iterate on both at the same time
+
# States
cat > ${S_DIR}/top.sls << EOFTSLS
base:
@@ -284,7 +287,16 @@ base:
- locale
EOFTSLS
-if [ -d "${SOURCE_STATES_DIR}" ]; then
+# Pillars
+cat > ${P_DIR}/top.sls << EOFPSLS
+base:
+ '*':
+ - locale
+ - arvados
+EOFPSLS
+
+# States, extra states
+if [ -d "${F_DIR}"/extra/extra ]; then
for f in "${F_DIR}"/extra/extra/*.sls; do
echo " - extra.$(basename ${f} | sed 's/.sls$//g')" >> ${S_DIR}/top.sls
done
@@ -293,58 +305,78 @@ fi
# If we want specific roles for a node, just add the desired states
# and its dependencies
if [ -z "${ROLES}" ]; then
- echo ' - nginx.passenger' >> ${S_DIR}/top.sls
- echo ' - postgres' >> ${S_DIR}/top.sls
- echo ' - docker' >> ${S_DIR}/top.sls
- echo ' - arvados' >> ${S_DIR}/top.sls
+ # States
+ echo " - nginx.passenger" >> ${S_DIR}/top.sls
+ echo " - postgres" >> ${S_DIR}/top.sls
+ echo " - docker" >> ${S_DIR}/top.sls
+ echo " - arvados" >> ${S_DIR}/top.sls
+
+ # Pillars
+ echo " - docker" >> ${P_DIR}/top.sls
+ echo " - nginx_api_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_controller_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_keepproxy_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_keepweb_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_passenger" >> ${P_DIR}/top.sls
+ echo " - nginx_websocket_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_webshell_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_workbench2_configuration" >> ${P_DIR}/top.sls
+ echo " - nginx_workbench_configuration" >> ${P_DIR}/top.sls
+ echo " - postgresql" >> ${P_DIR}/top.sls
else
# If we add individual roles, make sure we add the repo first
echo " - arvados.repo" >> ${S_DIR}/top.sls
for R in ${ROLES}; do
case "${R}" in
"database")
+ # States
echo " - postgres" >> ${S_DIR}/top.sls
+ # Pillars
+ echo ' - postgresql' >> ${P_DIR}/top.sls
;;
"api")
+ # States
# FIXME: https://dev.arvados.org/issues/17352
grep -q "postgres.client" ${S_DIR}/top.sls || echo " - postgres.client" >> ${S_DIR}/top.sls
grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ # Pillars
+ grep -q "docker" ${P_DIR}/top.sls || echo " - docker" >> ${P_DIR}/top.sls
+ grep -q "postgresql" ${P_DIR}/top.sls || echo " - postgresql" >> ${P_DIR}/top.sls
+ grep -q "nginx_passenger" ${P_DIR}/top.sls || echo " - nginx_passenger" >> ${P_DIR}/top.sls
+ grep -q "nginx_${R}_configuration" ${P_DIR}/top.sls || echo " - nginx_${R}_configuration" >> ${P_DIR}/top.sls
;;
"workbench" | "workbench2" | "keepweb" | "keepproxy")
+ # States
grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ # Pillars
+ grep -q "nginx_passenger" ${P_DIR}/top.sls || echo " - nginx_passenger" >> ${P_DIR}/top.sls
+ grep -q "nginx_${R}_configuration" ${P_DIR}/top.sls || echo " - nginx_${R}_configuration" >> ${P_DIR}/top.sls
;;
- "shell" | "dispatcher")
- grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ "shell")
+ # States
+ grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ # Pillars
+ grep -q "" ${P_DIR}/top.sls || echo " - docker" >> ${P_DIR}/top.sls
+ grep -q "nginx_webshell_configuration" ${P_DIR}/top.sls || echo " - nginx_webshell_configuration" >> ${P_DIR}/top.sls
+ ;;
+ "dispatcher")
+ # States
+ grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ # Pillars
+ # ATM, no specific pillar needed
;;
*)
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ echo "Unknown role ${R}"
+ exit 1
;;
esac
done
fi
-# Pillars
-cat > ${P_DIR}/top.sls << EOFPSLS
-base:
- '*':
- - arvados
- - docker
- - locale
- - nginx_api_configuration
- - nginx_controller_configuration
- - nginx_keepproxy_configuration
- - nginx_keepweb_configuration
- - nginx_passenger
- - nginx_websocket_configuration
- - nginx_webshell_configuration
- - nginx_workbench2_configuration
- - nginx_workbench_configuration
- - postgresql
-EOFPSLS
-
# FIXME! #16992 Temporary fix for psql call in arvados-api-server
if [ -e /root/.psqlrc ]; then
if ! ( grep 'pset pager off' /root/.psqlrc ); then
commit 8579c7a81e432a4c6fdb0e29947a3093783e63d9
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 09:35:19 2021 -0300
fix(provision): remove deprecated parameter
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
index 710c4da86..de9cd9648 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
@@ -80,7 +80,6 @@ arvados:
tokens:
system_root: __SYSTEM_ROOT_TOKEN__
management: __MANAGEMENT_TOKEN__
- rails_secret: __RAILS_SECRET_TOKEN__
anonymous_user: __ANONYMOUS_USER_TOKEN__
### KEYS
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
index f3d2bcb9e..31d3a0d50 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
@@ -80,7 +80,6 @@ arvados:
tokens:
system_root: __SYSTEM_ROOT_TOKEN__
management: __MANAGEMENT_TOKEN__
- rails_secret: __RAILS_SECRET_TOKEN__
anonymous_user: __ANONYMOUS_USER_TOKEN__
### KEYS
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example
index d02ba95c3..88d6a75d6 100644
--- a/tools/salt-install/local.params.example
+++ b/tools/salt-install/local.params.example
@@ -43,7 +43,6 @@ INITIAL_USER_PASSWORD="password"
BLOB_SIGNING_KEY=blobsigningkeymushaveatleast32characters
MANAGEMENT_TOKEN=managementtokenmushaveatleast32characters
SYSTEM_ROOT_TOKEN=systemroottokenmushaveatleast32characters
-RAILS_SECRET_TOKEN=railssecrettokenmushaveatleast32characters
ANONYMOUS_USER_TOKEN=anonymoususertokenmushaveatleast32characters
WORKBENCH_SECRET_KEY=workbenchsecretkeymushaveatleast32characters
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 1d9ae0fe9..d30ff88a4 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -220,7 +220,6 @@ for f in "${SOURCE_PILLARS_DIR}"/*; do
s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
- s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
s/__RELEASE__/${RELEASE}/g;
s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
s/__VERSION__/${VERSION}/g;
@@ -265,7 +264,6 @@ if [ -d "${SOURCE_STATES_DIR}" ]; then
s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
- s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
s/__RELEASE__/${RELEASE}/g;
s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
s/__VERSION__/${VERSION}/g;
commit d0218fb8eb1b5d87a33435a4bf09f5ea1e22af25
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 08:05:26 2021 -0300
fix(provision): check salt-call is installed
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index f5b986d1f..1d9ae0fe9 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -156,12 +156,12 @@ fi
apt-get update
apt-get install -y curl git jq
-dpkg -l |grep salt-minion
-if [ ${?} -eq 0 ]; then
+if [ which salt-call ]; then
echo "Salt already installed"
else
curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
sh /tmp/bootstrap_salt.sh -XdfP -x python3
+ /bin/systemctl stop salt-minion.service
/bin/systemctl disable salt-minion.service
fi
commit a396085e56ad9bec41468c745a4212367323dc18
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Feb 9 07:49:50 2021 -0300
fix(provision): add missing postgres dependency for arvados-api-server
refs #17246 & #17352
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 486f43057..f5b986d1f 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -307,11 +307,17 @@ else
"database")
echo " - postgres" >> ${S_DIR}/top.sls
;;
- "api","workbench","workbench2","keepweb","keepproxy")
+ "api")
+ # FIXME: https://dev.arvados.org/issues/17352
+ grep -q "postgres.client" ${S_DIR}/top.sls || echo " - postgres.client" >> ${S_DIR}/top.sls
grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
echo " - arvados.${R}" >> ${S_DIR}/top.sls
;;
- "shell","dispatcher")
+ "workbench" | "workbench2" | "keepweb" | "keepproxy")
+ grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ "shell" | "dispatcher")
grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
echo " - arvados.${R}" >> ${S_DIR}/top.sls
;;
commit d6ec1985ae59b64ca055a50f2a84f54682b27987
Author: Javier Bértoli <jbertoli at curii.com>
Date: Thu Feb 4 14:35:09 2021 -0300
fix(provision): refactor single host architectures
Allow to use a single-host/single-hostname or single-host/multiple-hostnames setup
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/Vagrantfile b/tools/salt-install/Vagrantfile
index 60f57ca66..666c6c48f 100644
--- a/tools/salt-install/Vagrantfile
+++ b/tools/salt-install/Vagrantfile
@@ -11,9 +11,10 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.ssh.insert_key = false
config.ssh.forward_x11 = true
- config.vm.define "arvados" do |arv|
+ # A single_host multiple_hostnames example
+ config.vm.define "arvados-sh-mn" do |arv|
arv.vm.box = "bento/debian-10"
- arv.vm.hostname = "vagrant.local"
+ arv.vm.hostname = "harpo.local"
# CPU/RAM
config.vm.provider :virtualbox do |v|
v.memory = 2048
@@ -21,23 +22,66 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
end
# Networking
+ # WEBUI PORT
arv.vm.network "forwarded_port", guest: 8443, host: 8443
- arv.vm.network "forwarded_port", guest: 25100, host: 25100
+ # KEEPPROXY
+ arv.vm.network "forwarded_port", guest: 25101, host: 25101
+ # KEEPWEB
arv.vm.network "forwarded_port", guest: 9002, host: 9002
- arv.vm.network "forwarded_port", guest: 9000, host: 9000
- arv.vm.network "forwarded_port", guest: 8900, host: 8900
+ # WEBSOCKET
arv.vm.network "forwarded_port", guest: 8002, host: 8002
- arv.vm.network "forwarded_port", guest: 8001, host: 8001
- arv.vm.network "forwarded_port", guest: 8000, host: 8000
- arv.vm.network "forwarded_port", guest: 3001, host: 3001
+ arv.vm.provision "shell",
+ inline: "sed 's#fixme#harpo#g;
+ s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=8443#g' \
+ /vagrant/local.params.example > /vagrant/local.params.single_host_multiple_hostnames"
arv.vm.provision "shell",
path: "provision.sh",
args: [
# "--debug",
- "--config /vagrant/local.params",
+ "--config /vagrant/local.params.single_host_multiple_hostnames",
"--test",
- "--vagrant",
- "--ssl-port=8443"
+ "--vagrant"
].join(" ")
end
+
+ ## # A single_host single_hostname example
+ ## config.vm.define "arvados-sh-sn" do |arv|
+ ## arv.vm.box = "bento/debian-10"
+ ## arv.vm.hostname = "zeppo.local"
+ ## # CPU/RAM
+ ## config.vm.provider :virtualbox do |v|
+ ## v.memory = 2048
+ ## v.cpus = 2
+ ## end
+
+ ## # Networking
+ ## arv.vm.network "forwarded_port", guest: 9443, host: 9443
+ ## arv.vm.network "forwarded_port", guest: 9444, host: 9444
+ ## arv.vm.network "forwarded_port", guest: 9445, host: 9445
+ ## arv.vm.network "forwarded_port", guest: 35101, host: 35101
+ ## arv.vm.network "forwarded_port", guest: 10002, host: 10002
+ ## arv.vm.network "forwarded_port", guest: 14202, host: 14202
+ ## arv.vm.network "forwarded_port", guest: 18002, host: 18002
+ ## arv.vm.provision "shell",
+ ## inline: "sed 's#HOSTNAME_EXT=\"\"#HOSTNAME_EXT=\"zeppo.local.cluster\"#g;
+ ## s#CLUSTER=\"fixme\"#CLUSTER=\"zeppo\"#g;
+ ## s#DOMAIN=\"some.domain\"#DOMAIN=\"local.cluster\"#g;
+ ## s#CONFIG_DIR=\"config_examples/single_host/multiple_hostnames\"#CONFIG_DIR=\"config_examples/single_host/single_hostname\"#g;
+ ## s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=9443#g;
+ ## s#KEEP_EXT_SSL_PORT=25101#KEEP_EXT_SSL_PORT=35101#g;
+ ## s#KEEPWEB_EXT_SSL_PORT=9002#KEEPWEB_EXT_SSL_PORT=11002#g;
+ ## s#WEBSHELL_EXT_SSL_PORT=4202#WEBSHELL_EXT_SSL_PORT=14202#g;
+ ## s#WEBSOCKET_EXT_SSL_PORT=8002#WEBSOCKET_EXT_SSL_PORT=18002#g;
+ ## s#WORKBENCH1_EXT_SSL_PORT=443#WORKBENCH1_EXT_SSL_PORT=9444#g;
+ ## s#WORKBENCH2_EXT_SSL_PORT=3001#WORKBENCH2_EXT_SSL_PORT=9445#g;' \
+ ## /vagrant/local.params.example > /vagrant/local.params.single_host_single_hostname"
+ ## arv.vm.provision "shell",
+ ## path: "provision.sh",
+ ## args: [
+ ## # "--debug",
+ ## "--config /vagrant/local.params.single_host_single_hostname",
+ ## "--test",
+ ## "--vagrant"
+ ## ].join(" ")
+ ## end
end
diff --git a/tools/salt-install/Vagrantfile.single_host_single_hostname.example b/tools/salt-install/Vagrantfile.single_host_single_hostname.example
new file mode 100644
index 000000000..666c6c48f
--- /dev/null
+++ b/tools/salt-install/Vagrantfile.single_host_single_hostname.example
@@ -0,0 +1,87 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+# Vagrantfile API/syntax version. Don"t touch unless you know what you"re doing!
+VAGRANTFILE_API_VERSION = "2".freeze
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+ config.ssh.insert_key = false
+ config.ssh.forward_x11 = true
+
+ # A single_host multiple_hostnames example
+ config.vm.define "arvados-sh-mn" do |arv|
+ arv.vm.box = "bento/debian-10"
+ arv.vm.hostname = "harpo.local"
+ # CPU/RAM
+ config.vm.provider :virtualbox do |v|
+ v.memory = 2048
+ v.cpus = 2
+ end
+
+ # Networking
+ # WEBUI PORT
+ arv.vm.network "forwarded_port", guest: 8443, host: 8443
+ # KEEPPROXY
+ arv.vm.network "forwarded_port", guest: 25101, host: 25101
+ # KEEPWEB
+ arv.vm.network "forwarded_port", guest: 9002, host: 9002
+ # WEBSOCKET
+ arv.vm.network "forwarded_port", guest: 8002, host: 8002
+ arv.vm.provision "shell",
+ inline: "sed 's#fixme#harpo#g;
+ s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=8443#g' \
+ /vagrant/local.params.example > /vagrant/local.params.single_host_multiple_hostnames"
+ arv.vm.provision "shell",
+ path: "provision.sh",
+ args: [
+ # "--debug",
+ "--config /vagrant/local.params.single_host_multiple_hostnames",
+ "--test",
+ "--vagrant"
+ ].join(" ")
+ end
+
+ ## # A single_host single_hostname example
+ ## config.vm.define "arvados-sh-sn" do |arv|
+ ## arv.vm.box = "bento/debian-10"
+ ## arv.vm.hostname = "zeppo.local"
+ ## # CPU/RAM
+ ## config.vm.provider :virtualbox do |v|
+ ## v.memory = 2048
+ ## v.cpus = 2
+ ## end
+
+ ## # Networking
+ ## arv.vm.network "forwarded_port", guest: 9443, host: 9443
+ ## arv.vm.network "forwarded_port", guest: 9444, host: 9444
+ ## arv.vm.network "forwarded_port", guest: 9445, host: 9445
+ ## arv.vm.network "forwarded_port", guest: 35101, host: 35101
+ ## arv.vm.network "forwarded_port", guest: 10002, host: 10002
+ ## arv.vm.network "forwarded_port", guest: 14202, host: 14202
+ ## arv.vm.network "forwarded_port", guest: 18002, host: 18002
+ ## arv.vm.provision "shell",
+ ## inline: "sed 's#HOSTNAME_EXT=\"\"#HOSTNAME_EXT=\"zeppo.local.cluster\"#g;
+ ## s#CLUSTER=\"fixme\"#CLUSTER=\"zeppo\"#g;
+ ## s#DOMAIN=\"some.domain\"#DOMAIN=\"local.cluster\"#g;
+ ## s#CONFIG_DIR=\"config_examples/single_host/multiple_hostnames\"#CONFIG_DIR=\"config_examples/single_host/single_hostname\"#g;
+ ## s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=9443#g;
+ ## s#KEEP_EXT_SSL_PORT=25101#KEEP_EXT_SSL_PORT=35101#g;
+ ## s#KEEPWEB_EXT_SSL_PORT=9002#KEEPWEB_EXT_SSL_PORT=11002#g;
+ ## s#WEBSHELL_EXT_SSL_PORT=4202#WEBSHELL_EXT_SSL_PORT=14202#g;
+ ## s#WEBSOCKET_EXT_SSL_PORT=8002#WEBSOCKET_EXT_SSL_PORT=18002#g;
+ ## s#WORKBENCH1_EXT_SSL_PORT=443#WORKBENCH1_EXT_SSL_PORT=9444#g;
+ ## s#WORKBENCH2_EXT_SSL_PORT=3001#WORKBENCH2_EXT_SSL_PORT=9445#g;' \
+ ## /vagrant/local.params.example > /vagrant/local.params.single_host_single_hostname"
+ ## arv.vm.provision "shell",
+ ## path: "provision.sh",
+ ## args: [
+ ## # "--debug",
+ ## "--config /vagrant/local.params.single_host_single_hostname",
+ ## "--test",
+ ## "--vagrant"
+ ## ].join(" ")
+ ## end
+end
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
index 6c6dec26f..710c4da86 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
@@ -118,7 +118,7 @@ arvados:
Services:
Controller:
- ExternalURL: 'https://__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
InternalURLs:
'http://controller.internal:8003': {}
DispatchCloud:
@@ -128,7 +128,7 @@ arvados:
InternalURLs:
'http://__CLUSTER__.__DOMAIN__:9005': {}
Keepproxy:
- ExternalURL: 'https://keep.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://keep.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
InternalURLs:
'http://keep.internal:25100': {}
Keepstore:
@@ -138,18 +138,18 @@ arvados:
InternalURLs:
'http://api.internal:8004': {}
WebDAV:
- ExternalURL: 'https://collections.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://collections.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
InternalURLs:
'http://collections.internal:9002': {}
WebDAVDownload:
- ExternalURL: 'https://download.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://download.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
WebShell:
- ExternalURL: 'https://webshell.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://webshell.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
Websocket:
ExternalURL: 'wss://ws.__CLUSTER__.__DOMAIN__/websocket'
InternalURLs:
'http://ws.internal:8005': {}
Workbench1:
- ExternalURL: 'https://workbench.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://workbench.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
Workbench2:
- ExternalURL: 'https://workbench2.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
+ ExternalURL: 'https://workbench2.__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls
index 00c3b3a13..3adf0580a 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls
@@ -40,7 +40,7 @@ nginx:
- server:
- server_name: __CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://controller_upstream'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls
index 6554f79a7..2d8922df9 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls
@@ -36,7 +36,7 @@ nginx:
- server:
- server_name: keep.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://keepproxy_upstream'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls
index cc871b9da..d180a3bad 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls
@@ -37,7 +37,7 @@ nginx:
- server:
- server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://collections_downloads_upstream'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls
index a0756b7ce..e75f04434 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls
@@ -37,7 +37,7 @@ nginx:
- server:
- server_name: webshell.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /shell.__CLUSTER__.__DOMAIN__:
- proxy_pass: 'http://webshell_upstream'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls
index ebe03f733..3a354ac29 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls
@@ -36,7 +36,7 @@ nginx:
- server:
- server_name: ws.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://websocket_upstream'
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls
index 8930be408..8fdd55399 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls
@@ -34,7 +34,7 @@ nginx:
- server:
- server_name: workbench2.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- root: /var/www/arvados-workbench2/workbench2
@@ -42,7 +42,7 @@ nginx:
- 'if (-f $document_root/maintenance.html)':
- return: 503
- location /config.json:
- - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__"}' ~ "'" }}
+ - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
- include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls
index be571ca77..649af10b6 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls
@@ -43,7 +43,7 @@ nginx:
- server:
- server_name: workbench.__CLUSTER__.__DOMAIN__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://workbench_upstream'
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
index f0e7a19a4..1b21aaaeb 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
@@ -22,11 +22,11 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__EXT__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __WEBSHELL_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- - location /__HOSTNAME__:
+ - location /__HOSTNAME_EXT__:
- proxy_pass: 'http://webshell_upstream'
- proxy_read_timeout: 90
- proxy_connect_timeout: 90
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
index f783e523f..462443c1f 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
@@ -28,7 +28,7 @@ nginx:
- 'if (-f $document_root/maintenance.html)':
- return: 503
- location /config.json:
- - return: {{ "200 '" ~ '{"API_HOST":"__HOSTNAME__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
+ - return: {{ "200 '" ~ '{"API_HOST":"__HOSTNAME_EXT__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
- include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example
index bd9b1c411..d02ba95c3 100644
--- a/tools/salt-install/local.params.example
+++ b/tools/salt-install/local.params.example
@@ -16,9 +16,14 @@ DOMAIN="some.domain"
# the EXTERNAL/PUBLIC hostname for the instance.
# If empty, the INTERNAL HOST IP will be used
HOSTNAME_EXT=""
-# The internal hostname for the host
+# The internal hostname for the host. In the example files, only used in the
+# single_host/single_hostname example
HOSTNAME_INT="127.0.1.1"
-CONTROLLER_EXT_SSL_PORT=8000
+# Host SSL port where you want to point your browser to access Arvados
+# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
+# You can point it to another port if desired
+# In Vagrant, make sure it matches what you set in the Vagrantfile (8443)
+CONTROLLER_EXT_SSL_PORT=443
KEEP_EXT_SSL_PORT=25101
# Both for collections and downloads
KEEPWEB_EXT_SSL_PORT=9002
@@ -42,9 +47,16 @@ RAILS_SECRET_TOKEN=railssecrettokenmushaveatleast32characters
ANONYMOUS_USER_TOKEN=anonymoususertokenmushaveatleast32characters
WORKBENCH_SECRET_KEY=workbenchsecretkeymushaveatleast32characters
-# The example config files you want to use. There are a few examples
-# under 'config_examples'
-CONFIG_DIR="config_examples/single_host/single_hostname"
+# The directory to check for the config files (pillars, states) you want to use.
+# There are a few examples under 'config_examples'. If you don't change this
+# variable, the single_host, multiple_hostnames config will be used
+# CONFIG_DIR="config_examples/single_host/single_hostname"
+CONFIG_DIR="config_examples/single_host/multiple_hostnames"
+# Extra states to pply. iIf you use your own subdir, change this value accordingly
+EXTRA_STATES_DIR="${F_DIR}/arvados-formula/test/salt/states/examples/single_host"
+
+# When using the single_host/single_hostname example, change to this one
+# EXTRA_STATES_DIR="${CONFIG_DIR}/states"
# Which release of Arvados repo you want to use
RELEASE="production"
@@ -52,12 +64,6 @@ RELEASE="production"
# in the desired repo
VERSION="latest"
-# Host SSL port where you want to point your browser to access Arvados
-# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
-# You can point it to another port if desired
-# In Vagrant, make sure it matches what you set in the Vagrantfile
-HOST_SSL_PORT=443
-
# This is an arvados-formula setting.
# If branch is set, the script will switch to it before running salt
# Usually not needed, only used for testing
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index a7605e1d9..486f43057 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/bin/bash -x
# Copyright (C) The Arvados Authors. All rights reserved.
#
@@ -55,7 +55,7 @@ arguments() {
while [ ${#} -ge 1 ]; do
case ${1} in
-c | --config)
- CONFIG=${2}
+ CONFIG_FILE=${2}
shift 2
;;
-d | --debug)
@@ -63,7 +63,7 @@ arguments() {
shift
;;
-p | --ssl-port)
- HOST_SSL_PORT=${2}
+ CONTROLLER_EXT_SSL_PORT=${2}
shift 2
;;
-r | --roles)
@@ -102,7 +102,7 @@ arguments() {
CONFIG="${SCRIPT_DIR}/local.params"
CONFIG_DIR="config_examples/single_host/multiple_hostnames"
LOG_LEVEL="info"
-HOST_SSL_PORT=443
+CONTROLLER_EXT_SSL_PORT=443
TESTS_DIR="tests"
CLUSTER=""
@@ -130,12 +130,20 @@ NGINX_TAG="v2.4.0"
DOCKER_TAG="v1.0.0"
LOCALE_TAG="v0.3.4"
+# Salt's dir
+## states
+S_DIR="/srv/salt"
+## formulas
+F_DIR="/srv/formulas"
+##pillars
+P_DIR="/srv/pillars"
+
arguments ${@}
-if [ -s ${CONFIG} ]; then
- source ${CONFIG}
+if [ -s ${CONFIG_FILE} ]; then
+ source ${CONFIG_FILE}
else
- echo >&2 "Please create a '${CONFIG}' file with initial values, as described in FIXME_URL_TO_DESCR"
+ echo >&2 "Please create a '${CONFIG_FILE}' file with initial values, as described in FIXME_URL_TO_DESCR"
exit 1
fi
@@ -145,14 +153,6 @@ if ! grep -E '^[[:alnum:]]{5}$' <<<${CLUSTER} ; then
exit 1
fi
-# Salt's dir
-## states
-S_DIR="/srv/salt"
-## formulas
-F_DIR="/srv/formulas"
-##pillars
-P_DIR="/srv/pillars"
-
apt-get update
apt-get install -y curl git jq
@@ -161,7 +161,7 @@ if [ ${?} -eq 0 ]; then
echo "Salt already installed"
else
curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
- sh /tmp/bootstrap_salt.sh -XUdfP -x python3
+ sh /tmp/bootstrap_salt.sh -XdfP -x python3
/bin/systemctl disable salt-minion.service
fi
@@ -172,7 +172,6 @@ file_roots:
base:
- ${S_DIR}
- ${F_DIR}/*
- - ${F_DIR}/*/test/salt/states/examples
pillar_roots:
base:
@@ -181,64 +180,6 @@ EOFSM
mkdir -p ${S_DIR} ${F_DIR} ${P_DIR}
-# States
-cat > ${S_DIR}/top.sls << EOFTSLS
-base:
- '*':
- # - single_host.host_entries
- # - single_host.snakeoil_certs
- - locale
-EOFTSLS
-
-# If we want specific roles for a node, just add the desired states
-# and its dependencies
-if [ -z "${ROLES}" ]; then
- echo ' - nginx.passenger' >> ${S_DIR}/top.sls
- echo ' - postgres' >> ${S_DIR}/top.sls
- echo ' - docker' >> ${S_DIR}/top.sls
- echo ' - arvados' >> ${S_DIR}/top.sls
-else
- # If we add individual roles, make sure we add the repo first
- echo " - arvados.repo" >> ${S_DIR}/top.sls
- for R in ${ROLES}; do
- case "${R}" in
- "database")
- echo " - postgres" >> ${S_DIR}/top.sls
- ::
- "api","workbench","workbench2","keepweb","keepproxy")
- grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
- ;;
- "shell","dispatcher")
- grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
- ;;
- *)
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
- ::
- esac
- done
-fi
-
-# Pillars
-cat > ${P_DIR}/top.sls << EOFPSLS
-base:
- '*':
- - arvados
- - docker
- - locale
- - nginx_api_configuration
- - nginx_controller_configuration
- - nginx_keepproxy_configuration
- - nginx_keepweb_configuration
- - nginx_passenger
- - nginx_websocket_configuration
- - nginx_webshell_configuration
- - nginx_workbench2_configuration
- - nginx_workbench_configuration
- - postgresql
-EOFPSLS
-
# Get the formula and dependencies
cd ${F_DIR} || exit 1
git clone --branch "${ARVADOS_TAG}" https://github.com/arvados/arvados-formula.git
@@ -255,39 +196,39 @@ fi
if [ "x${VAGRANT}" = "xyes" ]; then
SOURCE_PILLARS_DIR="/vagrant/${CONFIG_DIR}/pillars"
- SOURCE_STATES_DIR="/vagrant/${CONFIG_DIR}/states"
TESTS_DIR="/vagrant/${TESTS_DIR}"
else
SOURCE_PILLARS_DIR="${SCRIPT_DIR}/${CONFIG_DIR}/pillars"
- SOURCE_STATES_DIR="${SCRIPT_DIR}/${CONFIG_DIR}/states"
TESTS_DIR="${SCRIPT_DIR}/${TESTS_DIR}"
fi
-# Replace cluster and domain name in the example pillars
+SOURCE_STATES_DIR="${EXTRA_STATES_DIR}"
+
+# Replace variables (cluster, domain, etc) in the pillars, states and tests
+# to ease deployment for newcomers
for f in "${SOURCE_PILLARS_DIR}"/*; do
- sed "s/__CLUSTER__/${CLUSTER}/g;
- s/__DOMAIN__/${DOMAIN}/g;
- s/__RELEASE__/${RELEASE}/g;
+ sed "s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
+ s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
- s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
- s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
- s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
- s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
- s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__CLUSTER__/${CLUSTER}/g;
+ s/__DOMAIN__/${DOMAIN}/g;
s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
- s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
- s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
- s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
- s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
+ s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
- s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
- s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
- s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
- s/__VERSION__/${VERSION}/g" \
+ s/__RELEASE__/${RELEASE}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
+ s/__VERSION__/${VERSION}/g;
+ s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
+ s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
+ s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
+ s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g" \
"${f}" > "${P_DIR}"/$(basename "${f}")
done
@@ -295,48 +236,111 @@ mkdir -p /tmp/cluster_tests
# Replace cluster and domain name in the test files
for f in "${TESTS_DIR}"/*; do
sed "s/__CLUSTER__/${CLUSTER}/g;
+ s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
s/__DOMAIN__/${DOMAIN}/g;
s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
- s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
- s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
- s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
- s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
- s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g" \
+ s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g" \
"${f}" > "/tmp/cluster_tests"/$(basename "${f}")
done
chmod 755 /tmp/cluster_tests/run-test.sh
# Replace helper state files that differ from the formula's examples
-if -d "${SOURCE_STATES_DIR}"; then
+if [ -d "${SOURCE_STATES_DIR}" ]; then
+ mkdir -p "${F_DIR}"/extra/extra
+
for f in "${SOURCE_STATES_DIR}"/*; do
- sed "s/__CLUSTER__/${CLUSTER}/g;
- s/__DOMAIN__/${DOMAIN}/g;
- s/__RELEASE__/${RELEASE}/g;
+ sed "s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
+ s/__CLUSTER__/${CLUSTER}/g;
+ s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
- s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
- s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
- s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
- s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
- s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__DOMAIN__/${DOMAIN}/g;
s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
- s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
- s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
- s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
- s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
+ s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
- s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
- s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
- s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
- s/__VERSION__/${VERSION}/g" \
- "${f}" > "${F_DIR}/arvados-formula/test/salt/states/examples/single_host"/$(basename "${f}")
+ s/__RELEASE__/${RELEASE}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
+ s/__VERSION__/${VERSION}/g;
+ s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
+ s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
+ s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
+ s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g" \
+ "${f}" > "${F_DIR}/extra/extra"/$(basename "${f}")
+ done
+fi
+
+# Now, we build the SALT states/pillars trees
+# States
+cat > ${S_DIR}/top.sls << EOFTSLS
+base:
+ '*':
+ - locale
+EOFTSLS
+
+if [ -d "${SOURCE_STATES_DIR}" ]; then
+ for f in "${F_DIR}"/extra/extra/*.sls; do
+ echo " - extra.$(basename ${f} | sed 's/.sls$//g')" >> ${S_DIR}/top.sls
+ done
+fi
+
+# If we want specific roles for a node, just add the desired states
+# and its dependencies
+if [ -z "${ROLES}" ]; then
+ echo ' - nginx.passenger' >> ${S_DIR}/top.sls
+ echo ' - postgres' >> ${S_DIR}/top.sls
+ echo ' - docker' >> ${S_DIR}/top.sls
+ echo ' - arvados' >> ${S_DIR}/top.sls
+else
+ # If we add individual roles, make sure we add the repo first
+ echo " - arvados.repo" >> ${S_DIR}/top.sls
+ for R in ${ROLES}; do
+ case "${R}" in
+ "database")
+ echo " - postgres" >> ${S_DIR}/top.sls
+ ;;
+ "api","workbench","workbench2","keepweb","keepproxy")
+ grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ "shell","dispatcher")
+ grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ *)
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ esac
done
fi
+# Pillars
+cat > ${P_DIR}/top.sls << EOFPSLS
+base:
+ '*':
+ - arvados
+ - docker
+ - locale
+ - nginx_api_configuration
+ - nginx_controller_configuration
+ - nginx_keepproxy_configuration
+ - nginx_keepweb_configuration
+ - nginx_passenger
+ - nginx_websocket_configuration
+ - nginx_webshell_configuration
+ - nginx_workbench2_configuration
+ - nginx_workbench_configuration
+ - postgresql
+EOFPSLS
+
# FIXME! #16992 Temporary fix for psql call in arvados-api-server
if [ -e /root/.psqlrc ]; then
if ! ( grep 'pset pager off' /root/.psqlrc ); then
@@ -369,12 +373,12 @@ fi
echo "Copying the Arvados CA certificate to the installer dir, so you can import it"
# If running in a vagrant VM, also add default user to docker group
if [ "x${VAGRANT}" = "xyes" ]; then
- cp /etc/ssl/certs/arvados-snakeoil-ca.pem /vagrant
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem /vagrant/${CLUSTER}.${DOMAIN}-arvados-snakeoil-ca.pem
echo "Adding the vagrant user to the docker group"
usermod -a -G docker vagrant
else
- cp /etc/ssl/certs/arvados-snakeoil-ca.pem ${SCRIPT_DIR}
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem ${SCRIPT_DIR}/${CLUSTER}.${DOMAIN}-arvados-snakeoil-ca.pem
fi
# Test that the installation finished correctly
diff --git a/tools/salt-install/tests/run-test.sh b/tools/salt-install/tests/run-test.sh
index 16ee2851e..6bc8422f8 100755
--- a/tools/salt-install/tests/run-test.sh
+++ b/tools/salt-install/tests/run-test.sh
@@ -4,7 +4,7 @@
# SPDX-License-Identifier: Apache-2.0
export ARVADOS_API_TOKEN=__SYSTEM_ROOT_TOKEN__
-export ARVADOS_API_HOST=__HOSTNAME_INT__:__CONTROLLER_EXT_SSL_PORT__
+export ARVADOS_API_HOST=__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__
export ARVADOS_API_HOST_INSECURE=true
set -o pipefail
commit ffd332a1849c7fef8aecfb27c442d67a34e29294
Author: Javier Bértoli <jbertoli at curii.com>
Date: Wed Feb 3 16:57:53 2021 -0300
feat(provision): refactor to manage different infrastructure configurations
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index ce1588c19..a7605e1d9 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -70,7 +70,7 @@ arguments() {
for i in ${2//,/ }
do
# Verify the role exists
- if [[ ! "api,controller,keepstore,websocket,keepweb,workbench2,keepproxy,shell,workbench,dispatcher" == *"$i"* ]]; then
+ if [[ ! "database,api,controller,keepstore,websocket,keepweb,workbench2,keepproxy,shell,workbench,dispatcher" == *"$i"* ]]; then
echo "The role '${i}' is not a valid role"
usage
exit 1
@@ -185,20 +185,38 @@ mkdir -p ${S_DIR} ${F_DIR} ${P_DIR}
cat > ${S_DIR}/top.sls << EOFTSLS
base:
'*':
- - single_host.host_entries
- - single_host.snakeoil_certs
+ # - single_host.host_entries
+ # - single_host.snakeoil_certs
- locale
- - nginx.passenger
- - postgres
- - docker
EOFTSLS
-# If we want specific roles for a node, just add those states
+# If we want specific roles for a node, just add the desired states
+# and its dependencies
if [ -z "${ROLES}" ]; then
+ echo ' - nginx.passenger' >> ${S_DIR}/top.sls
+ echo ' - postgres' >> ${S_DIR}/top.sls
+ echo ' - docker' >> ${S_DIR}/top.sls
echo ' - arvados' >> ${S_DIR}/top.sls
else
+ # If we add individual roles, make sure we add the repo first
+ echo " - arvados.repo" >> ${S_DIR}/top.sls
for R in ${ROLES}; do
- echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ case "${R}" in
+ "database")
+ echo " - postgres" >> ${S_DIR}/top.sls
+ ::
+ "api","workbench","workbench2","keepweb","keepproxy")
+ grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ "shell","dispatcher")
+ grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ;;
+ *)
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ ::
+ esac
done
fi
@@ -285,37 +303,39 @@ for f in "${TESTS_DIR}"/*; do
s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g" \
- ${f} > /tmp/cluster_tests/$(basename ${f})
+ "${f}" > "/tmp/cluster_tests"/$(basename "${f}")
done
chmod 755 /tmp/cluster_tests/run-test.sh
# Replace helper state files that differ from the formula's examples
-for f in "${SOURCE_STATES_DIR}"/*; do
- sed "s/__CLUSTER__/${CLUSTER}/g;
- s/__DOMAIN__/${DOMAIN}/g;
- s/__RELEASE__/${RELEASE}/g;
- s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
- s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
- s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
- s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
- s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
- s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
- s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
- s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
- s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
- s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
- s/__INITIAL_USER__/${INITIAL_USER}/g;
- s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
- s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
- s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
- s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
- s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
- s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
- s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
- s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
- s/__VERSION__/${VERSION}/g" \
- "${f}" > "${F_DIR}"/arvados-formula/test/salt/states/examples/single_host/$(basename "${f}")
-done
+if -d "${SOURCE_STATES_DIR}"; then
+ for f in "${SOURCE_STATES_DIR}"/*; do
+ sed "s/__CLUSTER__/${CLUSTER}/g;
+ s/__DOMAIN__/${DOMAIN}/g;
+ s/__RELEASE__/${RELEASE}/g;
+ s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
+ s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
+ s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
+ s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
+ s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
+ s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
+ s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
+ s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
+ s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
+ s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
+ s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
+ s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
+ s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
+ s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
+ s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
+ s/__VERSION__/${VERSION}/g" \
+ "${f}" > "${F_DIR}/arvados-formula/test/salt/states/examples/single_host"/$(basename "${f}")
+ done
+fi
# FIXME! #16992 Temporary fix for psql call in arvados-api-server
if [ -e /root/.psqlrc ]; then
commit c590159bf245425b96187358b53498999fa64051
Author: Javier Bértoli <jbertoli at curii.com>
Date: Wed Jan 27 13:07:14 2021 -0300
feat(provision): check the cluster name provided is exactly 5 chars long
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index facb2e88e..ce1588c19 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -139,6 +139,12 @@ else
exit 1
fi
+if ! grep -E '^[[:alnum:]]{5}$' <<<${CLUSTER} ; then
+ echo >&2 "ERROR: <CLUSTER> must be exactly 5 alphanumeric characters long"
+ echo >&2 "Fix the cluster name in the 'local.params' file and re-run the provision script"
+ exit 1
+fi
+
# Salt's dir
## states
S_DIR="/srv/salt"
commit 6c058acc4ea63b29ef049715cc5bb104ca4e7bd7
Author: Javier Bértoli <jbertoli at curii.com>
Date: Wed Jan 27 09:54:49 2021 -0300
feat(provision): refactor to add other setup examples
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
similarity index 90%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
index 4aa4735d8..6c6dec26f 100644
--- a/tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls
@@ -78,19 +78,15 @@ arvados:
### TOKENS
tokens:
- system_root: changemesystemroottoken
- management: changememanagementtoken
- rails_secret: changemerailssecrettoken
- anonymous_user: changemeanonymoususertoken
+ system_root: __SYSTEM_ROOT_TOKEN__
+ management: __MANAGEMENT_TOKEN__
+ rails_secret: __RAILS_SECRET_TOKEN__
+ anonymous_user: __ANONYMOUS_USER_TOKEN__
### KEYS
secrets:
- blob_signing_key: changemeblobsigningkey
- workbench_secret_key: changemeworkbenchsecretkey
- dispatcher_access_key: changemedispatcheraccesskey
- dispatcher_secret_key: changeme_dispatchersecretkey
- keep_access_key: changemekeepaccesskey
- keep_secret_key: changemekeepsecretkey
+ blob_signing_key: __BLOB_SIGNING_KEY__
+ workbench_secret_key: __WORKBENCH_SECRET_KEY__
Login:
Test:
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/docker.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/docker.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/docker.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/docker.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/locale.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/locale.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/locale.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/locale.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_api_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_api_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_api_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_api_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_controller_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_controller_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepproxy_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepproxy_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepweb_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_keepweb_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_passenger.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_passenger.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_passenger.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_passenger.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_webshell_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_webshell_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_websocket_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_websocket_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench2_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench2_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench_configuration.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/nginx_workbench_configuration.sls
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/postgresql.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/postgresql.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/multiple_hostnames/postgresql.sls
rename to tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/postgresql.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/arvados.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
similarity index 68%
rename from tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
index e5e458665..f3d2bcb9e 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/arvados.sls
@@ -78,19 +78,15 @@ arvados:
### TOKENS
tokens:
- system_root: changemesystemroottoken
- management: changememanagementtoken
- rails_secret: changemerailssecrettoken
- anonymous_user: changemeanonymoususertoken
+ system_root: __SYSTEM_ROOT_TOKEN__
+ management: __MANAGEMENT_TOKEN__
+ rails_secret: __RAILS_SECRET_TOKEN__
+ anonymous_user: __ANONYMOUS_USER_TOKEN__
### KEYS
secrets:
- blob_signing_key: changemeblobsigningkey
- workbench_secret_key: changemeworkbenchsecretkey
- dispatcher_access_key: changemedispatcheraccesskey
- dispatcher_secret_key: changeme_dispatchersecretkey
- keep_access_key: changemekeepaccesskey
- keep_secret_key: changemekeepsecretkey
+ blob_signing_key: __BLOB_SIGNING_KEY__
+ workbench_secret_key: __WORKBENCH_SECRET_KEY__
Login:
Test:
@@ -107,7 +103,7 @@ arvados:
# <cluster>-nyw5e-<volume>
__CLUSTER__-nyw5e-000000000000000:
AccessViaHosts:
- 'http://__HOSTNAME__:25107':
+ 'http://__HOSTNAME_INT__:25107':
ReadOnly: false
Replication: 2
Driver: Directory
@@ -122,38 +118,32 @@ arvados:
Services:
Controller:
- ExternalURL: 'https://__HOSTNAME__:__CONTROLLER_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__CONTROLLER_EXT_SSL_PORT__'
InternalURLs:
- 'http://controller.internal:8003': {}
- DispatchCloud:
- InternalURLs:
- 'http://__HOSTNAME__:9006': {}
- Keepbalance:
- InternalURLs:
- 'http://__HOSTNAME__:9005': {}
+ 'http://__HOSTNAME_INT__:8003': {}
Keepproxy:
- ExternalURL: 'https://__HOSTNAME__:__KEEP_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__KEEP_EXT_SSL_PORT__'
InternalURLs:
- 'http://keep.internal:25100': {}
+ 'http://__HOSTNAME_INT__:25100': {}
Keepstore:
InternalURLs:
- 'http://keep0.internal:25107': {}
+ 'http://__HOSTNAME_INT__:25107': {}
RailsAPI:
InternalURLs:
- 'http://api.internal:8004': {}
+ 'http://__HOSTNAME_INT__:8004': {}
WebDAV:
- ExternalURL: 'https://__HOSTNAME__:__KEEPWEB_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__KEEPWEB_EXT_SSL_PORT__'
InternalURLs:
- 'http://collections.internal:9002': {}
+ 'http://__HOSTNAME_INT__:9003': {}
WebDAVDownload:
- ExternalURL: 'https://__HOSTNAME__:__KEEPWEB_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__KEEPWEB_EXT_SSL_PORT__'
WebShell:
- ExternalURL: 'https://__HOSTNAME__:__WEBSHELL_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__WEBSHELL_EXT_SSL_PORT__'
Websocket:
- ExternalURL: 'wss://__HOSTNAME__:__WEBSOCKET_EXT_SSL_PORT__/websocket'
+ ExternalURL: 'wss://__HOSTNAME_EXT__:__WEBSOCKET_EXT_SSL_PORT__/websocket'
InternalURLs:
- 'http://ws.internal:8005': {}
+ 'http://__HOSTNAME_INT__:8005': {}
Workbench1:
- ExternalURL: 'https://__HOSTNAME__:__WORKBENCH1_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__WORKBENCH1_EXT_SSL_PORT__'
Workbench2:
- ExternalURL: 'https://__HOSTNAME__:__WORKBENCH2_EXT_SSL_PORT__'
+ ExternalURL: 'https://__HOSTNAME_EXT__:__WORKBENCH2_EXT_SSL_PORT__'
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/docker.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/docker.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/single_hostname/docker.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/docker.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/locale.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/locale.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/single_hostname/locale.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/locale.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_api_configuration.sls
similarity index 93%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_api_configuration.sls
index b2f12c773..18f09af50 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_api_configuration.sls
@@ -18,7 +18,7 @@ nginx:
overwrite: true
config:
- server:
- - listen: 'api.internal:8004'
+ - listen: '__HOSTNAME_INT__:8004'
- server_name: api
- root: /var/www/arvados-api/current/public
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_controller_configuration.sls
similarity index 87%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_controller_configuration.sls
index 2eb33b835..b7b75ab9c 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_controller_configuration.sls
@@ -14,7 +14,7 @@ nginx:
default: 1
'127.0.0.0/8': 0
upstream controller_upstream:
- - server: 'controller.internal:8003 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:8003 fail_timeout=10s'
### SITES
servers:
@@ -25,9 +25,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: _
- listen:
- - 80 default
+ - 80 default_server
- location /.well-known:
- root: /var/www
- location /:
@@ -38,9 +38,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl default_server
- index: index.html index.htm
- location /:
- proxy_pass: 'http://controller_upstream'
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepproxy_configuration.sls
similarity index 73%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepproxy_configuration.sls
index b26de2710..81d72aac7 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepproxy_configuration.sls
@@ -11,30 +11,16 @@ nginx:
### STREAMS
http:
upstream keepproxy_upstream:
- - server: 'keep.internal:25100 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:25100 fail_timeout=10s'
servers:
managed:
- ### DEFAULT
- arvados_keepproxy_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __KEEP_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
arvados_keepproxy_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __KEEP_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepweb_configuration.sls
similarity index 72%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepweb_configuration.sls
index 98a3cdf94..fcb56c994 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_keepweb_configuration.sls
@@ -11,31 +11,17 @@ nginx:
### STREAMS
http:
upstream collections_downloads_upstream:
- - server: 'collections.internal:9002 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:9003 fail_timeout=10s'
servers:
managed:
- ### COLLECTIONS / DOWNLOAD
- arvados_collections_download_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __KEEPWEB_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
### COLLECTIONS / DOWNLOAD
arvados_collections_download_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __KEEPWEB_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_passenger.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_passenger.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_passenger.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_passenger.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
similarity index 84%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
index dac606123..f0e7a19a4 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_webshell_configuration.sls
@@ -12,30 +12,17 @@ nginx:
### STREAMS
http:
upstream webshell_upstream:
- - server: 'shell.internal:4200 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:4200 fail_timeout=10s'
### SITES
servers:
managed:
- arvados_webshell_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __WEBSHELL_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
arvados_webshell_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: __HOSTNAME__EXT__
- listen:
- __WEBSHELL_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_websocket_configuration.sls
similarity index 74%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_websocket_configuration.sls
index 827524cbe..7c4ff7835 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_websocket_configuration.sls
@@ -11,30 +11,16 @@ nginx:
### STREAMS
http:
upstream websocket_upstream:
- - server: 'ws.internal:8005 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:8005 fail_timeout=10s'
servers:
managed:
- ### DEFAULT
- arvados_websocket_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __WEBSOCKET_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
arvados_websocket_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: __HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __WEBSOCKET_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
similarity index 70%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
index 7f90cbc82..f783e523f 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench2_configuration.sls
@@ -13,26 +13,12 @@ nginx:
### SITES
servers:
managed:
- ### DEFAULT
- arvados_workbench2_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __WORKBENCH2_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
arvados_workbench2_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: workbench2.__HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __WORKBENCH2_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench_configuration.sls
similarity index 76%
rename from tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench_configuration.sls
index 0cbd3e14a..9ed6e3b87 100644
--- a/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/pillars/nginx_workbench_configuration.sls
@@ -17,31 +17,17 @@ nginx:
### STREAMS
http:
upstream workbench_upstream:
- - server: 'workbench.internal:9000 fail_timeout=10s'
+ - server: '__HOSTNAME_INT__:9000 fail_timeout=10s'
### SITES
servers:
managed:
- ### DEFAULT
- arvados_workbench_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: __HOSTNAME__
- - listen:
- - __WORKBENCH_EXT_SSL_PORT__
- - location /.well-known:
- - root: /var/www
- - location /:
- - return: '301 https://$host$request_uri'
-
arvados_workbench_ssl:
enabled: true
overwrite: true
config:
- server:
- - server_name: workbench.__HOSTNAME__
+ - server_name: __HOSTNAME_EXT__
- listen:
- __WORKBENCH1_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
@@ -63,7 +49,7 @@ nginx:
overwrite: true
config:
- server:
- - listen: 'workbench.internal:9000'
+ - listen: '__HOSTNAME_INT__:9000'
- server_name: workbench
- root: /var/www/arvados-workbench/current/public
- index: index.html index.htm
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/postgresql.sls b/tools/salt-install/config_examples/single_host/single_hostname/pillars/postgresql.sls
similarity index 100%
rename from tools/salt-install/config_examples/single_host/single_hostname/postgresql.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/pillars/postgresql.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/states/host_entries.sls b/tools/salt-install/config_examples/single_host/single_hostname/states/host_entries.sls
new file mode 100644
index 000000000..7e3957c57
--- /dev/null
+++ b/tools/salt-install/config_examples/single_host/single_hostname/states/host_entries.sls
@@ -0,0 +1,32 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+{%- set curr_tpldir = tpldir %}
+{%- set tpldir = 'arvados' %}
+{%- from "arvados/map.jinja" import arvados with context %}
+{%- set tpldir = curr_tpldir %}
+
+arvados_test_salt_states_examples_single_host_etc_hosts_host_present:
+ host.present:
+ - ip: 127.0.0.2
+ - names:
+ - {{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ # FIXME! This just works for our testings.
+ # Won't work if the cluster name != host name
+ {%- for entry in [
+ 'api',
+ 'collections',
+ 'controller',
+ 'download',
+ 'keep',
+ 'keepweb',
+ 'keep0',
+ 'shell',
+ 'workbench',
+ 'workbench2',
+ 'ws',
+ ]
+ %}
+ - {{ entry }}
+ {%- endfor %}
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/states/snakeoil_certs.sls b/tools/salt-install/config_examples/single_host/single_hostname/states/snakeoil_certs.sls
new file mode 100644
index 000000000..375cc84eb
--- /dev/null
+++ b/tools/salt-install/config_examples/single_host/single_hostname/states/snakeoil_certs.sls
@@ -0,0 +1,156 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+{%- set curr_tpldir = tpldir %}
+{%- set tpldir = 'arvados' %}
+{%- from "arvados/map.jinja" import arvados with context %}
+{%- set tpldir = curr_tpldir %}
+
+include:
+ - nginx.service
+
+{%- set arvados_ca_cert_file = '/etc/ssl/certs/arvados-snakeoil-ca.pem' %}
+{%- set arvados_ca_key_file = '/etc/ssl/private/arvados-snakeoil-ca.key' %}
+{%- set arvados_cert_file = '/etc/ssl/certs/arvados-snakeoil-cert.pem' %}
+{%- set arvados_csr_file = '/etc/ssl/private/arvados-snakeoil-cert.csr' %}
+{%- set arvados_key_file = '/etc/ssl/private/arvados-snakeoil-cert.key' %}
+
+{%- if grains.get('os_family') == 'Debian' %}
+ {%- set arvados_ca_cert_dest = '/usr/local/share/ca-certificates/arvados-snakeoil-ca.crt' %}
+ {%- set update_ca_cert = '/usr/sbin/update-ca-certificates' %}
+ {%- set openssl_conf = '/etc/ssl/openssl.cnf' %}
+{%- else %}
+ {%- set arvados_ca_cert_dest = '/etc/pki/ca-trust/source/anchors/arvados-snakeoil-ca.pem' %}
+ {%- set update_ca_cert = '/usr/bin/update-ca-trust' %}
+ {%- set openssl_conf = '/etc/pki/tls/openssl.cnf' %}
+{%- endif %}
+
+arvados_test_salt_states_examples_single_host_snakeoil_certs_dependencies_pkg_installed:
+ pkg.installed:
+ - pkgs:
+ - openssl
+ - ca-certificates
+
+arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_ca_cmd_run:
+ # Taken from https://github.com/arvados/arvados/blob/master/tools/arvbox/lib/arvbox/docker/service/certificate/run
+ cmd.run:
+ - name: |
+ # These dirs are not to CentOS-ish, but this is a helper script
+ # and they should be enough
+ mkdir -p /etc/ssl/certs/ /etc/ssl/private/ && \
+ openssl req \
+ -new \
+ -nodes \
+ -sha256 \
+ -x509 \
+ -subj "/C=CC/ST=Some State/O=Arvados Formula/OU=arvados-formula/CN=snakeoil-ca-{{ arvados.cluster.name }}.{{ arvados.cluster.domain }}" \
+ -extensions x509_ext \
+ -config <(cat {{ openssl_conf }} \
+ <(printf "\n[x509_ext]\nbasicConstraints=critical,CA:true,pathlen:0\nkeyUsage=critical,keyCertSign,cRLSign")) \
+ -out {{ arvados_ca_cert_file }} \
+ -keyout {{ arvados_ca_key_file }} \
+ -days 365 && \
+ cp {{ arvados_ca_cert_file }} {{ arvados_ca_cert_dest }} && \
+ {{ update_ca_cert }}
+ - unless:
+ - test -f {{ arvados_ca_cert_file }}
+ - openssl verify -CAfile {{ arvados_ca_cert_file }} {{ arvados_ca_cert_file }}
+ - require:
+ - pkg: arvados_test_salt_states_examples_single_host_snakeoil_certs_dependencies_pkg_installed
+
+arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_cert_cmd_run:
+ cmd.run:
+ - name: |
+ cat > /tmp/openssl.cnf <<-CNF
+ [req]
+ default_bits = 2048
+ prompt = no
+ default_md = sha256
+ req_extensions = rext
+ distinguished_name = dn
+ [dn]
+ C = CC
+ ST = Some State
+ L = Some Location
+ O = Arvados Formula
+ OU = arvados-formula
+ CN = {{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ emailAddress = admin@{{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ [rext]
+ subjectAltName = @alt_names
+ [alt_names]
+ {%- for entry in grains.get('ipv4') %}
+ IP.{{ loop.index }} = {{ entry }}
+ {%- endfor %}
+ {%- for entry in [
+ 'keep',
+ 'collections',
+ 'download',
+ 'keepweb',
+ 'ws',
+ 'workbench',
+ 'workbench2',
+ ]
+ %}
+ DNS.{{ loop.index }} = {{ entry }}
+ {%- endfor %}
+ DNS.8 = {{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ DNS.9 = '__HOSTNAME_EXT__'
+ DNS.10 = '__HOSTNAME_INT__'
+ CNF
+
+ # The req
+ openssl req \
+ -config /tmp/openssl.cnf \
+ -new \
+ -nodes \
+ -sha256 \
+ -out {{ arvados_csr_file }} \
+ -keyout {{ arvados_key_file }} > /tmp/snake_oil_certs.output 2>&1 && \
+ # The cert
+ openssl x509 \
+ -req \
+ -days 365 \
+ -in {{ arvados_csr_file }} \
+ -out {{ arvados_cert_file }} \
+ -extfile /tmp/openssl.cnf \
+ -extensions rext \
+ -CA {{ arvados_ca_cert_file }} \
+ -CAkey {{ arvados_ca_key_file }} \
+ -set_serial $(date +%s) && \
+ chmod 0644 {{ arvados_cert_file }} && \
+ chmod 0640 {{ arvados_key_file }}
+ - unless:
+ - test -f {{ arvados_key_file }}
+ - openssl verify -CAfile {{ arvados_ca_cert_file }} {{ arvados_cert_file }}
+ - require:
+ - pkg: arvados_test_salt_states_examples_single_host_snakeoil_certs_dependencies_pkg_installed
+ - cmd: arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_ca_cmd_run
+
+{%- if grains.get('os_family') == 'Debian' %}
+arvados_test_salt_states_examples_single_host_snakeoil_certs_ssl_cert_pkg_installed:
+ pkg.installed:
+ - name: ssl-cert
+ - require_in:
+ - sls: postgres
+
+arvados_test_salt_states_examples_single_host_snakeoil_certs_certs_permissions_cmd_run:
+ cmd.run:
+ - name: |
+ chown root:ssl-cert {{ arvados_key_file }}
+ - require:
+ - cmd: arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_cert_cmd_run
+ - pkg: arvados_test_salt_states_examples_single_host_snakeoil_certs_ssl_cert_pkg_installed
+{%- endif %}
+
+arvados_test_salt_states_examples_single_host_snakeoil_certs_nginx_snakeoil_file_managed:
+ file.managed:
+ - name: /etc/nginx/snippets/arvados-snakeoil.conf
+ - contents: |
+ ssl_certificate {{ arvados_cert_file }};
+ ssl_certificate_key {{ arvados_key_file }};
+ - watch_in:
+ - service: nginx_service
+
+
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example
index a88301b2a..bd9b1c411 100644
--- a/tools/salt-install/local.params.example
+++ b/tools/salt-install/local.params.example
@@ -13,9 +13,11 @@ DOMAIN="some.domain"
# When setting the cluster in a single host, you can use a single hostname
# to access all the instances. When using virtualization (ie AWS), this should be
-# the EXTERNAL hostname for the instance.
+# the EXTERNAL/PUBLIC hostname for the instance.
# If empty, the INTERNAL HOST IP will be used
-HOSTNAME=""
+HOSTNAME_EXT=""
+# The internal hostname for the host
+HOSTNAME_INT="127.0.1.1"
CONTROLLER_EXT_SSL_PORT=8000
KEEP_EXT_SSL_PORT=25101
# Both for collections and downloads
@@ -32,6 +34,14 @@ INITIAL_USER="admin"
INITIAL_USER_EMAIL="admin at fixme.localdomain"
INITIAL_USER_PASSWORD="password"
+# YOU SHOULD CHANGE THESE TO SOME RANDOM STRINGS
+BLOB_SIGNING_KEY=blobsigningkeymushaveatleast32characters
+MANAGEMENT_TOKEN=managementtokenmushaveatleast32characters
+SYSTEM_ROOT_TOKEN=systemroottokenmushaveatleast32characters
+RAILS_SECRET_TOKEN=railssecrettokenmushaveatleast32characters
+ANONYMOUS_USER_TOKEN=anonymoususertokenmushaveatleast32characters
+WORKBENCH_SECRET_KEY=workbenchsecretkeymushaveatleast32characters
+
# The example config files you want to use. There are a few examples
# under 'config_examples'
CONFIG_DIR="config_examples/single_host/single_hostname"
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index f3df4109a..facb2e88e 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -107,7 +107,8 @@ TESTS_DIR="tests"
CLUSTER=""
DOMAIN=""
-HOSTNAME=""
+HOSTNAME_EXT=""
+HOSTNAME_INT="127.0.1.1"
INITIAL_USER=""
INITIAL_USER_EMAIL=""
INITIAL_USER_PASSWORD=""
@@ -229,14 +230,16 @@ if [ "x${BRANCH}" != "x" ]; then
fi
if [ "x${VAGRANT}" = "xyes" ]; then
- SOURCE_PILLARS_DIR="/vagrant/${CONFIG_DIR}"
+ SOURCE_PILLARS_DIR="/vagrant/${CONFIG_DIR}/pillars"
+ SOURCE_STATES_DIR="/vagrant/${CONFIG_DIR}/states"
TESTS_DIR="/vagrant/${TESTS_DIR}"
else
- SOURCE_PILLARS_DIR="${SCRIPT_DIR}/${CONFIG_DIR}"
+ SOURCE_PILLARS_DIR="${SCRIPT_DIR}/${CONFIG_DIR}/pillars"
+ SOURCE_STATES_DIR="${SCRIPT_DIR}/${CONFIG_DIR}/states"
TESTS_DIR="${SCRIPT_DIR}/${TESTS_DIR}"
fi
-# Replace cluster and domain name in the example pillars and test files
+# Replace cluster and domain name in the example pillars
for f in "${SOURCE_PILLARS_DIR}"/*; do
sed "s/__CLUSTER__/${CLUSTER}/g;
s/__DOMAIN__/${DOMAIN}/g;
@@ -244,25 +247,35 @@ for f in "${SOURCE_PILLARS_DIR}"/*; do
s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
- s/__WORKBENCH1_EXT__SSL_PORT__/${WORKBENCH1_EXT__SSL_PORT}/g;
- s/__WORKBENCH2_EXT__SSL_PORT__/${WORKBENCH2_EXT__SSL_PORT}/g;
+ s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
+ s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
- s/__HOSTNAME__/${HOSTNAME}/g;
+ s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
+ s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
+ s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
+ s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
+ s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
+ s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
+ s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
s/__VERSION__/${VERSION}/g" \
"${f}" > "${P_DIR}"/$(basename "${f}")
done
mkdir -p /tmp/cluster_tests
-# Replace cluster and domain name in the example pillars and test files
+# Replace cluster and domain name in the test files
for f in "${TESTS_DIR}"/*; do
sed "s/__CLUSTER__/${CLUSTER}/g;
s/__DOMAIN__/${DOMAIN}/g;
+ s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
+ s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g" \
@@ -270,6 +283,34 @@ for f in "${TESTS_DIR}"/*; do
done
chmod 755 /tmp/cluster_tests/run-test.sh
+# Replace helper state files that differ from the formula's examples
+for f in "${SOURCE_STATES_DIR}"/*; do
+ sed "s/__CLUSTER__/${CLUSTER}/g;
+ s/__DOMAIN__/${DOMAIN}/g;
+ s/__RELEASE__/${RELEASE}/g;
+ s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
+ s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
+ s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
+ s/__WORKBENCH1_EXT_SSL_PORT__/${WORKBENCH1_EXT_SSL_PORT}/g;
+ s/__WORKBENCH2_EXT_SSL_PORT__/${WORKBENCH2_EXT_SSL_PORT}/g;
+ s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__HOSTNAME_EXT__/${HOSTNAME_EXT}/g;
+ s/__HOSTNAME_INT__/${HOSTNAME_INT}/g;
+ s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
+ s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
+ s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
+ s/__BLOB_SIGNING_KEY__/${BLOB_SIGNING_KEY}/g;
+ s/__MANAGEMENT_TOKEN__/${MANAGEMENT_TOKEN}/g;
+ s/__SYSTEM_ROOT_TOKEN__/${SYSTEM_ROOT_TOKEN}/g;
+ s/__RAILS_SECRET_TOKEN__/${RAILS_SECRET_TOKEN}/g;
+ s/__ANONYMOUS_USER_TOKEN__/${ANONYMOUS_USER_TOKEN}/g;
+ s/__WORKBENCH_SECRET_KEY__/${WORKBENCH_SECRET_KEY}/g;
+ s/__VERSION__/${VERSION}/g" \
+ "${f}" > "${F_DIR}"/arvados-formula/test/salt/states/examples/single_host/$(basename "${f}")
+done
+
# FIXME! #16992 Temporary fix for psql call in arvados-api-server
if [ -e /root/.psqlrc ]; then
if ! ( grep 'pset pager off' /root/.psqlrc ); then
diff --git a/tools/salt-install/tests/run-test.sh b/tools/salt-install/tests/run-test.sh
index 8d9de6fdf..16ee2851e 100755
--- a/tools/salt-install/tests/run-test.sh
+++ b/tools/salt-install/tests/run-test.sh
@@ -3,8 +3,8 @@
#
# SPDX-License-Identifier: Apache-2.0
-export ARVADOS_API_TOKEN=changemesystemroottoken
-export ARVADOS_API_HOST=__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+export ARVADOS_API_TOKEN=__SYSTEM_ROOT_TOKEN__
+export ARVADOS_API_HOST=__HOSTNAME_INT__:__CONTROLLER_EXT_SSL_PORT__
export ARVADOS_API_HOST_INSECURE=true
set -o pipefail
commit 33160dea02e7552732fe23cbcc3e061b1a5245bb
Author: Javier Bértoli <jbertoli at curii.com>
Date: Tue Jan 26 08:58:07 2021 -0300
feat(provision): refactor to add other setup examples
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/config_examples/single_host/multiple_hostnames/README.md b/tools/salt-install/config_examples/single_host/multiple_hostnames/README.md
new file mode 100644
index 000000000..17ca89a9f
--- /dev/null
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/README.md
@@ -0,0 +1,20 @@
+Single host with multiple hostnames
+===================================
+
+These files let you setup Arvados on a single host using different hostnames
+for each of its components nginx's virtualhosts.
+
+The hostnames are composed after the variables "CLUSTER" and "DOMAIN" set in
+the `local.params` file.
+
+The virtual hosts' hostnames that will be used are:
+
+* CLUSTER.DOMAIN
+* collections.CLUSTER.DOMAIN
+* download.CLUSTER.DOMAIN
+* keep.CLUSTER.DOMAIN
+* keep0.CLUSTER.DOMAIN
+* webshell.CLUSTER.DOMAIN
+* workbench.CLUSTER.DOMAIN
+* workbench2.CLUSTER.DOMAIN
+* ws.CLUSTER.DOMAIN
diff --git a/tools/salt-install/single_host/arvados.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
similarity index 78%
copy from tools/salt-install/single_host/arvados.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
index a06244270..4aa4735d8 100644
--- a/tools/salt-install/single_host/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/multiple_hostnames/arvados.sls
@@ -107,7 +107,7 @@ arvados:
# <cluster>-nyw5e-<volume>
__CLUSTER__-nyw5e-000000000000000:
AccessViaHosts:
- http://keep0.__CLUSTER__.__DOMAIN__:25107:
+ 'http://keep0.__CLUSTER__.__DOMAIN__:25107':
ReadOnly: false
Replication: 2
Driver: Directory
@@ -122,38 +122,38 @@ arvados:
Services:
Controller:
- ExternalURL: https://__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
InternalURLs:
- http://controller.internal:8003: {}
+ 'http://controller.internal:8003': {}
DispatchCloud:
InternalURLs:
- http://__CLUSTER__.__DOMAIN__:9006: {}
+ 'http://__CLUSTER__.__DOMAIN__:9006': {}
Keepbalance:
InternalURLs:
- http://__CLUSTER__.__DOMAIN__:9005: {}
+ 'http://__CLUSTER__.__DOMAIN__:9005': {}
Keepproxy:
- ExternalURL: https://keep.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://keep.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
InternalURLs:
- http://keep.internal:25100: {}
+ 'http://keep.internal:25100': {}
Keepstore:
InternalURLs:
- http://keep0.__CLUSTER__.__DOMAIN__:25107: {}
+ 'http://keep0.__CLUSTER__.__DOMAIN__:25107': {}
RailsAPI:
InternalURLs:
- http://api.internal:8004: {}
+ 'http://api.internal:8004': {}
WebDAV:
- ExternalURL: https://collections.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://collections.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
InternalURLs:
- http://collections.internal:9002: {}
+ 'http://collections.internal:9002': {}
WebDAVDownload:
- ExternalURL: https://download.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://download.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
WebShell:
- ExternalURL: https://webshell.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://webshell.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
Websocket:
- ExternalURL: wss://ws.__CLUSTER__.__DOMAIN__/websocket
+ ExternalURL: 'wss://ws.__CLUSTER__.__DOMAIN__/websocket'
InternalURLs:
- http://ws.internal:8005: {}
+ 'http://ws.internal:8005': {}
Workbench1:
- ExternalURL: https://workbench.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://workbench.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
Workbench2:
- ExternalURL: https://workbench2.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://workbench2.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__'
diff --git a/tools/salt-install/single_host/docker.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/docker.sls
similarity index 100%
copy from tools/salt-install/single_host/docker.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/docker.sls
diff --git a/tools/salt-install/single_host/locale.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/locale.sls
similarity index 100%
copy from tools/salt-install/single_host/locale.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/locale.sls
diff --git a/tools/salt-install/single_host/nginx_api_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_api_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_api_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_api_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_controller_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_controller_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_controller_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepproxy_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_keepproxy_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepproxy_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepweb_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_keepweb_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_keepweb_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_passenger.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_passenger.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_passenger.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_passenger.sls
diff --git a/tools/salt-install/single_host/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_webshell_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_webshell_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_webshell_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_websocket_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_websocket_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_websocket_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench2_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_workbench2_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench2_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench_configuration.sls
similarity index 100%
copy from tools/salt-install/single_host/nginx_workbench_configuration.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/nginx_workbench_configuration.sls
diff --git a/tools/salt-install/single_host/postgresql.sls b/tools/salt-install/config_examples/single_host/multiple_hostnames/postgresql.sls
similarity index 100%
copy from tools/salt-install/single_host/postgresql.sls
copy to tools/salt-install/config_examples/single_host/multiple_hostnames/postgresql.sls
diff --git a/tools/salt-install/config_examples/single_host/single_hostname/README.md b/tools/salt-install/config_examples/single_host/single_hostname/README.md
new file mode 100644
index 000000000..9c7ab96c3
--- /dev/null
+++ b/tools/salt-install/config_examples/single_host/single_hostname/README.md
@@ -0,0 +1,23 @@
+Single host with a single hostname
+==================================
+
+These files let you setup Arvados on a single host using a single hostname
+for all of its components nginx's virtualhosts.
+
+The hostname MUST be given in the `local.params` file. The script won't try
+to guess it because, depending on the network architecture where you're
+installing Arvados, things might not work as expected.
+
+The services will be available on the same hostname but different ports,
+which can be given on the `local.params` file or will default to the following
+values:
+
+* CLUSTER.DOMAIN
+* collections
+* download
+* keep
+* keep0
+* webshell
+* workbench
+* workbench2
+* ws
diff --git a/tools/salt-install/single_host/arvados.sls b/tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
similarity index 78%
rename from tools/salt-install/single_host/arvados.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
index a06244270..e5e458665 100644
--- a/tools/salt-install/single_host/arvados.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/arvados.sls
@@ -107,7 +107,7 @@ arvados:
# <cluster>-nyw5e-<volume>
__CLUSTER__-nyw5e-000000000000000:
AccessViaHosts:
- http://keep0.__CLUSTER__.__DOMAIN__:25107:
+ 'http://__HOSTNAME__:25107':
ReadOnly: false
Replication: 2
Driver: Directory
@@ -122,38 +122,38 @@ arvados:
Services:
Controller:
- ExternalURL: https://__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__CONTROLLER_EXT_SSL_PORT__'
InternalURLs:
- http://controller.internal:8003: {}
+ 'http://controller.internal:8003': {}
DispatchCloud:
InternalURLs:
- http://__CLUSTER__.__DOMAIN__:9006: {}
+ 'http://__HOSTNAME__:9006': {}
Keepbalance:
InternalURLs:
- http://__CLUSTER__.__DOMAIN__:9005: {}
+ 'http://__HOSTNAME__:9005': {}
Keepproxy:
- ExternalURL: https://keep.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__KEEP_EXT_SSL_PORT__'
InternalURLs:
- http://keep.internal:25100: {}
+ 'http://keep.internal:25100': {}
Keepstore:
InternalURLs:
- http://keep0.__CLUSTER__.__DOMAIN__:25107: {}
+ 'http://keep0.internal:25107': {}
RailsAPI:
InternalURLs:
- http://api.internal:8004: {}
+ 'http://api.internal:8004': {}
WebDAV:
- ExternalURL: https://collections.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__KEEPWEB_EXT_SSL_PORT__'
InternalURLs:
- http://collections.internal:9002: {}
+ 'http://collections.internal:9002': {}
WebDAVDownload:
- ExternalURL: https://download.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__KEEPWEB_EXT_SSL_PORT__'
WebShell:
- ExternalURL: https://webshell.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__WEBSHELL_EXT_SSL_PORT__'
Websocket:
- ExternalURL: wss://ws.__CLUSTER__.__DOMAIN__/websocket
+ ExternalURL: 'wss://__HOSTNAME__:__WEBSOCKET_EXT_SSL_PORT__/websocket'
InternalURLs:
- http://ws.internal:8005: {}
+ 'http://ws.internal:8005': {}
Workbench1:
- ExternalURL: https://workbench.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__WORKBENCH1_EXT_SSL_PORT__'
Workbench2:
- ExternalURL: https://workbench2.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ ExternalURL: 'https://__HOSTNAME__:__WORKBENCH2_EXT_SSL_PORT__'
diff --git a/tools/salt-install/single_host/docker.sls b/tools/salt-install/config_examples/single_host/single_hostname/docker.sls
similarity index 100%
rename from tools/salt-install/single_host/docker.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/docker.sls
diff --git a/tools/salt-install/single_host/locale.sls b/tools/salt-install/config_examples/single_host/single_hostname/locale.sls
similarity index 100%
rename from tools/salt-install/single_host/locale.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/locale.sls
diff --git a/tools/salt-install/single_host/nginx_api_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls
similarity index 100%
rename from tools/salt-install/single_host/nginx_api_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_api_configuration.sls
diff --git a/tools/salt-install/single_host/nginx_controller_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
similarity index 92%
rename from tools/salt-install/single_host/nginx_controller_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
index 00c3b3a13..2eb33b835 100644
--- a/tools/salt-install/single_host/nginx_controller_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_controller_configuration.sls
@@ -25,7 +25,7 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- 80 default
- location /.well-known:
@@ -38,9 +38,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: __CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://controller_upstream'
diff --git a/tools/salt-install/single_host/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
similarity index 90%
rename from tools/salt-install/single_host/nginx_keepproxy_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
index 6554f79a7..b26de2710 100644
--- a/tools/salt-install/single_host/nginx_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepproxy_configuration.sls
@@ -21,9 +21,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: keep.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __KEEP_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -34,9 +34,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: keep.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __KEEP_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://keepproxy_upstream'
diff --git a/tools/salt-install/single_host/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
similarity index 77%
rename from tools/salt-install/single_host/nginx_keepweb_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
index cc871b9da..98a3cdf94 100644
--- a/tools/salt-install/single_host/nginx_keepweb_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_keepweb_configuration.sls
@@ -15,15 +15,15 @@ nginx:
servers:
managed:
- ### DEFAULT
+ ### COLLECTIONS / DOWNLOAD
arvados_collections_download_default:
enabled: true
overwrite: true
config:
- server:
- - server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __KEEPWEB_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -35,9 +35,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __KEEPWEB_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://collections_downloads_upstream'
@@ -53,5 +53,5 @@ nginx:
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- include: 'snippets/arvados-snakeoil.conf'
- - access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
- - error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
+ - access_log: /var/log/nginx/keepweb.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/keepweb.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/single_host/nginx_passenger.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_passenger.sls
similarity index 100%
rename from tools/salt-install/single_host/nginx_passenger.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_passenger.sls
diff --git a/tools/salt-install/single_host/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
similarity index 92%
rename from tools/salt-install/single_host/nginx_webshell_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
index a0756b7ce..dac606123 100644
--- a/tools/salt-install/single_host/nginx_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_webshell_configuration.sls
@@ -22,9 +22,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: webshell.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WEBSHELL_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -35,11 +35,11 @@ nginx:
overwrite: true
config:
- server:
- - server_name: webshell.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WEBSHELL_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- - location /shell.__CLUSTER__.__DOMAIN__:
+ - location /__HOSTNAME__:
- proxy_pass: 'http://webshell_upstream'
- proxy_read_timeout: 90
- proxy_connect_timeout: 90
diff --git a/tools/salt-install/single_host/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
similarity index 90%
rename from tools/salt-install/single_host/nginx_websocket_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
index ebe03f733..827524cbe 100644
--- a/tools/salt-install/single_host/nginx_websocket_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_websocket_configuration.sls
@@ -21,9 +21,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: ws.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WEBSOCKET_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -34,9 +34,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: ws.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WEBSOCKET_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://websocket_upstream'
diff --git a/tools/salt-install/single_host/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
similarity index 80%
rename from tools/salt-install/single_host/nginx_workbench2_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
index 8930be408..7f90cbc82 100644
--- a/tools/salt-install/single_host/nginx_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench2_configuration.sls
@@ -19,9 +19,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench2.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WORKBENCH2_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -32,9 +32,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench2.__CLUSTER__.__DOMAIN__
+ - server_name: workbench2.__HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WORKBENCH2_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- root: /var/www/arvados-workbench2/workbench2
@@ -42,7 +42,7 @@ nginx:
- 'if (-f $document_root/maintenance.html)':
- return: 503
- location /config.json:
- - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__"}' ~ "'" }}
+ - return: {{ "200 '" ~ '{"API_HOST":"__HOSTNAME__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
- include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/single_host/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
similarity index 91%
rename from tools/salt-install/single_host/nginx_workbench_configuration.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
index be571ca77..0cbd3e14a 100644
--- a/tools/salt-install/single_host/nginx_workbench_configuration.sls
+++ b/tools/salt-install/config_examples/single_host/single_hostname/nginx_workbench_configuration.sls
@@ -28,9 +28,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench.__CLUSTER__.__DOMAIN__
+ - server_name: __HOSTNAME__
- listen:
- - 80
+ - __WORKBENCH_EXT_SSL_PORT__
- location /.well-known:
- root: /var/www
- location /:
@@ -41,9 +41,9 @@ nginx:
overwrite: true
config:
- server:
- - server_name: workbench.__CLUSTER__.__DOMAIN__
+ - server_name: workbench.__HOSTNAME__
- listen:
- - __HOST_SSL_PORT__ http2 ssl
+ - __WORKBENCH1_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://workbench_upstream'
diff --git a/tools/salt-install/single_host/postgresql.sls b/tools/salt-install/config_examples/single_host/single_hostname/postgresql.sls
similarity index 100%
rename from tools/salt-install/single_host/postgresql.sls
rename to tools/salt-install/config_examples/single_host/single_hostname/postgresql.sls
diff --git a/tools/salt-install/local.params.example b/tools/salt-install/local.params.example
new file mode 100644
index 000000000..a88301b2a
--- /dev/null
+++ b/tools/salt-install/local.params.example
@@ -0,0 +1,64 @@
+##########################################################
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: CC-BY-SA-3.0
+
+# These are the basic parameters to configure the installation
+
+# The 5 letters name you want to give your cluster
+CLUSTER="fixme"
+
+# The domainname you want tou give to your cluster's hosts
+DOMAIN="some.domain"
+
+# When setting the cluster in a single host, you can use a single hostname
+# to access all the instances. When using virtualization (ie AWS), this should be
+# the EXTERNAL hostname for the instance.
+# If empty, the INTERNAL HOST IP will be used
+HOSTNAME=""
+CONTROLLER_EXT_SSL_PORT=8000
+KEEP_EXT_SSL_PORT=25101
+# Both for collections and downloads
+KEEPWEB_EXT_SSL_PORT=9002
+WEBSHELL_EXT_SSL_PORT=4202
+WEBSOCKET_EXT_SSL_PORT=8002
+WORKBENCH1_EXT_SSL_PORT=443
+WORKBENCH2_EXT_SSL_PORT=3001
+
+INITIAL_USER="admin"
+
+# If not specified, the initial user email will be composed as
+# INITIAL_USER at CLUSTER.DOMAIN
+INITIAL_USER_EMAIL="admin at fixme.localdomain"
+INITIAL_USER_PASSWORD="password"
+
+# The example config files you want to use. There are a few examples
+# under 'config_examples'
+CONFIG_DIR="config_examples/single_host/single_hostname"
+
+# Which release of Arvados repo you want to use
+RELEASE="production"
+# Which version of Arvados you want to install. Defaults to 'latest'
+# in the desired repo
+VERSION="latest"
+
+# Host SSL port where you want to point your browser to access Arvados
+# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
+# You can point it to another port if desired
+# In Vagrant, make sure it matches what you set in the Vagrantfile
+HOST_SSL_PORT=443
+
+# This is an arvados-formula setting.
+# If branch is set, the script will switch to it before running salt
+# Usually not needed, only used for testing
+BRANCH="master"
+
+##########################################################
+# Usually there's no need to modify things below this line
+
+# Formulas versions
+ARVADOS_TAG="v1.1.4"
+POSTGRES_TAG="v0.41.3"
+NGINX_TAG="v2.4.0"
+DOCKER_TAG="v1.0.0"
+LOCALE_TAG="v0.3.4"
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index b97d71965..f3df4109a 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -15,15 +15,6 @@ set -o pipefail
# capture the directory that the script is running from
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-CONFIG_DIR="single_host"
-RELEASE="production"
-VERSION="latest"
-ARVADOS_TAG="v1.1.4"
-POSTGRES_TAG="v0.41.3"
-NGINX_TAG="v2.4.0"
-DOCKER_TAG="v1.0.0"
-LOCALE_TAG="v0.3.4"
-
usage() {
echo >&2
echo >&2 "Usage: ${0} [-h] [-h]"
@@ -109,10 +100,35 @@ arguments() {
}
CONFIG="${SCRIPT_DIR}/local.params"
+CONFIG_DIR="config_examples/single_host/multiple_hostnames"
LOG_LEVEL="info"
HOST_SSL_PORT=443
TESTS_DIR="tests"
+CLUSTER=""
+DOMAIN=""
+HOSTNAME=""
+INITIAL_USER=""
+INITIAL_USER_EMAIL=""
+INITIAL_USER_PASSWORD=""
+
+CONTROLLER_EXT_SSL_PORT=8000
+KEEP_EXT_SSL_PORT=25101
+# Both for collections and downloads
+KEEPWEB_EXT_SSL_PORT=9002
+WEBSHELL_EXT_SSL_PORT=4202
+WEBSOCKET_EXT_SSL_PORT=8002
+WORKBENCH1_EXT_SSL_PORT=443
+WORKBENCH2_EXT_SSL_PORT=3001
+
+RELEASE="production"
+VERSION="latest"
+ARVADOS_TAG="v1.1.4"
+POSTGRES_TAG="v0.41.3"
+NGINX_TAG="v2.4.0"
+DOCKER_TAG="v1.0.0"
+LOCALE_TAG="v0.3.4"
+
arguments ${@}
if [ -s ${CONFIG} ]; then
@@ -156,9 +172,7 @@ pillar_roots:
- ${P_DIR}
EOFSM
-mkdir -p ${S_DIR}
-mkdir -p ${F_DIR}
-mkdir -p ${P_DIR}
+mkdir -p ${S_DIR} ${F_DIR} ${P_DIR}
# States
cat > ${S_DIR}/top.sls << EOFTSLS
@@ -227,8 +241,15 @@ for f in "${SOURCE_PILLARS_DIR}"/*; do
sed "s/__CLUSTER__/${CLUSTER}/g;
s/__DOMAIN__/${DOMAIN}/g;
s/__RELEASE__/${RELEASE}/g;
+ s/__CONTROLLER_EXT_SSL_PORT__/${CONTROLLER_EXT_SSL_PORT}/g;
+ s/__KEEP_EXT_SSL_PORT__/${KEEP_EXT_SSL_PORT}/g;
+ s/__WEBSHELL_EXT_SSL_PORT__/${WEBSHELL_EXT_SSL_PORT}/g;
+ s/__WORKBENCH1_EXT__SSL_PORT__/${WORKBENCH1_EXT__SSL_PORT}/g;
+ s/__WORKBENCH2_EXT__SSL_PORT__/${WORKBENCH2_EXT__SSL_PORT}/g;
+ s/__WEBSOCKET_EXT_SSL_PORT__/${WEBSOCKET_EXT_SSL_PORT}/g;
+ s/__HOSTNAME__/${HOSTNAME}/g;
+ s/__KEEPWEB_EXT_SSL_PORT__/${KEEPWEB_EXT_SSL_PORT}/g;
s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
- s/__GUEST_SSL_PORT__/${GUEST_SSL_PORT}/g;
s/__INITIAL_USER__/${INITIAL_USER}/g;
s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
commit 2b9350438e027719deecf85bfdf9fc9ae4ef177d
Author: Javier Bértoli <jbertoli at curii.com>
Date: Mon Jan 25 07:10:01 2021 -0300
feat(provision): manage setup params from local file
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/Vagrantfile b/tools/salt-install/Vagrantfile
index 6966ea834..60f57ca66 100644
--- a/tools/salt-install/Vagrantfile
+++ b/tools/salt-install/Vagrantfile
@@ -34,6 +34,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
path: "provision.sh",
args: [
# "--debug",
+ "--config /vagrant/local.params",
"--test",
"--vagrant",
"--ssl-port=8443"
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 79712d3f9..b97d71965 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -24,43 +24,37 @@ NGINX_TAG="v2.4.0"
DOCKER_TAG="v1.0.0"
LOCALE_TAG="v0.3.4"
-if [ -s ${SCRIPT_DIR}/local.params ]; then
- source ${SCRIPT_DIR}/local.params
-else
- echo >&2 "Please create a '${SCRIPT_DIR}/local.params' file with initial values, as described in FIXME_URL_TO_DESCR"
- exit 1
-fi
-
usage() {
echo >&2
echo >&2 "Usage: ${0} [-h] [-h]"
echo >&2
echo >&2 "${0} options:"
- echo >&2 " -d, --debug Run salt installation in debug mode"
- echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
- echo >&2 " -t, --test Test installation running a CWL workflow"
- echo >&2 " -r, --roles List of Arvados roles to apply to the host, comma separated"
- echo >&2 " Possible values are:"
- echo >&2 " api"
- echo >&2 " controller"
- echo >&2 " keepstore"
- echo >&2 " websocket"
- echo >&2 " keepweb"
- echo >&2 " workbench2"
- echo >&2 " keepproxy"
- echo >&2 " shell"
- echo >&2 " workbench"
- echo >&2 " dispatcher"
- echo >&2 " Defaults to applying them all"
- echo >&2 " -h, --help Display this help and exit"
- echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
+ echo >&2 " -d, --debug Run salt installation in debug mode"
+ echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
+ echo >&2 " -c <local.params>, --config <local.params> Path to the local.params config file"
+ echo >&2 " -t, --test Test installation running a CWL workflow"
+ echo >&2 " -r, --roles List of Arvados roles to apply to the host, comma separated"
+ echo >&2 " Possible values are:"
+ echo >&2 " api"
+ echo >&2 " controller"
+ echo >&2 " keepstore"
+ echo >&2 " websocket"
+ echo >&2 " keepweb"
+ echo >&2 " workbench2"
+ echo >&2 " keepproxy"
+ echo >&2 " shell"
+ echo >&2 " workbench"
+ echo >&2 " dispatcher"
+ echo >&2 " Defaults to applying them all"
+ echo >&2 " -h, --help Display this help and exit"
+ echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
echo >&2
}
arguments() {
# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
- TEMP=$(getopt -o dhp:r:tv \
- --long debug,help,ssl-port:,roles:,test,vagrant \
+ TEMP=$(getopt -o c:dhp:r:tv \
+ --long config:,debug,help,ssl-port:,roles:,test,vagrant \
-n "${0}" -- "${@}")
if [ ${?} != 0 ] ; then echo "GNU getopt missing? Use -h for help"; exit 1 ; fi
@@ -69,6 +63,10 @@ arguments() {
while [ ${#} -ge 1 ]; do
case ${1} in
+ -c | --config)
+ CONFIG=${2}
+ shift 2
+ ;;
-d | --debug)
LOG_LEVEL="debug"
shift
@@ -110,12 +108,20 @@ arguments() {
done
}
+CONFIG="${SCRIPT_DIR}/local.params"
LOG_LEVEL="info"
HOST_SSL_PORT=443
TESTS_DIR="tests"
arguments ${@}
+if [ -s ${CONFIG} ]; then
+ source ${CONFIG}
+else
+ echo >&2 "Please create a '${CONFIG}' file with initial values, as described in FIXME_URL_TO_DESCR"
+ exit 1
+fi
+
# Salt's dir
## states
S_DIR="/srv/salt"
commit b57db2bcf9ba4ef5f74b9b1e4c0a0788f6439658
Author: Javier Bértoli <jbertoli at curii.com>
Date: Fri Jan 22 12:41:12 2021 -0300
feat(provision): allow to install individual roles
refs #17246
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/tools/salt-install/README.md b/tools/salt-install/README.md
index 10d08b414..58800dd6f 100644
--- a/tools/salt-install/README.md
+++ b/tools/salt-install/README.md
@@ -6,15 +6,21 @@
##### About
-This directory holds a small script to install Arvados on a single node, using the
+This directory holds a small script to help you get Arvados up and running, using the
[Saltstack arvados-formula](https://github.com/saltstack-formulas/arvados-formula)
in master-less mode.
-The fastest way to get it running is to modify the first lines in the `provision.sh`
-script to suit your needs, copy it in the host where you want to install Arvados
-and run it as root.
+There are a few preset examples that you can use:
-There's an example `Vagrantfile` also, to install it in a vagrant box if you want
+* `single_host`: Install all the Arvados components in a single host. Suitable for testing
+ or demo-ing, but not recommended for production use.
+* `multi_host/aws`: Let's you install different Arvados components in different hosts on AWS.
+
+The fastest way to get it running is to copy the `local.params.example` file to `local.params`,
+edit and modify the file to suit your needs, copy this file along with the `provision.sh` script
+into the host where you want to install Arvados and run the `provision.sh` script as root.
+
+There's an example `Vagrantfile` also, to install Arvados in a vagrant box if you want
to try it locally.
For more information, please read https://doc.arvados.org/main/install/salt-single-host.html
diff --git a/tools/salt-install/provision.sh b/tools/salt-install/provision.sh
index 31266c1b8..79712d3f9 100755
--- a/tools/salt-install/provision.sh
+++ b/tools/salt-install/provision.sh
@@ -10,55 +10,26 @@
#
# vagrant up
-##########################################################
-# This section are the basic parameters to configure the installation
-
-# The 5 letters name you want to give your cluster
-CLUSTER="arva2"
-DOMAIN="arv.local"
-
-INITIAL_USER="admin"
+set -o pipefail
-# If not specified, the initial user email will be composed as
-# INITIAL_USER at CLUSTER.DOMAIN
-INITIAL_USER_EMAIL="${INITIAL_USER}@${CLUSTER}.${DOMAIN}"
-INITIAL_USER_PASSWORD="password"
+# capture the directory that the script is running from
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-# The example config you want to use. Currently, only "single_host" is
-# available
CONFIG_DIR="single_host"
-
-# Which release of Arvados repo you want to use
RELEASE="production"
-# Which version of Arvados you want to install. Defaults to 'latest'
-# in the desired repo
VERSION="latest"
-
-# Host SSL port where you want to point your browser to access Arvados
-# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
-# You can point it to another port if desired
-# In Vagrant, make sure it matches what you set in the Vagrantfile
-# HOST_SSL_PORT=443
-
-# This is a arvados-formula setting.
-# If branch is set, the script will switch to it before running salt
-# Usually not needed, only used for testing
-# BRANCH="master"
-
-##########################################################
-# Usually there's no need to modify things below this line
-
-# Formulas versions
ARVADOS_TAG="v1.1.4"
POSTGRES_TAG="v0.41.3"
NGINX_TAG="v2.4.0"
DOCKER_TAG="v1.0.0"
LOCALE_TAG="v0.3.4"
-set -o pipefail
-
-# capture the directory that the script is running from
-SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+if [ -s ${SCRIPT_DIR}/local.params ]; then
+ source ${SCRIPT_DIR}/local.params
+else
+ echo >&2 "Please create a '${SCRIPT_DIR}/local.params' file with initial values, as described in FIXME_URL_TO_DESCR"
+ exit 1
+fi
usage() {
echo >&2
@@ -68,6 +39,19 @@ usage() {
echo >&2 " -d, --debug Run salt installation in debug mode"
echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
echo >&2 " -t, --test Test installation running a CWL workflow"
+ echo >&2 " -r, --roles List of Arvados roles to apply to the host, comma separated"
+ echo >&2 " Possible values are:"
+ echo >&2 " api"
+ echo >&2 " controller"
+ echo >&2 " keepstore"
+ echo >&2 " websocket"
+ echo >&2 " keepweb"
+ echo >&2 " workbench2"
+ echo >&2 " keepproxy"
+ echo >&2 " shell"
+ echo >&2 " workbench"
+ echo >&2 " dispatcher"
+ echo >&2 " Defaults to applying them all"
echo >&2 " -h, --help Display this help and exit"
echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
echo >&2
@@ -75,8 +59,8 @@ usage() {
arguments() {
# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
- TEMP=$(getopt -o dhp:tv \
- --long debug,help,ssl-port:,test,vagrant \
+ TEMP=$(getopt -o dhp:r:tv \
+ --long debug,help,ssl-port:,roles:,test,vagrant \
-n "${0}" -- "${@}")
if [ ${?} != 0 ] ; then echo "GNU getopt missing? Use -h for help"; exit 1 ; fi
@@ -89,6 +73,23 @@ arguments() {
LOG_LEVEL="debug"
shift
;;
+ -p | --ssl-port)
+ HOST_SSL_PORT=${2}
+ shift 2
+ ;;
+ -r | --roles)
+ for i in ${2//,/ }
+ do
+ # Verify the role exists
+ if [[ ! "api,controller,keepstore,websocket,keepweb,workbench2,keepproxy,shell,workbench,dispatcher" == *"$i"* ]]; then
+ echo "The role '${i}' is not a valid role"
+ usage
+ exit 1
+ fi
+ ROLES="${ROLES} ${i}"
+ done
+ shift 2
+ ;;
-t | --test)
TEST="yes"
shift
@@ -97,10 +98,6 @@ arguments() {
VAGRANT="yes"
shift
;;
- -p | --ssl-port)
- HOST_SSL_PORT=${2}
- shift 2
- ;;
--)
shift
break
@@ -167,9 +164,17 @@ base:
- nginx.passenger
- postgres
- docker
- - arvados
EOFTSLS
+# If we want specific roles for a node, just add those states
+if [ -z "${ROLES}" ]; then
+ echo ' - arvados' >> ${S_DIR}/top.sls
+else
+ for R in ${ROLES}; do
+ echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ done
+fi
+
# Pillars
cat > ${P_DIR}/top.sls << EOFPSLS
base:
@@ -191,7 +196,7 @@ EOFPSLS
# Get the formula and dependencies
cd ${F_DIR} || exit 1
-git clone --branch "${ARVADOS_TAG}" https://github.com/saltstack-formulas/arvados-formula.git
+git clone --branch "${ARVADOS_TAG}" https://github.com/arvados/arvados-formula.git
git clone --branch "${DOCKER_TAG}" https://github.com/saltstack-formulas/docker-formula.git
git clone --branch "${LOCALE_TAG}" https://github.com/saltstack-formulas/locale-formula.git
git clone --branch "${NGINX_TAG}" https://github.com/saltstack-formulas/nginx-formula.git
commit e15b1cd2d4c29381d0ccbdc33cb48e5034dab2f9
Author: Javier Bértoli <jbertoli at curii.com>
Date: Fri Jan 15 09:36:07 2021 -0300
Documentation: Explain that the canonical salt installer is hosted in Arvados' github account
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/doc/install/salt-multi-host.html.textile.liquid b/doc/install/salt-multi-host.html.textile.liquid
index 4ba153faf..50de6e439 100644
--- a/doc/install/salt-multi-host.html.textile.liquid
+++ b/doc/install/salt-multi-host.html.textile.liquid
@@ -40,25 +40,25 @@ The formulas we use are:
* "docker":https://github.com/saltstack-formulas/docker-formula.git
* "locale":https://github.com/saltstack-formulas/locale-formula.git
-There are example Salt pillar files for each of those formulas in the "arvados-formula's test/salt/pillar/examples":https://github.com/saltstack-formulas/arvados-formula/tree/master/test/salt/pillar/examples directory. As they are, they allow you to get all the main Arvados components up and running.
+There are example Salt pillar files for each of those formulas in the "arvados-formula's test/salt/pillar/examples":https://github.com/arvados/arvados-formula/tree/master/test/salt/pillar/examples directory. As they are, they allow you to get all the main Arvados components up and running.
h2(#saltstack). Install Arvados using Saltstack
This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
-The Arvados formula we maintain is located in the Saltstack's community repository of formulas:
+The Arvados formula we maintain is located in Arvados' Github account and should be considered the canonical place to download its most up-to-date version:
-* "arvados-formula":https://github.com/saltstack-formulas/arvados-formula.git
+* "arvados-formula":https://github.com/arvados/arvados-formula.git
-The @development@ version lives in our own repository
+As the Saltstack's community keeps a "repository of formulas":https://github.com/saltstack-formulas/ in Github, we also provide
-* "arvados-formula development":https://github.com/arvados/arvados-formula.git
+* "a copy of the formula":https://github.com/saltstack-formulas/arvados-formula.git
-This last one might break from time to time, as we try and add new features. Use with caution.
+there, and do our best effort to keep it in sync with ours.
-As much as possible, we try to keep it up to date, with example pillars to help you deploy Arvados.
+A @development@ branch exists which uses Arvados' development repositories. This last one might break from time to time, as we try and add new features. As much as possible, we try to keep it up to date, with example pillars to help you deploy Arvados. Use with caution.
-For those familiar with Saltstack, the process to get it deployed is similar to any other formula:
+For those familiar with Saltstack, the process to get Arvados deployed is similar to any other formula:
1. Fork/copy the formula to your Salt master host.
2. Edit the Arvados, nginx, postgres, locale and docker pillars to match your desired configuration.
diff --git a/doc/install/salt.html.textile.liquid b/doc/install/salt.html.textile.liquid
index 8f5ecc8c6..2b7aa6602 100644
--- a/doc/install/salt.html.textile.liquid
+++ b/doc/install/salt.html.textile.liquid
@@ -14,7 +14,7 @@ SPDX-License-Identifier: CC-BY-SA-3.0
h2(#introduction). Introduction
-To ease the installation of the various Arvados components, we have developed a "Saltstack":https://www.saltstack.com/ 's "arvados-formula":https://github.com/saltstack-formulas/arvados-formula which can help you get an Arvados cluster up and running.
+To ease the installation of the various Arvados components, we have developed a "Saltstack":https://www.saltstack.com/ 's "arvados-formula":https://github.com/arvados/arvados-formula which can help you get an Arvados cluster up and running.
Saltstack is a Python-based, open-source software for event-driven IT automation, remote task execution, and configuration management. It can be used in a master/minion setup or master-less.
@@ -24,6 +24,6 @@ h2(#installmethod). Choose an installation method
The salt formulas can be used in different ways. Choose one of these three options to install Arvados:
-* "Use Vagrant to install Arvados in a virtual machine":salt-vagrant.html
* "Arvados on a single host":salt-single-host.html
+* "Use Vagrant to install Arvados in a virtual machine":salt-vagrant.html
* "Arvados across multiple hosts":salt-multi-host.html
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list