[ARVADOS] updated: 2.1.0-1481-g15a2556dc

Git user git at public.arvados.org
Tue Oct 12 00:52:14 UTC 2021 

Summary of changes:
 doc/install/salt-multi-host.html.textile.liquid    | 23 ++++++-
 doc/install/salt-single-host.html.textile.liquid   | 21 ++++++
 tools/salt-install/Vagrantfile                     |  8 ++-
 .../letsencrypt_controller_configuration.sls       | 10 +--
 .../letsencrypt_keepproxy_configuration.sls        | 10 +--
 .../pillars/letsencrypt_keepweb_configuration.sls  | 11 ----
 .../pillars/letsencrypt_webshell_configuration.sls |  8 ---
 .../letsencrypt_websocket_configuration.sls        | 10 +--
 .../letsencrypt_workbench2_configuration.sls       |  8 ---
 .../letsencrypt_workbench_configuration.sls        |  8 ---
 .../aws/pillars/nginx_api_configuration.sls        |  2 +-
 ...ion.sls => nginx_collections_configuration.sls} | 33 ++++------
 .../aws/pillars/nginx_controller_configuration.sls | 11 ++--
 ...ration.sls => nginx_download_configuration.sls} | 33 ++++------
 .../aws/pillars/nginx_keepproxy_configuration.sls  | 11 ++--
 .../aws/pillars/nginx_keepweb_configuration.sls    | 75 +---------------------
 .../aws/pillars/nginx_webshell_configuration.sls   | 11 ++--
 .../aws/pillars/nginx_websocket_configuration.sls  |  9 +--
 .../aws/pillars/nginx_workbench2_configuration.sls |  9 +--
 .../aws/pillars/nginx_workbench_configuration.sls  |  9 +--
 .../local.params.example.multiple_hosts            | 22 ++++++-
 ...l.params.example.single_host_multiple_hostnames |  3 +-
 22 files changed, 135 insertions(+), 210 deletions(-)
 copy tools/salt-install/config_examples/multi_host/aws/pillars/{nginx_keepproxy_configuration.sls => nginx_collections_configuration.sls} (55%)
 copy tools/salt-install/config_examples/multi_host/aws/pillars/{nginx_keepproxy_configuration.sls => nginx_download_configuration.sls} (55%)

       via  15a2556dc2f45b393641a9ee012306dacb3edd5c (commit)
       via  3b0f80205c1942cc954eb891691c7c382aa9c87c (commit)
       via  2eeaf4eacf23e166b60cf95562ea2727a68e1e08 (commit)
      from  6f84dbb74b46470e937a52cbbb5de4c5b825e122 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.


commit 15a2556dc2f45b393641a9ee012306dacb3edd5c
Author: Javier Bértoli <jbertoli at curii.com>
Date:   Mon Oct 11 21:51:27 2021 -0300

    17742: add documentation about custom certs usage
    
    Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>

diff --git a/doc/install/salt-multi-host.html.textile.liquid b/doc/install/salt-multi-host.html.textile.liquid
index 0d7fb916e..f3afcd503 100644
--- a/doc/install/salt-multi-host.html.textile.liquid
+++ b/doc/install/salt-multi-host.html.textile.liquid
@@ -106,7 +106,28 @@ cp -r config_examples/multi_host/aws local_config_dir
 
 Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_INT_IP, *_TOKEN</b> and <b>*KEY</b> variables. Those variables will be used to do a search and replace on the <i>pillars/*</i> in place of any matching __VARIABLE__.
 
-The <i>multi_host</i> include LetsEncrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53. If you will provide custom certificates, please set the variable <i>USE_LETSENCRYPT=no</i>.
+The <i>multi_host</i> include LetsEncrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53.
+
+If you plan to use custom certificates, please set the variable <i>USE_LETSENCRYPT=no</i> and copy your certificates to the directory specified with the variable @CUSTOM_CERTS_DIR@ (usually "./certs") in the remote directory where you copied the @provision.sh@ script. From this dir, the provision script will install the certificates required for the role you're installing.
+
+The script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>downoad / collections</i>):
+
+* "controller"
+* "websocket"
+* "workbench"
+* "workbench2"
+* "webshell"
+* "download"         # Part of keepweb
+* "collections"      # Part of keepweb
+* "keepproxy"
+
+Ie., for 'keepproxy', the script will lookup for
+
+<notextile>
+<pre><code>${CUSTOM_CERTS_DIR}/keepproxy.crt
+${CUSTOM_CERTS_DIR}/keepproxy.key
+</code></pre>
+</notextile>
 
 h3(#further_customization). Further customization of the installation (modifying the salt pillars and states)
 
diff --git a/doc/install/salt-single-host.html.textile.liquid b/doc/install/salt-single-host.html.textile.liquid
index 6ca6738e3..857cdb0dc 100644
--- a/doc/install/salt-single-host.html.textile.liquid
+++ b/doc/install/salt-single-host.html.textile.liquid
@@ -55,6 +55,27 @@ cp -r config_examples/single_host/single_hostname local_config_dir
 
 Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_PORT, *_TOKEN</b> and <b>*KEY</b> variables.
 
+If you plan to use custom certificates, please set the variable <i>USE_LETSENCRYPT=no</i> and copy your certificates to the directory specified with the variable @CUSTOM_CERTS_DIR@ (usually "./certs") in the remote directory where you copied the @provision.sh@ script. From this dir, the provision script will install the certificates required for the role you're installing.
+
+The script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>downoad / collections</i>):
+
+* "controller"
+* "websocket"
+* "workbench"
+* "workbench2"
+* "webshell"
+* "download"         # Part of keepweb
+* "collections"      # Part of keepweb
+* "keepproxy"
+
+Ie., for 'keepproxy', the script will lookup for
+
+<notextile>
+<pre><code>${CUSTOM_CERTS_DIR}/keepproxy.crt
+${CUSTOM_CERTS_DIR}/keepproxy.key
+</code></pre>
+</notextile>
+
 h3(#single_host_multiple_hostnames). Single host / multiple hostnames (Alternative configuration)
 <notextile>
 <pre><code>cp local.params.example.single_host_multiple_hostnames local.params

commit 3b0f80205c1942cc954eb891691c7c382aa9c87c
Author: Javier Bértoli <jbertoli at curii.com>
Date:   Mon Oct 11 21:49:43 2021 -0300

    17742: add custom certs to the local.params files
    
    Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>

diff --git a/tools/salt-install/Vagrantfile b/tools/salt-install/Vagrantfile
index a3463bfc5..f7f8da3b1 100644
--- a/tools/salt-install/Vagrantfile
+++ b/tools/salt-install/Vagrantfile
@@ -35,14 +35,18 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
                                cp -vr /vagrant/tests /home/vagrant/tests;
                                sed 's#cluster_fixme_or_this_wont_work#harpo#g;
                                     s#domain_fixme_or_this_wont_work#local#g;
-                                    s/#\ BRANCH=\"main\"/\ BRANCH=\"main\"/g;
-                                    s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=8443#g' \
+                                    s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=8443#g;
+                                    s#RELEASE=\"production\"#RELEASE=\"development\"#g;
+                                    s/# VERSION=.*$/VERSION=\"latest\"/g;
+                                    s/#\ BRANCH=\"main\"/\ BRANCH=\"main\"/g' \
                                     /vagrant/local.params.example.single_host_multiple_hostnames > /tmp/local.params.single_host_multiple_hostnames"
+
      arv.vm.provision "shell",
                       path: "provision.sh",
                       args: [
                         # "--debug",
                         "--config /tmp/local.params.single_host_multiple_hostnames",
+                        "--development",
                         "--test",
                         "--vagrant"
                       ].join(" ")
diff --git a/tools/salt-install/local.params.example.multiple_hosts b/tools/salt-install/local.params.example.multiple_hosts
index 283c631ec..fdba88dbe 100644
--- a/tools/salt-install/local.params.example.multiple_hosts
+++ b/tools/salt-install/local.params.example.multiple_hosts
@@ -43,7 +43,6 @@ DATABASE_INT_IP=10.0.0.6
 SHELL_INT_IP=10.0.0.7
 
 INITIAL_USER="admin"
-INITIAL_USER_PASSWORD="password"
 
 # If not specified, the initial user email will be composed as
 # INITIAL_USER at CLUSTER.DOMAIN
@@ -64,7 +63,7 @@ DATABASE_PASSWORD=please_set_this_to_some_secure_value
 # salt formula (https://github.com/saltstack-formulas/letsencrypt-formula) to try to
 # automatically obtain and install SSL certificates for your instances or set this
 # variable to "no", provide and upload your own certificates to the instances and
-# modify the 'nginx_*' salt pillars accordingly
+# modify the 'nginx_*' salt pillars accordingly (see CUSTOM_CERTS_DIR below) 
 USE_LETSENCRYPT="yes"
 USE_LETSENCRYPT_IAM_USER="yes"
 # For collections, we need to obtain a wildcard certificate for
@@ -76,6 +75,25 @@ LE_AWS_REGION="us-east-1"
 LE_AWS_ACCESS_KEY_ID="AKIABCDEFGHIJKLMNOPQ"
 LE_AWS_SECRET_ACCESS_KEY="thisistherandomstringthatisyoursecretkey"
 
+# If you going to provide your own certificates for Arvados, the provision script can
+# help you deploy them. In order to do that, you need to set `USE_LETSENCRYPT=no` above,
+# and copy the required certificates under the directory specified in the next line.
+# The certs will be copied from this directory by the provision script.
+CUSTOM_CERTS_DIR="./certs"
+# The script expects cert/key files with these basenames (matching the role except for
+# keepweb, which is split in both downoad/collections):
+#  "controller"
+#  "websocket"
+#  "workbench"
+#  "workbench2"
+#  "webshell"
+#  "download"         # Part of keepweb
+#  "collections"      # Part of keepweb
+#  "keep"             # Keepproxy
+# Ie., 'keep', the script will lookup for
+# ${CUSTOM_CERTS_DIR}/keep.crt
+# ${CUSTOM_CERTS_DIR}/keep.key
+
 # The directory to check for the config files (pillars, states) you want to use.
 # There are a few examples under 'config_examples'.
 # CONFIG_DIR="local_config_dir"
diff --git a/tools/salt-install/local.params.example.single_host_multiple_hostnames b/tools/salt-install/local.params.example.single_host_multiple_hostnames
index 46ac88187..cf79fe244 100644
--- a/tools/salt-install/local.params.example.single_host_multiple_hostnames
+++ b/tools/salt-install/local.params.example.single_host_multiple_hostnames
@@ -45,7 +45,7 @@ DATABASE_PASSWORD=please_set_this_to_some_secure_value
 # salt formula (https://github.com/saltstack-formulas/letsencrypt-formula) to try to
 # automatically obtain and install SSL certificates for your instances or set this
 # variable to "no", provide and upload your own certificates to the instances and
-# modify the 'nginx_*' salt pillars accordingly (see below, param CUSTOM_CERTS_DIR)
+# modify the 'nginx_*' salt pillars accordingly (see CUSTOM_CERTS_DIR below)
 USE_LETSENCRYPT="no"
 
 # If you going to provide your own certificates for Arvados, the provision script can
@@ -55,7 +55,6 @@ USE_LETSENCRYPT="no"
 CUSTOM_CERTS_DIR="./certs"
 # The script expects cert/key files with these basenames (matching the role except for
 # keepweb, which is split in both downoad/collections):
-# CUSTOM_CERTS_DIR.
 #  "controller"
 #  "websocket"
 #  "workbench"

commit 2eeaf4eacf23e166b60cf95562ea2727a68e1e08
Author: Javier Bértoli <jbertoli at curii.com>
Date:   Mon Oct 11 19:03:26 2021 -0300

    17742: modify multi_host/aws example's nginx
    
    to use individual certificates and allow custom certs
    Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>

diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_controller_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_controller_configuration.sls
index 68c8512e7..1f088a8a7 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_controller_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_controller_configuration.sls
@@ -6,13 +6,5 @@
 ### LETSENCRYPT
 letsencrypt:
   domainsets:
-    __CLUSTER__.__DOMAIN__:
+    controller.__CLUSTER__.__DOMAIN__:
       - __CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
-  ### SNIPPETS
-  snippets:
-    __CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
-      - ssl_certificate: /etc/letsencrypt/live/__CLUSTER__.__DOMAIN__/fullchain.pem
-      - ssl_certificate_key: /etc/letsencrypt/live/__CLUSTER__.__DOMAIN__/privkey.pem
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_keepproxy_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_keepproxy_configuration.sls
index 3056b89d4..b2945e611 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_keepproxy_configuration.sls
@@ -6,13 +6,5 @@
 ### LETSENCRYPT
 letsencrypt:
   domainsets:
-    keep.__CLUSTER__.__DOMAIN__:
+    keepproxy.__CLUSTER__.__DOMAIN__:
       - keep.__CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
-  ### SNIPPETS
-  snippets:
-    keep.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
-      - ssl_certificate: /etc/letsencrypt/live/keep.__CLUSTER__.__DOMAIN__/fullchain.pem
-      - ssl_certificate_key: /etc/letsencrypt/live/keep.__CLUSTER__.__DOMAIN__/privkey.pem
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_keepweb_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_keepweb_configuration.sls
index c1720ad04..f95d7e619 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_keepweb_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_keepweb_configuration.sls
@@ -11,14 +11,3 @@ letsencrypt:
     collections.__CLUSTER__.__DOMAIN__:
       - collections.__CLUSTER__.__DOMAIN__
       - '*.collections.__CLUSTER__.__DOMAIN__'
-
-### NGINX
-nginx:
-  ### SNIPPETS
-  snippets:
-    download.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
-      - ssl_certificate: /etc/letsencrypt/live/download.__CLUSTER__.__DOMAIN__/fullchain.pem
-      - ssl_certificate_key: /etc/letsencrypt/live/download.__CLUSTER__.__DOMAIN__/privkey.pem
-    collections.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
-      - ssl_certificate: /etc/letsencrypt/live/collections.__CLUSTER__.__DOMAIN__/fullchain.pem
-      - ssl_certificate_key: /etc/letsencrypt/live/collections.__CLUSTER__.__DOMAIN__/privkey.pem
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_webshell_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_webshell_configuration.sls
index e9d2bb018..17e6422f4 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_webshell_configuration.sls
@@ -8,11 +8,3 @@ letsencrypt:
   domainsets:
     webshell.__CLUSTER__.__DOMAIN__:
       - webshell.__CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
-  ### SNIPPETS
-  snippets:
-    webshell.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
-      - ssl_certificate: /etc/letsencrypt/live/webshell.__CLUSTER__.__DOMAIN__/fullchain.pem
-      - ssl_certificate_key: /etc/letsencrypt/live/webshell.__CLUSTER__.__DOMAIN__/privkey.pem
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_websocket_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_websocket_configuration.sls
index d24431fac..6515b3bd0 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_websocket_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_websocket_configuration.sls
@@ -6,13 +6,5 @@
 ### LETSENCRYPT
 letsencrypt:
   domainsets:
-    ws.__CLUSTER__.__DOMAIN__:
+    websocket.__CLUSTER__.__DOMAIN__:
       - ws.__CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
-  ### SNIPPETS
-  snippets:
-    ws.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
-      - ssl_certificate: /etc/letsencrypt/live/ws.__CLUSTER__.__DOMAIN__/fullchain.pem
-      - ssl_certificate_key: /etc/letsencrypt/live/ws.__CLUSTER__.__DOMAIN__/privkey.pem
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_workbench2_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_workbench2_configuration.sls
index 5aa634286..2bcf2b784 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_workbench2_configuration.sls
@@ -8,11 +8,3 @@ letsencrypt:
   domainsets:
     workbench2.__CLUSTER__.__DOMAIN__:
       - workbench2.__CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
-  ### SNIPPETS
-  snippets:
-    workbench2.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
-      - ssl_certificate: /etc/letsencrypt/live/workbench2.__CLUSTER__.__DOMAIN__/fullchain.pem
-      - ssl_certificate_key: /etc/letsencrypt/live/workbench2.__CLUSTER__.__DOMAIN__/privkey.pem
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_workbench_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_workbench_configuration.sls
index 4620f79e3..9ef348719 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_workbench_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/letsencrypt_workbench_configuration.sls
@@ -8,11 +8,3 @@ letsencrypt:
   domainsets:
     workbench.__CLUSTER__.__DOMAIN__:
       - workbench.__CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
-  ### SNIPPETS
-  snippets:
-    workbench.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
-      - ssl_certificate: /etc/letsencrypt/live/workbench.__CLUSTER__.__DOMAIN__/fullchain.pem
-      - ssl_certificate_key: /etc/letsencrypt/live/workbench.__CLUSTER__.__DOMAIN__/privkey.pem
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_api_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_api_configuration.sls
index c0b087045..9fbf90dd2 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_api_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_api_configuration.sls
@@ -13,7 +13,7 @@ nginx:
   ### SITES
   servers:
     managed:
-      arvados_api:
+      arvados_api.conf:
         enabled: true
         overwrite: true
         config:
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_collections_configuration.sls
similarity index 55%
copy from tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepproxy_configuration.sls
copy to tools/salt-install/config_examples/multi_host/aws/pillars/nginx_collections_configuration.sls
index fac97f3c6..00be378c1 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_collections_configuration.sls
@@ -5,41 +5,34 @@
 
 ### NGINX
 nginx:
-  ### SERVER
-  server:
-    config:
-      ### STREAMS
-      http:
-        upstream keepproxy_upstream:
-          - server: 'localhost:25107 fail_timeout=10s'
-
   servers:
     managed:
       ### DEFAULT
-      arvados_keepproxy_default:
+      arvados_collections_default.conf:
         enabled: true
         overwrite: true
         config:
           - server:
-            - server_name: keep.__CLUSTER__.__DOMAIN__
+            - server_name: '~^(.*\.)?collections\.__CLUSTER__\.__DOMAIN__'
             - listen:
               - 80
             - location /:
               - return: '301 https://$host$request_uri'
 
-      arvados_keepproxy_ssl:
+      ### COLLECTIONS
+      arvados_collections_ssl.conf:
         enabled: true
         overwrite: true
         requires:
-          cmd: create-initial-cert-keep.__CLUSTER__.__DOMAIN__-keep.__CLUSTER__.__DOMAIN__
+          __CERT_REQUIRES__
         config:
           - server:
-            - server_name: keep.__CLUSTER__.__DOMAIN__
+            - server_name: '~^(.*\.)?collections\.__CLUSTER__\.__DOMAIN__'
             - listen:
-              - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
+              - __KEEPWEB_EXT_SSL_PORT__ http2 ssl
             - index: index.html index.htm
             - location /:
-              - proxy_pass: 'http://keepproxy_upstream'
+              - proxy_pass: 'http://collections_downloads_upstream'
               - proxy_read_timeout: 90
               - proxy_connect_timeout: 90
               - proxy_redirect: 'off'
@@ -48,11 +41,11 @@ nginx:
               - proxy_set_header: 'X-Real-IP $remote_addr'
               - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
               - proxy_buffering: 'off'
-            - client_body_buffer_size: 64M
-            - client_max_body_size: 64M
+            - client_max_body_size: 0
             - proxy_http_version: '1.1'
             - proxy_request_buffering: 'off'
             - include: snippets/ssl_hardening_default.conf
-            - include: snippets/keep.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
-            - access_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.access.log combined
-            - error_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.error.log
+            - ssl_certificate: __CERT_PEM__
+            - ssl_certificate_key: __CERT_KEY__
+            - access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
+            - error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_controller_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_controller_configuration.sls
index aa11cca74..41d6e1365 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_controller_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_controller_configuration.sls
@@ -21,7 +21,7 @@ nginx:
   servers:
     managed:
       ### DEFAULT
-      arvados_controller_default:
+      arvados_controller_default.conf:
         enabled: true
         overwrite: true
         config:
@@ -29,14 +29,16 @@ nginx:
             - server_name: __CLUSTER__.__DOMAIN__
             - listen:
               - 80 default
+            - location /.well-known:
+              - root: /var/www
             - location /:
               - return: '301 https://$host$request_uri'
 
-      arvados_controller_ssl:
+      arvados_controller_ssl.conf:
         enabled: true
         overwrite: true
         requires:
-          cmd: create-initial-cert-__CLUSTER__.__DOMAIN__-__CLUSTER__.__DOMAIN__
+          __CERT_REQUIRES__
         config:
           - server:
             - server_name: __CLUSTER__.__DOMAIN__
@@ -54,7 +56,8 @@ nginx:
               - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
               - proxy_set_header: 'X-External-Client $external_client'
             - include: snippets/ssl_hardening_default.conf
-            - include: snippets/__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+            - ssl_certificate: __CERT_PEM__
+            - ssl_certificate_key: __CERT_KEY__
             - access_log: /var/log/nginx/controller.__CLUSTER__.__DOMAIN__.access.log combined
             - error_log: /var/log/nginx/controller.__CLUSTER__.__DOMAIN__.error.log
             - client_max_body_size: 128m
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_download_configuration.sls
similarity index 55%
copy from tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepproxy_configuration.sls
copy to tools/salt-install/config_examples/multi_host/aws/pillars/nginx_download_configuration.sls
index fac97f3c6..9246fc11c 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_download_configuration.sls
@@ -5,41 +5,34 @@
 
 ### NGINX
 nginx:
-  ### SERVER
-  server:
-    config:
-      ### STREAMS
-      http:
-        upstream keepproxy_upstream:
-          - server: 'localhost:25107 fail_timeout=10s'
-
   servers:
     managed:
       ### DEFAULT
-      arvados_keepproxy_default:
+      arvados_download_default.conf:
         enabled: true
         overwrite: true
         config:
           - server:
-            - server_name: keep.__CLUSTER__.__DOMAIN__
+            - server_name: download.__CLUSTER__.__DOMAIN__
             - listen:
               - 80
             - location /:
               - return: '301 https://$host$request_uri'
 
-      arvados_keepproxy_ssl:
+      ### DOWNLOAD
+      arvados_download_ssl.conf:
         enabled: true
         overwrite: true
         requires:
-          cmd: create-initial-cert-keep.__CLUSTER__.__DOMAIN__-keep.__CLUSTER__.__DOMAIN__
+          __CERT_REQUIRES__
         config:
           - server:
-            - server_name: keep.__CLUSTER__.__DOMAIN__
+            - server_name: download.__CLUSTER__.__DOMAIN__
             - listen:
-              - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
+              - __KEEPWEB_EXT_SSL_PORT__ http2 ssl
             - index: index.html index.htm
             - location /:
-              - proxy_pass: 'http://keepproxy_upstream'
+              - proxy_pass: 'http://collections_downloads_upstream'
               - proxy_read_timeout: 90
               - proxy_connect_timeout: 90
               - proxy_redirect: 'off'
@@ -48,11 +41,11 @@ nginx:
               - proxy_set_header: 'X-Real-IP $remote_addr'
               - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
               - proxy_buffering: 'off'
-            - client_body_buffer_size: 64M
-            - client_max_body_size: 64M
+            - client_max_body_size: 0
             - proxy_http_version: '1.1'
             - proxy_request_buffering: 'off'
             - include: snippets/ssl_hardening_default.conf
-            - include: snippets/keep.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
-            - access_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.access.log combined
-            - error_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.error.log
+            - ssl_certificate: __CERT_PEM__
+            - ssl_certificate_key: __CERT_KEY__
+            - access_log: /var/log/nginx/download.__CLUSTER__.__DOMAIN__.access.log combined
+            - error_log: /var/log/nginx/download.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepproxy_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepproxy_configuration.sls
index fac97f3c6..2f00524f9 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepproxy_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepproxy_configuration.sls
@@ -16,7 +16,7 @@ nginx:
   servers:
     managed:
       ### DEFAULT
-      arvados_keepproxy_default:
+      arvados_keepproxy_default.conf:
         enabled: true
         overwrite: true
         config:
@@ -27,16 +27,16 @@ nginx:
             - location /:
               - return: '301 https://$host$request_uri'
 
-      arvados_keepproxy_ssl:
+      arvados_keepproxy_ssl.conf:
         enabled: true
         overwrite: true
         requires:
-          cmd: create-initial-cert-keep.__CLUSTER__.__DOMAIN__-keep.__CLUSTER__.__DOMAIN__
+          __CERT_REQUIRES__
         config:
           - server:
             - server_name: keep.__CLUSTER__.__DOMAIN__
             - listen:
-              - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
+              - __KEEP_EXT_SSL_PORT__ http2 ssl
             - index: index.html index.htm
             - location /:
               - proxy_pass: 'http://keepproxy_upstream'
@@ -53,6 +53,7 @@ nginx:
             - proxy_http_version: '1.1'
             - proxy_request_buffering: 'off'
             - include: snippets/ssl_hardening_default.conf
-            - include: snippets/keep.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+            - ssl_certificate: __CERT_PEM__
+            - ssl_certificate_key: __CERT_KEY__
             - access_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.access.log combined
             - error_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepweb_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepweb_configuration.sls
index e99295353..441140e80 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepweb_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_keepweb_configuration.sls
@@ -3,6 +3,7 @@
 #
 # SPDX-License-Identifier: AGPL-3.0
 
+# Keepweb upstream is common to both downloads and collections
 ### NGINX
 nginx:
   ### SERVER
@@ -12,77 +13,3 @@ nginx:
       http:
         upstream collections_downloads_upstream:
           - server: 'localhost:9002 fail_timeout=10s'
-
-  servers:
-    managed:
-      ### DEFAULT
-      arvados_collections_download_default:
-        enabled: true
-        overwrite: true
-        config:
-          - server:
-            - server_name: '~^((.*\.)?collections|download)\.__CLUSTER__\.__DOMAIN__'
-            - listen:
-              - 80
-            - location /:
-              - return: '301 https://$host$request_uri'
-
-      ### COLLECTIONS
-      arvados_collections_ssl:
-        enabled: true
-        overwrite: true
-        requires:
-          cmd: 'create-initial-cert-collections.__CLUSTER__.__DOMAIN__-collections.__CLUSTER__.__DOMAIN__+*.__CLUSTER__.__DOMAIN__'
-        config:
-          - server:
-            - server_name: '*.collections.__CLUSTER__.__DOMAIN__'
-            - listen:
-              - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
-            - index: index.html index.htm
-            - location /:
-              - proxy_pass: 'http://collections_downloads_upstream'
-              - proxy_read_timeout: 90
-              - proxy_connect_timeout: 90
-              - proxy_redirect: 'off'
-              - proxy_set_header: X-Forwarded-Proto https
-              - proxy_set_header: 'Host $http_host'
-              - proxy_set_header: 'X-Real-IP $remote_addr'
-              - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
-              - proxy_buffering: 'off'
-            - client_max_body_size: 0
-            - proxy_http_version: '1.1'
-            - proxy_request_buffering: 'off'
-            - include: snippets/ssl_hardening_default.conf
-            - include: snippets/collections.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
-            - access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
-            - error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
-
-      ### DOWNLOAD
-      arvados_download_ssl:
-        enabled: true
-        overwrite: true
-        requires:
-          cmd: create-initial-cert-download.__CLUSTER__.__DOMAIN__-download.__CLUSTER__.__DOMAIN__
-        config:
-          - server:
-            - server_name: download.__CLUSTER__.__DOMAIN__
-            - listen:
-              - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
-            - index: index.html index.htm
-            - location /:
-              - proxy_pass: 'http://collections_downloads_upstream'
-              - proxy_read_timeout: 90
-              - proxy_connect_timeout: 90
-              - proxy_redirect: 'off'
-              - proxy_set_header: X-Forwarded-Proto https
-              - proxy_set_header: 'Host $http_host'
-              - proxy_set_header: 'X-Real-IP $remote_addr'
-              - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
-              - proxy_buffering: 'off'
-            - client_max_body_size: 0
-            - proxy_http_version: '1.1'
-            - proxy_request_buffering: 'off'
-            - include: snippets/ssl_hardening_default.conf
-            - include: snippets/download.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
-            - access_log: /var/log/nginx/download.__CLUSTER__.__DOMAIN__.access.log combined
-            - error_log: /var/log/nginx/download.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_webshell_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_webshell_configuration.sls
index 49c86dd31..f2c88c83c 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_webshell_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_webshell_configuration.sls
@@ -17,7 +17,7 @@ nginx:
   ### SITES
   servers:
     managed:
-      arvados_webshell_default:
+      arvados_webshell_default.conf:
         enabled: true
         overwrite: true
         config:
@@ -28,16 +28,16 @@ nginx:
             - location /:
               - return: '301 https://$host$request_uri'
 
-      arvados_webshell_ssl:
+      arvados_webshell_ssl.conf:
         enabled: true
         overwrite: true
         requires:
-          cmd: create-initial-cert-webshell.__CLUSTER__.__DOMAIN__-webshell.__CLUSTER__.__DOMAIN__
+          __CERT_REQUIRES__
         config:
           - server:
             - server_name: webshell.__CLUSTER__.__DOMAIN__
             - listen:
-              - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
+              - __WEBSHELL_EXT_SSL_PORT__ http2 ssl
             - index: index.html index.htm
             - location /shell.__CLUSTER__.__DOMAIN__:
               - proxy_pass: 'http://webshell_upstream'
@@ -69,7 +69,8 @@ nginx:
                 - add_header: "'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type'"
 
             - include: snippets/ssl_hardening_default.conf
-            - include: snippets/webshell.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+            - ssl_certificate: __CERT_PEM__
+            - ssl_certificate_key: __CERT_KEY__
             - access_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.access.log combined
             - error_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.error.log
 
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_websocket_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_websocket_configuration.sls
index c9671cd0c..9658c620c 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_websocket_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_websocket_configuration.sls
@@ -16,7 +16,7 @@ nginx:
   servers:
     managed:
       ### DEFAULT
-      arvados_websocket_default:
+      arvados_websocket_default.conf:
         enabled: true
         overwrite: true
         config:
@@ -27,11 +27,11 @@ nginx:
             - location /:
               - return: '301 https://$host$request_uri'
 
-      arvados_websocket_ssl:
+      arvados_websocket_ssl.conf:
         enabled: true
         overwrite: true
         requires:
-          cmd: create-initial-cert-ws.__CLUSTER__.__DOMAIN__-ws.__CLUSTER__.__DOMAIN__
+          __CERT_REQUIRES__
         config:
           - server:
             - server_name: ws.__CLUSTER__.__DOMAIN__
@@ -54,6 +54,7 @@ nginx:
             - proxy_http_version: '1.1'
             - proxy_request_buffering: 'off'
             - include: snippets/ssl_hardening_default.conf
-            - include: snippets/ws.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+            - ssl_certificate: __CERT_PEM__
+            - ssl_certificate_key: __CERT_KEY__
             - access_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.access.log combined
             - error_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_workbench2_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_workbench2_configuration.sls
index bd4123539..a821b521f 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_workbench2_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_workbench2_configuration.sls
@@ -14,7 +14,7 @@ nginx:
   servers:
     managed:
       ### DEFAULT
-      arvados_workbench2_default:
+      arvados_workbench2_default.conf:
         enabled: true
         overwrite: true
         config:
@@ -25,11 +25,11 @@ nginx:
             - location /:
               - return: '301 https://$host$request_uri'
 
-      arvados_workbench2_ssl:
+      arvados_workbench2_ssl.conf:
         enabled: true
         overwrite: true
         requires:
-          cmd: create-initial-cert-workbench2.__CLUSTER__.__DOMAIN__-workbench2.__CLUSTER__.__DOMAIN__
+          __CERT_REQUIRES__
         config:
           - server:
             - server_name: workbench2.__CLUSTER__.__DOMAIN__
@@ -44,6 +44,7 @@ nginx:
             - location /config.json:
               - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
             - include: snippets/ssl_hardening_default.conf
-            - include: snippets/workbench2.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+            - ssl_certificate: __CERT_PEM__
+            - ssl_certificate_key: __CERT_KEY__
             - access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
             - error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_workbench_configuration.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_workbench_configuration.sls
index ec28b98c6..32904a12b 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_workbench_configuration.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/nginx_workbench_configuration.sls
@@ -23,7 +23,7 @@ nginx:
   servers:
     managed:
       ### DEFAULT
-      arvados_workbench_default:
+      arvados_workbench_default.conf:
         enabled: true
         overwrite: true
         config:
@@ -34,11 +34,11 @@ nginx:
             - location /:
               - return: '301 https://$host$request_uri'
 
-      arvados_workbench_ssl:
+      arvados_workbench_ssl.conf:
         enabled: true
         overwrite: true
         requires:
-          cmd: create-initial-cert-workbench.__CLUSTER__.__DOMAIN__-workbench.__CLUSTER__.__DOMAIN__
+          __CERT_REQUIRES__
         config:
           - server:
             - server_name: workbench.__CLUSTER__.__DOMAIN__
@@ -55,7 +55,8 @@ nginx:
               - proxy_set_header: 'X-Real-IP $remote_addr'
               - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
             - include: snippets/ssl_hardening_default.conf
-            - include: snippets/workbench.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+            - ssl_certificate: __CERT_PEM__
+            - ssl_certificate_key: __CERT_KEY__
             - access_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.access.log combined
             - error_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.error.log
 

-----------------------------------------------------------------------


hooks/post-receive
--

More information about the arvados-commits mailing list