[ARVADOS] updated: 2.1.0-470-g708544a82
Git user
git at public.arvados.org
Wed Mar 31 12:40:50 UTC 2021
Summary of changes:
tools/terraform/api_instance.tf | 8 ++++----
tools/terraform/db_instance.tf | 5 +++--
tools/terraform/keepproxy_instance.tf | 16 ++++++++--------
tools/terraform/keepstore_instance.tf | 4 ++--
tools/terraform/shell_instance.tf | 6 +++---
tools/terraform/workbench_instance.tf | 8 ++++----
6 files changed, 24 insertions(+), 23 deletions(-)
discards 375a1363be4b94b9682c0824b762033d58d608eb (commit)
via 708544a82c4524f7ca7ca899ef272453c754ef97 (commit)
This update added new revisions after undoing existing revisions. That is
to say, the old revision is not a strict subset of the new revision. This
situation occurs when you --force push a change and generate a repository
containing something like this:
* -- * -- B -- O -- O -- O (375a1363be4b94b9682c0824b762033d58d608eb)
\
N -- N -- N (708544a82c4524f7ca7ca899ef272453c754ef97)
When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 708544a82c4524f7ca7ca899ef272453c754ef97
Author: Javier Bértoli <jbertoli at curii.com>
Date: Wed Mar 24 10:07:01 2021 -0300
feat(deployment): add initial terraform code for AWS deployment
refs #17450
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli at curii.com>
diff --git a/doc/install/salt-multi-host.html.textile.liquid b/doc/install/salt-multi-host.html.textile.liquid
index 709c32e2a..0bba34427 100644
--- a/doc/install/salt-multi-host.html.textile.liquid
+++ b/doc/install/salt-multi-host.html.textile.liquid
@@ -9,90 +9,177 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-# "Install Saltstack":#saltstack
-# "Install dependencies":#dependencies
-# "Install Arvados using Saltstack":#saltstack
-# "DNS configuration":#final_steps
+# "Hosts preparation":#hosts_preparation
+## "Hosts setup using terraform (experimental)":#hosts_setup_using_terraform
+# "Multi host install using the provision.sh script":#multi_host
+# "Choose the desired configuration":#choose_configuration
+## "Multi host / multi hostname":#multi_host_multi_hostnames
+## "Multi host / multiple hostnames (Alternative configuration)":#multi_host_multiple_hostnames
+## "Further customization of the installation (modifying the salt pillars and states)":#further_customization
+# "Run the provision.sh script":#run_provision_script
+# "Final configuration steps":#final_steps
+## "Install the CA root certificate (required in both alternatives)":#ca_root_certificate
+## "DNS configuration (multi host / multiple hostnames)":#multi_host_multiple_hostnames_dns_configuration
# "Initial user and login":#initial_user
+# "Test the installed cluster running a simple workflow":#test_install
-h2(#saltstack). Install Saltstack
+h2(#hosts_preparation). Hosts preparation
-If you already have a Saltstack environment you can skip this section.
+In order to run Arvados on a multi-host installation, there are a few requirements that your infrastructure has to fulfill.
-The simplest way to get Salt up and running on a node is to use the bootstrap script they provide:
+These instructions explain how to setup a multi-host environment that is suitable for production use of Arvados.
+We suggest distributing the Arvados components in the following way, given at least 6 hosts:
+
+# Database server:
+## postgresql server
+# API node:
+## arvados api server
+## arvados controller
+## arvados websocket
+## arvados cloud dispatcher
+# WORKBENCH node:
+## arvados workbench
+## arvados workbench2
+# KEEPPROXY node:
+## arvados keepproxy
+## arvados keepweb
+# KEEPSTOREs (at least 2)
+## arvados keepstore
+# SHELL node (optional):
+## arvados shell
+
+Note that these hosts can be virtual machines in your infrastructure and they don't need to be real machines.
+
+h3(#hosts_setup_using_terraform). Hosts setup using terraform (experimental)
+
+
+
+
+
+
+
+
+
+h2(#multi_host). Multi host install using the provision.sh script
+
+<b>NOTE: The multi host installation is not recommended for production use.</b>
+
+This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+
+This procedure will install all the main Arvados components to get you up and running in a multi host. The whole installation procedure takes somewhere between 15 to 60 minutes, depending on the host resources and its network bandwidth. As a reference, on a virtual machine with 1 core and 1 GB RAM, it takes ~25 minutes to do the initial install.
+
+We suggest you to use the @provision.sh@ script to deploy Arvados, which is implemented with the @arvados-formula@ in a Saltstack master-less setup. After setting up a few variables in a config file (next step), you'll be ready to run it and get Arvados deployed.
+
+h2(#choose_configuration). Choose the desired configuration
+
+For documentation's sake, we will use the cluster name <i>arva2</i> and the domain <i>arv.local</i>. If you don't change them as required in the next steps, installation won't proceed.
+
+Arvados' multi host installation can be done in two fashions:
+
+* Using a multi hostname, asigning <i>a different port (other than 443) for each user-facing service</i>: This choice is easier to setup, but the user will need to know the port/s for the different services she wants to connect to.
+* Using multiple hostnames on the same IP: this setup involves a few extra steps but each service will have a meaningful hostname so it will make easier to access them later.
+
+Once you decide which of these choices you prefer, copy one the two example configuration files and directory, and edit them to suit your needs.
+
+h3(#multi_host_multi_hostnames). Multi host / multi hostname
<notextile>
-<pre><code>curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
-sudo sh /tmp/bootstrap_salt.sh -XUdfP -x python3
+<pre><code>cp local.params.example.multi_host_multi_hostname local.params
+cp -r config_examples/multi_host/multi_hostname local_config_dir
</code></pre>
</notextile>
-For more information check "Saltstack's documentation":https://docs.saltstack.com/en/latest/topics/installation/index.html
+Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_PORT, *_TOKEN</b> and <b>*KEY</b> variables.
-h2(#dependencies). Install dependencies
+h3(#multi_host_multiple_hostnames). Multi host / multiple hostnames (Alternative configuration)
+<notextile>
+<pre><code>cp local.params.example.multi_host_multiple_hostnames local.params
+cp -r config_examples/multi_host/multiple_hostnames local_config_dir
+</code></pre>
+</notextile>
-Arvados depends in a few applications and packages (postgresql, nginx+passenger, ruby) that can also be installed using their respective Saltstack formulas.
+Edit the variables in the <i>local.params</i> file.
-The formulas we use are:
+## "Further customization of the installation (modifying the salt pillars and states)":#further_customization
-* "postgres":https://github.com/saltstack-formulas/postgres-formula.git
-* "nginx":https://github.com/saltstack-formulas/nginx-formula.git
-* "docker":https://github.com/saltstack-formulas/docker-formula.git
-* "locale":https://github.com/saltstack-formulas/locale-formula.git
-* "letsencrypt":https://github.com/saltstack-formulas/letsencrypt-formula.git
+If you want or need further customization, you can edit the Saltstack pillars and states files. Pay particular attention to the <i>pillars/arvados.sls</i> one. Any extra <i>state</i> file you add under <i>local_config_dir/states</i> will be added to the salt run and applied to the host.
-There are example Salt pillar files for each of those formulas in the "arvados-formula's test/salt/pillar/examples":https://github.com/arvados/arvados-formula/tree/master/test/salt/pillar/examples directory. As they are, they allow you to get all the main Arvados components up and running.
+h2(#run_provision_script). Run the provision.sh script
-h2(#saltstack). Install Arvados using Saltstack
+When you finished customizing the configuration, you are ready to copy the files to the host (if needed) and run the @provision.sh@ script:
-This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+<notextile>
+<pre><code>scp -r provision.sh local* user at host:
+ssh user at host sudo provision.sh
+</code></pre>
+</notextile>
-The Arvados formula we maintain is located in Arvados' Github account and should be considered the canonical place to download its most up-to-date version:
+and wait for it to finish.
-* "arvados-formula":https://github.com/arvados/arvados-formula.git
+If everything goes OK, you'll get some final lines stating something like:
-As the Saltstack's community keeps a "repository of formulas":https://github.com/saltstack-formulas/ in Github, we also provide
+<notextile>
+<pre><code>arvados: Succeeded: 109 (changed=9)
+arvados: Failed: 0
+</code></pre>
+</notextile>
+
+h2(#final_steps). Final configuration steps
+
+Once the deployment went OK, you'll need to perform a few extra steps in your local browser/host to access the cluster.
-* "a copy of the formula":https://github.com/saltstack-formulas/arvados-formula.git
+h3(#ca_root_certificate). Install the CA root certificate (required in both alternatives)
-there, and do our best effort to keep it in sync with ours.
+Arvados uses SSL to encrypt communications. Its UI uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
-For those familiar with Saltstack, the process to get Arvados deployed is similar to any other formula:
+For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so you can add it to your workstation.
-1. Fork/copy the formula to your Salt master host.
-2. Edit the Arvados, nginx, postgres, locale and docker pillars to match your desired configuration.
-3. Run a @state.apply@ to get it deployed.
+Installing the root certificate into your web browser will prevent security errors when accessing Arvados services with your web browser.
-h2(#final_steps). DNS configuration
+# Go to the certificate manager in your browser.
+#* In Chrome, this can be found under "Settings → Advanced → Manage Certificates" or by entering @chrome://settings/certificates@ in the URL bar.
+#* In Firefox, this can be found under "Preferences → Privacy & Security" or entering @about:preferences#privacy@ in the URL bar and then choosing "View Certificates...".
+# Select the "Authorities" tab, then press the "Import" button. Choose @arvados-snakeoil-ca.pem@
-After the setup is done, you need to set up your DNS to be able to access the cluster's nodes.
+The certificate will be added under the "Arvados Formula".
-The simplest way to do this is to add entries in the @/etc/hosts@ file of every host:
+To access your Arvados instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage.
+
+* On Debian/Ubuntu:
<notextile>
-<pre><code>export CLUSTER="arva2"
-export DOMAIN="arv.local"
+<pre><code>cp arvados-root-cert.pem /usr/local/share/ca-certificates/
+/usr/sbin/update-ca-certificates
+</code></pre>
+</notextile>
+
+* On CentOS:
-echo A.B.C.a api ${CLUSTER}.${DOMAIN} api.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.b keep keep.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.c keep0 keep0.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.d collections collections.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.e download download.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.f ws ws.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.g workbench workbench.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.h workbench2 workbench2.${CLUSTER}.${DOMAIN}" >> /etc/hosts
+<notextile>
+<pre><code>cp arvados-root-cert.pem /etc/pki/ca-trust/source/anchors/
+/usr/bin/update-ca-trust
</code></pre>
</notextile>
-Replacing in each case de @A.B.C.x@ IP with the corresponding IP of the node.
+h3(#multi_host_multiple_hostnames_dns_configuration). DNS configuration (multi host / multiple hostnames)
+
+When using multiple hostnames, after the setup is done, you need to set up your DNS to be able to access the cluster.
-If your infrastructure uses another DNS service setup, add the corresponding entries accordingly.
+If you don't have access to the domain's DNS to add the required entries, the simplest way to do it is to edit your @/etc/hosts@ file (as root):
-h2(#initial_user). Initial user and login
+<notextile>
+<pre><code>export CLUSTER="arva2"
+export DOMAIN="arv.local"
+export HOST_IP="127.0.0.2" # This is valid either if installing in your computer directly
+ # or in a Vagrant VM. If you're installing it on a remote host
+ # just change the IP to match that of the host.
+echo "${HOST_IP} api keep keep0 collections download ws workbench workbench2 ${CLUSTER}.${DOMAIN} api.${CLUSTER}.${DOMAIN} keep.${CLUSTER}.${DOMAIN} keep0.${CLUSTER}.${DOMAIN} collections.${CLUSTER}.${DOMAIN} download.${CLUSTER}.${DOMAIN} ws.${CLUSTER}.${DOMAIN} workbench.${CLUSTER}.${DOMAIN} workbench2.${CLUSTER}.${DOMAIN}" >> /etc/hosts
+</code></pre>
+</notextile>
-At this point you should be able to log into the Arvados cluster.
+h2(#initial_user). Initial user and login
-If you did not change the defaults, the initial URL will be:
+At this point you should be able to log into the Arvados cluster. The initial URL will be:
* https://workbench.arva2.arv.local
@@ -102,8 +189,100 @@ or, in general, the url format will be:
By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.
-Assuming you didn't change the defaults, the initial credentials are:
+Assuming you didn't change these values in the @local.params@ file, the initial credentials are:
* User: 'admin'
* Password: 'password'
* Email: 'admin at arva2.arv.local'
+
+h2(#test_install). Test the installed cluster running a simple workflow
+
+The @provision.sh@ script saves a simple example test workflow in the @/tmp/cluster_tests@ directory in the node. If you want to run it, just ssh to the node, change to that directory and run:
+
+<notextile>
+<pre><code>cd /tmp/cluster_tests
+./run-test.sh
+</code></pre>
+</notextile>
+
+It will create a test user (by default, the same one as the admin user), upload a small workflow and run it. If everything goes OK, the output should similar to this (some output was shortened for clarity):
+
+<notextile>
+<pre><code>Creating Arvados Standard Docker Images project
+Arvados project uuid is 'arva2-j7d0g-0prd8cjlk6kfl7y'
+{
+ ...
+ "uuid":"arva2-o0j2j-n4zu4cak5iifq2a",
+ "owner_uuid":"arva2-tpzed-000000000000000",
+ ...
+}
+Uploading arvados/jobs' docker image to the project
+2.1.1: Pulling from arvados/jobs
+8559a31e96f4: Pulling fs layer
+...
+Status: Downloaded newer image for arvados/jobs:2.1.1
+docker.io/arvados/jobs:2.1.1
+2020-11-23 21:43:39 arvados.arv_put[32678] INFO: Creating new cache file at /home/vagrant/.cache/arvados/arv-put/c59256eda1829281424c80f588c7cc4d
+2020-11-23 21:43:46 arvados.arv_put[32678] INFO: Collection saved as 'Docker image arvados jobs:2.1.1 sha256:0dd50'
+arva2-4zz18-1u5pvbld7cvxuy2
+Creating initial user ('admin')
+Setting up user ('admin')
+{
+ "items":[
+ {
+ ...
+ "owner_uuid":"arva2-tpzed-000000000000000",
+ ...
+ "uuid":"arva2-o0j2j-1ownrdne0ok9iox"
+ },
+ {
+ ...
+ "owner_uuid":"arva2-tpzed-000000000000000",
+ ...
+ "uuid":"arva2-o0j2j-1zbeyhcwxc1tvb7"
+ },
+ {
+ ...
+ "email":"admin at arva2.arv.local",
+ ...
+ "owner_uuid":"arva2-tpzed-000000000000000",
+ ...
+ "username":"admin",
+ "uuid":"arva2-tpzed-3wrm93zmzpshrq2",
+ ...
+ }
+ ],
+ "kind":"arvados#HashList"
+}
+Activating user 'admin'
+{
+ ...
+ "email":"admin at arva2.arv.local",
+ ...
+ "username":"admin",
+ "uuid":"arva2-tpzed-3wrm93zmzpshrq2",
+ ...
+}
+Running test CWL workflow
+INFO /usr/bin/cwl-runner 2.1.1, arvados-python-client 2.1.1, cwltool 3.0.20200807132242
+INFO Resolved 'hasher-workflow.cwl' to 'file:///tmp/cluster_tests/hasher-workflow.cwl'
+...
+INFO Using cluster arva2 (https://arva2.arv.local:8443/)
+INFO Upload local files: "test.txt"
+INFO Uploaded to ea34d971b71d5536b4f6b7d6c69dc7f6+50 (arva2-4zz18-c8uvwqdry4r8jao)
+INFO Using collection cache size 256 MiB
+INFO [container hasher-workflow.cwl] submitted container_request arva2-xvhdp-v1bkywd58gyocwm
+INFO [container hasher-workflow.cwl] arva2-xvhdp-v1bkywd58gyocwm is Final
+INFO Overall process status is success
+INFO Final output collection d6c69a88147dde9d52a418d50ef788df+123
+{
+ "hasher_out": {
+ "basename": "hasher3.md5sum.txt",
+ "class": "File",
+ "location": "keep:d6c69a88147dde9d52a418d50ef788df+123/hasher3.md5sum.txt",
+ "size": 95
+ }
+}
+INFO Final process status is success
+</code></pre>
+</notextile>
diff --git a/doc/install/salt-multi-host.html.textile.liquid b/doc/install/terraform-multi-host.html.textile.liquid
similarity index 99%
copy from doc/install/salt-multi-host.html.textile.liquid
copy to doc/install/terraform-multi-host.html.textile.liquid
index 709c32e2a..c7f6fe554 100644
--- a/doc/install/salt-multi-host.html.textile.liquid
+++ b/doc/install/terraform-multi-host.html.textile.liquid
@@ -1,7 +1,7 @@
---
layout: default
navsection: installguide
-title: Multi host Arvados
+title: Terraform code to create the
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
diff --git a/tools/terraform/_provider.tf b/tools/terraform/_provider.tf
new file mode 100644
index 000000000..ce3eb03ec
--- /dev/null
+++ b/tools/terraform/_provider.tf
@@ -0,0 +1,16 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+provider "aws" {
+ region = var.aws_region
+ profile = var.aws_profile
+}
+
+terraform {
+ required_providers {
+ aws = {
+ version = "~> 3.31"
+ }
+ }
+}
diff --git a/tools/terraform/_user_data.sh b/tools/terraform/_user_data.sh
new file mode 100644
index 000000000..634420e3b
--- /dev/null
+++ b/tools/terraform/_user_data.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+apt update
+apt install -y curl xfsprogs
+
+# Find the unformatted disks with lsblk, getting those with no format ($2)
+# and which name has no number (for xv*) or 'p?' (for nmve*)
+UNFORMATTED_DISK="/dev/$(lsblk -o NAME,FSTYPE -dsn | awk '/xv.*[0-9].*/ || /nvme.*p.*/ { next; } $2 == "" {print $1}')"
+if ! grep -q '/data' /etc/fstab && [ "$${UNFORMATTED_DISK}" != "/dev/" ]; then
+ mkdir -p /data
+ mkfs.xfs -f $${UNFORMATTED_DISK} || exit 1
+ BLKID=$(blkid |grep xfs|awk '{print $2}')
+
+ echo "# Added by curii_run_once script" >> /etc/fstab
+ echo "$${BLKID} /data xfs auto 0 0" >> /etc/fstab
+ mount /data || exit 1
+fi
diff --git a/tools/terraform/api_iam_role.tf b/tools/terraform/api_iam_role.tf
new file mode 100644
index 000000000..118ac8c2f
--- /dev/null
+++ b/tools/terraform/api_iam_role.tf
@@ -0,0 +1,27 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# Assume role for the instance
+resource "aws_iam_role" "api_iam_role" {
+ name = "${var.cluster}-api-iam-role"
+ assume_role_policy = templatefile("${path.module}/iam_policy_assume_role.json", {})
+}
+
+# Associate the dispatcher policy to the role
+resource "aws_iam_role_policy_attachment" "api_dispatcher_policies_attachment" {
+ role = aws_iam_role.api_iam_role.name
+ policy_arn = aws_iam_policy.dispatcher_iam_policy.arn
+}
+
+# Associate letsencrypt modification policy to the role
+resource "aws_iam_role_policy_attachment" "api_letsencrypt_route53_policies_attachment" {
+ role = aws_iam_role.api_iam_role.name
+ policy_arn = aws_iam_policy.letsencrypt_route53_iam_policy.arn
+}
+
+# Add the role to the instance profile
+resource "aws_iam_instance_profile" "api_instance_profile" {
+ name = "api_instance_profile"
+ role = "${var.cluster}-api-iam-role"
+}
diff --git a/tools/terraform/api_iam_role_outputs.tf b/tools/terraform/api_iam_role_outputs.tf
new file mode 100644
index 000000000..33fa9f8f4
--- /dev/null
+++ b/tools/terraform/api_iam_role_outputs.tf
@@ -0,0 +1,22 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "dispatcher_iam_policy_id" {
+ value = aws_iam_policy.dispatcher_iam_policy.id
+}
+output "dispatcher_iam_policy_arn" {
+ value = aws_iam_policy.dispatcher_iam_policy.arn
+}
+output "letsencrypt_route53_iam_policy_id" {
+ value = aws_iam_policy.letsencrypt_route53_iam_policy.id
+}
+output "letsencrypt_route53_iam_policy_arn" {
+ value = aws_iam_policy.letsencrypt_route53_iam_policy.arn
+}
+output "api_iam_role_arn" {
+ value = aws_iam_role.api_iam_role.arn
+}
+output "api_iam_role_id" {
+ value = aws_iam_role.api_iam_role.id
+}
diff --git a/tools/terraform/api_instance.tf b/tools/terraform/api_instance.tf
new file mode 100644
index 000000000..285cf614b
--- /dev/null
+++ b/tools/terraform/api_instance.tf
@@ -0,0 +1,96 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+module "api" {
+ source = "terraform-aws-modules/ec2-instance/aws"
+ version = "~> 2.17.0"
+
+ name = "${var.cluster}-api"
+ instance_count = 1
+
+ iam_instance_profile = "api_instance_profile"
+ ami = try(var.instance_ami["api"], var.instance_ami["default"])
+ instance_type = try(var.instance_type["api"], var.instance_type["default"])
+ key_name = var.key_name
+ monitoring = true
+
+ tags = merge({"Name": "${var.cluster}-api",
+ "OsType": "LINUX"}, local.resource_tags)
+ volume_tags = merge({"Name": "${var.cluster}-api"}, local.resource_tags)
+
+ network_interface = [{
+ device_index = 0,
+ network_interface_id = aws_network_interface.api.id,
+ }]
+
+ # associate_public_ip_address = false
+ ebs_optimized = true
+ user_data = templatefile("_user_data.sh", {})
+
+ root_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = var.root_bd_size,
+ delete_on_termination = true,
+ }]
+ ebs_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = try(var.data_bd_size["api"], var.data_bd_size["default"])
+ delete_on_termination = true,
+ device_name = "xvdh",
+ }]
+}
+
+resource "aws_eip" "cluster_api_public_ip" {
+ vpc = true
+ instance = module.api.id[0]
+ network_interface = aws_network_interface.api.id
+ tags = merge({"Name": "${var.cluster}-api-ip"},local.resource_tags)
+}
+
+resource "aws_network_interface" "api" {
+ subnet_id = var.manage_vpc ? module.vpc.0.public_subnets[0] : var.public_subnets_ids[0]
+ # private_ips = [cidrhost(var.vpc_subnet_cidrs[0], var.host_number["api"])]
+ security_groups = [
+ local.ssh_sg,
+ local.http_sg,
+ local.https_sg,
+ ]
+ tags = merge({"Name": "${var.cluster}-api"}, local.resource_tags)
+}
+
+## Public A RRs
+module "api_route53_public_records_A" {
+ source = "./modules/aws/route53/records/a"
+ zone_id = module.r53_zone_public.id
+
+ zone_records_A = {
+ (var.r53_domain_name) = {
+ ttl = "300",
+ records = [aws_eip.cluster_api_public_ip.public_ip]
+ },
+ "ws" = {
+ ttl = "300",
+ records = [aws_eip.cluster_api_public_ip.public_ip]
+ }
+ }
+}
+
+## Private A RRs
+module "api_route53_private_records_A" {
+ source = "./modules/aws/route53/records/a"
+ zone_id = module.r53_zone_private.id
+
+ zone_records_A = {
+ (var.r53_domain_name) = {
+ ttl = "300",
+ records = aws_network_interface.api.private_ips
+ },
+ "ws" = {
+ ttl = "300",
+ records = aws_network_interface.api.private_ips
+ }
+ }
+}
diff --git a/tools/terraform/api_instance_outputs.tf b/tools/terraform/api_instance_outputs.tf
new file mode 100644
index 000000000..ba6dc6b70
--- /dev/null
+++ b/tools/terraform/api_instance_outputs.tf
@@ -0,0 +1,19 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "api_id" {
+ value = module.api.id
+}
+output "api_private_dns_names" {
+ value = aws_network_interface.api.private_dns_name
+}
+output "api_private_ip" {
+ value = module.api.private_ip
+}
+output "api_private_eni_id" {
+ value = aws_network_interface.api.id
+}
+output "api_public_ip" {
+ value = aws_eip.cluster_api_public_ip.public_ip
+}
diff --git a/tools/terraform/db_instance.tf b/tools/terraform/db_instance.tf
new file mode 100644
index 000000000..0377f484f
--- /dev/null
+++ b/tools/terraform/db_instance.tf
@@ -0,0 +1,70 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+module "database" {
+ source = "terraform-aws-modules/ec2-instance/aws"
+ version = "~> 2.17.0"
+
+ name = "${var.cluster}-database"
+ instance_count = 1
+
+ ami = try(var.instance_ami["database"], var.instance_ami["default"])
+ instance_type = try(var.instance_type["database"], var.instance_type["default"])
+ key_name = var.key_name
+ monitoring = true
+
+ tags = merge({"Name": "${var.cluster}-database",
+ "OsType": "LINUX"}, local.resource_tags)
+ volume_tags = merge({"Name": "${var.cluster}-database"}, local.resource_tags)
+
+ network_interface = [{
+ device_index = 0,
+ network_interface_id = aws_network_interface.database.id,
+ }]
+
+ ebs_optimized = true
+ user_data = templatefile("_user_data.sh", {})
+
+ root_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = var.root_bd_size,
+ delete_on_termination = true,
+ }]
+ ebs_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = try(var.data_bd_size["database"], var.data_bd_size["default"])
+ delete_on_termination = true,
+ device_name = "xvdh",
+ }]
+}
+
+resource "aws_network_interface" "database" {
+ subnet_id = var.manage_vpc ? module.vpc.0.private_subnets[0] : var.private_subnets_ids[0]
+ # private_ips = [cidrhost(var.vpc_subnet_cidrs[0], var.host_number["database"])]
+ security_groups = [
+ local.ssh_sg,
+ local.postgresql_sg,
+ ]
+ tags = merge({"Name": "${var.cluster}-database"}, local.resource_tags)
+}
+
+## Private A RRs
+module "database_route53_private_records_A" {
+ source = "./modules/aws/route53/records/a"
+ zone_id = module.r53_zone_private.id
+
+ zone_records_A = {
+ "database" = {
+ ttl = "300",
+ records = aws_network_interface.database.private_ips
+
+ },
+ "db" = {
+ ttl = "300",
+ records = aws_network_interface.database.private_ips
+ }
+ }
+}
diff --git a/tools/terraform/db_instance_outputs.tf b/tools/terraform/db_instance_outputs.tf
new file mode 100644
index 000000000..e03cdcaa6
--- /dev/null
+++ b/tools/terraform/db_instance_outputs.tf
@@ -0,0 +1,16 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "database_id" {
+ value = module.database.id
+}
+output "database_private_dns_names" {
+ value = module.database.private_dns
+}
+output "database_private_ip" {
+ value = module.database.private_ip
+}
+output "database_private_eni_id" {
+ value = aws_network_interface.database.id
+}
diff --git a/tools/terraform/dispatcher_iam_policy.json b/tools/terraform/dispatcher_iam_policy.json
new file mode 100644
index 000000000..7022c6579
--- /dev/null
+++ b/tools/terraform/dispatcher_iam_policy.json
@@ -0,0 +1,14 @@
+{
+ "Version": "2012-10-17",
+ "Id": "${cluster}-api-dispatcher-role policy",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": [
+ "ec2:*",
+ "kms:*"
+ ],
+ "Resource": "*"
+ }
+ ]
+}
diff --git a/tools/terraform/dispatcher_iam_policy.tf b/tools/terraform/dispatcher_iam_policy.tf
new file mode 100644
index 000000000..c8202842c
--- /dev/null
+++ b/tools/terraform/dispatcher_iam_policy.tf
@@ -0,0 +1,11 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_iam_policy" "dispatcher_iam_policy" {
+ name = "${var.cluster}-dispatcher-iam-role-policy"
+ description = "Policy to allow API to launch compute instances"
+ policy = templatefile("${path.module}/dispatcher_iam_policy.json", {
+ "cluster" = var.cluster
+ })
+}
diff --git a/tools/terraform/general_outputs.tf b/tools/terraform/general_outputs.tf
new file mode 100644
index 000000000..a46092dcf
--- /dev/null
+++ b/tools/terraform/general_outputs.tf
@@ -0,0 +1,23 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+### GENERAL
+output "aws_region" {
+ value = var.aws_region
+}
+output "aws_profile" {
+ value = var.aws_profile
+}
+output "environment" {
+ value = var.environment
+}
+output "namespace" {
+ value = var.namespace
+}
+output "resource_tags" {
+ value = local.resource_tags
+}
+output "ami" {
+ value = var.ami
+}
diff --git a/tools/terraform/iam_policy_assume_role.json b/tools/terraform/iam_policy_assume_role.json
new file mode 100644
index 000000000..e3b695d98
--- /dev/null
+++ b/tools/terraform/iam_policy_assume_role.json
@@ -0,0 +1,13 @@
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Action": "sts:AssumeRole",
+ "Principal": {
+ "Service": "ec2.amazonaws.com"
+ },
+ "Effect": "Allow",
+ "Sid": ""
+ }
+ ]
+}
diff --git a/tools/terraform/keepproxy_iam_role.tf b/tools/terraform/keepproxy_iam_role.tf
new file mode 100644
index 000000000..838d1f5ea
--- /dev/null
+++ b/tools/terraform/keepproxy_iam_role.tf
@@ -0,0 +1,21 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# Assume role for the instance
+resource "aws_iam_role" "keepproxy_iam_role" {
+ name = "${var.cluster}-keepproxy-iam-role"
+ assume_role_policy = templatefile("${path.module}/iam_policy_assume_role.json", {})
+}
+
+# Associate letsencrypt modification policy to the role
+resource "aws_iam_role_policy_attachment" "keepproxy_letsencrypt_route53_policies_attachment" {
+ role = aws_iam_role.keepproxy_iam_role.name
+ policy_arn = aws_iam_policy.letsencrypt_route53_iam_policy.arn
+}
+
+# Add the role to the instance profile
+resource "aws_iam_instance_profile" "keepproxy_instance_profile" {
+ name = "keepproxy_instance_profile"
+ role = "${var.cluster}-keepproxy-iam-role"
+}
diff --git a/tools/terraform/keepproxy_iam_role_outputs.tf b/tools/terraform/keepproxy_iam_role_outputs.tf
new file mode 100644
index 000000000..dc20142c6
--- /dev/null
+++ b/tools/terraform/keepproxy_iam_role_outputs.tf
@@ -0,0 +1,10 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "keepproxy_iam_role_arn" {
+ value = aws_iam_role.keepproxy_iam_role.arn
+}
+output "keepproxy_iam_role_id" {
+ value = aws_iam_role.keepproxy_iam_role.id
+}
diff --git a/tools/terraform/keepproxy_instance.tf b/tools/terraform/keepproxy_instance.tf
new file mode 100644
index 000000000..8fab4bb10
--- /dev/null
+++ b/tools/terraform/keepproxy_instance.tf
@@ -0,0 +1,112 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+module "keepproxy" {
+ source = "terraform-aws-modules/ec2-instance/aws"
+ version = "~> 2.16"
+
+ name = "${var.cluster}-keepproxy"
+ instance_count = 1
+
+ iam_instance_profile = "keepproxy_instance_profile"
+ ami = try(var.instance_ami["keepproxy"], var.instance_ami["default"])
+ instance_type = try(var.instance_type["keepproxy"], var.instance_type["default"])
+ key_name = var.key_name
+ monitoring = true
+
+ tags = merge({"Name": "${var.cluster}-keepproxy",
+ "OsType": "LINUX"}, local.resource_tags)
+ volume_tags = merge({"Name": "${var.cluster}-keepproxy"}, local.resource_tags)
+
+ network_interface = [{
+ device_index = 0,
+ network_interface_id = aws_network_interface.keepproxy.id,
+ }]
+
+ # associate_public_ip_address = false
+ ebs_optimized = true
+ user_data = templatefile("_user_data.sh", {})
+
+ root_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = var.root_bd_size,
+ delete_on_termination = true,
+ }]
+ ebs_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = try(var.data_bd_size["keeproxy"], var.data_bd_size["default"])
+ delete_on_termination = true,
+ device_name = "xvdh",
+ }]
+}
+
+resource "aws_eip" "cluster_keepproxy_public_ip" {
+ vpc = true
+ instance = module.keepproxy.id[0]
+ network_interface = aws_network_interface.keepproxy.id
+ tags = merge({"Name": "${var.cluster}-keepproxy-ip"},local.resource_tags)
+}
+
+resource "aws_network_interface" "keepproxy" {
+ subnet_id = var.manage_vpc ? module.vpc.0.public_subnets[0] : var.public_subnets_ids[0]
+ # private_ips = [cidrhost(var.vpc_subnet_cidrs[0], var.host_number["keepproxy"])]
+ security_groups = [
+ local.ssh_sg,
+ local.http_sg,
+ local.https_sg,
+ ]
+ tags = merge({"Name": "${var.cluster}-keepproxy"}, local.resource_tags)
+}
+
+## Public A RRs
+module "keepproxy_route53_public_records_A" {
+ source = "./modules/aws/route53/records/a"
+ zone_id = module.r53_zone_public.id
+
+ zone_records_A = {
+ "keep" = {
+ ttl = "300",
+ records = [aws_eip.cluster_keepproxy_public_ip.public_ip]
+ },
+ "collections" = {
+ ttl = "300",
+ records = [aws_eip.cluster_keepproxy_public_ip.public_ip]
+ },
+ "*.collections" = {
+ ttl = "300",
+ records = [aws_eip.cluster_keepproxy_public_ip.public_ip]
+ },
+ "download" = {
+ ttl = "300",
+ records = [aws_eip.cluster_keepproxy_public_ip.public_ip]
+ }
+ }
+}
+
+## Private A RRs
+module "keepproxy_route53_private_records_A" {
+ source = "./modules/aws/route53/records/a"
+ zone_id = module.r53_zone_private.id
+
+ zone_records_A = {
+ "keep" = {
+ ttl = "300",
+ records = aws_network_interface.keepproxy.private_ips
+ },
+ "collections" = {
+ ttl = "300",
+ records = aws_network_interface.keepproxy.private_ips
+ },
+ "*.collections" = {
+ ttl = "300",
+ records = aws_network_interface.keepproxy.private_ips
+ },
+ "download" = {
+ ttl = "300",
+ records = aws_network_interface.keepproxy.private_ips
+ }
+ }
+}
diff --git a/tools/terraform/keepproxy_instance_outputs.tf b/tools/terraform/keepproxy_instance_outputs.tf
new file mode 100644
index 000000000..4578a90fd
--- /dev/null
+++ b/tools/terraform/keepproxy_instance_outputs.tf
@@ -0,0 +1,19 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "keepproxy_id" {
+ value = module.keepproxy.id
+}
+output "keepproxy_private_dns_names" {
+ value = module.keepproxy.private_dns
+}
+output "keepproxy_private_ip" {
+ value = module.keepproxy.private_ip
+}
+output "keepproxy_private_eni_id" {
+ value = aws_network_interface.keepproxy.id
+}
+output "keepproxy_public_ip" {
+ value = aws_eip.cluster_keepproxy_public_ip.public_ip
+}
diff --git a/tools/terraform/keepstore_iam_policy.json b/tools/terraform/keepstore_iam_policy.json
new file mode 100644
index 000000000..9f7dd202d
--- /dev/null
+++ b/tools/terraform/keepstore_iam_policy.json
@@ -0,0 +1,19 @@
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": ["s3:ListBucket"],
+ "Resource": ["${bucket_arn}"]
+ },
+ {
+ "Effect": "Allow",
+ "Action": [
+ "s3:PutObject",
+ "s3:GetObject",
+ "s3:DeleteObject"
+ ],
+ "Resource": ["${bucket_arn}/*"]
+ }
+ ]
+}
diff --git a/tools/terraform/keepstore_iam_policy.tf b/tools/terraform/keepstore_iam_policy.tf
new file mode 100644
index 000000000..6a9d873ed
--- /dev/null
+++ b/tools/terraform/keepstore_iam_policy.tf
@@ -0,0 +1,13 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# IAM policy to access the bucket
+resource "aws_iam_policy" "keepstore_iam_policy" {
+ count = var.keepstore_count
+ name = "${var.cluster}-keepstore-${format("%02d", count.index)}-iam-role-policy"
+ description = "Policy to allow writing to the S3 bucket ${var.cluster}-nyw5e-${format("%016d", count.index)}-volume-policy"
+ policy = templatefile("${path.module}/keepstore_iam_policy.json", {
+ "bucket_arn" = aws_s3_bucket.keepstore.*.arn[count.index]
+ })
+}
diff --git a/tools/terraform/keepstore_iam_policy_s3_bucket.json b/tools/terraform/keepstore_iam_policy_s3_bucket.json
new file mode 100644
index 000000000..83c1ddaeb
--- /dev/null
+++ b/tools/terraform/keepstore_iam_policy_s3_bucket.json
@@ -0,0 +1,32 @@
+{
+ "Version": "2012-10-17",
+ "Id": "${id}",
+ "Statement": [
+ {
+ "Sid": "BucketAllow",
+ "Effect": "Allow",
+ "Principal": {
+ "AWS":
+ ${jsonencode([ for arn in access_arns : "${arn}" ], )}
+ },
+ "Action": [
+ "s3:ListBucket"
+ ],
+ "Resource": "${bucket_arn}"
+ },
+ {
+ "Sid": "BucketAllowObjects",
+ "Effect": "Allow",
+ "Principal": {
+ "AWS":
+ ${jsonencode([ for arn in access_arns : "${arn}" ], )}
+ },
+ "Action": [
+ "s3:PutObject",
+ "s3:GetObject",
+ "s3:DeleteObject"
+ ],
+ "Resource": "${bucket_arn}/*"
+ }
+ ]
+}
diff --git a/tools/terraform/keepstore_iam_role.tf b/tools/terraform/keepstore_iam_role.tf
new file mode 100644
index 000000000..512abbbed
--- /dev/null
+++ b/tools/terraform/keepstore_iam_role.tf
@@ -0,0 +1,24 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# Assume role for the instance
+resource "aws_iam_role" "keepstore_iam_assume_role" {
+ count = var.keepstore_count
+ name = "${var.cluster}-keepstore-${format("%02d", count.index)}-iam-role"
+ assume_role_policy = file("${path.module}/iam_policy_assume_role.json")
+}
+
+# Associate the access bucket policy to the role
+resource "aws_iam_role_policy_attachment" "keepstore_policies_attachment" {
+ count = var.keepstore_count
+ role = aws_iam_role.keepstore_iam_assume_role.*.name[count.index]
+ policy_arn = aws_iam_policy.keepstore_iam_policy.*.arn[count.index]
+}
+
+# Add the role to the instance profile
+resource "aws_iam_instance_profile" "keepstore_instance_profile" {
+ count = var.keepstore_count
+ name = "keepstore-${format("%02d", count.index)}_instance_profile"
+ role = "${var.cluster}-keepstore-${format("%02d", count.index)}-iam-role"
+}
diff --git a/tools/terraform/keepstore_iam_role_outputs.tf b/tools/terraform/keepstore_iam_role_outputs.tf
new file mode 100644
index 000000000..a1265d8e6
--- /dev/null
+++ b/tools/terraform/keepstore_iam_role_outputs.tf
@@ -0,0 +1,16 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "keepstore_iam_policy_id" {
+ value = aws_iam_policy.keepstore_iam_policy.*.id
+}
+output "keepstore_iam_policy_arn" {
+ value = aws_iam_policy.keepstore_iam_policy.*.arn
+}
+output "keepstore_iam_assume_role_arn" {
+ value = aws_iam_role.keepstore_iam_assume_role.*.arn
+}
+output "keepstore_iam_assume_role_id" {
+ value = aws_iam_role.keepstore_iam_assume_role.*.id
+}
diff --git a/tools/terraform/keepstore_instance.tf b/tools/terraform/keepstore_instance.tf
new file mode 100644
index 000000000..8e61fefc6
--- /dev/null
+++ b/tools/terraform/keepstore_instance.tf
@@ -0,0 +1,85 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+module "keepstore" {
+ count = var.keepstore_count
+ source = "terraform-aws-modules/ec2-instance/aws"
+ version = "~> 2.17.0"
+
+ name = "${var.cluster}-keepstore-${format("%02d", count.index)}"
+ instance_count = 1
+
+ ami = try(var.instance_ami["keepstore"], var.instance_ami["default"])
+ instance_type = try(var.instance_type["keepstore"], var.instance_type["default"])
+
+ iam_instance_profile = "keepstore-${format("%02d", count.index)}_instance_profile"
+ key_name = var.key_name
+ monitoring = true
+
+ tags = merge({"Name": "${var.cluster}-keepstore-${format("%02d", count.index)}",
+ "OsType": "LINUX"}, local.resource_tags)
+ volume_tags = merge({"Name": "${var.cluster}-keepstore-${format("%02d", count.index)}"},
+ local.resource_tags)
+
+ network_interface = [{
+ device_index = 0,
+ network_interface_id = aws_network_interface.keepstore.*.id[count.index],
+ }]
+
+ # associate_public_ip_address = false
+ ebs_optimized = true
+ user_data = templatefile("_user_data.sh", {})
+
+ root_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = var.root_bd_size,
+ delete_on_termination = true,
+ }]
+ ebs_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = try(var.data_bd_size["keepstore"], var.data_bd_size["default"])
+ delete_on_termination = true,
+ device_name = "xvdh",
+ }]
+}
+
+# resource "aws_eip" "cluster_keepstore_public_ip" {
+# count = var.keepstore_count
+# vpc = true
+# instance = module.keepstore.*.id[count.index][0]
+# network_interface = aws_network_interface.keepstore.*.id[count.index]
+# tags = merge({"Name": "${var.cluster}-keepstore-${format("%02d", count.index)}-ip"},local.resource_tags)
+# }
+
+resource "aws_network_interface" "keepstore" {
+ count = var.keepstore_count
+ subnet_id = var.manage_vpc ? module.vpc.0.private_subnets[0] : var.private_subnets_ids[0]
+ # private_ips = [cidrhost(var.vpc_subnet_cidrs[0], var.host_number["keepstore"])]
+ security_groups = [
+ local.ssh_sg,
+ local.keepstore_sg,
+ ]
+ tags = merge({"Name": "${var.cluster}-keepstore-${format("%02d", count.index)}"},
+ local.resource_tags)
+}
+
+### FIXME! Needs improvement
+## Private A RRs
+module "keepstore_route53_private_records_A" {
+ source = "./modules/aws/route53/records/a"
+ zone_id = module.r53_zone_private.id
+
+ zone_records_A = {
+ "keep0" = {
+ ttl = "300",
+ records = aws_network_interface.keepstore.0.private_ips
+ },
+ "keep1" = {
+ ttl = "300",
+ records = aws_network_interface.keepstore.1.private_ips
+ },
+ }
+}
diff --git a/tools/terraform/keepstore_instance_outputs.tf b/tools/terraform/keepstore_instance_outputs.tf
new file mode 100644
index 000000000..ac76a5ee3
--- /dev/null
+++ b/tools/terraform/keepstore_instance_outputs.tf
@@ -0,0 +1,16 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "keepstore_id" {
+ value = module.keepstore.*.id
+}
+output "keepstore_private_dns_names" {
+ value = module.keepstore.*.private_dns
+}
+output "keepstore_private_ip" {
+ value = module.keepstore.*.private_ip
+}
+output "keepstore_private_eni_id" {
+ value = aws_network_interface.keepstore.*.id
+}
diff --git a/tools/terraform/keepstore_s3_bucket.tf b/tools/terraform/keepstore_s3_bucket.tf
new file mode 100644
index 000000000..5aad658bd
--- /dev/null
+++ b/tools/terraform/keepstore_s3_bucket.tf
@@ -0,0 +1,35 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_s3_bucket" "keepstore" {
+ count = var.keepstore_count
+ bucket = "${var.cluster}-nyw5e-${format("%016d", count.index)}-volume"
+ acl = "private"
+
+ website {
+ index_document = "index.html"
+ error_document = "error.html"
+ }
+
+ server_side_encryption_configuration {
+ rule {
+ apply_server_side_encryption_by_default {
+ sse_algorithm = "AES256"
+ }
+ }
+ }
+ tags = merge({"Name": "${var.cluster}-nyw5e-${format("%016d", count.index)}-bucket"},
+ local.resource_tags)
+}
+
+resource "aws_s3_bucket_policy" "keepstore_bucket_policy" {
+ count = var.keepstore_count
+ bucket = aws_s3_bucket.keepstore.*.id[count.index]
+
+ policy = templatefile("${path.module}/keepstore_iam_policy_s3_bucket.json", {
+ id = "${var.cluster}-nyw5e-${format("%016d", count.index)}-volume-policy",
+ bucket_arn = aws_s3_bucket.keepstore.*.arn[count.index]
+ access_arns = [aws_iam_role.keepstore_iam_assume_role.*.arn[count.index]]
+ })
+}
diff --git a/tools/terraform/keepstore_s3_bucket_outputs.tf b/tools/terraform/keepstore_s3_bucket_outputs.tf
new file mode 100644
index 000000000..b816d22e4
--- /dev/null
+++ b/tools/terraform/keepstore_s3_bucket_outputs.tf
@@ -0,0 +1,13 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "keepstore_bucket_arn" {
+ value = aws_s3_bucket.keepstore.*.arn
+}
+output "keepstore_bucket_id" {
+ value = aws_s3_bucket.keepstore.*.id
+}
+output "keepstore_bucket_policy_id" {
+ value = aws_s3_bucket_policy.keepstore_bucket_policy.*.id
+}
diff --git a/tools/terraform/keypair.tf b/tools/terraform/keypair.tf
new file mode 100644
index 000000000..c9a6636eb
--- /dev/null
+++ b/tools/terraform/keypair.tf
@@ -0,0 +1,11 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_key_pair" "keypair" {
+ count = var.enable_key_pair == true ? 1 : 0
+
+ key_name = var.key_name
+ public_key = var.key_public_key == "" ? file(var.key_path) : var.key_public_key
+ tags = local.resource_tags
+}
diff --git a/tools/terraform/keypair_outputs.tf b/tools/terraform/keypair_outputs.tf
new file mode 100644
index 000000000..92d188f77
--- /dev/null
+++ b/tools/terraform/keypair_outputs.tf
@@ -0,0 +1,8 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "keypair_name" {
+ description = "Name of the SSH keypair applied to the instances."
+ value = var.key_name
+}
diff --git a/tools/terraform/letsencrypt_route53_iam_policy.json b/tools/terraform/letsencrypt_route53_iam_policy.json
new file mode 100644
index 000000000..282a63fd9
--- /dev/null
+++ b/tools/terraform/letsencrypt_route53_iam_policy.json
@@ -0,0 +1,25 @@
+{
+ "Version": "2012-10-17",
+ "Id": "${cluster}_letsencrypt_route53_policy",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": [
+ "route53:ListHostedZones",
+ "route53:GetChange"
+ ],
+ "Resource": [
+ "*"
+ ]
+ },
+ {
+ "Effect" : "Allow",
+ "Action" : [
+ "route53:ChangeResourceRecordSets"
+ ],
+ "Resource" : [
+ "arn:aws:route53:::hostedzone/${zone_id}"
+ ]
+ }
+ ]
+}
diff --git a/tools/terraform/letsencrypt_route53_iam_policy.tf b/tools/terraform/letsencrypt_route53_iam_policy.tf
new file mode 100644
index 000000000..c1b7fdcf2
--- /dev/null
+++ b/tools/terraform/letsencrypt_route53_iam_policy.tf
@@ -0,0 +1,12 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_iam_policy" "letsencrypt_route53_iam_policy" {
+ name = "${var.cluster}-letsencrypt_route53-iam-role-policy"
+ description = "Policy to allow API to add records to the public zone"
+ policy = templatefile("${path.module}/letsencrypt_route53_iam_policy.json", {
+ "cluster" = var.cluster
+ "zone_id" = module.r53_zone_public.id
+ })
+}
diff --git a/tools/terraform/locals.tf b/tools/terraform/locals.tf
new file mode 100644
index 000000000..c40dae11d
--- /dev/null
+++ b/tools/terraform/locals.tf
@@ -0,0 +1,20 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+locals {
+ resource_tags = merge(
+ { "Terraform" = "true" },
+ { "Environment" = var.environment },
+ { "Namespace" = var.namespace },
+ { "Cluster" = var.cluster },
+ var.tags,
+ )
+ ssh_sg = var.manage_security_groups ? module.arvados_ssh_sg.0.this_security_group_id : try(var.vpc_security_group_ids["ssh"], var.vpc_security_group_ids["default"])
+ http_sg = var.manage_security_groups ? module.arvados_http_sg.0.this_security_group_id : try(var.vpc_security_group_ids["http"], var.vpc_security_group_ids["default"])
+ https_sg = var.manage_security_groups ? module.arvados_https_sg.0.this_security_group_id : try(var.vpc_security_group_ids["https"], var.vpc_security_group_ids["default"])
+ webshell_sg = var.manage_security_groups ? module.arvados_webshell_sg.0.this_security_group_id : try(var.vpc_security_group_ids["webshell"], var.vpc_security_group_ids["default"])
+ postgresql_sg = var.manage_security_groups ? module.arvados_postgresql_sg.0.this_security_group_id : try(var.vpc_security_group_ids["postgresql"], var.vpc_security_group_ids["default"])
+ keepstore_sg = var.manage_security_groups ? module.arvados_keepstore_sg.0.this_security_group_id : try(var.vpc_security_group_ids["keepstore"], var.vpc_security_group_ids["default"])
+
+}
diff --git a/tools/terraform/modules/aws/route53/records/a/main.tf b/tools/terraform/modules/aws/route53/records/a/main.tf
new file mode 100644
index 000000000..3f71b39b4
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/a/main.tf
@@ -0,0 +1,15 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_route53_record" "a_record" {
+ for_each = var.zone_records_A
+
+ zone_id = var.zone_id
+ name = lookup(each.value, "name", each.key)
+ type = "A"
+ ttl = lookup(each.value, "ttl", 600)
+
+ records = lookup(each.value, "records", null)
+ allow_overwrite = lookup(each.value, "allow_overwrite", null)
+}
diff --git a/tools/terraform/modules/aws/route53/records/a/outputs.tf b/tools/terraform/modules/aws/route53/records/a/outputs.tf
new file mode 100644
index 000000000..9ae9cdf8b
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/a/outputs.tf
@@ -0,0 +1,7 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "a_records" {
+ value = aws_route53_record.a_record
+}
diff --git a/tools/terraform/modules/aws/route53/records/a/variables.tf b/tools/terraform/modules/aws/route53/records/a/variables.tf
new file mode 100644
index 000000000..d42acc150
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/a/variables.tf
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+variable "zone_id" {
+ type = string
+}
+# Sadly, terraform can't iterate over maps of mixed values without lots
+# of juggling with forced parameters in object definitions and
+# other similarly nasty workarounds, so trying to keep it simple, we'll
+# declare modules for each type of RRs, so code is simpler.
+# More info here https://github.com/hashicorp/terraform/issues/19898
+variable "zone_records_A" {
+ description = "Map of A RRs to add to the zone to create"
+ type = map
+ default = {}
+}
diff --git a/tools/terraform/modules/aws/route53/records/aaaa/main.tf b/tools/terraform/modules/aws/route53/records/aaaa/main.tf
new file mode 100644
index 000000000..cdd3e0d49
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/aaaa/main.tf
@@ -0,0 +1,15 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_route53_record" "aaaa_record" {
+ for_each = var.zone_records_AAAA
+
+ zone_id = var.zone_id
+ name = lookup(each.value, "name", each.key)
+ type = "AAAA"
+ ttl = lookup(each.value, "ttl", 600)
+
+ records = lookup(each.value, "records", null)
+ allow_overwrite = lookup(each.value, "allow_overwrite", null)
+}
diff --git a/tools/terraform/modules/aws/route53/records/aaaa/outputs.tf b/tools/terraform/modules/aws/route53/records/aaaa/outputs.tf
new file mode 100644
index 000000000..683183a96
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/aaaa/outputs.tf
@@ -0,0 +1,7 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "aaaa_records" {
+ value = aws_route53_record.aaaa_record
+}
diff --git a/tools/terraform/modules/aws/route53/records/aaaa/variables.tf b/tools/terraform/modules/aws/route53/records/aaaa/variables.tf
new file mode 100644
index 000000000..43a838c6b
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/aaaa/variables.tf
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+variable "zone_id" {
+ type = string
+}
+# Sadly, terraform can't iterate over maps of mixed values without lots
+# of juggling with forced parameters in object definitions and
+# other similarly nasty workarounds, so trying to keep it simple, we'll
+# declare modules for each type of RRs, so code is simpler.
+# More info here https://github.com/hashicorp/terraform/issues/19898
+variable "zone_records_AAAA" {
+ description = "Map of AAAA RRs to add to the zone to create"
+ type = map
+ default = {}
+}
diff --git a/tools/terraform/modules/aws/route53/records/alias/main.tf b/tools/terraform/modules/aws/route53/records/alias/main.tf
new file mode 100644
index 000000000..92834a3c4
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/alias/main.tf
@@ -0,0 +1,18 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_route53_record" "alias_record" {
+ for_each = var.zone_records_ALIAS
+
+ zone_id = var.zone_id
+ name = lookup(each.value, "name", each.key)
+ type = lookup(each.value, "type", "A")
+
+ alias {
+ name = each.value.alias.name
+ zone_id = each.value.alias.zone_id
+ evaluate_target_health = lookup(each.value.alias, "evaluate_target_health", false)
+ }
+ allow_overwrite = lookup(each.value, "allow_overwrite", null)
+}
diff --git a/tools/terraform/modules/aws/route53/records/alias/outputs.tf b/tools/terraform/modules/aws/route53/records/alias/outputs.tf
new file mode 100644
index 000000000..59b605456
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/alias/outputs.tf
@@ -0,0 +1,7 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "alias_records" {
+ value = aws_route53_record.alias_record
+}
diff --git a/tools/terraform/modules/aws/route53/records/alias/variables.tf b/tools/terraform/modules/aws/route53/records/alias/variables.tf
new file mode 100644
index 000000000..e9381e622
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/alias/variables.tf
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+variable "zone_id" {
+ type = string
+}
+# Sadly, terraform can't iterate over maps of mixed values without lots
+# of juggling with forced parameters in object definitions and
+# other similarly nasty workarounds, so trying to keep it simple, we'll
+# declare modules for each type of RRs, so code is simpler.
+# More info here https://github.com/hashicorp/terraform/issues/19898
+variable "zone_records_ALIAS" {
+ description = "Map of ALIAS RRs to add to the zone to create"
+ type = map
+ default = {}
+}
diff --git a/tools/terraform/modules/aws/route53/records/cname/main.tf b/tools/terraform/modules/aws/route53/records/cname/main.tf
new file mode 100644
index 000000000..9eef47a6c
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/cname/main.tf
@@ -0,0 +1,15 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_route53_record" "cname_record" {
+ for_each = var.zone_records_CNAME
+
+ zone_id = var.zone_id
+ name = lookup(each.value, "name", each.key)
+ type = "CNAME"
+ ttl = lookup(each.value, "ttl", 600)
+
+ records = each.value.records
+ allow_overwrite = lookup(each.value, "allow_overwrite", null)
+}
diff --git a/tools/terraform/modules/aws/route53/records/cname/outputs.tf b/tools/terraform/modules/aws/route53/records/cname/outputs.tf
new file mode 100644
index 000000000..db7757eee
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/cname/outputs.tf
@@ -0,0 +1,7 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "cname_records" {
+ value = aws_route53_record.cname_record
+}
diff --git a/tools/terraform/modules/aws/route53/records/cname/variables.tf b/tools/terraform/modules/aws/route53/records/cname/variables.tf
new file mode 100644
index 000000000..6d053dc08
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/cname/variables.tf
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+variable "zone_id" {
+ type = string
+}
+# Sadly, terraform can't iterate over maps of mixed values without lots
+# of juggling with forced parameters in object definitions and
+# other similarly nasty workarounds, so trying to keep it simple, we'll
+# declare modules for each type of RRs, so code is simpler.
+# More info here https://github.com/hashicorp/terraform/issues/19898
+variable "zone_records_CNAME" {
+ description = "Map of CNAME RRs to add to the zone to create"
+ type = map
+ default = {}
+}
diff --git a/tools/terraform/modules/aws/route53/records/mx/main.tf b/tools/terraform/modules/aws/route53/records/mx/main.tf
new file mode 100644
index 000000000..71ced88ba
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/mx/main.tf
@@ -0,0 +1,15 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_route53_record" "mx_record" {
+ for_each = var.zone_records_MX
+
+ zone_id = var.zone_id
+ name = lookup(each.value, "name", each.key)
+ type = "MX"
+ ttl = lookup(each.value, "ttl", 600)
+
+ records = lookup(each.value, "records", null)
+ allow_overwrite = lookup(each.value, "allow_overwrite", null)
+}
diff --git a/tools/terraform/modules/aws/route53/records/mx/outputs.tf b/tools/terraform/modules/aws/route53/records/mx/outputs.tf
new file mode 100644
index 000000000..f1c7e62f0
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/mx/outputs.tf
@@ -0,0 +1,7 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "mx_records" {
+ value = aws_route53_record.mx_record
+}
diff --git a/tools/terraform/modules/aws/route53/records/mx/variables.tf b/tools/terraform/modules/aws/route53/records/mx/variables.tf
new file mode 100644
index 000000000..afdb5d1c2
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/mx/variables.tf
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+variable "zone_id" {
+ type = string
+}
+# Sadly, terraform can't iterate over maps of mixed values without lots
+# of juggling with forced parameters in object definitions and
+# other similarly nasty workarounds, so trying to keep it simple, we'll
+# declare modules for each type of RRs, so code is simpler.
+# More info here https://github.com/hashicorp/terraform/issues/19898
+variable "zone_records_MX" {
+ description = "Map of MX RRs to add to the zone to create"
+ type = map
+ default = {}
+}
diff --git a/tools/terraform/modules/aws/route53/records/ns/main.tf b/tools/terraform/modules/aws/route53/records/ns/main.tf
new file mode 100644
index 000000000..6670fc29a
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/ns/main.tf
@@ -0,0 +1,15 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_route53_record" "ns_record" {
+ for_each = var.zone_records_NS
+
+ zone_id = var.zone_id
+ name = lookup(each.value, "name", each.key)
+ type = "NS"
+ ttl = lookup(each.value, "ttl", 600)
+
+ records = lookup(each.value, "records", null)
+ allow_overwrite = lookup(each.value, "allow_overwrite", null)
+}
diff --git a/tools/terraform/modules/aws/route53/records/ns/outputs.tf b/tools/terraform/modules/aws/route53/records/ns/outputs.tf
new file mode 100644
index 000000000..4e900e2b3
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/ns/outputs.tf
@@ -0,0 +1,7 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "ns_records" {
+ value = aws_route53_record.ns_record
+}
diff --git a/tools/terraform/modules/aws/route53/records/ns/variables.tf b/tools/terraform/modules/aws/route53/records/ns/variables.tf
new file mode 100644
index 000000000..c993ffcc2
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/ns/variables.tf
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+variable "zone_id" {
+ type = string
+}
+# Sadly, terraform can't iterate over maps of mixed values without lots
+# of juggling with forced parameters in object definitions and
+# other similarly nasty workarounds, so trying to keep it simple, we'll
+# declare modules for each type of RRs, so code is simpler.
+# More info here https://github.com/hashicorp/terraform/issues/19898
+variable "zone_records_NS" {
+ description = "Map of NS RRs to add to the zone to create"
+ type = map
+ default = {}
+}
diff --git a/tools/terraform/modules/aws/route53/records/txt/main.tf b/tools/terraform/modules/aws/route53/records/txt/main.tf
new file mode 100644
index 000000000..91a3969bc
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/txt/main.tf
@@ -0,0 +1,15 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_route53_record" "txt_record" {
+ for_each = var.zone_records_TXT
+
+ zone_id = var.zone_id
+ name = lookup(each.value, "name", each.key)
+ type = "TXT"
+ ttl = lookup(each.value, "ttl", 600)
+
+ records = lookup(each.value, "records", null)
+ allow_overwrite = lookup(each.value, "allow_overwrite", null)
+}
diff --git a/tools/terraform/modules/aws/route53/records/txt/outputs.tf b/tools/terraform/modules/aws/route53/records/txt/outputs.tf
new file mode 100644
index 000000000..fed21f21c
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/txt/outputs.tf
@@ -0,0 +1,7 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "txt_records" {
+ value = aws_route53_record.txt_record
+}
diff --git a/tools/terraform/modules/aws/route53/records/txt/variables.tf b/tools/terraform/modules/aws/route53/records/txt/variables.tf
new file mode 100644
index 000000000..17dacdd8e
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/records/txt/variables.tf
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+variable "zone_id" {
+ type = string
+}
+# Sadly, terraform can't iterate over maps of mixed values without lots
+# of juggling with forced parameters in object definitions and
+# other similarly nasty workarounds, so trying to keep it simple, we'll
+# declare modules for each type of RRs, so code is simpler.
+# More info here https://github.com/hashicorp/terraform/issues/19898
+variable "zone_records_TXT" {
+ description = "Map of TXT RRs to add to the zone to create"
+ type = map
+ default = {}
+}
diff --git a/tools/terraform/modules/aws/route53/zone/private/main.tf b/tools/terraform/modules/aws/route53/zone/private/main.tf
new file mode 100644
index 000000000..8aa125e06
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/zone/private/main.tf
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_route53_zone" "this" {
+name = var.zone_name
+ vpc {
+ vpc_id = lookup(var.zone_config, "vpc_id")
+ }
+
+ comment = lookup(var.zone_config, "comment", null)
+ force_destroy = lookup(var.zone_config, "force_destroy", null)
+ tags = merge(
+ var.tags,
+ {"ZoneScope" = "private"}
+ )
+}
diff --git a/tools/terraform/modules/aws/route53/zone/private/outputs.tf b/tools/terraform/modules/aws/route53/zone/private/outputs.tf
new file mode 100644
index 000000000..9765daaa0
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/zone/private/outputs.tf
@@ -0,0 +1,13 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "id" {
+ value = aws_route53_zone.this.id
+}
+output "name" {
+ value = aws_route53_zone.this.name
+}
+output "name_servers" {
+ value = aws_route53_zone.this.name_servers
+}
diff --git a/tools/terraform/modules/aws/route53/zone/private/variables.tf b/tools/terraform/modules/aws/route53/zone/private/variables.tf
new file mode 100644
index 000000000..7afb16eb5
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/zone/private/variables.tf
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+variable "zone_name" {
+ description = "Public zone to create"
+ type = string
+}
+variable "zone_config" {
+ description = "Zone's config parameters"
+ type = map
+ default = {}
+}
+variable "tags" {
+ type = map
+ default = {}
+}
diff --git a/tools/terraform/modules/aws/route53/zone/public/main.tf b/tools/terraform/modules/aws/route53/zone/public/main.tf
new file mode 100644
index 000000000..9697ea217
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/zone/public/main.tf
@@ -0,0 +1,13 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+resource "aws_route53_zone" "this" {
+ name = var.zone_name
+ comment = lookup(var.zone_config, "comment", null)
+ force_destroy = lookup(var.zone_config, "force_destroy", null)
+ tags = merge(
+ var.tags,
+ {"ZoneScope" = "public"}
+ )
+}
diff --git a/tools/terraform/modules/aws/route53/zone/public/outputs.tf b/tools/terraform/modules/aws/route53/zone/public/outputs.tf
new file mode 100644
index 000000000..9765daaa0
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/zone/public/outputs.tf
@@ -0,0 +1,13 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "id" {
+ value = aws_route53_zone.this.id
+}
+output "name" {
+ value = aws_route53_zone.this.name
+}
+output "name_servers" {
+ value = aws_route53_zone.this.name_servers
+}
diff --git a/tools/terraform/modules/aws/route53/zone/public/variables.tf b/tools/terraform/modules/aws/route53/zone/public/variables.tf
new file mode 100644
index 000000000..7afb16eb5
--- /dev/null
+++ b/tools/terraform/modules/aws/route53/zone/public/variables.tf
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+variable "zone_name" {
+ description = "Public zone to create"
+ type = string
+}
+variable "zone_config" {
+ description = "Zone's config parameters"
+ type = map
+ default = {}
+}
+variable "tags" {
+ type = map
+ default = {}
+}
diff --git a/tools/terraform/route53.tf b/tools/terraform/route53.tf
new file mode 100644
index 000000000..97763d5ce
--- /dev/null
+++ b/tools/terraform/route53.tf
@@ -0,0 +1,28 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+## ZONE definition
+module "r53_zone_public" {
+ source = "./modules/aws/route53/zone/public"
+ zone_name = var.r53_domain_name
+ tags = merge(
+ {"Name" = var.r53_domain_name,
+ "Cluster" = var.cluster,
+ },
+ local.resource_tags,
+ )
+}
+module "r53_zone_private" {
+ source = "./modules/aws/route53/zone/private"
+ zone_name = var.r53_domain_name
+ zone_config = {
+ vpc_id = var.manage_vpc ? module.vpc.*.vpc_id[0] : var.vpc_id
+ }
+ tags = merge(
+ {"Name" = var.r53_domain_name,
+ "Cluster" = var.cluster,
+ },
+ local.resource_tags,
+ )
+}
diff --git a/tools/terraform/route53_outputs.tf b/tools/terraform/route53_outputs.tf
new file mode 100644
index 000000000..fad655867
--- /dev/null
+++ b/tools/terraform/route53_outputs.tf
@@ -0,0 +1,22 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "vpc_route53_private_zone_id" {
+ value = module.r53_zone_private.id
+}
+output "vpc_route53_private_zone_name" {
+ value = module.r53_zone_private.name
+}
+output "vpc_route53_private_name_servers" {
+ value = module.r53_zone_private.name_servers
+}
+output "vpc_route53_public_zone_id" {
+ value = module.r53_zone_public.id
+}
+output "vpc_route53_public_zone_name" {
+ value = module.r53_zone_public.name
+}
+output "vpc_route53_public_name_servers" {
+ value = module.r53_zone_public.name_servers
+}
diff --git a/tools/terraform/security_groups.tf b/tools/terraform/security_groups.tf
new file mode 100644
index 000000000..4c24e5ef4
--- /dev/null
+++ b/tools/terraform/security_groups.tf
@@ -0,0 +1,112 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+module "arvados_ssh_sg" {
+ count = var.manage_security_groups ? 1 : 0
+ source = "terraform-aws-modules/security-group/aws//modules/ssh"
+
+ name = "${var.cluster}_ssh_sg"
+ description = "SSH to Arvados VPC"
+ vpc_id = var.manage_vpc ? module.vpc.*.vpc_id[0] : var.vpc_id
+
+ ingress_cidr_blocks = concat(
+ var.allowed_full_access_cidrs,
+ var.allowed_ssh_access_cidrs,
+ var.private_subnets,
+ var.public_subnets
+ )
+ tags = merge({"Name": "${var.cluster}-ssh-sg"},
+ local.resource_tags)
+}
+module "arvados_http_sg" {
+ count = var.manage_security_groups ? 1 : 0
+ source = "terraform-aws-modules/security-group/aws//modules/http-80"
+ name = "${var.cluster}_http_80_sg"
+ description = "HTTP security group"
+ vpc_id = var.manage_vpc ? module.vpc.*.vpc_id[0] : var.vpc_id
+ ingress_cidr_blocks = concat(
+ var.allowed_full_access_cidrs,
+ var.allowed_http_access_cidrs,
+ var.private_subnets,
+ var.public_subnets
+ )
+ tags = merge({"Name": "${var.cluster}-http-sg"},
+ local.resource_tags)
+}
+module "arvados_https_sg" {
+ count = var.manage_security_groups ? 1 : 0
+ source = "terraform-aws-modules/security-group/aws//modules/https-443"
+ name = "${var.cluster}_https_443_sg"
+ description = "HTTPs security group"
+ vpc_id = var.manage_vpc ? module.vpc.*.vpc_id[0] : var.vpc_id
+ ingress_cidr_blocks = concat(
+ var.allowed_full_access_cidrs,
+ var.allowed_http_access_cidrs,
+ var.private_subnets,
+ var.public_subnets
+ )
+ tags = merge({"Name": "${var.cluster}-https-sg"},
+ local.resource_tags)
+}
+module "arvados_webshell_sg" {
+ count = var.manage_security_groups ? 1 : 0
+ source = "terraform-aws-modules/security-group/aws"
+
+ name = "${var.cluster}_webshell_sg"
+ description = "Arvados access to webshell server"
+ vpc_id = var.manage_vpc ? module.vpc.*.vpc_id[0] : var.vpc_id
+
+ ingress_cidr_blocks = concat(
+ var.allowed_full_access_cidrs,
+ var.allowed_ssh_access_cidrs,
+ var.private_subnets,
+ )
+ ingress_with_cidr_blocks = [
+ {
+ from_port = 4200
+ to_port = 4200
+ protocol = "tcp"
+ description = "Webshell port"
+ cidr_blocks = var.cluster_cidr
+ },
+ ]
+ tags = merge({"Name": "${var.cluster}-webshell-sg"},
+ local.resource_tags)
+}
+module "arvados_postgresql_sg" {
+ count = var.manage_security_groups ? 1 : 0
+ source = "terraform-aws-modules/security-group/aws//modules/postgresql"
+
+ name = "${var.cluster}_postgresql_sg"
+ description = "Arvados postgresql security group"
+ vpc_id = var.manage_vpc ? module.vpc.*.vpc_id[0] : var.vpc_id
+
+ ingress_cidr_blocks = concat(
+ var.allowed_full_access_cidrs,
+ var.private_subnets,
+ )
+
+ tags = merge({"Name": "${var.cluster}-postgresql-sg"},
+ local.resource_tags)
+}
+module "arvados_keepstore_sg" {
+ count = var.manage_security_groups ? 1 : 0
+ source = "terraform-aws-modules/security-group/aws"
+
+ name = "keepstore_sg"
+ description = "Arvados security group for the keepstore service"
+ vpc_id = var.manage_vpc ? module.vpc.*.vpc_id[0] : var.vpc_id
+
+ ingress_cidr_blocks = [var.vpc_cidr]
+ ingress_with_cidr_blocks = [
+ {
+ from_port = 25107
+ to_port = 25107
+ protocol = "tcp"
+ description = "Keepstore port"
+ cidr_blocks = var.cluster_cidr
+ },
+ ]
+ tags = merge({"Name": "keepstore-sg"},local.resource_tags)
+}
diff --git a/tools/terraform/security_groups_outputs.tf b/tools/terraform/security_groups_outputs.tf
new file mode 100644
index 000000000..c0f0f9d49
--- /dev/null
+++ b/tools/terraform/security_groups_outputs.tf
@@ -0,0 +1,28 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "arvados_ssh_security_group_id" {
+ description = "The ID of the arvados SSH security group"
+ value = var.manage_security_groups ? module.arvados_ssh_sg.0.this_security_group_id : var.vpc_security_group_ids["ssh"]
+}
+output "arvados_webshell_security_group_id" {
+ description = "The ID of the arvados Webshell security group"
+ value = var.manage_security_groups ? module.arvados_webshell_sg.0.this_security_group_id : var.vpc_security_group_ids["webshell"]
+}
+output "arvados_http_security_group_id" {
+ description = "The ID of the arvados HTTP security group"
+ value = var.manage_security_groups ? module.arvados_http_sg.0.this_security_group_id : var.vpc_security_group_ids["http"]
+}
+output "arvados_https_security_group_id" {
+ description = "The ID of the arvados HTTPS security group"
+ value = var.manage_security_groups ? module.arvados_https_sg.0.this_security_group_id : var.vpc_security_group_ids["https"]
+}
+output "arvados_postgresql_security_group_id" {
+ description = "The ID of the arvados Postgresql security group"
+ value = var.manage_security_groups ? module.arvados_postgresql_sg.0.this_security_group_id : var.vpc_security_group_ids["postgresql"]
+}
+output "arvados_keepstore_security_group_id" {
+ description = "The ID of the arvados Keepstore security group"
+ value = var.manage_security_groups ? module.arvados_keepstore_sg.0.this_security_group_id : var.vpc_security_group_ids["keepstore"]
+}
diff --git a/tools/terraform/shell_iam_role.tf b/tools/terraform/shell_iam_role.tf
new file mode 100644
index 000000000..7e11adf73
--- /dev/null
+++ b/tools/terraform/shell_iam_role.tf
@@ -0,0 +1,21 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# Assume role for the instance
+resource "aws_iam_role" "shell_iam_role" {
+ name = "${var.cluster}-shell-iam-role"
+ assume_role_policy = templatefile("${path.module}/iam_policy_assume_role.json", {})
+}
+
+# Associate letsencrypt modification policy to the role
+resource "aws_iam_role_policy_attachment" "shell_letsencrypt_route53_policies_attachment" {
+ role = aws_iam_role.shell_iam_role.name
+ policy_arn = aws_iam_policy.letsencrypt_route53_iam_policy.arn
+}
+
+# Add the role to the instance profile
+resource "aws_iam_instance_profile" "shell_instance_profile" {
+ name = "shell_instance_profile"
+ role = "${var.cluster}-shell-iam-role"
+}
diff --git a/tools/terraform/shell_iam_role_outputs.tf b/tools/terraform/shell_iam_role_outputs.tf
new file mode 100644
index 000000000..23e364d5d
--- /dev/null
+++ b/tools/terraform/shell_iam_role_outputs.tf
@@ -0,0 +1,10 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "shell_iam_role_arn" {
+ value = aws_iam_role.shell_iam_role.arn
+}
+output "shell_iam_role_id" {
+ value = aws_iam_role.shell_iam_role.id
+}
diff --git a/tools/terraform/shell_instance.tf b/tools/terraform/shell_instance.tf
new file mode 100644
index 000000000..e7774612f
--- /dev/null
+++ b/tools/terraform/shell_instance.tf
@@ -0,0 +1,91 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+module "shell" {
+ source = "terraform-aws-modules/ec2-instance/aws"
+ version = "~> 2.17.0"
+
+ name = "${var.cluster}-shell"
+ instance_count = 1
+
+ iam_instance_profile = "shell_instance_profile"
+ ami = try(var.instance_ami["shell"], var.instance_ami["default"])
+ instance_type = try(var.instance_type["shell"], var.instance_type["default"])
+ key_name = var.key_name
+ monitoring = true
+
+ tags = merge({"Name": "${var.cluster}-shell",
+ "OsType": "LINUX"}, local.resource_tags)
+ volume_tags = merge({"Name": "${var.cluster}-shell"}, local.resource_tags)
+
+ network_interface = [{
+ device_index = 0,
+ network_interface_id = aws_network_interface.shell.id,
+ }]
+
+ # associate_public_ip_address = false
+ ebs_optimized = true
+ user_data = templatefile("_user_data.sh", {})
+
+ root_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = var.root_bd_size,
+ delete_on_termination = true,
+ }]
+ ebs_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = try(var.data_bd_size["shell"], var.data_bd_size["default"])
+ delete_on_termination = true,
+ device_name = "xvdh",
+ }]
+}
+
+resource "aws_eip" "cluster_shell_public_ip" {
+ vpc = true
+ instance = module.shell.id[0]
+ network_interface = aws_network_interface.shell.id
+ tags = merge({"Name": "${var.cluster}-shell-ip"},local.resource_tags)
+}
+
+resource "aws_network_interface" "shell" {
+ subnet_id = var.manage_vpc ? module.vpc.0.public_subnets[0] : var.public_subnets_ids[0]
+ # private_ips = [cidrhost(var.vpc_subnet_cidrs[0], var.host_number["shell"])]
+ security_groups = [
+ local.ssh_sg,
+ local.webshell_sg,
+ ]
+ tags = merge({"Name": "${var.cluster}-shell"}, local.resource_tags)
+}
+
+## Public A RRs
+module "shell_route53_public_records_A" {
+ source = "./modules/aws/route53/records/a"
+ zone_id = module.r53_zone_public.id
+
+ zone_records_A = {
+ "shell" = {
+ ttl = "300",
+ records = [aws_eip.cluster_shell_public_ip.public_ip]
+ },
+ }
+}
+
+## Private A RRs
+module "shell_route53_private_records_A" {
+ source = "./modules/aws/route53/records/a"
+ zone_id = module.r53_zone_private.id
+
+ zone_records_A = {
+ "shell" = {
+ ttl = "300",
+ records = aws_network_interface.shell.private_ips
+ },
+ "webshell" = {
+ ttl = "300",
+ records = aws_network_interface.shell.private_ips
+ },
+ }
+}
diff --git a/tools/terraform/shell_instance_outputs.tf b/tools/terraform/shell_instance_outputs.tf
new file mode 100644
index 000000000..6be74d481
--- /dev/null
+++ b/tools/terraform/shell_instance_outputs.tf
@@ -0,0 +1,19 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "shell_id" {
+ value = module.shell.id
+}
+output "shell_private_dns_names" {
+ value = module.shell.private_dns
+}
+output "shell_private_ip" {
+ value = module.shell.private_ip
+}
+output "shell_private_eni_id" {
+ value = aws_network_interface.shell.id
+}
+output "shell_public_ip" {
+ value = aws_eip.cluster_shell_public_ip.public_ip
+}
diff --git a/tools/terraform/terraform.tfvars b/tools/terraform/terraform.tfvars
new file mode 100644
index 000000000..46ea86971
--- /dev/null
+++ b/tools/terraform/terraform.tfvars
@@ -0,0 +1,84 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+### GENERAL
+aws_profile = "profile-to-use"
+aws_region = "us-east-1"
+environment = "production"
+namespace = "3rd-party deploy test"
+
+### KEYPAIR
+key_name = "keyname"
+key_path = "~/.ssh/id_rsa.pub"
+
+cluster = "vwxyz"
+r53_domain_name = "vwxyz.arvados.test"
+
+# VPC
+# If you have/want to use a VPC already defined, set this value to false
+# and uncomment and provide values for the following variables
+
+# manage_vpc = true
+# vpc_id = "vpc-12345678901234567"
+# private_subnets_ids = []
+# compute_subnets_ids = []
+# public_subnets_ids = []
+
+cluster_cidr = "10.0.0.0/16"
+azs = ["us-east-1a"]
+private_subnets = ["10.0.255.0/24"]
+compute_subnets = ["10.0.254.0/24"]
+public_subnets = ["10.0.0.0/24"]
+enable_nat_gateway = true
+enable_vpn_gateway = false
+single_nat_gateway = true
+one_nat_gateway_per_az = false
+enable_dhcp_options = true
+
+instance_type = {
+ "default" = "m5a.large",
+ # "api" = "m5a.large",
+ # "shell" = "m5a.large",
+ # "keepproxy" = "m5a.large",
+ # "keepstore" = "m5a.large",
+ # "workbench" = "m5a.large",
+ # "database" = "m5a.large",
+}
+instance_ami = {
+ "default" = "ami-07d02ee1eeb0c996c",
+ # "api" = "ami-07d02ee1eeb0c996c",
+ # "shell" = "ami-07d02ee1eeb0c996c",
+ # "keepstore" = "ami-07d02ee1eeb0c996c",
+ # "keepproxy" = "ami-07d02ee1eeb0c996c",
+ # "workbench" = "ami-07d02ee1eeb0c996c",
+ # "database" = "ami-07d02ee1eeb0c996c",
+}
+
+data_bd_size = {
+ "default" = 50,
+ # "api" = 50,
+ # "shell" = 50,
+ # "keepproxy" = 50,
+ # "keepstore" = 50,
+ # "workbench" = 50,
+ "database" = 250,
+}
+# KEEPSTORE/s
+keepstore_count = 2
+
+# SECURITY
+# CIDRs allowed unrestricted access to the instances
+# allowed_access_cidrs = "0.0.0.0/0"
+
+# If you have/want to use already defined security groups, set this value to false
+# and uncomment and provide values for the following variables
+# vpc_security_group_ids = {
+# "default" = "sg-01111111111111110",
+# "ssh" = "sg-01234567890123456",
+# "http" = "sg-12345678901234567",
+# "https" = "sg-23456789012345678",
+# "webshell" = "sg-34567890123456789",
+# "postgresql" = "sg-45678901234567890",
+# "keepstore" = "sg-56789012345678901",
+# }
diff --git a/tools/terraform/variables.tf b/tools/terraform/variables.tf
new file mode 100644
index 000000000..902ff4563
--- /dev/null
+++ b/tools/terraform/variables.tf
@@ -0,0 +1,243 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+### GENERAL
+variable "aws_region" {
+ description = "The AWS region where to deploy the cluster"
+ type = string
+ default = ""
+}
+variable "aws_profile" {
+ description = "The AWS profile to use"
+ type = string
+ default = ""
+}
+variable "ami" {
+ description = "The AMI to use when launching instances"
+ # This is Debian buster in us-east-1
+ type = string
+ default = "ami-07d02ee1eeb0c996c"
+}
+variable "namespace" {
+ description = "A descriptive name for the resources' namespace"
+ type = string
+ default = ""
+}
+variable "environment" {
+ description = "A descriptive name for the cluster's environment"
+ type = string
+ default = ""
+}
+variable "tags" {
+ description = "Tags to add to all the environment resources, beside the default ones (Environment, Namespace, Terraform)"
+ type = map
+ default = {}
+}
+
+### KEYPAIRS
+variable "key_path" {
+ description = "Where the keypair is localted (e.g. `~/.ssh/id_rsa.pub`)"
+ type = string
+ default = ""
+}
+variable "key_name" {
+ description = "Name to give to the keypair"
+ type = string
+ default = ""
+}
+variable "key_public_key" {
+ description = "Public key value if you didn't provide a path to the key"
+ type = string
+ default = ""
+}
+variable "enable_key_pair" {
+ description = "A boolean flag to enable/disable key pair"
+ type = bool
+ default = true
+}
+
+### VPC
+variable "cluster" {
+ type = string
+}
+variable "cluster_cidr" {
+ type = string
+ default = ""
+}
+variable "azs" {
+ type = list(string)
+ default = []
+}
+variable "private_subnets" {
+ type = list(string)
+ default = []
+}
+variable "public_subnets" {
+ type = list(string)
+ default = []
+}
+variable "compute_subnets" {
+ type = list(string)
+ default = []
+}
+variable "private_subnets_ids" {
+ type = list(string)
+ default = []
+}
+variable "public_subnets_ids" {
+ type = list(string)
+ default = []
+}
+variable "compute_subnets_ids" {
+ type = list(string)
+ default = []
+}
+variable "enable_nat_gateway" {
+ type = bool
+ default = true
+}
+variable "enable_vpn_gateway" {
+ type = bool
+ default = false
+}
+variable "single_nat_gateway" {
+ type = bool
+ default = true
+}
+variable "one_nat_gateway_per_az" {
+ type = bool
+ default = false
+}
+variable "route53_force_destroy" {
+ description = "Destroy R53 zone when VPC is destroyed"
+ type = bool
+ default = false
+}
+variable "enable_dhcp_options" {
+ type = bool
+ default = false
+}
+variable "allowed_http_access_cidrs" {
+ description = "CIDRs that will have HTTP/HTTPs access to the cluster's VPC instances"
+ type = list(string)
+ default = ["0.0.0.0/0"]
+}
+variable "allowed_ssh_access_cidrs" {
+ description = "CIDRs that will have ssh access to the cluster's VPC instances"
+ type = list(string)
+ default = []
+}
+variable "allowed_full_access_cidrs" {
+ description = "CIDRs that will have full access to the cluster's VPC instances"
+ type = list(string)
+ default = []
+}
+
+variable "kms_key_id" {
+ default = ""
+}
+
+variable "key_pair" {
+ default = ""
+}
+
+variable "instance_ami" {
+ type = map(string)
+ default = {}
+}
+
+variable "instance_type" {
+ type = map(string)
+ default = {}
+}
+
+variable "arvados_volume_name" {
+ type = string
+ default = ""
+}
+
+variable "keepstore_count" {
+ type = string
+ default = "1"
+}
+
+variable "root_bd_size" {
+ type = number
+ default = 50
+}
+
+variable "data_bd_size" {
+ type = map(number)
+ default = {}
+}
+
+variable "db_identifier" {
+ type = string
+ default = ""
+}
+
+variable "db_engine_version" {
+ type = string
+ default = "11.8"
+}
+
+variable "db_instance_class" {
+ type = string
+ default = "db.t2.large"
+}
+
+variable "db_allocated_storage" {
+ type = number
+ default = 512
+}
+
+variable "db_storage_encrypted" {
+ type = bool
+ default = true
+}
+
+variable "db_permissions_boundary" {
+ type = string
+ default = ""
+}
+
+variable "manage_vpc" {
+ description = "If you want to manage/create the VPC where the cluster will be deployed with terraform"
+ type = bool
+ default = true
+}
+
+variable "vpc_id" {
+ description = "If you are not managing the vpc with terraform, then you need to provide the VPC ID"
+ type = string
+ default = ""
+}
+
+variable "manage_security_groups" {
+ description = "If you want to manage/create the security groups with terraform"
+ type = bool
+ default = true
+}
+
+variable "vpc_security_group_ids" {
+ description = "If you are not managing the security groups, then you need to provide them"
+ type = map(string)
+ default = {}
+}
+
+variable "vpc_cidr" {
+ type = string
+ default = ""
+
+}
+variable "vpc_subnet_cidrs" {
+ type = list(string)
+ default = []
+}
+variable "r53_domain_name" {
+ description = "Domain which will be appended to the cluster name"
+ type = string
+ default = ""
+}
+
diff --git a/tools/terraform/vpc.tf b/tools/terraform/vpc.tf
new file mode 100644
index 000000000..539c24eb9
--- /dev/null
+++ b/tools/terraform/vpc.tf
@@ -0,0 +1,32 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+module "vpc" {
+ count = var.manage_vpc ? 1 : 0
+ source = "terraform-aws-modules/vpc/aws"
+ version = "2.77.0"
+
+ name = var.cluster
+ cidr = var.cluster_cidr
+
+ azs = var.azs
+ # we'll need internet access in the compute nodes, so the compute_subnets will
+ # go to the private_subnets
+ private_subnets = concat(var.private_subnets, var.compute_subnets)
+ public_subnets = var.public_subnets
+
+ enable_dns_hostnames = true
+ enable_dns_support = true
+
+ enable_s3_endpoint = true
+
+ enable_nat_gateway = var.enable_nat_gateway
+ enable_vpn_gateway = var.enable_vpn_gateway
+ single_nat_gateway = var.single_nat_gateway
+ one_nat_gateway_per_az = var.one_nat_gateway_per_az
+
+ enable_dhcp_options = var.enable_dhcp_options
+ dhcp_options_domain_name = var.r53_domain_name
+ tags = local.resource_tags
+}
diff --git a/tools/terraform/vpc_outputs.tf b/tools/terraform/vpc_outputs.tf
new file mode 100644
index 000000000..a2641199e
--- /dev/null
+++ b/tools/terraform/vpc_outputs.tf
@@ -0,0 +1,40 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "vpc_id" {
+ value = var.manage_vpc ? module.vpc.0.vpc_id : var.vpc_id
+}
+output "cluster" {
+ value = var.cluster
+}
+output "vpc_name" {
+ value = var.manage_vpc ? module.vpc.0.name : var.cluster
+}
+output "vpc_cidr" {
+ value = var.cluster_cidr
+}
+output "vpc_azs" {
+ value = var.manage_vpc ? module.vpc.0.azs : var.azs
+}
+output "vpc_private_subnets_ids" {
+ value = var.manage_vpc ? module.vpc.0.private_subnets : concat(var.private_subnets_ids, var.compute_subnets_ids)
+}
+output "vpc_compute_subnets_ids" {
+ value = var.manage_vpc ? [module.vpc.0.private_subnets[1]] : var.compute_subnets_ids
+}
+output "vpc_public_subnets_ids" {
+ value = var.manage_vpc ? module.vpc.0.public_subnets : var.public_subnets_ids
+}
+output "vpc_nat_public_ips" {
+ value = var.manage_vpc ? module.vpc.0.nat_public_ips : null
+}
+output "vpc_private_subnets_cidr_blocks" {
+ value = var.manage_vpc ? module.vpc.0.private_subnets_cidr_blocks : concat(var.private_subnets, var.compute_subnets)
+}
+output "vpc_compute_subnets_cidr_blocks" {
+ value = var.manage_vpc ? [module.vpc.0.private_subnets_cidr_blocks[1]] : var.compute_subnets
+}
+output "vpc_public_subnets_cidr_blocks" {
+ value = var.manage_vpc ? module.vpc.0.public_subnets_cidr_blocks : var.public_subnets
+}
diff --git a/tools/terraform/workbench_iam_role.tf b/tools/terraform/workbench_iam_role.tf
new file mode 100644
index 000000000..0774b9ca7
--- /dev/null
+++ b/tools/terraform/workbench_iam_role.tf
@@ -0,0 +1,21 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# Assume role for the instance
+resource "aws_iam_role" "workbench_iam_role" {
+ name = "${var.cluster}-workbench-iam-role"
+ assume_role_policy = templatefile("${path.module}/iam_policy_assume_role.json", {})
+}
+
+# Associate letsencrypt modification policy to the role
+resource "aws_iam_role_policy_attachment" "workbench_letsencrypt_route53_policies_attachment" {
+ role = aws_iam_role.workbench_iam_role.name
+ policy_arn = aws_iam_policy.letsencrypt_route53_iam_policy.arn
+}
+
+# Add the role to the instance profile
+resource "aws_iam_instance_profile" "workbench_instance_profile" {
+ name = "workbench_instance_profile"
+ role = "${var.cluster}-workbench-iam-role"
+}
diff --git a/tools/terraform/workbench_iam_role_outputs.tf b/tools/terraform/workbench_iam_role_outputs.tf
new file mode 100644
index 000000000..d8770705a
--- /dev/null
+++ b/tools/terraform/workbench_iam_role_outputs.tf
@@ -0,0 +1,10 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "workbench_iam_role_arn" {
+ value = aws_iam_role.workbench_iam_role.arn
+}
+output "workbench_iam_role_id" {
+ value = aws_iam_role.workbench_iam_role.id
+}
diff --git a/tools/terraform/workbench_instance.tf b/tools/terraform/workbench_instance.tf
new file mode 100644
index 000000000..31e147ee4
--- /dev/null
+++ b/tools/terraform/workbench_instance.tf
@@ -0,0 +1,96 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+module "workbench" {
+ source = "terraform-aws-modules/ec2-instance/aws"
+ version = "~> 2.17.0"
+
+ name = "${var.cluster}-workbench"
+ instance_count = 1
+
+ iam_instance_profile = "workbench_instance_profile"
+ ami = try(var.instance_ami["workbench"], var.instance_ami["default"])
+ instance_type = try(var.instance_type["workbench"], var.instance_type["default"])
+ key_name = var.key_name
+ monitoring = true
+
+ tags = merge({"Name": "${var.cluster}-workbench",
+ "OsType": "LINUX"}, local.resource_tags)
+ volume_tags = merge({"Name": "${var.cluster}-workbench"}, local.resource_tags)
+
+ network_interface = [{
+ device_index = 0,
+ network_interface_id = aws_network_interface.workbench.id,
+ }]
+
+ # associate_public_ip_address = false
+ ebs_optimized = true
+ user_data = templatefile("_user_data.sh", {})
+
+ root_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = var.root_bd_size,
+ delete_on_termination = true,
+ }]
+ ebs_block_device = [{
+ encrypted = true,
+ kms_key_id = var.kms_key_id,
+ volume_size = try(var.data_bd_size["workbench"], var.data_bd_size["default"])
+ delete_on_termination = true,
+ device_name = "xvdh",
+ }]
+}
+
+resource "aws_eip" "cluster_workbench_public_ip" {
+ vpc = true
+ instance = module.workbench.id[0]
+ network_interface = aws_network_interface.workbench.id
+ tags = merge({"Name": "${var.cluster}-workbench-ip"},local.resource_tags)
+}
+
+resource "aws_network_interface" "workbench" {
+ subnet_id = var.manage_vpc ? module.vpc.0.public_subnets[0] : var.public_subnets_ids[0]
+ # private_ips = [cidrhost(var.vpc_subnet_cidrs[0], var.host_number["workbench"])]
+ security_groups = [
+ local.ssh_sg,
+ local.http_sg,
+ local.https_sg,
+ ]
+ tags = merge({"Name": "${var.cluster}-workbench"}, local.resource_tags)
+}
+
+## Public A RRs
+module "workbench_route53_public_records_A" {
+ source = "./modules/aws/route53/records/a"
+ zone_id = module.r53_zone_public.id
+
+ zone_records_A = {
+ "workbench" = {
+ ttl = "300",
+ records = [aws_eip.cluster_workbench_public_ip.public_ip]
+ },
+ "workbench2" = {
+ ttl = "300",
+ records = [aws_eip.cluster_workbench_public_ip.public_ip]
+ },
+ }
+}
+
+## Private A RRs
+module "workbench_route53_private_records_A" {
+ source = "./modules/aws/route53/records/a"
+ zone_id = module.r53_zone_private.id
+
+ zone_records_A = {
+ "workbench" = {
+ ttl = "300",
+ records = aws_network_interface.workbench.private_ips
+ },
+ "workbench2" = {
+ ttl = "300",
+ records = aws_network_interface.workbench.private_ips
+ },
+ }
+}
diff --git a/tools/terraform/workbench_instance_outputs.tf b/tools/terraform/workbench_instance_outputs.tf
new file mode 100644
index 000000000..82e2b9717
--- /dev/null
+++ b/tools/terraform/workbench_instance_outputs.tf
@@ -0,0 +1,19 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+output "workbench_id" {
+ value = module.workbench.id
+}
+output "workbench_private_dns_names" {
+ value = module.workbench.private_dns
+}
+output "workbench_private_ip" {
+ value = module.workbench.private_ip
+}
+output "workbench_private_eni_id" {
+ value = aws_network_interface.workbench.id
+}
+output "workbench_public_ip" {
+ value = aws_eip.cluster_workbench_public_ip.public_ip
+}
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list