[ARVADOS] updated: 1.3.0-1987-g3e8f849f4
Git user
git at public.arvados.org
Tue Dec 10 21:40:56 UTC 2019
Summary of changes:
AUTHORS | 1 +
README.md | 8 +--
build/rails-package-scripts/postinst.sh | 8 +--
build/run-library.sh | 55 +++++++--------
build/version-at-commit.sh | 17 +++++
doc/_config.yml | 8 +--
doc/_includes/_install_compute_docker.liquid | 5 +-
doc/_includes/_install_compute_fuse.liquid | 5 --
doc/_includes/_install_docker_cleaner.liquid | 2 +-
doc/admin/upgrading.html.textile.liquid | 2 +-
doc/index.html.liquid | 6 +-
.../install-compute-node.html.textile.liquid | 6 +-
doc/install/index.html.textile.liquid | 9 ++-
.../install-dispatch-cloud.html.textile.liquid | 80 +++++++---------------
doc/user/cwl/cwl-runner.html.textile.liquid | 4 +-
.../cwl/federated-workflows.html.textile.liquid | 4 +-
.../getting_started/community.html.textile.liquid | 2 +-
lib/controller/federation/list.go | 11 ++-
lib/controller/federation/list_test.go | 6 +-
lib/controller/router/router.go | 6 +-
lib/controller/router/router_test.go | 10 ++-
sdk/cli/arvados-cli.gemspec | 5 +-
sdk/go/arvados/client.go | 9 ++-
sdk/go/arvados/fs_collection.go | 13 +++-
sdk/go/arvados/fs_collection_test.go | 24 +++++++
.../arvados/client/api/model/argument/Filter.java | 5 +-
.../arvados/client/logic/collection/FileToken.java | 6 +-
sdk/ruby/arvados.gemspec | 5 +-
.../api/app/models/api_client_authorization.rb | 1 +
services/api/fpm-info.sh | 4 +-
.../api_client_authorizations_controller_test.rb | 9 +++
.../test/integration/container_dispatch_test.rb | 33 +++++++++
services/login-sync/Gemfile.lock | 8 +--
services/login-sync/arvados-login-sync.gemspec | 5 +-
tools/arvbox/bin/arvbox | 1 +
tools/arvbox/lib/arvbox/docker/go-setup.sh | 4 +-
tools/arvbox/lib/arvbox/docker/keep-setup.sh | 2 +-
.../docker/service/arv-git-httpd/run-service | 2 +-
.../service/crunch-dispatch-local/run-service | 4 +-
.../lib/arvbox/docker/service/keep-web/run-service | 2 +-
.../arvbox/docker/service/keepproxy/run-service | 2 +-
.../arvbox/docker/service/websockets/run-service | 2 +-
42 files changed, 230 insertions(+), 171 deletions(-)
create mode 100755 build/version-at-commit.sh
create mode 100644 services/api/test/integration/container_dispatch_test.rb
via 3e8f849f40aed8fe2f6ce7e41c674195d177c52b (commit)
via db31bedb109a4d918830a910654685a9f591cf28 (commit)
via 75b025460ce334b77e116d3455add89f6a4642d4 (commit)
via 318a5819e7ac49305f2fe4a23ce6a07e5bd8b42e (commit)
via 4130c84e6a46d426c3ae0beabd1a8b0a74deda48 (commit)
via 9b3ed4805fda2af391677b8c77d318d916203dcb (commit)
via a8f0b080e63a5aa7fc3538859bfffd8a165ed869 (commit)
via fff0bd39af0a44febf5cfd6e7436c282f42b70a8 (commit)
via 408c99ff902e9318f6b0d21606a5c9294b017862 (commit)
via 38715d47a135a43e6141590d384b4794ef318b8f (commit)
via 69c206fb1c2fd54c7f8b2be008c587cb5e4a638e (commit)
via 9c2fb29f2c493fad7bfe15d3aa93ce3a24b0aa72 (commit)
via 7c633f8b55ddfbe82891ccb906ff1b92e64fe365 (commit)
via 32d4f9f58a9669d32fc846e49b08e41c5df9d299 (commit)
via de3dea3c1246be411a1a9789c70a91997b3cc8ea (commit)
via 5f9e11ae95dc2b768ccc7e5f165be06b6baabdb2 (commit)
via 9a62877a985fef90c729e6075076337e8d0600c0 (commit)
via d08871688722c5303b56e5a76456aec7d10e770b (commit)
via b9b602aac9122c3c956fc2c05fa5fb360d0b18fb (commit)
via e470737688e80651a4d0c11fa0caef2b63ad4fd0 (commit)
via 04e09f7a3f02db183f40933b944c1fff00360e09 (commit)
via 06d6bec4ce5b9cf5ff4af658d0f182e763a38783 (commit)
via c09adb3ad51eca81e21914582a3cbf92816a3700 (commit)
via 5ce55817039488a4445287191025212a14e50de8 (commit)
via e92e0f23b7b9bef63e581c3214a133d7dd3196ed (commit)
via dfcbc87f0e67febb94bc9fc39a3601928534f851 (commit)
via 278e53556cb9623c0450cae6b9e21bab682be58f (commit)
via fb72a4315bfcf64621d023a38d1544319aa3666c (commit)
via 7a2e86058222c4e4ff0a2a74898aa2a091f66021 (commit)
via 3d5e303970a342dd80a617058852a0dab686cfd0 (commit)
from bf2aee923894b06592fa87977787559bb14aac18 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 3e8f849f40aed8fe2f6ce7e41c674195d177c52b
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Dec 10 16:40:35 2019 -0500
15572: Updating dispatcher install
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/_config.yml b/doc/_config.yml
index 2003594da..d9c5ece74 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -212,17 +212,15 @@ navbar:
- Additional services:
- install/install-ws.html.textile.liquid
- install/install-arv-git-httpd.html.textile.liquid
- - install/install-shell-server.html.textile.liquid
+ - install/install-shell-server.html.textile.liquid
+ - Containers API support on cloud:
+ - install/install-dispatch-cloud.html.textile.liquid
- Containers API support on SLURM:
- install/crunch2-slurm/install-prerequisites.html.textile.liquid
- install/crunch2-slurm/install-slurm.html.textile.liquid
- install/crunch2-slurm/install-compute-node.html.textile.liquid
- install/crunch2-slurm/install-dispatch.html.textile.liquid
- install/crunch2-slurm/install-test.html.textile.liquid
- - install/install-nodemanager.html.textile.liquid
- - install/install-compute-ping.html.textile.liquid
- - Containers API support on cloud (beta):
- - install/install-dispatch-cloud.html.textile.liquid
- External dependencies:
- install/install-postgresql.html.textile.liquid
- install/ruby.html.textile.liquid
diff --git a/doc/_includes/_install_compute_docker.liquid b/doc/_includes/_install_compute_docker.liquid
index 69b49e83c..5a3efee74 100644
--- a/doc/_includes/_install_compute_docker.liquid
+++ b/doc/_includes/_install_compute_docker.liquid
@@ -8,16 +8,13 @@ h2. Install Docker
Compute nodes must have Docker installed to run containers. This requires a relatively recent version of Linux (at least upstream version 3.10, or a distribution version with the appropriate patches backported). Follow the "Docker Engine installation documentation":https://docs.docker.com/ for your distribution.
-For Debian-based systems, the Arvados package repository includes a backported @docker.io@ package with a known-good version you can install.
-
h2(#configure_docker_daemon). Configure the Docker daemon
Crunch runs Docker containers with relatively little configuration. You may need to start the Docker daemon with specific options to make sure these jobs run smoothly in your environment. This section highlights options that are useful to most installations. Refer to the "Docker daemon reference":https://docs.docker.com/reference/commandline/daemon/ for complete information about all available options.
The best way to configure these options varies by distribution.
-* If you're using our backported @docker.io@ package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker.io at .
-* If you're using another Debian-based package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker at .
+* If you're using the Debian package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker at .
* On Red Hat-based distributions, you can list these options in the @other_args@ setting in @/etc/sysconfig/docker at .
h3. Default ulimits
diff --git a/doc/_includes/_install_compute_fuse.liquid b/doc/_includes/_install_compute_fuse.liquid
index 449c32c26..b72536272 100644
--- a/doc/_includes/_install_compute_fuse.liquid
+++ b/doc/_includes/_install_compute_fuse.liquid
@@ -10,11 +10,6 @@ FUSE must be configured with the @user_allow_other@ option enabled for Crunch to
<notextile>
<pre>
-# Set the maximum number of FUSE mounts allowed to non-root users.
-# The default is 1000.
-#
-#mount_max = 1000
-
# Allow non-root users to specify the 'allow_other' or 'allow_root'
# mount options.
#
diff --git a/doc/_includes/_install_docker_cleaner.liquid b/doc/_includes/_install_docker_cleaner.liquid
index 5b0e1551e..ecd78390e 100644
--- a/doc/_includes/_install_docker_cleaner.liquid
+++ b/doc/_includes/_install_docker_cleaner.liquid
@@ -6,7 +6,7 @@ SPDX-License-Identifier: CC-BY-SA-3.0
h2. Configure the Docker cleaner
-The arvados-docker-cleaner program removes least recently used Docker images as needed to keep disk usage below a configured limit.
+The @arvados-docker-cleaner@ program removes least recently used Docker images as needed to keep disk usage below a configured limit.
{% include 'notebox_begin' %}
This also removes all containers as soon as they exit, as if they were run with @docker run --rm at . If you need to debug or inspect containers after they stop, temporarily stop arvados-docker-cleaner or configure it with @"RemoveStoppedContainers":"never"@.
diff --git a/doc/install/crunch2-slurm/install-compute-node.html.textile.liquid b/doc/install/crunch2-slurm/install-compute-node.html.textile.liquid
index d8a62a9c9..6c691f29a 100644
--- a/doc/install/crunch2-slurm/install-compute-node.html.textile.liquid
+++ b/doc/install/crunch2-slurm/install-compute-node.html.textile.liquid
@@ -11,19 +11,17 @@ SPDX-License-Identifier: CC-BY-SA-3.0
h2. Install dependencies
-First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
+h3. Centos 7
{% include 'note_python_sc' %}
-On Red Hat-based systems:
-
<notextile>
<pre><code>~$ <span class="userinput">echo 'exclude=python2-llfuse' | sudo tee -a /etc/yum.conf</span>
~$ <span class="userinput">sudo yum install python-arvados-fuse crunch-run arvados-docker-cleaner</span>
</code></pre>
</notextile>
-On Debian-based systems:
+h3. Debian and Ubuntu
<notextile>
<pre><code>~$ <span class="userinput">sudo apt-get install python-arvados-fuse crunch-run arvados-docker-cleaner</span>
diff --git a/doc/install/install-dispatch-cloud.html.textile.liquid b/doc/install/install-dispatch-cloud.html.textile.liquid
index 5d497cc11..772ba548c 100644
--- a/doc/install/install-dispatch-cloud.html.textile.liquid
+++ b/doc/install/install-dispatch-cloud.html.textile.liquid
@@ -2,7 +2,6 @@
layout: default
navsection: installguide
title: Install the cloud dispatcher
-
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
@@ -10,7 +9,11 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-The cloud dispatch service is an *experimental* service for running containers on cloud VMs. It eliminates the need for SLURM, Node Manager, and SLURM dispatcher. It works with Microsoft Azure and Amazon EC2; future versions will also support Google Compute Engine.
+# "Update config.yml":#update-config
+# "Install arvados-dispatch-cloud":#install-packages
+# "Confirm working installation":#confirm-working
+
+The cloud dispatch service is for running containers on cloud VMs. It works with Microsoft Azure and Amazon EC2; future versions will also support Google Compute Engine.
The cloud dispatch service can run on any node that can connect to the Arvados API service, the cloud provider's API, and the SSH service on cloud VMs. It is not resource-intensive, so you can run it on the API server node.
@@ -22,13 +25,9 @@ The cloud dispatch service can run on any node that can connect to the Arvados A
</code></pre>
</notextile>
-h2. Create a dispatcher token
-
-If you haven't already done so, create an Arvados superuser token to use as SystemRootToken in your cluster config file.
-
-{% include 'create_superuser_token' %}
+h2(#update-config). Update config.yml
-h2. Create a private key
+h3. Create a private key
Generate an SSH private key with no passphrase. Save it in the cluster configuration file (see @PrivateKey@ in the example below).
@@ -57,18 +56,12 @@ You can delete the key files after you have copied the private key to your confi
</code></pre>
</notextile>
-h2. Configure the dispatcher
+h3. Configure CloudVMs
Add or update the following portions of your cluster configuration file, @/etc/arvados/config.yml at . Refer to "config.defaults.yml":{{site.baseurl}}/admin/config.html for information about additional configuration options.
<notextile>
-<pre><code>Clusters:
- <span class="userinput">uuid_prefix</span>:
- ManagementToken: xyzzy
- SystemRootToken: <span class="userinput">zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz</span>
- Services:
- Controller:
- ExternalURL: "https://<span class="userinput">uuid_prefix.arvadosapi.com</span>"
+<pre><code> Services:
DispatchCloud:
InternalURLs:
"http://localhost:9006": {}
@@ -109,9 +102,7 @@ Add or update the following portions of your cluster configuration file, @/etc/a
Minimal configuration example for Amazon EC2:
<notextile>
-<pre><code>Clusters:
- <span class="userinput">uuid_prefix</span>:
- Containers:
+<pre><code> Containers:
CloudVMs:
ImageID: ami-01234567890abcdef
Driver: ec2
@@ -130,9 +121,7 @@ Minimal configuration example for Amazon EC2:
Minimal configuration example for Azure:
<notextile>
-<pre><code>Clusters:
- <span class="userinput">uuid_prefix</span>:
- Containers:
+<pre><code> Containers:
CloudVMs:
ImageID: "https://zzzzzzzz.blob.core.windows.net/system/Microsoft.Compute/Images/images/zzzzz-compute-osDisk.55555555-5555-5555-5555-555555555555.vhd"
Driver: azure
@@ -153,25 +142,7 @@ Minimal configuration example for Azure:
</code></pre>
</notextile>
-h2. Test your configuration
-
-First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
-
-Next, install the arvados-server package.
-
-On Red Hat-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo yum install arvados-server</span>
-</code></pre>
-</notextile>
-
-On Debian-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get install arvados-server</span>
-</code></pre>
-</notextile>
+h3. Test your configuration
Run the @cloudtest@ tool to verify that your configuration works. This creates a new cloud VM, confirms that it boots correctly and accepts your configured SSH private key, and shuts it down.
@@ -182,32 +153,31 @@ Run the @cloudtest@ tool to verify that your configuration works. This creates a
Refer to the "cloudtest tool documentation":../admin/cloudtest.html for more information.
-h2. Install the dispatcher
+h2. Install arvados-dispatch-cloud
+
+{% include 'notebox_begin' %}
+
+The arvados-dispatch-cloud package includes configuration files for systemd. If you're using a different init system, configure a service to start and stop an @arvados-dispatch-cloud@ process as desired.
-On Red Hat-based systems:
+{% include 'notebox_end' %}
+
+h3. Centos 7
<notextile>
-<pre><code>~$ <span class="userinput">sudo yum install arvados-dispatch-cloud</span>
-~$ <span class="userinput">sudo systemctl enable arvados-dispatch-cloud</span>
+<pre><code># <span class="userinput">yum install arvados-dispatch-cloud</span>
</code></pre>
</notextile>
-On Debian-based systems:
+h3. Debian and Ubuntu
<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get install arvados-dispatch-cloud</span>
+<pre><code># <span class="userinput">apt-get --no-install-recommends install arvados-dispatch-cloud</span>
</code></pre>
</notextile>
-{% include 'notebox_begin' %}
-
-The arvados-dispatch-cloud package includes configuration files for systemd. If you're using a different init system, configure a service to start and stop an @arvados-dispatch-cloud@ process as desired.
-
-{% include 'notebox_end' %}
-
-h2. Verify the dispatcher is running
+h2(#confirm-working). Confirm working installation
-Use your ManagementToken to test the dispatcher's metrics endpoint.
+Use your @ManagementToken@ to test the dispatcher's metrics endpoint.
<notextile>
<pre><code>~$ <span class="userinput">token="xyzzy"</span>
commit db31bedb109a4d918830a910654685a9f591cf28
Merge: bf2aee923 75b025460
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Dec 10 16:14:00 2019 -0500
Merge branch 'master' into 15572-new-install-docs
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --cc doc/install/index.html.textile.liquid
index fd1511468,527b62cf3..efa671db8
--- a/doc/install/index.html.textile.liquid
+++ b/doc/install/index.html.textile.liquid
@@@ -9,7 -9,7 +9,7 @@@ Copyright (C) The Arvados Authors. All
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
- Arvados components run on GNU/Linux systems, and supports AWS and Azure cloud platforms as well as on-premises installs. Arvados supports Debian and derivatives such as Ubuntu, as well as Red Hat and derivatives such as CentOS. Although Arvados development is sponsored by Veritas Genetics which offers commercial support, "Arvados is Free Software":{{site.baseurl}}/copying/copying.html and we encourage self supported/community supported installations.
-Arvados components run on GNU/Linux systems, and supports multiple cloud operating stacks. Arvados supports Debian and derivatives such as Ubuntu, as well as Red Hat and derivatives such as CentOS. "Arvados is Free Software":{{site.baseurl}}/copying/copying.html and self-install installations are not limited in any way. Commercial support and development are also available from "Curii Corporation.":mailto:info at curii.com
++Arvados components run on GNU/Linux systems, and supports AWS, GCP and Azure cloud platforms as well as on-premises installs. Arvados supports Debian and derivatives such as Ubuntu, as well as Red Hat and derivatives such as CentOS. "Arvados is Free Software":{{site.baseurl}}/copying/copying.html and self-install installations are not limited in any way. Commercial support and development are also available from "Curii Corporation.":mailto:info at curii.com
Arvados components can be installed and configured in a number of different ways.
diff --cc services/login-sync/Gemfile.lock
index 5f163e87c,5f163e87c..1d89604ae
--- a/services/login-sync/Gemfile.lock
+++ b/services/login-sync/Gemfile.lock
@@@ -1,7 -1,7 +1,7 @@@
PATH
remote: .
specs:
-- arvados-login-sync (1.4.1.20190930204434)
++ arvados-login-sync (1.3.1.pre20191210204053)
arvados (~> 1.3.0, >= 1.3.0)
faraday (< 0.16)
@@@ -23,7 -23,7 +23,7 @@@ GE
i18n (~> 0)
json (>= 1.7.7, < 3)
jwt (>= 0.1.5, < 2)
-- arvados-google-api-client (0.8.7.2)
++ arvados-google-api-client (0.8.7.3)
activesupport (>= 3.2, < 5.1)
addressable (~> 2.3)
autoparse (~> 0.3)
@@@ -60,7 -60,7 +60,7 @@@
minitest (5.11.3)
mocha (1.8.0)
metaclass (~> 0.0.1)
-- multi_json (1.13.1)
++ multi_json (1.14.1)
multipart-post (2.1.1)
os (1.0.1)
public_suffix (4.0.1)
@@@ -85,4 -85,4 +85,4 @@@ DEPENDENCIE
rake
BUNDLED WITH
-- 1.17.3
++ 2.0.2
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list