[ARVADOS] updated: fa92bc75955f1a81652241addf3c6b24c594b55e
git at public.curoverse.com
git at public.curoverse.com
Mon Jul 6 17:37:15 EDT 2015
Summary of changes:
doc/install/install-compute-node.html.textile.liquid | 4 ++--
doc/sdk/perl/index.html.textile.liquid | 2 +-
doc/sdk/python/sdk-python.html.textile.liquid | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
discards 5298486af0f743cf5d4b2096698d06a799c0d58a (commit)
discards d79d6e77e5ee1da3a5d2d63b2a9fd9414df7314a (commit)
discards f6f12a6e838465ca3261a2ffdcb853c35f9eee67 (commit)
via fa92bc75955f1a81652241addf3c6b24c594b55e (commit)
via 80a9fa664c8370436a0190f483410e5af05e26c3 (commit)
via f721928e89bb06f6df21432da521c9163ff722e3 (commit)
This update added new revisions after undoing existing revisions. That is
to say, the old revision is not a strict subset of the new revision. This
situation occurs when you --force push a change and generate a repository
containing something like this:
* -- * -- B -- O -- O -- O (5298486af0f743cf5d4b2096698d06a799c0d58a)
\
N -- N -- N (fa92bc75955f1a81652241addf3c6b24c594b55e)
When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit fa92bc75955f1a81652241addf3c6b24c594b55e
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jul 6 17:29:03 2015 -0400
6455: Include more steps in Crunch install docs.
* Document the necessary API server configuration change for
crunch-dispatch.
* Document Docker installation.
* Note compute node crunch users need to be in the fuse group.
diff --git a/doc/install/install-compute-node.html.textile.liquid b/doc/install/install-compute-node.html.textile.liquid
index b3ab375..767b8e3 100644
--- a/doc/install/install-compute-node.html.textile.liquid
+++ b/doc/install/install-compute-node.html.textile.liquid
@@ -6,7 +6,7 @@ title: Install a compute node
h2. Install dependencies
-First, "add the appropriate package repository for your distribution":/install/install-manual-prerequisites.html#repos.
+First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
On Debian-based systems:
@@ -15,24 +15,30 @@ On Debian-based systems:
</code></pre>
</notextile>
-h2. Install slurm and munge
+On Red Hat-based systems:
<notextile>
-<pre><code>~$ <span class="userinput">sudo /usr/bin/apt-get install slurm-llnl munge</span>
+<pre><code>~$ <span class="userinput">sudo yum install perl python27-python-virtualenv fuse python27-python-arvados-python-client python27-python-arvados-fuse crunchstat iptables ca-certificates</span>
</code></pre>
</notextile>
-h2. Copy configuration files from the dispatcher (api)
+{% include 'note_python27_sc' %}
-The @/etc/slurm-llnl/slurm.conf@ and @/etc/munge/munge.key@ files need to be identicaly across the dispatcher and all compute nodes. Copy the files you created in the "Install the Crunch dispatcher":{{site.baseurl}} step to this compute node.
+h2. Set up Docker
-h2. Crunch user account
+Compute nodes must have Docker installed to run jobs inside containers. This requires a relatively recent version of Linux (at least upstream version 3.10, or a distribution version with the appropriate patches backported). Follow the "Docker Engine installation documentation":https://docs.docker.com/ for your distribution.
-* @adduser crunch@
+For Debian-based systems, the Arvados package repository includes a backported @docker.io@ package with a known-good version you can install.
-The crunch user should have the same UID, GID, and home directory on all compute nodes and on the dispatcher (api server).
+h2. Set up SLURM
-h2. Configure fuse
+Install SLURM following "the same process you used to install the Crunch dispatcher":{{ site.baseurl }}/install/install-crunch-dispatch.html#slurm.
+
+h2. Copy configuration files from the dispatcher (API server)
+
+The @/etc/slurm-llnl/slurm.conf@ and @/etc/munge/munge.key@ files need to be identicaly across the dispatcher and all compute nodes. Copy the files you created in the "Install the Crunch dispatcher":{{site.baseurl}} step to this compute node.
+
+h2. Configure FUSE
Install this file as @/etc/fuse.conf@:
@@ -50,6 +56,17 @@ user_allow_other
</pre>
</notextile>
+h2. Crunch user account
+
+Create a Crunch user account, and add it to the @fuse@ and @docker@ groups so it can use those tools:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo useradd --groups fuse,docker crunch</span>
+</code></pre>
+</notextile>
+
+The crunch user should have the same UID, GID, and home directory across all compute nodes and the dispatcher (API server).
+
h2. Tell the API server about this compute node
Load your API superuser token on the compute node:
diff --git a/doc/install/install-crunch-dispatch.html.textile.liquid b/doc/install/install-crunch-dispatch.html.textile.liquid
index afb67cd..300f55a 100644
--- a/doc/install/install-crunch-dispatch.html.textile.liquid
+++ b/doc/install/install-crunch-dispatch.html.textile.liquid
@@ -91,13 +91,13 @@ Please make sure to update the value of the @ControlMachine@ parameter to the ho
h2. Enable SLURM job dispatch
-h4. Crunch user account
+In your API server's @application.yml@ configuration file, add the line @crunch_job_wrapper: :slurm_immediate@ under the appropriate section. (The second colon is not a typo. It denotes a Ruby symbol.)
-* @adduser crunch@
+h2. Crunch user account
-The crunch user should have the same UID, GID, and home directory on all compute nodes and on the dispatcher (api server).
+Run @sudo adduser crunch at . The crunch user should have the same UID, GID, and home directory on all compute nodes and on the dispatcher (API server).
-h4. Repositories
+h2. Git Repositories
Crunch scripts must be in Git repositories in the directory configured as @git_repositories_dir@/*.git (see the "API server installation":install-api-server.html#git_repositories_dir).
commit 80a9fa664c8370436a0190f483410e5af05e26c3
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jul 6 17:27:08 2015 -0400
6438: Modernize Rails server installation guides.
Document the package-based installation procedure, and associated
Nginx configuration.
diff --git a/doc/install/install-api-server.html.textile.liquid b/doc/install/install-api-server.html.textile.liquid
index b65fe69..8f9de33 100644
--- a/doc/install/install-api-server.html.textile.liquid
+++ b/doc/install/install-api-server.html.textile.liquid
@@ -4,72 +4,116 @@ navsection: installguide
title: Install the API server
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
h2. Install prerequisites
-<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get install \
- bison build-essential gettext libcurl3 libcurl3-gnutls \
- libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
- libssl-dev libxslt1.1 postgresql git wget zlib1g-dev
-</span></code></pre></notextile>
+The Arvados package repository includes an API server package that can help automate much of the deployment. It requires:
+
+* PostgreSQL 9.0+
+* "Ruby 2.1 and bundler":install-manual-prerequisites-ruby.html
+* Build tools and the curl and PostgreSQL development libraries, to build gem dependencies
+* Nginx
-Also make sure you have "Ruby and bundler":install-manual-prerequisites-ruby.html installed.
+On older distributions, you may need to use a backports repository to satisfy these requirements. For example, on older Red Hat-based systems, consider using the "postgresql92":https://www.softwarecollections.org/en/scls/rhscl/postgresql92/ and "nginx16":https://www.softwarecollections.org/en/scls/rhscl/nginx16/ Software Collections.
-h2. Download the source tree
+On a Debian-based system, install the following packages:
<notextile>
-<pre><code>~$ <span class="userinput">cd $HOME</span> # (or wherever you want to install)
-~$ <span class="userinput">git clone https://github.com/curoverse/arvados.git</span>
-</code></pre></notextile>
+<pre><code>~$ <span class="userinput">sudo apt-get install bison build-essential libpq-dev libcurl4-openssl-dev postgresql git nginx arvados-api-server</span>
+</code></pre>
+</notextile>
+
+On a Red Hat-based system, install the following packages:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo yum install bison make automake gcc gcc-c++ libcurl-devel postgresql-server postgresql-devel nginx git arvados-api-server</span>
+</code></pre>
+</notextile>
-See also: "Downloading the source code":https://arvados.org/projects/arvados/wiki/Download on the Arvados wiki.
+{% include 'notebox_begin' %}
+
+If you intend to use specific versions of these packages from Software Collections, you may have to adapt some of the package names to match; e.g., @postgresql92-postgresql-server postgresql92-postgresql-devel nginx16 at .
+
+{% include 'notebox_end' %}
+
+h2. Set up the database
-The API server is in @services/api@ in the source tree.
+Generate a new database password. Nobody ever needs to memorize it or type it, so we'll make a strong one:
+
+<notextile>
+<pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**128).to_s(36)'</span>
+6gqa1vu492idd7yca9tfandj3
+</code></pre></notextile>
-h2. Install gem dependencies
+Create a new database user.
<notextile>
-<pre><code>~$ <span class="userinput">cd arvados/services/api</span>
-~/arvados/services/api$ <span class="userinput">bundle install</span>
+<pre><code>~$ <span class="userinput">sudo -u postgres createuser --encrypted -R -S --pwprompt arvados</span>
+[sudo] password for <b>you</b>: <span class="userinput">yourpassword</span>
+Enter password for new role: <span class="userinput">paste-password-you-generated</span>
+Enter it again: <span class="userinput">paste-password-again</span>
</code></pre></notextile>
-h2. Choose your environment
+{% include 'notebox_begin' %}
+
+This user setup assumes that your PostgreSQL is configured to accept password authentication. Red Hat systems use ident-based authentication by default. You may need to either adapt the user creation, or reconfigure PostgreSQL (in @pg_hba.conf@) to accept password authentication.
-The API server can be run in @development@ or in @production@ mode. Unless this installation is going to be used for development on the Arvados API server itself, you should run it in @production@ mode.
+{% include 'notebox_end' %}
-Copy the example environment file for your environment. For example, if you choose @production@:
+Create the database:
<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">cp -i config/environments/production.rb.example config/environments/production.rb</span>
-</code></pre></notextile>
+<pre><code>~$ <span class="userinput">sudo -u postgres createdb arvados_production -T template0 -E UTF8 -O arvados</span>
+</code></pre>
+</notextile>
-h2. Configure the API server
+h2. Set up configuration files
-First, copy the example configuration file:
+The API server package uses configuration files that you write to @/etc/arvados/api@ and ensures they're consistently deployed. Create this directory and copy the example configuration files to it:
<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">cp -i config/application.yml.example config/application.yml</span>
-</code></pre></notextile>
+<pre><code>~$ <span class="userinput">sudo mkdir -p /etc/arvados/api</span>
+~$ <span class="userinput">sudo chmod 700 /etc/arvados/api</span>
+~$ <span class="userinput">cd /var/www/arvados-api/current</span>
+/var/www/arvados-api/current$ <span class="userinput">sudo cp config/initializers/omniauth.rb.example /etc/arvados/api/omniauth.rb</span>
+/var/www/arvados-api/current$ <span class="userinput">sudo cp config/database.yml.sample /etc/arvados/api/database.yml</span>
+/var/www/arvados-api/current$ <span class="userinput">sudo cp config/application.yml.example /etc/arvados/api/application.yml</span>
+</code></pre>
+</notextile>
+
+h2(#omniauth). Configure Omniauth
+
+Edit @/etc/arvados/api/omniauth.rb@ to configure the SSO server for authentication. @APP_ID@ and @APP_SECRET@ correspond to the @app_id@ and @app_secret@ set in "Create arvados-server client for Single Sign On (SSO)":install-sso.html#client and @CUSTOM_PROVIDER_URL@ is the address of your SSO server.
+
+<notextile>
+<pre><code>APP_ID = 'arvados-server'
+APP_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
+CUSTOM_PROVIDER_URL = 'https://sso.example.com/'
+</code></pre>
+</notextile>
-The API server reads the @config/application.yml@ file, as well as the @config/application.defaults.yml@ file. Values in @config/application.yml@ take precedence over the defaults that are defined in @config/application.defaults.yml at . The @config/application.yml.example@ file is not read by the API server and is provided for installation convenience, only.
+h2. Configure the database connection
-Consult @config/application.default.yml@ for a full list of configuration options. Always put your local configuration in @config/application.yml@, never edit @config/application.default.yml at .
+Edit @/etc/arvados/api/database.yml@ and replace the @xxxxxxxx@ database password placeholders with the PostgreSQL password you generated above.
+
+h2. Configure the API server
+
+Edit @/etc/arvados/api/application.yml@ following the instructions below. The deployment script will consistently deploy this to the API server's configuration directory. The API server reads both @application.yml@ and its own @config/application.defaults.yml@ file. Values in @application.yml@ take precedence over the defaults that are defined in @config/application.defaults.yml at . The @config/application.yml.example@ file is not read by the API server and is provided for installation convenience only.
+
+Consult @config/application.default.yml@ for a full list of configuration options. Always put your local configuration in @/etc/arvados/api/application.yml@—never edit @config/application.default.yml at .
h3(#uuid_prefix). uuid_prefix
-Define your @uuid_prefix@ in @config/application.yml@ by setting the @uuid_prefix@ field in the section for your environment. This prefix is used for all database identifiers to identify the record as originating from this site. It must be exactly 5 alphanumeric characters (lowercase ASCII letters and digits).
+Define your @uuid_prefix@ in @application.yml@ by setting the @uuid_prefix@ field in the section for your environment. This prefix is used for all database identifiers to identify the record as originating from this site. It must be exactly 5 alphanumeric characters (lowercase ASCII letters and digits).
h3(#git_repositories_dir). git_repositories_dir
-This field defaults to @/var/lib/arvados/git at . You can override the value by defining it in @config/application.yml at .
+This field defaults to @/var/lib/arvados/git at . You can override the value by defining it in @application.yml at .
Make sure a clone of the arvados repository exists in @git_repositories_dir at .
<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git</span>
-~/arvados/services/api$ <span class="userinput">sudo git clone --bare ../../.git /var/lib/arvados/git/arvados.git</span>
+<pre><code>~$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git</span>
+~$ <span class="userinput">sudo git clone --bare ../../.git /var/lib/arvados/git/arvados.git</span>
</code></pre></notextile>
h3. secret_token
@@ -77,7 +121,7 @@ h3. secret_token
Generate a new secret token for signing cookies:
<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">ruby -e 'puts rand(2**400).to_s(36)'</span>
+<pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**400).to_s(36)'</span>
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
</code></pre></notextile>
@@ -93,91 +137,120 @@ Fill in the url of your workbench application in @workbench_address@, for exampl
https://workbench.@prefix_uuid@.your.domain
-h3. other options
+h3. Other options
-Consult @application.default.yml@ for a full list of configuration options. Always put your local configuration in @application.yml@ instead of editing @application.default.yml at .
+Consult @/var/www/arvados-api/current/config/application.default.yml@ for a full list of configuration options. Always put your local configuration in @application.yml@ instead of editing @application.default.yml at .
-h2. Set up the database
+h2. Prepare the API server deployment
-Generate a new database password. Nobody ever needs to memorize it or type it, so we'll make a strong one:
+Now that all your configuration is in place, run @/usr/local/bin/arvados-api-server-upgrade.sh at . This will install and check your configuration, install necessary gems, and run any necessary database setup.
+{% include 'notebox_begin' %}
+You can safely ignore the following error message you may see when loading the database structure:
<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">ruby -e 'puts rand(2**128).to_s(36)'</span>
-6gqa1vu492idd7yca9tfandj3
-</code></pre></notextile>
+<pre><code>ERROR: must be owner of extension plpgsql</code></pre></notextile>
+{% include 'notebox_end' %}
-Create a new database user with permission to create its own databases.
+This command aborts when it encounters an error. It's safe to rerun multiple times, so if there's a problem with your configuration, you can fix that and try again.
-<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">sudo -u postgres createuser --createdb --encrypted -R -S --pwprompt arvados</span>
-[sudo] password for <b>you</b>: <span class="userinput">yourpassword</span>
-Enter password for new role: <span class="userinput">paste-password-you-generated</span>
-Enter it again: <span class="userinput">paste-password-again</span>
-</code></pre></notextile>
+h2. Set up Web servers
-Configure API server to connect to your database by creating and updating @config/database.yml at . Replace the @xxxxxxxx@ database password placeholders with the new password you generated above.
+For best performance, we recommend you use Nginx as your Web server front-end, with a Passenger backend for the main API server and a Puma backend for API server Websockets. To do that:
<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">cp -i config/database.yml.sample config/database.yml</span>
-~/arvados/services/api$ <span class="userinput">edit config/database.yml</span>
-</code></pre></notextile>
+<ol>
+<li>Install Nginx via your distribution or a backports repository.</li>
-Create and initialize the database. If you are planning a production system, choose the @production@ rails environment, otherwise use @development at .
+<li><a href="https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html">Install Phusion Passenger for Nginx</a>.</li>
-<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">RAILS_ENV=production bundle exec rake db:setup</span>
-</code></pre></notextile>
+<li><p>Puma is already included with the API server's gems. We recommend you use a tool like <a href="http://smarden.org/runit/">runit</a> or something similar. Here's a sample run script for that:</p>
-Alternatively, if the database user you intend to use for the API server is not allowed to create new databases, you can create the database first and then populate it with rake. Be sure to adjust the database name if you are using the @development@ environment. This sequence of commands is functionally equivalent to the rake db:setup command above.
+<pre><code>#!/bin/bash
-<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">su postgres createdb arvados_production -E UTF8 -O arvados</span>
-~/arvados/services/api$ <span class="userinput">RAILS_ENV=production bundle exec rake db:structure:load</span>
-~/arvados/services/api$ <span class="userinput">RAILS_ENV=production bundle exec rake db:seed</span>
-</code></pre></notextile>
+set -e
+# Uncomment the line below if you're using RVM.
+#source /etc/profile.d/rvm.sh
-{% include 'notebox_begin' %}
-You can safely ignore the following error message you may see when loading the database structure:
-<notextile>
-<pre><code>ERROR: must be owner of extension plpgsql</code></pre></notextile>
-{% include 'notebox_end' %}
+envdir="/etc/sv/puma/env"
+root=/etc/sv/puma
+echo "Starting puma from ${root}"
+cd $root
+mkdir -p "${envdir}"
+exec 2>&1
+cd /var/www/arvados-api/current
+# You may need to change arguments below to match your deployment, especially -u.
+exec chpst -e "${envdir}" -m 1073741824 -u www-data:www-data bundle exec puma -t 0:512 -e production -b tcp://127.0.0.1:8100
+</code></pre>
+</li>
-h2(#omniauth). Set up omniauth
+<li><p>Edit the http section of your Nginx configuration to run the Passenger server, and act as a front-end for both it and Puma. You might add a block like the following, adding SSL and logging parameters to taste:</p>
-First copy the omniauth configuration file:
+<pre><code>server {
+ listen 127.0.0.1:8000;
+ server_name localhost-api;
-<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">cp -i config/initializers/omniauth.rb.example config/initializers/omniauth.rb
-</code></pre></notextile>
+ root /var/www/arvados-api/current/public;
+ index index.html index.htm index.php;
-Edit @config/initializers/omniauth.rb@ to configure the SSO server for authentication. @APP_ID@ and @APP_SECRET@ correspond to the @app_id@ and @app_secret@ set in "Create arvados-server client for Single Sign On (SSO)":install-sso.html#client and @CUSTOM_PROVIDER_URL@ is the address of your SSO server.
+ passenger_enabled on;
+ # If you're using RVM, uncomment the line below.
+ #passenger_ruby /usr/local/rvm/wrappers/default/ruby;
+}
-<notextile>
-<pre><code>APP_ID = 'arvados-server'
-APP_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
-CUSTOM_PROVIDER_URL = 'https://sso.example.com/'
-</code></pre>
-</notextile>
+upstream api {
+ server 127.0.0.1:8000 fail_timeout=10s;
+}
-h2. Start the API server
+upstream websockets {
+ # The address below must match the one specified in puma's -b option.
+ server 127.0.0.1:8100 fail_timeout=10s;
+}
-h3. Development environment
+proxy_http_version 1.1;
-If you plan to run in development mode, you can now run the development server this way:
+server {
+ listen <span class="userinput">[your public IP address]</span>:443 ssl;
+ server_name <span class="userinput">uuid-prefix.your.domain</span>;
-<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">bundle exec rails server --port=3030
-</code></pre></notextile>
+ ssl on;
-h3. Production environment
+ index index.html index.htm index.php;
-We recommend "Passenger":https://www.phusionpassenger.com/ to run the API server in production.
+ location / {
+ proxy_pass http://api;
+ proxy_redirect off;
-Point it to the services/api directory in the source tree.
+ proxy_set_header X-Forwarded-Proto https;
+ proxy_set_header Host $http_host;
+ proxy_set_header X-External-Client $external_client;
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ }
+}
-To enable streaming so users can monitor crunch jobs in real time, make sure to add the following to your Passenger configuration:
+server {
+ listen <span class="userinput">[your public IP address]</span>:443 ssl;
+ server_name ws.<span class="userinput">uuid-prefix.your.domain</span>;
-<notextile>
-<pre><code><span class="userinput">PassengerBufferResponse off</span>
+ ssl on;
+
+ index index.html index.htm index.php;
+
+ location / {
+ proxy_pass http://websockets;
+ proxy_redirect off;
+
+ proxy_set_header Upgrade $http_upgrade;
+ proxy_set_header Connection "upgrade";
+ proxy_set_header Host $host;
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ }
+}
</code></pre>
+</li>
+
+<li>Restart Nginx.</li>
+
+</ol>
</notextile>
diff --git a/doc/install/install-workbench-app.html.textile.liquid b/doc/install/install-workbench-app.html.textile.liquid
index 4131849..1a65f2d 100644
--- a/doc/install/install-workbench-app.html.textile.liquid
+++ b/doc/install/install-workbench-app.html.textile.liquid
@@ -4,86 +4,65 @@ navsection: installguide
title: Install Workbench
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
h2. Install prerequisites
-<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get install \
- bison build-essential gettext libcurl3 libcurl3-gnutls \
- libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
- libssl-dev libxslt1.1 git wget zlib1g-dev graphviz
-</span></code></pre></notextile>
+The Arvados package repository includes Workbench server package that can help automate much of the deployment. It requires:
-Also make sure you have "Ruby and bundler":install-manual-prerequisites-ruby.html installed.
+* "Ruby 2.1 and bundler":install-manual-prerequisites-ruby.html
+* The Arvados Python SDK
+* Graphviz
+* Build tools to build gem dependencies
+* Nginx
Workbench doesn't need its own database, so it does not need to have PostgreSQL installed.
-h2. Download the source tree
-
-<notextile>
-<pre><code>~$ <span class="userinput">cd $HOME</span> # (or wherever you want to install)
-~$ <span class="userinput">git clone https://github.com/curoverse/arvados.git</span>
-</code></pre></notextile>
-
-See also: "Downloading the source code":https://arvados.org/projects/arvados/wiki/Download on the Arvados wiki.
-
-The Workbench application is in @apps/workbench@ in the source tree.
+On older distributions, you may need to use a backports repository to satisfy these requirements. For example, on older Red Hat-based systems, consider using the "nginx16":https://www.softwarecollections.org/en/scls/rhscl/nginx16/ Software Collection.
-h2. Install gem dependencies
+On a Debian-based system, install the following packages:
<notextile>
-<pre><code>~$ <span class="userinput">cd arvados/apps/workbench</span>
-~/arvados/apps/workbench$ <span class="userinput">bundle install</span>
+<pre><code>~$ <span class="userinput">sudo apt-get install bison build-essential graphviz git nginx python-arvados-python-client arvados-workbench</span>
</code></pre>
</notextile>
-Alternatively, if you don't have sudo/root privileges on the host, install the gems in your own directory instead of installing them system-wide:
+On a Red Hat-based system, install the following packages:
<notextile>
-<pre><code>~$ <span class="userinput">cd arvados/apps/workbench</span>
-~/arvados/apps/workbench$ <span class="userinput">bundle install --path=vendor/bundle</span>
-</code></pre></notextile>
-
-The @bundle install@ command might produce a warning about the themes_for_rails gem. This is OK:
-
-<notextile>
-<pre><code>themes_for_rails at /home/<b>you</b>/.rvm/gems/ruby-2.1.1/bundler/gems/themes_for_rails-1fd2d7897d75 did not have a valid gemspec.
-This prevents bundler from installing bins or native extensions, but that may not affect its functionality.
-The validation message from Rubygems was:
- duplicate dependency on rails (= 3.0.11, development), (>= 3.0.0) use:
- add_runtime_dependency 'rails', '= 3.0.11', '>= 3.0.0'
-Using themes_for_rails (0.5.1) from https://github.com/holtkampw/themes_for_rails (at 1fd2d78)
-</code></pre></notextile>
+<pre><code>~$ <span class="userinput">sudo yum install bison make automake gcc gcc-c++ graphviz git nginx python27-python-arvados-python-client arvados-workbench</span>
+</code></pre>
+</notextile>
-h2. Choose your environment
+{% include 'notebox_begin' %}
-The Workbench application can be run in @development@ or in @production@ mode. Unless this installation is going to be used for development on the Workbench applicatoin itself, you should run it in @production@ mode.
+If you intend to use specific versions of these packages from Software Collections, you may have to adapt some of the package names to match; e.g., @nginx16 at .
-Copy the example environment file for your environment. For example, if you choose @production@:
+{% include 'notebox_end' %}
-<notextile>
-<pre><code>~/arvados/apps/workbench$ <span class="userinput">cp -i config/environments/production.rb.example config/environments/production.rb</span>
-</code></pre></notextile>
+{% include 'note_python27_sc' %}
-h2. Configure the Workbench application
+h2. Set up configuration files
-First, copy the example configuration file:
+The Workbench server package uses configuration files that you write to @/etc/arvados/workbench@ and ensures they're consistently deployed. Create this directory and copy the example configuration files to it:
<notextile>
-<pre><code>~/arvados/apps/workbench$ <span class="userinput">cp -i config/application.yml.example config/application.yml</span>
-</code></pre></notextile>
+<pre><code>~$ <span class="userinput">sudo mkdir -p /etc/arvados/workbench</span>
+~$ <span class="userinput">sudo chmod 700 /etc/arvados/workbench</span>
+~$ <span class="userinput">sudo cp /var/www/arvados-workbench/current/config/application.yml.example /etc/arvados/workbench/application.yml</span>
+</code></pre>
+</notextile>
+
+h2. Configure Workbench
-The Workbench application reads the @config/application.yml@ file, as well as the @config/application.defaults.yml@ file. Values in @config/application.yml@ take precedence over the defaults that are defined in @config/application.defaults.yml at . The @config/application.yml.example@ file is not read by the Workbench application and is provided for installation convenience, only.
+Edit @/etc/arvados/workbench/application.yml@ following the instructions below. The deployment script will consistently deploy this to Workbench's configuration directory. Workbench reads both @application.yml@ and its own @config/application.defaults.yml@ file. Values in @application.yml@ take precedence over the defaults that are defined in @config/application.defaults.yml at . The @config/application.yml.example@ file is not read by Workbench and is provided for installation convenience only.
-Consult @config/application.default.yml@ for a full list of configuration options. Always put your local configuration in @config/application.yml@, never edit @config/application.default.yml at .
+Consult @config/application.default.yml@ for a full list of configuration options. Always put your local configuration in @/etc/arvados/workbench/application.yml@—never edit @config/application.default.yml at .
h3. secret_token
This application needs a secret token. Generate a new secret:
<notextile>
-<pre><code>~/arvados/apps/workbench$ <span class="userinput">ruby -e 'puts rand(2**400).to_s(36)'</span>
+<pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**400).to_s(36)'</span>
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
</code></pre>
</notextile>
@@ -108,27 +87,89 @@ h3. arvados_insecure_https
If the SSL certificate you use for your API server isn't an official certificate signed by a CA, make sure @arvados_insecure_https@ is @true at .
-h3. other options
+h3. Other options
Consult @application.default.yml@ for a full list of configuration options. Always put your local configuration in @application.yml@ instead of editing @application.default.yml at .
-Copy @config/piwik.yml.example@ to @config/piwik.yml@ and edit to suit.
+h2. Configure Piwik
-h2. Start the Workbench application
+In @/var/www/arvados-workbench/current/config@, copy @piwik.yml.example@ to @piwik.yml@ and edit to suit.
-h3. Development environment
+h2. Prepare the Workbench deployment
-If you plan to run in development mode, you can now run the development server this way:
+Now that all your configuration is in place, run @/usr/local/bin/arvados-workbench-upgrade.sh at . This will install and check your configuration, and install necessary gems.
+{% include 'notebox_begin' %}
+You can safely ignore the following error message you may see when installing gems:
<notextile>
-<pre><code>~/arvados/apps/workbench$ <span class="userinput">bundle exec rails server --port=3031</span>
-</code></pre></notextile>
+<pre><code>themes_for_rails at /usr/local/rvm/gems/ruby-2.1.1/bundler/gems/themes_for_rails-1fd2d7897d75 did not have a valid gemspec.
+This prevents bundler from installing bins or native extensions, but that may not affect its functionality.
+The validation message from Rubygems was:
+ duplicate dependency on rails (= 3.0.11, development), (>= 3.0.0) use:
+ add_runtime_dependency 'rails', '= 3.0.11', '>= 3.0.0'
+Using themes_for_rails (0.5.1) from https://github.com/holtkampw/themes_for_rails (at 1fd2d78)
+</code></pre>
+</notextile>
+{% include 'notebox_end' %}
+
+This command aborts when it encounters an error. It's safe to rerun multiple times, so if there's a problem with your configuration, you can fix that and try again.
+
+h2. Set up Web server
+
+For best performance, we recommend you use Nginx as your Web server front-end, with a Passenger backend to serve Workbench. To do that:
+
+<notextile>
+<ol>
+<li>Install Nginx via your distribution or a backports repository.</li>
+
+<li><a href="https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html">Install Phusion Passenger for Nginx</a>.</li>
-h3. Production environment
+<li><p>Edit the http section of your Nginx configuration to run the Passenger server, and act as a front-end for it. You might add a block like the following, adding SSL and logging parameters to taste:</p>
-We recommend "Passenger":https://www.phusionpassenger.com/ to run the API server in production.
+<pre><code>server {
+ listen 127.0.0.1:9000;
+ server_name localhost-workbench;
-Point it to the apps/workbench directory in the source tree.
+ root /var/www/arvados-workbench/current/public;
+ index index.html index.htm index.php;
+
+ passenger_enabled on;
+ # If you're using RVM, uncomment the line below.
+ #passenger_ruby /usr/local/rvm/wrappers/default/ruby;
+}
+
+upstream workbench {
+ server 127.0.0.1:9000 fail_timeout=10s;
+}
+
+proxy_http_version 1.1;
+
+server {
+ listen <span class="userinput">[your public IP address]</span>:443 ssl;
+ server_name workbench.<span class="userinput">uuid-prefix.your.domain</span>;
+
+ ssl on;
+
+ index index.html index.htm index.php;
+
+ location / {
+ proxy_pass http://workbench;
+ proxy_redirect off;
+
+ proxy_set_header X-Forwarded-Proto https;
+ proxy_set_header Host $http_host;
+ proxy_set_header X-External-Client $external_client;
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ }
+}
+</code></pre>
+</li>
+
+<li>Restart Nginx.</li>
+
+</ol>
+</notextile>
h2. Trusted client setting
@@ -136,7 +177,7 @@ Log in to Workbench once to ensure that the Arvados API server has a record of t
In the <strong>API server</strong> project root, start the rails console. Locate the ApiClient record for your Workbench installation (typically, while you're setting this up, the @last@ one in the database is the one you want), then set the @is_trusted@ flag for the appropriate client record:
-<notextile><pre><code>~/arvados/services/api$ <span class="userinput">RAILS_ENV=production bundle exec rails console</span>
+<notextile><pre><code>/var/www/arvados-api/current$ <span class="userinput">RAILS_ENV=production bundle exec rails console</span>
irb(main):001:0> <span class="userinput">wb = ApiClient.all.last; [wb.url_prefix, wb.created_at]</span>
=> ["https://workbench.example.com/", Sat, 19 Apr 2014 03:35:12 UTC +00:00]
irb(main):002:0> <span class="userinput">include CurrentApiClient</span>
@@ -151,7 +192,7 @@ h2(#admin-user). Add an admin user
Next, we're going to use the rails console on the <strong>API server</strong> to activate our own account and give yourself admin privileges:
<notextile>
-<pre><code>~/arvados/services/api$ <span class="userinput">RAILS_ENV=production bundle exec rails console</span>
+<pre><code>/var/www/arvados-api/current$ <span class="userinput">RAILS_ENV=production bundle exec rails console</span>
irb(main):001:0> <span class="userinput">Thread.current[:user] = User.all.select(&:identity_url).last</span>
irb(main):002:0> <span class="userinput">Thread.current[:user].is_admin = true</span>
irb(main):003:0> <span class="userinput">Thread.current[:user].update_attributes is_admin: true, is_active: true</span>
commit f721928e89bb06f6df21432da521c9163ff722e3
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jul 6 17:26:10 2015 -0400
6388: Add CentOS 6.6 instructions to install docs.
diff --git a/doc/_includes/_install_debian_key.liquid b/doc/_includes/_install_debian_key.liquid
new file mode 100644
index 0000000..2b4793c
--- /dev/null
+++ b/doc/_includes/_install_debian_key.liquid
@@ -0,0 +1,4 @@
+<notextile>
+<pre><code>~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
+</code></pre>
+</notextile>
diff --git a/doc/_includes/_install_redhat_key.liquid b/doc/_includes/_install_redhat_key.liquid
new file mode 100644
index 0000000..9fd219b
--- /dev/null
+++ b/doc/_includes/_install_redhat_key.liquid
@@ -0,0 +1,6 @@
+<notextile>
+<pre><code>~$ <span class="userinput">gpg --keyserver pool.sks-keyservers.net --recv-keys 1078ECD7</span>
+~$ <span class="userinput">gpg --armor --export 1078ECD7 >/tmp/curoverse.key</span>
+~$ <span class="userinput">sudo rpm --import /tmp/curoverse.key</span>
+</code></pre>
+</notextile>
diff --git a/doc/_includes/_note_python27_sc.liquid b/doc/_includes/_note_python27_sc.liquid
new file mode 100644
index 0000000..7aad695
--- /dev/null
+++ b/doc/_includes/_note_python27_sc.liquid
@@ -0,0 +1,5 @@
+{% include 'notebox_begin' %}
+
+On older Red Hat-based systems, these packages require the "python27 Software Collection":https://www.softwarecollections.org/en/scls/rhscl/python27/.
+
+{% include 'notebox_end' %}
diff --git a/doc/install/install-arv-git-httpd.html.textile.liquid b/doc/install/install-arv-git-httpd.html.textile.liquid
index 85f643f..44423e1 100644
--- a/doc/install/install-arv-git-httpd.html.textile.liquid
+++ b/doc/install/install-arv-git-httpd.html.textile.liquid
@@ -4,8 +4,6 @@ navsection: installguide
title: Install Git server
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
The arv-git-httpd server provides HTTP access to hosted git repositories, using Arvados authentication tokens instead of passwords. It is intended to be installed on the system where your git repositories are stored, and accessed through a web proxy that provides SSL support.
By convention, we use the following hostname for the git service:
@@ -19,13 +17,17 @@ This hostname should resolve from anywhere on the internet.
h2. Install arv-git-httpd
-First add the Arvados apt repository, and then install the arv-git-httpd package.
+On Debian-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo apt-get install git arv-git-httpd</span>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
<notextile>
-<pre><code>~$ <span class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</span>
-~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get update</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get install arv-git-httpd</span>
+<pre><code>~$ <span class="userinput">sudo yum install git arv-git-httpd</span>
</code></pre>
</notextile>
@@ -47,7 +49,7 @@ fatal: No REQUEST_METHOD from server
</code></pre>
</notextile>
-We recommend running @arv-git-httpd@ under "runit":https://packages.debian.org/search?keywords=runit or something similar.
+We recommend running @arv-git-httpd@ under "runit":http://smarden.org/runit/ or something similar.
Your @run@ script should look something like this:
@@ -65,7 +67,7 @@ This is best achieved by putting a reverse proxy with SSL support in front of ar
h3. Tell the API server about the arv-git-httpd service
-In your API server's config/application.yml file, add the following entry:
+In your API server's @config/application.yml@ file, add the following entry:
<notextile>
<pre><code>git_http_base: git.<span class="userinput">uuid_prefix</span>.your.domain
diff --git a/doc/install/install-compute-node.html.textile.liquid b/doc/install/install-compute-node.html.textile.liquid
index 9a64ac3..b3ab375 100644
--- a/doc/install/install-compute-node.html.textile.liquid
+++ b/doc/install/install-compute-node.html.textile.liquid
@@ -4,17 +4,14 @@ navsection: installguide
title: Install a compute node
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
h2. Install dependencies
-First add the Arvados apt repository, and then install a number of packages.
+First, "add the appropriate package repository for your distribution":/install/install-manual-prerequisites.html#repos.
+
+On Debian-based systems:
<notextile>
-<pre><code>~$ <span class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</span>
-~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get update</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get install python-pip python-pyvcf python-gflags python-google-api-python-client python-virtualenv libattr1-dev libfuse-dev python-dev python-llfuse fuse crunchstat python-arvados-fuse iptables ca-certificates lxc apt-transport-https docker.io</span>
+<pre><code>~$ <span class="userinput">sudo apt-get install perl python-virtualenv fuse python-arvados-python-client python-arvados-fuse crunchstat iptables ca-certificates</span>
</code></pre>
</notextile>
diff --git a/doc/install/install-crunch-dispatch.html.textile.liquid b/doc/install/install-crunch-dispatch.html.textile.liquid
index 4a695ca..afb67cd 100644
--- a/doc/install/install-crunch-dispatch.html.textile.liquid
+++ b/doc/install/install-crunch-dispatch.html.textile.liquid
@@ -5,25 +5,25 @@ title: Install the Crunch dispatcher
...
-
-
The dispatcher normally runs on the same host/VM as the API server.
-h4. Perl SDK dependencies
+h2. Perl SDK dependencies
Install the Perl SDK on the controller.
* See "Perl SDK":{{site.baseurl}}/sdk/perl/index.html page for details.
-h4. Python SDK dependencies
+h2. Python SDK dependencies
Install the Python SDK and CLI tools on controller and all compute nodes.
* See "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html page for details.
-h4. Slurm
+h2(#slurm). Set up SLURM
+
+On the API server, install SLURM and munge, and generate a munge key.
-On the API server, install slurm and munge, and generate a munge key:
+On Debian-based systems:
<notextile>
<pre><code>~$ <span class="userinput">sudo /usr/bin/apt-get install slurm-llnl munge</span>
@@ -31,7 +31,9 @@ On the API server, install slurm and munge, and generate a munge key:
</code></pre>
</notextile>
-Now we need to give slurm a configuration file in @/etc/slurm-llnl/slurm.conf at . Here's an example:
+On Red Hat-based systems, "install SLURM and munge from source following their installation guide":https://computing.llnl.gov/linux/slurm/quickstart_admin.html.
+
+Now we need to give SLURM a configuration file in @/etc/slurm-llnl/slurm.conf at . Here's an example:
<notextile>
<pre>
@@ -85,7 +87,9 @@ PartitionName=compute Nodes=compute[0-255]
</pre>
</notextile>
-Please make sure to update the value of the @ControlMachine@ parameter to the hostname of your dispatcher (api server).
+Please make sure to update the value of the @ControlMachine@ parameter to the hostname of your dispatcher (API server).
+
+h2. Enable SLURM job dispatch
h4. Crunch user account
@@ -114,12 +118,12 @@ Without getting this error:
ArgumentError: Specified script_version does not resolve to a commit
</pre>
-h4. Running jobs
+h2. Running jobs
* @services/api/script/crunch-dispatch.rb@ must be running.
* @crunch-dispatch.rb@ needs @services/crunch/crunch-job@ in its @PATH at .
-* @crunch-job@ needs @sdk/perl/lib@ and @warehouse-apps/libwarehouse-perl/lib@ in its @PERLLIB@
-* @crunch-job@ needs @ARVADOS_API_HOST@ (and, if necessary in a development environment, @ARVADOS_API_HOST_INSECURE@)
+* @crunch-job@ needs the installation path of the Perl SDK in its @PERLLIB at .
+* @crunch-job@ needs the @ARVADOS_API_HOST@ (and, if necessary in a development environment, @ARVADOS_API_HOST_INSECURE@) environment variable set.
Example @/var/service/arvados_crunch_dispatch/run@ script:
diff --git a/doc/install/install-keepproxy.html.textile.liquid b/doc/install/install-keepproxy.html.textile.liquid
index 9e8f878..e800bf4 100644
--- a/doc/install/install-keepproxy.html.textile.liquid
+++ b/doc/install/install-keepproxy.html.textile.liquid
@@ -4,8 +4,6 @@ navsection: installguide
title: Install Keepproxy server
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
The Keepproxy server is a gateway into your Keep storage. Unlike the Keepstore servers, which are only accessible on the local LAN, Keepproxy is designed to provide secure access into Keep from anywhere on the internet.
By convention, we use the following hostname for the Keepproxy:
@@ -20,13 +18,17 @@ This hostname should resolve from anywhere on the internet.
h2. Install Keepproxy
-First add the Arvados apt repository, and then install the Keepproxy package.
+On Debian-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo apt-get install keepproxy</span>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
<notextile>
-<pre><code>~$ <span class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</span>
-~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get update</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get install keepproxy</span>
+<pre><code>~$ <span class="userinput">sudo yum install keepproxy</span>
</code></pre>
</notextile>
@@ -43,7 +45,7 @@ Usage of default:
</code></pre>
</notextile>
-It's recommended to run Keepproxy under "runit":https://packages.debian.org/search?keywords=runit or something similar.
+It's recommended to run Keepproxy under "runit":http://smarden.org/runit/ or something similar.
h3. Create an API token for the Keepproxy server
diff --git a/doc/install/install-keepstore.html.textile.liquid b/doc/install/install-keepstore.html.textile.liquid
index eb53783..72b3c98 100644
--- a/doc/install/install-keepstore.html.textile.liquid
+++ b/doc/install/install-keepstore.html.textile.liquid
@@ -4,8 +4,6 @@ navsection: installguide
title: Install Keepstore servers
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
We are going to install two Keepstore servers. By convention, we use the following hostname pattern:
<div class="offset1">
@@ -19,13 +17,17 @@ Because the Keepstore servers are not directly accessible from the internet, the
h2. Install Keepstore
-First add the Arvados apt repository, and then install the Keepstore package.
+On Debian-based systems:
<notextile>
-<pre><code>~$ <span class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</span>
-~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get update</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get install keepstore</span>
+<pre><code>~$ <span class="userinput">sudo apt-get install keepstore</span>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo yum install keepstore</span>
</code></pre>
</notextile>
@@ -63,11 +65,10 @@ Prepare one or more volumes for Keepstore to use. Simply create a /keep director
2015/05/08 13:44:26 keepstore starting, pid 2765
2015/05/08 13:44:26 Using volume [UnixVolume /mnt/keep] (writable=true)
2015/05/08 13:44:26 listening at :25107
-
</code></pre>
</notextile>
-It's recommended to run Keepstore under "runit":https://packages.debian.org/search?keywords=runit or something similar.
+It's recommended to run Keepstore under "runit":http://smarden.org/runit/ or something similar.
Repeat this section for each Keepstore server you are setting up.
diff --git a/doc/install/install-manual-prerequisites-ruby.html.textile.liquid b/doc/install/install-manual-prerequisites-ruby.html.textile.liquid
index 44c8460..a4f71ba 100644
--- a/doc/install/install-manual-prerequisites-ruby.html.textile.liquid
+++ b/doc/install/install-manual-prerequisites-ruby.html.textile.liquid
@@ -4,14 +4,14 @@ navsection: installguide
title: Install Ruby and bundler
...
-We recommend Ruby >= 2.1.
+We recommend Ruby >= 2.1. We currently require Bundler 1.9 (not 1.10 or above) because of a limitation in some of our gem dependencies.
h2(#rvm). Option 1: Install with rvm
<notextile>
<pre><code>~$ <span class="userinput">gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3</span>
~$ <span class="userinput">\curl -sSL https://get.rvm.io | bash -s stable --ruby=2.1</span>
-~$ <span class="userinput">gem install bundler
+~$ <span class="userinput">gem install -v 1.9.9 bundler
</span></code></pre></notextile>
h2(#fromsource). Option 2: Install from source
@@ -26,5 +26,5 @@ cd ruby-2.1.5
make
sudo make install
-sudo gem install bundler</span>
+sudo gem install -v 1.9.9 bundler</span>
</code></pre></notextile>
diff --git a/doc/install/install-manual-prerequisites.html.textile.liquid b/doc/install/install-manual-prerequisites.html.textile.liquid
index 74a8647..afe2830 100644
--- a/doc/install/install-manual-prerequisites.html.textile.liquid
+++ b/doc/install/install-manual-prerequisites.html.textile.liquid
@@ -21,6 +21,56 @@ table(table table-bordered table-condensed).
The number of Keepstore, shell and compute nodes listed above is a minimum. In a real production installation, you will likely run many more of each of those types of nodes. In such a scenario, you would probably also want to dedicate a node to the Workbench server and Crunch dispatcher, respectively. For performance reasons, you may want to run the database server on a separate node as well.
+h2(#repos). Arvados package repositories
+
+On any host where you install Arvados software, you'll need to set up an Arvados package repository. They're available for several popular distributions.
+
+h3. CentOS
+
+Packages are available for CentOS 6. First, register the Curoverse signing key in RPM's database:
+
+{% include 'install_redhat_key' %}
+
+Then save this configuration block in @/etc/yum.repos.d/arvados.repo@:
+
+<notextile>
+<pre><code>[arvados]
+name=Arvados
+baseurl=http://rpm.arvados.org/CentOS/$releasever/os/$basearch/
+</code></pre>
+</notextile>
+
+h3. Debian
+
+Packages are available for Debian 7 ("wheezy"). First, register the Curoverse signing key in apt's database:
+
+{% include 'install_debian_key' %}
+
+Then save the configuration line listed for your version of Debian in @/etc/apt/sources.list.d/arvados.list@:
+
+table(table table-bordered table-condensed).
+|*Debian version*|*@/etc/apt/sources.list.d/arvados.list@*|
+|7 ("wheezy")|@deb http://apt.arvados.org/ wheezy main@|
+
+Finally, add this new repository to apt's database:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo /usr/bin/apt-get update</span>
+</code></pre>
+</notextile>
+
+h3. Ubuntu
+
+Packages are available for Ubuntu 12.04 ("precise"). First, register the Curoverse signing key in apt's database:
+
+{% include 'install_debian_key' %}
+
+Then save the configuration line listed for your version of Ubuntu in @/etc/apt/sources.list.d/arvados.list@:
+
+table(table table-bordered table-condensed).
+|*Ubuntu version*|*@/etc/apt/sources.list.d/arvados.list@*|
+|12.04 ("precise")|@deb http://apt.arvados.org/ precise main@|
+
h2. A unique identifier
Each Arvados installation should have a globally unique identifier, which is a unique 5-character alphanumeric string. Here is a snippet of ruby that generates such a string based on the hostname of your computer:
@@ -29,7 +79,7 @@ Each Arvados installation should have a globally unique identifier, which is a u
Digest::MD5.hexdigest(`hostname`).to_i(16).to_s(36)[0..4]
</pre>
-You may also use a different method to pick the unique identifier. The unique identifier will be part of the hostname of the services in your Arvados cluster. The rest of this documentation will refer to it as your @uuid_prefix at .
+You may also use a different method to pick the unique identifier. The unique identifier will be part of the hostname of the services in your Arvados cluster. The rest of this documentation will refer to it as your @uuid_prefix at .
h2. SSL certificates
diff --git a/doc/install/install-shell-server.html.textile.liquid b/doc/install/install-shell-server.html.textile.liquid
index 25ddf7b..1e72e5e 100644
--- a/doc/install/install-shell-server.html.textile.liquid
+++ b/doc/install/install-shell-server.html.textile.liquid
@@ -4,14 +4,42 @@ navsection: installguide
title: Install a shell server
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
-There is nothing inherently special about an Arvados shell server. It is just a GNU/Linux machine with the Arvados SDKs installed. For optimal performance, the Arvados shell server should be on the same LAN as the Arvados cluster, but that is not required.
+There is nothing inherently special about an Arvados shell server. It is just a GNU/Linux machine with Arvados utilites and SDKs installed. For optimal performance, the Arvados shell server should be on the same LAN as the Arvados cluster, but that is not required.
h2. Install API tokens
Please follow the "API token guide":{{site.baseurl}}/user/reference/api-tokens.html to get API tokens for your user and install them on your shell server. We will use those tokens to test the SDKs as we install them.
-h2. Install the SDKs
+h2. Install the Ruby SDK and utilities
+
+If you're using RVM:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo rvm-exec gem install arvados-cli</span>
+</code></pre>
+</notextile>
+
+If you're not using RVM:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo gem install arvados-cli</span>
+</code></pre>
+</notextile>
+
+h2. Install the Python SDK and utilities
+
+On Debian-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo apt-get install python-arvados-python-client python-arvados-fuse</span>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo yum install python27-python-arvados-python-client python27-python-arvados-fuse</span>
+</code></pre>
+</notextile>
-Install the "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html and the "Command line SDK":{{site.baseurl}}/sdk/cli/install.html
+{% include 'note_python27_sc' %}
diff --git a/doc/sdk/perl/index.html.textile.liquid b/doc/sdk/perl/index.html.textile.liquid
index 448cbb1..cdbb7d3 100644
--- a/doc/sdk/perl/index.html.textile.liquid
+++ b/doc/sdk/perl/index.html.textile.liquid
@@ -15,24 +15,46 @@ It should be treated as alpha/experimental. Currently, limitations include:
h3. Installation
+h4. Option 1: Install from distribution packages
+
+First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
+
+On Debian-based systems:
+
<notextile>
-<pre>
-$ <code class="userinput">sudo apt-get install libjson-perl libio-socket-ssl-perl libwww-perl libipc-system-simple-perl</code>
-$ <code class="userinput">git clone https://github.com/curoverse/arvados.git</code>
-$ <code class="userinput">cd arvados/sdk/perl</code>
-$ <code class="userinput">perl Makefile.PL</code>
-$ <code class="userinput">sudo make install</code>
-</pre>
+<pre><code>~$ <span class="userinput">sudo apt-get install libjson-perl libio-socket-ssl-perl libwww-perl libipc-system-simple-perl libarvados-perl</code>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo yum install perl-ExtUtils-MakeMaker perl-JSON perl-IO-Socket-SSL perl-WWW-Curl libarvados-perl</code>
+</code></pre>
+</notextile>
+
+h4. Option 2: Install from source
+
+First, install dependencies from your distribution. Refer to the package lists above, but don't install @libarvados-perl at .
+
+Then run the following:
+
+<notextile>
+<pre><code>~$ <span class="userinput">git clone https://github.com/curoverse/arvados.git</span>
+~$ <span class="userinput">cd arvados/sdk/perl</span>
+~$ <span class="userinput">perl Makefile.PL</span>
+~$ <span class="userinput">sudo make install</span>
+</code></pre>
</notextile>
-h4. Test installation
+h3. Test installation
If the SDK is installed, @perl -MArvados -e ''@ should produce no errors.
If your @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ environment variables are set up correctly (see "api-tokens":{{site.baseurl}}/user/reference/api-tokens.html for details), the following test script should work:
<notextile>
-<pre>$ <code class="userinput">perl <<'EOF'
+<pre>~$ <code class="userinput">perl <<'EOF'
use Arvados;
my $arv = Arvados->new('apiVersion' => 'v1');
my $me = $arv->{'users'}->{'current'}->execute;
diff --git a/doc/sdk/python/sdk-python.html.textile.liquid b/doc/sdk/python/sdk-python.html.textile.liquid
index ead804e..75769d7 100644
--- a/doc/sdk/python/sdk-python.html.textile.liquid
+++ b/doc/sdk/python/sdk-python.html.textile.liquid
@@ -14,66 +14,58 @@ h3. Installation
If you are logged in to an Arvados VM, the Python SDK should be installed.
-To use the Python SDK elsewhere, you can either install the Python SDK via PyPI or build and install the package using the arvados source tree.
+To use the Python SDK elsewhere, you can install from a distribution package, PyPI, or source.
{% include 'notebox_begin' %}
-The Python SDK requires Python 2.7
+The Python SDK requires Python 2.7.
{% include 'notebox_end' %}
-h4. Option 1: install with PyPI
+h4. Option 1: Install from distribution packages
+
+First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
+
+On Debian-based systems:
<notextile>
-<pre>
-$ <code class="userinput">sudo apt-get install python-pip python-dev libattr1-dev libfuse-dev pkg-config python-yaml</code>
-$ <code class="userinput">sudo pip install arvados-python-client</code>
-</pre>
+<pre><code>~$ <span class="userinput">sudo apt-get install python-arvados-python-client</code>
+</code></pre>
</notextile>
-_If your version of @pip@ is 1.4 or newer, the @pip install@ command might give an error: "Could not find a version that satisfies the requirement arvados-python-client". If this happens, fix it by adding a @--pre@ flag:_
+On Red Hat-based systems:
<notextile>
-<pre>
-$ <code class="userinput">sudo pip install --pre arvados-python-client</code>
-</pre>
+<pre><code>~$ <span class="userinput">sudo yum install python27-python-arvados-python-client</code>
+</code></pre>
</notextile>
-h4. Option 2: install from distribution packages (Debian/Ubuntu only)
+{% include 'note_python27_sc' %}
-First add @http://apt.arvados.org@ to your list of apt repositories:
+h4. Option 2: Install with pip
-<notextile>
-<pre>
-$ <code class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</code>
-</pre>
-</notextile>
+Run @pip-2.7 install arvados-python-client@ in an appropriate installation environment, such as a virtualenv.
-Then install the package:
+If your version of @pip@ is 1.4 or newer, the @pip install@ command might give an error: "Could not find a version that satisfies the requirement arvados-python-client". If this happens, try @pip-2.7 install --pre arvados-python-client at .
-<notextile>
-<pre>
-$ <code class="userinput">sudo apt-get update</code>
-$ <code class="userinput">sudo apt-get install python-arvados-python-client</code>
-</pre>
-</notextile>
+h4. Option 3: Install from source
-h4. Option 3: build and install from source
+Install the @python-setuptools@ package from your distribution. Then run the following:
<notextile>
-<pre>
-~$ <code class="userinput">sudo apt-get install python-dev libattr1-dev libfuse-dev pkg-config</code>
-~$ <code class="userinput">git clone https://github.com/curoverse/arvados.git</code>
-~$ <code class="userinput">cd arvados/sdk/python</code>
-~/arvados/sdk/python$ <code class="userinput">sudo python setup.py install</code>
-</pre>
+<pre><code>~$ <span class="userinput">git clone https://github.com/curoverse/arvados.git</span>
+~$ <span class="userinput">cd arvados/sdk/python</span>
+~$ <span class="userinput">python2.7 setup.py install</span>
+</code></pre>
</notextile>
+You may optionally run the final installation command in a virtualenv, or with the @--user@ option.
+
h4. Test installation
If the SDK is installed and your @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ environment variables are set up correctly (see "api-tokens":{{site.baseurl}}/user/reference/api-tokens.html for details), @import arvados@ should produce no errors:
<notextile>
-<pre>$ <code class="userinput">python</code>
-Python 2.7.4 (default, Sep 26 2013, 03:20:26)
+<pre>~$ <code class="userinput">python2.7</code>
+Python 2.7.4 (default, Sep 26 2013, 03:20:26)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> <code class="userinput">import arvados</code>
@@ -160,4 +152,3 @@ j = arv.jobs().list().execute()
</notextile>
The SDK retrieves the list of API methods from the server at run time. Therefore, the set of available methods is determined by the server version rather than the SDK version.
-
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list