[ARVADOS] updated: 7af49bece83dab21129766c55a3ce9ec7f1bcaad

git at public.curoverse.com git at public.curoverse.com
Tue Jul 28 18:13:46 EDT 2015


Summary of changes:
 doc/_includes/_install_postgres.liquid             |   4 +-
 .../create-standard-objects.html.textile.liquid    |   9 +
 doc/install/install-api-server.html.textile.liquid |  14 +-
 .../install-arv-git-httpd.html.textile.liquid      | 232 ++++++++++++++++-----
 ...nstall-manual-prerequisites.html.textile.liquid |   9 +-
 .../install-shell-server.html.textile.liquid       |   2 +-
 doc/sdk/cli/subcommands.html.textile.liquid        |   2 +-
 doc/user/topics/arv-run.html.textile.liquid        |   2 +-
 docker/api/Dockerfile                              |   4 +-
 docker/api/setup-gitolite.sh.in                    |   2 +-
 sdk/python/arvados/commands/run.py                 |   9 +-
 sdk/python/arvados/events.py                       |  27 ++-
 sdk/python/bin/arv-get                             |  73 ++++---
 sdk/python/tests/test_websockets.py                |   6 +-
 .../arvados/v1/repositories_controller.rb          |  62 +++---
 services/api/app/models/arvados_model.rb           |  11 +-
 services/api/app/models/authorized_key.rb          |   3 +-
 services/api/lib/eventbus.rb                       |   7 +-
 .../api/script/arvados-git-sync.rb                 |   7 +-
 services/api/test/fixtures/links.yml               |  14 ++
 .../arvados/v1/repositories_controller_test.rb     |  97 ++++++++-
 services/api/test/integration/websocket_test.rb    |  47 ++++-
 services/api/test/unit/authorized_key_test.rb      |  45 +++-
 services/fuse/arvados_fuse/__init__.py             |  23 +-
 24 files changed, 548 insertions(+), 163 deletions(-)
 rename docker/api/update-gitolite.rb => services/api/script/arvados-git-sync.rb (96%)

  discards  923dcaf9eb662f0ba616ae8245d9a5504d6be04b (commit)
       via  7af49bece83dab21129766c55a3ce9ec7f1bcaad (commit)
       via  3f2ceb31279a18bd0f85bcc39e7c36af4a1e9b8c (commit)
       via  7bb74b6131565244c3e0b6cb9edfd34248e88557 (commit)
       via  b5e687f68d4204c54044c9786e1c923453515b5f (commit)
       via  c8f0866051391c9932a8d39bbd3cacb83e60c20e (commit)
       via  76c38b6bf63fa7329abb135a40710cbdbea18d44 (commit)
       via  0f0108a652d852b0bc165bf43e41fc378c49315a (commit)
       via  e69333d856baabf8bbbc27602850dcfe86309d23 (commit)
       via  66af20886def83f6a20cc1e6587de00cbf2f8b59 (commit)
       via  4a5adfa084b3c3a8e586df5ac0acc0b3fc6150db (commit)
       via  a71c0bc8685f71dc9ebb2804626d2d12741eebc3 (commit)
       via  f4180d151eacadf1455b2ebe43ecb61cb095df7c (commit)
       via  171ac077e7335978007daf8199559290e73b8180 (commit)
       via  753c1446ea70d70043be0913e52bb270d28ecded (commit)
       via  3e6ba5fa5f225c8aa431ce9a2796369c1e1dda2d (commit)
       via  aeb481ad2a9b9c3c090b15b317d6ce262ca95da9 (commit)
       via  17ce65cd493d8040640f5a5c3d2a97a5175a0465 (commit)
       via  51641ba5579cb9ebe14234e0888a162b46d1627d (commit)
       via  0ae899078093ac04cfdf416940f4faa821400641 (commit)
       via  a467cb24c0a2db71c39ce1bf86507bff6f3cea05 (commit)
       via  ce128902e008420f453eb29986280d72777bec32 (commit)
       via  997b13a64e1e224f77c4f1f39f0033d4750413de (commit)
       via  e302c2a74072ebe734adfb45fc6b525f299bb9fb (commit)

This update added new revisions after undoing existing revisions.  That is
to say, the old revision is not a strict subset of the new revision.  This
situation occurs when you --force push a change and generate a repository
containing something like this:

 * -- * -- B -- O -- O -- O (923dcaf9eb662f0ba616ae8245d9a5504d6be04b)
            \
             N -- N -- N (7af49bece83dab21129766c55a3ce9ec7f1bcaad)

When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.


commit 7af49bece83dab21129766c55a3ce9ec7f1bcaad
Author: Tom Clegg <tom at curoverse.com>
Date:   Sun Jul 26 13:08:55 2015 -0400

    6663: Document git setup.

diff --git a/doc/install/create-standard-objects.html.textile.liquid b/doc/install/create-standard-objects.html.textile.liquid
index 1ce4fae..9d688d4 100644
--- a/doc/install/create-standard-objects.html.textile.liquid
+++ b/doc/install/create-standard-objects.html.textile.liquid
@@ -30,6 +30,15 @@ Secondly, create a link object to make the repository object readable by the "Al
 EOF</span>
 </code></pre></notextile>
 
+In a couple of minutes, your arvados-git-sync cron job will create an empty repository on your git server. Seed it with the real arvados repository.
+
+<notextile>
+<pre><code>~$ <span class="userinput">cd /tmp</span>
+/tmp$ <span class="userinput">git clone --bare https://github.com/curoverse/arvados.git</span>
+/tmp/arvados$ <span class="userinput">git --git-dir arvados.git push https://git.<b>uuid_prefix.your.domain</b>/arvados.git '*:*'</span>
+</code></pre>
+</notextile>
+
 Next, create a default project for the standard Arvados Docker images, and give all users read access to it. The project is owned by the system user.
 
 <notextile>
diff --git a/doc/install/install-api-server.html.textile.liquid b/doc/install/install-api-server.html.textile.liquid
index 0503609..f5d5825 100644
--- a/doc/install/install-api-server.html.textile.liquid
+++ b/doc/install/install-api-server.html.textile.liquid
@@ -94,13 +94,10 @@ Define your @uuid_prefix@ in @application.yml@ by setting the @uuid_prefix@ fiel
 
 h3(#git_repositories_dir). git_repositories_dir
 
-This field defaults to @/var/lib/arvados/git at . You can override the value by defining it in @application.yml at .
-
-Make sure a clone of the arvados repository exists in @git_repositories_dir at .
+Additional git setup will be done on the "next page":install-arv-git-httpd.html. For now, just create the repository storage directory.
 
 <notextile>
-<pre><code>~$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git</span>
-~$ <span class="userinput">sudo git clone --bare git://git.curoverse.com/arvados.git /var/lib/arvados/git/arvados.git</span>
+<pre><code>~$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git/repositories</span>
 </code></pre></notextile>
 
 h3. secret_token
@@ -146,9 +143,10 @@ h2. Prepare the API server deployment
 Now that all your configuration is in place, run @/usr/local/bin/arvados-api-server-upgrade.sh at .  This will install and check your configuration, install necessary gems, and run any necessary database setup.
 
 {% include 'notebox_begin' %}
-You can safely ignore the following error message you may see when loading the database structure:
-<notextile>
-<pre><code>ERROR:  must be owner of extension plpgsql</code></pre></notextile>
+You can safely ignore the following messages if they appear while this script runs:
+<pre>Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will
+break this application for all non-root users on this machine.</pre>
+<pre>fatal: Not a git repository (or any of the parent directories): .git</pre>
 {% include 'notebox_end' %}
 
 This command aborts when it encounters an error.  It's safe to rerun multiple times, so if there's a problem with your configuration, you can fix that and try again.
diff --git a/doc/install/install-arv-git-httpd.html.textile.liquid b/doc/install/install-arv-git-httpd.html.textile.liquid
index 33b1124..de61722 100644
--- a/doc/install/install-arv-git-httpd.html.textile.liquid
+++ b/doc/install/install-arv-git-httpd.html.textile.liquid
@@ -1,21 +1,224 @@
 ---
 layout: default
 navsection: installguide
-title: Install Git server
+title: Install the Git server
 ...
 
-The arvados-git-httpd server provides HTTP access to hosted git repositories, using Arvados authentication tokens instead of passwords. It is intended to be installed on the system where your git repositories are stored, and accessed through a web proxy that provides SSL support.
+Arvados allows users to create their own private and public git repositories, and clone/push them using SSH and HTTPS.
+
+The git hosting setup involves three components.
+* The "update-git.rb" script polls the API server for the current list of repositories, creates bare repositories, and updates the local permission cache used by gitolite.
+* Gitolite provides SSH access.
+* arvados-git-http provides HTTPS access.
+
+It is not strictly necessary to deploy _both_ SSH and HTTPS access, but we recommend deploying both:
+* SSH is a more appropriate way to authenticate from a user's workstation because it does not require managing tokens on the client side;
+* HTTPS is a more appropriate way to authenticate from a shell VM because it does not depend on SSH agent forwarding (SSH clients' agent forwarding features tend to behave as if the remote machine is fully trusted).
+
+The HTTPS instructions given below will not work if you skip the SSH setup steps.
+
+h2. Set up DNS
 
 By convention, we use the following hostname for the git service:
 
-<div class="offset1">
-table(table table-bordered table-condensed).
-|git. at uuid_prefix@.your.domain|
-</div>
+<notextile>
+<pre><code>git.<span class="userinput">uuid_prefix</span>.your.domain
+</code></pre>
+</notextile>
+
+{% include 'notebox_begin' %}
+Here, we show how to install the git hosting services *on the same host as your API server.* Using a different host is not yet fully supported. On this page we will refer to it as your git server.
+{% include 'notebox_end' %}
+
+DNS and network configuration should be set up so port 443 reaches your HTTPS proxy, and port 22 reaches your git server.
+
+h2. Generate an API token
+
+On the API server:
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">cd /var/www/arvados-api/current</span>
+gitserver:/var/www/arvados-api/current$ <span class="userinput">sudo -u www-data RAILS_ENV=production bundle exec ./script/create_superuser_token.rb</span>
+4hdqaixi5a027jqn0vyjbwa3xmcue8logzhtsmk1bplgp064fe
+</code></pre>
+</notextile>
+
+Copy that token; you'll need it in a minute.
+
+h2. Install git and other dependencies
+
+On Debian-based systems:
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">sudo apt-get install git openssh-server</span>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">sudo yum install git perl-Data-Dumper openssh-server</span>
+</code></pre>
+</notextile>
+
+h2. Create a "git" user and a storage directory
+
+Hosted repositories will be stored under @/var/lib/arvados/git/@. If you choose a different location, make sure to update the @git_repositories_dir@ entry in your API server's @config/application.yml@ file, preserving the trailing @repositories/@ part.
+
+A new UNIX account called "git" will own the files. This makes git URLs look familiar to users (<code>git@[...]:username/reponame.git</code>).
+
+On Debian- or Red Hat-based systems:
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git/repositories</span>
+gitserver:~$ <span class="userinput">sudo useradd --comment git --home-dir /var/lib/arvados/git git</span>
+gitserver:~$ <span class="userinput">sudo chown -R git:git ~git</span>
+</code></pre>
+</notextile>
+
+The git user needs its own SSH key. (It must be able to run @ssh git at localhost@ from scripts.)
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">sudo -u git -i bash</span>
+git at gitserver:~$ <span class="userinput">ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa</span>
+git at gitserver:~$ <span class="userinput">ssh -o stricthostkeychecking=no localhost cat .ssh/id_rsa.pub</span>
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7aBIDAAgMQN16Pg6eHmvc+D+6TljwCGr4YGUBphSdVb25UyBCeAEgzqRiqy0IjQR2BLtSirXr+1SJAcQfBgI/jwR7FG+YIzJ4ND9JFEfcpq20FvWnMMQ6XD3y3xrZ1/h/RdBNwy4QCqjiXuxDpDB7VNP9/oeAzoATPZGhqjPfNS+RRVEQpC6BzZdsR+S838E53URguBOf9yrPwdHvosZn7VC0akeWQerHqaBIpSfDMtaM4+9s1Gdsz0iP85rtj/6U/K/XOuv2CZsuVZZ52nu3soHnEX2nx2IaXMS3L8Z+lfOXB2T6EaJgXF7Z9ME5K1tx9TSNTRcYCiKztXLNLSbp git at gitserver
+git at gitserver:~$ <span class="userinput">exit</span>
+</code></pre>
+</notextile>
+
+h2. Install gitolite
+
+Check https://github.com/sitaramc/gitolite/tags for the latest stable version (_e.g.,_ @v3.6.3@).
+
+Download and install the version you selected.
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">sudo -u git -i bash</span>
+git at gitserver:~$ <span class="userinput">echo 'PATH=$HOME/bin:$PATH' >.profile</span>
+git at gitserver:~$ <span class="userinput">source .profile</span>
+git at gitserver:~$ <span class="userinput">git clone --branch <b>v3.6.3</b> git://github.com/sitaramc/gitolite</span>
+...
+Note: checking out '5d24ae666bfd2fa9093d67c840eb8d686992083f'.
+...
+git at gitserver:~$ <span class="userinput">mkdir bin</span>
+git at gitserver:~$ <span class="userinput">gitolite/install -ln ~git/bin</span>
+git at gitserver:~$ <span class="userinput">bin/gitolite setup -pk .ssh/id_rsa.pub</span>
+Initialized empty Git repository in /var/lib/arvados/git/repositories/gitolite-admin.git/
+Initialized empty Git repository in /var/lib/arvados/git/repositories/testing.git/
+WARNING: /var/lib/arvados/git/.ssh missing; creating a new one
+    (this is normal on a brand new install)
+WARNING: /var/lib/arvados/git/.ssh/authorized_keys missing; creating a new one
+    (this is normal on a brand new install)
+</code></pre>
+</notextile>
+
+Create the base configuration files.
+
+<notextile>
+<pre><code>git at gitserver:~$ <span class="userinput">git clone git at localhost:gitolite-admin</span>
+Cloning into 'gitolite-admin'...
+warning: You appear to have cloned an empty repository.
+done.
+git at gitserver:~$ <span class="userinput">cd gitolite-admin</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git config user.email arvados</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git config user.name arvados</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git config push.default simple</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">mkdir -p keydir/arvados conf/admin conf/auto</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">cat >conf/admin/arvados.conf <<EOF
+ at arvados_git_user = arvados_git_user
+
+repo @all
+     RW+                 = @arvados_git_user
+
+EOF</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">cat >conf/gitolite.conf <<EOF
+include "auto/*.conf"
+include "admin/*.conf"
+EOF</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git add .</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git commit -m -</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git push</span>
+[master a4d083b] -
+ 4 files changed, 10 insertions(+), 5 deletions(-)
+ create mode 100644 arvadosaliases.pl
+ create mode 100644 conf/admin/arvados.conf
+ create mode 100644 keydir/arvados_git_user.pub
+git at gitserver:~/gitolite-admin$ <span class="userinput">exit</span>
+</code></pre>
+</notextile>
+
+h2. Configure gitolite
+
+Configure gitolite to look up a repository name like @username/reponame.git@ and find the appropriate bare repository storage directory.
+
+Add the following lines to the top of @~git/.gitolite.rc@:
+
+<notextile>
+<pre><code><span class="userinput">my $repo_aliases;
+my $aliases_src = "$ENV{HOME}/.gitolite/arvadosaliases.pl";
+if ($ENV{HOME} && (-e $aliases_src)) {
+    $repo_aliases = do $aliases_src;
+}
+$repo_aliases ||= {};
+</span></code></pre>
+</notextile>
+
+Add the following lines inside the section that begins @%RC = (@:
+
+<notextile>
+<pre><code><span class="userinput">    REPO_ALIASES => $repo_aliases,
+</span></code></pre>
+</notextile>
+
+Uncomment the 'Alias' line in the section that begins @ENABLE => [@:
+
+<notextile>
+<pre><code><span class="userinput">            # access a repo by another (possibly legacy) name
+            'Alias',
+</span></code></pre>
+</notextile>
+
+h2. Configure git synchronization
+
+Create a configuration file @/var/www/arvados-api/current/config/arvados-clients.yml@ using the following template, filling in the appropriate values for your system.
+* For @arvados_api_token@, use the token you generated above.
+* For @gitolite_arvados_git_user_key@, provide the public key you generated above, i.e., the contents of @~git/.ssh/id_rsa.pub at .
+
+<notextile>
+<pre><code>production:
+  gitolite_url: /var/lib/arvados/git/repositories/gitolite-admin.git
+  gitolite_tmp: /var/lib/arvados/git
+  arvados_api_host: <span class="userinput">uuid_prefix.example.com</span>
+  arvados_api_token: "<span class="userinput">4hdqaixi5a027jqn0vyjbwa3xmcue8logzhtsmk1bplgp064fe</span>"
+  arvados_api_host_insecure: <span class="userinput">false</span>
+  gitolite_arvados_git_user_key: "<span class="userinput">ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7aBIDAAgMQN16Pg6eHmvc+D+6TljwCGr4YGUBphSdVb25UyBCeAEgzqRiqy0IjQR2BLtSirXr+1SJAcQfBgI/jwR7FG+YIzJ4ND9JFEfcpq20FvWnMMQ6XD3y3xrZ1/h/RdBNwy4QCqjiXuxDpDB7VNP9/oeAzoATPZGhqjPfNS+RRVEQpC6BzZdsR+S838E53URguBOf9yrPwdHvosZn7VC0akeWQerHqaBIpSfDMtaM4+9s1Gdsz0iP85rtj/6U/K/XOuv2CZsuVZZ52nu3soHnEX2nx2IaXMS3L8Z+lfOXB2T6EaJgXF7Z9ME5K1tx9TSNTRcYCiKztXLNLSbp git at gitserver</span>"
+</code></pre>
+</notextile>
+
+h2. Enable the synchronization script
+
+The API server package includes a script that retrieves the current set of repository names and permissions from the API, writes names and permissions to @arvadosaliases.pl@ in a format usable by gitolite, and creates new empty repositories if needed. This script should run every 2 to 5 minutes.
+
+If you are using RVM, create @/etc/cron.d/arvados-git-sync@ with the following content:
+
+<notextile>
+<pre><code><span class="userinput">*/5 * * * * git cd /var/www/arvados-api/current && /usr/local/rvm/bin/rvm-exec default bundle exec script/arvados-git-sync.rb production</span>
+</code></pre>
+</notextile>
+
+Otherwise, create @/etc/cron.d/arvados-git-sync@ with the following content:
+
+<notextile>
+<pre><code><span class="userinput">*/5 * * * * git cd /var/www/arvados-api/current && bundle exec script/arvados-git-sync.rb production</span>
+</code></pre>
+</notextile>
+
+h2. Install the arvados-git-httpd package
 
-This hostname should resolve from anywhere on the internet.
+This is needed only for HTTPS access.
 
-h2. Install arvados-git-httpd
+The arvados-git-httpd package provides HTTP access, using Arvados authentication tokens instead of passwords. It is intended to be installed on the system where your git repositories are stored, and accessed through a web proxy that provides SSL support.
 
 On Debian-based systems:
 
@@ -31,11 +234,11 @@ On Red Hat-based systems:
 </code></pre>
 </notextile>
 
-Verify that @arvados-git-httpd@ and @git-http-backend@ are functional:
+Verify that @arvados-git-httpd@ and @git-http-backend@ can be run:
 
 <notextile>
 <pre><code>~$ <span class="userinput">arvados-git-httpd -h</span>
-Usage of arv-git-httpd:
+Usage of arvados-git-httpd:
   -address="0.0.0.0:80": Address to listen on, "host:port".
   -git-command="/usr/bin/git": Path to git executable. Each authenticated request will execute this program with a single argument, "http-backend".
   -repo-root="/path/to/cwd": Path to git repositories.
@@ -49,21 +252,58 @@ fatal: No REQUEST_METHOD from server
 </code></pre>
 </notextile>
 
-We recommend running @arvados-git-httpd@ under "runit":http://smarden.org/runit/ or something similar.
+h3. Enable arvados-git-httpd
 
-Your @run@ script should look something like this:
+Install "runit":http://smarden.org/runit/ (if it's not already installed) and configure it to run arvados-git-httpd. Update the API host to match your site.
 
 <notextile>
-<pre><code>export ARVADOS_API_HOST=<span class="userinput">uuid_prefix</span>.your.domain
-exec sudo -u git arvados-git-httpd -address=:9001 -git-command="$(which git)" -repo-root=<span class="userinput">/var/lib/arvados/git</span> 2>&1
+<pre><code>~$ <span class="userinput">sudo apt-get install runit</span>
+~$ <span class="userinput">cd /etc/sv</span>
+/etc/sv$ <span class="userinput">sudo mkdir arvados-git-httpd; cd arvados-git-httpd</span>
+/etc/sv/arvados-git-httpd$ <span class="userinput">sudo mkdir log</span>
+/etc/sv/arvados-git-httpd$ <span class="userinput">sudo sh -c 'cat >log/run' <<'EOF'
+#!/bin/sh
+mkdir -p main
+chown git:git main
+exec chpst -u git:git svlogd -tt main
+EOF</span>
+/etc/sv/arvados-git-httpd$ <span class="userinput">sudo sh -c 'cat >run' <<'EOF'
+#!/bin/sh
+export ARVADOS_API_HOST=<b>uuid_prefix.your.domain</b>
+export GITOLITE_HTTP_HOME=/var/lib/arvados/git
+export PATH="$PATH:/var/lib/arvados/git/bin"
+exec chpst -u git:git arvados-git-httpd -address=:9001 -git-command="$(which git)" -repo-root=<b>/var/lib/arvados/git/repositories</b> 2>&1
+EOF</span>
+/etc/sv/arvados-git-httpd$ <span class="userinput">sudo chmod +x run log/run</span>
 </code></pre>
 </notextile>
 
-h3. Set up a reverse proxy with SSL support
+h3. Set up a reverse proxy to provide SSL service
+
+The arvados-git-httpd service will be accessible from anywhere on the internet, so we recommend using SSL.
+
+This is best achieved by putting a reverse proxy with SSL support in front of arvados-git-httpd, running on port 443 and passing requests to @arvados-git-httpd@ on port 9001 (or whatever port you chose in your run script).
 
-The arvados-git-httpd service will be accessible from anywhere on the internet, so we recommend using SSL for transport encryption.
+<notextile>
+<pre><code><span class="userinput">http {
+  upstream arvados-git-httpd {
+    server localhost:9001;
+  }
+  server {
+    listen *:443 ssl;
+    server_name git.uuid_prefix.example.com;
+    ssl_certificate /root/git.uuid_prefix.example.com.crt;
+    ssl_certificate_key /root/git.uuid_prefix.example.com.key;
+    location  / {
+      proxy_pass http://arvados-git-httpd;
+      proxy_set_header X-Forwarded-For $remote_addr;
+    }
+  }
+}
+</span>
+</code></pre>
+</notextile>
 
-This is best achieved by putting a reverse proxy with SSL support in front of arvados-git-httpd, running on port 443 and passing requests to arvados-git-httpd on port 9001 (or whatever port you chose in your run script).
 
 h3. Tell the API server about the arvados-git-httpd service
 
diff --git a/doc/install/install-shell-server.html.textile.liquid b/doc/install/install-shell-server.html.textile.liquid
index 71e6b01..df5045e 100644
--- a/doc/install/install-shell-server.html.textile.liquid
+++ b/doc/install/install-shell-server.html.textile.liquid
@@ -15,7 +15,7 @@ h2. Install the Ruby SDK and utilities
 If you're using RVM:
 
 <notextile>
-<pre><code>~$ <span class="userinput">sudo rvm-exec gem install arvados-cli</span>
+<pre><code>~$ <span class="userinput">sudo /usr/local/rvm/bin/rvm-exec default gem install arvados-cli</span>
 </code></pre>
 </notextile>
 
diff --git a/docker/api/Dockerfile b/docker/api/Dockerfile
index 758c015..6a3428c 100644
--- a/docker/api/Dockerfile
+++ b/docker/api/Dockerfile
@@ -59,9 +59,9 @@ ADD keep_proxy.json /root/
 
 # Set up update-gitolite.rb
 RUN mkdir /usr/local/arvados/config -p
-ADD generated/arvados-clients.yml /usr/local/arvados/config/
+ADD generated/arvados-clients.yml /usr/src/arvados/services/api/config/
 ADD .gitolite.rc /usr/local/arvados/config/
-ADD update-gitolite.rb /usr/local/arvados/
+RUN ln /usr/src/arvados/services/api/script/arvados-git-sync.rb /usr/local/bin/
 
 # Supervisor.
 ADD supervisor.conf /etc/supervisor/conf.d/arvados.conf
diff --git a/docker/api/setup-gitolite.sh.in b/docker/api/setup-gitolite.sh.in
index d953fa7..023ca5d 100755
--- a/docker/api/setup-gitolite.sh.in
+++ b/docker/api/setup-gitolite.sh.in
@@ -60,7 +60,7 @@ git push
 su - git -c "git clone --bare git://github.com/curoverse/arvados.git /home/git/repositories/arvados.git"
 
 echo "ARVADOS_API_HOST_INSECURE=yes" > /etc/cron.d/gitolite-update
-echo "*/2 * * * * root /bin/bash -c 'source /etc/profile.d/rvm.sh && /usr/local/arvados/update-gitolite.rb production'" >> /etc/cron.d/gitolite-update
+echo "*/2 * * * * root /bin/bash -c 'source /etc/profile.d/rvm.sh && /usr/src/arvados/services/api/script/arvados-git-sync.rb production'" >> /etc/cron.d/gitolite-update
 
 # Create/update the repos now
 . /etc/profile.d/rvm.sh
diff --git a/services/api/config/application.default.yml b/services/api/config/application.default.yml
index 409dea6..5d1402d 100644
--- a/services/api/config/application.default.yml
+++ b/services/api/config/application.default.yml
@@ -86,7 +86,7 @@ common:
   # able to submit crunch jobs. To pass the test suites, put a clone
   # of the arvados tree in {git_repositories_dir}/arvados.git or
   # {git_repositories_dir}/arvados/.git
-  git_repositories_dir: /var/lib/arvados/git
+  git_repositories_dir: /var/lib/gitolite/repositories
 
   # This is a (bare) repository that stores commits used in jobs.  When a job
   # runs, the source commits are first fetched into this repository, then this
diff --git a/docker/api/update-gitolite.rb b/services/api/script/arvados-git-sync.rb
similarity index 96%
rename from docker/api/update-gitolite.rb
rename to services/api/script/arvados-git-sync.rb
index 1f178cb..3a8ed27 100755
--- a/docker/api/update-gitolite.rb
+++ b/services/api/script/arvados-git-sync.rb
@@ -21,19 +21,18 @@ DEBUG = 1
 
 # load and merge in the environment-specific application config info
 # if present, overriding base config parameters as specified
-path = File.dirname(__FILE__) + '/config/arvados-clients.yml'
+path = File.absolute_path('../../config/arvados-clients.yml', __FILE__)
 if File.exists?(path) then
   cp_config = YAML.load_file(path)[ENV['RAILS_ENV']]
 else
-  puts "Please create a\n " + File.dirname(__FILE__) + "/config/arvados-clients.yml\n file"
+  puts "Please create a\n #{path}\n file"
   exit 1
 end
 
 gitolite_url = cp_config['gitolite_url']
 gitolite_arvados_git_user_key = cp_config['gitolite_arvados_git_user_key']
 
-gitolite_tmpdir = File.join(File.absolute_path(File.dirname(__FILE__)),
-                            cp_config['gitolite_tmp'])
+gitolite_tmpdir = cp_config['gitolite_tmp']
 gitolite_admin = File.join(gitolite_tmpdir, 'gitolite-admin')
 gitolite_admin_keydir = File.join(gitolite_admin, 'keydir')
 gitolite_keydir = File.join(gitolite_admin, 'keydir', 'arvados')

commit 3f2ceb31279a18bd0f85bcc39e7c36af4a1e9b8c
Author: Tom Clegg <tom at curoverse.com>
Date:   Tue Jul 28 18:11:59 2015 -0400

    6663: Fix repository permissions.
    
    Users with is_active==false (except anonymous_user) are not listed by
    get_all_permissions.
    
    A repository that is shared with a group is only shared with users who
    have permissions on that group, not all users who have permissions on
    *any* group.
    
    {User --can_write--> group --can_read--> repo} is correctly listed as
    {User --can_read--> repo}, not can_write.
    
    Tests check that the permissions reported by get_all_permissions do
    not exceed the API server's interpretation of the current permission
    set.

diff --git a/services/api/app/controllers/arvados/v1/repositories_controller.rb b/services/api/app/controllers/arvados/v1/repositories_controller.rb
index fd6ab58..998a060 100644
--- a/services/api/app/controllers/arvados/v1/repositories_controller.rb
+++ b/services/api/app/controllers/arvados/v1/repositories_controller.rb
@@ -3,14 +3,23 @@ class Arvados::V1::RepositoriesController < ApplicationController
   skip_before_filter :render_404_if_no_object, :only => :get_all_permissions
   before_filter :admin_required, :only => :get_all_permissions
   def get_all_permissions
-    @users = {}
-    User.includes(:authorized_keys).find_each do |u|
-      @users[u.uuid] = u
+    users = {}
+    user_aks = {}
+    admins = []
+    User.eager_load(:authorized_keys).find_each do |u|
+      next unless u.is_active or u.uuid == anonymous_user_uuid
+      users[u.uuid] = u
+      user_aks[u.uuid] ||= u.authorized_keys.
+        collect do |ak|
+        {
+          public_key: ak.public_key,
+          authorized_key_uuid: ak.uuid
+        }
+      end
+      admins << u.uuid if u.is_admin
     end
-    admins = @users.select { |k,v| v.is_admin }
-    @user_aks = {}
     @repo_info = {}
-    Repository.includes(:permissions).find_each do |repo|
+    Repository.eager_load(:permissions).find_each do |repo|
       @repo_info[repo.uuid] = {
         uuid: repo.uuid,
         name: repo.name,
@@ -22,38 +31,33 @@ class Arvados::V1::RepositoriesController < ApplicationController
       perms = []
       repo.permissions.each do |perm|
         if ArvadosModel::resource_class_for_uuid(perm.tail_uuid) == Group
-          @users.each do |user_uuid, user|
-            user.group_permissions.each do |group_uuid, perm_mask|
-              if perm_mask[:manage]
-                perms << {name: 'can_manage', user_uuid: user_uuid}
-              elsif perm_mask[:write]
-                perms << {name: 'can_write', user_uuid: user_uuid}
-              elsif perm_mask[:read]
-                perms << {name: 'can_read', user_uuid: user_uuid}
-              end
+          users.each do |user_uuid, user|
+            perm_mask = user.group_permissions[perm.tail_uuid]
+            if not perm_mask
+              next
+            elsif perm_mask[:manage] and perm.name == 'can_manage'
+              perms << {name: 'can_manage', user_uuid: user_uuid}
+            elsif perm_mask[:write] and ['can_manage', 'can_write'].index perm.name
+              perms << {name: 'can_write', user_uuid: user_uuid}
+            elsif perm_mask[:read]
+              perms << {name: 'can_read', user_uuid: user_uuid}
             end
           end
-        else
+        elsif users[perm.tail_uuid]
+          # user exists and is (active or the anonymous user)
           perms << {name: perm.name, user_uuid: perm.tail_uuid}
         end
       end
       # Owner of the repository, and all admins, can RW
-      ([repo.owner_uuid] + admins.keys).each do |user_uuid|
+      ([repo.owner_uuid] | admins).each do |user_uuid|
+        # no permissions for inactive user, even when owner of repo
+        next unless users[user_uuid]
         perms << {name: 'can_write', user_uuid: user_uuid}
       end
       perms.each do |perm|
         user_uuid = perm[:user_uuid]
-        @user_aks[user_uuid] = @users[user_uuid].andand.authorized_keys.andand.
-          collect do |ak|
-          {
-            public_key: ak.public_key,
-            authorized_key_uuid: ak.uuid
-          }
-        end || []
-        if @user_aks[user_uuid].any?
-          ri = (@repo_info[repo.uuid][:user_permissions][user_uuid] ||= {})
-          ri[perm[:name]] = true
-        end
+        ri = (@repo_info[repo.uuid][:user_permissions][user_uuid] ||= {})
+        ri[perm[:name]] = true
       end
     end
     @repo_info.values.each do |repo_users|
@@ -72,6 +76,6 @@ class Arvados::V1::RepositoriesController < ApplicationController
     end
     send_json(kind: 'arvados#RepositoryPermissionSnapshot',
               repositories: @repo_info.values,
-              user_keys: @user_aks)
+              user_keys: user_aks)
   end
 end
diff --git a/services/api/test/fixtures/links.yml b/services/api/test/fixtures/links.yml
index 434f9c7..925e466 100644
--- a/services/api/test/fixtures/links.yml
+++ b/services/api/test/fixtures/links.yml
@@ -405,6 +405,20 @@ admin_can_write_aproject:
   head_uuid: zzzzz-j7d0g-v955i6s2oi1cbso
   properties: {}
 
+project_viewer_member_of_all_users_group:
+  uuid: zzzzz-o0j2j-cdnq6627g0h0r2x
+  owner_uuid: zzzzz-tpzed-000000000000000
+  created_at: 2015-07-28T21:34:41.361747000Z
+  modified_by_client_uuid: zzzzz-ozdt8-brczlopd8u8d0jr
+  modified_by_user_uuid: zzzzz-tpzed-000000000000000
+  modified_at: 2015-07-28T21:34:41.361747000Z
+  updated_at: 2015-07-28T21:34:41.361747000Z
+  tail_uuid: zzzzz-tpzed-projectviewer1a
+  link_class: permission
+  name: can_read
+  head_uuid: zzzzz-j7d0g-fffffffffffffff
+  properties: {}
+
 project_viewer_can_read_project:
   uuid: zzzzz-o0j2j-projviewerreadp
   owner_uuid: zzzzz-tpzed-000000000000000
diff --git a/services/api/test/functional/arvados/v1/repositories_controller_test.rb b/services/api/test/functional/arvados/v1/repositories_controller_test.rb
index 7ba2183..514bb66 100644
--- a/services/api/test/functional/arvados/v1/repositories_controller_test.rb
+++ b/services/api/test/functional/arvados/v1/repositories_controller_test.rb
@@ -42,6 +42,26 @@ class Arvados::V1::RepositoriesControllerTest < ActionController::TestCase
     end
   end
 
+  test "get_all_permissions takes into account is_active flag" do
+    r = nil
+    act_as_user users(:active) do
+      r = Repository.create! name: 'active/testrepo'
+    end
+    act_as_system_user do
+      u = users(:active)
+      u.is_active = false
+      u.save!
+    end
+    authorize_with :admin
+    get :get_all_permissions
+    assert_response :success
+    json_response['repositories'].each do |r|
+      r['user_permissions'].each do |user_uuid, perms|
+        refute_equal user_uuid, users(:active).uuid
+      end
+    end
+  end
+
   test "get_all_permissions does not give any access to user without permission" do
     viewer_uuid = users(:project_viewer).uuid
     assert_equal(authorized_keys(:project_viewer).authorized_user_uuid,
@@ -88,15 +108,84 @@ class Arvados::V1::RepositoriesControllerTest < ActionController::TestCase
     end
   end
 
-  test "get_all_permissions lists repos with no authorized keys" do
+  test "get_all_permissions lists all repos regardless of permissions" do
+    act_as_system_user do
+      # Create repos that could potentially be left out of the
+      # permission list by accident.
+
+      # No authorized_key, no username (this can't even be done
+      # without skipping validations)
+      r = Repository.create name: 'root/testrepo'
+      assert r.save validate: false
+
+      r = Repository.create name: 'invalid username / repo name', owner_uuid: users(:inactive).uuid
+      assert r.save validate: false
+    end
+    authorize_with :admin
+    get :get_all_permissions
+    assert_response :success
+    assert_equal(Repository.count, json_response["repositories"].size)
+  end
+
+  test "get_all_permissions lists user permissions for users with no authorized keys" do
     authorize_with :admin
     AuthorizedKey.destroy_all
     get :get_all_permissions
     assert_response :success
     assert_equal(Repository.count, json_response["repositories"].size)
-    assert(json_response["repositories"].any? do |repo|
-             repo["user_permissions"].empty?
-           end, "test is invalid - all repositories have authorized keys")
+    repos_with_perms = []
+    json_response['repositories'].each do |repo|
+      if repo['user_permissions'].any?
+        repos_with_perms << repo['uuid']
+      end
+    end
+    assert_not_empty repos_with_perms, 'permissions are missing'
+  end
+
+  # Ensure get_all_permissions correctly describes what the normal
+  # permission system would do.
+  test "get_all_permissions obeys group permissions" do
+    act_as_user system_user do
+      r = Repository.create!(name: 'admin/groupcanwrite', owner_uuid: users(:admin).uuid)
+      g = Group.create!(group_class: 'group', name: 'repo-writers')
+      u1 = users(:active)
+      u2 = users(:spectator)
+      Link.create!(tail_uuid: g.uuid, head_uuid: r.uuid, link_class: 'permission', name: 'can_manage')
+      Link.create!(tail_uuid: u1.uuid, head_uuid: g.uuid, link_class: 'permission', name: 'can_write')
+      Link.create!(tail_uuid: u2.uuid, head_uuid: g.uuid, link_class: 'permission', name: 'can_read')
+
+      r = Repository.create!(name: 'admin/groupreadonly', owner_uuid: users(:admin).uuid)
+      g = Group.create!(group_class: 'group', name: 'repo-readers')
+      u1 = users(:active)
+      u2 = users(:spectator)
+      Link.create!(tail_uuid: g.uuid, head_uuid: r.uuid, link_class: 'permission', name: 'can_read')
+      Link.create!(tail_uuid: u1.uuid, head_uuid: g.uuid, link_class: 'permission', name: 'can_write')
+      Link.create!(tail_uuid: u2.uuid, head_uuid: g.uuid, link_class: 'permission', name: 'can_read')
+    end
+    authorize_with :admin
+    get :get_all_permissions
+    assert_response :success
+    json_response['repositories'].each do |repo|
+      repo['user_permissions'].each do |user_uuid, perms|
+        u = User.find_by_uuid(user_uuid)
+        if perms['can_read']
+          assert u.can? read: repo['uuid']
+          assert_match /R/, perms['gitolite_permissions']
+        else
+          refute_match /R/, perms['gitolite_permissions']
+        end
+        if perms['can_write']
+          assert u.can? write: repo['uuid']
+          assert_match /RW/, perms['gitolite_permissions']
+        else
+          refute_match /W/, perms['gitolite_permissions']
+        end
+        if perms['can_manage']
+          assert u.can? manage: repo['uuid']
+          assert_match /RW/, perms['gitolite_permissions']
+        end
+      end
+    end
   end
 
   test "default index includes fetch_url" do

commit 7bb74b6131565244c3e0b6cb9edfd34248e88557
Author: Tom Clegg <tom at curoverse.com>
Date:   Tue Jul 28 17:58:25 2015 -0400

    6663: Fix "duplicate public_key" test: OK if the key being updated has the same public_key.

diff --git a/services/api/app/models/authorized_key.rb b/services/api/app/models/authorized_key.rb
index b156a1d..e5f691c 100644
--- a/services/api/app/models/authorized_key.rb
+++ b/services/api/app/models/authorized_key.rb
@@ -40,7 +40,8 @@ class AuthorizedKey < ArvadosModel
         errors.add(:public_key, "does not appear to be a valid ssh-rsa or dsa public key")
       else
         # Valid if no other rows have this public key
-        if self.class.where('public_key like ?', "%#{self.public_key}%").any?
+        if self.class.where('not (uuid = ?) and public_key like ?',
+                            uuid, "%#{self.public_key}%").any?
           errors.add(:public_key, "already exists in the database, use a different key.")
           return false
         end
diff --git a/services/api/test/unit/authorized_key_test.rb b/services/api/test/unit/authorized_key_test.rb
index b8d9b67..9ce3929 100644
--- a/services/api/test/unit/authorized_key_test.rb
+++ b/services/api/test/unit/authorized_key_test.rb
@@ -1,7 +1,46 @@
 require 'test_helper'
 
 class AuthorizedKeyTest < ActiveSupport::TestCase
-  # test "the truth" do
-  #   assert true
-  # end
+  TEST_KEY = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCf5aTI55uyWr44TckP/ELUAyPsdnf5fTZDcSDN4qiMZYAL7TYV2ixwnbPObLObM0GmHSSFLV1KqsuFICUPgkyKoHbAH6XPgmtfOLU60VkGf1v5uxQ/kXCECRCJmPb3K9dIXGEw+1DXPdOV/xG7rJNvo4a9WK9iqqZr8p+VGKM6C017b8BDLk0tuEEjZ5jXcT/ka/hTScxWkKgF6auPOVQ79OA5+0VaYm4uQLzVUdgwVUPWQQecRrtnc08XYM1htpcLDIAbWfUNK7uE6XR3/OhtrJGf05FGbtGguPgi33F9W3Q3yw6saOK5Y3TfLbskgFaEdLgzqK/QSBRk2zBF49Tj test at localhost"
+
+  test 'create and update key' do
+    u1 = users(:active)
+    act_as_user u1 do
+      ak = AuthorizedKey.create(name: "foo", public_key: TEST_KEY, authorized_user_uuid: u1.uuid)
+      assert ak.save, ak.errors.full_messages.to_s
+      ak.name = "bar"
+      assert ak.valid?, ak.errors.full_messages.to_s
+      assert ak.save, ak.errors.full_messages.to_s
+    end
+  end
+
+  test 'duplicate key not permitted' do
+    u1 = users(:active)
+    act_as_user u1 do
+      ak = AuthorizedKey.create!(name: "foo", public_key: TEST_KEY, authorized_user_uuid: u1.uuid)
+    end
+    u2 = users(:spectator)
+    act_as_user u2 do
+      ak2 = AuthorizedKey.create(name: "bar", public_key: TEST_KEY, authorized_user_uuid: u2.uuid)
+      refute ak2.valid?
+      refute ak2.save
+      assert_match /already exists/, ak2.errors.full_messages.to_s
+    end
+  end
+
+  test 'attach key to wrong user account' do
+    act_as_user users(:active) do
+      ak = AuthorizedKey.create(name: "foo", public_key: TEST_KEY)
+      ak.authorized_user_uuid = users(:spectator).uuid
+      refute ak.save
+      ak.uuid = nil
+      ak.authorized_user_uuid = users(:admin).uuid
+      refute ak.save
+      ak.uuid = nil
+      ak.authorized_user_uuid = users(:active).uuid
+      assert ak.save, ak.errors.full_messages.to_s
+      ak.authorized_user_uuid = users(:admin).uuid
+      refute ak.save
+    end
+  end
 end

commit b5e687f68d4204c54044c9786e1c923453515b5f
Author: Tom Clegg <tom at curoverse.com>
Date:   Tue Jul 28 17:52:30 2015 -0400

    6663: Fix using default owner_uuid in repositories#create. refs #6652

diff --git a/services/api/app/models/arvados_model.rb b/services/api/app/models/arvados_model.rb
index d5b2f87..35dd1a9 100644
--- a/services/api/app/models/arvados_model.rb
+++ b/services/api/app/models/arvados_model.rb
@@ -23,6 +23,7 @@ class ArvadosModel < ActiveRecord::Base
   after_destroy :log_destroy
   after_find :convert_serialized_symbols_to_strings
   before_validation :normalize_collection_uuids
+  before_validation :set_default_owner
   validate :ensure_serialized_attribute_type
   validate :ensure_valid_uuids
 
@@ -276,12 +277,14 @@ class ArvadosModel < ActiveRecord::Base
     true
   end
 
-  def ensure_owner_uuid_is_permitted
-    raise PermissionDeniedError if !current_user
-
-    if new_record? and respond_to? :owner_uuid=
+  def set_default_owner
+    if new_record? and current_user and respond_to? :owner_uuid=
       self.owner_uuid ||= current_user.uuid
     end
+  end
+
+  def ensure_owner_uuid_is_permitted
+    raise PermissionDeniedError if !current_user
 
     if self.owner_uuid.nil?
       errors.add :owner_uuid, "cannot be nil"

-----------------------------------------------------------------------


hooks/post-receive
-- 




More information about the arvados-commits mailing list