[ARVADOS] updated: 42b24c29c8dc75cb61e1690305f399080bb97b80
git at public.curoverse.com
git at public.curoverse.com
Wed Jul 29 10:25:06 EDT 2015
Summary of changes:
.../test/integration/user_manage_account_test.rb | 6 ++--
doc/_includes/_install_ruby_and_bundler.liquid | 2 +-
doc/install/install-sso.html.textile.liquid | 2 +-
sdk/python/arvados/commands/put.py | 19 +++++++---
services/api/app/models/authorized_key.rb | 5 ++-
services/fuse/arvados_fuse/__init__.py | 41 +++++++++++++---------
services/fuse/arvados_fuse/fusedir.py | 11 ++++--
services/fuse/tests/mount_test_base.py | 4 +--
services/fuse/tests/test_mount.py | 27 ++++++++++++--
9 files changed, 80 insertions(+), 37 deletions(-)
discards 7af49bece83dab21129766c55a3ce9ec7f1bcaad (commit)
discards 3f2ceb31279a18bd0f85bcc39e7c36af4a1e9b8c (commit)
discards 7bb74b6131565244c3e0b6cb9edfd34248e88557 (commit)
discards b5e687f68d4204c54044c9786e1c923453515b5f (commit)
via 42b24c29c8dc75cb61e1690305f399080bb97b80 (commit)
via 52cf70df320d02117bd76a6f5af82aa6b95b5960 (commit)
via bf8ec1dd86f6d558941a131940680e7dc330c4e3 (commit)
via 9fd9565ea0411b43943076add49d5bca2da8e379 (commit)
via d22b2efd246ec82a5888d500bcbc7669654b842d (commit)
via 6152230e1f77b925c2041d688805ab6764919805 (commit)
via ff24bb94fdfc9dad61821aef56898e2c196c0aea (commit)
via ed5230f0e823445f56e7d6940b6fb178e328718b (commit)
via 004bafda085386b50e074eedf370ec40a386d6e5 (commit)
via c21473474aba30c12921961483990908ba525b82 (commit)
via 130cbc5cd46272834c2971b40bdba8c32eeee614 (commit)
via 15ac0086cc6623fa3e4c601c19a14fdcd6c139ca (commit)
via 229773c2aa3ddc798ae4eb1eff62f8f77db85391 (commit)
This update added new revisions after undoing existing revisions. That is
to say, the old revision is not a strict subset of the new revision. This
situation occurs when you --force push a change and generate a repository
containing something like this:
* -- * -- B -- O -- O -- O (7af49bece83dab21129766c55a3ce9ec7f1bcaad)
\
N -- N -- N (42b24c29c8dc75cb61e1690305f399080bb97b80)
When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 42b24c29c8dc75cb61e1690305f399080bb97b80
Author: Tom Clegg <tom at curoverse.com>
Date: Sun Jul 26 13:08:55 2015 -0400
6663: Document git setup.
diff --git a/doc/install/create-standard-objects.html.textile.liquid b/doc/install/create-standard-objects.html.textile.liquid
index 1ce4fae..9d688d4 100644
--- a/doc/install/create-standard-objects.html.textile.liquid
+++ b/doc/install/create-standard-objects.html.textile.liquid
@@ -30,6 +30,15 @@ Secondly, create a link object to make the repository object readable by the "Al
EOF</span>
</code></pre></notextile>
+In a couple of minutes, your arvados-git-sync cron job will create an empty repository on your git server. Seed it with the real arvados repository.
+
+<notextile>
+<pre><code>~$ <span class="userinput">cd /tmp</span>
+/tmp$ <span class="userinput">git clone --bare https://github.com/curoverse/arvados.git</span>
+/tmp/arvados$ <span class="userinput">git --git-dir arvados.git push https://git.<b>uuid_prefix.your.domain</b>/arvados.git '*:*'</span>
+</code></pre>
+</notextile>
+
Next, create a default project for the standard Arvados Docker images, and give all users read access to it. The project is owned by the system user.
<notextile>
diff --git a/doc/install/install-api-server.html.textile.liquid b/doc/install/install-api-server.html.textile.liquid
index 0503609..f5d5825 100644
--- a/doc/install/install-api-server.html.textile.liquid
+++ b/doc/install/install-api-server.html.textile.liquid
@@ -94,13 +94,10 @@ Define your @uuid_prefix@ in @application.yml@ by setting the @uuid_prefix@ fiel
h3(#git_repositories_dir). git_repositories_dir
-This field defaults to @/var/lib/arvados/git at . You can override the value by defining it in @application.yml at .
-
-Make sure a clone of the arvados repository exists in @git_repositories_dir at .
+Additional git setup will be done on the "next page":install-arv-git-httpd.html. For now, just create the repository storage directory.
<notextile>
-<pre><code>~$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git</span>
-~$ <span class="userinput">sudo git clone --bare git://git.curoverse.com/arvados.git /var/lib/arvados/git/arvados.git</span>
+<pre><code>~$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git/repositories</span>
</code></pre></notextile>
h3. secret_token
@@ -146,9 +143,10 @@ h2. Prepare the API server deployment
Now that all your configuration is in place, run @/usr/local/bin/arvados-api-server-upgrade.sh at . This will install and check your configuration, install necessary gems, and run any necessary database setup.
{% include 'notebox_begin' %}
-You can safely ignore the following error message you may see when loading the database structure:
-<notextile>
-<pre><code>ERROR: must be owner of extension plpgsql</code></pre></notextile>
+You can safely ignore the following messages if they appear while this script runs:
+<pre>Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will
+break this application for all non-root users on this machine.</pre>
+<pre>fatal: Not a git repository (or any of the parent directories): .git</pre>
{% include 'notebox_end' %}
This command aborts when it encounters an error. It's safe to rerun multiple times, so if there's a problem with your configuration, you can fix that and try again.
diff --git a/doc/install/install-arv-git-httpd.html.textile.liquid b/doc/install/install-arv-git-httpd.html.textile.liquid
index 33b1124..de61722 100644
--- a/doc/install/install-arv-git-httpd.html.textile.liquid
+++ b/doc/install/install-arv-git-httpd.html.textile.liquid
@@ -1,21 +1,224 @@
---
layout: default
navsection: installguide
-title: Install Git server
+title: Install the Git server
...
-The arvados-git-httpd server provides HTTP access to hosted git repositories, using Arvados authentication tokens instead of passwords. It is intended to be installed on the system where your git repositories are stored, and accessed through a web proxy that provides SSL support.
+Arvados allows users to create their own private and public git repositories, and clone/push them using SSH and HTTPS.
+
+The git hosting setup involves three components.
+* The "update-git.rb" script polls the API server for the current list of repositories, creates bare repositories, and updates the local permission cache used by gitolite.
+* Gitolite provides SSH access.
+* arvados-git-http provides HTTPS access.
+
+It is not strictly necessary to deploy _both_ SSH and HTTPS access, but we recommend deploying both:
+* SSH is a more appropriate way to authenticate from a user's workstation because it does not require managing tokens on the client side;
+* HTTPS is a more appropriate way to authenticate from a shell VM because it does not depend on SSH agent forwarding (SSH clients' agent forwarding features tend to behave as if the remote machine is fully trusted).
+
+The HTTPS instructions given below will not work if you skip the SSH setup steps.
+
+h2. Set up DNS
By convention, we use the following hostname for the git service:
-<div class="offset1">
-table(table table-bordered table-condensed).
-|git. at uuid_prefix@.your.domain|
-</div>
+<notextile>
+<pre><code>git.<span class="userinput">uuid_prefix</span>.your.domain
+</code></pre>
+</notextile>
+
+{% include 'notebox_begin' %}
+Here, we show how to install the git hosting services *on the same host as your API server.* Using a different host is not yet fully supported. On this page we will refer to it as your git server.
+{% include 'notebox_end' %}
+
+DNS and network configuration should be set up so port 443 reaches your HTTPS proxy, and port 22 reaches your git server.
+
+h2. Generate an API token
+
+On the API server:
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">cd /var/www/arvados-api/current</span>
+gitserver:/var/www/arvados-api/current$ <span class="userinput">sudo -u www-data RAILS_ENV=production bundle exec ./script/create_superuser_token.rb</span>
+4hdqaixi5a027jqn0vyjbwa3xmcue8logzhtsmk1bplgp064fe
+</code></pre>
+</notextile>
+
+Copy that token; you'll need it in a minute.
+
+h2. Install git and other dependencies
+
+On Debian-based systems:
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">sudo apt-get install git openssh-server</span>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">sudo yum install git perl-Data-Dumper openssh-server</span>
+</code></pre>
+</notextile>
+
+h2. Create a "git" user and a storage directory
+
+Hosted repositories will be stored under @/var/lib/arvados/git/@. If you choose a different location, make sure to update the @git_repositories_dir@ entry in your API server's @config/application.yml@ file, preserving the trailing @repositories/@ part.
+
+A new UNIX account called "git" will own the files. This makes git URLs look familiar to users (<code>git@[...]:username/reponame.git</code>).
+
+On Debian- or Red Hat-based systems:
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git/repositories</span>
+gitserver:~$ <span class="userinput">sudo useradd --comment git --home-dir /var/lib/arvados/git git</span>
+gitserver:~$ <span class="userinput">sudo chown -R git:git ~git</span>
+</code></pre>
+</notextile>
+
+The git user needs its own SSH key. (It must be able to run @ssh git at localhost@ from scripts.)
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">sudo -u git -i bash</span>
+git at gitserver:~$ <span class="userinput">ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa</span>
+git at gitserver:~$ <span class="userinput">ssh -o stricthostkeychecking=no localhost cat .ssh/id_rsa.pub</span>
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7aBIDAAgMQN16Pg6eHmvc+D+6TljwCGr4YGUBphSdVb25UyBCeAEgzqRiqy0IjQR2BLtSirXr+1SJAcQfBgI/jwR7FG+YIzJ4ND9JFEfcpq20FvWnMMQ6XD3y3xrZ1/h/RdBNwy4QCqjiXuxDpDB7VNP9/oeAzoATPZGhqjPfNS+RRVEQpC6BzZdsR+S838E53URguBOf9yrPwdHvosZn7VC0akeWQerHqaBIpSfDMtaM4+9s1Gdsz0iP85rtj/6U/K/XOuv2CZsuVZZ52nu3soHnEX2nx2IaXMS3L8Z+lfOXB2T6EaJgXF7Z9ME5K1tx9TSNTRcYCiKztXLNLSbp git at gitserver
+git at gitserver:~$ <span class="userinput">exit</span>
+</code></pre>
+</notextile>
+
+h2. Install gitolite
+
+Check https://github.com/sitaramc/gitolite/tags for the latest stable version (_e.g.,_ @v3.6.3@).
+
+Download and install the version you selected.
+
+<notextile>
+<pre><code>gitserver:~$ <span class="userinput">sudo -u git -i bash</span>
+git at gitserver:~$ <span class="userinput">echo 'PATH=$HOME/bin:$PATH' >.profile</span>
+git at gitserver:~$ <span class="userinput">source .profile</span>
+git at gitserver:~$ <span class="userinput">git clone --branch <b>v3.6.3</b> git://github.com/sitaramc/gitolite</span>
+...
+Note: checking out '5d24ae666bfd2fa9093d67c840eb8d686992083f'.
+...
+git at gitserver:~$ <span class="userinput">mkdir bin</span>
+git at gitserver:~$ <span class="userinput">gitolite/install -ln ~git/bin</span>
+git at gitserver:~$ <span class="userinput">bin/gitolite setup -pk .ssh/id_rsa.pub</span>
+Initialized empty Git repository in /var/lib/arvados/git/repositories/gitolite-admin.git/
+Initialized empty Git repository in /var/lib/arvados/git/repositories/testing.git/
+WARNING: /var/lib/arvados/git/.ssh missing; creating a new one
+ (this is normal on a brand new install)
+WARNING: /var/lib/arvados/git/.ssh/authorized_keys missing; creating a new one
+ (this is normal on a brand new install)
+</code></pre>
+</notextile>
+
+Create the base configuration files.
+
+<notextile>
+<pre><code>git at gitserver:~$ <span class="userinput">git clone git at localhost:gitolite-admin</span>
+Cloning into 'gitolite-admin'...
+warning: You appear to have cloned an empty repository.
+done.
+git at gitserver:~$ <span class="userinput">cd gitolite-admin</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git config user.email arvados</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git config user.name arvados</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git config push.default simple</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">mkdir -p keydir/arvados conf/admin conf/auto</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">cat >conf/admin/arvados.conf <<EOF
+ at arvados_git_user = arvados_git_user
+
+repo @all
+ RW+ = @arvados_git_user
+
+EOF</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">cat >conf/gitolite.conf <<EOF
+include "auto/*.conf"
+include "admin/*.conf"
+EOF</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git add .</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git commit -m -</span>
+git at gitserver:~/gitolite-admin$ <span class="userinput">git push</span>
+[master a4d083b] -
+ 4 files changed, 10 insertions(+), 5 deletions(-)
+ create mode 100644 arvadosaliases.pl
+ create mode 100644 conf/admin/arvados.conf
+ create mode 100644 keydir/arvados_git_user.pub
+git at gitserver:~/gitolite-admin$ <span class="userinput">exit</span>
+</code></pre>
+</notextile>
+
+h2. Configure gitolite
+
+Configure gitolite to look up a repository name like @username/reponame.git@ and find the appropriate bare repository storage directory.
+
+Add the following lines to the top of @~git/.gitolite.rc@:
+
+<notextile>
+<pre><code><span class="userinput">my $repo_aliases;
+my $aliases_src = "$ENV{HOME}/.gitolite/arvadosaliases.pl";
+if ($ENV{HOME} && (-e $aliases_src)) {
+ $repo_aliases = do $aliases_src;
+}
+$repo_aliases ||= {};
+</span></code></pre>
+</notextile>
+
+Add the following lines inside the section that begins @%RC = (@:
+
+<notextile>
+<pre><code><span class="userinput"> REPO_ALIASES => $repo_aliases,
+</span></code></pre>
+</notextile>
+
+Uncomment the 'Alias' line in the section that begins @ENABLE => [@:
+
+<notextile>
+<pre><code><span class="userinput"> # access a repo by another (possibly legacy) name
+ 'Alias',
+</span></code></pre>
+</notextile>
+
+h2. Configure git synchronization
+
+Create a configuration file @/var/www/arvados-api/current/config/arvados-clients.yml@ using the following template, filling in the appropriate values for your system.
+* For @arvados_api_token@, use the token you generated above.
+* For @gitolite_arvados_git_user_key@, provide the public key you generated above, i.e., the contents of @~git/.ssh/id_rsa.pub at .
+
+<notextile>
+<pre><code>production:
+ gitolite_url: /var/lib/arvados/git/repositories/gitolite-admin.git
+ gitolite_tmp: /var/lib/arvados/git
+ arvados_api_host: <span class="userinput">uuid_prefix.example.com</span>
+ arvados_api_token: "<span class="userinput">4hdqaixi5a027jqn0vyjbwa3xmcue8logzhtsmk1bplgp064fe</span>"
+ arvados_api_host_insecure: <span class="userinput">false</span>
+ gitolite_arvados_git_user_key: "<span class="userinput">ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7aBIDAAgMQN16Pg6eHmvc+D+6TljwCGr4YGUBphSdVb25UyBCeAEgzqRiqy0IjQR2BLtSirXr+1SJAcQfBgI/jwR7FG+YIzJ4ND9JFEfcpq20FvWnMMQ6XD3y3xrZ1/h/RdBNwy4QCqjiXuxDpDB7VNP9/oeAzoATPZGhqjPfNS+RRVEQpC6BzZdsR+S838E53URguBOf9yrPwdHvosZn7VC0akeWQerHqaBIpSfDMtaM4+9s1Gdsz0iP85rtj/6U/K/XOuv2CZsuVZZ52nu3soHnEX2nx2IaXMS3L8Z+lfOXB2T6EaJgXF7Z9ME5K1tx9TSNTRcYCiKztXLNLSbp git at gitserver</span>"
+</code></pre>
+</notextile>
+
+h2. Enable the synchronization script
+
+The API server package includes a script that retrieves the current set of repository names and permissions from the API, writes names and permissions to @arvadosaliases.pl@ in a format usable by gitolite, and creates new empty repositories if needed. This script should run every 2 to 5 minutes.
+
+If you are using RVM, create @/etc/cron.d/arvados-git-sync@ with the following content:
+
+<notextile>
+<pre><code><span class="userinput">*/5 * * * * git cd /var/www/arvados-api/current && /usr/local/rvm/bin/rvm-exec default bundle exec script/arvados-git-sync.rb production</span>
+</code></pre>
+</notextile>
+
+Otherwise, create @/etc/cron.d/arvados-git-sync@ with the following content:
+
+<notextile>
+<pre><code><span class="userinput">*/5 * * * * git cd /var/www/arvados-api/current && bundle exec script/arvados-git-sync.rb production</span>
+</code></pre>
+</notextile>
+
+h2. Install the arvados-git-httpd package
-This hostname should resolve from anywhere on the internet.
+This is needed only for HTTPS access.
-h2. Install arvados-git-httpd
+The arvados-git-httpd package provides HTTP access, using Arvados authentication tokens instead of passwords. It is intended to be installed on the system where your git repositories are stored, and accessed through a web proxy that provides SSL support.
On Debian-based systems:
@@ -31,11 +234,11 @@ On Red Hat-based systems:
</code></pre>
</notextile>
-Verify that @arvados-git-httpd@ and @git-http-backend@ are functional:
+Verify that @arvados-git-httpd@ and @git-http-backend@ can be run:
<notextile>
<pre><code>~$ <span class="userinput">arvados-git-httpd -h</span>
-Usage of arv-git-httpd:
+Usage of arvados-git-httpd:
-address="0.0.0.0:80": Address to listen on, "host:port".
-git-command="/usr/bin/git": Path to git executable. Each authenticated request will execute this program with a single argument, "http-backend".
-repo-root="/path/to/cwd": Path to git repositories.
@@ -49,21 +252,58 @@ fatal: No REQUEST_METHOD from server
</code></pre>
</notextile>
-We recommend running @arvados-git-httpd@ under "runit":http://smarden.org/runit/ or something similar.
+h3. Enable arvados-git-httpd
-Your @run@ script should look something like this:
+Install "runit":http://smarden.org/runit/ (if it's not already installed) and configure it to run arvados-git-httpd. Update the API host to match your site.
<notextile>
-<pre><code>export ARVADOS_API_HOST=<span class="userinput">uuid_prefix</span>.your.domain
-exec sudo -u git arvados-git-httpd -address=:9001 -git-command="$(which git)" -repo-root=<span class="userinput">/var/lib/arvados/git</span> 2>&1
+<pre><code>~$ <span class="userinput">sudo apt-get install runit</span>
+~$ <span class="userinput">cd /etc/sv</span>
+/etc/sv$ <span class="userinput">sudo mkdir arvados-git-httpd; cd arvados-git-httpd</span>
+/etc/sv/arvados-git-httpd$ <span class="userinput">sudo mkdir log</span>
+/etc/sv/arvados-git-httpd$ <span class="userinput">sudo sh -c 'cat >log/run' <<'EOF'
+#!/bin/sh
+mkdir -p main
+chown git:git main
+exec chpst -u git:git svlogd -tt main
+EOF</span>
+/etc/sv/arvados-git-httpd$ <span class="userinput">sudo sh -c 'cat >run' <<'EOF'
+#!/bin/sh
+export ARVADOS_API_HOST=<b>uuid_prefix.your.domain</b>
+export GITOLITE_HTTP_HOME=/var/lib/arvados/git
+export PATH="$PATH:/var/lib/arvados/git/bin"
+exec chpst -u git:git arvados-git-httpd -address=:9001 -git-command="$(which git)" -repo-root=<b>/var/lib/arvados/git/repositories</b> 2>&1
+EOF</span>
+/etc/sv/arvados-git-httpd$ <span class="userinput">sudo chmod +x run log/run</span>
</code></pre>
</notextile>
-h3. Set up a reverse proxy with SSL support
+h3. Set up a reverse proxy to provide SSL service
+
+The arvados-git-httpd service will be accessible from anywhere on the internet, so we recommend using SSL.
+
+This is best achieved by putting a reverse proxy with SSL support in front of arvados-git-httpd, running on port 443 and passing requests to @arvados-git-httpd@ on port 9001 (or whatever port you chose in your run script).
-The arvados-git-httpd service will be accessible from anywhere on the internet, so we recommend using SSL for transport encryption.
+<notextile>
+<pre><code><span class="userinput">http {
+ upstream arvados-git-httpd {
+ server localhost:9001;
+ }
+ server {
+ listen *:443 ssl;
+ server_name git.uuid_prefix.example.com;
+ ssl_certificate /root/git.uuid_prefix.example.com.crt;
+ ssl_certificate_key /root/git.uuid_prefix.example.com.key;
+ location / {
+ proxy_pass http://arvados-git-httpd;
+ proxy_set_header X-Forwarded-For $remote_addr;
+ }
+ }
+}
+</span>
+</code></pre>
+</notextile>
-This is best achieved by putting a reverse proxy with SSL support in front of arvados-git-httpd, running on port 443 and passing requests to arvados-git-httpd on port 9001 (or whatever port you chose in your run script).
h3. Tell the API server about the arvados-git-httpd service
diff --git a/doc/install/install-shell-server.html.textile.liquid b/doc/install/install-shell-server.html.textile.liquid
index 71e6b01..df5045e 100644
--- a/doc/install/install-shell-server.html.textile.liquid
+++ b/doc/install/install-shell-server.html.textile.liquid
@@ -15,7 +15,7 @@ h2. Install the Ruby SDK and utilities
If you're using RVM:
<notextile>
-<pre><code>~$ <span class="userinput">sudo rvm-exec gem install arvados-cli</span>
+<pre><code>~$ <span class="userinput">sudo /usr/local/rvm/bin/rvm-exec default gem install arvados-cli</span>
</code></pre>
</notextile>
diff --git a/docker/api/Dockerfile b/docker/api/Dockerfile
index 758c015..6a3428c 100644
--- a/docker/api/Dockerfile
+++ b/docker/api/Dockerfile
@@ -59,9 +59,9 @@ ADD keep_proxy.json /root/
# Set up update-gitolite.rb
RUN mkdir /usr/local/arvados/config -p
-ADD generated/arvados-clients.yml /usr/local/arvados/config/
+ADD generated/arvados-clients.yml /usr/src/arvados/services/api/config/
ADD .gitolite.rc /usr/local/arvados/config/
-ADD update-gitolite.rb /usr/local/arvados/
+RUN ln /usr/src/arvados/services/api/script/arvados-git-sync.rb /usr/local/bin/
# Supervisor.
ADD supervisor.conf /etc/supervisor/conf.d/arvados.conf
diff --git a/docker/api/setup-gitolite.sh.in b/docker/api/setup-gitolite.sh.in
index d953fa7..023ca5d 100755
--- a/docker/api/setup-gitolite.sh.in
+++ b/docker/api/setup-gitolite.sh.in
@@ -60,7 +60,7 @@ git push
su - git -c "git clone --bare git://github.com/curoverse/arvados.git /home/git/repositories/arvados.git"
echo "ARVADOS_API_HOST_INSECURE=yes" > /etc/cron.d/gitolite-update
-echo "*/2 * * * * root /bin/bash -c 'source /etc/profile.d/rvm.sh && /usr/local/arvados/update-gitolite.rb production'" >> /etc/cron.d/gitolite-update
+echo "*/2 * * * * root /bin/bash -c 'source /etc/profile.d/rvm.sh && /usr/src/arvados/services/api/script/arvados-git-sync.rb production'" >> /etc/cron.d/gitolite-update
# Create/update the repos now
. /etc/profile.d/rvm.sh
diff --git a/services/api/config/application.default.yml b/services/api/config/application.default.yml
index 409dea6..5d1402d 100644
--- a/services/api/config/application.default.yml
+++ b/services/api/config/application.default.yml
@@ -86,7 +86,7 @@ common:
# able to submit crunch jobs. To pass the test suites, put a clone
# of the arvados tree in {git_repositories_dir}/arvados.git or
# {git_repositories_dir}/arvados/.git
- git_repositories_dir: /var/lib/arvados/git
+ git_repositories_dir: /var/lib/gitolite/repositories
# This is a (bare) repository that stores commits used in jobs. When a job
# runs, the source commits are first fetched into this repository, then this
diff --git a/docker/api/update-gitolite.rb b/services/api/script/arvados-git-sync.rb
similarity index 96%
rename from docker/api/update-gitolite.rb
rename to services/api/script/arvados-git-sync.rb
index 1f178cb..3a8ed27 100755
--- a/docker/api/update-gitolite.rb
+++ b/services/api/script/arvados-git-sync.rb
@@ -21,19 +21,18 @@ DEBUG = 1
# load and merge in the environment-specific application config info
# if present, overriding base config parameters as specified
-path = File.dirname(__FILE__) + '/config/arvados-clients.yml'
+path = File.absolute_path('../../config/arvados-clients.yml', __FILE__)
if File.exists?(path) then
cp_config = YAML.load_file(path)[ENV['RAILS_ENV']]
else
- puts "Please create a\n " + File.dirname(__FILE__) + "/config/arvados-clients.yml\n file"
+ puts "Please create a\n #{path}\n file"
exit 1
end
gitolite_url = cp_config['gitolite_url']
gitolite_arvados_git_user_key = cp_config['gitolite_arvados_git_user_key']
-gitolite_tmpdir = File.join(File.absolute_path(File.dirname(__FILE__)),
- cp_config['gitolite_tmp'])
+gitolite_tmpdir = cp_config['gitolite_tmp']
gitolite_admin = File.join(gitolite_tmpdir, 'gitolite-admin')
gitolite_admin_keydir = File.join(gitolite_admin, 'keydir')
gitolite_keydir = File.join(gitolite_admin, 'keydir', 'arvados')
commit 52cf70df320d02117bd76a6f5af82aa6b95b5960
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Jul 28 18:11:59 2015 -0400
6663: Fix repository permissions.
Users with is_active==false (except anonymous_user) are not listed by
get_all_permissions.
A repository that is shared with a group is only shared with users who
have permissions on that group, not all users who have permissions on
*any* group.
{User --can_write--> group --can_read--> repo} is correctly listed as
{User --can_read--> repo}, not can_write.
Tests check that the permissions reported by get_all_permissions do
not exceed the API server's interpretation of the current permission
set.
diff --git a/services/api/app/controllers/arvados/v1/repositories_controller.rb b/services/api/app/controllers/arvados/v1/repositories_controller.rb
index fd6ab58..998a060 100644
--- a/services/api/app/controllers/arvados/v1/repositories_controller.rb
+++ b/services/api/app/controllers/arvados/v1/repositories_controller.rb
@@ -3,14 +3,23 @@ class Arvados::V1::RepositoriesController < ApplicationController
skip_before_filter :render_404_if_no_object, :only => :get_all_permissions
before_filter :admin_required, :only => :get_all_permissions
def get_all_permissions
- @users = {}
- User.includes(:authorized_keys).find_each do |u|
- @users[u.uuid] = u
+ users = {}
+ user_aks = {}
+ admins = []
+ User.eager_load(:authorized_keys).find_each do |u|
+ next unless u.is_active or u.uuid == anonymous_user_uuid
+ users[u.uuid] = u
+ user_aks[u.uuid] ||= u.authorized_keys.
+ collect do |ak|
+ {
+ public_key: ak.public_key,
+ authorized_key_uuid: ak.uuid
+ }
+ end
+ admins << u.uuid if u.is_admin
end
- admins = @users.select { |k,v| v.is_admin }
- @user_aks = {}
@repo_info = {}
- Repository.includes(:permissions).find_each do |repo|
+ Repository.eager_load(:permissions).find_each do |repo|
@repo_info[repo.uuid] = {
uuid: repo.uuid,
name: repo.name,
@@ -22,38 +31,33 @@ class Arvados::V1::RepositoriesController < ApplicationController
perms = []
repo.permissions.each do |perm|
if ArvadosModel::resource_class_for_uuid(perm.tail_uuid) == Group
- @users.each do |user_uuid, user|
- user.group_permissions.each do |group_uuid, perm_mask|
- if perm_mask[:manage]
- perms << {name: 'can_manage', user_uuid: user_uuid}
- elsif perm_mask[:write]
- perms << {name: 'can_write', user_uuid: user_uuid}
- elsif perm_mask[:read]
- perms << {name: 'can_read', user_uuid: user_uuid}
- end
+ users.each do |user_uuid, user|
+ perm_mask = user.group_permissions[perm.tail_uuid]
+ if not perm_mask
+ next
+ elsif perm_mask[:manage] and perm.name == 'can_manage'
+ perms << {name: 'can_manage', user_uuid: user_uuid}
+ elsif perm_mask[:write] and ['can_manage', 'can_write'].index perm.name
+ perms << {name: 'can_write', user_uuid: user_uuid}
+ elsif perm_mask[:read]
+ perms << {name: 'can_read', user_uuid: user_uuid}
end
end
- else
+ elsif users[perm.tail_uuid]
+ # user exists and is (active or the anonymous user)
perms << {name: perm.name, user_uuid: perm.tail_uuid}
end
end
# Owner of the repository, and all admins, can RW
- ([repo.owner_uuid] + admins.keys).each do |user_uuid|
+ ([repo.owner_uuid] | admins).each do |user_uuid|
+ # no permissions for inactive user, even when owner of repo
+ next unless users[user_uuid]
perms << {name: 'can_write', user_uuid: user_uuid}
end
perms.each do |perm|
user_uuid = perm[:user_uuid]
- @user_aks[user_uuid] = @users[user_uuid].andand.authorized_keys.andand.
- collect do |ak|
- {
- public_key: ak.public_key,
- authorized_key_uuid: ak.uuid
- }
- end || []
- if @user_aks[user_uuid].any?
- ri = (@repo_info[repo.uuid][:user_permissions][user_uuid] ||= {})
- ri[perm[:name]] = true
- end
+ ri = (@repo_info[repo.uuid][:user_permissions][user_uuid] ||= {})
+ ri[perm[:name]] = true
end
end
@repo_info.values.each do |repo_users|
@@ -72,6 +76,6 @@ class Arvados::V1::RepositoriesController < ApplicationController
end
send_json(kind: 'arvados#RepositoryPermissionSnapshot',
repositories: @repo_info.values,
- user_keys: @user_aks)
+ user_keys: user_aks)
end
end
diff --git a/services/api/test/fixtures/links.yml b/services/api/test/fixtures/links.yml
index 434f9c7..925e466 100644
--- a/services/api/test/fixtures/links.yml
+++ b/services/api/test/fixtures/links.yml
@@ -405,6 +405,20 @@ admin_can_write_aproject:
head_uuid: zzzzz-j7d0g-v955i6s2oi1cbso
properties: {}
+project_viewer_member_of_all_users_group:
+ uuid: zzzzz-o0j2j-cdnq6627g0h0r2x
+ owner_uuid: zzzzz-tpzed-000000000000000
+ created_at: 2015-07-28T21:34:41.361747000Z
+ modified_by_client_uuid: zzzzz-ozdt8-brczlopd8u8d0jr
+ modified_by_user_uuid: zzzzz-tpzed-000000000000000
+ modified_at: 2015-07-28T21:34:41.361747000Z
+ updated_at: 2015-07-28T21:34:41.361747000Z
+ tail_uuid: zzzzz-tpzed-projectviewer1a
+ link_class: permission
+ name: can_read
+ head_uuid: zzzzz-j7d0g-fffffffffffffff
+ properties: {}
+
project_viewer_can_read_project:
uuid: zzzzz-o0j2j-projviewerreadp
owner_uuid: zzzzz-tpzed-000000000000000
diff --git a/services/api/test/functional/arvados/v1/repositories_controller_test.rb b/services/api/test/functional/arvados/v1/repositories_controller_test.rb
index 7ba2183..514bb66 100644
--- a/services/api/test/functional/arvados/v1/repositories_controller_test.rb
+++ b/services/api/test/functional/arvados/v1/repositories_controller_test.rb
@@ -42,6 +42,26 @@ class Arvados::V1::RepositoriesControllerTest < ActionController::TestCase
end
end
+ test "get_all_permissions takes into account is_active flag" do
+ r = nil
+ act_as_user users(:active) do
+ r = Repository.create! name: 'active/testrepo'
+ end
+ act_as_system_user do
+ u = users(:active)
+ u.is_active = false
+ u.save!
+ end
+ authorize_with :admin
+ get :get_all_permissions
+ assert_response :success
+ json_response['repositories'].each do |r|
+ r['user_permissions'].each do |user_uuid, perms|
+ refute_equal user_uuid, users(:active).uuid
+ end
+ end
+ end
+
test "get_all_permissions does not give any access to user without permission" do
viewer_uuid = users(:project_viewer).uuid
assert_equal(authorized_keys(:project_viewer).authorized_user_uuid,
@@ -88,15 +108,84 @@ class Arvados::V1::RepositoriesControllerTest < ActionController::TestCase
end
end
- test "get_all_permissions lists repos with no authorized keys" do
+ test "get_all_permissions lists all repos regardless of permissions" do
+ act_as_system_user do
+ # Create repos that could potentially be left out of the
+ # permission list by accident.
+
+ # No authorized_key, no username (this can't even be done
+ # without skipping validations)
+ r = Repository.create name: 'root/testrepo'
+ assert r.save validate: false
+
+ r = Repository.create name: 'invalid username / repo name', owner_uuid: users(:inactive).uuid
+ assert r.save validate: false
+ end
+ authorize_with :admin
+ get :get_all_permissions
+ assert_response :success
+ assert_equal(Repository.count, json_response["repositories"].size)
+ end
+
+ test "get_all_permissions lists user permissions for users with no authorized keys" do
authorize_with :admin
AuthorizedKey.destroy_all
get :get_all_permissions
assert_response :success
assert_equal(Repository.count, json_response["repositories"].size)
- assert(json_response["repositories"].any? do |repo|
- repo["user_permissions"].empty?
- end, "test is invalid - all repositories have authorized keys")
+ repos_with_perms = []
+ json_response['repositories'].each do |repo|
+ if repo['user_permissions'].any?
+ repos_with_perms << repo['uuid']
+ end
+ end
+ assert_not_empty repos_with_perms, 'permissions are missing'
+ end
+
+ # Ensure get_all_permissions correctly describes what the normal
+ # permission system would do.
+ test "get_all_permissions obeys group permissions" do
+ act_as_user system_user do
+ r = Repository.create!(name: 'admin/groupcanwrite', owner_uuid: users(:admin).uuid)
+ g = Group.create!(group_class: 'group', name: 'repo-writers')
+ u1 = users(:active)
+ u2 = users(:spectator)
+ Link.create!(tail_uuid: g.uuid, head_uuid: r.uuid, link_class: 'permission', name: 'can_manage')
+ Link.create!(tail_uuid: u1.uuid, head_uuid: g.uuid, link_class: 'permission', name: 'can_write')
+ Link.create!(tail_uuid: u2.uuid, head_uuid: g.uuid, link_class: 'permission', name: 'can_read')
+
+ r = Repository.create!(name: 'admin/groupreadonly', owner_uuid: users(:admin).uuid)
+ g = Group.create!(group_class: 'group', name: 'repo-readers')
+ u1 = users(:active)
+ u2 = users(:spectator)
+ Link.create!(tail_uuid: g.uuid, head_uuid: r.uuid, link_class: 'permission', name: 'can_read')
+ Link.create!(tail_uuid: u1.uuid, head_uuid: g.uuid, link_class: 'permission', name: 'can_write')
+ Link.create!(tail_uuid: u2.uuid, head_uuid: g.uuid, link_class: 'permission', name: 'can_read')
+ end
+ authorize_with :admin
+ get :get_all_permissions
+ assert_response :success
+ json_response['repositories'].each do |repo|
+ repo['user_permissions'].each do |user_uuid, perms|
+ u = User.find_by_uuid(user_uuid)
+ if perms['can_read']
+ assert u.can? read: repo['uuid']
+ assert_match /R/, perms['gitolite_permissions']
+ else
+ refute_match /R/, perms['gitolite_permissions']
+ end
+ if perms['can_write']
+ assert u.can? write: repo['uuid']
+ assert_match /RW/, perms['gitolite_permissions']
+ else
+ refute_match /W/, perms['gitolite_permissions']
+ end
+ if perms['can_manage']
+ assert u.can? manage: repo['uuid']
+ assert_match /RW/, perms['gitolite_permissions']
+ end
+ end
+ end
end
test "default index includes fetch_url" do
commit bf8ec1dd86f6d558941a131940680e7dc330c4e3
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Jul 28 17:58:25 2015 -0400
6663: Fix "duplicate public_key" test: OK if the key being updated has the same public_key.
diff --git a/apps/workbench/test/integration/user_manage_account_test.rb b/apps/workbench/test/integration/user_manage_account_test.rb
index cc28b27..1b80daf 100644
--- a/apps/workbench/test/integration/user_manage_account_test.rb
+++ b/apps/workbench/test/integration/user_manage_account_test.rb
@@ -42,13 +42,13 @@ class UserManageAccountTest < ActionDispatch::IntegrationTest
page.find_field('public_key').set 'first test with an incorrect ssh key value'
click_button 'Submit'
- assert page.has_text?('Public key does not appear to be a valid ssh-rsa or dsa public key'), 'No text - Public key does not appear to be a valid'
+ assert_text 'Public key does not appear to be a valid ssh-rsa or dsa public key'
public_key_str = api_fixture('authorized_keys')['active']['public_key']
page.find_field('public_key').set public_key_str
page.find_field('name').set 'added_in_test'
click_button 'Submit'
- assert page.has_text?('Public key already exists in the database, use a different key.'), 'No text - Public key already exists'
+ assert_text 'Public key already exists in the database, use a different key.'
new_key = SSHKey.generate
page.find_field('public_key').set new_key.ssh_public_key
@@ -57,7 +57,7 @@ class UserManageAccountTest < ActionDispatch::IntegrationTest
end
# key must be added. look for it in the refreshed page
- assert page.has_text?('added_in_test'), 'No text - added_in_test'
+ assert_text 'added_in_test'
end
[
diff --git a/services/api/app/models/authorized_key.rb b/services/api/app/models/authorized_key.rb
index b156a1d..452cd69 100644
--- a/services/api/app/models/authorized_key.rb
+++ b/services/api/app/models/authorized_key.rb
@@ -33,14 +33,14 @@ class AuthorizedKey < ArvadosModel
def public_key_must_be_unique
if self.public_key
- #key = /^ssh-(rsa|dss) [A-Za-z0-9+\/=\+]+\b/.match(self.public_key)
valid_key = SSHKey.valid_ssh_public_key? self.public_key
if not valid_key
errors.add(:public_key, "does not appear to be a valid ssh-rsa or dsa public key")
else
# Valid if no other rows have this public key
- if self.class.where('public_key like ?', "%#{self.public_key}%").any?
+ if self.class.where('uuid != ? and public_key like ?',
+ uuid || '', "%#{self.public_key}%").any?
errors.add(:public_key, "already exists in the database, use a different key.")
return false
end
diff --git a/services/api/test/unit/authorized_key_test.rb b/services/api/test/unit/authorized_key_test.rb
index b8d9b67..9ce3929 100644
--- a/services/api/test/unit/authorized_key_test.rb
+++ b/services/api/test/unit/authorized_key_test.rb
@@ -1,7 +1,46 @@
require 'test_helper'
class AuthorizedKeyTest < ActiveSupport::TestCase
- # test "the truth" do
- # assert true
- # end
+ TEST_KEY = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCf5aTI55uyWr44TckP/ELUAyPsdnf5fTZDcSDN4qiMZYAL7TYV2ixwnbPObLObM0GmHSSFLV1KqsuFICUPgkyKoHbAH6XPgmtfOLU60VkGf1v5uxQ/kXCECRCJmPb3K9dIXGEw+1DXPdOV/xG7rJNvo4a9WK9iqqZr8p+VGKM6C017b8BDLk0tuEEjZ5jXcT/ka/hTScxWkKgF6auPOVQ79OA5+0VaYm4uQLzVUdgwVUPWQQecRrtnc08XYM1htpcLDIAbWfUNK7uE6XR3/OhtrJGf05FGbtGguPgi33F9W3Q3yw6saOK5Y3TfLbskgFaEdLgzqK/QSBRk2zBF49Tj test at localhost"
+
+ test 'create and update key' do
+ u1 = users(:active)
+ act_as_user u1 do
+ ak = AuthorizedKey.create(name: "foo", public_key: TEST_KEY, authorized_user_uuid: u1.uuid)
+ assert ak.save, ak.errors.full_messages.to_s
+ ak.name = "bar"
+ assert ak.valid?, ak.errors.full_messages.to_s
+ assert ak.save, ak.errors.full_messages.to_s
+ end
+ end
+
+ test 'duplicate key not permitted' do
+ u1 = users(:active)
+ act_as_user u1 do
+ ak = AuthorizedKey.create!(name: "foo", public_key: TEST_KEY, authorized_user_uuid: u1.uuid)
+ end
+ u2 = users(:spectator)
+ act_as_user u2 do
+ ak2 = AuthorizedKey.create(name: "bar", public_key: TEST_KEY, authorized_user_uuid: u2.uuid)
+ refute ak2.valid?
+ refute ak2.save
+ assert_match /already exists/, ak2.errors.full_messages.to_s
+ end
+ end
+
+ test 'attach key to wrong user account' do
+ act_as_user users(:active) do
+ ak = AuthorizedKey.create(name: "foo", public_key: TEST_KEY)
+ ak.authorized_user_uuid = users(:spectator).uuid
+ refute ak.save
+ ak.uuid = nil
+ ak.authorized_user_uuid = users(:admin).uuid
+ refute ak.save
+ ak.uuid = nil
+ ak.authorized_user_uuid = users(:active).uuid
+ assert ak.save, ak.errors.full_messages.to_s
+ ak.authorized_user_uuid = users(:admin).uuid
+ refute ak.save
+ end
+ end
end
commit 9fd9565ea0411b43943076add49d5bca2da8e379
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Jul 28 17:52:30 2015 -0400
6663: Fix using default owner_uuid in repositories#create. refs #6652
diff --git a/services/api/app/models/arvados_model.rb b/services/api/app/models/arvados_model.rb
index d5b2f87..35dd1a9 100644
--- a/services/api/app/models/arvados_model.rb
+++ b/services/api/app/models/arvados_model.rb
@@ -23,6 +23,7 @@ class ArvadosModel < ActiveRecord::Base
after_destroy :log_destroy
after_find :convert_serialized_symbols_to_strings
before_validation :normalize_collection_uuids
+ before_validation :set_default_owner
validate :ensure_serialized_attribute_type
validate :ensure_valid_uuids
@@ -276,12 +277,14 @@ class ArvadosModel < ActiveRecord::Base
true
end
- def ensure_owner_uuid_is_permitted
- raise PermissionDeniedError if !current_user
-
- if new_record? and respond_to? :owner_uuid=
+ def set_default_owner
+ if new_record? and current_user and respond_to? :owner_uuid=
self.owner_uuid ||= current_user.uuid
end
+ end
+
+ def ensure_owner_uuid_is_permitted
+ raise PermissionDeniedError if !current_user
if self.owner_uuid.nil?
errors.add :owner_uuid, "cannot be nil"
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list