[ARVADOS] created: 2.0.4-3-gdc5d5d67e
Git user
git at public.arvados.org
Thu Jan 6 17:51:24 UTC 2022
at dc5d5d67ea7a4c58e55fc5116948e1ffe12304f4 (commit)
commit dc5d5d67ea7a4c58e55fc5116948e1ffe12304f4
Author: Ward Vandewege <ward at curii.com>
Date: Thu Sep 23 09:19:59 2021 -0400
Fix typo.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid b/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
index 2375e8b3d..a6f64e139 100644
--- a/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
@@ -29,7 +29,7 @@ table(table table-bordered table-condensed).
|_. collection state|_. is_trashed|_. trash_at|_. delete_at|_. get|_. list|_. list?include_trash=true|_. can be modified|
|persisted collection|false |null |null |yes |yes |yes |yes |
|expiring collection|false |future |future |yes |yes |yes |yes |
-|trashed collection|true |past |future |no |no |yes |only is_trashed, trash_at and delete_at attribtues|
+|trashed collection|true |past |future |no |no |yes |only is_trashed, trash_at and delete_at attributes|
|deleted collection|true|past |past |no |no |no |no |
h2(#delete-collection). Deleting / trashing collections
commit 9b6a214f4feb90845640bce53efa34d544c701e9
Author: Ward Vandewege <ward at curii.com>
Date: Mon Aug 17 15:18:53 2020 -0400
16434: fix packaging bug for python3-arvados-cwl-runner, for real.
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/sdk/cwl/fpm-info.sh b/sdk/cwl/fpm-info.sh
index 977f53da6..569dfd216 100644
--- a/sdk/cwl/fpm-info.sh
+++ b/sdk/cwl/fpm-info.sh
@@ -6,11 +6,11 @@ case "$TARGET" in
debian8)
fpm_depends+=(libgnutls-deb0-28 libcurl3-gnutls)
;;
- debian* | ubuntu*)
+ debian9 | ubuntu1604)
fpm_depends+=(libcurl3-gnutls libpython2.7)
;;
debian* | ubuntu*)
- fpm_depends+=(libcurl3-gnutls python3-distutils)
+ fpm_depends+=(libcurl3-gnutls libpython2.7 python3-distutils)
;;
esac
commit c610ee2ac5bdcdcbc2f460584f94cdb8948689b2
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Aug 17 14:36:28 2020 -0400
Add link from upgrade notes to general release notes.
no issue #
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/admin/upgrading.html.textile.liquid b/doc/admin/upgrading.html.textile.liquid
index df64904a8..cf526d6f4 100644
--- a/doc/admin/upgrading.html.textile.liquid
+++ b/doc/admin/upgrading.html.textile.liquid
@@ -10,14 +10,14 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-What you need to know and do in order to upgrade your Arvados installation.
+For Arvados administrators, this page will cover what you need to know and do in order to ensure a smooth upgrade of your Arvados installation. For general release notes covering features added and bugs fixed, see "Arvados releases":https://arvados.org/releases .
h2. General process
# Consult upgrade notes below to see if any manual configuration updates are necessary.
# Wait for the cluster to be idle and stop Arvados services.
# Install new packages using @apt-get upgrade@ or @yum upgrade at .
-# Package installation scripts will perform any necessary data migrations.
+# Wait for package installation scripts as they perform any necessary data migrations.
# Restart Arvados services.
h2. Upgrade notes
commit 95594ca8821d6c8f7e13fa788b4dc160fc3bbb53
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu Aug 13 15:22:49 2020 -0400
16683: Check that remote cluster id is presumed valid, add test
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/services/api/app/models/link.rb b/services/api/app/models/link.rb
index c580d63b0..bf107f575 100644
--- a/services/api/app/models/link.rb
+++ b/services/api/app/models/link.rb
@@ -47,6 +47,7 @@ class Link < ArvadosModel
!attr_value.nil? &&
self.link_class == 'permission' &&
attr_value[0..4] != Rails.configuration.ClusterID &&
+ ApiClientAuthorization.remote_host(uuid_prefix: attr_value[0..4]) &&
ArvadosModel::resource_class_for_uuid(attr_value) == User
# Permission link tail is a remote user (the user permissions
# are being granted to), so bypass the standard check that a
diff --git a/services/api/test/unit/link_test.rb b/services/api/test/unit/link_test.rb
index 00f3cc291..c7d21bdc4 100644
--- a/services/api/test/unit/link_test.rb
+++ b/services/api/test/unit/link_test.rb
@@ -58,6 +58,14 @@ class LinkTest < ActiveSupport::TestCase
users(:active).uuid.sub(/-\w+$/, "-#{'z' * 15}"))
end
+ test "link granting permission to remote user is valid" do
+ refute new_active_link_valid?(tail_uuid:
+ users(:active).uuid.sub(/^\w+-/, "foooo-"))
+ Rails.configuration.RemoteClusters = Rails.configuration.RemoteClusters.merge({foooo: ActiveSupport::InheritableOptions.new({Host: "bar.com"})})
+ assert new_active_link_valid?(tail_uuid:
+ users(:active).uuid.sub(/^\w+-/, "foooo-"))
+ end
+
test "link granting non-project permission to unreadable user is invalid" do
refute new_active_link_valid?(tail_uuid: users(:admin).uuid,
head_uuid: collections(:bar_file).uuid)
commit 35f0b6522e32e4f7601760efaca3e900ca918072
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Wed Aug 12 17:32:06 2020 -0400
16683: Permit granting permissions to remote users
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index ec04cecec..4695a9504 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -85,7 +85,7 @@ type ListOptions struct {
IncludeTrash bool `json:"include_trash"`
IncludeOldVersions bool `json:"include_old_versions"`
BypassFederation bool `json:"bypass_federation"`
- ForwardedFor string `json:"forwarded_for,omitempty"`
+ ForwardedFor string `json:"forwarded_for,omitempty"`
}
type CreateOptions struct {
diff --git a/services/api/app/models/arvados_model.rb b/services/api/app/models/arvados_model.rb
index 816dbf475..7f6ab462d 100644
--- a/services/api/app/models/arvados_model.rb
+++ b/services/api/app/models/arvados_model.rb
@@ -716,6 +716,20 @@ class ArvadosModel < ApplicationRecord
%r/[a-z0-9]{5}-#{uuid_prefix}-[a-z0-9]{15}/
end
+ def check_readable_uuid attr, attr_value
+ return if attr_value.nil?
+ if (r = ArvadosModel::resource_class_for_uuid attr_value)
+ unless skip_uuid_read_permission_check.include? attr
+ r = r.readable_by(current_user)
+ end
+ if r.where(uuid: attr_value).count == 0
+ errors.add(attr, "'#{attr_value}' not found")
+ end
+ else
+ # Not a valid uuid or PDH, but that (currently) is not an error.
+ end
+ end
+
def ensure_valid_uuids
specials = [system_user_uuid]
@@ -724,16 +738,7 @@ class ArvadosModel < ApplicationRecord
next if skip_uuid_existence_check.include? attr
attr_value = send attr
next if specials.include? attr_value
- if attr_value
- if (r = ArvadosModel::resource_class_for_uuid attr_value)
- unless skip_uuid_read_permission_check.include? attr
- r = r.readable_by(current_user)
- end
- if r.where(uuid: attr_value).count == 0
- errors.add(attr, "'#{attr_value}' not found")
- end
- end
- end
+ check_readable_uuid attr, attr_value
end
end
end
diff --git a/services/api/app/models/link.rb b/services/api/app/models/link.rb
index ad7800fe6..c580d63b0 100644
--- a/services/api/app/models/link.rb
+++ b/services/api/app/models/link.rb
@@ -42,6 +42,27 @@ class Link < ArvadosModel
protected
+ def check_readable_uuid attr, attr_value
+ if attr == 'tail_uuid' &&
+ !attr_value.nil? &&
+ self.link_class == 'permission' &&
+ attr_value[0..4] != Rails.configuration.ClusterID &&
+ ArvadosModel::resource_class_for_uuid(attr_value) == User
+ # Permission link tail is a remote user (the user permissions
+ # are being granted to), so bypass the standard check that a
+ # referenced object uuid is readable by current user.
+ #
+ # We could do a call to the remote cluster to check if the user
+ # in tail_uuid exists. This would detect copy-and-paste errors,
+ # but add another way for the request to fail, and I don't think
+ # it would improve security. It doesn't seem to be worth the
+ # complexity tradeoff.
+ true
+ else
+ super
+ end
+ end
+
def permission_to_attach_to_objects
# Anonymous users cannot write links
return false if !current_user
@@ -54,6 +75,11 @@ class Link < ArvadosModel
head_obj = ArvadosModel.find_by_uuid(head_uuid)
+ if head_obj.nil?
+ errors.add(:head_uuid, "does not exist")
+ return false
+ end
+
# No permission links can be pointed to past collection versions
return false if head_obj.is_a?(Collection) && head_obj.current_version_uuid != head_uuid
commit 0cc5bd9af9f936c2eeffb31c4f417b0f43eed036
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Wed Aug 12 16:20:19 2020 -0400
16683: Sharing dialog query is compatible with federated user list
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/apps/workbench/app/views/application/_show_sharing.html.erb b/apps/workbench/app/views/application/_show_sharing.html.erb
index 7877e60d3..75773ab90 100644
--- a/apps/workbench/app/views/application/_show_sharing.html.erb
+++ b/apps/workbench/app/views/application/_show_sharing.html.erb
@@ -8,6 +8,8 @@ SPDX-License-Identifier: AGPL-3.0 %>
[User, Group].each do |type|
type
.filter([['uuid','in', at share_links.collect(&:tail_uuid)]])
+ .with_count("none")
+ .fetch_multiple_pages(false)
.each do |o|
uuid_map[o.uuid] = o
end
commit 93480ec377d8d78352cb900e9d894593daa4b3ef
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Wed Aug 12 16:10:32 2020 -0400
16683: Set and check ForwardedFor on federated list requests
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/federation/generated.go b/lib/controller/federation/generated.go
index 20edd90b9..8745f3b97 100755
--- a/lib/controller/federation/generated.go
+++ b/lib/controller/federation/generated.go
@@ -23,6 +23,7 @@ func (conn *Conn) generated_ContainerList(ctx context.Context, options arvados.L
var needSort atomic.Value
needSort.Store(false)
err := conn.splitListRequest(ctx, options, func(ctx context.Context, _ string, backend arvados.API, options arvados.ListOptions) ([]string, error) {
+ options.ForwardedFor = conn.cluster.ClusterID + "-" + options.ForwardedFor
cl, err := backend.ContainerList(ctx, options)
if err != nil {
return nil, err
@@ -63,6 +64,7 @@ func (conn *Conn) generated_SpecimenList(ctx context.Context, options arvados.Li
var needSort atomic.Value
needSort.Store(false)
err := conn.splitListRequest(ctx, options, func(ctx context.Context, _ string, backend arvados.API, options arvados.ListOptions) ([]string, error) {
+ options.ForwardedFor = conn.cluster.ClusterID + "-" + options.ForwardedFor
cl, err := backend.SpecimenList(ctx, options)
if err != nil {
return nil, err
@@ -103,6 +105,7 @@ func (conn *Conn) generated_UserList(ctx context.Context, options arvados.ListOp
var needSort atomic.Value
needSort.Store(false)
err := conn.splitListRequest(ctx, options, func(ctx context.Context, _ string, backend arvados.API, options arvados.ListOptions) ([]string, error) {
+ options.ForwardedFor = conn.cluster.ClusterID + "-" + options.ForwardedFor
cl, err := backend.UserList(ctx, options)
if err != nil {
return nil, err
diff --git a/lib/controller/federation/list.go b/lib/controller/federation/list.go
index 0a596eb9c..bc6d3e00a 100644
--- a/lib/controller/federation/list.go
+++ b/lib/controller/federation/list.go
@@ -27,6 +27,7 @@ func (conn *Conn) generated_CollectionList(ctx context.Context, options arvados.
var needSort atomic.Value
needSort.Store(false)
err := conn.splitListRequest(ctx, options, func(ctx context.Context, _ string, backend arvados.API, options arvados.ListOptions) ([]string, error) {
+ options.ForwardedFor = conn.cluster.ClusterID + "-" + options.ForwardedFor
cl, err := backend.CollectionList(ctx, options)
if err != nil {
return nil, err
@@ -107,7 +108,7 @@ func (conn *Conn) generated_CollectionList(ctx context.Context, options arvados.
// backend.
func (conn *Conn) splitListRequest(ctx context.Context, opts arvados.ListOptions, fn func(context.Context, string, arvados.API, arvados.ListOptions) ([]string, error)) error {
- if opts.BypassFederation {
+ if opts.BypassFederation || opts.ForwardedFor != "" {
// Client requested no federation. Pass through.
_, err := fn(ctx, conn.cluster.ClusterID, conn.local, opts)
return err
@@ -249,7 +250,7 @@ func (conn *Conn) splitListRequest(ctx context.Context, opts arvados.ListOptions
done, err := fn(ctx, clusterID, backend, remoteOpts)
if err != nil {
- errs <- httpErrorf(http.StatusBadGateway, err.Error())
+ errs <- httpErrorf(http.StatusBadGateway, "%s", err.Error())
return
}
progress := false
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index e60108335..ec04cecec 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -85,6 +85,7 @@ type ListOptions struct {
IncludeTrash bool `json:"include_trash"`
IncludeOldVersions bool `json:"include_old_versions"`
BypassFederation bool `json:"bypass_federation"`
+ ForwardedFor string `json:"forwarded_for,omitempty"`
}
type CreateOptions struct {
commit 65803831b766cf0abf8286937a78a8fe05419d3c
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu Aug 13 17:21:35 2020 -0400
Merge federation-migrate fixes refs #16589
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/python/arvados/commands/federation_migrate.py b/sdk/python/arvados/commands/federation_migrate.py
index 445775cce..5c1bb29e7 100755
--- a/sdk/python/arvados/commands/federation_migrate.py
+++ b/sdk/python/arvados/commands/federation_migrate.py
@@ -22,6 +22,7 @@ import hmac
import urllib.parse
import os
import hashlib
+import re
from arvados._version import __version__
EMAIL=0
@@ -169,19 +170,20 @@ def read_migrations(args, by_email, by_username):
def update_username(args, email, user_uuid, username, migratecluster, migratearv):
print("(%s) Updating username of %s to '%s' on %s" % (email, user_uuid, username, migratecluster))
- if not args.dry_run:
- try:
- conflicts = migratearv.users().list(filters=[["username", "=", username]], bypass_federation=True).execute()
- if conflicts["items"]:
- # There's already a user with the username, move the old user out of the way
- migratearv.users().update(uuid=conflicts["items"][0]["uuid"],
- bypass_federation=True,
- body={"user": {"username": username+"migrate"}}).execute()
- migratearv.users().update(uuid=user_uuid,
- bypass_federation=True,
- body={"user": {"username": username}}).execute()
- except arvados.errors.ApiError as e:
- print("(%s) Error updating username of %s to '%s' on %s: %s" % (email, user_uuid, username, migratecluster, e))
+ if args.dry_run:
+ return
+ try:
+ conflicts = migratearv.users().list(filters=[["username", "=", username]], bypass_federation=True).execute()
+ if conflicts["items"]:
+ # There's already a user with the username, move the old user out of the way
+ migratearv.users().update(uuid=conflicts["items"][0]["uuid"],
+ bypass_federation=True,
+ body={"user": {"username": username+"migrate"}}).execute()
+ migratearv.users().update(uuid=user_uuid,
+ bypass_federation=True,
+ body={"user": {"username": username}}).execute()
+ except arvados.errors.ApiError as e:
+ print("(%s) Error updating username of %s to '%s' on %s: %s" % (email, user_uuid, username, migratecluster, e))
def choose_new_user(args, by_email, email, userhome, username, old_user_uuid, clusters):
@@ -212,11 +214,17 @@ def choose_new_user(args, by_email, email, userhome, username, old_user_uuid, cl
conflicts = homearv.users().list(filters=[["username", "=", username]],
bypass_federation=True).execute()
if conflicts["items"]:
- homearv.users().update(uuid=conflicts["items"][0]["uuid"],
- bypass_federation=True,
- body={"user": {"username": username+"migrate"}}).execute()
- user = homearv.users().create(body={"user": {"email": email, "username": username,
- "is_active": olduser["is_active"]}}).execute()
+ homearv.users().update(
+ uuid=conflicts["items"][0]["uuid"],
+ bypass_federation=True,
+ body={"user": {"username": username+"migrate"}}).execute()
+ user = homearv.users().create(
+ body={"user": {
+ "email": email,
+ "first_name": olduser["first_name"],
+ "last_name": olduser["last_name"],
+ "username": username,
+ "is_active": olduser["is_active"]}}).execute()
except arvados.errors.ApiError as e:
print("(%s) Could not create user: %s" % (email, str(e)))
return None
@@ -271,7 +279,7 @@ def activate_remote_user(args, email, homearv, migratearv, old_user_uuid, new_us
newuser = arvados.api(host=ru.netloc, token=salted,
insecure=os.environ.get("ARVADOS_API_HOST_INSECURE")).users().current().execute()
else:
- newuser = {"is_active": True, "username": username}
+ newuser = {"is_active": True, "username": email.split('@')[0], "is_admin": False}
except arvados.errors.ApiError as e:
print("(%s) Error getting user info for %s from %s: %s" % (email, new_user_uuid, migratecluster, e))
return None
@@ -287,39 +295,48 @@ def activate_remote_user(args, email, homearv, migratearv, old_user_uuid, new_us
return None
if olduser["is_admin"] and not newuser["is_admin"]:
- print("(%s) Not migrating %s because user is admin but target user %s is not admin on %s" % (email, old_user_uuid, new_user_uuid, migratecluster))
+ print("(%s) Not migrating %s because user is admin but target user %s is not admin on %s. Please ensure the user admin status is the same on both clusters. Note that a federated admin account has admin privileges on the entire federation." % (email, old_user_uuid, new_user_uuid, migratecluster))
return None
return newuser
def migrate_user(args, migratearv, email, new_user_uuid, old_user_uuid):
+ if args.dry_run:
+ return
try:
- if not args.dry_run:
+ new_owner_uuid = new_user_uuid
+ if args.data_into_subproject:
grp = migratearv.groups().create(body={
"owner_uuid": new_user_uuid,
"name": "Migrated from %s (%s)" % (email, old_user_uuid),
"group_class": "project"
}, ensure_unique_name=True).execute()
- migratearv.users().merge(old_user_uuid=old_user_uuid,
- new_user_uuid=new_user_uuid,
- new_owner_uuid=grp["uuid"],
- redirect_to_new_user=True).execute()
+ new_owner_uuid = grp["uuid"]
+ migratearv.users().merge(old_user_uuid=old_user_uuid,
+ new_user_uuid=new_user_uuid,
+ new_owner_uuid=new_owner_uuid,
+ redirect_to_new_user=True).execute()
except arvados.errors.ApiError as e:
- print("(%s) Error migrating user: %s" % (email, e))
+ name_collision = re.search(r'Key \(owner_uuid, name\)=\((.*?), (.*?)\) already exists\.\n.*UPDATE "(.*?)"', e._get_reason())
+ if name_collision:
+ target_owner, rsc_name, rsc_type = name_collision.groups()
+ print("(%s) Cannot migrate to %s because both origin and target users have a %s named '%s'. Please rename the conflicting items or use --data-into-subproject to migrate all users' data into a special subproject." % (email, target_owner, rsc_type[:-1], rsc_name))
+ else:
+ print("(%s) Skipping user migration because of error: %s" % (email, e))
def main():
-
parser = argparse.ArgumentParser(description='Migrate users to federated identity, see https://doc.arvados.org/admin/merge-remote-account.html')
parser.add_argument(
'--version', action='version', version="%s %s" % (sys.argv[0], __version__),
help='Print version and exit.')
- parser.add_argument('--tokens', type=str, required=False)
+ parser.add_argument('--tokens', type=str, metavar='FILE', required=False, help="Read tokens from FILE. Not needed when using LoginCluster.")
+ parser.add_argument('--data-into-subproject', action="store_true", help="Migrate user's data into a separate subproject. This can be used to avoid name collisions from within an account.")
group = parser.add_mutually_exclusive_group(required=True)
- group.add_argument('--report', type=str, help="Generate report .csv file listing users by email address and their associated Arvados accounts")
- group.add_argument('--migrate', type=str, help="Consume report .csv and migrate users to designated Arvados accounts")
- group.add_argument('--dry-run', type=str, help="Consume report .csv and report how user would be migrated to designated Arvados accounts")
- group.add_argument('--check', action="store_true", help="Check that tokens are usable and the federation is well connected")
+ group.add_argument('--report', type=str, metavar='FILE', help="Generate report .csv file listing users by email address and their associated Arvados accounts.")
+ group.add_argument('--migrate', type=str, metavar='FILE', help="Consume report .csv and migrate users to designated Arvados accounts.")
+ group.add_argument('--dry-run', type=str, metavar='FILE', help="Consume report .csv and report how user would be migrated to designated Arvados accounts.")
+ group.add_argument('--check', action="store_true", help="Check that tokens are usable and the federation is well connected.")
args = parser.parse_args()
clusters, errors, loginCluster = connect_clusters(args)
commit 00964013f61975cd03ce2aed810c0543402ca707
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Wed Jul 22 14:58:59 2020 -0400
16623: Add NetworkResourceGroup option for Azure
refs #16623
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/cloud/azure/azure.go b/lib/cloud/azure/azure.go
index 752de152d..6de367aa2 100644
--- a/lib/cloud/azure/azure.go
+++ b/lib/cloud/azure/azure.go
@@ -43,6 +43,7 @@ type azureInstanceSetConfig struct {
ResourceGroup string
Location string
Network string
+ NetworkResourceGroup string
Subnet string
StorageAccount string
BlobContainer string
@@ -356,6 +357,11 @@ func (az *azureInstanceSet) Create(
}
tags["created-at"] = to.StringPtr(time.Now().Format(time.RFC3339Nano))
+ networkResourceGroup := az.azconfig.NetworkResourceGroup
+ if networkResourceGroup == "" {
+ networkResourceGroup = az.azconfig.ResourceGroup
+ }
+
nicParameters := network.Interface{
Location: &az.azconfig.Location,
Tags: tags,
@@ -368,7 +374,7 @@ func (az *azureInstanceSet) Create(
ID: to.StringPtr(fmt.Sprintf("/subscriptions/%s/resourceGroups/%s/providers"+
"/Microsoft.Network/virtualnetworks/%s/subnets/%s",
az.azconfig.SubscriptionID,
- az.azconfig.ResourceGroup,
+ networkResourceGroup,
az.azconfig.Network,
az.azconfig.Subnet)),
},
diff --git a/lib/config/config.default.yml b/lib/config/config.default.yml
index a0def71f7..08df3acb4 100644
--- a/lib/config/config.default.yml
+++ b/lib/config/config.default.yml
@@ -869,13 +869,29 @@ Clusters:
# (azure) Instance configuration.
CloudEnvironment: AzurePublicCloud
- ResourceGroup: ""
Location: centralus
+
+ # (azure) The resource group where the VM and virtual NIC will be
+ # created.
+ ResourceGroup: ""
+
+ # (azure) The resource group of the Network to use for the virtual
+ # NIC (if different from ResourceGroup)
+ NetworkResourceGroup: ""
Network: ""
Subnet: ""
+
+ # (azure) Where to store the VM VHD blobs
StorageAccount: ""
BlobContainer: ""
+
+ # (azure) How long to wait before deleting VHD and NIC
+ # objects that are no longer being used.
DeleteDanglingResourcesAfter: 20s
+
+ # Account (that already exists in the VM image) that will be
+ # set up with an ssh authorized key to allow the compute
+ # dispatcher to connect.
AdminUsername: arvados
InstanceTypes:
diff --git a/lib/config/generated_config.go b/lib/config/generated_config.go
index 7b7c3c20d..c99a23b58 100644
--- a/lib/config/generated_config.go
+++ b/lib/config/generated_config.go
@@ -875,13 +875,29 @@ Clusters:
# (azure) Instance configuration.
CloudEnvironment: AzurePublicCloud
- ResourceGroup: ""
Location: centralus
+
+ # (azure) The resource group where the VM and virtual NIC will be
+ # created.
+ ResourceGroup: ""
+
+ # (azure) The resource group of the Network to use for the virtual
+ # NIC (if different from ResourceGroup)
+ NetworkResourceGroup: ""
Network: ""
Subnet: ""
+
+ # (azure) Where to store the VM VHD blobs
StorageAccount: ""
BlobContainer: ""
+
+ # (azure) How long to wait before deleting VHD and NIC
+ # objects that are no longer being used.
DeleteDanglingResourcesAfter: 20s
+
+ # Account (that already exists in the VM image) that will be
+ # set up with an ssh authorized key to allow the compute
+ # dispatcher to connect.
AdminUsername: arvados
InstanceTypes:
commit 6d644fd3475576fe3648315c62b5003365add9da
Author: Ward Vandewege <ward at curii.com>
Date: Wed Aug 12 10:19:07 2020 -0400
documentation: disable download buffering in our sample nginx config for
keep-web and keepproxy. This helps when the clients are far away/slow.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/install/install-keep-web.html.textile.liquid b/doc/install/install-keep-web.html.textile.liquid
index 899f2dec6..e0456da04 100644
--- a/doc/install/install-keep-web.html.textile.liquid
+++ b/doc/install/install-keep-web.html.textile.liquid
@@ -142,6 +142,7 @@ server {
client_max_body_size 0;
proxy_http_version 1.1;
proxy_request_buffering off;
+ proxy_max_temp_file_size 0;
}
}
</pre></notextile>
diff --git a/doc/install/install-keepproxy.html.textile.liquid b/doc/install/install-keepproxy.html.textile.liquid
index ae6bd3989..b4edd4f57 100644
--- a/doc/install/install-keepproxy.html.textile.liquid
+++ b/doc/install/install-keepproxy.html.textile.liquid
@@ -66,6 +66,7 @@ server {
proxy_set_header X-Real-IP $remote_addr;
proxy_http_version 1.1;
proxy_request_buffering off;
+ proxy_max_temp_file_size 0;
ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
commit d3ee03dbd2677264ab336710ccae074f2f5388d5
Author: Ward Vandewege <ward at curii.com>
Date: Mon Aug 10 13:53:44 2020 -0400
The arvados k8s repo has changed name.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/install/arvados-on-kubernetes-GKE.html.textile.liquid b/doc/install/arvados-on-kubernetes-GKE.html.textile.liquid
index c4236b1c2..0801b7d4e 100644
--- a/doc/install/arvados-on-kubernetes-GKE.html.textile.liquid
+++ b/doc/install/arvados-on-kubernetes-GKE.html.textile.liquid
@@ -75,11 +75,11 @@ There should be no errors. The command will return nothing.
h2(#git). Clone the repository
-Clone the repository and nagivate to the @arvados-kubernetes/charts/arvados@ directory:
+Clone the repository and nagivate to the @arvados-k8s/charts/arvados@ directory:
<pre>
-$ git clone https://github.com/arvados/arvados-kubernetes.git
-$ cd arvados-kubernetes/charts/arvados
+$ git clone https://github.com/arvados/arvados-k8s.git
+$ cd arvados-k8s/charts/arvados
</pre>
h2(#Start). Start the Arvados cluster
diff --git a/doc/install/arvados-on-kubernetes-minikube.html.textile.liquid b/doc/install/arvados-on-kubernetes-minikube.html.textile.liquid
index 56a51b035..86aaf08f9 100644
--- a/doc/install/arvados-on-kubernetes-minikube.html.textile.liquid
+++ b/doc/install/arvados-on-kubernetes-minikube.html.textile.liquid
@@ -47,11 +47,11 @@ There should be no errors. The command will return nothing.
h2(#git). Clone the repository
-Clone the repository and nagivate to the @arvados-kubernetes/charts/arvados@ directory:
+Clone the repository and nagivate to the @arvados-k8s/charts/arvados@ directory:
<pre>
-$ git clone https://github.com/arvados/arvados-kubernetes.git
-$ cd arvados-kubernetes/charts/arvados
+$ git clone https://github.com/arvados/arvados-k8s.git
+$ cd arvados-k8s/charts/arvados
</pre>
h2(#Start). Start the Arvados cluster
commit 01649f280c2bd4631358c0a7284194d9d6fa4a97
Author: Ward Vandewege <ward at curii.com>
Date: Wed Jul 29 17:36:30 2020 -0400
documentation: mention API/MaxKeepBlobBuffers on the Keepstore
installation page.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/install/install-keepstore.html.textile.liquid b/doc/install/install-keepstore.html.textile.liquid
index 869ca15d9..3cb922642 100644
--- a/doc/install/install-keepstore.html.textile.liquid
+++ b/doc/install/install-keepstore.html.textile.liquid
@@ -52,6 +52,8 @@ Fill in the @Volumes@ section of @config.yml@ for each storage volume. Availabl
* If you are using S3-compatible object storage (including Amazon S3, Google Cloud Storage, and Ceph RADOS), follow the setup instructions on "S3 Object Storage":configure-s3-object-storage.html
* If you are using Azure Blob Storage, follow the setup instructions on "Azure Blob Storage":configure-azure-blob-storage.html
+There are a number of general configuration parameters for Keepstore. They are described in the "configuration reference":{{site.baseurl}}/admin/config.html. In particular, you probably want to change @API/MaxKeepBlobBuffers@ to align Keepstore's memory usage with the available memory on the machine that hosts it.
+
h3. List services
Add each keepstore server to the @Services.Keepstore@ section of @/etc/arvados/config.yml@ .
commit 52232876afcc99efe624af2959e6519f4c10472a
Author: Ward Vandewege <ward at curii.com>
Date: Fri Jul 24 17:13:49 2020 -0400
documentation: arvados-dispatch-cloud: add missing curl command.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/install/install-dispatch-cloud.html.textile.liquid b/doc/install/install-dispatch-cloud.html.textile.liquid
index 7bff6a4a2..860b23159 100644
--- a/doc/install/install-dispatch-cloud.html.textile.liquid
+++ b/doc/install/install-dispatch-cloud.html.textile.liquid
@@ -248,13 +248,46 @@ Submit a simple container request:
This command should return a record with a @container_uuid@ field. Once @arvados-dispatch-cloud@ polls the API server for new containers to run, you should see it dispatch that same container.
-The @arvados-dispatch-cloud@ API a list of queued and running jobs. For example:
+The @arvados-dispatch-cloud@ API provides a list of queued and running jobs and cloud instances. Use your @ManagementToken@ to test the dispatcher's endpoint. For example, when one container is running:
<notextile>
-<pre><code>~$ <span class="userinput">curl ...</span>
+<pre><code>~$ <span class="userinput">curl -sH "Authorization: Bearer $token" http://localhost:9006/arvados/v1/dispatch/containers</span>
+{
+ "items": [
+ {
+ "container": {
+ "uuid": "zzzzz-dz642-hdp2vpu9nq14tx0",
+ ...
+ "state": "Running",
+ "scheduling_parameters": {
+ "partitions": null,
+ "preemptible": false,
+ "max_run_time": 0
+ },
+ "exit_code": 0,
+ "runtime_status": null,
+ "started_at": null,
+ "finished_at": null
+ },
+ "instance_type": {
+ "Name": "Standard_D2s_v3",
+ "ProviderType": "Standard_D2s_v3",
+ "VCPUs": 2,
+ "RAM": 8589934592,
+ "Scratch": 16000000000,
+ "IncludedScratch": 16000000000,
+ "AddedScratch": 0,
+ "Price": 0.11,
+ "Preemptible": false
+ }
+ }
+ ]
+}
</code></pre>
</notextile>
+A similar request can be made to the @http://localhost:9006/arvados/v1/dispatch/instances@ endpoint.
+
When the container finishes, the dispatcher will log it.
After the container finishes, you can get the container record by UUID *from a shell server* to see its results:
commit 38a0c90de3d292eeb2f74a0d1c397798d007ada2
Author: Ward Vandewege <ward at curii.com>
Date: Tue Jul 7 09:27:43 2020 -0400
Documentation: fix nginx snippets, remove warnings.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/install/install-api-server.html.textile.liquid b/doc/install/install-api-server.html.textile.liquid
index e64c38266..b8442eb06 100644
--- a/doc/install/install-api-server.html.textile.liquid
+++ b/doc/install/install-api-server.html.textile.liquid
@@ -142,10 +142,9 @@ server {
# This configures the public https port that clients will actually connect to,
# the request is reverse proxied to the upstream 'controller'
- listen *:443 ssl;
- server_name <span class="userinput">xxxxx.example.com</span>;
+ listen 443 ssl;
+ server_name <span class="userinput">ClusterID.example.com</span>;
- ssl on;
ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
diff --git a/doc/install/install-arv-git-httpd.html.textile.liquid b/doc/install/install-arv-git-httpd.html.textile.liquid
index d501f46b7..3d70fc4de 100644
--- a/doc/install/install-arv-git-httpd.html.textile.liquid
+++ b/doc/install/install-arv-git-httpd.html.textile.liquid
@@ -224,12 +224,11 @@ Use a text editor to create a new file @/etc/nginx/conf.d/arvados-git.conf@ with
server 127.0.0.1:<span class="userinput">9001</span>;
}
server {
- listen *:443 ssl;
+ listen 443 ssl;
server_name git.<span class="userinput">ClusterID.example.com</span>;
proxy_connect_timeout 90s;
proxy_read_timeout 300s;
- ssl on;
ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
diff --git a/doc/install/install-keep-web.html.textile.liquid b/doc/install/install-keep-web.html.textile.liquid
index 1ac387b64..899f2dec6 100644
--- a/doc/install/install-keep-web.html.textile.liquid
+++ b/doc/install/install-keep-web.html.textile.liquid
@@ -121,7 +121,7 @@ upstream keep-web {
}
server {
- listen *:443 ssl;
+ listen 443 ssl;
server_name <span class="userinput">download.ClusterID.example.com</span>
<span class="userinput">collections.ClusterID.example.com</span>
<span class="userinput">*.collections.ClusterID.example.com</span>
diff --git a/doc/install/install-keepproxy.html.textile.liquid b/doc/install/install-keepproxy.html.textile.liquid
index 0839c0e52..ae6bd3989 100644
--- a/doc/install/install-keepproxy.html.textile.liquid
+++ b/doc/install/install-keepproxy.html.textile.liquid
@@ -58,7 +58,7 @@ Use a text editor to create a new file @/etc/nginx/conf.d/keepproxy.conf@ with t
}
server {
- listen *:443 ssl;
+ listen 443 ssl;
server_name <span class="userinput">keep.ClusterID.example.com</span>;
proxy_connect_timeout 90s;
@@ -67,7 +67,6 @@ server {
proxy_http_version 1.1;
proxy_request_buffering off;
- ssl on;
ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
diff --git a/doc/install/install-workbench-app.html.textile.liquid b/doc/install/install-workbench-app.html.textile.liquid
index 3d391724d..7ee8db92f 100644
--- a/doc/install/install-workbench-app.html.textile.liquid
+++ b/doc/install/install-workbench-app.html.textile.liquid
@@ -62,10 +62,9 @@ Use a text editor to create a new file @/etc/nginx/conf.d/arvados-workbench.conf
}
server {
- listen *:443 ssl;
+ listen 443 ssl;
server_name workbench.<span class="userinput">ClusterID.example.com</span>;
- ssl on;
ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
diff --git a/doc/install/install-workbench2-app.html.textile.liquid b/doc/install/install-workbench2-app.html.textile.liquid
index b59799c43..f3a320b10 100644
--- a/doc/install/install-workbench2-app.html.textile.liquid
+++ b/doc/install/install-workbench2-app.html.textile.liquid
@@ -47,10 +47,9 @@ Use a text editor to create a new file @/etc/nginx/conf.d/arvados-workbench2.con
}
server {
- listen *:443 ssl;
+ listen 443 ssl;
server_name workbench2.<span class="userinput">ClusterID.example.com</span>;
- ssl on;
ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
diff --git a/doc/install/install-ws.html.textile.liquid b/doc/install/install-ws.html.textile.liquid
index e7b20f45a..9b2fce162 100644
--- a/doc/install/install-ws.html.textile.liquid
+++ b/doc/install/install-ws.html.textile.liquid
@@ -43,7 +43,7 @@ upstream arvados-ws {
}
server {
- listen *:443 ssl;
+ listen 443 ssl;
server_name ws.<span class="userinput">ClusterID.example.com</span>;
proxy_connect_timeout 90s;
commit 5b589eec74ef5baaf451afa239ce986e54ef5496
Author: Ward Vandewege <ward at curii.com>
Date: Thu Jun 25 12:04:20 2020 -0400
6649: Document webshell installation.
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/_config.yml b/doc/_config.yml
index d053da122..b05737f8c 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -216,6 +216,7 @@ navbar:
- install/install-ws.html.textile.liquid
- install/install-arv-git-httpd.html.textile.liquid
- install/install-shell-server.html.textile.liquid
+ - install/install-webshell.html.textile.liquid
- Containers API:
- install/crunch2-slurm/install-compute-node.html.textile.liquid
- install/install-jobs-image.html.textile.liquid
diff --git a/doc/install/install-webshell.html.textile.liquid b/doc/install/install-webshell.html.textile.liquid
new file mode 100644
index 000000000..4040fcf54
--- /dev/null
+++ b/doc/install/install-webshell.html.textile.liquid
@@ -0,0 +1,184 @@
+---
+layout: default
+navsection: installguide
+title: Configure webshell
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+# "Introduction":#introduction
+# "Prerequisites":#prerequisites
+# "Update config.yml":#configure
+# "Update nginx configuration":#update-nginx
+# "Install packages":#install-packages
+# "Configure shellinabox":#config-shellinabox
+# "Configure pam":#config-pam
+# "Confirm working installation":#confirm-working
+
+h2(#introduction). Introduction
+
+Arvados supports @webshell@, which allows ssh access to shell nodes via the browser. This functionality is integrated in @Workbench at .
+
+ at Webshell@ is provided by the @shellinabox@ package which runs on each shell node for which webshell is enabled. For authentication, a supported @pam library@ that allows authentication against Arvados is also required. One Nginx (or similar web server) virtualhost is also needed to expose all the @shellinabox@ instances via https.
+
+h2(#prerequisites). Prerequisites
+
+# "Install workbench":{{site.baseurl}}/install/install-workbench-app.html
+# "Set up a shell node":{{site.baseurl}}/install/install-shell-server.html
+
+h2(#configure). Update config.yml
+
+Edit the cluster config at @config.yml@ and set @Services.WebShell.ExternalURL at . Replace @zzzzz@ with your cluster id. Workbench will use this information to activate its support for webshell.
+
+<notextile>
+<pre><code> Services:
+ WebShell:
+ InternalURLs: {}
+ ExternalURL: <span class="userinput">https://webshell.ClusterID.example.com/</span>
+</span></code></pre>
+</notextile>
+
+h2(#update-nginx). Update Nginx configuration
+
+The arvados-webshell service will be accessible from anywhere on the internet, so we recommend using SSL for transport encryption. This Nginx virtualhost could live on your Workbench server, or any other server that is reachable by your Workbench users and can access the @shell-in-a-box@ service on the shell node(s) on port 4200.
+
+Use a text editor to create a new file @/etc/nginx/conf.d/arvados-webshell.conf@ with the following configuration. Options that need attention are marked in <span class="userinput">red</span>.
+
+<notextile><pre>
+upstream arvados-webshell {
+ server <span class="userinput">shell.ClusterID.example.com</span>:<span class="userinput">4200</span>;
+}
+
+server {
+ listen 443 ssl;
+ server_name webshell.<span class="userinput">ClusterID.example.com</span>;
+
+ proxy_connect_timeout 90s;
+ proxy_read_timeout 300s;
+
+ ssl on;
+ ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
+ ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
+
+ location /<span class="userinput">shell.ClusterID</span> {
+ if ($request_method = 'OPTIONS') {
+ add_header 'Access-Control-Allow-Origin' '*';
+ add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
+ add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
+ add_header 'Access-Control-Max-Age' 1728000;
+ add_header 'Content-Type' 'text/plain charset=UTF-8';
+ add_header 'Content-Length' 0;
+ return 204;
+ }
+ if ($request_method = 'POST') {
+ add_header 'Access-Control-Allow-Origin' '*';
+ add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
+ add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
+ }
+ if ($request_method = 'GET') {
+ add_header 'Access-Control-Allow-Origin' '*';
+ add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
+ add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
+ }
+
+ proxy_ssl_session_reuse off;
+ proxy_read_timeout 90;
+ proxy_set_header X-Forwarded-Proto https;
+ proxy_set_header Host $http_host;
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ proxy_pass http://arvados-webshell;
+ }
+}
+</pre></notextile>
+
+Note that the location line in the nginx config matches your shell node hostname *without domain*, because that is how the shell node was defined in the "Set up a shell node":{{site.baseurl}}/install/install-shell-server.html#vm-record instructions. It makes for a more user friendly experience in Workbench.
+
+For additional shell nodes with @shell-in-a-box@, add @location@ and @upstream@ sections as needed.
+
+{% assign arvados_component = 'shellinabox libpam-arvados' %}
+
+{% include 'install_packages' %}
+
+h2(#config-shellinabox). Configure shellinabox
+
+h3. Red Hat and Centos
+
+Edit @/etc/sysconfig/shellinaboxd@:
+
+<notextile><pre>
+# TCP port that shellinboxd's webserver listens on
+PORT=4200
+
+# SSL is disabled because it is terminated in Nginx. Adjust as needed.
+OPTS="--disable-ssl --no-beep --service=/<span class="userinput">shell.ClusterID.example.com</span>:AUTH:HOME:SHELL"
+</pre></notextile>
+
+<notextile>
+<pre>
+<code># <span class="userinput">systemctl enable shellinabox</span></code>
+<code># <span class="userinput">systemctl start shellinabox</span></code>
+</pre>
+</notextile>
+
+h3. Debian and Ubuntu
+
+Edit @/etc/default/shellinabox@:
+
+<notextile><pre>
+# TCP port that shellinboxd's webserver listens on
+SHELLINABOX_PORT=4200
+
+# SSL is disabled because it is terminated in Nginx. Adjust as needed.
+SHELLINABOX_ARGS="--disable-ssl --no-beep --service=/<span class="userinput">shell.ClusterID.example.com</span>:AUTH:HOME:SHELL"
+</pre></notextile>
+
+<notextile>
+<pre>
+<code># <span class="userinput">systemctl enable shellinabox</span></code>
+<code># <span class="userinput">systemctl start shellinabox</span></code>
+</pre>
+</notextile>
+
+
+h2(#config-pam). Configure pam
+
+Use a text editor to create a new file @/etc/pam.d/shellinabox@ with the following configuration. Options that need attention are marked in <span class="userinput">red</span>.
+
+<notextile><pre>
+# This example is a stock debian "login" file with libpam_arvados
+# replacing pam_unix, and the "noprompt" option in use. It can be
+# installed as /etc/pam.d/shellinabox .
+
+auth optional pam_faildelay.so delay=3000000
+auth [success=ok new_authtok_reqd=ok ignore=ignore user_unknown=bad default=die] pam_securetty.so
+auth requisite pam_nologin.so
+session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
+session required pam_env.so readenv=1
+session required pam_env.so readenv=1 envfile=/etc/default/locale
+
+auth [success=1 default=ignore] pam_python.so /usr/lib/security/libpam_arvados.py <span class="userinput">ClusterID.example.com</span> <span class="userinput">shell.ClusterID.example.com</span> noprompt
+auth requisite pam_deny.so
+auth required pam_permit.so
+
+auth optional pam_group.so
+session required pam_limits.so
+session optional pam_lastlog.so
+session optional pam_motd.so motd=/run/motd.dynamic
+session optional pam_motd.so
+session optional pam_mail.so standard
+
+ at include common-account
+ at include common-session
+ at include common-password
+
+session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so open
+</pre></notextile>
+
+h2(#confirm-working). Confirm working installation
+
+A user should be able to log in to the shell server, using webshell via workbench. Please refer to "Accessing an Arvados VM with Webshell":{{site.baseurl}}/user/getting_started/vm-login-with-webshell.html
+
commit 65e1b1ec2815c3e23bd3705a29c9832edbd07cb1
Author: Ward Vandewege <ward at curii.com>
Date: Thu Jun 25 20:38:04 2020 -0400
Documentation: add missing span in the 'Set up a shell node' page, to
fix coloration bug in the 'Create scoped token' section.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/install/install-shell-server.html.textile.liquid b/doc/install/install-shell-server.html.textile.liquid
index f57d0690c..d30ed8587 100644
--- a/doc/install/install-shell-server.html.textile.liquid
+++ b/doc/install/install-shell-server.html.textile.liquid
@@ -69,7 +69,7 @@ As an admin arvados user (such as the system root user), create a token that is
<notextile>
<pre>
-<code>apiserver:~$ <span class="userinput">arv api_client_authorization create --api-client-authorization '{"scopes":["GET /arvados/v1/virtual_machines/<b>zzzzz-2x53u-zzzzzzzzzzzzzzz</b>/logins"]}'
+<code>apiserver:~$ <span class="userinput">arv api_client_authorization create --api-client-authorization '{"scopes":["GET /arvados/v1/virtual_machines/<b>zzzzz-2x53u-zzzzzzzzzzzzzzz</b>/logins"]}'</span>
{
...
"api_token":"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz",
commit d52ff9609b07af06d63bb58839fdca1df7c59105
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Tue Jun 23 14:09:50 2020 -0300
Fixes template usage on SSO install page.
No issue #
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/doc/install/install-sso.html.textile.liquid b/doc/install/install-sso.html.textile.liquid
index 4d91b18c0..cb6470066 100644
--- a/doc/install/install-sso.html.textile.liquid
+++ b/doc/install/install-sso.html.textile.liquid
@@ -116,6 +116,8 @@ There are two configuration options for local accounts:
For more information about configuring backend support for sending email (required to send email confirmations) see "Configuring Action Mailer":http://guides.rubyonrails.org/configuring.html#configuring-action-mailer
+{% assign railshost = "" %}
+{% assign railsdir = "/var/www/arvados-sso/current" %}
If @allow_account_registration@ is false, you may manually create local accounts on the SSO server from the Rails console. {% include 'install_rails_command' %}
Enter the following commands at the console.
commit 9d898d5e08d7326ea2c26607a0e39ee6630befe6
Author: Ward Vandewege <ward at curii.com>
Date: Mon Jun 22 12:08:24 2020 -0400
Make it clear that the API token is used by all Arvados command line
tools, not just the 'arv' tool.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/user/reference/api-tokens.html.textile.liquid b/doc/user/reference/api-tokens.html.textile.liquid
index d5172f0c5..6afc20bf4 100644
--- a/doc/user/reference/api-tokens.html.textile.liquid
+++ b/doc/user/reference/api-tokens.html.textile.liquid
@@ -9,7 +9,7 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-The Arvados API token is a secret key that enables the @arv@ command line client to access Arvados with the proper permissions.
+The Arvados API token is a secret key that enables the Arvados command line tools to authenticate themselves.
Access the Arvados Workbench using this link: "{{site.arvados_workbench_host}}/":{{site.arvados_workbench_host}}/ (Replace the hostname portion with the hostname of your local Arvados instance if necessary.)
commit f56512e09e1579ed822ab32efd4f2bba89d9e1ed
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Jun 19 17:20:59 2020 -0400
Add links to the Arvados forum refs #16521
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/index.html.liquid b/doc/index.html.liquid
index f81cd5901..c7fd46eb2 100644
--- a/doc/index.html.liquid
+++ b/doc/index.html.liquid
@@ -34,7 +34,8 @@ SPDX-License-Identifier: CC-BY-SA-3.0
<a name="Support"></a>
<p><strong>Support and Community</strong></p>
- <p>The <a href="https://gitter.im/arvados/community">arvados community channel</a> at gitter.im is available for live discussion and community support. There is also a <a href="http://lists.arvados.org/mailman/listinfo/arvados">mailing list</a>.
+ <p>Interact with the Arvados community on the <a href="https://forum.arvados.org">Arvados Forum</a>
+ and the <a href="https://gitter.im/arvados/community">arvados/community</a> channel at gitter.im.
</p>
<p>Curii Corporation provides managed Arvados installations as well as commercial support for Arvados. Please contact <a href="mailto:info at curii.com">info at curii.com</a> for more information.</p>
diff --git a/doc/user/getting_started/community.html.textile.liquid b/doc/user/getting_started/community.html.textile.liquid
index b85b55612..6dc0b4d81 100644
--- a/doc/user/getting_started/community.html.textile.liquid
+++ b/doc/user/getting_started/community.html.textile.liquid
@@ -11,15 +11,19 @@ SPDX-License-Identifier: CC-BY-SA-3.0
h2. On the web
-The Arvados Free Sofware project page is located at "https://arvados.org":https://arvados.org . The "Arvados Wiki":https://dev.arvados.org/projects/arvados/wiki is a collaborative site for documenting Arvados and provides an overview of the Arvados Platform and Components. The "Arvados blog":https://dev.arvados.org/projects/arvados/blogs posts articles of interest about Arvados.
+The Arvados Free Sofware project page is located at "https://arvados.org":https://arvados.org . The "Arvados Wiki":https://dev.arvados.org/projects/arvados/wiki is a collaborative site for documenting Arvados and provides an overview of the Arvados Platform and Components.
-h2. Mailing lists
+h2. Forum
-The "Arvados user mailing list":http://lists.arvados.org/mailman/listinfo/arvados is a forum for general discussion, questions, and news about Arvados development. The "Arvados developer mailing list":http://lists.arvados.org/mailman/listinfo/arvados-dev is a forum for more technical discussion, intended for developers and contributers to Arvados.
+The "Arvados Forum":https://forum.arvados.org has topic-based discussion, Q&A and community support.
h2. Chat
-The "arvados community channel":https://gitter.im/arvados/community channel at "gitter.im":https://gitter.im is available for live discussion and support.
+The "arvados/community":https://gitter.im/arvados/community channel at "gitter.im":https://gitter.im is available for live discussion and support.
+
+h2. Mailing list
+
+The "Arvados user mailing list":http://lists.arvados.org/mailman/listinfo/arvados is a low-volume list used mainly to announce new releases of Arvados.
h2. Bug tracking
commit 19a22a743e63cf54dfb104e19123f2aa8a5bcbd6
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Jun 12 10:13:24 2020 -0400
16524: Pin rsa package to < 4.1
Because >= 4.1 does not support python 2 any more. refs #16524
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/python/setup.py b/sdk/python/setup.py
index ff68e2a7f..4adadc353 100644
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -54,6 +54,7 @@ setup(name='arvados-python-client',
'ruamel.yaml >=0.15.54, <=0.15.77',
'setuptools',
'ws4py >=0.4.2',
+ 'rsa < 4.1'
],
extras_require={
':os.name=="posix" and python_version<"3"': ['subprocess32 >= 3.5.1'],
commit db83f805e4ccf82eef22fe2c9a3db38ebd772a83
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed May 13 17:07:11 2020 -0400
16391: Fix log message.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/crunch-dispatch-slurm/crunch-dispatch-slurm.go b/services/crunch-dispatch-slurm/crunch-dispatch-slurm.go
index 36bcef4f2..4115482d8 100644
--- a/services/crunch-dispatch-slurm/crunch-dispatch-slurm.go
+++ b/services/crunch-dispatch-slurm/crunch-dispatch-slurm.go
@@ -280,7 +280,8 @@ func (disp *Dispatcher) runContainer(_ *dispatch.Dispatcher, ctr arvados.Contain
cmd = append(cmd, disp.cluster.Containers.CrunchRunArgumentsList...)
if err := disp.submit(ctr, cmd); err != nil {
var text string
- if err, ok := err.(dispatchcloud.ConstraintsNotSatisfiableError); ok {
+ switch err := err.(type) {
+ case dispatchcloud.ConstraintsNotSatisfiableError:
var logBuf bytes.Buffer
fmt.Fprintf(&logBuf, "cannot run container %s: %s\n", ctr.UUID, err)
if len(err.AvailableTypes) == 0 {
@@ -296,7 +297,7 @@ func (disp *Dispatcher) runContainer(_ *dispatch.Dispatcher, ctr arvados.Contain
}
text = logBuf.String()
disp.UpdateState(ctr.UUID, dispatch.Cancelled)
- } else {
+ default:
text = fmt.Sprintf("Error submitting container %s to slurm: %s", ctr.UUID, err)
}
log.Print(text)
commit 3c0a5f77f2bea2206aa3fe9d5ffa8245c85c9cbc
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Fri Jun 5 15:13:44 2020 -0300
16492: Upgrades websocket-extensions gem to fix a security alert.
We really don't use this gem as it's requested by actioncable from Rails 5
that we don't even load at runtime. But, as we cannot remove actioncable
from our Gemfile.lock files because of some other gems depending on 'rails',
we upgrade it to make GitHub happy.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/apps/workbench/Gemfile.lock b/apps/workbench/Gemfile.lock
index e722fa241..2420fee24 100644
--- a/apps/workbench/Gemfile.lock
+++ b/apps/workbench/Gemfile.lock
@@ -315,7 +315,7 @@ GEM
json (>= 1.8.0)
websocket-driver (0.6.5)
websocket-extensions (>= 0.1.0)
- websocket-extensions (0.1.3)
+ websocket-extensions (0.1.5)
xpath (2.1.0)
nokogiri (~> 1.3)
diff --git a/services/api/Gemfile.lock b/services/api/Gemfile.lock
index 9f3a5fb2b..c8a1a27b7 100644
--- a/services/api/Gemfile.lock
+++ b/services/api/Gemfile.lock
@@ -273,7 +273,7 @@ GEM
json (>= 1.8.0)
websocket-driver (0.6.5)
websocket-extensions (>= 0.1.0)
- websocket-extensions (0.1.3)
+ websocket-extensions (0.1.5)
PLATFORMS
ruby
commit 49e0afcce46dcba1e1402f2524b4b88f6522d216
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon May 11 11:08:01 2020 -0400
16339: Fix $schemas rewriting bug
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cwl/arvados_cwl/runner.py b/sdk/cwl/arvados_cwl/runner.py
index 2239e0f9d..4c2e6697d 100644
--- a/sdk/cwl/arvados_cwl/runner.py
+++ b/sdk/cwl/arvados_cwl/runner.py
@@ -42,6 +42,7 @@ import schema_salad.validate as validate
import arvados.collection
from .util import collectionUUID
import ruamel.yaml as yaml
+from ruamel.yaml.comments import CommentedMap, CommentedSeq
import arvados_cwl.arvdocker
from .pathmapper import ArvPathMapper, trim_listing, collection_pdh_pattern, collection_uuid_pattern
@@ -392,7 +393,7 @@ def upload_dependencies(arvrunner, name, document_loader,
discovered_secondaryfiles[mapper.mapper(d).resolved] = discovered[d]
if "$schemas" in workflowobj:
- sch = []
+ sch = CommentedSeq()
for s in workflowobj["$schemas"]:
sch.append(mapper.mapper(s).resolved)
workflowobj["$schemas"] = sch
commit 5a39bdfd6afdd0f0423adf76d26eeba7f3daf0cc
Author: Ward Vandewege <ward at jhvc.com>
Date: Mon Apr 13 17:57:03 2020 -0400
16328: If config.yml is available, use the keepstores defined there instead of
the legacy autodiscover mechanism via the API server.
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/services/keepproxy/keepproxy.go b/services/keepproxy/keepproxy.go
index 79aa992aa..188173274 100644
--- a/services/keepproxy/keepproxy.go
+++ b/services/keepproxy/keepproxy.go
@@ -116,6 +116,12 @@ func run(logger log.FieldLogger, cluster *arvados.Cluster) error {
return fmt.Errorf("Error setting up arvados client %v", err)
}
+ // If a config file is available, use the keepstores defined there
+ // instead of the legacy autodiscover mechanism via the API server
+ for k := range cluster.Services.Keepstore.InternalURLs {
+ arv.KeepServiceURIs = append(arv.KeepServiceURIs, k.String())
+ }
+
if cluster.SystemLogs.LogLevel == "debug" {
keepclient.DebugPrintf = log.Printf
}
diff --git a/services/keepproxy/keepproxy_test.go b/services/keepproxy/keepproxy_test.go
index aa3235680..94ed05bff 100644
--- a/services/keepproxy/keepproxy_test.go
+++ b/services/keepproxy/keepproxy_test.go
@@ -40,6 +40,12 @@ var _ = Suite(&ServerRequiredSuite{})
// Tests that require the Keep server running
type ServerRequiredSuite struct{}
+// Gocheck boilerplate
+var _ = Suite(&ServerRequiredConfigYmlSuite{})
+
+// Tests that require the Keep servers running as defined in config.yml
+type ServerRequiredConfigYmlSuite struct{}
+
// Gocheck boilerplate
var _ = Suite(&NoKeepServerSuite{})
@@ -83,6 +89,21 @@ func (s *ServerRequiredSuite) TearDownSuite(c *C) {
arvadostest.StopAPI()
}
+func (s *ServerRequiredConfigYmlSuite) SetUpSuite(c *C) {
+ arvadostest.StartAPI()
+ // config.yml defines 4 keepstores
+ arvadostest.StartKeep(4, false)
+}
+
+func (s *ServerRequiredConfigYmlSuite) SetUpTest(c *C) {
+ arvadostest.ResetEnv()
+}
+
+func (s *ServerRequiredConfigYmlSuite) TearDownSuite(c *C) {
+ arvadostest.StopKeep(4)
+ arvadostest.StopAPI()
+}
+
func (s *NoKeepServerSuite) SetUpSuite(c *C) {
arvadostest.StartAPI()
// We need API to have some keep services listed, but the
@@ -99,12 +120,17 @@ func (s *NoKeepServerSuite) TearDownSuite(c *C) {
arvadostest.StopAPI()
}
-func runProxy(c *C, bogusClientToken bool) *keepclient.KeepClient {
+func runProxy(c *C, bogusClientToken bool, loadKeepstoresFromConfig bool) *keepclient.KeepClient {
cfg, err := config.NewLoader(nil, ctxlog.TestLogger(c)).Load()
c.Assert(err, Equals, nil)
cluster, err := cfg.GetCluster("")
c.Assert(err, Equals, nil)
+ if !loadKeepstoresFromConfig {
+ // Do not load Keepstore InternalURLs from the config file
+ cluster.Services.Keepstore.InternalURLs = make(map[arvados.URL]arvados.ServiceInstance)
+ }
+
cluster.Services.Keepproxy.InternalURLs = map[arvados.URL]arvados.ServiceInstance{arvados.URL{Host: ":0"}: arvados.ServiceInstance{}}
listener = nil
@@ -131,7 +157,7 @@ func runProxy(c *C, bogusClientToken bool) *keepclient.KeepClient {
}
func (s *ServerRequiredSuite) TestResponseViaHeader(c *C) {
- runProxy(c, false)
+ runProxy(c, false, false)
defer closeListener()
req, err := http.NewRequest("POST",
@@ -158,7 +184,7 @@ func (s *ServerRequiredSuite) TestResponseViaHeader(c *C) {
}
func (s *ServerRequiredSuite) TestLoopDetection(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
sr := map[string]string{
@@ -176,7 +202,7 @@ func (s *ServerRequiredSuite) TestLoopDetection(c *C) {
}
func (s *ServerRequiredSuite) TestStorageClassesHeader(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
// Set up fake keepstore to record request headers
@@ -203,7 +229,7 @@ func (s *ServerRequiredSuite) TestStorageClassesHeader(c *C) {
}
func (s *ServerRequiredSuite) TestDesiredReplicas(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
content := []byte("TestDesiredReplicas")
@@ -220,7 +246,7 @@ func (s *ServerRequiredSuite) TestDesiredReplicas(c *C) {
}
func (s *ServerRequiredSuite) TestPutWrongContentLength(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
content := []byte("TestPutWrongContentLength")
@@ -259,7 +285,7 @@ func (s *ServerRequiredSuite) TestPutWrongContentLength(c *C) {
}
func (s *ServerRequiredSuite) TestManyFailedPuts(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
router.(*proxyHandler).timeout = time.Nanosecond
@@ -286,7 +312,7 @@ func (s *ServerRequiredSuite) TestManyFailedPuts(c *C) {
}
func (s *ServerRequiredSuite) TestPutAskGet(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
hash := fmt.Sprintf("%x", md5.Sum([]byte("foo")))
@@ -363,7 +389,7 @@ func (s *ServerRequiredSuite) TestPutAskGet(c *C) {
}
func (s *ServerRequiredSuite) TestPutAskGetForbidden(c *C) {
- kc := runProxy(c, true)
+ kc := runProxy(c, true, false)
defer closeListener()
hash := fmt.Sprintf("%x+3", md5.Sum([]byte("bar")))
@@ -389,7 +415,7 @@ func (s *ServerRequiredSuite) TestPutAskGetForbidden(c *C) {
}
func (s *ServerRequiredSuite) TestCorsHeaders(c *C) {
- runProxy(c, false)
+ runProxy(c, false, false)
defer closeListener()
{
@@ -420,7 +446,7 @@ func (s *ServerRequiredSuite) TestCorsHeaders(c *C) {
}
func (s *ServerRequiredSuite) TestPostWithoutHash(c *C) {
- runProxy(c, false)
+ runProxy(c, false, false)
defer closeListener()
{
@@ -463,7 +489,22 @@ func (s *ServerRequiredSuite) TestStripHint(c *C) {
// With a valid but non-existing prefix (expect "\n")
// With an invalid prefix (expect error)
func (s *ServerRequiredSuite) TestGetIndex(c *C) {
- kc := runProxy(c, false)
+ getIndexWorker(c, false)
+}
+
+// Test GetIndex
+// Uses config.yml
+// Put one block, with 2 replicas
+// With no prefix (expect the block locator, twice)
+// With an existing prefix (expect the block locator, twice)
+// With a valid but non-existing prefix (expect "\n")
+// With an invalid prefix (expect error)
+func (s *ServerRequiredConfigYmlSuite) TestGetIndex(c *C) {
+ getIndexWorker(c, true)
+}
+
+func getIndexWorker(c *C, useConfig bool) {
+ kc := runProxy(c, false, useConfig)
defer closeListener()
// Put "index-data" blocks
@@ -526,7 +567,7 @@ func (s *ServerRequiredSuite) TestGetIndex(c *C) {
}
func (s *ServerRequiredSuite) TestCollectionSharingToken(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
hash, _, err := kc.PutB([]byte("shareddata"))
c.Check(err, IsNil)
@@ -539,7 +580,7 @@ func (s *ServerRequiredSuite) TestCollectionSharingToken(c *C) {
}
func (s *ServerRequiredSuite) TestPutAskGetInvalidToken(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
// Put a test block
@@ -576,7 +617,7 @@ func (s *ServerRequiredSuite) TestPutAskGetInvalidToken(c *C) {
}
func (s *ServerRequiredSuite) TestAskGetKeepProxyConnectionError(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
// Point keepproxy at a non-existent keepstore
@@ -602,7 +643,7 @@ func (s *ServerRequiredSuite) TestAskGetKeepProxyConnectionError(c *C) {
}
func (s *NoKeepServerSuite) TestAskGetNoKeepServerError(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
hash := fmt.Sprintf("%x", md5.Sum([]byte("foo")))
@@ -625,7 +666,7 @@ func (s *NoKeepServerSuite) TestAskGetNoKeepServerError(c *C) {
}
func (s *ServerRequiredSuite) TestPing(c *C) {
- kc := runProxy(c, false)
+ kc := runProxy(c, false, false)
defer closeListener()
rtr := MakeRESTRouter(kc, 10*time.Second, arvadostest.ManagementToken)
commit 132c3152aac95f67f439bad68aceab9f60ad5aaa
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Tue Apr 21 13:55:10 2020 -0300
16341: Upgrades rack on API & Workbench.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/apps/workbench/Gemfile.lock b/apps/workbench/Gemfile.lock
index 2af9c8b16..e722fa241 100644
--- a/apps/workbench/Gemfile.lock
+++ b/apps/workbench/Gemfile.lock
@@ -214,7 +214,7 @@ GEM
multi_json (~> 1.0)
websocket-driver (>= 0.2.0)
public_suffix (4.0.3)
- rack (2.0.7)
+ rack (2.2.2)
rack-mini-profiler (1.0.2)
rack (>= 1.2.0)
rack-test (0.6.3)
@@ -375,4 +375,4 @@ DEPENDENCIES
uglifier (~> 2.0)
BUNDLED WITH
- 1.11
+ 1.16.6
diff --git a/services/api/Gemfile.lock b/services/api/Gemfile.lock
index 24d5ad5b6..9f3a5fb2b 100644
--- a/services/api/Gemfile.lock
+++ b/services/api/Gemfile.lock
@@ -180,7 +180,7 @@ GEM
pg (1.1.4)
power_assert (1.1.4)
public_suffix (4.0.3)
- rack (2.0.7)
+ rack (2.2.2)
rack-test (0.6.3)
rack (>= 1.0)
rails (5.0.7.2)
@@ -317,4 +317,4 @@ DEPENDENCIES
uglifier (~> 2.0)
BUNDLED WITH
- 1.11
+ 1.16.6
commit 7f9c18ecb14a435bf29b38a3a63450ef88d33784
Author: Dante Tsang <waichuntwc at gmail.com>
Date: Mon Apr 20 18:23:56 2020 +0800
Fixed FetchAll method for Container queue in order to make use of preemptible instance
Updated AUTHORS file
Arvados-DCO-1.1-Signed-off-by: Dante Tsang <dante at dantetsang.com>
diff --git a/AUTHORS b/AUTHORS
index 436a504c3..93bdb37dc 100644
--- a/AUTHORS
+++ b/AUTHORS
@@ -19,3 +19,5 @@ Thomas Mooney <tmooney at genome.wustl.edu>
Chen Chen <aflyhorse at gmail.com>
Veritas Genetics, Inc. <*@veritasgenetics.com>
Curii Corporation, Inc. <*@curii.com>
+Dante Tsang <dante at dantetsang.com>
+Codex Genetics Ltd <info at codexgenetics.com>
\ No newline at end of file
diff --git a/lib/dispatchcloud/container/queue.go b/lib/dispatchcloud/container/queue.go
index d128c265f..45b346383 100644
--- a/lib/dispatchcloud/container/queue.go
+++ b/lib/dispatchcloud/container/queue.go
@@ -382,7 +382,7 @@ func (cq *Queue) poll() (map[string]*arvados.Container, error) {
*next[upd.UUID] = upd
}
}
- selectParam := []string{"uuid", "state", "priority", "runtime_constraints", "container_image", "mounts"}
+ selectParam := []string{"uuid", "state", "priority", "runtime_constraints", "container_image", "mounts", "scheduling_parameters"}
limitParam := 1000
mine, err := cq.fetchAll(arvados.ResourceListParams{
commit b405ab07d342827e2bcb70ab4072d30eded90537
Author: Ward Vandewege <ward at jhvc.com>
Date: Fri May 1 16:55:01 2020 -0400
16393: keepproxy used cluster.API.KeepServiceRequestTimeout (defaults to
15s) as the timeout on its connection to the keepstores. When a slow client
sends blocks to keepproxy, they get streamed through to keepstore, and
the upload can take more than cluster.API.KeepServiceRequestTimeout
seconds. Update keepproxy to use keepclient.DefaultProxyRequestTimeout
(300s) instead when it connects to the keepstores.
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/services/keepproxy/keepproxy.go b/services/keepproxy/keepproxy.go
index 58e4a8534..79aa992aa 100644
--- a/services/keepproxy/keepproxy.go
+++ b/services/keepproxy/keepproxy.go
@@ -157,7 +157,7 @@ func run(logger log.FieldLogger, cluster *arvados.Cluster) error {
signal.Notify(term, syscall.SIGINT)
// Start serving requests.
- router = MakeRESTRouter(kc, time.Duration(cluster.API.KeepServiceRequestTimeout), cluster.SystemRootToken)
+ router = MakeRESTRouter(kc, time.Duration(keepclient.DefaultProxyRequestTimeout), cluster.ManagementToken)
return http.Serve(listener, httpserver.AddRequestIDs(httpserver.LogRequests(router)))
}
commit a788b2a6c38b0a5d5887f44f45ff44d258f1b723
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed Apr 29 15:37:32 2020 -0400
16349: Set time zone to UTC at db connection level.
This covers the trash time comparisons in materialized_permission_view
as well as explicit uses of CURRENT_TIMESTAMP in Rails queries.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/api/app/controllers/application_controller.rb b/services/api/app/controllers/application_controller.rb
index c7d7ad814..83a233cd5 100644
--- a/services/api/app/controllers/application_controller.rb
+++ b/services/api/app/controllers/application_controller.rb
@@ -386,8 +386,8 @@ class ApplicationController < ActionController::Base
@read_auths += ApiClientAuthorization
.includes(:user)
.where('api_token IN (?) AND
- (expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP AT TIME ZONE ?)',
- secrets, 'UTC')
+ (expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP)',
+ secrets)
.to_a
end
@read_auths.select! { |auth| auth.scopes_allow_request? request }
diff --git a/services/api/app/models/api_client_authorization.rb b/services/api/app/models/api_client_authorization.rb
index a4bf0cd5b..503dfc50d 100644
--- a/services/api/app/models/api_client_authorization.rb
+++ b/services/api/app/models/api_client_authorization.rb
@@ -158,7 +158,7 @@ class ApiClientAuthorization < ArvadosModel
# fast path: look up the token in the local database
auth = ApiClientAuthorization.
includes(:user, :api_client).
- where('uuid=? and (expires_at is null or expires_at > CURRENT_TIMESTAMP AT TIME ZONE ?)', token_uuid, 'UTC').
+ where('uuid=? and (expires_at is null or expires_at > CURRENT_TIMESTAMP)', token_uuid).
first
if auth && auth.user &&
(secret == auth.api_token ||
@@ -284,7 +284,7 @@ class ApiClientAuthorization < ArvadosModel
# token is not a 'v2' token
auth = ApiClientAuthorization.
includes(:user, :api_client).
- where('api_token=? and (expires_at is null or expires_at > CURRENT_TIMESTAMP AT TIME ZONE ?)', token, 'UTC').
+ where('api_token=? and (expires_at is null or expires_at > CURRENT_TIMESTAMP)', token).
first
if auth && auth.user
return auth
diff --git a/services/api/config/initializers/time_zone.rb b/services/api/config/initializers/time_zone.rb
new file mode 100644
index 000000000..cedd8f3e4
--- /dev/null
+++ b/services/api/config/initializers/time_zone.rb
@@ -0,0 +1,15 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+ActiveRecord::Base.connection.class.set_callback :checkout, :after do
+ # If the database connection is in a time zone other than UTC,
+ # "timestamp" values don't behave as desired.
+ #
+ # For example, ['select now() > ?', Time.now] returns true in time
+ # zones +0100 and UTC (which makes sense since Time.now is evaluated
+ # before now()), but false in time zone -0100 (now() returns an
+ # earlier clock time, and its time zone is dropped when comparing to
+ # a "timestamp without time zone").
+ raw_connection.sync_exec("SET TIME ZONE 'UTC'")
+end
diff --git a/services/api/lib/create_superuser_token.rb b/services/api/lib/create_superuser_token.rb
index c1530162e..57eac048a 100755
--- a/services/api/lib/create_superuser_token.rb
+++ b/services/api/lib/create_superuser_token.rb
@@ -40,7 +40,7 @@ module CreateSuperUserToken
where(user_id: system_user.id).
where(api_client_id: apiClient.id).
where_serialized(:scopes, ['all']).
- where('(expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP AT TIME ZONE ?)', 'UTC').
+ where('(expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP)').
first
# none exist; create one with the supplied token
diff --git a/services/api/test/unit/time_zone_test.rb b/services/api/test/unit/time_zone_test.rb
new file mode 100644
index 000000000..60ca6b50b
--- /dev/null
+++ b/services/api/test/unit/time_zone_test.rb
@@ -0,0 +1,15 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+require 'test_helper'
+
+class TimeZoneTest < ActiveSupport::TestCase
+ test "Database connection time zone" do
+ # This is pointless if the testing host is already using the UTC
+ # time zone. But if not, the test confirms that
+ # config/initializers/time_zone.rb has successfully changed the
+ # database connection time zone to UTC.
+ assert_equal('UTC', ActiveRecord::Base.connection.select_value("show timezone"))
+ end
+end
commit 04e3809b09008c4009a9bc74120952d753244884
Author: Tom Clegg <tom at tomclegg.ca>
Date: Mon Apr 27 16:57:55 2020 -0400
16349: Skip converting DB timestamps to & from localtime.
The previous code already returned the correct time. This change
merely avoids converting the time to the local time zone in
PostgreSQL, and back to UTC again in Rails.
This might also avoid rare problems in unusual edge cases where Rails
and PostgreSQL session use different time zones, like a configuration
mishap or a race during a daylight savings time transition.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/api/lib/db_current_time.rb b/services/api/lib/db_current_time.rb
index fdb664152..5e1634ecb 100644
--- a/services/api/lib/db_current_time.rb
+++ b/services/api/lib/db_current_time.rb
@@ -3,9 +3,13 @@
# SPDX-License-Identifier: AGPL-3.0
module DbCurrentTime
- CURRENT_TIME_SQL = "SELECT clock_timestamp()"
+ CURRENT_TIME_SQL = "SELECT clock_timestamp() AT TIME ZONE 'UTC'"
def db_current_time
- Time.parse(ActiveRecord::Base.connection.select_value(CURRENT_TIME_SQL)).to_time
+ Time.parse(ActiveRecord::Base.connection.select_value(CURRENT_TIME_SQL) + " +0000")
+ end
+
+ def db_transaction_time
+ Time.parse(ActiveRecord::Base.connection.select_value("SELECT current_timestamp AT TIME ZONE 'UTC'") + " +0000")
end
end
commit 15f61e8bce0d67014f02852a05bc462e09b80435
Author: Tom Clegg <tom at tomclegg.ca>
Date: Mon Apr 27 16:37:49 2020 -0400
16349: Fix TZ-sensitive comparison in token expiry checks.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/lib/controller/federation.go b/lib/controller/federation.go
index c0d127284..ac239fb9b 100644
--- a/lib/controller/federation.go
+++ b/lib/controller/federation.go
@@ -166,7 +166,7 @@ func (h *Handler) validateAPItoken(req *http.Request, token string) (*CurrentUse
}
user.Authorization.APIToken = token
var scopes string
- err = db.QueryRowContext(req.Context(), `SELECT api_client_authorizations.uuid, api_client_authorizations.scopes, users.uuid FROM api_client_authorizations JOIN users on api_client_authorizations.user_id=users.id WHERE api_token=$1 AND (expires_at IS NULL OR expires_at > current_timestamp) LIMIT 1`, token).Scan(&user.Authorization.UUID, &scopes, &user.UUID)
+ err = db.QueryRowContext(req.Context(), `SELECT api_client_authorizations.uuid, api_client_authorizations.scopes, users.uuid FROM api_client_authorizations JOIN users on api_client_authorizations.user_id=users.id WHERE api_token=$1 AND (expires_at IS NULL OR expires_at > current_timestamp AT TIME ZONE 'UTC') LIMIT 1`, token).Scan(&user.Authorization.UUID, &scopes, &user.UUID)
if err == sql.ErrNoRows {
ctxlog.FromContext(req.Context()).Debugf("validateAPItoken(%s): not found in database", token)
return nil, false, nil
@@ -214,9 +214,9 @@ func (h *Handler) createAPItoken(req *http.Request, userUUID string, scopes []st
(uuid, api_token, expires_at, scopes,
user_id,
api_client_id, created_at, updated_at)
-VALUES ($1, $2, CURRENT_TIMESTAMP + INTERVAL '2 weeks', $3,
+VALUES ($1, $2, CURRENT_TIMESTAMP AT TIME ZONE 'UTC' + INTERVAL '2 weeks', $3,
(SELECT id FROM users WHERE users.uuid=$4 LIMIT 1),
-0, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`,
+0, CURRENT_TIMESTAMP AT TIME ZONE 'UTC', CURRENT_TIMESTAMP AT TIME ZONE 'UTC')`,
uuid, token, string(scopesjson), userUUID)
if err != nil {
diff --git a/services/api/app/controllers/application_controller.rb b/services/api/app/controllers/application_controller.rb
index 83a233cd5..c7d7ad814 100644
--- a/services/api/app/controllers/application_controller.rb
+++ b/services/api/app/controllers/application_controller.rb
@@ -386,8 +386,8 @@ class ApplicationController < ActionController::Base
@read_auths += ApiClientAuthorization
.includes(:user)
.where('api_token IN (?) AND
- (expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP)',
- secrets)
+ (expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP AT TIME ZONE ?)',
+ secrets, 'UTC')
.to_a
end
@read_auths.select! { |auth| auth.scopes_allow_request? request }
diff --git a/services/api/app/models/api_client_authorization.rb b/services/api/app/models/api_client_authorization.rb
index 503dfc50d..a4bf0cd5b 100644
--- a/services/api/app/models/api_client_authorization.rb
+++ b/services/api/app/models/api_client_authorization.rb
@@ -158,7 +158,7 @@ class ApiClientAuthorization < ArvadosModel
# fast path: look up the token in the local database
auth = ApiClientAuthorization.
includes(:user, :api_client).
- where('uuid=? and (expires_at is null or expires_at > CURRENT_TIMESTAMP)', token_uuid).
+ where('uuid=? and (expires_at is null or expires_at > CURRENT_TIMESTAMP AT TIME ZONE ?)', token_uuid, 'UTC').
first
if auth && auth.user &&
(secret == auth.api_token ||
@@ -284,7 +284,7 @@ class ApiClientAuthorization < ArvadosModel
# token is not a 'v2' token
auth = ApiClientAuthorization.
includes(:user, :api_client).
- where('api_token=? and (expires_at is null or expires_at > CURRENT_TIMESTAMP)', token).
+ where('api_token=? and (expires_at is null or expires_at > CURRENT_TIMESTAMP AT TIME ZONE ?)', token, 'UTC').
first
if auth && auth.user
return auth
diff --git a/services/api/lib/create_superuser_token.rb b/services/api/lib/create_superuser_token.rb
index 57eac048a..c1530162e 100755
--- a/services/api/lib/create_superuser_token.rb
+++ b/services/api/lib/create_superuser_token.rb
@@ -40,7 +40,7 @@ module CreateSuperUserToken
where(user_id: system_user.id).
where(api_client_id: apiClient.id).
where_serialized(:scopes, ['all']).
- where('(expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP)').
+ where('(expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP AT TIME ZONE ?)', 'UTC').
first
# none exist; create one with the supplied token
commit cc6d74141a399b9f8401e1388bf6913114cad5bc
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed Apr 22 16:50:06 2020 -0400
16343: Fix handling of local CR creation when LoginCluster is used.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/lib/controller/fed_containers.go b/lib/controller/fed_containers.go
index a923f757f..c62cea116 100644
--- a/lib/controller/fed_containers.go
+++ b/lib/controller/fed_containers.go
@@ -42,13 +42,11 @@ func remoteContainerRequestCreate(
return true
}
- if *clusterId == "" {
- *clusterId = h.handler.Cluster.ClusterID
- }
-
- if strings.HasPrefix(currentUser.Authorization.UUID, h.handler.Cluster.ClusterID) &&
- *clusterId == h.handler.Cluster.ClusterID {
- // local user submitting container request to local cluster
+ if *clusterId == "" || *clusterId == h.handler.Cluster.ClusterID {
+ // Submitting container request to local cluster. No
+ // need to set a runtime_token (rails api will create
+ // one when the container runs) or do a remote cluster
+ // request.
return false
}
commit 1334a03fb592d13f72ee5379cc4ac1199240d2e7
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed Apr 22 16:31:06 2020 -0400
16343: Add debug logs in token checking code.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/lib/controller/federation.go b/lib/controller/federation.go
index 674183dcc..c0d127284 100644
--- a/lib/controller/federation.go
+++ b/lib/controller/federation.go
@@ -19,6 +19,7 @@ import (
"git.arvados.org/arvados.git/sdk/go/arvados"
"git.arvados.org/arvados.git/sdk/go/auth"
+ "git.arvados.org/arvados.git/sdk/go/ctxlog"
"github.com/jmcvetta/randutil"
)
@@ -153,6 +154,7 @@ func (h *Handler) validateAPItoken(req *http.Request, token string) (*CurrentUse
user := CurrentUser{Authorization: arvados.APIClientAuthorization{APIToken: token}}
db, err := h.db(req)
if err != nil {
+ ctxlog.FromContext(req.Context()).WithError(err).Debugf("validateAPItoken(%s): database error", token)
return nil, false, err
}
@@ -166,18 +168,23 @@ func (h *Handler) validateAPItoken(req *http.Request, token string) (*CurrentUse
var scopes string
err = db.QueryRowContext(req.Context(), `SELECT api_client_authorizations.uuid, api_client_authorizations.scopes, users.uuid FROM api_client_authorizations JOIN users on api_client_authorizations.user_id=users.id WHERE api_token=$1 AND (expires_at IS NULL OR expires_at > current_timestamp) LIMIT 1`, token).Scan(&user.Authorization.UUID, &scopes, &user.UUID)
if err == sql.ErrNoRows {
+ ctxlog.FromContext(req.Context()).Debugf("validateAPItoken(%s): not found in database", token)
return nil, false, nil
} else if err != nil {
+ ctxlog.FromContext(req.Context()).WithError(err).Debugf("validateAPItoken(%s): database error", token)
return nil, false, err
}
if uuid != "" && user.Authorization.UUID != uuid {
// secret part matches, but UUID doesn't -- somewhat surprising
+ ctxlog.FromContext(req.Context()).Debugf("validateAPItoken(%s): secret part found, but with different UUID: %s", token, user.Authorization.UUID)
return nil, false, nil
}
err = json.Unmarshal([]byte(scopes), &user.Authorization.Scopes)
if err != nil {
+ ctxlog.FromContext(req.Context()).WithError(err).Debugf("validateAPItoken(%s): error parsing scopes from db", token)
return nil, false, err
}
+ ctxlog.FromContext(req.Context()).Debugf("validateAPItoken(%s): ok", token)
return &user, true, nil
}
diff --git a/services/api/app/models/api_client_authorization.rb b/services/api/app/models/api_client_authorization.rb
index 77fc0a45a..503dfc50d 100644
--- a/services/api/app/models/api_client_authorization.rb
+++ b/services/api/app/models/api_client_authorization.rb
@@ -164,6 +164,9 @@ class ApiClientAuthorization < ArvadosModel
(secret == auth.api_token ||
secret == OpenSSL::HMAC.hexdigest('sha1', auth.api_token, remote))
# found it
+ if token_uuid[0..4] != Rails.configuration.ClusterID
+ Rails.logger.debug "found cached remote token #{token_uuid} with secret #{secret} in local db"
+ end
return auth
end
@@ -274,6 +277,7 @@ class ApiClientAuthorization < ArvadosModel
api_token: secret,
api_client_id: 0,
expires_at: Time.now + Rails.configuration.Login.RemoteTokenRefresh)
+ Rails.logger.debug "cached remote token #{token_uuid} with secret #{secret} in local db"
end
return auth
else
commit cfa70fa206cef453a736187c1b23c8360f010cb8
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Apr 28 16:44:30 2020 -0400
16366: "InternalURLs" and "AccessViaHosts" are consistent in install
refs #16366
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/install/configure-azure-blob-storage.html.textile.liquid b/doc/install/configure-azure-blob-storage.html.textile.liquid
index 8c5098abe..2ccec586e 100644
--- a/doc/install/configure-azure-blob-storage.html.textile.liquid
+++ b/doc/install/configure-azure-blob-storage.html.textile.liquid
@@ -67,8 +67,8 @@ Volumes are configured in the @Volumes@ section of the cluster configuration fil
# If the AccessViaHosts section is empty or omitted, all
# keepstore servers will have read/write access to the
# volume.
- "http://<span class="userinput">keep0.ClusterID.example.com</span>:25107/": {}
- "http://<span class="userinput">keep1.ClusterID.example.com</span>:25107/": {ReadOnly: true}
+ "http://<span class="userinput">keep0.ClusterID.example.com</span>:25107": {}
+ "http://<span class="userinput">keep1.ClusterID.example.com</span>:25107": {ReadOnly: true}
Driver: <span class="userinput">Azure</span>
DriverParameters:
diff --git a/doc/install/configure-s3-object-storage.html.textile.liquid b/doc/install/configure-s3-object-storage.html.textile.liquid
index 9708ea5cd..e953f660f 100644
--- a/doc/install/configure-s3-object-storage.html.textile.liquid
+++ b/doc/install/configure-s3-object-storage.html.textile.liquid
@@ -25,8 +25,8 @@ Volumes are configured in the @Volumes@ section of the cluster configuration fil
# If the AccessViaHosts section is empty or omitted, all
# keepstore servers will have read/write access to the
# volume.
- "http://<span class="userinput">keep0.ClusterID.example.com</span>:25107/": {}
- "http://<span class="userinput">keep1.ClusterID.example.com</span>:25107/": {ReadOnly: true}
+ "http://<span class="userinput">keep0.ClusterID.example.com</span>:25107": {}
+ "http://<span class="userinput">keep1.ClusterID.example.com</span>:25107": {ReadOnly: true}
Driver: <span class="userinput">S3</span>
DriverParameters:
diff --git a/doc/install/install-keepstore.html.textile.liquid b/doc/install/install-keepstore.html.textile.liquid
index fedeb3c3f..869ca15d9 100644
--- a/doc/install/install-keepstore.html.textile.liquid
+++ b/doc/install/install-keepstore.html.textile.liquid
@@ -61,8 +61,8 @@ Add each keepstore server to the @Services.Keepstore@ section of @/etc/arvados/c
Keepstore:
# No ExternalURL because they are only accessed by the internal subnet.
InternalURLs:
- "http://<span class="userinput">keep0.ClusterID.example.com</span>:25107/": {}
- "http://<span class="userinput">keep1.ClusterID.example.com</span>:25107/": {}
+ "http://<span class="userinput">keep0.ClusterID.example.com</span>:25107": {}
+ "http://<span class="userinput">keep1.ClusterID.example.com</span>:25107": {}
# and so forth
</code></pre>
</notextile>
commit 6260e1b0d7778d247880357556debe51acf71cf4
Author: Ward Vandewege <ward at jhvc.com>
Date: Thu May 14 20:39:13 2020 -0400
16434: the python3-arvados-cwl-runner packages on Debian 10 and Ubuntu
18.04 (and later, presumably) have a dependency on the
python3-distutils package.
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/sdk/cwl/fpm-info.sh b/sdk/cwl/fpm-info.sh
index 5c47532db..977f53da6 100644
--- a/sdk/cwl/fpm-info.sh
+++ b/sdk/cwl/fpm-info.sh
@@ -9,6 +9,9 @@ case "$TARGET" in
debian* | ubuntu*)
fpm_depends+=(libcurl3-gnutls libpython2.7)
;;
+ debian* | ubuntu*)
+ fpm_depends+=(libcurl3-gnutls python3-distutils)
+ ;;
esac
fpm_args+=(--conflicts=python-cwltool --conflicts=cwltool)
commit e58c5263c3529a9f0fe586852612c991b5a8ceec
Author: Ward Vandewege <ward at jhvc.com>
Date: Thu May 14 20:40:14 2020 -0400
Fix comment typos in run-library.sh
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/build/run-library.sh b/build/run-library.sh
index eb6ee1b1f..de51d19bc 100755
--- a/build/run-library.sh
+++ b/build/run-library.sh
@@ -465,7 +465,7 @@ fpm_build_virtualenv () {
fi
# arvados-python-client sdist should always be built, to be available
- # for other dependant packages.
+ # for other dependent packages.
if [[ -n "$ONLY_BUILD" ]] && [[ "arvados-python-client" != "$PKG" ]] && [[ "$PYTHON_PKG" != "$ONLY_BUILD" ]] && [[ "$PKG" != "$ONLY_BUILD" ]]; then
return 0
fi
@@ -613,7 +613,7 @@ fpm_build_virtualenv () {
# 12271 - As FPM-generated packages don't include scripts by default, the
# packages cleanup on upgrade depends on files being listed on the %files
# section in the generated SPEC files. To remove DIRECTORIES, they need to
- # be listed in that sectiontoo, so we need to add this parameter to properly
+ # be listed in that section too, so we need to add this parameter to properly
# remove lingering dirs. But this only works for python2: if used on
# python33, it includes dirs like /opt/rh/python33 that belong to
# other packages.
commit 0267966c0452f398b6f56077e426584446ad403e
Author: Ward Vandewege <ward at jhvc.com>
Date: Thu May 14 20:40:55 2020 -0400
16434: add a basic execution test for the python3-arvados-cwl-runner packages
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/build/package-testing/test-package-python3-arvados-cwl-runner.sh b/build/package-testing/test-package-python3-arvados-cwl-runner.sh
new file mode 100755
index 000000000..99327c016
--- /dev/null
+++ b/build/package-testing/test-package-python3-arvados-cwl-runner.sh
@@ -0,0 +1,8 @@
+#!/bin/sh
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+set -e
+
+arvados-cwl-runner --version
commit 5c2cfee366a58d54fc39f47ccd84a5db5d96d856
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed May 6 15:39:30 2020 -0400
16387: Allow setting is_active=false only on LoginCluster users.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/api/app/controllers/arvados/v1/users_controller.rb b/services/api/app/controllers/arvados/v1/users_controller.rb
index 647b62fea..ede7572e6 100644
--- a/services/api/app/controllers/arvados/v1/users_controller.rb
+++ b/services/api/app/controllers/arvados/v1/users_controller.rb
@@ -51,9 +51,10 @@ class Arvados::V1::UsersController < ApplicationController
@object = current_user
end
if not @object.is_active
- if @object.uuid[0..4] != Rails.configuration.ClusterID
- logger.warn "Remote user #{@object.uuid} called users.activate"
- raise ArgumentError.new "cannot activate remote account"
+ if @object.uuid[0..4] == Rails.configuration.Login.LoginCluster &&
+ @object.uuid[0..4] != Rails.configuration.ClusterID
+ logger.warn "Local user #{@object.uuid} called users#activate but only LoginCluster can do that"
+ raise ArgumentError.new "cannot activate user #{@object.uuid} here, only the #{@object.uuid[0..4]} cluster can do that"
elsif not (current_user.is_admin or @object.is_invited)
logger.warn "User #{@object.uuid} called users.activate " +
"but is not invited"
diff --git a/services/api/app/models/user.rb b/services/api/app/models/user.rb
index 3f0a97062..609e26413 100644
--- a/services/api/app/models/user.rb
+++ b/services/api/app/models/user.rb
@@ -238,10 +238,15 @@ class User < ArvadosModel
end
def must_unsetup_to_deactivate
- if self.is_active_changed? &&
+ if !self.new_record? &&
+ self.uuid[0..4] == Rails.configuration.Login.LoginCluster &&
+ self.uuid[0..4] != Rails.configuration.ClusterID
+ # OK to update our local record to whatever the LoginCluster
+ # reports, because self-activate is not allowed.
+ return
+ elsif self.is_active_changed? &&
self.is_active_was &&
- !self.is_active &&
- self.uuid[0..4] == Rails.configuration.ClusterID
+ !self.is_active
group = Group.where(name: 'All users').select do |g|
g[:uuid].match(/-f+$/)
commit e94ed22c9f40ecafd3cb1ee1349eaee018b4d883
Author: Tom Clegg <tom at tomclegg.ca>
Date: Mon May 4 10:53:21 2020 -0400
16387: Allow batch update to set is_active=false for a remote user.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/api/app/controllers/arvados/v1/users_controller.rb b/services/api/app/controllers/arvados/v1/users_controller.rb
index 6a5fbbc50..647b62fea 100644
--- a/services/api/app/controllers/arvados/v1/users_controller.rb
+++ b/services/api/app/controllers/arvados/v1/users_controller.rb
@@ -51,7 +51,10 @@ class Arvados::V1::UsersController < ApplicationController
@object = current_user
end
if not @object.is_active
- if not (current_user.is_admin or @object.is_invited)
+ if @object.uuid[0..4] != Rails.configuration.ClusterID
+ logger.warn "Remote user #{@object.uuid} called users.activate"
+ raise ArgumentError.new "cannot activate remote account"
+ elsif not (current_user.is_admin or @object.is_invited)
logger.warn "User #{@object.uuid} called users.activate " +
"but is not invited"
raise ArgumentError.new "Cannot activate without being invited."
diff --git a/services/api/app/models/user.rb b/services/api/app/models/user.rb
index 310c2ca69..3f0a97062 100644
--- a/services/api/app/models/user.rb
+++ b/services/api/app/models/user.rb
@@ -239,8 +239,9 @@ class User < ArvadosModel
def must_unsetup_to_deactivate
if self.is_active_changed? &&
- self.is_active_was == true &&
- !self.is_active
+ self.is_active_was &&
+ !self.is_active &&
+ self.uuid[0..4] == Rails.configuration.ClusterID
group = Group.where(name: 'All users').select do |g|
g[:uuid].match(/-f+$/)
commit d7597b98e702ad5ab9c059ee3c3528b9d3bda292
Author: Tom Clegg <tom at tomclegg.ca>
Date: Fri May 1 15:58:58 2020 -0400
Add StartedAt and FinishedAt fields to arvados.Container.
No issue #
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/sdk/go/arvados/container.go b/sdk/go/arvados/container.go
index 653312d86..fddea2c00 100644
--- a/sdk/go/arvados/container.go
+++ b/sdk/go/arvados/container.go
@@ -28,6 +28,8 @@ type Container struct {
SchedulingParameters SchedulingParameters `json:"scheduling_parameters"`
ExitCode int `json:"exit_code"`
RuntimeStatus map[string]interface{} `json:"runtime_status"`
+ StartedAt *time.Time `json:"started_at"` // nil if not yet started
+ FinishedAt *time.Time `json:"finished_at"` // nil if not yet finished
}
// Container is an arvados#container resource.
commit 6b11d6756340eb9ba557b4148c4e5df8d46660b9
Author: Ward Vandewege <ward at jhvc.com>
Date: Thu May 14 20:57:49 2020 -0400
bugfix: fix arvados-node-manager package build
refs #16373
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/build/run-library.sh b/build/run-library.sh
index 3a1ed1c1a..eb6ee1b1f 100755
--- a/build/run-library.sh
+++ b/build/run-library.sh
@@ -673,9 +673,9 @@ fpm_build_virtualenv () {
done
# make sure the systemd service file ends up in the right place
- # currently only used by arvados-docker-cleaner
+ # used by arvados-docker-cleaner and arvados-node-manager
if [[ -e "${systemd_unit}" ]]; then
- COMMAND_ARR+=("usr/share/python3/dist/$PKG/share/doc/$PKG/$PKG.service=/lib/systemd/system/$PKG.service")
+ COMMAND_ARR+=("usr/share/$python/dist/$PKG/share/doc/$PKG/$PKG.service=/lib/systemd/system/$PKG.service")
fi
COMMAND_ARR+=("${fpm_args[@]}")
commit 26f06e7a69aaf7c84995d452c16f7a97ddf6032d
Author: Ward Vandewege <ward at jhvc.com>
Date: Thu May 7 10:20:50 2020 -0400
Bugfix: make the arvados-docker-cleaner packages install their systemd
unit file in the correct directory.
closes #16373
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/build/run-library.sh b/build/run-library.sh
index ac5dc718b..3a1ed1c1a 100755
--- a/build/run-library.sh
+++ b/build/run-library.sh
@@ -638,7 +638,8 @@ fpm_build_virtualenv () {
COMMAND_ARR+=('-n' "$PYTHON_PKG")
COMMAND_ARR+=('-C' "build")
- if [[ -e "$WORKSPACE/$PKG_DIR/$PKG.service" ]]; then
+ systemd_unit="$WORKSPACE/$PKG_DIR/$PKG.service"
+ if [[ -e "${systemd_unit}" ]]; then
COMMAND_ARR+=('--after-install' "${WORKSPACE}/build/go-python-package-scripts/postinst")
COMMAND_ARR+=('--before-remove' "${WORKSPACE}/build/go-python-package-scripts/prerm")
fi
@@ -671,6 +672,12 @@ fpm_build_virtualenv () {
COMMAND_ARR+=('--depends' "$i")
done
+ # make sure the systemd service file ends up in the right place
+ # currently only used by arvados-docker-cleaner
+ if [[ -e "${systemd_unit}" ]]; then
+ COMMAND_ARR+=("usr/share/python3/dist/$PKG/share/doc/$PKG/$PKG.service=/lib/systemd/system/$PKG.service")
+ fi
+
COMMAND_ARR+=("${fpm_args[@]}")
# Make sure to install all our package binaries in /usr/bin.
commit 1b9451fc5c3765191cb17bfb4ed4468ecbf077aa
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Thu May 21 20:08:40 2020 -0300
16435: Updates the documentation.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/doc/architecture/index.html.textile.liquid b/doc/architecture/index.html.textile.liquid
index c7ea3268e..705048cd6 100644
--- a/doc/architecture/index.html.textile.liquid
+++ b/doc/architecture/index.html.textile.liquid
@@ -56,4 +56,4 @@ table(table table-bordered table-condensed).
|keep-block-check|Given a list of keep block locators, check that each block exists on one of the configured keepstore servers and verify the block hash.|
|keep-exercise|Benchmarking tool to test throughput and reliability of keepstores under various usage patterns.|
|keep-rsync|Get lists of blocks from two clusters, copy blocks which exist on source cluster but are missing from destination cluster.|
-|sync-groups|Take a CSV file listing (group, username) pairs and synchronize membership in Arvados groups.|
+|sync-groups|Take a CSV file listing with (group, user, permission) records and synchronize membership in Arvados groups.|
diff --git a/doc/user/topics/arvados-sync-groups.html.textile.liquid b/doc/user/topics/arvados-sync-groups.html.textile.liquid
index 9a609039b..7d831bf04 100644
--- a/doc/user/topics/arvados-sync-groups.html.textile.liquid
+++ b/doc/user/topics/arvados-sync-groups.html.textile.liquid
@@ -15,10 +15,12 @@ h1. Using arvados-sync-groups
This tool reads a CSV (comma-separated values) file having information about external groups and their members. When running it for the first time, it'll create a special group named 'Externally synchronized groups' meant to be the parent of all the remote groups.
-Every line on the file should have 2 values: a group name and a local user identifier, meaning that the named user is a member of the group. The tool will create the group if it doesn't exist, and add the user to it. If group member is not present on the input file, the account will be removed from the group.
+Every line on the file should have 3 values: a group name, a local user identifier and a permission level, meaning that the named user is a member of the group with the provided permission. The tool will create the group if it doesn't exist, and add the user to it. If any group member is not present on the input file, it will be removed from the group.
Users can be identified by their email address or username: the tool will check if every user exist on the system, and report back when not found. Groups on the other hand, are identified by their name.
+Permission level can be one of the following: @can_read@, @can_write@ or @can_manage@, giving the group member read, read/write or managing privileges on the group. For backwards compatibility purposes, if any record omits the third (permission) field, it will default to @can_write@ permission. You can read more about permissions on the "group management admin guide":/admin/group-management.html.
+
This tool is designed to be run periodically reading a file created by a remote auth system (ie: LDAP) dump script, applying what's included on the file as the source of truth.
commit 3889fe0ff14d710817f53ba4d517445df89c2342
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Thu May 21 19:00:37 2020 -0300
16435: Adds & updates tests.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/tools/sync-groups/sync-groups_test.go b/tools/sync-groups/sync-groups_test.go
index 77d9756ff..9eec6b6d9 100644
--- a/tools/sync-groups/sync-groups_test.go
+++ b/tools/sync-groups/sync-groups_test.go
@@ -106,7 +106,7 @@ func MakeTempCSVFile(data [][]string) (f *os.File, err error) {
}
// GroupMembershipExists checks that both needed links exist between user and group
-func GroupMembershipExists(ac *arvados.Client, userUUID string, groupUUID string) bool {
+func GroupMembershipExists(ac *arvados.Client, userUUID string, groupUUID string, perm string) bool {
ll := LinkList{}
// Check Group -> User can_read permission
params := arvados.ResourceListParams{
@@ -145,7 +145,7 @@ func GroupMembershipExists(ac *arvados.Client, userUUID string, groupUUID string
}, {
Attr: "name",
Operator: "=",
- Operand: "can_write",
+ Operand: perm,
}, {
Attr: "tail_uuid",
Operator: "=",
@@ -259,7 +259,7 @@ func (s *TestSuite) TestIgnoreSpaces(c *C) {
groupUUID, err := RemoteGroupExists(s.cfg, groupName)
c.Assert(err, IsNil)
c.Assert(groupUUID, Not(Equals), "")
- c.Assert(GroupMembershipExists(s.cfg.Client, activeUserUUID, groupUUID), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, activeUserUUID, groupUUID, "can_write"), Equals, true)
}
}
@@ -279,6 +279,83 @@ func (s *TestSuite) TestWrongNumberOfFields(c *C) {
}
}
+// Check different membership permissions
+func (s *TestSuite) TestMembershipLevels(c *C) {
+ userEmail := s.users[arvadostest.ActiveUserUUID].Email
+ userUUID := s.users[arvadostest.ActiveUserUUID].UUID
+ data := [][]string{
+ {"TestGroup1", userEmail, "can_read"},
+ {"TestGroup2", userEmail, "can_write"},
+ {"TestGroup3", userEmail, "can_manage"},
+ {"TestGroup4", userEmail, "invalid_permission"},
+ }
+ tmpfile, err := MakeTempCSVFile(data)
+ c.Assert(err, IsNil)
+ defer os.Remove(tmpfile.Name()) // clean up
+ s.cfg.Path = tmpfile.Name()
+ err = doMain(s.cfg)
+ c.Assert(err, IsNil)
+ for _, record := range data {
+ groupName := record[0]
+ permLevel := record[2]
+ if permLevel != "invalid_permission" {
+ groupUUID, err := RemoteGroupExists(s.cfg, groupName)
+ c.Assert(err, IsNil)
+ c.Assert(groupUUID, Not(Equals), "")
+ c.Assert(GroupMembershipExists(s.cfg.Client, userUUID, groupUUID, permLevel), Equals, true)
+ } else {
+ groupUUID, err := RemoteGroupExists(s.cfg, groupName)
+ c.Assert(err, IsNil)
+ c.Assert(groupUUID, Equals, "")
+ }
+ }
+}
+
+// Check membership level change
+func (s *TestSuite) TestMembershipLevelUpdate(c *C) {
+ userEmail := s.users[arvadostest.ActiveUserUUID].Email
+ userUUID := s.users[arvadostest.ActiveUserUUID].UUID
+ groupName := "TestGroup1"
+ // Give read permissions
+ tmpfile, err := MakeTempCSVFile([][]string{{groupName, userEmail, "can_read"}})
+ c.Assert(err, IsNil)
+ defer os.Remove(tmpfile.Name()) // clean up
+ s.cfg.Path = tmpfile.Name()
+ err = doMain(s.cfg)
+ c.Assert(err, IsNil)
+ // Check permissions
+ groupUUID, err := RemoteGroupExists(s.cfg, groupName)
+ c.Assert(err, IsNil)
+ c.Assert(groupUUID, Not(Equals), "")
+ c.Assert(GroupMembershipExists(s.cfg.Client, userUUID, groupUUID, "can_read"), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, userUUID, groupUUID, "can_write"), Equals, false)
+ c.Assert(GroupMembershipExists(s.cfg.Client, userUUID, groupUUID, "can_manage"), Equals, false)
+
+ // Give write permissions
+ tmpfile, err = MakeTempCSVFile([][]string{{groupName, userEmail, "can_write"}})
+ c.Assert(err, IsNil)
+ defer os.Remove(tmpfile.Name()) // clean up
+ s.cfg.Path = tmpfile.Name()
+ err = doMain(s.cfg)
+ c.Assert(err, IsNil)
+ // Check permissions
+ c.Assert(GroupMembershipExists(s.cfg.Client, userUUID, groupUUID, "can_read"), Equals, false)
+ c.Assert(GroupMembershipExists(s.cfg.Client, userUUID, groupUUID, "can_write"), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, userUUID, groupUUID, "can_manage"), Equals, false)
+
+ // Give manage permissions
+ tmpfile, err = MakeTempCSVFile([][]string{{groupName, userEmail, "can_manage"}})
+ c.Assert(err, IsNil)
+ defer os.Remove(tmpfile.Name()) // clean up
+ s.cfg.Path = tmpfile.Name()
+ err = doMain(s.cfg)
+ c.Assert(err, IsNil)
+ // Check permissions
+ c.Assert(GroupMembershipExists(s.cfg.Client, userUUID, groupUUID, "can_read"), Equals, false)
+ c.Assert(GroupMembershipExists(s.cfg.Client, userUUID, groupUUID, "can_write"), Equals, false)
+ c.Assert(GroupMembershipExists(s.cfg.Client, userUUID, groupUUID, "can_manage"), Equals, true)
+}
+
// The absence of a user membership on the CSV file implies its removal
func (s *TestSuite) TestMembershipRemoval(c *C) {
localUserEmail := s.users[arvadostest.ActiveUserUUID].Email
@@ -302,8 +379,8 @@ func (s *TestSuite) TestMembershipRemoval(c *C) {
groupUUID, err := RemoteGroupExists(s.cfg, groupName)
c.Assert(err, IsNil)
c.Assert(groupUUID, Not(Equals), "")
- c.Assert(GroupMembershipExists(s.cfg.Client, localUserUUID, groupUUID), Equals, true)
- c.Assert(GroupMembershipExists(s.cfg.Client, remoteUserUUID, groupUUID), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, localUserUUID, groupUUID, "can_write"), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, remoteUserUUID, groupUUID, "can_write"), Equals, true)
}
// New CSV with some previous membership missing
data = [][]string{
@@ -320,14 +397,14 @@ func (s *TestSuite) TestMembershipRemoval(c *C) {
groupUUID, err := RemoteGroupExists(s.cfg, "TestGroup1")
c.Assert(err, IsNil)
c.Assert(groupUUID, Not(Equals), "")
- c.Assert(GroupMembershipExists(s.cfg.Client, localUserUUID, groupUUID), Equals, true)
- c.Assert(GroupMembershipExists(s.cfg.Client, remoteUserUUID, groupUUID), Equals, false)
+ c.Assert(GroupMembershipExists(s.cfg.Client, localUserUUID, groupUUID, "can_write"), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, remoteUserUUID, groupUUID, "can_write"), Equals, false)
// Confirm TestGroup1 memberships
groupUUID, err = RemoteGroupExists(s.cfg, "TestGroup2")
c.Assert(err, IsNil)
c.Assert(groupUUID, Not(Equals), "")
- c.Assert(GroupMembershipExists(s.cfg.Client, localUserUUID, groupUUID), Equals, false)
- c.Assert(GroupMembershipExists(s.cfg.Client, remoteUserUUID, groupUUID), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, localUserUUID, groupUUID, "can_write"), Equals, false)
+ c.Assert(GroupMembershipExists(s.cfg.Client, remoteUserUUID, groupUUID, "can_write"), Equals, true)
}
// If a group doesn't exist on the system, create it before adding users
@@ -352,7 +429,7 @@ func (s *TestSuite) TestAutoCreateGroupWhenNotExisting(c *C) {
c.Assert(err, IsNil)
c.Assert(groupUUID, Not(Equals), "")
// active user should be a member
- c.Assert(GroupMembershipExists(s.cfg.Client, arvadostest.ActiveUserUUID, groupUUID), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, arvadostest.ActiveUserUUID, groupUUID, "can_write"), Equals, true)
}
// Users listed on the file that don't exist on the system are ignored
@@ -378,7 +455,7 @@ func (s *TestSuite) TestIgnoreNonexistantUsers(c *C) {
groupUUID, err = RemoteGroupExists(s.cfg, "TestGroup4")
c.Assert(err, IsNil)
c.Assert(groupUUID, Not(Equals), "")
- c.Assert(GroupMembershipExists(s.cfg.Client, activeUserUUID, groupUUID), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, activeUserUUID, groupUUID, "can_write"), Equals, true)
}
// Users listed on the file that don't exist on the system are ignored
@@ -386,13 +463,16 @@ func (s *TestSuite) TestIgnoreEmptyFields(c *C) {
activeUserEmail := s.users[arvadostest.ActiveUserUUID].Email
activeUserUUID := s.users[arvadostest.ActiveUserUUID].UUID
// Confirm that group doesn't exist
- groupUUID, err := RemoteGroupExists(s.cfg, "TestGroup4")
- c.Assert(err, IsNil)
- c.Assert(groupUUID, Equals, "")
+ for _, groupName := range []string{"TestGroup4", "TestGroup5"} {
+ groupUUID, err := RemoteGroupExists(s.cfg, groupName)
+ c.Assert(err, IsNil)
+ c.Assert(groupUUID, Equals, "")
+ }
// Create file & run command
data := [][]string{
- {"", activeUserEmail}, // Empty field
- {"TestGroup5", ""}, // Empty field
+ {"", activeUserEmail}, // Empty field
+ {"TestGroup5", ""}, // Empty field
+ {"TestGroup5", activeUserEmail, ""}, // Empty 3rd field: is optional but cannot be empty
{"TestGroup4", activeUserEmail},
}
tmpfile, err := MakeTempCSVFile(data)
@@ -401,11 +481,15 @@ func (s *TestSuite) TestIgnoreEmptyFields(c *C) {
s.cfg.Path = tmpfile.Name()
err = doMain(s.cfg)
c.Assert(err, IsNil)
- // Confirm that memberships exist
+ // Confirm that records about TestGroup5 were skipped
+ groupUUID, err := RemoteGroupExists(s.cfg, "TestGroup5")
+ c.Assert(err, IsNil)
+ c.Assert(groupUUID, Equals, "")
+ // Confirm that membership exists
groupUUID, err = RemoteGroupExists(s.cfg, "TestGroup4")
c.Assert(err, IsNil)
c.Assert(groupUUID, Not(Equals), "")
- c.Assert(GroupMembershipExists(s.cfg.Client, activeUserUUID, groupUUID), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, activeUserUUID, groupUUID, "can_write"), Equals, true)
}
// Instead of emails, use username as identifier
@@ -432,5 +516,5 @@ func (s *TestSuite) TestUseUsernames(c *C) {
groupUUID, err = RemoteGroupExists(s.cfg, "TestGroup1")
c.Assert(err, IsNil)
c.Assert(groupUUID, Not(Equals), "")
- c.Assert(GroupMembershipExists(s.cfg.Client, activeUserUUID, groupUUID), Equals, true)
+ c.Assert(GroupMembershipExists(s.cfg.Client, activeUserUUID, groupUUID, "can_write"), Equals, true)
}
commit 2ea16ba491194afca89500fcee80e91dac0d3d95
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Thu May 21 18:11:10 2020 -0300
16435: Avoids creating duplicated group->user links.
When a user needs a permission change, the g->u link already exists.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/tools/sync-groups/sync-groups.go b/tools/sync-groups/sync-groups.go
index 2ffd61fec..9e2307b7a 100644
--- a/tools/sync-groups/sync-groups.go
+++ b/tools/sync-groups/sync-groups.go
@@ -450,8 +450,13 @@ func ProcessFile(
if cfg.Verbose {
log.Printf("Adding %q to group %q", groupMember, groupName)
}
- // User wasn't a member, but should be.
- if e := AddMemberToGroup(cfg, allUsers[userIDToUUID[groupMember]], gi.Group, groupPermission); e != nil {
+ // User permissionwasn't there, but should be. Avoid duplicating the
+ // group->user link when necessary.
+ createG2ULink := true
+ if _, ok := gi.PreviousMembers[groupMember]; ok {
+ createG2ULink = false // User is already member of the group
+ }
+ if e := AddMemberToGroup(cfg, allUsers[userIDToUUID[groupMember]], gi.Group, groupPermission, createG2ULink); e != nil {
err = e
return
}
@@ -498,7 +503,7 @@ func subtract(setA map[string]GroupPermissions, setB map[string]GroupPermissions
} else {
for perm := range setA[element] {
if _, ok := setB[element][perm]; !ok {
- result[element][perm] = true
+ result[element] = GroupPermissions{perm: true}
}
}
}
@@ -716,18 +721,21 @@ func RemoveMemberLinksFromGroup(cfg *ConfigParams, user arvados.User, linkNames
}
// AddMemberToGroup create membership links
-func AddMemberToGroup(cfg *ConfigParams, user arvados.User, group arvados.Group, perm string) error {
+func AddMemberToGroup(cfg *ConfigParams, user arvados.User, group arvados.Group, perm string, createG2ULink bool) error {
var newLink arvados.Link
- linkData := map[string]string{
- "owner_uuid": cfg.SysUserUUID,
- "link_class": "permission",
- "name": "can_read",
- "tail_uuid": group.UUID,
- "head_uuid": user.UUID,
- }
- if err := CreateLink(cfg, &newLink, linkData); err != nil {
- userID, _ := GetUserID(user, cfg.UserID)
- return fmt.Errorf("error adding group %q -> user %q read permission: %s", group.Name, userID, err)
+ var linkData map[string]string
+ if createG2ULink {
+ linkData = map[string]string{
+ "owner_uuid": cfg.SysUserUUID,
+ "link_class": "permission",
+ "name": "can_read",
+ "tail_uuid": group.UUID,
+ "head_uuid": user.UUID,
+ }
+ if err := CreateLink(cfg, &newLink, linkData); err != nil {
+ userID, _ := GetUserID(user, cfg.UserID)
+ return fmt.Errorf("error adding group %q -> user %q read permission: %s", group.Name, userID, err)
+ }
}
linkData = map[string]string{
"owner_uuid": cfg.SysUserUUID,
commit 5958df985bfc8293dc94f06b2c23824cb4c8f41b
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Thu May 21 16:21:12 2020 -0300
16435: Adds support for different permission levels: read, write & manage.
If the 3rd field isn't present on any record, it will fallback to 'can_write'
to maintain backwards compatibility.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/tools/sync-groups/sync-groups.go b/tools/sync-groups/sync-groups.go
index b77d8ccdd..2ffd61fec 100644
--- a/tools/sync-groups/sync-groups.go
+++ b/tools/sync-groups/sync-groups.go
@@ -26,11 +26,14 @@ type resourceList interface {
GetItems() []interface{}
}
-// GroupInfo tracks previous and current members of a particular Group
+// GroupPermissions maps permission levels on groups (can_read, can_write, can_manage)
+type GroupPermissions map[string]bool
+
+// GroupInfo tracks previous and current member's permissions on a particular Group
type GroupInfo struct {
Group arvados.Group
- PreviousMembers map[string]bool
- CurrentMembers map[string]bool
+ PreviousMembers map[string]GroupPermissions
+ CurrentMembers map[string]GroupPermissions
}
// GetUserID returns the correct user id value depending on the selector
@@ -137,7 +140,7 @@ func ParseFlags(config *ConfigParams) error {
usageStr := `Synchronize remote groups into Arvados from a CSV format file with 3 columns:
* 1st: Group name
* 2nd: User identifier
- * 3rd (Optional): User permission on the group: read, write or manage. (Default: write)`
+ * 3rd (Optional): User permission on the group: can_read, can_write or can_manage. (Default: can_write)`
fmt.Fprintf(os.Stderr, "%s\n\n", usageStr)
fmt.Fprintf(os.Stderr, "Usage:\n%s [OPTIONS] <input-file.csv>\n\n", os.Args[0])
fmt.Fprintf(os.Stderr, "Options:\n")
@@ -335,16 +338,30 @@ func doMain(cfg *ConfigParams) error {
// Remove previous members not listed on this run
for groupUUID := range remoteGroups {
gi := remoteGroups[groupUUID]
- evictedMembers := subtract(gi.PreviousMembers, gi.CurrentMembers)
+ evictedMemberPerms := subtract(gi.PreviousMembers, gi.CurrentMembers)
groupName := gi.Group.Name
- if len(evictedMembers) > 0 {
- log.Printf("Removing %d users from group %q", len(evictedMembers), groupName)
- }
- for evictedUser := range evictedMembers {
- if err := RemoveMemberFromGroup(cfg, allUsers[userIDToUUID[evictedUser]], gi.Group); err != nil {
+ if len(evictedMemberPerms) > 0 {
+ log.Printf("Removing permissions from %d users on group %q", len(evictedMemberPerms), groupName)
+ }
+ for member := range evictedMemberPerms {
+ var perms []string
+ completeMembershipRemoval := false
+ if _, ok := gi.CurrentMembers[member]; !ok {
+ completeMembershipRemoval = true
+ membershipsRemoved++
+ } else {
+ // Collect which user->group permission links should be removed
+ for p := range evictedMemberPerms[member] {
+ if evictedMemberPerms[member][p] {
+ perms = append(perms, p)
+ }
+ }
+ membershipsRemoved += len(perms)
+ }
+ if err := RemoveMemberLinksFromGroup(cfg, allUsers[userIDToUUID[member]],
+ perms, completeMembershipRemoval, gi.Group); err != nil {
return err
}
- membershipsRemoved++
}
}
log.Printf("Groups created: %d. Memberships added: %d, removed: %d, skipped: %d", groupsCreated, membershipsAdded, membershipsRemoved, membershipsSkipped)
@@ -382,8 +399,17 @@ func ProcessFile(
}
groupName := strings.TrimSpace(record[0])
groupMember := strings.TrimSpace(record[1]) // User ID (username or email)
- if groupName == "" || groupMember == "" {
- log.Printf("Warning: CSV record has at least one empty field (%s, %s). Skipping", groupName, groupMember)
+ groupPermission := "can_write"
+ if len(record) == 3 {
+ groupPermission = strings.ToLower(record[2])
+ }
+ if groupName == "" || groupMember == "" || groupPermission == "" {
+ log.Printf("Warning: CSV record has at least one empty field (%s, %s, %s). Skipping", groupName, groupMember, groupPermission)
+ membersSkipped++
+ continue
+ }
+ if !(groupPermission == "can_read" || groupPermission == "can_write" || groupPermission == "can_manage") {
+ log.Printf("Warning: 3rd field should be 'can_read', 'can_write' or 'can_manage'. Found: %q at line %d, skipping.", groupPermission, lineNo)
membersSkipped++
continue
}
@@ -412,26 +438,31 @@ func ProcessFile(
groupNameToUUID[groupName] = newGroup.UUID
remoteGroups[newGroup.UUID] = &GroupInfo{
Group: newGroup,
- PreviousMembers: make(map[string]bool), // Empty set
- CurrentMembers: make(map[string]bool), // Empty set
+ PreviousMembers: make(map[string]GroupPermissions),
+ CurrentMembers: make(map[string]GroupPermissions),
}
groupsCreated++
}
// Both group & user exist, check if user is a member
groupUUID := groupNameToUUID[groupName]
gi := remoteGroups[groupUUID]
- if !gi.PreviousMembers[groupMember] && !gi.CurrentMembers[groupMember] {
+ if !gi.PreviousMembers[groupMember][groupPermission] && !gi.CurrentMembers[groupMember][groupPermission] {
if cfg.Verbose {
log.Printf("Adding %q to group %q", groupMember, groupName)
}
// User wasn't a member, but should be.
- if e := AddMemberToGroup(cfg, allUsers[userIDToUUID[groupMember]], gi.Group); e != nil {
+ if e := AddMemberToGroup(cfg, allUsers[userIDToUUID[groupMember]], gi.Group, groupPermission); e != nil {
err = e
return
}
membersAdded++
}
- gi.CurrentMembers[groupMember] = true
+ if _, ok := gi.CurrentMembers[groupMember]; ok {
+ gi.CurrentMembers[groupMember][groupPermission] = true
+ } else {
+ gi.CurrentMembers[groupMember] = GroupPermissions{groupPermission: true}
+ }
+
}
return
}
@@ -459,11 +490,17 @@ func GetAll(c *arvados.Client, res string, params arvados.ResourceListParams, pa
return allItems, nil
}
-func subtract(setA map[string]bool, setB map[string]bool) map[string]bool {
- result := make(map[string]bool)
+func subtract(setA map[string]GroupPermissions, setB map[string]GroupPermissions) map[string]GroupPermissions {
+ result := make(map[string]GroupPermissions)
for element := range setA {
- if !setB[element] {
- result[element] = true
+ if _, ok := setB[element]; !ok {
+ result[element] = setA[element]
+ } else {
+ for perm := range setA[element] {
+ if _, ok := setB[element][perm]; !ok {
+ result[element][perm] = true
+ }
+ }
}
}
return result
@@ -533,8 +570,8 @@ func GetRemoteGroups(cfg *ConfigParams, allUsers map[string]arvados.User) (remot
Operand: "permission",
}, {
Attr: "name",
- Operator: "=",
- Operand: "can_write",
+ Operator: "in",
+ Operand: []string{"can_read", "can_write", "can_manage"},
}, {
Attr: "head_uuid",
Operator: "=",
@@ -547,18 +584,23 @@ func GetRemoteGroups(cfg *ConfigParams, allUsers map[string]arvados.User) (remot
}
g2uLinks, err := GetAll(cfg.Client, "links", g2uFilter, &LinkList{})
if err != nil {
- return remoteGroups, groupNameToUUID, fmt.Errorf("error getting member (can_read) links for group %q: %s", group.Name, err)
+ return remoteGroups, groupNameToUUID, fmt.Errorf("error getting group->user 'can_read' links for group %q: %s", group.Name, err)
}
u2gLinks, err := GetAll(cfg.Client, "links", u2gFilter, &LinkList{})
if err != nil {
- return remoteGroups, groupNameToUUID, fmt.Errorf("error getting member (can_write) links for group %q: %s", group.Name, err)
+ return remoteGroups, groupNameToUUID, fmt.Errorf("error getting user->group links for group %q: %s", group.Name, err)
}
- // Build a list of user ids (email or username) belonging to this group
- membersSet := make(map[string]bool)
- u2gLinkSet := make(map[string]bool)
+ // Build a list of user ids (email or username) belonging to this group.
+ membersSet := make(map[string]GroupPermissions)
+ u2gLinkSet := make(map[string]GroupPermissions)
for _, l := range u2gLinks {
- linkedMemberUUID := l.(arvados.Link).TailUUID
- u2gLinkSet[linkedMemberUUID] = true
+ link := l.(arvados.Link)
+ // Also save the member's group access level.
+ if _, ok := u2gLinkSet[link.TailUUID]; ok {
+ u2gLinkSet[link.TailUUID][link.Name] = true
+ } else {
+ u2gLinkSet[link.TailUUID] = GroupPermissions{link.Name: true}
+ }
}
for _, item := range g2uLinks {
link := item.(arvados.Link)
@@ -576,55 +618,81 @@ func GetRemoteGroups(cfg *ConfigParams, allUsers map[string]arvados.User) (remot
if err != nil {
return remoteGroups, groupNameToUUID, err
}
- membersSet[memberID] = true
+ membersSet[memberID] = u2gLinkSet[link.HeadUUID]
}
remoteGroups[group.UUID] = &GroupInfo{
Group: group,
PreviousMembers: membersSet,
- CurrentMembers: make(map[string]bool), // Empty set
+ CurrentMembers: make(map[string]GroupPermissions),
}
groupNameToUUID[group.Name] = group.UUID
}
return remoteGroups, groupNameToUUID, nil
}
-// RemoveMemberFromGroup remove all links related to the membership
-func RemoveMemberFromGroup(cfg *ConfigParams, user arvados.User, group arvados.Group) error {
+// RemoveMemberLinksFromGroup remove all links related to the membership
+func RemoveMemberLinksFromGroup(cfg *ConfigParams, user arvados.User, linkNames []string, completeRemoval bool, group arvados.Group) error {
if cfg.Verbose {
log.Printf("Getting group membership links for user %q (%s) on group %q (%s)", user.Username, user.UUID, group.Name, group.UUID)
}
var links []interface{}
- // Search for all group<->user links (both ways)
- for _, filterset := range [][]arvados.Filter{
- // Group -> User
- {{
- Attr: "link_class",
- Operator: "=",
- Operand: "permission",
- }, {
- Attr: "tail_uuid",
- Operator: "=",
- Operand: group.UUID,
- }, {
- Attr: "head_uuid",
- Operator: "=",
- Operand: user.UUID,
- }},
- // Group <- User
- {{
- Attr: "link_class",
- Operator: "=",
- Operand: "permission",
- }, {
- Attr: "tail_uuid",
- Operator: "=",
- Operand: user.UUID,
- }, {
- Attr: "head_uuid",
- Operator: "=",
- Operand: group.UUID,
- }},
- } {
+ var filters [][]arvados.Filter
+ if completeRemoval {
+ // Search for all group<->user links (both ways)
+ filters = [][]arvados.Filter{
+ // Group -> User
+ {{
+ Attr: "link_class",
+ Operator: "=",
+ Operand: "permission",
+ }, {
+ Attr: "tail_uuid",
+ Operator: "=",
+ Operand: group.UUID,
+ }, {
+ Attr: "head_uuid",
+ Operator: "=",
+ Operand: user.UUID,
+ }},
+ // Group <- User
+ {{
+ Attr: "link_class",
+ Operator: "=",
+ Operand: "permission",
+ }, {
+ Attr: "tail_uuid",
+ Operator: "=",
+ Operand: user.UUID,
+ }, {
+ Attr: "head_uuid",
+ Operator: "=",
+ Operand: group.UUID,
+ }},
+ }
+ } else {
+ // Search only for the requested Group <- User permission links
+ filters = [][]arvados.Filter{
+ {{
+ Attr: "link_class",
+ Operator: "=",
+ Operand: "permission",
+ }, {
+ Attr: "tail_uuid",
+ Operator: "=",
+ Operand: user.UUID,
+ }, {
+ Attr: "head_uuid",
+ Operator: "=",
+ Operand: group.UUID,
+ }, {
+ Attr: "name",
+ Operator: "in",
+ Operand: linkNames,
+ }},
+ }
+ }
+
+ for _, filterset := range filters {
l, err := GetAll(cfg.Client, "links", arvados.ResourceListParams{Filters: filterset}, &LinkList{})
if err != nil {
userID, _ := GetUserID(user, cfg.UserID)
@@ -648,7 +716,7 @@ func RemoveMemberFromGroup(cfg *ConfigParams, user arvados.User, group arvados.G
}
// AddMemberToGroup create membership links
-func AddMemberToGroup(cfg *ConfigParams, user arvados.User, group arvados.Group) error {
+func AddMemberToGroup(cfg *ConfigParams, user arvados.User, group arvados.Group, perm string) error {
var newLink arvados.Link
linkData := map[string]string{
"owner_uuid": cfg.SysUserUUID,
@@ -664,13 +732,13 @@ func AddMemberToGroup(cfg *ConfigParams, user arvados.User, group arvados.Group)
linkData = map[string]string{
"owner_uuid": cfg.SysUserUUID,
"link_class": "permission",
- "name": "can_write",
+ "name": perm,
"tail_uuid": user.UUID,
"head_uuid": group.UUID,
}
if err := CreateLink(cfg, &newLink, linkData); err != nil {
userID, _ := GetUserID(user, cfg.UserID)
- return fmt.Errorf("error adding user %q -> group %q write permission: %s", userID, group.Name, err)
+ return fmt.Errorf("error adding user %q -> group %q %s permission: %s", userID, group.Name, perm, err)
}
return nil
}
commit 8b6836581b1ce61075f6e9020d18acd858a86cdf
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Wed May 20 18:23:08 2020 -0300
16435: Allows 2 or 3 fields per record on the CSV file.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/tools/sync-groups/sync-groups.go b/tools/sync-groups/sync-groups.go
index 7c5cd0558..b77d8ccdd 100644
--- a/tools/sync-groups/sync-groups.go
+++ b/tools/sync-groups/sync-groups.go
@@ -134,9 +134,10 @@ func ParseFlags(config *ConfigParams) error {
// Set up usage message
flags.Usage = func() {
- usageStr := `Synchronize remote groups into Arvados from a CSV format file with 2 columns:
- * 1st column: Group name
- * 2nd column: User identifier`
+ usageStr := `Synchronize remote groups into Arvados from a CSV format file with 3 columns:
+ * 1st: Group name
+ * 2nd: User identifier
+ * 3rd (Optional): User permission on the group: read, write or manage. (Default: write)`
fmt.Fprintf(os.Stderr, "%s\n\n", usageStr)
fmt.Fprintf(os.Stderr, "Usage:\n%s [OPTIONS] <input-file.csv>\n\n", os.Args[0])
fmt.Fprintf(os.Stderr, "Options:\n")
@@ -362,7 +363,8 @@ func ProcessFile(
) (groupsCreated, membersAdded, membersSkipped int, err error) {
lineNo := 0
csvReader := csv.NewReader(f)
- csvReader.FieldsPerRecord = 2
+ // Allow variable number of fields.
+ csvReader.FieldsPerRecord = -1
for {
record, e := csvReader.Read()
if e == io.EOF {
@@ -373,6 +375,11 @@ func ProcessFile(
err = fmt.Errorf("error parsing %q, line %d", cfg.Path, lineNo)
return
}
+ // Only allow 2 or 3 fields per record for backwards compatibility.
+ if len(record) < 2 || len(record) > 3 {
+ err = fmt.Errorf("error parsing %q, line %d: found %d fields but only 2 or 3 are allowed", cfg.Path, lineNo, len(record))
+ return
+ }
groupName := strings.TrimSpace(record[0])
groupMember := strings.TrimSpace(record[1]) // User ID (username or email)
if groupName == "" || groupMember == "" {
diff --git a/tools/sync-groups/sync-groups_test.go b/tools/sync-groups/sync-groups_test.go
index 3ef360079..77d9756ff 100644
--- a/tools/sync-groups/sync-groups_test.go
+++ b/tools/sync-groups/sync-groups_test.go
@@ -263,6 +263,22 @@ func (s *TestSuite) TestIgnoreSpaces(c *C) {
}
}
+// Error out when records have <2 or >3 records
+func (s *TestSuite) TestWrongNumberOfFields(c *C) {
+ for _, testCase := range [][][]string{
+ {{"field1"}},
+ {{"field1", "field2", "field3", "field4"}},
+ {{"field1", "field2", "field3", "field4", "field5"}},
+ } {
+ tmpfile, err := MakeTempCSVFile(testCase)
+ c.Assert(err, IsNil)
+ defer os.Remove(tmpfile.Name())
+ s.cfg.Path = tmpfile.Name()
+ err = doMain(s.cfg)
+ c.Assert(err, Not(IsNil))
+ }
+}
+
// The absence of a user membership on the CSV file implies its removal
func (s *TestSuite) TestMembershipRemoval(c *C) {
localUserEmail := s.users[arvadostest.ActiveUserUUID].Email
commit ccc6ca8fb944d45c94f2ae994d3c0f0567c1b91b
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu May 21 09:51:03 2020 -0400
16419: Use CAINFO instead of CAPATH
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/python/arvados/keep.py b/sdk/python/arvados/keep.py
index 9601601d4..bc43b849c 100644
--- a/sdk/python/arvados/keep.py
+++ b/sdk/python/arvados/keep.py
@@ -376,7 +376,7 @@ class KeepClient(object):
if self.insecure:
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
else:
- curl.setopt(pycurl.CAPATH,os.path.dirname(arvados.util.ca_certs_path()))
+ curl.setopt(pycurl.CAINFO, arvados.util.ca_certs_path())
if method == "HEAD":
curl.setopt(pycurl.NOBODY, True)
self._setcurltimeouts(curl, timeout, method=="HEAD")
@@ -476,7 +476,7 @@ class KeepClient(object):
if self.insecure:
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
else:
- curl.setopt(pycurl.CAPATH,os.path.dirname(arvados.util.ca_certs_path()))
+ curl.setopt(pycurl.CAINFO, arvados.util.ca_certs_path())
self._setcurltimeouts(curl, timeout)
try:
curl.perform()
commit c6523d3fdf49b9d7b0fc998f64473725ad9346fa
Author: Pjotr Prins <pjotr.public01 at thebird.nl>
Date: Wed May 20 14:35:54 2020 -0500
keep.py: python-api https certificate align pycurl with httplib2 certificate finder
Arvados-DCO-1.1-Signed-off-by: Pjotr Prins <pjotr.public01 at thebird.nl>
diff --git a/sdk/python/arvados/keep.py b/sdk/python/arvados/keep.py
index 86a28f54c..9601601d4 100644
--- a/sdk/python/arvados/keep.py
+++ b/sdk/python/arvados/keep.py
@@ -375,6 +375,8 @@ class KeepClient(object):
curl.setopt(pycurl.HEADERFUNCTION, self._headerfunction)
if self.insecure:
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
+ else:
+ curl.setopt(pycurl.CAPATH,os.path.dirname(arvados.util.ca_certs_path()))
if method == "HEAD":
curl.setopt(pycurl.NOBODY, True)
self._setcurltimeouts(curl, timeout, method=="HEAD")
@@ -473,6 +475,8 @@ class KeepClient(object):
curl.setopt(pycurl.HEADERFUNCTION, self._headerfunction)
if self.insecure:
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
+ else:
+ curl.setopt(pycurl.CAPATH,os.path.dirname(arvados.util.ca_certs_path()))
self._setcurltimeouts(curl, timeout)
try:
curl.perform()
commit 4506c6105f716cceb091954d8589dbea12696bea
Author: Pjotr Prins <pjotr.public01 at thebird.nl>
Date: Wed May 20 11:12:20 2020 -0500
util.py: python-api https certificate openssl override as is used in GNU Guix
Arvados-DCO-1.1-Signed-off-by: Pjotr Prins <pjotr.public01 at thebird.nl>
diff --git a/sdk/python/arvados/util.py b/sdk/python/arvados/util.py
index dcc0417c1..6c9822e9f 100644
--- a/sdk/python/arvados/util.py
+++ b/sdk/python/arvados/util.py
@@ -396,6 +396,9 @@ def ca_certs_path(fallback=httplib2.CA_CERTS):
it returns the value of `fallback` (httplib2's CA certs by default).
"""
for ca_certs_path in [
+ # SSL_CERT_FILE and SSL_CERT_DIR are openssl overrides - note
+ # that httplib2 itself also supports HTTPLIB2_CA_CERTS.
+ os.environ.get('SSL_CERT_FILE'),
# Arvados specific:
'/etc/arvados/ca-certificates.crt',
# Debian:
@@ -403,7 +406,7 @@ def ca_certs_path(fallback=httplib2.CA_CERTS):
# Red Hat:
'/etc/pki/tls/certs/ca-bundle.crt',
]:
- if os.path.exists(ca_certs_path):
+ if ca_certs_path and os.path.exists(ca_certs_path):
return ca_certs_path
return fallback
commit f9fe0abf384122c003cac247840f2113fccec59b
Author: Ward Vandewege <ward at jhvc.com>
Date: Thu Apr 30 15:32:57 2020 -0400
Arvados on Kubernetes now has workbench2 as well.
closes #16384
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/doc/install/arvados-on-kubernetes.html.textile.liquid b/doc/install/arvados-on-kubernetes.html.textile.liquid
index 3169b3247..ff52aa171 100644
--- a/doc/install/arvados-on-kubernetes.html.textile.liquid
+++ b/doc/install/arvados-on-kubernetes.html.textile.liquid
@@ -22,7 +22,6 @@ This Helm chart provides a basic, small Arvados cluster.
Current limitations, to be addressed in the future:
* An Arvados cluster created with this Helm chart is entirely ephemeral, and all data stored on the cluster will be deleted when it is shut down.
-* Workbench2 is not present yet
* No dynamic scaling of compute nodes (but you can adjust @values.yaml@ and reload the Helm chart)
* All compute nodes are the same size
* Compute nodes have no cpu/memory/disk constraints yet
commit 3921a3d0418cefb63f3b2e87b4f837ae43dbb275
Author: Ward Vandewege <ward at jhvc.com>
Date: Mon Apr 27 15:58:34 2020 -0400
Updated Arvados on Kubernetes documentation.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/doc/_config.yml b/doc/_config.yml
index 01a19c16b..d053da122 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -186,6 +186,8 @@ navbar:
- install/arvbox.html.textile.liquid
- Arvados on Kubernetes:
- install/arvados-on-kubernetes.html.textile.liquid
+ - install/arvados-on-kubernetes-minikube.html.textile.liquid
+ - install/arvados-on-kubernetes-GKE.html.textile.liquid
- Manual installation:
- install/install-manual-prerequisites.html.textile.liquid
- install/packages.html.textile.liquid
diff --git a/doc/install/arvados-on-kubernetes-GKE.html.textile.liquid b/doc/install/arvados-on-kubernetes-GKE.html.textile.liquid
index 88b2d5730..c4236b1c2 100644
--- a/doc/install/arvados-on-kubernetes-GKE.html.textile.liquid
+++ b/doc/install/arvados-on-kubernetes-GKE.html.textile.liquid
@@ -1,7 +1,7 @@
---
layout: default
navsection: installguide
-title: Arvados on Kubernetes - Google Kubernetes Engine
+title: Arvados on GKE
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
@@ -9,7 +9,13 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-This page documents the setup of the prerequisites to run the "Arvados on Kubernetes":/install/arvados-on-kubernetes.html @Helm@ chart on @Google Kubernetes Engine@ (GKE).
+This page documents setting up and running the "Arvados on Kubernetes":/install/arvados-on-kubernetes.html @Helm@ chart on @Google Kubernetes Engine@ (GKE).
+
+{% include 'notebox_begin_warning' %}
+This Helm chart does not retain any state after it is deleted. An Arvados cluster created with this Helm chart is entirely ephemeral, and all data stored on the cluster will be deleted when it is shut down. This will be fixed in a future version.
+{% include 'notebox_end' %}
+
+h2. Prerequisites
h3. Install tooling
@@ -27,12 +33,12 @@ Install @helm@:
* Follow the instructions at "https://docs.helm.sh/using_helm/#installing-helm":https://docs.helm.sh/using_helm/#installing-helm
-h3. Boot the GKE cluster
+h3. Create the GKE cluster
This can be done via the "cloud console":https://console.cloud.google.com/kubernetes/ or via the command line:
<pre>
-$ gcloud container clusters create <CLUSTERNAME> --zone us-central1-a --machine-type n1-standard-2 --cluster-version 1.10
+$ gcloud container clusters create <CLUSTERNAME> --zone us-central1-a --machine-type n1-standard-2 --cluster-version 1.15
</pre>
It takes a few minutes for the cluster to be initialized.
@@ -59,4 +65,91 @@ Test the connection:
$ kubectl get nodes
</pre>
-Now proceed to the "Initialize helm on the Kubernetes cluster":/install/arvados-on-kubernetes.html#helm section.
+Test @helm@ by running
+
+<pre>
+$ helm ls
+</pre>
+
+There should be no errors. The command will return nothing.
+
+h2(#git). Clone the repository
+
+Clone the repository and nagivate to the @arvados-kubernetes/charts/arvados@ directory:
+
+<pre>
+$ git clone https://github.com/arvados/arvados-kubernetes.git
+$ cd arvados-kubernetes/charts/arvados
+</pre>
+
+h2(#Start). Start the Arvados cluster
+
+Next, determine the IP address that the Arvados cluster will use to expose its API, Workbench, etc. If you want this Arvados cluster to be reachable from places other than the local machine, the IP address will need to be routable as appropriate.
+
+<pre>
+$ ./cert-gen.sh <IP ADDRESS>
+</pre>
+
+The @values.yaml@ file contains a number of variables that can be modified. At a minimum, review and/or modify the values for
+
+<pre>
+ adminUserEmail
+ adminUserPassword
+ superUserSecret
+ anonymousUserSecret
+</pre>
+
+Now start the Arvados cluster:
+
+<pre>
+$ helm install arvados . --set externalIP=<IP ADDRESS>
+</pre>
+
+At this point, you can use kubectl to see the Arvados cluster boot:
+
+<pre>
+$ kubectl get pods
+$ kubectl get svc
+</pre>
+
+After a few minutes, there shouldn't be any services listed with a 'Pending' external IP address. At that point you can access Arvados Workbench at the IP address specified
+
+* https://<IP ADDRESS>
+
+with the username and password specified in the @values.yaml@ file.
+
+Alternatively, use the Arvados cli tools or SDKs. First set the environment variables:
+
+<pre>
+$ export ARVADOS_API_TOKEN=<superUserSecret from values.yaml>
+$ export ARVADOS_API_HOST=<STATIC IP>:444
+$ export ARVADOS_API_HOST_INSECURE=true
+</pre>
+
+Test access with:
+
+<pre>
+$ arv user current
+</pre>
+
+h2(#reload). Reload
+
+If you make changes to the Helm chart (e.g. to @values.yaml@), you can reload Arvados with
+
+<pre>
+$ helm upgrade arvados .
+</pre>
+
+h2. Shut down
+
+{% include 'notebox_begin_warning' %}
+This Helm chart does not retain any state after it is deleted. An Arvados cluster created with this Helm chart is entirely ephemeral, and <strong>all data stored on the Arvados cluster will be deleted</strong> when it is shut down. This will be fixed in a future version.
+{% include 'notebox_end' %}
+
+<pre>
+$ helm del arvados
+</pre>
+
+<pre>
+$ gcloud container clusters delete <CLUSTERNAME> --zone us-central1-a
+</pre>
diff --git a/doc/install/arvados-on-kubernetes-minikube.html.textile.liquid b/doc/install/arvados-on-kubernetes-minikube.html.textile.liquid
index 132b443df..56a51b035 100644
--- a/doc/install/arvados-on-kubernetes-minikube.html.textile.liquid
+++ b/doc/install/arvados-on-kubernetes-minikube.html.textile.liquid
@@ -1,7 +1,7 @@
---
layout: default
navsection: installguide
-title: Arvados on Kubernetes - Minikube
+title: Arvados on Minikube
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
@@ -9,7 +9,13 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-This page documents the setup of the prerequisites to run the "Arvados on Kubernetes":/install/arvados-on-kubernetes.html @Helm@ chart on @Minikube at .
+This page documents setting up and running the "Arvados on Kubernetes":/install/arvados-on-kubernetes.html @Helm@ chart on @Minikube at .
+
+{% include 'notebox_begin_warning' %}
+This Helm chart does not retain any state after it is deleted. An Arvados cluster created with this Helm chart is entirely ephemeral, and all data stored on the cluster will be deleted when it is shut down. This will be fixed in a future version.
+{% include 'notebox_end' %}
+
+h2. Prerequisites
h3. Install tooling
@@ -31,4 +37,100 @@ Test the connection:
$ kubectl get nodes
</pre>
-Now proceed to the "Initialize helm on the Kubernetes cluster":/install/arvados-on-kubernetes.html#helm section.
+Test @helm@ by running
+
+<pre>
+$ helm ls
+</pre>
+
+There should be no errors. The command will return nothing.
+
+h2(#git). Clone the repository
+
+Clone the repository and nagivate to the @arvados-kubernetes/charts/arvados@ directory:
+
+<pre>
+$ git clone https://github.com/arvados/arvados-kubernetes.git
+$ cd arvados-kubernetes/charts/arvados
+</pre>
+
+h2(#Start). Start the Arvados cluster
+
+All Arvados services will be accessible on Minikube's IP address. This will be a local IP address, you can see what it is by running
+
+<pre>
+$ minikube ip
+192.168.39.15
+</pre>
+
+Generate self-signed SSL certificates for the Arvados services:
+
+<pre>
+$ ./cert-gen.sh `minikube ip`
+</pre>
+
+The @values.yaml@ file contains a number of variables that can be modified. At a minimum, review and/or modify the values for
+
+<pre>
+ adminUserEmail
+ adminUserPassword
+ superUserSecret
+ anonymousUserSecret
+</pre>
+
+Now start the Arvados cluster:
+
+<pre>
+$ helm install arvados . --set externalIP=`minikube ip`
+</pre>
+
+And update the Kubernetes services to have the Minikube IP as their 'external' IP:
+
+<pre>
+$ ./minikube-external-ip.sh
+</pre>
+
+At this point, you can use kubectl to see the Arvados cluster boot:
+
+<pre>
+$ kubectl get pods
+$ kubectl get svc
+</pre>
+
+After a few minutes, you can access Arvados Workbench at the Minikube IP address:
+
+* https://<MINIKUBE IP>
+
+with the username and password specified in the @values.yaml@ file.
+
+Alternatively, use the Arvados cli tools or SDKs. First set the environment variables:
+
+<pre>
+$ export ARVADOS_API_TOKEN=<superUserSecret from values.yaml>
+$ export ARVADOS_API_HOST=<MINIKUBE IP>:444
+$ export ARVADOS_API_HOST_INSECURE=true
+</pre>
+
+Test access with:
+
+<pre>
+$ arv user current
+</pre>
+
+h2(#reload). Reload
+
+If you make changes to the Helm chart (e.g. to @values.yaml@), you can reload Arvados with
+
+<pre>
+$ helm upgrade arvados .
+</pre>
+
+h2. Shut down
+
+{% include 'notebox_begin_warning' %}
+This Helm chart does not retain any state after it is deleted. An Arvados cluster created with this Helm chart is entirely ephemeral, and <strong>all data stored on the Arvados cluster will be deleted</strong> when it is shut down. This will be fixed in a future version.
+{% include 'notebox_end' %}
+
+<pre>
+$ helm del arvados
+</pre>
diff --git a/doc/install/arvados-on-kubernetes.html.textile.liquid b/doc/install/arvados-on-kubernetes.html.textile.liquid
index 08b2c7329..3169b3247 100644
--- a/doc/install/arvados-on-kubernetes.html.textile.liquid
+++ b/doc/install/arvados-on-kubernetes.html.textile.liquid
@@ -9,125 +9,29 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Arvados on Kubernetes is implemented as a Helm Chart.
+Arvados on Kubernetes is implemented as a @Helm 3@ chart.
{% include 'notebox_begin_warning' %}
-This Helm Chart does not retain any state after it is deleted. An Arvados cluster created with this Helm Chart is entirely ephemeral, and all data stored on the cluster will be deleted when it is shut down. This will be fixed in a future version.
+This Helm chart does not retain any state after it is deleted. An Arvados cluster created with this Helm chart is entirely ephemeral, and all data stored on the cluster will be deleted when it is shut down. This will be fixed in a future version.
{% include 'notebox_end' %}
h2(#overview). Overview
-This Helm Chart provides a basic, small Arvados cluster.
+This Helm chart provides a basic, small Arvados cluster.
Current limitations, to be addressed in the future:
-* An Arvados cluster created with this Helm Chart is entirely ephemeral, and all data stored on the cluster will be deleted when it is shut down.
-* No dynamic scaling of compute nodes (but you can adjust @values.yaml@ and "reload the Helm Chart":#reload
+* An Arvados cluster created with this Helm chart is entirely ephemeral, and all data stored on the cluster will be deleted when it is shut down.
+* Workbench2 is not present yet
+* No dynamic scaling of compute nodes (but you can adjust @values.yaml@ and reload the Helm chart)
* All compute nodes are the same size
* Compute nodes have no cpu/memory/disk constraints yet
* No git server
h2. Requirements
-* Kubernetes 1.10+ cluster with at least 3 nodes, 2 or more cores per node
-* @kubectl@ and @helm@ installed locally, and able to connect to your Kubernetes cluster
+* Minikube or Google Kubernetes Engine (Kubernetes 1.10+ with at least 3 nodes, 2+ cores per node)
+* @kubectl@ and @Helm 3@ installed locally, and able to connect to your Kubernetes cluster
-If you do not have a Kubernetes cluster already set up, you can use "Google Kubernetes Engine":/install/arvados-on-kubernetes-GKE.html for multi-node development and testing or "another Kubernetes solution":https://kubernetes.io/docs/setup/pick-right-solution/. Minikube is not supported yet.
+Please refer to "Arvados on Minikube":/install/arvados-on-kubernetes-minikube.html or "Arvados on GKE":/install/arvados-on-kubernetes-GKE.html for detailed installation instructions.
-h2(#helm). Initialize helm on the Kubernetes cluster
-
-If you already have helm running on the Kubernetes cluster, proceed directly to "Start the Arvados cluster":#Start below.
-
-<pre>
-$ helm init
-$ kubectl create serviceaccount --namespace kube-system tiller
-$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
-$ kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
-</pre>
-
-Test @helm@ by running
-
-<pre>
-$ helm ls
-</pre>
-
-There should be no errors. The command will return nothing.
-
-h2(#git). Clone the repository
-
-Clone the repository and nagivate to the @arvados-kubernetes/charts/arvados@ directory:
-
-<pre>
-$ git clone https://github.com/arvados/arvados-kubernetes.git
-$ cd arvados-kubernetes/charts/arvados
-</pre>
-
-h2(#Start). Start the Arvados cluster
-
-Next, determine the IP address that the Arvados cluster will use to expose its API, Workbench, etc. If you want this Arvados cluster to be reachable from places other than the local machine, the IP address will need to be routable as appropriate.
-
-<pre>
-$ ./cert-gen.sh <IP ADDRESS>
-</pre>
-
-The @values.yaml@ file contains a number of variables that can be modified. At a minimum, review and/or modify the values for
-
-<pre>
- adminUserEmail
- adminUserPassword
- superUserSecret
- anonymousUserSecret
-</pre>
-
-Now start the Arvados cluster:
-
-<pre>
-$ helm install --name arvados . --set externalIP=<IP ADDRESS>
-</pre>
-
-At this point, you can use kubectl to see the Arvados cluster boot:
-
-<pre>
-$ kubectl get pods
-$ kubectl get svc
-</pre>
-
-After a few minutes, you can access Arvados Workbench at the IP address specified
-
-* https://<IP ADDRESS>
-
-with the username and password specified in the @values.yaml@ file.
-
-Alternatively, use the Arvados cli tools or SDKs:
-
-Set the environment variables:
-
-<pre>
-$ export ARVADOS_API_TOKEN=<superUserSecret from values.yaml>
-$ export ARVADOS_API_HOST=<STATIC IP>:444
-$ export ARVADOS_API_HOST_INSECURE=true
-</pre>
-
-Test access with:
-
-<pre>
-$ arv user current
-</pre>
-
-h2(#reload). Reload
-
-If you make changes to the Helm Chart (e.g. to @values.yaml@), you can reload Arvados with
-
-<pre>
-$ helm upgrade arvados .
-</pre>
-
-h2. Shut down
-
-{% include 'notebox_begin_warning' %}
-This Helm Chart does not retain any state after it is deleted. An Arvados cluster created with this Helm Chart is entirely ephemeral, and <strong>all data stored on the Arvados cluster will be deleted</strong> when it is shut down. This will be fixed in a future version.
-{% include 'notebox_end' %}
-
-<pre>
-$ helm del arvados --purge
-</pre>
commit 5f300020c51e8073a9cb6e45ee49991386244510
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu Mar 5 18:00:15 2020 -0500
Pin version of launchy gem while we still support Ruby 2.3
refs #16215
Need to do it for login-sync too.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/services/login-sync/arvados-login-sync.gemspec b/services/login-sync/arvados-login-sync.gemspec
index 17f671c6a..b45f8692b 100644
--- a/services/login-sync/arvados-login-sync.gemspec
+++ b/services/login-sync/arvados-login-sync.gemspec
@@ -37,6 +37,7 @@ Gem::Specification.new do |s|
s.executables << "arvados-login-sync"
s.required_ruby_version = '>= 2.1.0'
s.add_runtime_dependency 'arvados', '~> 1.3.0', '>= 1.3.0'
+ s.add_runtime_dependency 'launchy', '< 2.5'
# arvados-google-api-client 0.8.7.2 is incompatible with faraday 0.16.2
s.add_dependency('faraday', '< 0.16')
# arvados-google-api-client (and thus arvados) gems
commit b51b3fc069dfbf88de0ca064c2018628d9b9737f
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu Mar 5 16:27:33 2020 -0500
Pin version of launchy gem while we still support Ruby 2.3
refs #16215
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cli/arvados-cli.gemspec b/sdk/cli/arvados-cli.gemspec
index 2cfc2f438..88a5ceece 100644
--- a/sdk/cli/arvados-cli.gemspec
+++ b/sdk/cli/arvados-cli.gemspec
@@ -48,6 +48,7 @@ Gem::Specification.new do |s|
s.add_runtime_dependency 'andand', '~> 1.3', '>= 1.3.3'
s.add_runtime_dependency 'oj', '~> 3.0'
s.add_runtime_dependency 'curb', '~> 0.8'
+ s.add_runtime_dependency 'launchy', '< 2.5'
# arvados-google-api-client 0.8.7.2 is incompatible with faraday 0.16.2
s.add_dependency('faraday', '< 0.16')
s.homepage =
commit 87c40af4f0f1936ff992d080b23acf4fcb96c10f
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Mon Apr 13 15:34:12 2020 -0300
16313: Updates minimum requestd ram on docker containers from 4 to 16 MB.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/lib/crunchrun/crunchrun.go b/lib/crunchrun/crunchrun.go
index b0a4007f7..c8f171ca9 100644
--- a/lib/crunchrun/crunchrun.go
+++ b/lib/crunchrun/crunchrun.go
@@ -1074,9 +1074,10 @@ func (runner *ContainerRunner) CreateContainer() error {
runner.ContainerConfig.Volumes = runner.Volumes
maxRAM := int64(runner.Container.RuntimeConstraints.RAM)
- if maxRAM < 4*1024*1024 {
- // Docker daemon won't let you set a limit less than 4 MiB
- maxRAM = 4 * 1024 * 1024
+ minDockerRAM := int64(16)
+ if maxRAM < minDockerRAM*1024*1024 {
+ // Docker daemon won't let you set a limit less than ~10 MiB
+ maxRAM = minDockerRAM * 1024 * 1024
}
runner.HostConfig = dockercontainer.HostConfig{
Binds: runner.Binds,
commit 64c991ac1aa7524a3062cc77037e14c2b751a67d
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Apr 13 14:38:15 2020 -0400
Bump mithril for advisory https://www.npmjs.com/advisories/1094
no issue #
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/apps/workbench/npm_packages b/apps/workbench/npm_packages
index 63608ee37..05802b4ba 100644
--- a/apps/workbench/npm_packages
+++ b/apps/workbench/npm_packages
@@ -10,5 +10,5 @@ npm 'jquery'
npm 'awesomplete'
npm 'jssha', '2.4.2'
-npm 'mithril', '1.1.6'
+npm 'mithril', '1.1.7'
npm 'es6-object-assign'
commit 3e4ab085a7a05d5f528fc25dcc7915f1f183fe8b
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Apr 13 14:00:32 2020 -0400
Pin jssha to 2.4.2
no issue #
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/apps/workbench/npm_packages b/apps/workbench/npm_packages
index 7c757db35..63608ee37 100644
--- a/apps/workbench/npm_packages
+++ b/apps/workbench/npm_packages
@@ -8,7 +8,7 @@
npm 'browserify', require: false
npm 'jquery'
npm 'awesomplete'
-npm 'jssha'
+npm 'jssha', '2.4.2'
npm 'mithril', '1.1.6'
npm 'es6-object-assign'
commit f39904077030634f31fb0d1288fc963fb4cd22fb
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Apr 13 10:25:11 2020 -0400
16263: Add test for BypassFederation
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/federation/user_test.go b/lib/controller/federation/user_test.go
index c55ec24d4..09aa5086d 100644
--- a/lib/controller/federation/user_test.go
+++ b/lib/controller/federation/user_test.go
@@ -5,6 +5,7 @@
package federation
import (
+ "context"
"encoding/json"
"errors"
"math"
@@ -15,6 +16,8 @@ import (
"git.arvados.org/arvados.git/lib/controller/rpc"
"git.arvados.org/arvados.git/sdk/go/arvados"
"git.arvados.org/arvados.git/sdk/go/arvadostest"
+ "git.arvados.org/arvados.git/sdk/go/auth"
+ "git.arvados.org/arvados.git/sdk/go/ctxlog"
check "gopkg.in/check.v1"
)
@@ -114,6 +117,36 @@ func (s *UserSuite) TestLoginClusterUserList(c *check.C) {
}
}
+func (s *UserSuite) TestLoginClusterUserListBypassFederation(c *check.C) {
+ s.cluster.ClusterID = "local"
+ s.cluster.Login.LoginCluster = "zzzzz"
+ s.fed = New(s.cluster)
+ s.addDirectRemote(c, "zzzzz", rpc.NewConn("zzzzz", &url.URL{Scheme: "https", Host: os.Getenv("ARVADOS_API_HOST")},
+ true, rpc.PassthroughTokenProvider))
+
+ spy := arvadostest.NewProxy(c, s.cluster.Services.RailsAPI)
+ s.fed.local = rpc.NewConn(s.cluster.ClusterID, spy.URL, true, rpc.PassthroughTokenProvider)
+
+ _, err := s.fed.UserList(s.ctx, arvados.ListOptions{Offset: 0, Limit: math.MaxInt64, Select: nil, BypassFederation: true})
+ // this will fail because it is not using a root token
+ c.Check(err.(*arvados.TransactionError).StatusCode, check.Equals, 403)
+
+ // Now use SystemRootToken
+ ctx := context.Background()
+ ctx = ctxlog.Context(ctx, ctxlog.TestLogger(c))
+ ctx = auth.NewContext(ctx, &auth.Credentials{Tokens: []string{arvadostest.SystemRootToken}})
+
+ // Assert that it did not try to batch update users.
+ _, err = s.fed.UserList(ctx, arvados.ListOptions{Offset: 0, Limit: math.MaxInt64, Select: nil, BypassFederation: true})
+ for _, d := range spy.RequestDumps {
+ d := string(d)
+ if strings.Contains(d, "PATCH /arvados/v1/users/batch") {
+ c.Fail()
+ }
+ }
+ c.Check(err, check.IsNil)
+}
+
// userAttrsCachedFromLoginCluster must have an entry for every field
// in the User struct.
func (s *UserSuite) TestUserAttrsUpdateWhitelist(c *check.C) {
commit 2014c3c987831c25866906a9f8450f5c02fdae2c
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Mon Apr 6 11:01:18 2020 -0300
16263: Pulls from official git repo when running federation tests.
This is helpful to ask for specific branches that aren't published on github.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/sdk/cwl/tests/federation/arvboxcwl/start.cwl b/sdk/cwl/tests/federation/arvboxcwl/start.cwl
index 5d9aaba2d..a7f46d6b2 100644
--- a/sdk/cwl/tests/federation/arvboxcwl/start.cwl
+++ b/sdk/cwl/tests/federation/arvboxcwl/start.cwl
@@ -74,7 +74,7 @@ arguments:
mkdir -p $ARVBOX_DATA
if ! test -d $ARVBOX_DATA/arvados ; then
cd $ARVBOX_DATA
- git clone https://github.com/arvados/arvados.git
+ git clone https://git.arvados.org/arvados.git
fi
cd $ARVBOX_DATA/arvados
gitver=`git rev-parse HEAD`
commit 51a627a92a84783adcc895fdf652be95fce1e3f0
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu Apr 2 10:52:44 2020 -0400
16263: User migration test also checks federated user behavior
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/python/tests/fed-migrate/check.py b/sdk/python/tests/fed-migrate/check.py
index 8165b3eba..a2c009616 100644
--- a/sdk/python/tests/fed-migrate/check.py
+++ b/sdk/python/tests/fed-migrate/check.py
@@ -5,47 +5,54 @@ import sys
j = json.load(open(sys.argv[1]))
apiA = arvados.api(host=j["arvados_api_hosts"][0], token=j["superuser_tokens"][0], insecure=True)
-apiB = arvados.api(host=j["arvados_api_hosts"][1], token=j["superuser_tokens"][1], insecure=True)
-apiC = arvados.api(host=j["arvados_api_hosts"][2], token=j["superuser_tokens"][2], insecure=True)
+tok = apiA.api_client_authorizations().current().execute()
+v2_token = "v2/%s/%s" % (tok["uuid"], tok["api_token"])
+
+apiB = arvados.api(host=j["arvados_api_hosts"][1], token=v2_token, insecure=True)
+apiC = arvados.api(host=j["arvados_api_hosts"][2], token=v2_token, insecure=True)
###
### Check users on API server "A" (the LoginCluster) ###
###
-
-users = apiA.users().list(bypass_federation=True).execute()
-
-assert len(users["items"]) == 11
-
by_username = {}
-
-for i in range(1, 10):
+def check_A(users):
+ assert len(users["items"]) == 11
+
+ for i in range(1, 10):
+ found = False
+ for u in users["items"]:
+ if u["username"] == ("case%d" % i) and u["email"] == ("case%d at test" % i):
+ found = True
+ by_username[u["username"]] = u["uuid"]
+ assert found
+
+ # Should be active
+ for i in (1, 2, 3, 4, 5, 6, 7, 8):
+ found = False
+ for u in users["items"]:
+ if u["username"] == ("case%d" % i) and u["email"] == ("case%d at test" % i) and u["is_active"] is True:
+ found = True
+ assert found, "Not found case%i" % i
+
+ # case9 should not be active
found = False
for u in users["items"]:
- if u["username"] == ("case%d" % i) and u["email"] == ("case%d at test" % i):
+ if (u["username"] == "case9" and u["email"] == "case9 at test" and
+ u["uuid"] == by_username[u["username"]] and u["is_active"] is False):
found = True
- by_username[u["username"]] = u["uuid"]
assert found
-# Should be active
-for i in (1, 2, 3, 4, 5, 6, 7, 8):
- found = False
- for u in users["items"]:
- if u["username"] == ("case%d" % i) and u["email"] == ("case%d at test" % i) and u["is_active"] is True:
- found = True
- assert found, "Not found case%i" % i
-
-# case9 should not be active
-found = False
-for u in users["items"]:
- if (u["username"] == "case9" and u["email"] == "case9 at test" and
- u["uuid"] == by_username[u["username"]] and u["is_active"] is False):
- found = True
-assert found
+users = apiA.users().list().execute()
+check_A(users)
+users = apiA.users().list(bypass_federation=True).execute()
+check_A(users)
###
### Check users on API server "B" (federation member) ###
###
+
+# check for expected migrations on B
users = apiB.users().list(bypass_federation=True).execute()
assert len(users["items"]) == 11
@@ -64,10 +71,15 @@ for u in users["items"]:
found = True
assert found
+# check that federated user listing works
+users = apiB.users().list().execute()
+check_A(users)
###
### Check users on API server "C" (federation member) ###
###
+
+# check for expected migrations on C
users = apiC.users().list(bypass_federation=True).execute()
assert len(users["items"]) == 8
@@ -89,4 +101,8 @@ for i in (3, 5, 9):
found = True
assert not found
+# check that federated user listing works
+users = apiC.users().list().execute()
+check_A(users)
+
print("Passed checks")
commit d383273ba261256afae6112d3b0d4e2d03cc8240
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Mar 31 21:03:42 2020 -0400
16263: Tweak federation tests, use CWL 1.1
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cwl/tests/federation/README b/sdk/cwl/tests/federation/README
index e5eb04c60..f97ca9726 100644
--- a/sdk/cwl/tests/federation/README
+++ b/sdk/cwl/tests/federation/README
@@ -26,7 +26,7 @@ Create main-test.json:
Or create an arvbox test cluster:
-$ cwltool --enable-ext arvbox-make-federation.cwl --arvbox_base ~/.arvbox/ --in_acr /path/to/arvados-cwl-runner > main-test.json
+$ cwltool arvbox-make-federation.cwl --arvbox_base ~/.arvbox/ --in_acr /path/to/arvados-cwl-runner > main-test.json
Run tests:
diff --git a/sdk/cwl/tests/federation/arvbox-make-federation.cwl b/sdk/cwl/tests/federation/arvbox-make-federation.cwl
index 5872dbef5..593f2399f 100644
--- a/sdk/cwl/tests/federation/arvbox-make-federation.cwl
+++ b/sdk/cwl/tests/federation/arvbox-make-federation.cwl
@@ -2,7 +2,7 @@
#
# SPDX-License-Identifier: Apache-2.0
-cwlVersion: v1.0
+cwlVersion: v1.1
class: Workflow
$namespaces:
arv: "http://arvados.org/cwl#"
@@ -10,7 +10,7 @@ $namespaces:
requirements:
ScatterFeatureRequirement: {}
StepInputExpressionRequirement: {}
- cwltool:LoadListingRequirement:
+ LoadListingRequirement:
loadListing: no_listing
InlineJavascriptRequirement: {}
inputs:
@@ -64,7 +64,7 @@ steps:
containers: containers
arvbox_base: arvbox_base
out: [arvbox_data]
- run: arvbox/mkdir.cwl
+ run: arvboxcwl/mkdir.cwl
start:
in:
container_name: containers
@@ -74,7 +74,7 @@ steps:
out: [cluster_id, container_host, arvbox_data_out, superuser_token]
scatter: [container_name, arvbox_data]
scatterMethod: dotproduct
- run: arvbox/start.cwl
+ run: arvboxcwl/start.cwl
fed-config:
in:
container_name: containers
@@ -87,10 +87,10 @@ steps:
out: []
scatter: [container_name, this_cluster_id, arvbox_data]
scatterMethod: dotproduct
- run: arvbox/fed-config.cwl
+ run: arvboxcwl/fed-config.cwl
setup-user:
in:
container_host: {source: start/container_host, valueFrom: "$(self[0])"}
superuser_token: {source: start/superuser_token, valueFrom: "$(self[0])"}
out: [test_user_uuid, test_user_token]
- run: arvbox/setup-user.cwl
+ run: arvboxcwl/setup-user.cwl
diff --git a/sdk/cwl/tests/federation/arvbox/fed-config.cwl b/sdk/cwl/tests/federation/arvboxcwl/fed-config.cwl
similarity index 96%
rename from sdk/cwl/tests/federation/arvbox/fed-config.cwl
rename to sdk/cwl/tests/federation/arvboxcwl/fed-config.cwl
index 37936df63..e1cacdcaf 100644
--- a/sdk/cwl/tests/federation/arvbox/fed-config.cwl
+++ b/sdk/cwl/tests/federation/arvboxcwl/fed-config.cwl
@@ -2,7 +2,7 @@
#
# SPDX-License-Identifier: Apache-2.0
-cwlVersion: v1.0
+cwlVersion: v1.1
class: CommandLineTool
$namespaces:
arv: "http://arvados.org/cwl#"
@@ -56,11 +56,11 @@ requirements:
}
return JSON.stringify({"development": {"remote_hosts": remoteClusters}});
}
- cwltool:LoadListingRequirement:
+ LoadListingRequirement:
loadListing: no_listing
ShellCommandRequirement: {}
InlineJavascriptRequirement: {}
- cwltool:InplaceUpdateRequirement:
+ InplaceUpdateRequirement:
inplaceUpdate: true
arguments:
- shellQuote: false
diff --git a/sdk/cwl/tests/federation/arvbox/mkdir.cwl b/sdk/cwl/tests/federation/arvboxcwl/mkdir.cwl
similarity index 91%
rename from sdk/cwl/tests/federation/arvbox/mkdir.cwl
rename to sdk/cwl/tests/federation/arvboxcwl/mkdir.cwl
index 727d491a3..854a727c6 100644
--- a/sdk/cwl/tests/federation/arvbox/mkdir.cwl
+++ b/sdk/cwl/tests/federation/arvboxcwl/mkdir.cwl
@@ -2,7 +2,7 @@
#
# SPDX-License-Identifier: Apache-2.0
-cwlVersion: v1.0
+cwlVersion: v1.1
class: CommandLineTool
$namespaces:
arv: "http://arvados.org/cwl#"
@@ -37,10 +37,10 @@ requirements:
- entry: $(inputs.arvbox_base)
entryname: base
writable: true
- cwltool:LoadListingRequirement:
+ LoadListingRequirement:
loadListing: no_listing
InlineJavascriptRequirement: {}
- cwltool:InplaceUpdateRequirement:
+ InplaceUpdateRequirement:
inplaceUpdate: true
arguments:
- mkdir
diff --git a/sdk/cwl/tests/federation/arvbox/setup-user.cwl b/sdk/cwl/tests/federation/arvboxcwl/setup-user.cwl
similarity index 87%
rename from sdk/cwl/tests/federation/arvbox/setup-user.cwl
rename to sdk/cwl/tests/federation/arvboxcwl/setup-user.cwl
index a3ad6e575..d8f2d9e12 100644
--- a/sdk/cwl/tests/federation/arvbox/setup-user.cwl
+++ b/sdk/cwl/tests/federation/arvboxcwl/setup-user.cwl
@@ -2,7 +2,7 @@
#
# SPDX-License-Identifier: Apache-2.0
-cwlVersion: v1.0
+cwlVersion: v1.1
class: CommandLineTool
$namespaces:
arv: "http://arvados.org/cwl#"
@@ -13,13 +13,15 @@ requirements:
ARVADOS_API_HOST: $(inputs.container_host)
ARVADOS_API_TOKEN: $(inputs.superuser_token)
ARVADOS_API_HOST_INSECURE: "true"
- cwltool:LoadListingRequirement:
+ LoadListingRequirement:
loadListing: no_listing
InlineJavascriptRequirement: {}
- cwltool:InplaceUpdateRequirement:
+ InplaceUpdateRequirement:
inplaceUpdate: true
DockerRequirement:
dockerPull: arvados/jobs
+ NetworkAccess:
+ networkAccess: true
inputs:
container_host: string
superuser_token: string
diff --git a/sdk/cwl/tests/federation/arvbox/setup_user.py b/sdk/cwl/tests/federation/arvboxcwl/setup_user.py
similarity index 100%
rename from sdk/cwl/tests/federation/arvbox/setup_user.py
rename to sdk/cwl/tests/federation/arvboxcwl/setup_user.py
diff --git a/sdk/cwl/tests/federation/arvbox/start.cwl b/sdk/cwl/tests/federation/arvboxcwl/start.cwl
similarity index 97%
rename from sdk/cwl/tests/federation/arvbox/start.cwl
rename to sdk/cwl/tests/federation/arvboxcwl/start.cwl
index 57b348973..5d9aaba2d 100644
--- a/sdk/cwl/tests/federation/arvbox/start.cwl
+++ b/sdk/cwl/tests/federation/arvboxcwl/start.cwl
@@ -2,7 +2,7 @@
#
# SPDX-License-Identifier: Apache-2.0
-cwlVersion: v1.0
+cwlVersion: v1.1
class: CommandLineTool
$namespaces:
arv: "http://arvados.org/cwl#"
@@ -64,7 +64,7 @@ requirements:
- entry: $(inputs.arvbox_data)
entryname: $(inputs.container_name)
writable: true
- cwltool:InplaceUpdateRequirement:
+ InplaceUpdateRequirement:
inplaceUpdate: true
InlineJavascriptRequirement: {}
arguments:
diff --git a/sdk/cwl/tests/federation/arvbox/stop.cwl b/sdk/cwl/tests/federation/arvboxcwl/stop.cwl
similarity index 100%
rename from sdk/cwl/tests/federation/arvbox/stop.cwl
rename to sdk/cwl/tests/federation/arvboxcwl/stop.cwl
diff --git a/sdk/python/tests/fed-migrate/README b/sdk/python/tests/fed-migrate/README
index 83d659d4d..1591b7e17 100644
--- a/sdk/python/tests/fed-migrate/README
+++ b/sdk/python/tests/fed-migrate/README
@@ -6,7 +6,7 @@ arv-federation-migrate should be in the path or the full path supplied
in the 'fed_migrate' input parameter.
# Create arvbox containers fedbox(1,2,3) for the federation
-$ cwltool --enable-ext arvbox-make-federation.cwl --arvbox_base ~/.arvbox > fed.json
+$ cwltool arvbox-make-federation.cwl --arvbox_base ~/.arvbox > fed.json
# Configure containers and run tests
$ cwltool fed-migrate.cwl fed.json
diff --git a/sdk/python/tests/fed-migrate/arvbox-make-federation.cwl b/sdk/python/tests/fed-migrate/arvbox-make-federation.cwl
index 0aa6f177a..aa859cba4 100644
--- a/sdk/python/tests/fed-migrate/arvbox-make-federation.cwl
+++ b/sdk/python/tests/fed-migrate/arvbox-make-federation.cwl
@@ -1,4 +1,4 @@
-cwlVersion: v1.0
+cwlVersion: v1.1
class: Workflow
$namespaces:
arv: "http://arvados.org/cwl#"
@@ -32,7 +32,7 @@ requirements:
SubworkflowFeatureRequirement: {}
ScatterFeatureRequirement: {}
StepInputExpressionRequirement: {}
- cwltool:LoadListingRequirement:
+ LoadListingRequirement:
loadListing: no_listing
steps:
start:
commit 733215b7022250633485f599ee80f6aa36425343
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Mar 31 17:42:03 2020 -0400
16263: Add bypass_federation test
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/services/api/test/integration/users_test.rb b/services/api/test/integration/users_test.rb
index ee230d514..ccc82a67f 100644
--- a/services/api/test/integration/users_test.rb
+++ b/services/api/test/integration/users_test.rb
@@ -440,5 +440,22 @@ class UsersTest < ActionDispatch::IntegrationTest
assert_match(/Cannot activate without being invited/, json_response['errors'][0])
end
+ test "bypass_federation only accepted for admins" do
+ get "/arvados/v1/users",
+ params: {
+ bypass_federation: true
+ },
+ headers: auth(:admin)
+
+ assert_response :success
+
+ get "/arvados/v1/users",
+ params: {
+ bypass_federation: true
+ },
+ headers: auth(:active)
+
+ assert_response 403
+ end
end
commit 60661ac34a5a3af23f0db792c58401d8c5051ad1
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Mar 31 17:16:57 2020 -0400
16263: Fix no_federation -> bypass_federation in boolParms
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/router/request.go b/lib/controller/router/request.go
index 88763524a..977a15f3a 100644
--- a/lib/controller/router/request.go
+++ b/lib/controller/router/request.go
@@ -169,7 +169,7 @@ var boolParams = map[string]bool{
"include_old_versions": true,
"redirect_to_new_user": true,
"send_notification_email": true,
- "no_federation": true,
+ "bypass_federation": true,
}
func stringToBool(s string) bool {
commit bbf58b8ed64c47900c7204e70fd342db90eb8348
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Mar 31 17:12:31 2020 -0400
16263: Fix only_admin_can_bypass_federation
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/services/api/app/controllers/application_controller.rb b/services/api/app/controllers/application_controller.rb
index a3435d0b6..83a233cd5 100644
--- a/services/api/app/controllers/application_controller.rb
+++ b/services/api/app/controllers/application_controller.rb
@@ -141,7 +141,7 @@ class ApplicationController < ActionController::Base
end
def only_admin_can_bypass_federation
- if params[:bypass_federation] && current_user.nil? or !current_user.is_admin
+ unless !params[:bypass_federation] || current_user.andand.is_admin
send_error("The bypass_federation parameter is only permitted when current user is admin", status: 403)
end
end
commit b79ae856450ab7442954b0454061173f8d3f540c
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Mar 31 16:45:49 2020 -0400
16263: Missed rename
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/federation/list.go b/lib/controller/federation/list.go
index d1d52cfc6..0a596eb9c 100644
--- a/lib/controller/federation/list.go
+++ b/lib/controller/federation/list.go
@@ -107,7 +107,7 @@ func (conn *Conn) generated_CollectionList(ctx context.Context, options arvados.
// backend.
func (conn *Conn) splitListRequest(ctx context.Context, opts arvados.ListOptions, fn func(context.Context, string, arvados.API, arvados.ListOptions) ([]string, error)) error {
- if opts.NoFederation {
+ if opts.BypassFederation {
// Client requested no federation. Pass through.
_, err := fn(ctx, conn.cluster.ClusterID, conn.local, opts)
return err
commit 9574055ff8e6a1a96cc76bdccb139652bc158f27
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Mar 31 16:33:27 2020 -0400
16263: Rename no_federation -> bypass_federation
Enforce if bypass_federation is true that user is admin.
Update API revision and make federation migrate check for it.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/federation/conn.go b/lib/controller/federation/conn.go
index a7c5518d5..8f5c9e0a3 100644
--- a/lib/controller/federation/conn.go
+++ b/lib/controller/federation/conn.go
@@ -448,7 +448,7 @@ func (conn *Conn) batchUpdateUsers(ctx context.Context,
}
func (conn *Conn) UserList(ctx context.Context, options arvados.ListOptions) (arvados.UserList, error) {
- if id := conn.cluster.Login.LoginCluster; id != "" && id != conn.cluster.ClusterID && !options.NoFederation {
+ if id := conn.cluster.Login.LoginCluster; id != "" && id != conn.cluster.ClusterID && !options.BypassFederation {
resp, err := conn.chooseBackend(id).UserList(ctx, options)
if err != nil {
return resp, err
@@ -468,7 +468,7 @@ func (conn *Conn) UserCreate(ctx context.Context, options arvados.CreateOptions)
}
func (conn *Conn) UserUpdate(ctx context.Context, options arvados.UpdateOptions) (arvados.User, error) {
- if options.NoFederation {
+ if options.BypassFederation {
return conn.local.UserUpdate(ctx, options)
}
return conn.chooseBackend(options.UUID).UserUpdate(ctx, options)
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index f3fdd254f..e60108335 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -84,7 +84,7 @@ type ListOptions struct {
Count string `json:"count"`
IncludeTrash bool `json:"include_trash"`
IncludeOldVersions bool `json:"include_old_versions"`
- NoFederation bool `json:"no_federation"`
+ BypassFederation bool `json:"bypass_federation"`
}
type CreateOptions struct {
@@ -95,9 +95,9 @@ type CreateOptions struct {
}
type UpdateOptions struct {
- UUID string `json:"uuid"`
- Attrs map[string]interface{} `json:"attrs"`
- NoFederation bool `json:"no_federation"`
+ UUID string `json:"uuid"`
+ Attrs map[string]interface{} `json:"attrs"`
+ BypassFederation bool `json:"bypass_federation"`
}
type UpdateUUIDOptions struct {
diff --git a/sdk/python/arvados/commands/federation_migrate.py b/sdk/python/arvados/commands/federation_migrate.py
index 0eaf1c03e..445775cce 100755
--- a/sdk/python/arvados/commands/federation_migrate.py
+++ b/sdk/python/arvados/commands/federation_migrate.py
@@ -66,8 +66,8 @@ def connect_clusters(args):
errors.append("Inconsistent login cluster configuration, expected '%s' on %s but was '%s'" % (loginCluster, config["ClusterID"], config["Login"]["LoginCluster"]))
continue
- if arv._rootDesc["revision"] < "20190926":
- errors.append("Arvados API server revision on cluster '%s' is too old, must be updated to at least Arvados 1.5 before running migration." % config["ClusterID"])
+ if arv._rootDesc["revision"] < "20200331":
+ errors.append("Arvados API server revision on cluster '%s' is too old, must be updated to at least Arvados 2.0.2 before running migration." % config["ClusterID"])
continue
try:
@@ -98,7 +98,7 @@ def fetch_users(clusters, loginCluster):
users = []
for c, arv in clusters.items():
print("Getting user list from %s" % c)
- ul = arvados.util.list_all(arv.users().list, no_federation=True)
+ ul = arvados.util.list_all(arv.users().list, bypass_federation=True)
for l in ul:
if l["uuid"].startswith(c):
users.append(l)
@@ -171,14 +171,14 @@ def update_username(args, email, user_uuid, username, migratecluster, migratearv
print("(%s) Updating username of %s to '%s' on %s" % (email, user_uuid, username, migratecluster))
if not args.dry_run:
try:
- conflicts = migratearv.users().list(filters=[["username", "=", username]], no_federation=True).execute()
+ conflicts = migratearv.users().list(filters=[["username", "=", username]], bypass_federation=True).execute()
if conflicts["items"]:
# There's already a user with the username, move the old user out of the way
migratearv.users().update(uuid=conflicts["items"][0]["uuid"],
- no_federation=True,
+ bypass_federation=True,
body={"user": {"username": username+"migrate"}}).execute()
migratearv.users().update(uuid=user_uuid,
- no_federation=True,
+ bypass_federation=True,
body={"user": {"username": username}}).execute()
except arvados.errors.ApiError as e:
print("(%s) Error updating username of %s to '%s' on %s: %s" % (email, user_uuid, username, migratecluster, e))
@@ -210,10 +210,10 @@ def choose_new_user(args, by_email, email, userhome, username, old_user_uuid, cl
try:
olduser = oldhomearv.users().get(uuid=old_user_uuid).execute()
conflicts = homearv.users().list(filters=[["username", "=", username]],
- no_federation=True).execute()
+ bypass_federation=True).execute()
if conflicts["items"]:
homearv.users().update(uuid=conflicts["items"][0]["uuid"],
- no_federation=True,
+ bypass_federation=True,
body={"user": {"username": username+"migrate"}}).execute()
user = homearv.users().create(body={"user": {"email": email, "username": username,
"is_active": olduser["is_active"]}}).execute()
@@ -250,7 +250,7 @@ def activate_remote_user(args, email, homearv, migratearv, old_user_uuid, new_us
return None
try:
- findolduser = migratearv.users().list(filters=[["uuid", "=", old_user_uuid]], no_federation=True).execute()
+ findolduser = migratearv.users().list(filters=[["uuid", "=", old_user_uuid]], bypass_federation=True).execute()
if len(findolduser["items"]) == 0:
return False
if len(findolduser["items"]) == 1:
@@ -280,7 +280,7 @@ def activate_remote_user(args, email, homearv, migratearv, old_user_uuid, new_us
print("(%s) Activating user %s on %s" % (email, new_user_uuid, migratecluster))
try:
if not args.dry_run:
- migratearv.users().update(uuid=new_user_uuid, no_federation=True,
+ migratearv.users().update(uuid=new_user_uuid, bypass_federation=True,
body={"is_active": True}).execute()
except arvados.errors.ApiError as e:
print("(%s) Could not activate user %s on %s: %s" % (email, new_user_uuid, migratecluster, e))
diff --git a/sdk/python/tests/fed-migrate/check.py b/sdk/python/tests/fed-migrate/check.py
index 85d2d31f2..8165b3eba 100644
--- a/sdk/python/tests/fed-migrate/check.py
+++ b/sdk/python/tests/fed-migrate/check.py
@@ -12,7 +12,7 @@ apiC = arvados.api(host=j["arvados_api_hosts"][2], token=j["superuser_tokens"][2
### Check users on API server "A" (the LoginCluster) ###
###
-users = apiA.users().list().execute()
+users = apiA.users().list(bypass_federation=True).execute()
assert len(users["items"]) == 11
@@ -46,7 +46,7 @@ assert found
###
### Check users on API server "B" (federation member) ###
###
-users = apiB.users().list().execute()
+users = apiB.users().list(bypass_federation=True).execute()
assert len(users["items"]) == 11
for i in range(2, 9):
@@ -68,7 +68,7 @@ assert found
###
### Check users on API server "C" (federation member) ###
###
-users = apiC.users().list().execute()
+users = apiC.users().list(bypass_federation=True).execute()
assert len(users["items"]) == 8
for i in (2, 4, 6, 7, 8):
diff --git a/services/api/app/controllers/application_controller.rb b/services/api/app/controllers/application_controller.rb
index 68fa7d880..a3435d0b6 100644
--- a/services/api/app/controllers/application_controller.rb
+++ b/services/api/app/controllers/application_controller.rb
@@ -53,6 +53,7 @@ class ApplicationController < ActionController::Base
before_action :reload_object_before_update, :only => :update
before_action(:render_404_if_no_object,
except: [:index, :create] + ERROR_ACTIONS)
+ before_action :only_admin_can_bypass_federation
attr_writer :resource_attrs
@@ -139,6 +140,12 @@ class ApplicationController < ActionController::Base
render_not_found "Object not found" if !@object
end
+ def only_admin_can_bypass_federation
+ if params[:bypass_federation] && current_user.nil? or !current_user.is_admin
+ send_error("The bypass_federation parameter is only permitted when current user is admin", status: 403)
+ end
+ end
+
def render_error(e)
logger.error e.inspect
if e.respond_to? :backtrace and e.backtrace
@@ -656,7 +663,7 @@ class ApplicationController < ActionController::Base
location: "query",
required: false,
},
- no_federation: {
+ bypass_federation: {
type: 'boolean',
required: false,
description: 'bypass federation behavior, list items from local instance database only'
diff --git a/services/api/app/controllers/arvados/v1/schema_controller.rb b/services/api/app/controllers/arvados/v1/schema_controller.rb
index aee5d1f95..b9aba2726 100644
--- a/services/api/app/controllers/arvados/v1/schema_controller.rb
+++ b/services/api/app/controllers/arvados/v1/schema_controller.rb
@@ -33,10 +33,10 @@ class Arvados::V1::SchemaController < ApplicationController
id: "arvados:v1",
name: "arvados",
version: "v1",
- # format is YYYYMMDD, must be fixed with (needs to be linearly
+ # format is YYYYMMDD, must be fixed width (needs to be lexically
# sortable), updated manually, may be used by clients to
# determine availability of API server features.
- revision: "20190926",
+ revision: "20200331",
source_version: AppVersion.hash,
sourceVersion: AppVersion.hash, # source_version should be deprecated in the future
packageVersion: AppVersion.package_version,
diff --git a/services/api/app/controllers/arvados/v1/users_controller.rb b/services/api/app/controllers/arvados/v1/users_controller.rb
index adb65bd7d..6a5fbbc50 100644
--- a/services/api/app/controllers/arvados/v1/users_controller.rb
+++ b/services/api/app/controllers/arvados/v1/users_controller.rb
@@ -248,7 +248,7 @@ class Arvados::V1::UsersController < ApplicationController
def self._update_requires_parameters
super.merge({
- no_federation: {
+ bypass_federation: {
type: 'boolean', required: false,
},
})
commit b9836aad61698e653fbf7c941c1f818d412b69d2
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Mar 31 14:42:46 2020 -0400
16263: Add no_federation to user update
We might agree on a different API but try this and see if it helps
pass the test.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/federation/conn.go b/lib/controller/federation/conn.go
index 89b335e0d..a7c5518d5 100644
--- a/lib/controller/federation/conn.go
+++ b/lib/controller/federation/conn.go
@@ -468,6 +468,9 @@ func (conn *Conn) UserCreate(ctx context.Context, options arvados.CreateOptions)
}
func (conn *Conn) UserUpdate(ctx context.Context, options arvados.UpdateOptions) (arvados.User, error) {
+ if options.NoFederation {
+ return conn.local.UserUpdate(ctx, options)
+ }
return conn.chooseBackend(options.UUID).UserUpdate(ctx, options)
}
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index a30da6242..f3fdd254f 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -95,8 +95,9 @@ type CreateOptions struct {
}
type UpdateOptions struct {
- UUID string `json:"uuid"`
- Attrs map[string]interface{} `json:"attrs"`
+ UUID string `json:"uuid"`
+ Attrs map[string]interface{} `json:"attrs"`
+ NoFederation bool `json:"no_federation"`
}
type UpdateUUIDOptions struct {
diff --git a/sdk/python/arvados/commands/federation_migrate.py b/sdk/python/arvados/commands/federation_migrate.py
index e1b8ee7d8..0eaf1c03e 100755
--- a/sdk/python/arvados/commands/federation_migrate.py
+++ b/sdk/python/arvados/commands/federation_migrate.py
@@ -173,8 +173,13 @@ def update_username(args, email, user_uuid, username, migratecluster, migratearv
try:
conflicts = migratearv.users().list(filters=[["username", "=", username]], no_federation=True).execute()
if conflicts["items"]:
- migratearv.users().update(uuid=conflicts["items"][0]["uuid"], body={"user": {"username": username+"migrate"}}).execute()
- migratearv.users().update(uuid=user_uuid, body={"user": {"username": username}}).execute()
+ # There's already a user with the username, move the old user out of the way
+ migratearv.users().update(uuid=conflicts["items"][0]["uuid"],
+ no_federation=True,
+ body={"user": {"username": username+"migrate"}}).execute()
+ migratearv.users().update(uuid=user_uuid,
+ no_federation=True,
+ body={"user": {"username": username}}).execute()
except arvados.errors.ApiError as e:
print("(%s) Error updating username of %s to '%s' on %s: %s" % (email, user_uuid, username, migratecluster, e))
@@ -204,10 +209,14 @@ def choose_new_user(args, by_email, email, userhome, username, old_user_uuid, cl
user = None
try:
olduser = oldhomearv.users().get(uuid=old_user_uuid).execute()
- conflicts = homearv.users().list(filters=[["username", "=", username]], no_federation=True).execute()
+ conflicts = homearv.users().list(filters=[["username", "=", username]],
+ no_federation=True).execute()
if conflicts["items"]:
- homearv.users().update(uuid=conflicts["items"][0]["uuid"], body={"user": {"username": username+"migrate"}}).execute()
- user = homearv.users().create(body={"user": {"email": email, "username": username, "is_active": olduser["is_active"]}}).execute()
+ homearv.users().update(uuid=conflicts["items"][0]["uuid"],
+ no_federation=True,
+ body={"user": {"username": username+"migrate"}}).execute()
+ user = homearv.users().create(body={"user": {"email": email, "username": username,
+ "is_active": olduser["is_active"]}}).execute()
except arvados.errors.ApiError as e:
print("(%s) Could not create user: %s" % (email, str(e)))
return None
@@ -259,7 +268,8 @@ def activate_remote_user(args, email, homearv, migratearv, old_user_uuid, new_us
try:
ru = urllib.parse.urlparse(migratearv._rootDesc["rootUrl"])
if not args.dry_run:
- newuser = arvados.api(host=ru.netloc, token=salted, insecure=os.environ.get("ARVADOS_API_HOST_INSECURE")).users().current().execute()
+ newuser = arvados.api(host=ru.netloc, token=salted,
+ insecure=os.environ.get("ARVADOS_API_HOST_INSECURE")).users().current().execute()
else:
newuser = {"is_active": True, "username": username}
except arvados.errors.ApiError as e:
@@ -270,7 +280,8 @@ def activate_remote_user(args, email, homearv, migratearv, old_user_uuid, new_us
print("(%s) Activating user %s on %s" % (email, new_user_uuid, migratecluster))
try:
if not args.dry_run:
- migratearv.users().update(uuid=new_user_uuid, body={"is_active": True}).execute()
+ migratearv.users().update(uuid=new_user_uuid, no_federation=True,
+ body={"is_active": True}).execute()
except arvados.errors.ApiError as e:
print("(%s) Could not activate user %s on %s: %s" % (email, new_user_uuid, migratecluster, e))
return None
diff --git a/services/api/app/controllers/arvados/v1/users_controller.rb b/services/api/app/controllers/arvados/v1/users_controller.rb
index eb2402b80..adb65bd7d 100644
--- a/services/api/app/controllers/arvados/v1/users_controller.rb
+++ b/services/api/app/controllers/arvados/v1/users_controller.rb
@@ -246,6 +246,14 @@ class Arvados::V1::UsersController < ApplicationController
}
end
+ def self._update_requires_parameters
+ super.merge({
+ no_federation: {
+ type: 'boolean', required: false,
+ },
+ })
+ end
+
def self._update_uuid_requires_parameters
{
new_uuid: {
commit 31943856d8150a70c3262d70f818af700ed2f177
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Mar 30 22:30:10 2020 -0400
16263: local_user_list -> no_federation in boolParams
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/router/request.go b/lib/controller/router/request.go
index d769c5006..88763524a 100644
--- a/lib/controller/router/request.go
+++ b/lib/controller/router/request.go
@@ -169,7 +169,7 @@ var boolParams = map[string]bool{
"include_old_versions": true,
"redirect_to_new_user": true,
"send_notification_email": true,
- "local_user_list": true,
+ "no_federation": true,
}
func stringToBool(s string) bool {
commit a9546ab1a73665896dc4d6c6ad0ea5da3c5eaa66
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Mar 30 22:14:23 2020 -0400
16263: Generalize "local_user_list" flag to "no_federation"
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/federation/conn.go b/lib/controller/federation/conn.go
index 21c156088..89b335e0d 100644
--- a/lib/controller/federation/conn.go
+++ b/lib/controller/federation/conn.go
@@ -386,64 +386,76 @@ var userAttrsCachedFromLoginCluster = map[string]bool{
"writable_by": false,
}
-func (conn *Conn) UserList(ctx context.Context, options arvados.ListOptions) (arvados.UserList, error) {
+func (conn *Conn) batchUpdateUsers(ctx context.Context,
+ options arvados.ListOptions,
+ items []arvados.User) (err error) {
+
+ id := conn.cluster.Login.LoginCluster
logger := ctxlog.FromContext(ctx)
- if id := conn.cluster.Login.LoginCluster; id != "" && id != conn.cluster.ClusterID && !options.LocalUserList {
- resp, err := conn.chooseBackend(id).UserList(ctx, options)
- if err != nil {
- return resp, err
+ batchOpts := arvados.UserBatchUpdateOptions{Updates: map[string]map[string]interface{}{}}
+ for _, user := range items {
+ if !strings.HasPrefix(user.UUID, id) {
+ continue
+ }
+ logger.Debugf("cache user info for uuid %q", user.UUID)
+
+ // If the remote cluster has null timestamps
+ // (e.g., test server with incomplete
+ // fixtures) use dummy timestamps (instead of
+ // the zero time, which causes a Rails API
+ // error "year too big to marshal: 1 UTC").
+ if user.ModifiedAt.IsZero() {
+ user.ModifiedAt = time.Now()
+ }
+ if user.CreatedAt.IsZero() {
+ user.CreatedAt = time.Now()
}
- batchOpts := arvados.UserBatchUpdateOptions{Updates: map[string]map[string]interface{}{}}
- for _, user := range resp.Items {
- if !strings.HasPrefix(user.UUID, id) {
- continue
- }
- logger.Debugf("cache user info for uuid %q", user.UUID)
-
- // If the remote cluster has null timestamps
- // (e.g., test server with incomplete
- // fixtures) use dummy timestamps (instead of
- // the zero time, which causes a Rails API
- // error "year too big to marshal: 1 UTC").
- if user.ModifiedAt.IsZero() {
- user.ModifiedAt = time.Now()
- }
- if user.CreatedAt.IsZero() {
- user.CreatedAt = time.Now()
- }
- var allFields map[string]interface{}
- buf, err := json.Marshal(user)
- if err != nil {
- return arvados.UserList{}, fmt.Errorf("error encoding user record from remote response: %s", err)
- }
- err = json.Unmarshal(buf, &allFields)
- if err != nil {
- return arvados.UserList{}, fmt.Errorf("error transcoding user record from remote response: %s", err)
- }
- updates := allFields
- if len(options.Select) > 0 {
- updates = map[string]interface{}{}
- for _, k := range options.Select {
- if v, ok := allFields[k]; ok && userAttrsCachedFromLoginCluster[k] {
- updates[k] = v
- }
+ var allFields map[string]interface{}
+ buf, err := json.Marshal(user)
+ if err != nil {
+ return fmt.Errorf("error encoding user record from remote response: %s", err)
+ }
+ err = json.Unmarshal(buf, &allFields)
+ if err != nil {
+ return fmt.Errorf("error transcoding user record from remote response: %s", err)
+ }
+ updates := allFields
+ if len(options.Select) > 0 {
+ updates = map[string]interface{}{}
+ for _, k := range options.Select {
+ if v, ok := allFields[k]; ok && userAttrsCachedFromLoginCluster[k] {
+ updates[k] = v
}
- } else {
- for k := range updates {
- if !userAttrsCachedFromLoginCluster[k] {
- delete(updates, k)
- }
+ }
+ } else {
+ for k := range updates {
+ if !userAttrsCachedFromLoginCluster[k] {
+ delete(updates, k)
}
}
- batchOpts.Updates[user.UUID] = updates
}
- if len(batchOpts.Updates) > 0 {
- ctxRoot := auth.NewContext(ctx, &auth.Credentials{Tokens: []string{conn.cluster.SystemRootToken}})
- _, err = conn.local.UserBatchUpdate(ctxRoot, batchOpts)
- if err != nil {
- return arvados.UserList{}, fmt.Errorf("error updating local user records: %s", err)
- }
+ batchOpts.Updates[user.UUID] = updates
+ }
+ if len(batchOpts.Updates) > 0 {
+ ctxRoot := auth.NewContext(ctx, &auth.Credentials{Tokens: []string{conn.cluster.SystemRootToken}})
+ _, err = conn.local.UserBatchUpdate(ctxRoot, batchOpts)
+ if err != nil {
+ return fmt.Errorf("error updating local user records: %s", err)
+ }
+ }
+ return nil
+}
+
+func (conn *Conn) UserList(ctx context.Context, options arvados.ListOptions) (arvados.UserList, error) {
+ if id := conn.cluster.Login.LoginCluster; id != "" && id != conn.cluster.ClusterID && !options.NoFederation {
+ resp, err := conn.chooseBackend(id).UserList(ctx, options)
+ if err != nil {
+ return resp, err
+ }
+ err = conn.batchUpdateUsers(ctx, options, resp.Items)
+ if err != nil {
+ return arvados.UserList{}, err
}
return resp, nil
} else {
@@ -460,7 +472,7 @@ func (conn *Conn) UserUpdate(ctx context.Context, options arvados.UpdateOptions)
}
func (conn *Conn) UserUpdateUUID(ctx context.Context, options arvados.UpdateUUIDOptions) (arvados.User, error) {
- return conn.chooseBackend(options.UUID).UserUpdateUUID(ctx, options)
+ return conn.local.UserUpdateUUID(ctx, options)
}
func (conn *Conn) UserMerge(ctx context.Context, options arvados.UserMergeOptions) (arvados.User, error) {
diff --git a/lib/controller/federation/list.go b/lib/controller/federation/list.go
index 6ee813317..d1d52cfc6 100644
--- a/lib/controller/federation/list.go
+++ b/lib/controller/federation/list.go
@@ -106,6 +106,13 @@ func (conn *Conn) generated_CollectionList(ctx context.Context, options arvados.
// corresponding options argument suitable for sending to that
// backend.
func (conn *Conn) splitListRequest(ctx context.Context, opts arvados.ListOptions, fn func(context.Context, string, arvados.API, arvados.ListOptions) ([]string, error)) error {
+
+ if opts.NoFederation {
+ // Client requested no federation. Pass through.
+ _, err := fn(ctx, conn.cluster.ClusterID, conn.local, opts)
+ return err
+ }
+
cannotSplit := false
var matchAllFilters map[string]bool
for _, f := range opts.Filters {
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index 3b20955a1..a30da6242 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -84,7 +84,7 @@ type ListOptions struct {
Count string `json:"count"`
IncludeTrash bool `json:"include_trash"`
IncludeOldVersions bool `json:"include_old_versions"`
- LocalUserList bool `json:"local_user_list"`
+ NoFederation bool `json:"no_federation"`
}
type CreateOptions struct {
diff --git a/sdk/python/arvados/commands/federation_migrate.py b/sdk/python/arvados/commands/federation_migrate.py
index 344390b48..e1b8ee7d8 100755
--- a/sdk/python/arvados/commands/federation_migrate.py
+++ b/sdk/python/arvados/commands/federation_migrate.py
@@ -98,7 +98,7 @@ def fetch_users(clusters, loginCluster):
users = []
for c, arv in clusters.items():
print("Getting user list from %s" % c)
- ul = arvados.util.list_all(arv.users().list, local_user_list=True)
+ ul = arvados.util.list_all(arv.users().list, no_federation=True)
for l in ul:
if l["uuid"].startswith(c):
users.append(l)
@@ -171,7 +171,7 @@ def update_username(args, email, user_uuid, username, migratecluster, migratearv
print("(%s) Updating username of %s to '%s' on %s" % (email, user_uuid, username, migratecluster))
if not args.dry_run:
try:
- conflicts = migratearv.users().list(filters=[["username", "=", username]], local_user_list=True).execute()
+ conflicts = migratearv.users().list(filters=[["username", "=", username]], no_federation=True).execute()
if conflicts["items"]:
migratearv.users().update(uuid=conflicts["items"][0]["uuid"], body={"user": {"username": username+"migrate"}}).execute()
migratearv.users().update(uuid=user_uuid, body={"user": {"username": username}}).execute()
@@ -204,7 +204,7 @@ def choose_new_user(args, by_email, email, userhome, username, old_user_uuid, cl
user = None
try:
olduser = oldhomearv.users().get(uuid=old_user_uuid).execute()
- conflicts = homearv.users().list(filters=[["username", "=", username]], local_user_list=True).execute()
+ conflicts = homearv.users().list(filters=[["username", "=", username]], no_federation=True).execute()
if conflicts["items"]:
homearv.users().update(uuid=conflicts["items"][0]["uuid"], body={"user": {"username": username+"migrate"}}).execute()
user = homearv.users().create(body={"user": {"email": email, "username": username, "is_active": olduser["is_active"]}}).execute()
@@ -241,10 +241,16 @@ def activate_remote_user(args, email, homearv, migratearv, old_user_uuid, new_us
return None
try:
- olduser = migratearv.users().get(uuid=old_user_uuid).execute()
+ findolduser = migratearv.users().list(filters=[["uuid", "=", old_user_uuid]], no_federation=True).execute()
+ if len(findolduser["items"]) == 0:
+ return False
+ if len(findolduser["items"]) == 1:
+ olduser = findolduser["items"][0]
+ else:
+ print("(%s) Unexpected result" % (email))
+ return None
except arvados.errors.ApiError as e:
- if e.resp.status != 404:
- print("(%s) Could not retrieve user %s from %s, user may have already been migrated: %s" % (email, old_user_uuid, migratecluster, e))
+ print("(%s) Could not retrieve user %s from %s, user may have already been migrated: %s" % (email, old_user_uuid, migratecluster, e))
return None
salted = 'v2/' + newtok["uuid"] + '/' + hmac.new(newtok["api_token"].encode(),
@@ -351,10 +357,10 @@ def main():
if new_user_uuid is None:
continue
- # cluster where the migration is happening
remote_users = {}
got_error = False
for migratecluster in clusters:
+ # cluster where the migration is happening
migratearv = clusters[migratecluster]
# the user's new home cluster
@@ -370,6 +376,8 @@ def main():
for migratecluster in clusters:
migratearv = clusters[migratecluster]
newuser = remote_users[migratecluster]
+ if newuser is False:
+ continue
print("(%s) Migrating %s to %s on %s" % (email, old_user_uuid, new_user_uuid, migratecluster))
diff --git a/services/api/app/controllers/application_controller.rb b/services/api/app/controllers/application_controller.rb
index 7b82cdb61..68fa7d880 100644
--- a/services/api/app/controllers/application_controller.rb
+++ b/services/api/app/controllers/application_controller.rb
@@ -656,6 +656,11 @@ class ApplicationController < ActionController::Base
location: "query",
required: false,
},
+ no_federation: {
+ type: 'boolean',
+ required: false,
+ description: 'bypass federation behavior, list items from local instance database only'
+ }
}
end
diff --git a/services/api/app/controllers/arvados/v1/users_controller.rb b/services/api/app/controllers/arvados/v1/users_controller.rb
index c94fa113e..eb2402b80 100644
--- a/services/api/app/controllers/arvados/v1/users_controller.rb
+++ b/services/api/app/controllers/arvados/v1/users_controller.rb
@@ -254,12 +254,6 @@ class Arvados::V1::UsersController < ApplicationController
}
end
- def self._index_requires_parameters
- super.merge(
- { local_user_list: {required: false, type: 'boolean',
- description: 'only list users from local database, no effect if LoginCluster is not set'} })
- end
-
def apply_filters(model_class=nil)
return super if @read_users.any?(&:is_admin)
if params[:uuid] != current_user.andand.uuid
commit 7a33961d2067a625d240a600318dd008eebe7361
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Mar 30 17:12:17 2020 -0400
16263: UserMerge shouldn't be federated
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/federation/conn.go b/lib/controller/federation/conn.go
index dc676c263..21c156088 100644
--- a/lib/controller/federation/conn.go
+++ b/lib/controller/federation/conn.go
@@ -464,7 +464,7 @@ func (conn *Conn) UserUpdateUUID(ctx context.Context, options arvados.UpdateUUID
}
func (conn *Conn) UserMerge(ctx context.Context, options arvados.UserMergeOptions) (arvados.User, error) {
- return conn.chooseBackend(options.OldUserUUID).UserMerge(ctx, options)
+ return conn.local.UserMerge(ctx, options)
}
func (conn *Conn) UserActivate(ctx context.Context, options arvados.UserActivateOptions) (arvados.User, error) {
commit c65d4fe115b500c6b248a1a8faa50eca3034a91e
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Mar 30 14:50:06 2020 -0400
16263: Fix omitempty placement
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index d5f5dcaf8..3b20955a1 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -60,11 +60,11 @@ var (
)
type GetOptions struct {
- UUID string `json:"uuid",omitempty`
+ UUID string `json:"uuid,omitempty"`
Select []string `json:"select"`
IncludeTrash bool `json:"include_trash"`
- ForwardedFor string `json:"forwarded_for",omitempty`
- Remote string `json:"remote",omitempty`
+ ForwardedFor string `json:"forwarded_for,omitempty"`
+ Remote string `json:"remote,omitempty"`
}
type UntrashOptions struct {
commit ede1354bb84dc5687d6cc588f16121165636de88
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Mar 30 14:43:36 2020 -0400
16263: Add omitempty to GetOptions
Don't try to migrate if activate_remote_user fails.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index 7bb3bfaa7..d5f5dcaf8 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -60,11 +60,11 @@ var (
)
type GetOptions struct {
- UUID string `json:"uuid"`
+ UUID string `json:"uuid",omitempty`
Select []string `json:"select"`
IncludeTrash bool `json:"include_trash"`
- ForwardedFor string `json:"forwarded_for"`
- Remote string `json:"remote"`
+ ForwardedFor string `json:"forwarded_for",omitempty`
+ Remote string `json:"remote",omitempty`
}
type UntrashOptions struct {
diff --git a/sdk/python/arvados/commands/federation_migrate.py b/sdk/python/arvados/commands/federation_migrate.py
index b66aa5d7c..344390b48 100755
--- a/sdk/python/arvados/commands/federation_migrate.py
+++ b/sdk/python/arvados/commands/federation_migrate.py
@@ -352,6 +352,8 @@ def main():
continue
# cluster where the migration is happening
+ remote_users = {}
+ got_error = False
for migratecluster in clusters:
migratearv = clusters[migratecluster]
@@ -361,14 +363,20 @@ def main():
newuser = activate_remote_user(args, email, homearv, migratearv, old_user_uuid, new_user_uuid)
if newuser is None:
- continue
+ got_error = True
+ remote_users[migratecluster] = newuser
+
+ if not got_error:
+ for migratecluster in clusters:
+ migratearv = clusters[migratecluster]
+ newuser = remote_users[migratecluster]
- print("(%s) Migrating %s to %s on %s" % (email, old_user_uuid, new_user_uuid, migratecluster))
+ print("(%s) Migrating %s to %s on %s" % (email, old_user_uuid, new_user_uuid, migratecluster))
- migrate_user(args, migratearv, email, new_user_uuid, old_user_uuid)
+ migrate_user(args, migratearv, email, new_user_uuid, old_user_uuid)
- if newuser['username'] != username:
- update_username(args, email, new_user_uuid, username, migratecluster, migratearv)
+ if newuser['username'] != username:
+ update_username(args, email, new_user_uuid, username, migratecluster, migratearv)
if __name__ == "__main__":
main()
commit bf76fe28bf456e93e29b77804f9ba59539323235
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Mar 27 21:36:09 2020 -0400
16263: Add local_user_list to boolParams
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/router/request.go b/lib/controller/router/request.go
index 39b4c5100..d769c5006 100644
--- a/lib/controller/router/request.go
+++ b/lib/controller/router/request.go
@@ -169,6 +169,7 @@ var boolParams = map[string]bool{
"include_old_versions": true,
"redirect_to_new_user": true,
"send_notification_email": true,
+ "local_user_list": true,
}
func stringToBool(s string) bool {
commit a19837776bf215aa62cd53187cb60064c6205679
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Mar 27 17:08:48 2020 -0400
16263: Add local_user_list flag to bypass LoginCluster behavior
Required by federation migrate script.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/lib/controller/federation/conn.go b/lib/controller/federation/conn.go
index 226bfabf1..dc676c263 100644
--- a/lib/controller/federation/conn.go
+++ b/lib/controller/federation/conn.go
@@ -388,7 +388,7 @@ var userAttrsCachedFromLoginCluster = map[string]bool{
func (conn *Conn) UserList(ctx context.Context, options arvados.ListOptions) (arvados.UserList, error) {
logger := ctxlog.FromContext(ctx)
- if id := conn.cluster.Login.LoginCluster; id != "" && id != conn.cluster.ClusterID {
+ if id := conn.cluster.Login.LoginCluster; id != "" && id != conn.cluster.ClusterID && !options.LocalUserList {
resp, err := conn.chooseBackend(id).UserList(ctx, options)
if err != nil {
return resp, err
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index 4eb5b61b3..7bb3bfaa7 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -84,6 +84,7 @@ type ListOptions struct {
Count string `json:"count"`
IncludeTrash bool `json:"include_trash"`
IncludeOldVersions bool `json:"include_old_versions"`
+ LocalUserList bool `json:"local_user_list"`
}
type CreateOptions struct {
diff --git a/sdk/python/arvados/commands/federation_migrate.py b/sdk/python/arvados/commands/federation_migrate.py
index e74d6215c..b66aa5d7c 100755
--- a/sdk/python/arvados/commands/federation_migrate.py
+++ b/sdk/python/arvados/commands/federation_migrate.py
@@ -98,7 +98,7 @@ def fetch_users(clusters, loginCluster):
users = []
for c, arv in clusters.items():
print("Getting user list from %s" % c)
- ul = arvados.util.list_all(arv.users().list)
+ ul = arvados.util.list_all(arv.users().list, local_user_list=True)
for l in ul:
if l["uuid"].startswith(c):
users.append(l)
@@ -171,7 +171,7 @@ def update_username(args, email, user_uuid, username, migratecluster, migratearv
print("(%s) Updating username of %s to '%s' on %s" % (email, user_uuid, username, migratecluster))
if not args.dry_run:
try:
- conflicts = migratearv.users().list(filters=[["username", "=", username]]).execute()
+ conflicts = migratearv.users().list(filters=[["username", "=", username]], local_user_list=True).execute()
if conflicts["items"]:
migratearv.users().update(uuid=conflicts["items"][0]["uuid"], body={"user": {"username": username+"migrate"}}).execute()
migratearv.users().update(uuid=user_uuid, body={"user": {"username": username}}).execute()
@@ -204,7 +204,7 @@ def choose_new_user(args, by_email, email, userhome, username, old_user_uuid, cl
user = None
try:
olduser = oldhomearv.users().get(uuid=old_user_uuid).execute()
- conflicts = homearv.users().list(filters=[["username", "=", username]]).execute()
+ conflicts = homearv.users().list(filters=[["username", "=", username]], local_user_list=True).execute()
if conflicts["items"]:
homearv.users().update(uuid=conflicts["items"][0]["uuid"], body={"user": {"username": username+"migrate"}}).execute()
user = homearv.users().create(body={"user": {"email": email, "username": username, "is_active": olduser["is_active"]}}).execute()
diff --git a/services/api/app/controllers/arvados/v1/users_controller.rb b/services/api/app/controllers/arvados/v1/users_controller.rb
index eb2402b80..c94fa113e 100644
--- a/services/api/app/controllers/arvados/v1/users_controller.rb
+++ b/services/api/app/controllers/arvados/v1/users_controller.rb
@@ -254,6 +254,12 @@ class Arvados::V1::UsersController < ApplicationController
}
end
+ def self._index_requires_parameters
+ super.merge(
+ { local_user_list: {required: false, type: 'boolean',
+ description: 'only list users from local database, no effect if LoginCluster is not set'} })
+ end
+
def apply_filters(model_class=nil)
return super if @read_users.any?(&:is_admin)
if params[:uuid] != current_user.andand.uuid
commit 5862a51baa10297165a3c49098f312f3e95a828e
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Thu Mar 26 11:46:34 2020 -0300
16263: Decodes JSON numbers as strings instead of float64.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/lib/controller/rpc/conn.go b/lib/controller/rpc/conn.go
index 4b143b770..b5c56dbc4 100644
--- a/lib/controller/rpc/conn.go
+++ b/lib/controller/rpc/conn.go
@@ -5,6 +5,7 @@
package rpc
import (
+ "bytes"
"context"
"crypto/tls"
"encoding/json"
@@ -14,6 +15,7 @@ import (
"net"
"net/http"
"net/url"
+ "strconv"
"strings"
"time"
@@ -100,19 +102,23 @@ func (conn *Conn) requestAndDecode(ctx context.Context, dst interface{}, ep arva
return fmt.Errorf("%T: requestAndDecode: Marshal opts: %s", conn, err)
}
var params map[string]interface{}
- err = json.Unmarshal(j, ¶ms)
+ dec := json.NewDecoder(bytes.NewBuffer(j))
+ dec.UseNumber()
+ err = dec.Decode(¶ms)
if err != nil {
- return fmt.Errorf("%T: requestAndDecode: Unmarshal opts: %s", conn, err)
+ return fmt.Errorf("%T: requestAndDecode: Decode opts: %s", conn, err)
}
if attrs, ok := params["attrs"]; ok && ep.AttrsKey != "" {
params[ep.AttrsKey] = attrs
delete(params, "attrs")
}
- if limit, ok := params["limit"].(float64); ok && limit < 0 {
- // Negative limit means "not specified" here, but some
- // servers/versions do not accept that, so we need to
- // remove it entirely.
- delete(params, "limit")
+ if limitStr, ok := params["limit"]; ok {
+ if limit, err := strconv.ParseInt(string(limitStr.(json.Number)), 10, 64); err == nil && limit < 0 {
+ // Negative limit means "not specified" here, but some
+ // servers/versions do not accept that, so we need to
+ // remove it entirely.
+ delete(params, "limit")
+ }
}
if len(tokens) > 1 {
params["reader_tokens"] = tokens[1:]
commit 0ed3f6e33bdc9d8debfd94f2cba8f388539aad32
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Wed Mar 25 16:35:18 2020 -0300
16263: Adds unit test case confirming 'limit' bug.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/lib/controller/federation/list_test.go b/lib/controller/federation/list_test.go
index ce84378a3..e6d2816f6 100644
--- a/lib/controller/federation/list_test.go
+++ b/lib/controller/federation/list_test.go
@@ -58,7 +58,7 @@ func (cl *collectionLister) CollectionList(ctx context.Context, options arvados.
if cl.MaxPageSize > 0 && len(resp.Items) >= cl.MaxPageSize {
break
}
- if options.Limit >= 0 && len(resp.Items) >= options.Limit {
+ if options.Limit >= 0 && int64(len(resp.Items)) >= options.Limit {
break
}
if cl.matchFilters(c, options.Filters) {
@@ -115,8 +115,8 @@ func (s *CollectionListSuite) SetUpTest(c *check.C) {
type listTrial struct {
count string
- limit int
- offset int
+ limit int64
+ offset int64
order []string
filters []arvados.Filter
selectfields []string
@@ -314,7 +314,7 @@ func (s *CollectionListSuite) TestCollectionListMultiSiteWithCount(c *check.C) {
}
func (s *CollectionListSuite) TestCollectionListMultiSiteWithLimit(c *check.C) {
- for _, limit := range []int{0, 1, 2} {
+ for _, limit := range []int64{0, 1, 2} {
s.test(c, listTrial{
count: "none",
limit: limit,
diff --git a/lib/controller/federation/user_test.go b/lib/controller/federation/user_test.go
index c087273af..c55ec24d4 100644
--- a/lib/controller/federation/user_test.go
+++ b/lib/controller/federation/user_test.go
@@ -7,6 +7,7 @@ package federation
import (
"encoding/json"
"errors"
+ "math"
"net/url"
"os"
"strings"
@@ -32,6 +33,7 @@ func (s *UserSuite) TestLoginClusterUserList(c *check.C) {
for _, updateFail := range []bool{false, true} {
for _, opts := range []arvados.ListOptions{
{Offset: 0, Limit: -1, Select: nil},
+ {Offset: 0, Limit: math.MaxInt64, Select: nil},
{Offset: 1, Limit: 1, Select: nil},
{Offset: 0, Limit: 2, Select: []string{"uuid"}},
{Offset: 0, Limit: 2, Select: []string{"uuid", "email"}},
@@ -45,6 +47,9 @@ func (s *UserSuite) TestLoginClusterUserList(c *check.C) {
s.fed.local = rpc.NewConn(s.cluster.ClusterID, spy.URL, true, rpc.PassthroughTokenProvider)
}
userlist, err := s.fed.UserList(s.ctx, opts)
+ if err != nil {
+ c.Logf("... UserList failed %q", err)
+ }
if updateFail && err == nil {
// All local updates fail, so the only
// cases expected to succeed are the
commit 5efd7f8a01d6efc4480e2d6f460489a890ae913c
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Wed Mar 25 11:26:55 2020 -0300
16263: Adds test exposing a bug when using 'limit' with the max int64 value.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index 0c5d32e8b..4eb5b61b3 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -77,8 +77,8 @@ type ListOptions struct {
Select []string `json:"select"`
Filters []Filter `json:"filters"`
Where map[string]interface{} `json:"where"`
- Limit int `json:"limit"`
- Offset int `json:"offset"`
+ Limit int64 `json:"limit"`
+ Offset int64 `json:"offset"`
Order []string `json:"order"`
Distinct bool `json:"distinct"`
Count string `json:"count"`
commit 2cc2d6440dd0be3557af76f1670192268d87f82b
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Tue Mar 24 17:39:01 2020 -0300
16263: Bool filter bug fix.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/sdk/go/arvados/resource_list.go b/sdk/go/arvados/resource_list.go
index d1a25c438..a5cc7d3b9 100644
--- a/sdk/go/arvados/resource_list.go
+++ b/sdk/go/arvados/resource_list.go
@@ -55,7 +55,7 @@ func (f *Filter) UnmarshalJSON(data []byte) error {
}
operand := elements[2]
switch operand.(type) {
- case string, float64, []interface{}, nil:
+ case string, float64, []interface{}, nil, bool:
default:
return fmt.Errorf("invalid filter operand %q", elements[2])
}
commit 38db937185cba295ecffe0f4d78a65fab6ea686c
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Tue Mar 24 18:03:34 2020 -0300
16263: Adds test exposing a bug on filter unmarshalling.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/sdk/go/arvados/resource_list_test.go b/sdk/go/arvados/resource_list_test.go
index 4e09c5375..b36e82c91 100644
--- a/sdk/go/arvados/resource_list_test.go
+++ b/sdk/go/arvados/resource_list_test.go
@@ -34,3 +34,40 @@ func TestMarshalFiltersWithNil(t *testing.T) {
t.Errorf("Encoded as %q, expected %q", buf, expect)
}
}
+
+func TestUnmarshalFiltersWithNil(t *testing.T) {
+ buf := []byte(`["modified_at","=",null]`)
+ f := &Filter{}
+ err := f.UnmarshalJSON(buf)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expect := Filter{Attr: "modified_at", Operator: "=", Operand: nil}
+ if f.Attr != expect.Attr || f.Operator != expect.Operator || f.Operand != expect.Operand {
+ t.Errorf("Decoded as %q, expected %q", f, expect)
+ }
+}
+
+func TestMarshalFiltersWithBoolean(t *testing.T) {
+ buf, err := json.Marshal([]Filter{
+ {Attr: "is_active", Operator: "=", Operand: true}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if expect := []byte(`[["is_active","=",true]]`); 0 != bytes.Compare(buf, expect) {
+ t.Errorf("Encoded as %q, expected %q", buf, expect)
+ }
+}
+
+func TestUnmarshalFiltersWithBoolean(t *testing.T) {
+ buf := []byte(`["is_active","=",true]`)
+ f := &Filter{}
+ err := f.UnmarshalJSON(buf)
+ if err != nil {
+ t.Fatal(err)
+ }
+ expect := Filter{Attr: "is_active", Operator: "=", Operand: true}
+ if f.Attr != expect.Attr || f.Operator != expect.Operator || f.Operand != expect.Operand {
+ t.Errorf("Decoded as %q, expected %q", f, expect)
+ }
+}
commit d97308ee0db8ad22f8eb03de75ce55cf7e228afd
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Mon Mar 23 16:20:41 2020 -0300
16263: Makes gofmt happy.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/lib/controller/federation/conn.go b/lib/controller/federation/conn.go
index 42a1a5727..226bfabf1 100644
--- a/lib/controller/federation/conn.go
+++ b/lib/controller/federation/conn.go
@@ -365,15 +365,15 @@ func (conn *Conn) SpecimenDelete(ctx context.Context, options arvados.DeleteOpti
}
var userAttrsCachedFromLoginCluster = map[string]bool{
- "created_at": true,
- "email": true,
- "first_name": true,
- "is_active": true,
- "is_admin": true,
- "last_name": true,
- "modified_at": true,
- "prefs": true,
- "username": true,
+ "created_at": true,
+ "email": true,
+ "first_name": true,
+ "is_active": true,
+ "is_admin": true,
+ "last_name": true,
+ "modified_at": true,
+ "prefs": true,
+ "username": true,
"etag": false,
"full_name": false,
commit 01baf16387609f936e45940561c0c289046a39d0
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Mon Mar 23 15:32:59 2020 -0300
16263: Don't cache modified_by_*_uuid fields when using LoginCluster.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/lib/controller/federation/conn.go b/lib/controller/federation/conn.go
index 56f117ee7..42a1a5727 100644
--- a/lib/controller/federation/conn.go
+++ b/lib/controller/federation/conn.go
@@ -372,18 +372,18 @@ var userAttrsCachedFromLoginCluster = map[string]bool{
"is_admin": true,
"last_name": true,
"modified_at": true,
- "modified_by_client_uuid": true,
- "modified_by_user_uuid": true,
"prefs": true,
"username": true,
- "etag": false,
- "full_name": false,
- "identity_url": false,
- "is_invited": false,
- "owner_uuid": false,
- "uuid": false,
- "writable_by": false,
+ "etag": false,
+ "full_name": false,
+ "identity_url": false,
+ "is_invited": false,
+ "modified_by_client_uuid": false,
+ "modified_by_user_uuid": false,
+ "owner_uuid": false,
+ "uuid": false,
+ "writable_by": false,
}
func (conn *Conn) UserList(ctx context.Context, options arvados.ListOptions) (arvados.UserList, error) {
commit 29486979fc8a48788f5c12fec399d8d41588cba0
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Mon Mar 23 12:27:18 2020 -0300
16263: Adds nullify behavior to users's batch_update endpoint.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/services/api/app/controllers/application_controller.rb b/services/api/app/controllers/application_controller.rb
index fbf177b01..7b82cdb61 100644
--- a/services/api/app/controllers/application_controller.rb
+++ b/services/api/app/controllers/application_controller.rb
@@ -486,12 +486,20 @@ class ApplicationController < ActionController::Base
# Go code may send empty values (ie: empty string instead of NULL) that
# should be translated to NULL on the database.
def set_nullable_attrs_to_null
- (resource_attrs.keys & nullable_attributes).each do |attr|
- val = resource_attrs[attr]
+ nullify_attrs(resource_attrs.to_hash).each do |k, v|
+ resource_attrs[k] = v
+ end
+ end
+
+ def nullify_attrs(a = {})
+ new_attrs = a.to_hash.symbolize_keys
+ (new_attrs.keys & nullable_attributes).each do |attr|
+ val = new_attrs[attr]
if (val.class == Integer && val == 0) || (val.class == String && val == "")
- resource_attrs[attr] = nil
+ new_attrs[attr] = nil
end
end
+ return new_attrs
end
def reload_object_before_update
diff --git a/services/api/app/controllers/arvados/v1/users_controller.rb b/services/api/app/controllers/arvados/v1/users_controller.rb
index fecc0620c..eb2402b80 100644
--- a/services/api/app/controllers/arvados/v1/users_controller.rb
+++ b/services/api/app/controllers/arvados/v1/users_controller.rb
@@ -22,7 +22,7 @@ class Arvados::V1::UsersController < ApplicationController
rescue ActiveRecord::RecordNotUnique
retry
end
- u.update_attributes!(attrs)
+ u.update_attributes!(nullify_attrs(attrs))
@objects << u
end
@offset = 0
diff --git a/services/api/test/functional/arvados/v1/users_controller_test.rb b/services/api/test/functional/arvados/v1/users_controller_test.rb
index b38f0d52f..817a1c9ef 100644
--- a/services/api/test/functional/arvados/v1/users_controller_test.rb
+++ b/services/api/test/functional/arvados/v1/users_controller_test.rb
@@ -1057,6 +1057,7 @@ class Arvados::V1::UsersControllerTest < ActionController::TestCase
newuuid => {
'first_name' => 'noot',
'email' => 'root at remot.example.com',
+ 'username' => '',
},
}})
assert_response(:success)
commit 4fbb5b4751488d9751710f03ebf99406d4c55410
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Thu Mar 19 18:15:52 2020 -0300
16263: Assigns nil to select attributes when receiving empty values.
Controller may translate NULL values to "" on certain object string fields.
The same with integers, NULLs are converted to 0.
When controller retrieves objects from railsAPI and uses the data to create
or update objects, some of those fields should get converted back to NULL.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/services/api/app/controllers/application_controller.rb b/services/api/app/controllers/application_controller.rb
index 369043e78..fbf177b01 100644
--- a/services/api/app/controllers/application_controller.rb
+++ b/services/api/app/controllers/application_controller.rb
@@ -45,6 +45,7 @@ class ApplicationController < ActionController::Base
before_action :load_required_parameters
before_action(:find_object_by_uuid,
except: [:index, :create] + ERROR_ACTIONS)
+ before_action(:set_nullable_attrs_to_null, only: [:update, :create])
before_action :load_limit_offset_order_params, only: [:index, :contents]
before_action :load_where_param, only: [:index, :contents]
before_action :load_filters_param, only: [:index, :contents]
@@ -478,6 +479,21 @@ class ApplicationController < ActionController::Base
@object = @objects.first
end
+ def nullable_attributes
+ []
+ end
+
+ # Go code may send empty values (ie: empty string instead of NULL) that
+ # should be translated to NULL on the database.
+ def set_nullable_attrs_to_null
+ (resource_attrs.keys & nullable_attributes).each do |attr|
+ val = resource_attrs[attr]
+ if (val.class == Integer && val == 0) || (val.class == String && val == "")
+ resource_attrs[attr] = nil
+ end
+ end
+ end
+
def reload_object_before_update
# This is necessary to prevent an ActiveRecord::ReadOnlyRecord
# error when updating an object which was retrieved using a join.
diff --git a/services/api/app/controllers/arvados/v1/users_controller.rb b/services/api/app/controllers/arvados/v1/users_controller.rb
index 1cf3b9d78..fecc0620c 100644
--- a/services/api/app/controllers/arvados/v1/users_controller.rb
+++ b/services/api/app/controllers/arvados/v1/users_controller.rb
@@ -268,4 +268,8 @@ class Arvados::V1::UsersController < ApplicationController
end
super
end
+
+ def nullable_attributes
+ super + [:email, :first_name, :last_name, :username]
+ end
end
diff --git a/services/api/test/functional/arvados/v1/users_controller_test.rb b/services/api/test/functional/arvados/v1/users_controller_test.rb
index 753e707b6..b38f0d52f 100644
--- a/services/api/test/functional/arvados/v1/users_controller_test.rb
+++ b/services/api/test/functional/arvados/v1/users_controller_test.rb
@@ -88,6 +88,38 @@ class Arvados::V1::UsersControllerTest < ActionController::TestCase
assert_nil created['identity_url'], 'expected no identity_url'
end
+ test "create new user with empty username" do
+ authorize_with :admin
+ post :create, params: {
+ user: {
+ first_name: "test_first_name",
+ last_name: "test_last_name",
+ username: ""
+ }
+ }
+ assert_response :success
+ created = JSON.parse(@response.body)
+ assert_equal 'test_first_name', created['first_name']
+ assert_not_nil created['uuid'], 'expected uuid for the newly created user'
+ assert_nil created['email'], 'expected no email'
+ assert_nil created['username'], 'expected no username'
+ end
+
+ test "update user with empty username" do
+ authorize_with :admin
+ user = users('spectator')
+ assert_not_nil user['username']
+ put :update, params: {
+ id: users('spectator')['uuid'],
+ user: {
+ username: ""
+ }
+ }
+ assert_response :success
+ updated = JSON.parse(@response.body)
+ assert_nil updated['username'], 'expected no username'
+ end
+
test "create user with user, vm and repo as input" do
authorize_with :admin
repo_name = 'usertestrepo'
commit da84332fc7b4a549e08e0be87d4b52ffe2eeddd9
Author: Ward Vandewege <ward at jhvc.com>
Date: Fri Apr 10 08:56:56 2020 -0400
Fix tests.
refs #16326
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/lib/config/generated_config.go b/lib/config/generated_config.go
index 6813bee40..7b7c3c20d 100644
--- a/lib/config/generated_config.go
+++ b/lib/config/generated_config.go
@@ -190,12 +190,21 @@ Clusters:
MaxItemsPerResponse: 1000
# Maximum number of concurrent requests to accept in a single
- # service process, or 0 for no limit. Currently supported only
- # by keepstore.
+ # service process, or 0 for no limit.
MaxConcurrentRequests: 0
- # Maximum number of 64MiB memory buffers per keepstore server
- # process, or 0 for no limit.
+ # Maximum number of 64MiB memory buffers per Keepstore server process, or
+ # 0 for no limit. When this limit is reached, up to
+ # (MaxConcurrentRequests - MaxKeepBlobBuffers) HTTP requests requiring
+ # buffers (like GET and PUT) will wait for buffer space to be released.
+ # Any HTTP requests beyond MaxConcurrentRequests will receive an
+ # immediate 503 response.
+ #
+ # MaxKeepBlobBuffers should be set such that (MaxKeepBlobBuffers * 64MiB
+ # * 1.1) fits comfortably in memory. On a host dedicated to running
+ # Keepstore, divide total memory by 88MiB to suggest a suitable value.
+ # For example, if grep MemTotal /proc/meminfo reports MemTotal: 7125440
+ # kB, compute 7125440 / (88 * 1024)=79 and configure MaxBuffers: 79
MaxKeepBlobBuffers: 128
# API methods to disable. Disabled methods are not listed in the
commit 845381a133ebecfaff0335e432fda2fc0ac9f280
Author: Ward Vandewege <ward at jhvc.com>
Date: Thu Apr 9 17:53:31 2020 -0400
documentation: update descriptions for MaxKeepBlobBuffers and MaxConcurrentRequests
keepstore: MaxConcurrentRequests set to zero should mean no limit
refs #16326
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/lib/config/config.default.yml b/lib/config/config.default.yml
index 3750adcab..a0def71f7 100644
--- a/lib/config/config.default.yml
+++ b/lib/config/config.default.yml
@@ -184,12 +184,21 @@ Clusters:
MaxItemsPerResponse: 1000
# Maximum number of concurrent requests to accept in a single
- # service process, or 0 for no limit. Currently supported only
- # by keepstore.
+ # service process, or 0 for no limit.
MaxConcurrentRequests: 0
- # Maximum number of 64MiB memory buffers per keepstore server
- # process, or 0 for no limit.
+ # Maximum number of 64MiB memory buffers per Keepstore server process, or
+ # 0 for no limit. When this limit is reached, up to
+ # (MaxConcurrentRequests - MaxKeepBlobBuffers) HTTP requests requiring
+ # buffers (like GET and PUT) will wait for buffer space to be released.
+ # Any HTTP requests beyond MaxConcurrentRequests will receive an
+ # immediate 503 response.
+ #
+ # MaxKeepBlobBuffers should be set such that (MaxKeepBlobBuffers * 64MiB
+ # * 1.1) fits comfortably in memory. On a host dedicated to running
+ # Keepstore, divide total memory by 88MiB to suggest a suitable value.
+ # For example, if grep MemTotal /proc/meminfo reports MemTotal: 7125440
+ # kB, compute 7125440 / (88 * 1024)=79 and configure MaxBuffers: 79
MaxKeepBlobBuffers: 128
# API methods to disable. Disabled methods are not listed in the
diff --git a/services/keepstore/command.go b/services/keepstore/command.go
index 0593460a2..c17ee35a3 100644
--- a/services/keepstore/command.go
+++ b/services/keepstore/command.go
@@ -153,10 +153,6 @@ func (h *handler) setup(ctx context.Context, cluster *arvados.Cluster, token str
}
bufs = newBufferPool(h.Logger, h.Cluster.API.MaxKeepBlobBuffers, BlockSize)
- if h.Cluster.API.MaxConcurrentRequests < 1 {
- h.Cluster.API.MaxConcurrentRequests = h.Cluster.API.MaxKeepBlobBuffers * 2
- h.Logger.Warnf("API.MaxConcurrentRequests <1 or not specified; defaulting to MaxKeepBlobBuffers * 2 == %d", h.Cluster.API.MaxConcurrentRequests)
- }
if h.Cluster.API.MaxConcurrentRequests > 0 && h.Cluster.API.MaxConcurrentRequests < h.Cluster.API.MaxKeepBlobBuffers {
h.Logger.Warnf("Possible configuration mistake: not useful to set API.MaxKeepBlobBuffers (%d) higher than API.MaxConcurrentRequests (%d)", h.Cluster.API.MaxKeepBlobBuffers, h.Cluster.API.MaxConcurrentRequests)
}
commit 8a125907af9a75bbf905de4b715309e562092704
Author: Tom Clegg <tom at tomclegg.ca>
Date: Thu Apr 9 16:58:36 2020 -0400
Warn if MaxKeepBlobBuffers > MaxConcurrentRequests.
No issue #
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/keepstore/command.go b/services/keepstore/command.go
index e0509393c..0593460a2 100644
--- a/services/keepstore/command.go
+++ b/services/keepstore/command.go
@@ -157,6 +157,9 @@ func (h *handler) setup(ctx context.Context, cluster *arvados.Cluster, token str
h.Cluster.API.MaxConcurrentRequests = h.Cluster.API.MaxKeepBlobBuffers * 2
h.Logger.Warnf("API.MaxConcurrentRequests <1 or not specified; defaulting to MaxKeepBlobBuffers * 2 == %d", h.Cluster.API.MaxConcurrentRequests)
}
+ if h.Cluster.API.MaxConcurrentRequests > 0 && h.Cluster.API.MaxConcurrentRequests < h.Cluster.API.MaxKeepBlobBuffers {
+ h.Logger.Warnf("Possible configuration mistake: not useful to set API.MaxKeepBlobBuffers (%d) higher than API.MaxConcurrentRequests (%d)", h.Cluster.API.MaxKeepBlobBuffers, h.Cluster.API.MaxConcurrentRequests)
+ }
if h.Cluster.Collections.BlobSigningKey != "" {
} else if h.Cluster.Collections.BlobSigning {
commit 3eeae9396640383abeecfb3ea5289b75a144d969
Author: Tom Clegg <tom at tomclegg.ca>
Date: Thu Mar 5 16:42:39 2020 -0500
16219: Test populated container fields.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/lib/dispatchcloud/container/queue_test.go b/lib/dispatchcloud/container/queue_test.go
index 31f321488..0075ee324 100644
--- a/lib/dispatchcloud/container/queue_test.go
+++ b/lib/dispatchcloud/container/queue_test.go
@@ -106,6 +106,13 @@ func (suite *IntegrationSuite) TestGetLockUnlockCancel(c *check.C) {
func (suite *IntegrationSuite) TestCancelIfNoInstanceType(c *check.C) {
errorTypeChooser := func(ctr *arvados.Container) (arvados.InstanceType, error) {
+ // Make sure the relevant container fields are
+ // actually populated.
+ c.Check(ctr.ContainerImage, check.Equals, "test")
+ c.Check(ctr.RuntimeConstraints.VCPUs, check.Equals, 4)
+ c.Check(ctr.RuntimeConstraints.RAM, check.Equals, int64(12000000000))
+ c.Check(ctr.Mounts["/tmp"].Capacity, check.Equals, int64(24000000000))
+ c.Check(ctr.Mounts["/var/spool/cwl"].Capacity, check.Equals, int64(24000000000))
return arvados.InstanceType{}, errors.New("no suitable instance type")
}
commit 7781042d8602072fe95b607ad9c64713a46f1637
Author: Tom Clegg <tom at tomclegg.ca>
Date: Thu Mar 5 14:44:22 2020 -0500
16219: Load all fields needed to compute node size.
Without ContainerImage and Mounts, the scratch size requirement can't
be computed correctly.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/lib/dispatchcloud/container/queue.go b/lib/dispatchcloud/container/queue.go
index a4a270dd1..d128c265f 100644
--- a/lib/dispatchcloud/container/queue.go
+++ b/lib/dispatchcloud/container/queue.go
@@ -26,8 +26,9 @@ type APIClient interface {
// A QueueEnt is an entry in the queue, consisting of a container
// record and the instance type that should be used to run it.
type QueueEnt struct {
- // The container to run. Only the UUID, State, Priority, and
- // RuntimeConstraints fields are populated.
+ // The container to run. Only the UUID, State, Priority,
+ // RuntimeConstraints, Mounts, and ContainerImage fields are
+ // populated.
Container arvados.Container `json:"container"`
InstanceType arvados.InstanceType `json:"instance_type"`
}
@@ -381,7 +382,7 @@ func (cq *Queue) poll() (map[string]*arvados.Container, error) {
*next[upd.UUID] = upd
}
}
- selectParam := []string{"uuid", "state", "priority", "runtime_constraints"}
+ selectParam := []string{"uuid", "state", "priority", "runtime_constraints", "container_image", "mounts"}
limitParam := 1000
mine, err := cq.fetchAll(arvados.ResourceListParams{
commit 6278100f84d7cc021932ce230ae7e713ed129b01
Author: Tom Clegg <tom at tomclegg.ca>
Date: Thu Mar 12 00:15:16 2020 -0400
16221: Fix test for config endpoint.
x['configs'] raises KeyError on an old discovery doc -- test
x.get('configs', False) instead.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/sdk/python/arvados/util.py b/sdk/python/arvados/util.py
index 9e0a31783..dcc0417c1 100644
--- a/sdk/python/arvados/util.py
+++ b/sdk/python/arvados/util.py
@@ -421,7 +421,7 @@ def new_request_id():
return rid
def get_config_once(svc):
- if not svc._rootDesc.get('resources')['configs']:
+ if not svc._rootDesc.get('resources').get('configs', False):
# Old API server version, no config export endpoint
return {}
if not hasattr(svc, '_cached_config'):
commit 891f67ea72aa35895c57421531b2b5cbbd25e70d
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed Mar 11 16:52:04 2020 -0400
Fix instance IDs in arvados-dispatch-cloud log messages.
Some logs had {"Instance": {}} instead of the provider's instance ID.
No issue #
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/lib/dispatchcloud/worker/pool.go b/lib/dispatchcloud/worker/pool.go
index c52422c41..14b14ea5d 100644
--- a/lib/dispatchcloud/worker/pool.go
+++ b/lib/dispatchcloud/worker/pool.go
@@ -435,7 +435,7 @@ func (wp *Pool) Shutdown(it arvados.InstanceType) bool {
// time (Idle) or the earliest create time (Booting)
for _, wkr := range wp.workers {
if wkr.idleBehavior != IdleBehaviorHold && wkr.state == tryState && wkr.instType == it {
- logger.WithField("Instance", wkr.instance).Info("shutting down")
+ logger.WithField("Instance", wkr.instance.ID()).Info("shutting down")
wkr.shutdown()
return true
}
@@ -835,13 +835,13 @@ func (wp *Pool) sync(threshold time.Time, instances []cloud.Instance) {
itTag := inst.Tags()[wp.tagKeyPrefix+tagKeyInstanceType]
it, ok := wp.instanceTypes[itTag]
if !ok {
- wp.logger.WithField("Instance", inst).Errorf("unknown InstanceType tag %q --- ignoring", itTag)
+ wp.logger.WithField("Instance", inst.ID()).Errorf("unknown InstanceType tag %q --- ignoring", itTag)
continue
}
if wkr, isNew := wp.updateWorker(inst, it); isNew {
notify = true
} else if wkr.state == StateShutdown && time.Since(wkr.destroyed) > wp.timeoutShutdown {
- wp.logger.WithField("Instance", inst).Info("worker still listed after shutdown; retrying")
+ wp.logger.WithField("Instance", inst.ID()).Info("worker still listed after shutdown; retrying")
wkr.shutdown()
}
}
commit eec034bf27818af1c3bd998e51c85772e7cdc9dd
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed Apr 1 16:44:02 2020 -0400
16270: Fill in missing scratch fields on InstanceType entries.
Previously they were being filled in correctly when written as an
array in the config file, but not when written as a map.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/sdk/go/arvados/config.go b/sdk/go/arvados/config.go
index a70980cbd..79e47ba5d 100644
--- a/sdk/go/arvados/config.go
+++ b/sdk/go/arvados/config.go
@@ -421,6 +421,24 @@ var errDuplicateInstanceTypeName = errors.New("duplicate instance type name")
// UnmarshalJSON handles old config files that provide an array of
// instance types instead of a hash.
func (it *InstanceTypeMap) UnmarshalJSON(data []byte) error {
+ fixup := func(t InstanceType) (InstanceType, error) {
+ if t.ProviderType == "" {
+ t.ProviderType = t.Name
+ }
+ if t.Scratch == 0 {
+ t.Scratch = t.IncludedScratch + t.AddedScratch
+ } else if t.AddedScratch == 0 {
+ t.AddedScratch = t.Scratch - t.IncludedScratch
+ } else if t.IncludedScratch == 0 {
+ t.IncludedScratch = t.Scratch - t.AddedScratch
+ }
+
+ if t.Scratch != (t.IncludedScratch + t.AddedScratch) {
+ return t, fmt.Errorf("InstanceType %q: Scratch != (IncludedScratch + AddedScratch)", t.Name)
+ }
+ return t, nil
+ }
+
if len(data) > 0 && data[0] == '[' {
var arr []InstanceType
err := json.Unmarshal(data, &arr)
@@ -436,19 +454,9 @@ func (it *InstanceTypeMap) UnmarshalJSON(data []byte) error {
if _, ok := (*it)[t.Name]; ok {
return errDuplicateInstanceTypeName
}
- if t.ProviderType == "" {
- t.ProviderType = t.Name
- }
- if t.Scratch == 0 {
- t.Scratch = t.IncludedScratch + t.AddedScratch
- } else if t.AddedScratch == 0 {
- t.AddedScratch = t.Scratch - t.IncludedScratch
- } else if t.IncludedScratch == 0 {
- t.IncludedScratch = t.Scratch - t.AddedScratch
- }
-
- if t.Scratch != (t.IncludedScratch + t.AddedScratch) {
- return fmt.Errorf("%v: Scratch != (IncludedScratch + AddedScratch)", t.Name)
+ t, err := fixup(t)
+ if err != nil {
+ return err
}
(*it)[t.Name] = t
}
@@ -464,8 +472,9 @@ func (it *InstanceTypeMap) UnmarshalJSON(data []byte) error {
*it = InstanceTypeMap(hash)
for name, t := range *it {
t.Name = name
- if t.ProviderType == "" {
- t.ProviderType = name
+ t, err := fixup(t)
+ if err != nil {
+ return err
}
(*it)[name] = t
}
diff --git a/sdk/go/arvados/config_test.go b/sdk/go/arvados/config_test.go
index b984cb566..e4d26e03f 100644
--- a/sdk/go/arvados/config_test.go
+++ b/sdk/go/arvados/config_test.go
@@ -45,3 +45,29 @@ func (s *ConfigSuite) TestInstanceTypeSize(c *check.C) {
c.Check(int64(it.Scratch), check.Equals, int64(4000000000))
c.Check(int64(it.RAM), check.Equals, int64(4294967296))
}
+
+func (s *ConfigSuite) TestInstanceTypeFixup(c *check.C) {
+ for _, confdata := range []string{
+ // Current format: map of entries
+ `{foo4: {IncludedScratch: 4GB}, foo8: {ProviderType: foo_8, Scratch: 8GB}}`,
+ // Legacy format: array of entries with key in "Name" field
+ `[{Name: foo4, IncludedScratch: 4GB}, {Name: foo8, ProviderType: foo_8, Scratch: 8GB}]`,
+ } {
+ c.Log(confdata)
+ var itm InstanceTypeMap
+ err := yaml.Unmarshal([]byte(confdata), &itm)
+ c.Check(err, check.IsNil)
+
+ c.Check(itm["foo4"].Name, check.Equals, "foo4")
+ c.Check(itm["foo4"].ProviderType, check.Equals, "foo4")
+ c.Check(itm["foo4"].Scratch, check.Equals, ByteSize(4000000000))
+ c.Check(itm["foo4"].AddedScratch, check.Equals, ByteSize(0))
+ c.Check(itm["foo4"].IncludedScratch, check.Equals, ByteSize(4000000000))
+
+ c.Check(itm["foo8"].Name, check.Equals, "foo8")
+ c.Check(itm["foo8"].ProviderType, check.Equals, "foo_8")
+ c.Check(itm["foo8"].Scratch, check.Equals, ByteSize(8000000000))
+ c.Check(itm["foo8"].AddedScratch, check.Equals, ByteSize(8000000000))
+ c.Check(itm["foo8"].IncludedScratch, check.Equals, ByteSize(0))
+ }
+}
commit 89705b6140489b7badc42b6a67367dafaeca78d0
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Mar 3 21:27:27 2020 +0000
Bump rake from 12.3.2 to 13.0.1 in /services/api
Bumps [rake](https://github.com/ruby/rake) from 12.3.2 to 13.0.1.
- [Release notes](https://github.com/ruby/rake/releases)
- [Changelog](https://github.com/ruby/rake/blob/master/History.rdoc)
- [Commits](https://github.com/ruby/rake/compare/v12.3.2...v13.0.1)
Signed-off-by: dependabot[bot] <support at github.com>
diff --git a/services/api/Gemfile.lock b/services/api/Gemfile.lock
index 53cdc906e..24d5ad5b6 100644
--- a/services/api/Gemfile.lock
+++ b/services/api/Gemfile.lock
@@ -213,7 +213,7 @@ GEM
method_source
rake (>= 0.8.7)
thor (>= 0.18.1, < 2.0)
- rake (12.3.2)
+ rake (13.0.1)
rb-fsevent (0.10.3)
rb-inotify (0.9.10)
ffi (>= 0.5.0, < 2)
commit b7e6ce193a9158b9e4700fe9334018702dedb3d8
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Mar 3 21:27:30 2020 +0000
Bump rake from 12.3.2 to 13.0.1 in /apps/workbench
Bumps [rake](https://github.com/ruby/rake) from 12.3.2 to 13.0.1.
- [Release notes](https://github.com/ruby/rake/releases)
- [Changelog](https://github.com/ruby/rake/blob/master/History.rdoc)
- [Commits](https://github.com/ruby/rake/compare/v12.3.2...v13.0.1)
Signed-off-by: dependabot[bot] <support at github.com>
diff --git a/apps/workbench/Gemfile.lock b/apps/workbench/Gemfile.lock
index 6f36def0c..2af9c8b16 100644
--- a/apps/workbench/Gemfile.lock
+++ b/apps/workbench/Gemfile.lock
@@ -247,7 +247,7 @@ GEM
method_source
rake (>= 0.8.7)
thor (>= 0.18.1, < 2.0)
- rake (12.3.2)
+ rake (13.0.1)
raphael-rails (2.1.2)
rb-fsevent (0.10.3)
rb-inotify (0.10.0)
commit fca7b8b5b8d11938f2f1c2684a9a34814968f86a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Feb 25 08:04:58 2020 +0000
Bump nokogiri from 1.10.4 to 1.10.8 in /apps/workbench
Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.10.4 to 1.10.8.
- [Release notes](https://github.com/sparklemotion/nokogiri/releases)
- [Changelog](https://github.com/sparklemotion/nokogiri/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sparklemotion/nokogiri/compare/v1.10.4...v1.10.8)
Signed-off-by: dependabot[bot] <support at github.com>
diff --git a/apps/workbench/Gemfile.lock b/apps/workbench/Gemfile.lock
index 829bc8584..6f36def0c 100644
--- a/apps/workbench/Gemfile.lock
+++ b/apps/workbench/Gemfile.lock
@@ -195,7 +195,7 @@ GEM
net-ssh-gateway (2.0.0)
net-ssh (>= 4.0.0)
nio4r (2.3.1)
- nokogiri (1.10.5)
+ nokogiri (1.10.8)
mini_portile2 (~> 2.4.0)
npm-rails (0.2.1)
rails (>= 3.2)
commit eeb4aa1bcf4462466170a4d3053d6c37f3497d7c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri Nov 8 21:48:47 2019 +0000
Bump loofah from 2.2.3 to 2.3.1 in /apps/workbench
Bumps [loofah](https://github.com/flavorjones/loofah) from 2.2.3 to 2.3.1.
- [Release notes](https://github.com/flavorjones/loofah/releases)
- [Changelog](https://github.com/flavorjones/loofah/blob/master/CHANGELOG.md)
- [Commits](https://github.com/flavorjones/loofah/compare/v2.2.3...v2.3.1)
Signed-off-by: dependabot[bot] <support at github.com>
diff --git a/apps/workbench/Gemfile.lock b/apps/workbench/Gemfile.lock
index b02161c59..829bc8584 100644
--- a/apps/workbench/Gemfile.lock
+++ b/apps/workbench/Gemfile.lock
@@ -122,7 +122,7 @@ GEM
coffee-script-source (1.12.2)
commonjs (0.2.7)
concurrent-ruby (1.1.5)
- crass (1.0.4)
+ crass (1.0.5)
deep_merge (1.2.1)
docile (1.3.1)
erubis (2.7.0)
@@ -167,7 +167,7 @@ GEM
railties (>= 4)
request_store (~> 1.0)
logstash-event (1.2.02)
- loofah (2.2.3)
+ loofah (2.3.1)
crass (~> 1.0.2)
nokogiri (>= 1.5.9)
mail (2.7.1)
@@ -195,7 +195,7 @@ GEM
net-ssh-gateway (2.0.0)
net-ssh (>= 4.0.0)
nio4r (2.3.1)
- nokogiri (1.10.4)
+ nokogiri (1.10.5)
mini_portile2 (~> 2.4.0)
npm-rails (0.2.1)
rails (>= 3.2)
commit 445465e4e859de272bee53c2eed3cd8f54e33d34
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Feb 25 08:04:57 2020 +0000
Bump nokogiri from 1.10.2 to 1.10.8 in /services/api
Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.10.2 to 1.10.8.
- [Release notes](https://github.com/sparklemotion/nokogiri/releases)
- [Changelog](https://github.com/sparklemotion/nokogiri/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sparklemotion/nokogiri/compare/v1.10.2...v1.10.8)
Signed-off-by: dependabot[bot] <support at github.com>
diff --git a/services/api/Gemfile.lock b/services/api/Gemfile.lock
index 2c780d477..53cdc906e 100644
--- a/services/api/Gemfile.lock
+++ b/services/api/Gemfile.lock
@@ -157,7 +157,7 @@ GEM
net-ssh-gateway (2.0.0)
net-ssh (>= 4.0.0)
nio4r (2.3.1)
- nokogiri (1.10.2)
+ nokogiri (1.10.8)
mini_portile2 (~> 2.4.0)
oauth2 (1.4.1)
faraday (>= 0.8, < 0.16.0)
commit 7d1defd3d60fe165522a736855346c9d34eae28d
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Mon Mar 23 18:07:56 2020 -0300
16266: Applies monkeypatch to fix CVE-2020-5267 on workbench1.
As adviced on https://github.com/advisories/GHSA-65cv-r6x7-79hv
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/apps/workbench/config/initializers/actionview_xss_fix.rb b/apps/workbench/config/initializers/actionview_xss_fix.rb
new file mode 100644
index 000000000..3f5e239ef
--- /dev/null
+++ b/apps/workbench/config/initializers/actionview_xss_fix.rb
@@ -0,0 +1,32 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+# This is related to:
+# * https://github.com/advisories/GHSA-65cv-r6x7-79hv
+# * https://nvd.nist.gov/vuln/detail/CVE-2020-5267
+#
+# Until we upgrade to rails 5.2, this monkeypatch should be enough
+ActionView::Helpers::JavaScriptHelper::JS_ESCAPE_MAP.merge!(
+ {
+ "`" => "\\`",
+ "$" => "\\$"
+ }
+)
+
+module ActionView::Helpers::JavaScriptHelper
+ alias :old_ej :escape_javascript
+ alias :old_j :j
+
+ def escape_javascript(javascript)
+ javascript = javascript.to_s
+ if javascript.empty?
+ result = ""
+ else
+ result = javascript.gsub(/(\\|<\/|\r\n|\342\200\250|\342\200\251|[\n\r"']|[`]|[$])/u, JS_ESCAPE_MAP)
+ end
+ javascript.html_safe? ? result.html_safe : result
+ end
+
+ alias :j :escape_javascript
+end
\ No newline at end of file
commit 883f51266e8ac9275b7842704c56ff195f853080
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Thu Mar 26 13:29:40 2020 -0300
16266: Adds tests exposing potential XSS vulnerability on escape_javascript()
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/apps/workbench/test/unit/helpers/javascript_helper_test.rb b/apps/workbench/test/unit/helpers/javascript_helper_test.rb
new file mode 100644
index 000000000..9d5a55345
--- /dev/null
+++ b/apps/workbench/test/unit/helpers/javascript_helper_test.rb
@@ -0,0 +1,17 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+require 'test_helper'
+
+# Tests XSS vulnerability monkeypatch
+# See: https://github.com/advisories/GHSA-65cv-r6x7-79hv
+class JavascriptHelperTest < ActionView::TestCase
+ def test_escape_backtick
+ assert_equal "\\`", escape_javascript("`")
+ end
+
+ def test_escape_dollar_sign
+ assert_equal "\\$", escape_javascript("$")
+ end
+end
commit 2713cc305c5065959da84b61e3a1b8867c72f4e8
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Mar 24 15:35:38 2020 -0400
16138: Remove additional/code references to copying pipelines
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/python/arvados/commands/arv_copy.py b/sdk/python/arvados/commands/arv_copy.py
index d39ed3387..5f12b62ee 100755
--- a/sdk/python/arvados/commands/arv_copy.py
+++ b/sdk/python/arvados/commands/arv_copy.py
@@ -98,18 +98,9 @@ def main():
copy_opts.add_argument(
'--no-recursive', dest='recursive', action='store_false',
help='Do not copy any dependencies. NOTE: if this option is given, the copied object will need to be updated manually in order to be functional.')
- copy_opts.add_argument(
- '--dst-git-repo', dest='dst_git_repo',
- help='The name of the destination git repository. Required when copying a pipeline recursively.')
copy_opts.add_argument(
'--project-uuid', dest='project_uuid',
- help='The UUID of the project at the destination to which the pipeline should be copied.')
- copy_opts.add_argument(
- '--allow-git-http-src', action="store_true",
- help='Allow cloning git repositories over insecure http')
- copy_opts.add_argument(
- '--allow-git-http-dst', action="store_true",
- help='Allow pushing git repositories over insecure http')
+ help='The UUID of the project at the destination to which the collection or workflow should be copied.')
copy_opts.add_argument(
'object_uuid',
@@ -118,7 +109,7 @@ def main():
copy_opts.set_defaults(recursive=True)
parser = argparse.ArgumentParser(
- description='Copy a pipeline instance, template, workflow, or collection from one Arvados instance to another.',
+ description='Copy a workflow or collection from one Arvados instance to another.',
parents=[copy_opts, arv_cmd.retry_opt])
args = parser.parse_args()
@@ -468,7 +459,7 @@ def copy_collection(obj_uuid, src, dst, args):
c = items[0]
if not c:
# See if there is a collection that's in the same project
- # as the root item (usually a pipeline) being copied.
+ # as the root item (usually a workflow) being copied.
for i in items:
if i.get("owner_uuid") == src_owner_uuid and i.get("name"):
c = i
@@ -618,55 +609,6 @@ def select_git_url(api, repo_name, retries, allow_insecure_http, allow_insecure_
return (git_url, git_config)
-# copy_git_repo(src_git_repo, src, dst, dst_git_repo, script_version, args)
-#
-# Copies commits from git repository 'src_git_repo' on Arvados
-# instance 'src' to 'dst_git_repo' on 'dst'. Both src_git_repo
-# and dst_git_repo are repository names, not UUIDs (i.e. "arvados"
-# or "jsmith")
-#
-# All commits will be copied to a destination branch named for the
-# source repository URL.
-#
-# The destination repository must already exist.
-#
-# The user running this command must be authenticated
-# to both repositories.
-#
-def copy_git_repo(src_git_repo, src, dst, dst_git_repo, script_version, args):
- # Identify the fetch and push URLs for the git repositories.
-
- (src_git_url, src_git_config) = select_git_url(src, src_git_repo, args.retries, args.allow_git_http_src, "--allow-git-http-src")
- (dst_git_url, dst_git_config) = select_git_url(dst, dst_git_repo, args.retries, args.allow_git_http_dst, "--allow-git-http-dst")
-
- logger.debug('src_git_url: {}'.format(src_git_url))
- logger.debug('dst_git_url: {}'.format(dst_git_url))
-
- dst_branch = re.sub(r'\W+', '_', "{}_{}".format(src_git_url, script_version))
-
- # Copy git commits from src repo to dst repo.
- if src_git_repo not in local_repo_dir:
- local_repo_dir[src_git_repo] = tempfile.mkdtemp()
- arvados.util.run_command(
- ["git"] + src_git_config + ["clone", "--bare", src_git_url,
- local_repo_dir[src_git_repo]],
- cwd=os.path.dirname(local_repo_dir[src_git_repo]),
- env={"HOME": os.environ["HOME"],
- "ARVADOS_API_TOKEN": src.api_token,
- "GIT_ASKPASS": "/bin/false"})
- arvados.util.run_command(
- ["git", "remote", "add", "dst", dst_git_url],
- cwd=local_repo_dir[src_git_repo])
- arvados.util.run_command(
- ["git", "branch", dst_branch, script_version],
- cwd=local_repo_dir[src_git_repo])
- arvados.util.run_command(["git"] + dst_git_config + ["push", "dst", dst_branch],
- cwd=local_repo_dir[src_git_repo],
- env={"HOME": os.environ["HOME"],
- "ARVADOS_API_TOKEN": dst.api_token,
- "GIT_ASKPASS": "/bin/false"})
-
-
def copy_docker_image(docker_image, docker_image_tag, src, dst, args):
"""Copy the docker image identified by docker_image and
docker_image_tag from src to dst. Create appropriate
@@ -707,7 +649,7 @@ def git_rev_parse(rev, repo):
# the second field of the uuid. This function consults the api's
# schema to identify the object class.
#
-# It returns a string such as 'Collection', 'PipelineInstance', etc.
+# It returns a string such as 'Collection', 'Workflow', etc.
#
# Special case: if handed a Keep locator hash, return 'Collection'.
#
commit a9866cc99faa6de3df2884b8ad97e858b922036c
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Mar 23 11:00:07 2020 -0400
16138: Remove additional pipelines related code
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/python/arvados/commands/arv_copy.py b/sdk/python/arvados/commands/arv_copy.py
index d94d19efc..d39ed3387 100755
--- a/sdk/python/arvados/commands/arv_copy.py
+++ b/sdk/python/arvados/commands/arv_copy.py
@@ -8,8 +8,8 @@
#
# By default, arv-copy recursively copies any dependent objects
# necessary to make the object functional in the new instance
-# (e.g. for a pipeline instance, arv-copy copies the pipeline
-# template, input collection, docker images, git repositories). If
+# (e.g. for a workflow, arv-copy copies the workflow,
+# input collections, and docker images). If
# --no-recursive is given, arv-copy copies only the single record
# identified by object-uuid.
#
@@ -86,9 +86,6 @@ def main():
copy_opts.add_argument(
'-f', '--force', dest='force', action='store_true',
help='Perform copy even if the object appears to exist at the remote destination.')
- copy_opts.add_argument(
- '--force-filters', action='store_true', default=False,
- help="Copy pipeline template filters verbatim, even if they act differently on the destination cluster.")
copy_opts.add_argument(
'--src', dest='source_arvados', required=True,
help='The name of the source Arvados instance (required) - points at an Arvados config file. May be either a pathname to a config file, or (for example) "foo" as shorthand for $HOME/.config/arvados/foo.conf.')
@@ -270,41 +267,6 @@ def exception_handler(handler, *exc_types):
except exc_types as error:
handler(error)
-def migrate_components_filters(template_components, dst_git_repo):
- """Update template component filters in-place for the destination.
-
- template_components is a dictionary of components in a pipeline template.
- This method walks over each component's filters, and updates them to have
- identical semantics on the destination cluster. It returns a list of
- error strings that describe what filters could not be updated safely.
-
- dst_git_repo is the name of the destination Git repository, which can
- be None if that is not known.
- """
- errors = []
- for cname, cspec in template_components.items():
- def add_error(errmsg):
- errors.append("{}: {}".format(cname, errmsg))
- if not isinstance(cspec, dict):
- add_error("value is not a component definition")
- continue
- src_repository = cspec.get('repository')
- filters = cspec.get('filters', [])
- if not isinstance(filters, list):
- add_error("filters are not a list")
- continue
- for cfilter in filters:
- if not (isinstance(cfilter, list) and (len(cfilter) == 3)):
- add_error("malformed filter {!r}".format(cfilter))
- continue
- if attr_filtered(cfilter, 'repository'):
- with exception_handler(add_error, ValueError):
- migrate_repository_filter(cfilter, src_repository, dst_git_repo)
- if attr_filtered(cfilter, 'script_version'):
- with exception_handler(add_error, ValueError):
- migrate_script_version_filter(cfilter)
- return errors
-
# copy_workflow(wf_uuid, src, dst, args)
#
@@ -407,53 +369,6 @@ def copy_collections(obj, src, dst, args):
return type(obj)(copy_collections(v, src, dst, args) for v in obj)
return obj
-def migrate_jobspec(jobspec, src, dst, dst_repo, args):
- """Copy a job's script to the destination repository, and update its record.
-
- Given a jobspec dictionary, this function finds the referenced script from
- src and copies it to dst and dst_repo. It also updates jobspec in place to
- refer to names on the destination.
- """
- repo = jobspec.get('repository')
- if repo is None:
- return
- # script_version is the "script_version" parameter from the source
- # component or job. If no script_version was supplied in the
- # component or job, it is a mistake in the pipeline, but for the
- # purposes of copying the repository, default to "master".
- script_version = jobspec.get('script_version') or 'master'
- script_key = (repo, script_version)
- if script_key not in scripts_copied:
- copy_git_repo(repo, src, dst, dst_repo, script_version, args)
- scripts_copied.add(script_key)
- jobspec['repository'] = dst_repo
- repo_dir = local_repo_dir[repo]
- for version_key in ['script_version', 'supplied_script_version']:
- if version_key in jobspec:
- jobspec[version_key] = git_rev_parse(jobspec[version_key], repo_dir)
-
-# copy_git_repos(p, src, dst, dst_repo, args)
-#
-# Copies all git repositories referenced by pipeline instance or
-# template 'p' from src to dst.
-#
-# For each component c in the pipeline:
-# * Copy git repositories named in c['repository'] and c['job']['repository'] if present
-# * Rename script versions:
-# * c['script_version']
-# * c['job']['script_version']
-# * c['job']['supplied_script_version']
-# to the commit hashes they resolve to, since any symbolic
-# names (tags, branches) are not preserved in the destination repo.
-#
-# The pipeline object is updated in place with the new repository
-# names. The return value is undefined.
-#
-def copy_git_repos(p, src, dst, dst_repo, args):
- for component in p['components'].values():
- migrate_jobspec(component, src, dst, dst_repo, args)
- if 'job' in component:
- migrate_jobspec(component['job'], src, dst, dst_repo, args)
def total_collection_size(manifest_text):
"""Return the total number of bytes in this collection (excluding
@@ -751,19 +666,6 @@ def copy_git_repo(src_git_repo, src, dst, dst_git_repo, script_version, args):
"ARVADOS_API_TOKEN": dst.api_token,
"GIT_ASKPASS": "/bin/false"})
-def copy_docker_images(pipeline, src, dst, args):
- """Copy any docker images named in the pipeline components'
- runtime_constraints field from src to dst."""
-
- logger.debug('copy_docker_images: {}'.format(pipeline['uuid']))
- for c_name, c_info in pipeline['components'].items():
- if ('runtime_constraints' in c_info and
- 'docker_image' in c_info['runtime_constraints']):
- copy_docker_image(
- c_info['runtime_constraints']['docker_image'],
- c_info['runtime_constraints'].get('docker_image_tag', 'latest'),
- src, dst, args)
-
def copy_docker_image(docker_image, docker_image_tag, src, dst, args):
"""Copy the docker image identified by docker_image and
commit c4f5611d0e67028b86b5c468ee1a7f200866cb32
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Mar 20 15:47:00 2020 -0400
16138: whitelist fields for arv-copy
Also removed code for copying
PipelineTemplate/PipelineInstance (obsolete jobs API).
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/python/arvados/commands/arv_copy.py b/sdk/python/arvados/commands/arv_copy.py
index 0ba3f0a48..d94d19efc 100755
--- a/sdk/python/arvados/commands/arv_copy.py
+++ b/sdk/python/arvados/commands/arv_copy.py
@@ -144,15 +144,6 @@ def main():
result = copy_collection(args.object_uuid,
src_arv, dst_arv,
args)
- elif t == 'PipelineInstance':
- set_src_owner_uuid(src_arv.pipeline_instances(), args.object_uuid, args)
- result = copy_pipeline_instance(args.object_uuid,
- src_arv, dst_arv,
- args)
- elif t == 'PipelineTemplate':
- set_src_owner_uuid(src_arv.pipeline_templates(), args.object_uuid, args)
- result = copy_pipeline_template(args.object_uuid,
- src_arv, dst_arv, args)
elif t == 'Workflow':
set_src_owner_uuid(src_arv.workflows(), args.object_uuid, args)
result = copy_workflow(args.object_uuid, src_arv, dst_arv, args)
@@ -225,67 +216,6 @@ def check_git_availability():
except Exception:
abort('git command is not available. Please ensure git is installed.')
-# copy_pipeline_instance(pi_uuid, src, dst, args)
-#
-# Copies a pipeline instance identified by pi_uuid from src to dst.
-#
-# If the args.recursive option is set:
-# 1. Copies all input collections
-# * For each component in the pipeline, include all collections
-# listed as job dependencies for that component)
-# 2. Copy docker images
-# 3. Copy git repositories
-# 4. Copy the pipeline template
-#
-# The only changes made to the copied pipeline instance are:
-# 1. The original pipeline instance UUID is preserved in
-# the 'properties' hash as 'copied_from_pipeline_instance_uuid'.
-# 2. The pipeline_template_uuid is changed to the new template uuid.
-# 3. The owner_uuid of the instance is changed to the user who
-# copied it.
-#
-def copy_pipeline_instance(pi_uuid, src, dst, args):
- # Fetch the pipeline instance record.
- pi = src.pipeline_instances().get(uuid=pi_uuid).execute(num_retries=args.retries)
-
- if args.recursive:
- check_git_availability()
-
- if not args.dst_git_repo:
- abort('--dst-git-repo is required when copying a pipeline recursively.')
- # Copy the pipeline template and save the copied template.
- if pi.get('pipeline_template_uuid', None):
- pt = copy_pipeline_template(pi['pipeline_template_uuid'],
- src, dst, args)
-
- # Copy input collections, docker images and git repos.
- pi = copy_collections(pi, src, dst, args)
- copy_git_repos(pi, src, dst, args.dst_git_repo, args)
- copy_docker_images(pi, src, dst, args)
-
- # Update the fields of the pipeline instance with the copied
- # pipeline template.
- if pi.get('pipeline_template_uuid', None):
- pi['pipeline_template_uuid'] = pt['uuid']
-
- else:
- # not recursive
- logger.info("Copying only pipeline instance %s.", pi_uuid)
- logger.info("You are responsible for making sure all pipeline dependencies have been updated.")
-
- # Update the pipeline instance properties, and create the new
- # instance at dst.
- pi['properties']['copied_from_pipeline_instance_uuid'] = pi_uuid
- pi['description'] = "Pipeline copied from {}\n\n{}".format(
- pi_uuid,
- pi['description'] if pi.get('description', None) else '')
-
- pi['owner_uuid'] = args.project_uuid
-
- del pi['uuid']
-
- new_pi = dst.pipeline_instances().create(body=pi, ensure_unique_name=True).execute(num_retries=args.retries)
- return new_pi
def filter_iter(arg):
"""Iterate a filter string-or-list.
@@ -375,47 +305,6 @@ def migrate_components_filters(template_components, dst_git_repo):
migrate_script_version_filter(cfilter)
return errors
-# copy_pipeline_template(pt_uuid, src, dst, args)
-#
-# Copies a pipeline template identified by pt_uuid from src to dst.
-#
-# If args.recursive is True, also copy any collections, docker
-# images and git repositories that this template references.
-#
-# The owner_uuid of the new template is changed to that of the user
-# who copied the template.
-#
-# Returns the copied pipeline template object.
-#
-def copy_pipeline_template(pt_uuid, src, dst, args):
- # fetch the pipeline template from the source instance
- pt = src.pipeline_templates().get(uuid=pt_uuid).execute(num_retries=args.retries)
-
- if not args.force_filters:
- filter_errors = migrate_components_filters(pt['components'], args.dst_git_repo)
- if filter_errors:
- abort("Template filters cannot be copied safely. Use --force-filters to copy anyway.\n" +
- "\n".join(filter_errors))
-
- if args.recursive:
- check_git_availability()
-
- if not args.dst_git_repo:
- abort('--dst-git-repo is required when copying a pipeline recursively.')
- # Copy input collections, docker images and git repos.
- pt = copy_collections(pt, src, dst, args)
- copy_git_repos(pt, src, dst, args.dst_git_repo, args)
- copy_docker_images(pt, src, dst, args)
-
- pt['description'] = "Pipeline template copied from {}\n\n{}".format(
- pt_uuid,
- pt['description'] if pt.get('description', None) else '')
- pt['name'] = "{} copied from {}".format(pt.get('name', ''), pt_uuid)
- del pt['uuid']
-
- pt['owner_uuid'] = args.project_uuid
-
- return dst.pipeline_templates().create(body=pt, ensure_unique_name=True).execute(num_retries=args.retries)
# copy_workflow(wf_uuid, src, dst, args)
#
@@ -590,17 +479,16 @@ def create_collection_from(c, src, dst, args):
available."""
collection_uuid = c['uuid']
- del c['uuid']
-
- if not c["name"]:
- c['name'] = "copied from " + collection_uuid
+ body = {}
+ for d in ('description', 'manifest_text', 'name', 'portable_data_hash', 'properties'):
+ body[d] = c[d]
- if 'properties' in c:
- del c['properties']
+ if not body["name"]:
+ body['name'] = "copied from " + collection_uuid
- c['owner_uuid'] = args.project_uuid
+ body['owner_uuid'] = args.project_uuid
- dst_collection = dst.collections().create(body=c, ensure_unique_name=True).execute(num_retries=args.retries)
+ dst_collection = dst.collections().create(body=body, ensure_unique_name=True).execute(num_retries=args.retries)
# Create docker_image_repo+tag and docker_image_hash links
# at the destination.
commit 7ad5beea6c92dbb13af52a380a86f8ca1b7e0ff8
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Mar 3 16:17:38 2020 -0500
Remove Gemfile.lock from login-sync
no issue #
To be consistent with sdk/ruby sdk/cli behavior.
There's a circular dependency where the version number of the gem is
embedded in Gemfile.lock, but the version number is derived from the
git commit timestamp, so when it updates Gemfile.lock it creates a new
version, which requires a updating Gemfile.lock, and so on.
However, unlike the Ruby on Rails deb/rpm packages, when we create
gems (dev or production) it does not include Gemfile.lock, so there's
no benefit to having it checked in.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/services/login-sync/.gitignore b/services/login-sync/.gitignore
index c111b3313..369ba8ec0 100644
--- a/services/login-sync/.gitignore
+++ b/services/login-sync/.gitignore
@@ -1 +1,2 @@
*.gem
+Gemfile.lock
\ No newline at end of file
diff --git a/services/login-sync/Gemfile.lock b/services/login-sync/Gemfile.lock
deleted file mode 100644
index 28cdff6fd..000000000
--- a/services/login-sync/Gemfile.lock
+++ /dev/null
@@ -1,89 +0,0 @@
-PATH
- remote: .
- specs:
- arvados-login-sync (1.5.0.dev20200114213539)
- arvados (~> 1.3.0, >= 1.3.0)
- faraday (< 0.16)
- signet (< 0.12)
-
-GEM
- remote: https://rubygems.org/
- specs:
- activesupport (5.0.7.2)
- concurrent-ruby (~> 1.0, >= 1.0.2)
- i18n (>= 0.7, < 2)
- minitest (~> 5.1)
- tzinfo (~> 1.1)
- addressable (2.7.0)
- public_suffix (>= 2.0.2, < 5.0)
- andand (1.3.3)
- arvados (1.3.3.20190320201707)
- activesupport (>= 3)
- andand (~> 1.3, >= 1.3.3)
- arvados-google-api-client (>= 0.7, < 0.8.9)
- i18n (~> 0)
- json (>= 1.7.7, < 3)
- jwt (>= 0.1.5, < 2)
- arvados-google-api-client (0.8.7.3)
- activesupport (>= 3.2, < 5.1)
- addressable (~> 2.3)
- autoparse (~> 0.3)
- extlib (~> 0.9)
- faraday (~> 0.9)
- googleauth (~> 0.3)
- launchy (~> 2.4)
- multi_json (~> 1.10)
- retriable (~> 1.4)
- signet (~> 0.6)
- autoparse (0.3.3)
- addressable (>= 2.3.1)
- extlib (>= 0.9.15)
- multi_json (>= 1.0.0)
- concurrent-ruby (1.1.5)
- extlib (0.9.16)
- faraday (0.15.4)
- multipart-post (>= 1.2, < 3)
- googleauth (0.9.0)
- faraday (~> 0.12)
- jwt (>= 1.4, < 3.0)
- memoist (~> 0.16)
- multi_json (~> 1.11)
- os (>= 0.9, < 2.0)
- signet (~> 0.7)
- i18n (0.9.5)
- concurrent-ruby (~> 1.0)
- json (2.3.0)
- jwt (1.5.6)
- launchy (2.4.3)
- addressable (~> 2.3)
- memoist (0.16.2)
- metaclass (0.0.4)
- minitest (5.11.3)
- mocha (1.8.0)
- metaclass (~> 0.0.1)
- multi_json (1.14.1)
- multipart-post (2.1.1)
- os (1.0.1)
- public_suffix (4.0.3)
- rake (12.3.2)
- retriable (1.4.1)
- signet (0.11.0)
- addressable (~> 2.3)
- faraday (~> 0.9)
- jwt (>= 1.5, < 3.0)
- multi_json (~> 1.10)
- thread_safe (0.3.6)
- tzinfo (1.2.6)
- thread_safe (~> 0.1)
-
-PLATFORMS
- ruby
-
-DEPENDENCIES
- arvados-login-sync!
- minitest (>= 5.0.0)
- mocha (>= 1.5.0)
- rake
-
-BUNDLED WITH
- 1.11
commit 8014ce0457ef848bba8ce18ad5980b9883105cc9
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Mar 3 15:34:05 2020 -0500
Ruby gems respect ARVADOS_BUILDING_VERSION
no issue #
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cli/arvados-cli.gemspec b/sdk/cli/arvados-cli.gemspec
index 3db613c29..2cfc2f438 100644
--- a/sdk/cli/arvados-cli.gemspec
+++ b/sdk/cli/arvados-cli.gemspec
@@ -13,7 +13,11 @@ begin
ENV["GIT_DIR"] = File.expand_path "#{__dir__}/../../.git"
ENV["GIT_WORK_TREE"] = File.expand_path "#{__dir__}/../.."
git_timestamp, git_hash = `git log -n1 --first-parent --format=%ct:%H #{__dir__}`.chomp.split(":")
- version = `#{__dir__}/../../build/version-at-commit.sh #{git_hash}`.encode('utf-8').strip
+ if ENV["ARVADOS_BUILDING_VERSION"]
+ version = ENV["ARVADOS_BUILDING_VERSION"]
+ else
+ version = `#{__dir__}/../../build/version-at-commit.sh #{git_hash}`.encode('utf-8').strip
+ end
git_timestamp = Time.at(git_timestamp.to_i).utc
ensure
ENV["GIT_DIR"] = git_dir
diff --git a/sdk/ruby/arvados.gemspec b/sdk/ruby/arvados.gemspec
index 9a24414bb..019e156a5 100644
--- a/sdk/ruby/arvados.gemspec
+++ b/sdk/ruby/arvados.gemspec
@@ -13,7 +13,11 @@ begin
ENV["GIT_DIR"] = File.expand_path "#{__dir__}/../../.git"
ENV["GIT_WORK_TREE"] = File.expand_path "#{__dir__}/../.."
git_timestamp, git_hash = `git log -n1 --first-parent --format=%ct:%H #{__dir__}`.chomp.split(":")
- version = `#{__dir__}/../../build/version-at-commit.sh #{git_hash}`.encode('utf-8').strip
+ if ENV["ARVADOS_BUILDING_VERSION"]
+ version = ENV["ARVADOS_BUILDING_VERSION"]
+ else
+ version = `#{__dir__}/../../build/version-at-commit.sh #{git_hash}`.encode('utf-8').strip
+ end
git_timestamp = Time.at(git_timestamp.to_i).utc
ensure
ENV["GIT_DIR"] = git_dir
diff --git a/services/login-sync/arvados-login-sync.gemspec b/services/login-sync/arvados-login-sync.gemspec
index 2a91eed71..17f671c6a 100644
--- a/services/login-sync/arvados-login-sync.gemspec
+++ b/services/login-sync/arvados-login-sync.gemspec
@@ -13,7 +13,11 @@ begin
ENV["GIT_DIR"] = File.expand_path "#{__dir__}/../../.git"
ENV["GIT_WORK_TREE"] = File.expand_path "#{__dir__}/../.."
git_timestamp, git_hash = `git log -n1 --first-parent --format=%ct:%H #{__dir__}`.chomp.split(":")
- version = `#{__dir__}/../../build/version-at-commit.sh #{git_hash}`.encode('utf-8').strip
+ if ENV["ARVADOS_BUILDING_VERSION"]
+ version = ENV["ARVADOS_BUILDING_VERSION"]
+ else
+ version = `#{__dir__}/../../build/version-at-commit.sh #{git_hash}`.encode('utf-8').strip
+ end
git_timestamp = Time.at(git_timestamp.to_i).utc
ensure
ENV["GIT_DIR"] = git_dir
commit 7658be2456100d0ad860b42495325e32e654147f
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu Feb 27 13:56:49 2020 -0500
Nodemanager and crunchstat-summary sync with pysdk version
refs #16151
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/services/nodemanager/arvados_version.py b/services/nodemanager/arvados_version.py
index 9aabff429..0c653694f 100644
--- a/services/nodemanager/arvados_version.py
+++ b/services/nodemanager/arvados_version.py
@@ -7,8 +7,23 @@ import time
import os
import re
+SETUP_DIR = os.path.dirname(os.path.abspath(__file__))
+
+def choose_version_from():
+ sdk_ts = subprocess.check_output(
+ ['git', 'log', '--first-parent', '--max-count=1',
+ '--format=format:%ct', os.path.join(SETUP_DIR, "../../sdk/python")]).strip()
+ cwl_ts = subprocess.check_output(
+ ['git', 'log', '--first-parent', '--max-count=1',
+ '--format=format:%ct', SETUP_DIR]).strip()
+ if int(sdk_ts) > int(cwl_ts):
+ getver = os.path.join(SETUP_DIR, "../../sdk/python")
+ else:
+ getver = SETUP_DIR
+ return getver
+
def git_version_at_commit():
- curdir = os.path.dirname(os.path.abspath(__file__))
+ curdir = choose_version_from()
myhash = subprocess.check_output(['git', 'log', '-n1', '--first-parent',
'--format=%H', curdir]).strip()
myversion = subprocess.check_output([curdir+'/../../build/version-at-commit.sh', myhash]).strip().decode()
diff --git a/tools/crunchstat-summary/arvados_version.py b/tools/crunchstat-summary/arvados_version.py
index 9aabff429..0c653694f 100644
--- a/tools/crunchstat-summary/arvados_version.py
+++ b/tools/crunchstat-summary/arvados_version.py
@@ -7,8 +7,23 @@ import time
import os
import re
+SETUP_DIR = os.path.dirname(os.path.abspath(__file__))
+
+def choose_version_from():
+ sdk_ts = subprocess.check_output(
+ ['git', 'log', '--first-parent', '--max-count=1',
+ '--format=format:%ct', os.path.join(SETUP_DIR, "../../sdk/python")]).strip()
+ cwl_ts = subprocess.check_output(
+ ['git', 'log', '--first-parent', '--max-count=1',
+ '--format=format:%ct', SETUP_DIR]).strip()
+ if int(sdk_ts) > int(cwl_ts):
+ getver = os.path.join(SETUP_DIR, "../../sdk/python")
+ else:
+ getver = SETUP_DIR
+ return getver
+
def git_version_at_commit():
- curdir = os.path.dirname(os.path.abspath(__file__))
+ curdir = choose_version_from()
myhash = subprocess.check_output(['git', 'log', '-n1', '--first-parent',
'--format=%H', curdir]).strip()
myversion = subprocess.check_output([curdir+'/../../build/version-at-commit.sh', myhash]).strip().decode()
commit 85cbbadc846bca172398a8be42f49ff8de91d2e9
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Feb 28 12:07:35 2020 -0500
16169: Monkey patch load_tool.resolve_and_validate_document to fix bug
There is a bug in upstream cwltool where the version updater needs to
replace the document fragments in the loader index with the updated
ones, but actually it only does it for the root document. Normally we
just fix the bug in upstream but that's challenging because current
cwltool dropped support for Python 2.7 and we're still supporting py2
in Arvados 2.0 (although py2 support will most likely be dropped in
Arvados 2.1). Making a bugfix fork comes with its own
complications (it would need to be added to PyPi) so monkey patching
is the least disruptive fix (and is relatively safe because our
cwltool dependency is pinned to a specific version). This
should be removed as soon as a bugfix goes into upstream cwltool and
we upgrade to it.
diff --git a/sdk/cwl/arvados_cwl/__init__.py b/sdk/cwl/arvados_cwl/__init__.py
index 3dd04040a..2b2acd568 100644
--- a/sdk/cwl/arvados_cwl/__init__.py
+++ b/sdk/cwl/arvados_cwl/__init__.py
@@ -16,6 +16,43 @@ import sys
import re
import pkg_resources # part of setuptools
+### begin monkey patch ###
+# Monkey patch solution for bug #16169
+#
+# There is a bug in upstream cwltool where the version updater needs
+# to replace the document fragments in the loader index with the
+# updated ones, but actually it only does it for the root document.
+# Normally we just fix the bug in upstream but that's challenging
+# because current cwltool dropped support for Python 2.7 and we're
+# still supporting py2 in Arvados 2.0 (although py2 support will most
+# likely be dropped in Arvados 2.1). Making a bugfix fork comes with
+# its own complications (it would need to be added to PyPi) so monkey
+# patching is the least disruptive fix (and is relatively safe because
+# our cwltool dependency is pinned to a specific version). This
+# should be removed as soon as a bugfix goes into upstream cwltool and
+# we upgrade to it.
+#
+import cwltool.load_tool
+from cwltool.utils import visit_class
+from six.moves import urllib
+original_resolve_and_validate_document = cwltool.load_tool.resolve_and_validate_document
+def wrapped_resolve_and_validate_document(
+ loadingContext, # type: LoadingContext
+ workflowobj, # type: Union[CommentedMap, CommentedSeq]
+ uri, # type: Text
+ preprocess_only=False, # type: bool
+ skip_schemas=None, # type: Optional[bool]
+ ):
+ loadingContext, uri = original_resolve_and_validate_document(loadingContext, workflowobj, uri, preprocess_only, skip_schemas)
+ if loadingContext.do_update in (True, None):
+ fileuri = urllib.parse.urldefrag(uri)[0]
+ def update_index(pr):
+ loadingContext.loader.idx[pr["id"]] = pr
+ visit_class(loadingContext.loader.idx[fileuri], ("CommandLineTool", "Workflow", "ExpressionTool"), update_index)
+ return loadingContext, uri
+cwltool.load_tool.resolve_and_validate_document = wrapped_resolve_and_validate_document
+### end monkey patch ###
+
from schema_salad.sourceline import SourceLine
import schema_salad.validate as validate
import cwltool.main
diff --git a/sdk/cwl/tests/wf/16169-step.cwl b/sdk/cwl/tests/wf/16169-step.cwl
index 8386df517..ce6f2c0c9 100644
--- a/sdk/cwl/tests/wf/16169-step.cwl
+++ b/sdk/cwl/tests/wf/16169-step.cwl
@@ -6,10 +6,12 @@ class: CommandLineTool
cwlVersion: v1.0
requirements:
InlineJavascriptRequirement: {}
+ DockerRequirement:
+ dockerPull: debian:stretch-slim
inputs:
d: Directory
outputs:
out: stdout
stdout: output.txt
arguments:
- [echo, "${if(inputs.d.listing === undefined) {return 'true';} else {return 'false';}}"]
\ No newline at end of file
+ [echo, "${if(inputs.d.listing === undefined) {return 'true';} else {return 'false';}}"]
diff --git a/sdk/python/arvados/commands/keepdocker.py b/sdk/python/arvados/commands/keepdocker.py
index c89fa644c..6673888ab 100644
--- a/sdk/python/arvados/commands/keepdocker.py
+++ b/sdk/python/arvados/commands/keepdocker.py
@@ -390,7 +390,7 @@ def main(arguments=None, stdout=sys.stdout, install_sig_handlers=True, api=None)
try:
image_hash = find_one_image_hash(args.image, args.tag)
except DockerError as error:
- logger.error(error.message)
+ logger.error(str(error))
sys.exit(1)
if not docker_image_compatible(api, image_hash):
commit b6ad6faad9a0750538cdd6ed68862d6c7d772b2d
Author: Lucas Di Pentima <lucas at di-pentima.com.ar>
Date: Wed Feb 26 20:58:59 2020 -0300
16169: Adds test exposing the requirement propagation bug.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas at di-pentima.com.ar>
diff --git a/sdk/cwl/tests/16169-no-listing-hint.cwl b/sdk/cwl/tests/16169-no-listing-hint.cwl
new file mode 100644
index 000000000..fe4f991ff
--- /dev/null
+++ b/sdk/cwl/tests/16169-no-listing-hint.cwl
@@ -0,0 +1,24 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+cwlVersion: v1.0
+class: Workflow
+$namespaces:
+ arv: "http://arvados.org/cwl#"
+ cwltool: "http://commonwl.org/cwltool#"
+requirements:
+ cwltool:LoadListingRequirement:
+ loadListing: no_listing
+inputs:
+ d: Directory
+steps:
+ step1:
+ in:
+ d: d
+ out: [out]
+ run: wf/16169-step.cwl
+outputs:
+ out:
+ type: File
+ outputSource: step1/out
diff --git a/sdk/cwl/tests/arvados-tests.yml b/sdk/cwl/tests/arvados-tests.yml
index 99aee3795..df9fac842 100644
--- a/sdk/cwl/tests/arvados-tests.yml
+++ b/sdk/cwl/tests/arvados-tests.yml
@@ -310,3 +310,15 @@
should_fail: true
tool: 15295-bad-keep-ref.cwl
doc: Test checking for invalid keepref
+
+- job: listing-job.yml
+ output: {
+ "out": {
+ "class": "File",
+ "location": "output.txt",
+ "size": 5,
+ "checksum": "sha1$724ba28f4a9a1b472057ff99511ed393a45552e1"
+ }
+ }
+ tool: 16169-no-listing-hint.cwl
+ doc: "Test cwltool:LoadListingRequirement propagation"
diff --git a/sdk/cwl/tests/wf/16169-step.cwl b/sdk/cwl/tests/wf/16169-step.cwl
new file mode 100644
index 000000000..8386df517
--- /dev/null
+++ b/sdk/cwl/tests/wf/16169-step.cwl
@@ -0,0 +1,15 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+class: CommandLineTool
+cwlVersion: v1.0
+requirements:
+ InlineJavascriptRequirement: {}
+inputs:
+ d: Directory
+outputs:
+ out: stdout
+stdout: output.txt
+arguments:
+ [echo, "${if(inputs.d.listing === undefined) {return 'true';} else {return 'false';}}"]
\ No newline at end of file
commit eb444cd34bf9b57a2844fe7ca4482f50ca094ee9
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu Feb 27 12:21:34 2020 -0500
16177: Fix tests, test server config now has TrustAllContent: false
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/apps/workbench/test/controllers/collections_controller_test.rb b/apps/workbench/test/controllers/collections_controller_test.rb
index 4fce54a8a..a95b64994 100644
--- a/apps/workbench/test/controllers/collections_controller_test.rb
+++ b/apps/workbench/test/controllers/collections_controller_test.rb
@@ -532,7 +532,7 @@ class CollectionsControllerTest < ActionController::TestCase
end
test "Redirect to keep_web_url via #{id_type} when trust_all_content enabled" do
- Rails.configuration.Workbench.TrustAllContent = true
+ Rails.configuration.Collections.TrustAllContent = true
setup_for_keep_web('https://collections.example',
'https://download.example')
tok = api_token('active')
@@ -583,7 +583,7 @@ class CollectionsControllerTest < ActionController::TestCase
[false, true].each do |trust_all_content|
test "Redirect preview to keep_web_download_url when preview is disabled and trust_all_content is #{trust_all_content}" do
- Rails.configuration.Workbench.TrustAllContent = trust_all_content
+ Rails.configuration.Collections.TrustAllContent = trust_all_content
setup_for_keep_web "", 'https://download.example/'
tok = api_token('active')
id = api_fixture('collections')['w_a_z_file']['uuid']
diff --git a/sdk/python/tests/run_test_server.py b/sdk/python/tests/run_test_server.py
index 9e9b12f98..f5528081f 100644
--- a/sdk/python/tests/run_test_server.py
+++ b/sdk/python/tests/run_test_server.py
@@ -737,7 +737,7 @@ def setup_config():
},
"Collections": {
"BlobSigningKey": "zfhgfenhffzltr9dixws36j1yhksjoll2grmku38mi7yxd66h5j4q9w4jzanezacp8s6q0ro3hxakfye02152hncy6zml2ed0uc",
- "TrustAllContent": True,
+ "TrustAllContent": False,
"ForwardSlashNameSubstitution": "/",
},
"Git": {
commit 0fd5ff742d62382275f1f74d663cfc00d5e0298d
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Wed Feb 26 15:31:58 2020 -0500
16177: Use correct config item Collections.TrustAllContent
Also added fallback error handler, if view or download link doesn't
work, render error text (otherwise you get a very confusing Rails
crash page).
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/apps/workbench/app/controllers/application_controller.rb b/apps/workbench/app/controllers/application_controller.rb
index 540770061..8d6f897bb 100644
--- a/apps/workbench/app/controllers/application_controller.rb
+++ b/apps/workbench/app/controllers/application_controller.rb
@@ -62,6 +62,7 @@ class ApplicationController < ActionController::Base
# the browser can't.
f.json { render opts.merge(json: {success: false, errors: @errors}) }
f.html { render({action: 'error'}.merge(opts)) }
+ f.all { render({action: 'error', formats: 'text'}.merge(opts)) }
end
end
diff --git a/apps/workbench/app/controllers/collections_controller.rb b/apps/workbench/app/controllers/collections_controller.rb
index 10e026ae6..9073d06c1 100644
--- a/apps/workbench/app/controllers/collections_controller.rb
+++ b/apps/workbench/app/controllers/collections_controller.rb
@@ -343,7 +343,7 @@ class CollectionsController < ApplicationController
# Prefer the attachment-only-host when we want an attachment
# (and when there is no preview link configured)
tmpl = Rails.configuration.Services.WebDAVDownload.ExternalURL.to_s
- elsif not Rails.configuration.Workbench.TrustAllContent
+ elsif not Rails.configuration.Collections.TrustAllContent
check_uri = URI.parse(tmpl.sub("*", munged_id))
if opts[:query_token] and
(check_uri.host.nil? or (
diff --git a/apps/workbench/app/views/application/error.text.erb b/apps/workbench/app/views/application/error.text.erb
new file mode 100644
index 000000000..103518274
--- /dev/null
+++ b/apps/workbench/app/views/application/error.text.erb
@@ -0,0 +1,11 @@
+<%# Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: AGPL-3.0 %>
+
+Oh... fiddlesticks.
+
+Sorry, I had some trouble handling your request.
+
+<% if @errors.is_a? Array then @errors.each do |error| %>
+<%= error %>
+<% end end %>
commit 77f2e73c03d263e1cda8ca2f07a35dbc53f6dd90
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Wed Feb 26 16:12:22 2020 -0500
16202: Add Workbench.SSHHelpHostSuffix
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/apps/workbench/app/views/users/_virtual_machines.html.erb b/apps/workbench/app/views/users/_virtual_machines.html.erb
index 38458593f..e2ce5b39b 100644
--- a/apps/workbench/app/views/users/_virtual_machines.html.erb
+++ b/apps/workbench/app/views/users/_virtual_machines.html.erb
@@ -85,7 +85,7 @@ SPDX-License-Identifier: AGPL-3.0 %>
<td style="word-break:break-all;">
<% if @my_vm_logins[vm[:uuid]] %>
<% @my_vm_logins[vm[:uuid]].each do |login| %>
- <code>ssh <%= login %>@<%= vm[:hostname] %></code>
+ <code>ssh <%= login %>@<%= vm[:hostname] %><%=Rails.configuration.Workbench.SSHHelpHostSuffix%></code>
<% end %>
<% end %>
</td>
diff --git a/lib/config/config.default.yml b/lib/config/config.default.yml
index 41af15073..3750adcab 100644
--- a/lib/config/config.default.yml
+++ b/lib/config/config.default.yml
@@ -1169,6 +1169,27 @@ Clusters:
<a href="https://doc.arvados.org/user/getting_started/ssh-access-unix.html">Accessing an Arvados VM with SSH</a> (generic instructions).
Site configurations vary. Contact your local cluster administrator if you have difficulty accessing an Arvados shell node.
+ # Sample text if you are using a "switchyard" ssh proxy.
+ # Replace "zzzzz" with your Cluster ID.
+ #SSHHelpPageHTML: |
+ # <p>Add a section like this to your SSH configuration file ( <i>~/.ssh/config</i>):</p>
+ # <pre>Host *.zzzzz
+ # TCPKeepAlive yes
+ # ServerAliveInterval 60
+ # ProxyCommand ssh -p2222 turnout at switchyard.zzzzz.arvadosapi.com -x -a $SSH_PROXY_FLAGS %h
+ # </pre>
+
+ # If you are using a switchyard ssh proxy, shell node hostnames
+ # may require a special hostname suffix. In the sample ssh
+ # configuration above, this would be ".zzzzz"
+ # This is added to the hostname in the "command line" column
+ # the Workbench "shell VMs" page.
+ #
+ # If your shell nodes are directly accessible by users without a
+ # proxy and have fully qualified host names, you should leave
+ # this blank.
+ SSHHelpHostSuffix: ""
+
# Bypass new (Arvados 1.5) API implementations, and hand off
# requests directly to Rails instead. This can provide a temporary
# workaround for clients that are incompatible with the new API
diff --git a/lib/config/export.go b/lib/config/export.go
index 44c69b6e2..88794140a 100644
--- a/lib/config/export.go
+++ b/lib/config/export.go
@@ -215,6 +215,7 @@ var whitelist = map[string]bool{
"Workbench.WelcomePageHTML": true,
"Workbench.InactivePageHTML": true,
"Workbench.SSHHelpPageHTML": true,
+ "Workbench.SSHHelpHostSuffix": true,
}
func redactUnsafe(m map[string]interface{}, mPrefix, lookupPrefix string) error {
diff --git a/lib/config/generated_config.go b/lib/config/generated_config.go
index 25fa89394..6813bee40 100644
--- a/lib/config/generated_config.go
+++ b/lib/config/generated_config.go
@@ -1175,6 +1175,27 @@ Clusters:
<a href="https://doc.arvados.org/user/getting_started/ssh-access-unix.html">Accessing an Arvados VM with SSH</a> (generic instructions).
Site configurations vary. Contact your local cluster administrator if you have difficulty accessing an Arvados shell node.
+ # Sample text if you are using a "switchyard" ssh proxy.
+ # Replace "zzzzz" with your Cluster ID.
+ #SSHHelpPageHTML: |
+ # <p>Add a section like this to your SSH configuration file ( <i>~/.ssh/config</i>):</p>
+ # <pre>Host *.zzzzz
+ # TCPKeepAlive yes
+ # ServerAliveInterval 60
+ # ProxyCommand ssh -p2222 turnout at switchyard.zzzzz.arvadosapi.com -x -a $SSH_PROXY_FLAGS %h
+ # </pre>
+
+ # If you are using a switchyard ssh proxy, shell node hostnames
+ # may require a special hostname suffix. In the sample ssh
+ # configuration above, this would be ".zzzzz"
+ # This is added to the hostname in the "command line" column
+ # the Workbench "shell VMs" page.
+ #
+ # If your shell nodes are directly accessible by users without a
+ # proxy and have fully qualified host names, you should leave
+ # this blank.
+ SSHHelpHostSuffix: ""
+
# Bypass new (Arvados 1.5) API implementations, and hand off
# requests directly to Rails instead. This can provide a temporary
# workaround for clients that are incompatible with the new API
diff --git a/sdk/go/arvados/config.go b/sdk/go/arvados/config.go
index 176f1dd2a..a70980cbd 100644
--- a/sdk/go/arvados/config.go
+++ b/sdk/go/arvados/config.go
@@ -215,6 +215,7 @@ type Cluster struct {
WelcomePageHTML string
InactivePageHTML string
SSHHelpPageHTML string
+ SSHHelpHostSuffix string
}
ForceLegacyAPI14 bool
commit c5f7bfacd8d7b5e33239434fee3e98b7c364f49b
Author: Ward Vandewege <ward at jhvc.com>
Date: Wed Feb 26 13:49:28 2020 -0500
Fix web shell.
refs #16203
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at jhvc.com>
diff --git a/apps/workbench/app/controllers/virtual_machines_controller.rb b/apps/workbench/app/controllers/virtual_machines_controller.rb
index 1427e3cc7..c74377314 100644
--- a/apps/workbench/app/controllers/virtual_machines_controller.rb
+++ b/apps/workbench/app/controllers/virtual_machines_controller.rb
@@ -25,8 +25,8 @@ class VirtualMachinesController < ApplicationController
end
def webshell
- return render_not_found if Rails.configuration.Workbench.ShellInABoxURL == URI("")
- webshell_url = URI(Rails.configuration.Workbench.ShellInABoxURL)
+ return render_not_found if Rails.configuration.Services.WebShell.ExternalURL == URI("")
+ webshell_url = URI(Rails.configuration.Services.WebShell.ExternalURL)
if webshell_url.host.index("*") != nil
webshell_url.host = webshell_url.host.sub("*", @object.hostname)
else
commit c1777f201a5dee0f69f063dfb3a2287ffd789c97
Author: Tom Clegg <tom at tomclegg.ca>
Date: Tue Feb 25 12:14:32 2020 -0500
15954: Propagate remote param.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/lib/controller/handler_test.go b/lib/controller/handler_test.go
index 6b1fa1a0b..f09203f72 100644
--- a/lib/controller/handler_test.go
+++ b/lib/controller/handler_test.go
@@ -19,6 +19,7 @@ import (
"git.arvados.org/arvados.git/sdk/go/arvados"
"git.arvados.org/arvados.git/sdk/go/arvadostest"
+ "git.arvados.org/arvados.git/sdk/go/auth"
"git.arvados.org/arvados.git/sdk/go/ctxlog"
"git.arvados.org/arvados.git/sdk/go/httpserver"
"github.com/prometheus/client_golang/prometheus"
@@ -229,6 +230,26 @@ func (s *HandlerSuite) TestValidateV2APIToken(c *check.C) {
c.Check(user.Authorization.TokenV2(), check.Equals, arvadostest.ActiveTokenV2)
}
+func (s *HandlerSuite) TestValidateRemoteToken(c *check.C) {
+ saltedToken, err := auth.SaltToken(arvadostest.ActiveTokenV2, "abcde")
+ c.Assert(err, check.IsNil)
+ for _, trial := range []struct {
+ code int
+ token string
+ }{
+ {http.StatusOK, saltedToken},
+ {http.StatusUnauthorized, "bogus"},
+ } {
+ req := httptest.NewRequest("GET", "https://0.0.0.0:1/arvados/v1/users/current?remote=abcde", nil)
+ req.Header.Set("Authorization", "Bearer "+trial.token)
+ resp := httptest.NewRecorder()
+ s.handler.ServeHTTP(resp, req)
+ if !c.Check(resp.Code, check.Equals, trial.code) {
+ c.Logf("HTTP %d: %s", resp.Code, resp.Body.String())
+ }
+ }
+}
+
func (s *HandlerSuite) TestCreateAPIToken(c *check.C) {
req := httptest.NewRequest("GET", "/arvados/v1/users/current", nil)
auth, err := s.handler.(*Handler).createAPItoken(req, arvadostest.ActiveUserUUID, nil)
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index 5c8d4f629..0c5d32e8b 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -64,6 +64,7 @@ type GetOptions struct {
Select []string `json:"select"`
IncludeTrash bool `json:"include_trash"`
ForwardedFor string `json:"forwarded_for"`
+ Remote string `json:"remote"`
}
type UntrashOptions struct {
commit 97302d14f8ace76ab6abb04f50d8952330b65cea
Author: Tom Clegg <tom at tomclegg.ca>
Date: Thu Feb 20 15:43:50 2020 -0500
16101: Redirect logout to Workbench if return_to param missing.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/lib/controller/federation/login_test.go b/lib/controller/federation/login_test.go
index 3cc5cb842..1d6e12e01 100644
--- a/lib/controller/federation/login_test.go
+++ b/lib/controller/federation/login_test.go
@@ -41,26 +41,30 @@ func (s *LoginSuite) TestDeferToLoginCluster(c *check.C) {
}
func (s *LoginSuite) TestLogout(c *check.C) {
+ s.cluster.Services.Workbench1.ExternalURL = arvados.URL{Scheme: "https", Host: "workbench1.example.com"}
+ s.cluster.Services.Workbench2.ExternalURL = arvados.URL{Scheme: "https", Host: "workbench2.example.com"}
s.cluster.Login.GoogleClientID = "zzzzzzzzzzzzzz"
s.addHTTPRemote(c, "zhome", &arvadostest.APIStub{})
s.cluster.Login.LoginCluster = "zhome"
returnTo := "https://app.example.com/foo?bar"
for _, trial := range []struct {
- token string
- target string
+ token string
+ returnTo string
+ target string
}{
- {token: "", target: returnTo},
- {token: "zzzzzzzzzzzzzzzzzzzzz", target: returnTo},
- {token: "v2/zzzzz-aaaaa-aaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", target: returnTo},
- {token: "v2/zhome-aaaaa-aaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", target: "http://" + s.cluster.RemoteClusters["zhome"].Host + "/logout?" + url.Values{"return_to": {returnTo}}.Encode()},
+ {token: "", returnTo: "", target: s.cluster.Services.Workbench2.ExternalURL.String()},
+ {token: "", returnTo: returnTo, target: returnTo},
+ {token: "zzzzzzzzzzzzzzzzzzzzz", returnTo: returnTo, target: returnTo},
+ {token: "v2/zzzzz-aaaaa-aaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", returnTo: returnTo, target: returnTo},
+ {token: "v2/zhome-aaaaa-aaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", returnTo: returnTo, target: "http://" + s.cluster.RemoteClusters["zhome"].Host + "/logout?" + url.Values{"return_to": {returnTo}}.Encode()},
} {
c.Logf("trial %#v", trial)
ctx := context.Background()
if trial.token != "" {
ctx = auth.NewContext(ctx, &auth.Credentials{Tokens: []string{trial.token}})
}
- resp, err := s.fed.Logout(ctx, arvados.LogoutOptions{ReturnTo: returnTo})
+ resp, err := s.fed.Logout(ctx, arvados.LogoutOptions{ReturnTo: trial.returnTo})
c.Assert(err, check.IsNil)
c.Logf(" RedirectLocation %q", resp.RedirectLocation)
target, err := url.Parse(resp.RedirectLocation)
diff --git a/lib/controller/localdb/login.go b/lib/controller/localdb/login.go
index e96b940ef..2e50b84f4 100644
--- a/lib/controller/localdb/login.go
+++ b/lib/controller/localdb/login.go
@@ -53,7 +53,15 @@ func (ctrl *googleLoginController) getProvider() (*oidc.Provider, error) {
}
func (ctrl *googleLoginController) Logout(ctx context.Context, cluster *arvados.Cluster, railsproxy *railsProxy, opts arvados.LogoutOptions) (arvados.LogoutResponse, error) {
- return arvados.LogoutResponse{RedirectLocation: opts.ReturnTo}, nil
+ target := opts.ReturnTo
+ if target == "" {
+ if cluster.Services.Workbench2.ExternalURL.Host != "" {
+ target = cluster.Services.Workbench2.ExternalURL.String()
+ } else {
+ target = cluster.Services.Workbench1.ExternalURL.String()
+ }
+ }
+ return arvados.LogoutResponse{RedirectLocation: target}, nil
}
func (ctrl *googleLoginController) Login(ctx context.Context, cluster *arvados.Cluster, railsproxy *railsProxy, opts arvados.LoginOptions) (arvados.LoginResponse, error) {
commit cac986324a271a0e82cfb1e2bb51bafc9504eae2
Author: Tom Clegg <tom at tomclegg.ca>
Date: Mon Feb 17 13:55:27 2020 -0500
16101: Handle logout without sso-provider.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/lib/controller/federation/conn.go b/lib/controller/federation/conn.go
index 42083cb83..56f117ee7 100644
--- a/lib/controller/federation/conn.go
+++ b/lib/controller/federation/conn.go
@@ -222,6 +222,32 @@ func (conn *Conn) Login(ctx context.Context, options arvados.LoginOptions) (arva
}
}
+func (conn *Conn) Logout(ctx context.Context, options arvados.LogoutOptions) (arvados.LogoutResponse, error) {
+ // If the logout request comes with an API token from a known
+ // remote cluster, redirect to that cluster's logout handler
+ // so it has an opportunity to clear sessions, expire tokens,
+ // etc. Otherwise use the local endpoint.
+ reqauth, ok := auth.FromContext(ctx)
+ if !ok || len(reqauth.Tokens) == 0 || len(reqauth.Tokens[0]) < 8 || !strings.HasPrefix(reqauth.Tokens[0], "v2/") {
+ return conn.local.Logout(ctx, options)
+ }
+ id := reqauth.Tokens[0][3:8]
+ if id == conn.cluster.ClusterID {
+ return conn.local.Logout(ctx, options)
+ }
+ remote, ok := conn.remotes[id]
+ if !ok {
+ return conn.local.Logout(ctx, options)
+ }
+ baseURL := remote.BaseURL()
+ target, err := baseURL.Parse(arvados.EndpointLogout.Path)
+ if err != nil {
+ return arvados.LogoutResponse{}, fmt.Errorf("internal error getting redirect target: %s", err)
+ }
+ target.RawQuery = url.Values{"return_to": {options.ReturnTo}}.Encode()
+ return arvados.LogoutResponse{RedirectLocation: target.String()}, nil
+}
+
func (conn *Conn) CollectionGet(ctx context.Context, options arvados.GetOptions) (arvados.Collection, error) {
if len(options.UUID) == 27 {
// UUID is really a UUID
diff --git a/lib/controller/federation/login_test.go b/lib/controller/federation/login_test.go
index ab39619c7..3cc5cb842 100644
--- a/lib/controller/federation/login_test.go
+++ b/lib/controller/federation/login_test.go
@@ -10,6 +10,7 @@ import (
"git.arvados.org/arvados.git/sdk/go/arvados"
"git.arvados.org/arvados.git/sdk/go/arvadostest"
+ "git.arvados.org/arvados.git/sdk/go/auth"
check "gopkg.in/check.v1"
)
@@ -38,3 +39,32 @@ func (s *LoginSuite) TestDeferToLoginCluster(c *check.C) {
c.Check(remotePresent, check.Equals, remote != "")
}
}
+
+func (s *LoginSuite) TestLogout(c *check.C) {
+ s.cluster.Login.GoogleClientID = "zzzzzzzzzzzzzz"
+ s.addHTTPRemote(c, "zhome", &arvadostest.APIStub{})
+ s.cluster.Login.LoginCluster = "zhome"
+
+ returnTo := "https://app.example.com/foo?bar"
+ for _, trial := range []struct {
+ token string
+ target string
+ }{
+ {token: "", target: returnTo},
+ {token: "zzzzzzzzzzzzzzzzzzzzz", target: returnTo},
+ {token: "v2/zzzzz-aaaaa-aaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", target: returnTo},
+ {token: "v2/zhome-aaaaa-aaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", target: "http://" + s.cluster.RemoteClusters["zhome"].Host + "/logout?" + url.Values{"return_to": {returnTo}}.Encode()},
+ } {
+ c.Logf("trial %#v", trial)
+ ctx := context.Background()
+ if trial.token != "" {
+ ctx = auth.NewContext(ctx, &auth.Credentials{Tokens: []string{trial.token}})
+ }
+ resp, err := s.fed.Logout(ctx, arvados.LogoutOptions{ReturnTo: returnTo})
+ c.Assert(err, check.IsNil)
+ c.Logf(" RedirectLocation %q", resp.RedirectLocation)
+ target, err := url.Parse(resp.RedirectLocation)
+ c.Check(err, check.IsNil)
+ c.Check(target.String(), check.Equals, trial.target)
+ }
+}
diff --git a/lib/controller/handler.go b/lib/controller/handler.go
index 935a1b6cb..e3869261a 100644
--- a/lib/controller/handler.go
+++ b/lib/controller/handler.go
@@ -86,6 +86,7 @@ func (h *Handler) setup() {
mux.Handle("/arvados/v1/users", rtr)
mux.Handle("/arvados/v1/users/", rtr)
mux.Handle("/login", rtr)
+ mux.Handle("/logout", rtr)
}
hs := http.NotFoundHandler()
diff --git a/lib/controller/handler_test.go b/lib/controller/handler_test.go
index d54f50cf1..6b1fa1a0b 100644
--- a/lib/controller/handler_test.go
+++ b/lib/controller/handler_test.go
@@ -180,6 +180,32 @@ func (s *HandlerSuite) TestProxyRedirect(c *check.C) {
c.Check(resp.Header().Get("Location"), check.Matches, `(https://0.0.0.0:1)?/auth/joshid\?return_to=%2Cfoo&?`)
}
+func (s *HandlerSuite) TestLogoutSSO(c *check.C) {
+ s.cluster.Login.ProviderAppID = "test"
+ req := httptest.NewRequest("GET", "https://0.0.0.0:1/logout?return_to=https://example.com/foo", nil)
+ resp := httptest.NewRecorder()
+ s.handler.ServeHTTP(resp, req)
+ if !c.Check(resp.Code, check.Equals, http.StatusFound) {
+ c.Log(resp.Body.String())
+ }
+ c.Check(resp.Header().Get("Location"), check.Equals, "http://localhost:3002/users/sign_out?"+url.Values{"redirect_uri": {"https://example.com/foo"}}.Encode())
+}
+
+func (s *HandlerSuite) TestLogoutGoogle(c *check.C) {
+ if s.cluster.ForceLegacyAPI14 {
+ // Google login N/A
+ return
+ }
+ s.cluster.Login.GoogleClientID = "test"
+ req := httptest.NewRequest("GET", "https://0.0.0.0:1/logout?return_to=https://example.com/foo", nil)
+ resp := httptest.NewRecorder()
+ s.handler.ServeHTTP(resp, req)
+ if !c.Check(resp.Code, check.Equals, http.StatusFound) {
+ c.Log(resp.Body.String())
+ }
+ c.Check(resp.Header().Get("Location"), check.Equals, "https://example.com/foo")
+}
+
func (s *HandlerSuite) TestValidateV1APIToken(c *check.C) {
req := httptest.NewRequest("GET", "/arvados/v1/users/current", nil)
user, ok, err := s.handler.(*Handler).validateAPItoken(req, arvadostest.ActiveToken)
diff --git a/lib/controller/localdb/conn.go b/lib/controller/localdb/conn.go
index 4139b270c..ac092382d 100644
--- a/lib/controller/localdb/conn.go
+++ b/lib/controller/localdb/conn.go
@@ -29,6 +29,15 @@ func NewConn(cluster *arvados.Cluster) *Conn {
}
}
+func (conn *Conn) Logout(ctx context.Context, opts arvados.LogoutOptions) (arvados.LogoutResponse, error) {
+ if conn.cluster.Login.ProviderAppID != "" {
+ // Proxy to RailsAPI, which hands off to sso-provider.
+ return conn.railsProxy.Logout(ctx, opts)
+ } else {
+ return conn.googleLoginController.Logout(ctx, conn.cluster, conn.railsProxy, opts)
+ }
+}
+
func (conn *Conn) Login(ctx context.Context, opts arvados.LoginOptions) (arvados.LoginResponse, error) {
wantGoogle := conn.cluster.Login.GoogleClientID != ""
wantSSO := conn.cluster.Login.ProviderAppID != ""
diff --git a/lib/controller/localdb/login.go b/lib/controller/localdb/login.go
index b1ebb27e4..e96b940ef 100644
--- a/lib/controller/localdb/login.go
+++ b/lib/controller/localdb/login.go
@@ -52,6 +52,10 @@ func (ctrl *googleLoginController) getProvider() (*oidc.Provider, error) {
return ctrl.provider, nil
}
+func (ctrl *googleLoginController) Logout(ctx context.Context, cluster *arvados.Cluster, railsproxy *railsProxy, opts arvados.LogoutOptions) (arvados.LogoutResponse, error) {
+ return arvados.LogoutResponse{RedirectLocation: opts.ReturnTo}, nil
+}
+
func (ctrl *googleLoginController) Login(ctx context.Context, cluster *arvados.Cluster, railsproxy *railsProxy, opts arvados.LoginOptions) (arvados.LoginResponse, error) {
provider, err := ctrl.getProvider()
if err != nil {
diff --git a/lib/controller/localdb/login_test.go b/lib/controller/localdb/login_test.go
index 9f3267cef..4fb0fbcee 100644
--- a/lib/controller/localdb/login_test.go
+++ b/lib/controller/localdb/login_test.go
@@ -163,6 +163,12 @@ func (s *LoginSuite) TearDownTest(c *check.C) {
s.railsSpy.Close()
}
+func (s *LoginSuite) TestGoogleLogout(c *check.C) {
+ resp, err := s.localdb.Logout(context.Background(), arvados.LogoutOptions{ReturnTo: "https://foo.example.com/bar"})
+ c.Check(err, check.IsNil)
+ c.Check(resp.RedirectLocation, check.Equals, "https://foo.example.com/bar")
+}
+
func (s *LoginSuite) TestGoogleLogin_Start_Bogus(c *check.C) {
resp, err := s.localdb.Login(context.Background(), arvados.LoginOptions{})
c.Check(err, check.IsNil)
diff --git a/lib/controller/router/router.go b/lib/controller/router/router.go
index c41f103dc..69d707703 100644
--- a/lib/controller/router/router.go
+++ b/lib/controller/router/router.go
@@ -54,6 +54,13 @@ func (rtr *router) addRoutes() {
return rtr.fed.Login(ctx, *opts.(*arvados.LoginOptions))
},
},
+ {
+ arvados.EndpointLogout,
+ func() interface{} { return &arvados.LogoutOptions{} },
+ func(ctx context.Context, opts interface{}) (interface{}, error) {
+ return rtr.fed.Logout(ctx, *opts.(*arvados.LogoutOptions))
+ },
+ },
{
arvados.EndpointCollectionCreate,
func() interface{} { return &arvados.CreateOptions{} },
diff --git a/lib/controller/rpc/conn.go b/lib/controller/rpc/conn.go
index bf6166f44..4b143b770 100644
--- a/lib/controller/rpc/conn.go
+++ b/lib/controller/rpc/conn.go
@@ -145,6 +145,14 @@ func (conn *Conn) Login(ctx context.Context, options arvados.LoginOptions) (arva
return resp, err
}
+func (conn *Conn) Logout(ctx context.Context, options arvados.LogoutOptions) (arvados.LogoutResponse, error) {
+ ep := arvados.EndpointLogout
+ var resp arvados.LogoutResponse
+ err := conn.requestAndDecode(ctx, &resp, ep, nil, options)
+ resp.RedirectLocation = conn.relativeToBaseURL(resp.RedirectLocation)
+ return resp, err
+}
+
// If the given location is a valid URL and its origin is the same as
// conn.baseURL, return it as a relative URL. Otherwise, return it
// unmodified.
diff --git a/lib/controller/rpc/conn_test.go b/lib/controller/rpc/conn_test.go
index 83a80e878..b97c0f87b 100644
--- a/lib/controller/rpc/conn_test.go
+++ b/lib/controller/rpc/conn_test.go
@@ -51,6 +51,16 @@ func (s *RPCSuite) TestLogin(c *check.C) {
c.Check(resp.RedirectLocation, check.Equals, "/auth/joshid?return_to="+url.QueryEscape(","+opts.ReturnTo))
}
+func (s *RPCSuite) TestLogout(c *check.C) {
+ s.ctx = context.Background()
+ opts := arvados.LogoutOptions{
+ ReturnTo: "https://foo.example.com/bar",
+ }
+ resp, err := s.conn.Logout(s.ctx, opts)
+ c.Check(err, check.IsNil)
+ c.Check(resp.RedirectLocation, check.Equals, "http://localhost:3002/users/sign_out?redirect_uri="+url.QueryEscape(opts.ReturnTo))
+}
+
func (s *RPCSuite) TestCollectionCreate(c *check.C) {
coll, err := s.conn.CollectionCreate(s.ctx, arvados.CreateOptions{Attrs: map[string]interface{}{
"owner_uuid": arvadostest.ActiveUserUUID,
diff --git a/sdk/go/arvados/api.go b/sdk/go/arvados/api.go
index aa670c539..5c8d4f629 100644
--- a/sdk/go/arvados/api.go
+++ b/sdk/go/arvados/api.go
@@ -19,6 +19,7 @@ type APIEndpoint struct {
var (
EndpointConfigGet = APIEndpoint{"GET", "arvados/v1/config", ""}
EndpointLogin = APIEndpoint{"GET", "login", ""}
+ EndpointLogout = APIEndpoint{"GET", "logout", ""}
EndpointCollectionCreate = APIEndpoint{"POST", "arvados/v1/collections", "collection"}
EndpointCollectionUpdate = APIEndpoint{"PATCH", "arvados/v1/collections/{uuid}", "collection"}
EndpointCollectionGet = APIEndpoint{"GET", "arvados/v1/collections/{uuid}", ""}
@@ -140,9 +141,14 @@ type LoginOptions struct {
State string `json:"state,omitempty"` // OAuth2 callback state
}
+type LogoutOptions struct {
+ ReturnTo string `json:"return_to"` // Redirect to this URL after logging out
+}
+
type API interface {
ConfigGet(ctx context.Context) (json.RawMessage, error)
Login(ctx context.Context, options LoginOptions) (LoginResponse, error)
+ Logout(ctx context.Context, options LogoutOptions) (LogoutResponse, error)
CollectionCreate(ctx context.Context, options CreateOptions) (Collection, error)
CollectionUpdate(ctx context.Context, options UpdateOptions) (Collection, error)
CollectionGet(ctx context.Context, options GetOptions) (Collection, error)
diff --git a/sdk/go/arvados/login.go b/sdk/go/arvados/login.go
index 7107ac57a..75ebc81c1 100644
--- a/sdk/go/arvados/login.go
+++ b/sdk/go/arvados/login.go
@@ -24,3 +24,12 @@ func (resp LoginResponse) ServeHTTP(w http.ResponseWriter, req *http.Request) {
w.Write(resp.HTML.Bytes())
}
}
+
+type LogoutResponse struct {
+ RedirectLocation string
+}
+
+func (resp LogoutResponse) ServeHTTP(w http.ResponseWriter, req *http.Request) {
+ w.Header().Set("Location", resp.RedirectLocation)
+ w.WriteHeader(http.StatusFound)
+}
diff --git a/sdk/go/arvadostest/api.go b/sdk/go/arvadostest/api.go
index b5cea5c18..9019d33cf 100644
--- a/sdk/go/arvadostest/api.go
+++ b/sdk/go/arvadostest/api.go
@@ -37,6 +37,10 @@ func (as *APIStub) Login(ctx context.Context, options arvados.LoginOptions) (arv
as.appendCall(as.Login, ctx, options)
return arvados.LoginResponse{}, as.Error
}
+func (as *APIStub) Logout(ctx context.Context, options arvados.LogoutOptions) (arvados.LogoutResponse, error) {
+ as.appendCall(as.Logout, ctx, options)
+ return arvados.LogoutResponse{}, as.Error
+}
func (as *APIStub) CollectionCreate(ctx context.Context, options arvados.CreateOptions) (arvados.Collection, error) {
as.appendCall(as.CollectionCreate, ctx, options)
return arvados.Collection{}, as.Error
commit 8da425a41233ce425f84e0e78166ac97358a5417
Author: Tom Clegg <tom at tomclegg.ca>
Date: Sat Feb 15 16:47:34 2020 -0500
16100: Move test to integration suite to avoid logging errors.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/keep-web/handler_test.go b/services/keep-web/handler_test.go
index 488c91cdc..f6f3de887 100644
--- a/services/keep-web/handler_test.go
+++ b/services/keep-web/handler_test.go
@@ -41,24 +41,6 @@ func (s *UnitSuite) SetUpTest(c *check.C) {
s.Config = cfg
}
-func (s *UnitSuite) TestKeepClientBlockCache(c *check.C) {
- cfg := newConfig(s.Config)
- cfg.cluster.Collections.WebDAVCache.MaxBlockEntries = 42
- h := handler{Config: cfg}
- c.Check(keepclient.DefaultBlockCache.MaxBlocks, check.Not(check.Equals), cfg.cluster.Collections.WebDAVCache.MaxBlockEntries)
- u := mustParseURL("http://keep-web.example/c=" + arvadostest.FooCollection + "/t=" + arvadostest.ActiveToken + "/foo")
- req := &http.Request{
- Method: "GET",
- Host: u.Host,
- URL: u,
- RequestURI: u.RequestURI(),
- }
- resp := httptest.NewRecorder()
- h.ServeHTTP(resp, req)
- c.Check(resp.Code, check.Equals, http.StatusOK)
- c.Check(keepclient.DefaultBlockCache.MaxBlocks, check.Equals, cfg.cluster.Collections.WebDAVCache.MaxBlockEntries)
-}
-
func (s *UnitSuite) TestCORSPreflight(c *check.C) {
h := handler{Config: newConfig(s.Config)}
u := mustParseURL("http://keep-web.example/c=" + arvadostest.FooCollection + "/foo")
@@ -994,6 +976,22 @@ func (s *IntegrationSuite) TestFileContentType(c *check.C) {
}
}
+func (s *IntegrationSuite) TestKeepClientBlockCache(c *check.C) {
+ s.testServer.Config.cluster.Collections.WebDAVCache.MaxBlockEntries = 42
+ c.Check(keepclient.DefaultBlockCache.MaxBlocks, check.Not(check.Equals), 42)
+ u := mustParseURL("http://keep-web.example/c=" + arvadostest.FooCollection + "/t=" + arvadostest.ActiveToken + "/foo")
+ req := &http.Request{
+ Method: "GET",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ }
+ resp := httptest.NewRecorder()
+ s.testServer.Handler.ServeHTTP(resp, req)
+ c.Check(resp.Code, check.Equals, http.StatusOK)
+ c.Check(keepclient.DefaultBlockCache.MaxBlocks, check.Equals, 42)
+}
+
func copyHeader(h http.Header) http.Header {
hc := http.Header{}
for k, v := range h {
commit c229744b5941ca76138d527578dcc20cd98ce1c6
Author: Tom Clegg <tom at tomclegg.ca>
Date: Fri Feb 14 13:40:01 2020 -0500
16100: Add package dependency for /etc/mime.types.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/keep-web/fpm-info.sh b/services/keep-web/fpm-info.sh
new file mode 100644
index 000000000..6bcbf67fe
--- /dev/null
+++ b/services/keep-web/fpm-info.sh
@@ -0,0 +1,12 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+case "$TARGET" in
+ centos*)
+ fpm_depends+=(mailcap)
+ ;;
+ debian* | ubuntu*)
+ fpm_depends+=(mime-support)
+ ;;
+esac
commit ff0bf85aeaf3eeaec8465394fb3748e06cfc2ac4
Author: Tom Clegg <tom at tomclegg.ca>
Date: Fri Feb 14 13:39:38 2020 -0500
16100: Log a warning if /etc/mime.types is missing.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/keep-web/main.go b/services/keep-web/main.go
index f028cca22..e4028842f 100644
--- a/services/keep-web/main.go
+++ b/services/keep-web/main.go
@@ -7,6 +7,7 @@ package main
import (
"flag"
"fmt"
+ "mime"
"os"
"git.arvados.org/arvados.git/lib/config"
@@ -104,6 +105,10 @@ func main() {
log.Printf("keep-web %s started", version)
+ if ext := ".txt"; mime.TypeByExtension(ext) == "" {
+ log.Warnf("cannot look up MIME type for %q -- this probably means /etc/mime.types is missing -- clients will see incorrect content types", ext)
+ }
+
os.Setenv("ARVADOS_API_HOST", cfg.cluster.Services.Controller.ExternalURL.Host)
srv := &server{Config: cfg}
if err := srv.Start(); err != nil {
commit da2385723280b6deb1fcb58ecfbbf7cf952e930e
Author: Tom Clegg <tom at tomclegg.ca>
Date: Fri Feb 14 13:25:30 2020 -0500
16100: Test content-type detection.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/keep-web/handler_test.go b/services/keep-web/handler_test.go
index 29bcdac1b..488c91cdc 100644
--- a/services/keep-web/handler_test.go
+++ b/services/keep-web/handler_test.go
@@ -19,6 +19,7 @@ import (
"git.arvados.org/arvados.git/lib/config"
"git.arvados.org/arvados.git/sdk/go/arvados"
+ "git.arvados.org/arvados.git/sdk/go/arvadosclient"
"git.arvados.org/arvados.git/sdk/go/arvadostest"
"git.arvados.org/arvados.git/sdk/go/auth"
"git.arvados.org/arvados.git/sdk/go/ctxlog"
@@ -933,6 +934,66 @@ func (s *IntegrationSuite) TestHealthCheckPing(c *check.C) {
c.Check(resp.Body.String(), check.Matches, `{"health":"OK"}\n`)
}
+func (s *IntegrationSuite) TestFileContentType(c *check.C) {
+ s.testServer.Config.cluster.Services.WebDAVDownload.ExternalURL.Host = "download.example.com"
+
+ client := s.testServer.Config.Client
+ client.AuthToken = arvadostest.ActiveToken
+ arv, err := arvadosclient.New(&client)
+ c.Assert(err, check.Equals, nil)
+ kc, err := keepclient.MakeKeepClient(arv)
+ c.Assert(err, check.Equals, nil)
+
+ fs, err := (&arvados.Collection{}).FileSystem(&client, kc)
+ c.Assert(err, check.IsNil)
+
+ trials := []struct {
+ filename string
+ content string
+ contentType string
+ }{
+ {"picture.txt", "BMX bikes are small this year\n", "text/plain; charset=utf-8"},
+ {"picture.bmp", "BMX bikes are small this year\n", "image/x-ms-bmp"},
+ {"picture.jpg", "BMX bikes are small this year\n", "image/jpeg"},
+ {"picture1", "BMX bikes are small this year\n", "image/bmp"}, // content sniff; "BM" is the magic signature for .bmp
+ {"picture2", "Cars are small this year\n", "text/plain; charset=utf-8"}, // content sniff
+ }
+ for _, trial := range trials {
+ f, err := fs.OpenFile(trial.filename, os.O_CREATE|os.O_WRONLY, 0777)
+ c.Assert(err, check.IsNil)
+ _, err = f.Write([]byte(trial.content))
+ c.Assert(err, check.IsNil)
+ c.Assert(f.Close(), check.IsNil)
+ }
+ mtxt, err := fs.MarshalManifest(".")
+ c.Assert(err, check.IsNil)
+ var coll arvados.Collection
+ err = client.RequestAndDecode(&coll, "POST", "arvados/v1/collections", nil, map[string]interface{}{
+ "collection": map[string]string{
+ "manifest_text": mtxt,
+ },
+ })
+ c.Assert(err, check.IsNil)
+
+ for _, trial := range trials {
+ u, _ := url.Parse("http://download.example.com/by_id/" + coll.UUID + "/" + trial.filename)
+ req := &http.Request{
+ Method: "GET",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ Header: http.Header{
+ "Authorization": {"Bearer " + client.AuthToken},
+ },
+ }
+ resp := httptest.NewRecorder()
+ s.testServer.Handler.ServeHTTP(resp, req)
+ c.Check(resp.Code, check.Equals, http.StatusOK)
+ c.Check(resp.Header().Get("Content-Type"), check.Equals, trial.contentType)
+ c.Check(resp.Body.String(), check.Equals, trial.content)
+ }
+}
+
func copyHeader(h http.Header) http.Header {
hc := http.Header{}
for k, v := range h {
commit a4b2ddad996a6ef81dcbc852bb03d319422f93b3
Author: Tom Clegg <tom at tomclegg.ca>
Date: Thu Feb 13 02:29:22 2020 -0500
16100: Fix bogus timing stats for empty responses.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/sdk/go/httpserver/logger.go b/sdk/go/httpserver/logger.go
index 30f5e2612..8886f9517 100644
--- a/sdk/go/httpserver/logger.go
+++ b/sdk/go/httpserver/logger.go
@@ -68,10 +68,16 @@ func logRequest(w *responseTimer, req *http.Request, lgr *logrus.Entry) {
func logResponse(w *responseTimer, req *http.Request, lgr *logrus.Entry) {
if tStart, ok := req.Context().Value(&requestTimeContextKey).(time.Time); ok {
tDone := time.Now()
+ writeTime := w.writeTime
+ if !w.wrote {
+ // Empty response body. Header was sent when
+ // handler exited.
+ writeTime = tDone
+ }
lgr = lgr.WithFields(logrus.Fields{
"timeTotal": stats.Duration(tDone.Sub(tStart)),
- "timeToStatus": stats.Duration(w.writeTime.Sub(tStart)),
- "timeWriteBody": stats.Duration(tDone.Sub(w.writeTime)),
+ "timeToStatus": stats.Duration(writeTime.Sub(tStart)),
+ "timeWriteBody": stats.Duration(tDone.Sub(writeTime)),
})
}
respCode := w.WroteStatus()
commit 137f2e31166c454d4536ba713347e8af3eb0176b
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed Feb 12 21:50:07 2020 -0500
16039: Touch python clients so they run tests with the latest SDK.
Otherwise, "pip install" (during "install services/nodemanager", for
example) downgrades the SDK from the current version to the latest
version published on pip, and all test suites run with that version,
instead of the version we think we're testing.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/sdk/cwl/setup.py b/sdk/cwl/setup.py
index 62ceab2fa..d4bb6d102 100644
--- a/sdk/cwl/setup.py
+++ b/sdk/cwl/setup.py
@@ -36,7 +36,8 @@ setup(name='arvados-cwl-runner',
'bin/arvados-cwl-runner',
],
# Note that arvados/build/run-build-packages.sh looks at this
- # file to determine what version of cwltool and schema-salad to build.
+ # file to determine what version of cwltool and schema-salad to
+ # build.
install_requires=[
'cwltool==1.0.20190831161204',
'schema-salad==4.5.20190815125611',
@@ -63,5 +64,5 @@ setup(name='arvados-cwl-runner',
'mock>=1.0,<4',
'subprocess32>=3.5.1',
],
- zip_safe=True
- )
+ zip_safe=True,
+)
diff --git a/sdk/pam/setup.py b/sdk/pam/setup.py
index af00142a0..59b49a19f 100755
--- a/sdk/pam/setup.py
+++ b/sdk/pam/setup.py
@@ -53,5 +53,5 @@ setup(name='arvados-pam',
],
test_suite='tests',
tests_require=['pbr<1.7.0', 'mock>=1.0', 'python-pam'],
- zip_safe=False
- )
+ zip_safe=False,
+)
diff --git a/services/nodemanager/setup.py b/services/nodemanager/setup.py
index a2b9a0ca9..75e8f85fb 100644
--- a/services/nodemanager/setup.py
+++ b/services/nodemanager/setup.py
@@ -56,5 +56,5 @@ setup(name='arvados-node-manager',
'apache-libcloud==2.5.0',
'subprocess32>=3.5.1',
],
- zip_safe=False
- )
+ zip_safe=False,
+)
diff --git a/tools/crunchstat-summary/setup.py b/tools/crunchstat-summary/setup.py
index 40c5a2f9a..557b6d3f4 100755
--- a/tools/crunchstat-summary/setup.py
+++ b/tools/crunchstat-summary/setup.py
@@ -42,5 +42,5 @@ setup(name='crunchstat_summary',
],
test_suite='tests',
tests_require=['pbr<1.7.0', 'mock>=1.0'],
- zip_safe=False
- )
+ zip_safe=False,
+)
commit cfd64e9f9714d755a80e96b71b7ab796c0710872
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed Feb 12 21:47:44 2020 -0500
16039: Install missing python dep.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/build/run-tests.sh b/build/run-tests.sh
index 43dd3f4eb..c4c533559 100755
--- a/build/run-tests.sh
+++ b/build/run-tests.sh
@@ -647,8 +647,8 @@ install_env() {
. "$VENVDIR/bin/activate"
# Needed for run_test_server.py which is used by certain (non-Python) tests.
- pip install --no-cache-dir PyYAML future \
- || fatal "pip install PyYAML failed"
+ pip install --no-cache-dir PyYAML future httplib2 \
+ || fatal "`pip install PyYAML future httplib2` failed"
# Preinstall libcloud if using a fork; otherwise nodemanager "pip
# install" won't pick it up by default.
commit 2ad995236f3584635b7a2e80c62a31323b9e65b8
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed Feb 12 13:49:50 2020 -0500
16039: Use current python sdk in py3 tests.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/build/run-tests.sh b/build/run-tests.sh
index f21861762..43dd3f4eb 100755
--- a/build/run-tests.sh
+++ b/build/run-tests.sh
@@ -1099,6 +1099,7 @@ install_deps() {
do_install sdk/cli
do_install sdk/perl
do_install sdk/python pip
+ do_install sdk/python pip3
do_install sdk/ruby
do_install services/api
do_install services/arv-git-httpd go
commit be568b1c98420f920c9f602d7e1aa47d401bcd39
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed Feb 12 10:48:05 2020 -0500
16039: Fix & add test for sanitized/unsanitized name conflict.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/services/fuse/arvados_fuse/fusedir.py b/services/fuse/arvados_fuse/fusedir.py
index 3f6430973..8b12f73e8 100644
--- a/services/fuse/arvados_fuse/fusedir.py
+++ b/services/fuse/arvados_fuse/fusedir.py
@@ -941,7 +941,7 @@ class ProjectDirectory(Directory):
namefilter],
limit=2).execute(num_retries=self.num_retries)["items"]
if contents:
- if len(contents) > 1 and contents[1].name == k:
+ if len(contents) > 1 and contents[1]['name'] == k:
# If "foo/bar" and "foo[SUBST]bar" both exist, use
# "foo[SUBST]bar".
contents = [contents[1]]
diff --git a/services/fuse/tests/test_mount.py b/services/fuse/tests/test_mount.py
index 1e63b9f4d..593d945cf 100644
--- a/services/fuse/tests/test_mount.py
+++ b/services/fuse/tests/test_mount.py
@@ -1239,3 +1239,17 @@ class SlashSubstitutionTest(IntegrationTest):
def checkContents(self):
self.assertRegexpMatches(self.api.collections().get(uuid=self.testcoll['uuid']).execute()['manifest_text'], ' acbd18db') # md5(foo)
self.assertRegexpMatches(self.api.collections().get(uuid=self.testcolleasy['uuid']).execute()['manifest_text'], ' f561aaf6') # md5(xxx)
+
+ @IntegrationTest.mount(argv=mnt_args)
+ @mock.patch('arvados.util.get_config_once')
+ def test_slash_substitution_conflict(self, get_config_once):
+ self.testcollconflict = self.api.collections().create(body={"name": self.fusename}).execute()
+ get_config_once.return_value = {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
+ self.pool_test(os.path.join(self.mnt, 'zzz'), self.fusename)
+ self.assertRegexpMatches(self.api.collections().get(uuid=self.testcollconflict['uuid']).execute()['manifest_text'], ' acbd18db') # md5(foo)
+ # foo/bar/baz collection unchanged, because it is masked by foo[SLASH]bar[SLASH]baz
+ self.assertEqual(self.api.collections().get(uuid=self.testcoll['uuid']).execute()['manifest_text'], '')
+ @staticmethod
+ def _test_slash_substitution_conflict(self, tmpdir, fusename):
+ with open(os.path.join(tmpdir, fusename, 'waz'), 'w') as f:
+ f.write('foo')
commit 8a7a749b027a115c2cef269ab7fdfc85f40c2f63
Author: Tom Clegg <tom at tomclegg.ca>
Date: Wed Feb 12 10:37:48 2020 -0500
16039: Accommodate API servers with no config export.
Add more checks in test cases.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/sdk/python/arvados/util.py b/sdk/python/arvados/util.py
index b27b64e59..9e0a31783 100644
--- a/sdk/python/arvados/util.py
+++ b/sdk/python/arvados/util.py
@@ -421,6 +421,9 @@ def new_request_id():
return rid
def get_config_once(svc):
+ if not svc._rootDesc.get('resources')['configs']:
+ # Old API server version, no config export endpoint
+ return {}
if not hasattr(svc, '_cached_config'):
svc._cached_config = svc.configs().get().execute()
return svc._cached_config
diff --git a/services/fuse/arvados_fuse/fusedir.py b/services/fuse/arvados_fuse/fusedir.py
index 9853f6ac2..3f6430973 100644
--- a/services/fuse/arvados_fuse/fusedir.py
+++ b/services/fuse/arvados_fuse/fusedir.py
@@ -55,10 +55,24 @@ class Directory(FreshBase):
self._entries = {}
self._mtime = time.time()
+ def forward_slash_subst(self):
+ if not hasattr(self, '_fsns'):
+ self._fsns = None
+ config = self.apiconfig()
+ try:
+ self._fsns = config["Collections"]["ForwardSlashNameSubstitution"]
+ except KeyError:
+ # old API server with no FSNS config
+ self._fsns = '_'
+ else:
+ if self._fsns == '' or self._fsns == '/':
+ self._fsns = None
+ return self._fsns
+
def unsanitize_filename(self, incoming):
"""Replace ForwardSlashNameSubstitution value with /"""
- fsns = self.apiconfig()["Collections"]["ForwardSlashNameSubstitution"]
- if isinstance(fsns, str) and fsns != '' and fsns != '/':
+ fsns = self.forward_slash_subst()
+ if isinstance(fsns, str):
return incoming.replace(fsns, '/')
else:
return incoming
@@ -76,8 +90,8 @@ class Directory(FreshBase):
elif dirty == '..':
return '__'
else:
- fsns = self.apiconfig()["Collections"]["ForwardSlashNameSubstitution"]
- if isinstance(fsns, str) and fsns != '':
+ fsns = self.forward_slash_subst()
+ if isinstance(fsns, str):
dirty = dirty.replace('/', fsns)
return _disallowed_filename_characters.sub('_', dirty)
diff --git a/services/fuse/tests/test_mount.py b/services/fuse/tests/test_mount.py
index 3ec03fa22..1e63b9f4d 100644
--- a/services/fuse/tests/test_mount.py
+++ b/services/fuse/tests/test_mount.py
@@ -1214,10 +1214,11 @@ class SlashSubstitutionTest(IntegrationTest):
def test_slash_substitution_before_listing(self, get_config_once):
get_config_once.return_value = {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
self.pool_test(os.path.join(self.mnt, 'zzz'), self.fusename)
+ self.checkContents()
@staticmethod
def _test_slash_substitution_before_listing(self, tmpdir, fusename):
with open(os.path.join(tmpdir, 'foo-bar-baz', 'waz'), 'w') as f:
- f.write('foo')
+ f.write('xxx')
with open(os.path.join(tmpdir, fusename, 'waz'), 'w') as f:
f.write('foo')
@@ -1226,10 +1227,15 @@ class SlashSubstitutionTest(IntegrationTest):
def test_slash_substitution_after_listing(self, get_config_once):
get_config_once.return_value = {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
self.pool_test(os.path.join(self.mnt, 'zzz'), self.fusename)
+ self.checkContents()
@staticmethod
def _test_slash_substitution_after_listing(self, tmpdir, fusename):
with open(os.path.join(tmpdir, 'foo-bar-baz', 'waz'), 'w') as f:
- f.write('foo')
+ f.write('xxx')
os.listdir(tmpdir)
with open(os.path.join(tmpdir, fusename, 'waz'), 'w') as f:
f.write('foo')
+
+ def checkContents(self):
+ self.assertRegexpMatches(self.api.collections().get(uuid=self.testcoll['uuid']).execute()['manifest_text'], ' acbd18db') # md5(foo)
+ self.assertRegexpMatches(self.api.collections().get(uuid=self.testcolleasy['uuid']).execute()['manifest_text'], ' f561aaf6') # md5(xxx)
commit 503160684305dccec44ad4cf309893404bf817e1
Author: Tom Clegg <tom at tomclegg.ca>
Date: Tue Feb 11 10:58:28 2020 -0500
16039: Obey ForwardSlashNameSubstitution config in arv-mount.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at tomclegg.ca>
diff --git a/sdk/python/arvados/api.py b/sdk/python/arvados/api.py
index b18ce25fd..ae687c50b 100644
--- a/sdk/python/arvados/api.py
+++ b/sdk/python/arvados/api.py
@@ -237,6 +237,7 @@ def api(version=None, cache=True, host=None, token=None, insecure=False,
svc.api_token = token
svc.insecure = insecure
svc.request_id = request_id
+ svc.config = lambda: util.get_config_once(svc)
kwargs['http'].max_request_size = svc._rootDesc.get('maxRequestSize', 0)
kwargs['http'].cache = None
kwargs['http']._request_id = lambda: svc.request_id or util.new_request_id()
diff --git a/sdk/python/arvados/util.py b/sdk/python/arvados/util.py
index fd29a3dc1..b27b64e59 100644
--- a/sdk/python/arvados/util.py
+++ b/sdk/python/arvados/util.py
@@ -419,3 +419,8 @@ def new_request_id():
rid += chr(c+ord('a')-10)
n = n // 36
return rid
+
+def get_config_once(svc):
+ if not hasattr(svc, '_cached_config'):
+ svc._cached_config = svc.configs().get().execute()
+ return svc._cached_config
diff --git a/services/fuse/arvados_fuse/__init__.py b/services/fuse/arvados_fuse/__init__.py
index 0944a3187..3a0316cf9 100644
--- a/services/fuse/arvados_fuse/__init__.py
+++ b/services/fuse/arvados_fuse/__init__.py
@@ -98,7 +98,7 @@ else:
LLFUSE_VERSION_0 = llfuse.__version__.startswith('0')
-from .fusedir import sanitize_filename, Directory, CollectionDirectory, TmpCollectionDirectory, MagicDirectory, TagsDirectory, ProjectDirectory, SharedDirectory, CollectionDirectoryBase
+from .fusedir import Directory, CollectionDirectory, TmpCollectionDirectory, MagicDirectory, TagsDirectory, ProjectDirectory, SharedDirectory, CollectionDirectoryBase
from .fusefile import StringFile, FuseArvadosFile
_logger = logging.getLogger('arvados.arvados_fuse')
diff --git a/services/fuse/arvados_fuse/command.py b/services/fuse/arvados_fuse/command.py
index 528336753..7bef8a269 100644
--- a/services/fuse/arvados_fuse/command.py
+++ b/services/fuse/arvados_fuse/command.py
@@ -301,7 +301,7 @@ class Mount(object):
return
e = self.operations.inodes.add_entry(Directory(
- llfuse.ROOT_INODE, self.operations.inodes))
+ llfuse.ROOT_INODE, self.operations.inodes, self.api.config))
dir_args[0] = e.inode
for name in self.args.mount_by_id:
diff --git a/services/fuse/arvados_fuse/fusedir.py b/services/fuse/arvados_fuse/fusedir.py
index 328765744..9853f6ac2 100644
--- a/services/fuse/arvados_fuse/fusedir.py
+++ b/services/fuse/arvados_fuse/fusedir.py
@@ -33,20 +33,6 @@ _logger = logging.getLogger('arvados.arvados_fuse')
# appear as underscores in the fuse mount.)
_disallowed_filename_characters = re.compile('[\x00/]')
-# '.' and '..' are not reachable if API server is newer than #6277
-def sanitize_filename(dirty):
- """Replace disallowed filename characters with harmless "_"."""
- if dirty is None:
- return None
- elif dirty == '':
- return '_'
- elif dirty == '.':
- return '_'
- elif dirty == '..':
- return '__'
- else:
- return _disallowed_filename_characters.sub('_', dirty)
-
class Directory(FreshBase):
"""Generic directory object, backed by a dict.
@@ -55,7 +41,7 @@ class Directory(FreshBase):
and the value referencing a File or Directory object.
"""
- def __init__(self, parent_inode, inodes):
+ def __init__(self, parent_inode, inodes, apiconfig):
"""parent_inode is the integer inode number"""
super(Directory, self).__init__()
@@ -65,11 +51,39 @@ class Directory(FreshBase):
raise Exception("parent_inode should be an int")
self.parent_inode = parent_inode
self.inodes = inodes
+ self.apiconfig = apiconfig
self._entries = {}
self._mtime = time.time()
- # Overriden by subclasses to implement logic to update the entries dict
- # when the directory is stale
+ def unsanitize_filename(self, incoming):
+ """Replace ForwardSlashNameSubstitution value with /"""
+ fsns = self.apiconfig()["Collections"]["ForwardSlashNameSubstitution"]
+ if isinstance(fsns, str) and fsns != '' and fsns != '/':
+ return incoming.replace(fsns, '/')
+ else:
+ return incoming
+
+ def sanitize_filename(self, dirty):
+ """Replace disallowed filename characters according to
+ ForwardSlashNameSubstitution in self.api_config."""
+ # '.' and '..' are not reachable if API server is newer than #6277
+ if dirty is None:
+ return None
+ elif dirty == '':
+ return '_'
+ elif dirty == '.':
+ return '_'
+ elif dirty == '..':
+ return '__'
+ else:
+ fsns = self.apiconfig()["Collections"]["ForwardSlashNameSubstitution"]
+ if isinstance(fsns, str) and fsns != '':
+ dirty = dirty.replace('/', fsns)
+ return _disallowed_filename_characters.sub('_', dirty)
+
+
+ # Overridden by subclasses to implement logic to update the
+ # entries dict when the directory is stale
@use_counter
def update(self):
pass
@@ -138,7 +152,7 @@ class Directory(FreshBase):
self._entries = {}
changed = False
for i in items:
- name = sanitize_filename(fn(i))
+ name = self.sanitize_filename(fn(i))
if name:
if name in oldentries and same(oldentries[name], i):
# move existing directory entry over
@@ -246,12 +260,13 @@ class CollectionDirectoryBase(Directory):
"""
- def __init__(self, parent_inode, inodes, collection):
- super(CollectionDirectoryBase, self).__init__(parent_inode, inodes)
+ def __init__(self, parent_inode, inodes, apiconfig, collection):
+ super(CollectionDirectoryBase, self).__init__(parent_inode, inodes, apiconfig)
+ self.apiconfig = apiconfig
self.collection = collection
def new_entry(self, name, item, mtime):
- name = sanitize_filename(name)
+ name = self.sanitize_filename(name)
if hasattr(item, "fuse_entry") and item.fuse_entry is not None:
if item.fuse_entry.dead is not True:
raise Exception("Can only reparent dead inode entry")
@@ -260,7 +275,7 @@ class CollectionDirectoryBase(Directory):
item.fuse_entry.dead = False
self._entries[name] = item.fuse_entry
elif isinstance(item, arvados.collection.RichCollectionBase):
- self._entries[name] = self.inodes.add_entry(CollectionDirectoryBase(self.inode, self.inodes, item))
+ self._entries[name] = self.inodes.add_entry(CollectionDirectoryBase(self.inode, self.inodes, self.apiconfig, item))
self._entries[name].populate(mtime)
else:
self._entries[name] = self.inodes.add_entry(FuseArvadosFile(self.inode, item, mtime))
@@ -268,7 +283,7 @@ class CollectionDirectoryBase(Directory):
def on_event(self, event, collection, name, item):
if collection == self.collection:
- name = sanitize_filename(name)
+ name = self.sanitize_filename(name)
_logger.debug("collection notify %s %s %s %s", event, collection, name, item)
with llfuse.lock:
if event == arvados.collection.ADD:
@@ -357,7 +372,7 @@ class CollectionDirectory(CollectionDirectoryBase):
"""Represents the root of a directory tree representing a collection."""
def __init__(self, parent_inode, inodes, api, num_retries, collection_record=None, explicit_collection=None):
- super(CollectionDirectory, self).__init__(parent_inode, inodes, None)
+ super(CollectionDirectory, self).__init__(parent_inode, inodes, api.config, None)
self.api = api
self.num_retries = num_retries
self.collection_record_file = None
@@ -548,7 +563,7 @@ class TmpCollectionDirectory(CollectionDirectoryBase):
keep_client=api_client.keep,
num_retries=num_retries)
super(TmpCollectionDirectory, self).__init__(
- parent_inode, inodes, collection)
+ parent_inode, inodes, api_client.config, collection)
self.collection_record_file = None
self.populate(self.mtime())
@@ -625,7 +640,7 @@ and the directory will appear if it exists.
""".lstrip()
def __init__(self, parent_inode, inodes, api, num_retries, pdh_only=False):
- super(MagicDirectory, self).__init__(parent_inode, inodes)
+ super(MagicDirectory, self).__init__(parent_inode, inodes, api.config)
self.api = api
self.num_retries = num_retries
self.pdh_only = pdh_only
@@ -660,6 +675,7 @@ and the directory will appear if it exists.
e = self.inodes.add_entry(ProjectDirectory(
self.inode, self.inodes, self.api, self.num_retries, project[u'items'][0]))
else:
+ import sys
e = self.inodes.add_entry(CollectionDirectory(
self.inode, self.inodes, self.api, self.num_retries, k))
@@ -696,7 +712,7 @@ class TagsDirectory(Directory):
"""A special directory that contains as subdirectories all tags visible to the user."""
def __init__(self, parent_inode, inodes, api, num_retries, poll_time=60):
- super(TagsDirectory, self).__init__(parent_inode, inodes)
+ super(TagsDirectory, self).__init__(parent_inode, inodes, api.config)
self.api = api
self.num_retries = num_retries
self._poll = True
@@ -753,7 +769,7 @@ class TagDirectory(Directory):
def __init__(self, parent_inode, inodes, api, num_retries, tag,
poll=False, poll_time=60):
- super(TagDirectory, self).__init__(parent_inode, inodes)
+ super(TagDirectory, self).__init__(parent_inode, inodes, api.config)
self.api = api
self.num_retries = num_retries
self.tag = tag
@@ -783,7 +799,7 @@ class ProjectDirectory(Directory):
def __init__(self, parent_inode, inodes, api, num_retries, project_object,
poll=False, poll_time=60):
- super(ProjectDirectory, self).__init__(parent_inode, inodes)
+ super(ProjectDirectory, self).__init__(parent_inode, inodes, api.config)
self.api = api
self.num_retries = num_retries
self.project_object = project_object
@@ -897,16 +913,25 @@ class ProjectDirectory(Directory):
elif self._full_listing or super(ProjectDirectory, self).__contains__(k):
return super(ProjectDirectory, self).__getitem__(k)
with llfuse.lock_released:
+ k2 = self.unsanitize_filename(k)
+ if k2 == k:
+ namefilter = ["name", "=", k]
+ else:
+ namefilter = ["name", "in", [k, k2]]
contents = self.api.groups().list(filters=[["owner_uuid", "=", self.project_uuid],
["group_class", "=", "project"],
- ["name", "=", k]],
- limit=1).execute(num_retries=self.num_retries)["items"]
+ namefilter],
+ limit=2).execute(num_retries=self.num_retries)["items"]
if not contents:
contents = self.api.collections().list(filters=[["owner_uuid", "=", self.project_uuid],
- ["name", "=", k]],
- limit=1).execute(num_retries=self.num_retries)["items"]
+ namefilter],
+ limit=2).execute(num_retries=self.num_retries)["items"]
if contents:
- name = sanitize_filename(self.namefn(contents[0]))
+ if len(contents) > 1 and contents[1].name == k:
+ # If "foo/bar" and "foo[SUBST]bar" both exist, use
+ # "foo[SUBST]bar".
+ contents = [contents[1]]
+ name = self.sanitize_filename(self.namefn(contents[0]))
if name != k:
raise KeyError(k)
return self._add_entry(contents[0], name)
@@ -995,8 +1020,8 @@ class ProjectDirectory(Directory):
new_attrs = properties.get("new_attributes") or {}
old_attrs["uuid"] = ev["object_uuid"]
new_attrs["uuid"] = ev["object_uuid"]
- old_name = sanitize_filename(self.namefn(old_attrs))
- new_name = sanitize_filename(self.namefn(new_attrs))
+ old_name = self.sanitize_filename(self.namefn(old_attrs))
+ new_name = self.sanitize_filename(self.namefn(new_attrs))
# create events will have a new name, but not an old name
# delete events will have an old name, but not a new name
@@ -1038,7 +1063,7 @@ class SharedDirectory(Directory):
def __init__(self, parent_inode, inodes, api, num_retries, exclude,
poll=False, poll_time=60):
- super(SharedDirectory, self).__init__(parent_inode, inodes)
+ super(SharedDirectory, self).__init__(parent_inode, inodes, api.config)
self.api = api
self.num_retries = num_retries
self.current_user = api.users().current().execute(num_retries=num_retries)
diff --git a/services/fuse/tests/test_mount.py b/services/fuse/tests/test_mount.py
index f539b3f7d..3ec03fa22 100644
--- a/services/fuse/tests/test_mount.py
+++ b/services/fuse/tests/test_mount.py
@@ -20,6 +20,7 @@ import arvados
import arvados_fuse as fuse
from . import run_test_server
+from .integration_test import IntegrationTest
from .mount_test_base import MountTestBase
logger = logging.getLogger('arvados.arv-mount')
@@ -1098,8 +1099,9 @@ class MagicDirApiError(FuseMagicTest):
llfuse.listdir(os.path.join(self.mounttmp, self.testcollection))
-class FuseUnitTest(unittest.TestCase):
+class SanitizeFilenameTest(MountTestBase):
def test_sanitize_filename(self):
+ pdir = fuse.ProjectDirectory(1, {}, self.api, 0, project_object=self.api.users().current().execute())
acceptable = [
"foo.txt",
".foo",
@@ -1119,15 +1121,15 @@ class FuseUnitTest(unittest.TestCase):
"//",
]
for f in acceptable:
- self.assertEqual(f, fuse.sanitize_filename(f))
+ self.assertEqual(f, pdir.sanitize_filename(f))
for f in unacceptable:
- self.assertNotEqual(f, fuse.sanitize_filename(f))
+ self.assertNotEqual(f, pdir.sanitize_filename(f))
# The sanitized filename should be the same length, though.
- self.assertEqual(len(f), len(fuse.sanitize_filename(f)))
+ self.assertEqual(len(f), len(pdir.sanitize_filename(f)))
# Special cases
- self.assertEqual("_", fuse.sanitize_filename(""))
- self.assertEqual("_", fuse.sanitize_filename("."))
- self.assertEqual("__", fuse.sanitize_filename(".."))
+ self.assertEqual("_", pdir.sanitize_filename(""))
+ self.assertEqual("_", pdir.sanitize_filename("."))
+ self.assertEqual("__", pdir.sanitize_filename(".."))
class FuseMagicTestPDHOnly(MountTestBase):
@@ -1191,3 +1193,43 @@ class FuseMagicTestPDHOnly(MountTestBase):
def test_with_default_by_id(self):
self.verify_pdh_only(skip_pdh_only=True)
+
+
+class SlashSubstitutionTest(IntegrationTest):
+ mnt_args = [
+ '--read-write',
+ '--mount-home', 'zzz',
+ ]
+
+ def setUp(self):
+ super(SlashSubstitutionTest, self).setUp()
+ self.api = arvados.safeapi.ThreadSafeApiCache(arvados.config.settings())
+ self.api.config = lambda: {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
+ self.testcoll = self.api.collections().create(body={"name": "foo/bar/baz"}).execute()
+ self.testcolleasy = self.api.collections().create(body={"name": "foo-bar-baz"}).execute()
+ self.fusename = 'foo[SLASH]bar[SLASH]baz'
+
+ @IntegrationTest.mount(argv=mnt_args)
+ @mock.patch('arvados.util.get_config_once')
+ def test_slash_substitution_before_listing(self, get_config_once):
+ get_config_once.return_value = {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
+ self.pool_test(os.path.join(self.mnt, 'zzz'), self.fusename)
+ @staticmethod
+ def _test_slash_substitution_before_listing(self, tmpdir, fusename):
+ with open(os.path.join(tmpdir, 'foo-bar-baz', 'waz'), 'w') as f:
+ f.write('foo')
+ with open(os.path.join(tmpdir, fusename, 'waz'), 'w') as f:
+ f.write('foo')
+
+ @IntegrationTest.mount(argv=mnt_args)
+ @mock.patch('arvados.util.get_config_once')
+ def test_slash_substitution_after_listing(self, get_config_once):
+ get_config_once.return_value = {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
+ self.pool_test(os.path.join(self.mnt, 'zzz'), self.fusename)
+ @staticmethod
+ def _test_slash_substitution_after_listing(self, tmpdir, fusename):
+ with open(os.path.join(tmpdir, 'foo-bar-baz', 'waz'), 'w') as f:
+ f.write('foo')
+ os.listdir(tmpdir)
+ with open(os.path.join(tmpdir, fusename, 'waz'), 'w') as f:
+ f.write('foo')
commit d6cac9806211dc5321cdc4fc24583f71e504cc84
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Feb 14 13:54:08 2020 -0500
Tweak upgrade notes
no issue #
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/admin/upgrading.html.textile.liquid b/doc/admin/upgrading.html.textile.liquid
index 23d712043..df64904a8 100644
--- a/doc/admin/upgrading.html.textile.liquid
+++ b/doc/admin/upgrading.html.textile.liquid
@@ -34,13 +34,7 @@ TODO: extract this information based on git commit messages and generate changel
<div class="releasenotes">
</notextile>
-h2(#master). development master (as of 2020-02-07)
-
-"Upgrading from 2.0.0":#v2_0_0
-
-None in current development master.
-
-h2(#v2_0_0). v2.0.0 (2020-02-07)
+h2(#v2_0_0). v2.0.0 (2020-02-14)
"Upgrading from 1.4":#v1_4_1
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list