[ARVADOS] updated: 1.3.0-1620-gce0df2b96
Git user
git at public.curoverse.com
Thu Sep 19 13:08:10 UTC 2019
Summary of changes:
doc/admin/config-migration.html.textile.liquid | 8 ++++++--
doc/admin/upgrading.html.textile.liquid | 19 +++++++++++++++----
services/api/app/models/keep_service.rb | 11 ++++++-----
.../arvados/v1/keep_services_controller_test.rb | 12 +++++-------
4 files changed, 32 insertions(+), 18 deletions(-)
via ce0df2b962b83137d97b75f594aa28fd694fb015 (commit)
via 55dc07e524cf221c3e572fb86de01255be6c759c (commit)
from 9dcec2ce4a077f14204fdfd6c4b1ec208ea281ab (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit ce0df2b962b83137d97b75f594aa28fd694fb015
Author: Tom Clegg <tclegg at veritasgenetics.com>
Date: Thu Sep 19 09:07:07 2019 -0400
13647: Export keepproxy's ExternalURL, not InternalURLs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg at veritasgenetics.com>
diff --git a/services/api/app/models/keep_service.rb b/services/api/app/models/keep_service.rb
index 777f6bfb2..60f263202 100644
--- a/services/api/app/models/keep_service.rb
+++ b/services/api/app/models/keep_service.rb
@@ -51,11 +51,12 @@ class KeepService < ArvadosModel
values = []
id = 1
Rails.configuration.Services.Keepstore.InternalURLs.each do |url, info|
- values << "(#{id}, " + quoted_column_values_from_url(url: url.to_s, info: info).join(", ") + ", 'disk', 'f'::bool, #{config_time}, #{config_time}, #{owner}, #{owner}, null)"
+ values << "(#{id}, " + quoted_column_values_from_url(url: url.to_s, rendezvous: info.Rendezvous).join(", ") + ", 'disk', 'f'::bool, #{config_time}, #{config_time}, #{owner}, #{owner}, null)"
id += 1
end
- Rails.configuration.Services.Keepproxy.InternalURLs.each do |url, info|
- values << "(#{id}, " + quoted_column_values_from_url(url: url.to_s, info: info).join(", ") + ", 'proxy', 'f'::bool, #{config_time}, #{config_time}, #{owner}, #{owner}, null)"
+ url = Rails.configuration.Services.Keepproxy.ExternalURL.to_s
+ if !url.blank?
+ values << "(#{id}, " + quoted_column_values_from_url(url: url, rendezvous: "").join(", ") + ", 'proxy', 'f'::bool, #{config_time}, #{config_time}, #{owner}, #{owner}, null)"
id += 1
end
if values.length == 0
@@ -69,8 +70,8 @@ class KeepService < ArvadosModel
private
- def self.quoted_column_values_from_url(url:, info:)
- rvz = info.Rendezvous
+ def self.quoted_column_values_from_url(url:, rendezvous:)
+ rvz = rendezvous
rvz = url if rvz.blank?
if /^[a-zA-Z0-9]{15}$/ !~ rvz
# If rvz is an URL (either the real service URL, or an alternate
diff --git a/services/api/test/functional/arvados/v1/keep_services_controller_test.rb b/services/api/test/functional/arvados/v1/keep_services_controller_test.rb
index f41a1d679..867ab35e7 100644
--- a/services/api/test/functional/arvados/v1/keep_services_controller_test.rb
+++ b/services/api/test/functional/arvados/v1/keep_services_controller_test.rb
@@ -46,12 +46,7 @@ class Arvados::V1::KeepServicesControllerTest < ActionController::TestCase
expect_rvz[k.to_s] = rvz
Rails.configuration.Services.Keepstore.InternalURLs[k].Rendezvous = rvz
end
- Rails.configuration.Services.Keepproxy.InternalURLs.each do |k,v|
- n += 1
- rvz = "%015x" % n
- expect_rvz[k.to_s] = rvz
- Rails.configuration.Services.Keepproxy.InternalURLs[k].Rendezvous = rvz
- end
+ expect_rvz[Rails.configuration.Services.Keepproxy.ExternalURL] = true
refute_empty expect_rvz
authorize_with :active
get :index,
@@ -61,7 +56,10 @@ class Arvados::V1::KeepServicesControllerTest < ActionController::TestCase
json_response['items'].each do |svc|
url = "#{svc['service_ssl_flag'] ? 'https' : 'http'}://#{svc['service_host']}:#{svc['service_port']}"
assert_equal true, expect_rvz.has_key?(url), "#{url} does not match any configured service: expecting #{expect_rvz}"
- assert_equal "zzzzz-bi6l4-#{expect_rvz[url]}", svc['uuid'], "exported service UUID should match InternalURLs.*.Rendezvous value"
+ rvz = expect_rvz[url]
+ if rvz.is_a? String
+ assert_equal "zzzzz-bi6l4-#{rvz}", svc['uuid'], "exported service UUID should match InternalURLs.*.Rendezvous value"
+ end
expect_rvz.delete(url)
end
assert_equal({}, expect_rvz, "all configured Keepstore and Keepproxy services should be returned")
commit 55dc07e524cf221c3e572fb86de01255be6c759c
Author: Tom Clegg <tclegg at veritasgenetics.com>
Date: Wed Sep 18 21:21:18 2019 -0400
13647: Update keepstore/keep_services config migration instructions.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg at veritasgenetics.com>
diff --git a/doc/admin/config-migration.html.textile.liquid b/doc/admin/config-migration.html.textile.liquid
index 4e2fd81af..442d7374e 100644
--- a/doc/admin/config-migration.html.textile.liquid
+++ b/doc/admin/config-migration.html.textile.liquid
@@ -58,9 +58,13 @@ h2. crunch-dispatch-slurm
Currently only reads @InstanceTypes@ from centralized configuration. Still requires component-specific configuration file.
-h2. keepstore
+h2(#keepstore). keepstore
-Currently only reads @RemoteClusters@ from centralized configuration. Still requires component-specific configuration file.
+The legacy keepstore config (loaded from @/etc/arvados/keepstore/keepstore.yml@ or a different location specified via -legacy-keepstore-config command line argument) takes precedence over the centralized config. After you migrate everything from the legacy config to the centralized config, you should delete @/etc/arvados/keepstore/keepstore.yml@ and stop using the -legacy-keepstore-config argument.
+
+If a legacy config file exists when a keepstore process starts up, its log messages will provide instructions for migrating configuration values from the legacy config file to the new centralized config file. Installing the @arvados-server@ package and running @arvados-server config-check@ will also display these migration instructions. After making these changes, run @arvados-server config-check@ (or restart keepstore and check startup logs) and repeat as needed. When this process is complete, you should delete the legacy config file, copy the updated centralized config file to your next keepstore server, and repeat the same process there.
+
+After migrating and removing all legacy keepstore config files, make sure the @/etc/arvados/config.yml@ file is identical across all system nodes -- API server, keepstore, etc. -- and restart all services to make sure they are using the latest configuration.
h2(#keepproxy). keepproxy
diff --git a/doc/admin/upgrading.html.textile.liquid b/doc/admin/upgrading.html.textile.liquid
index b98a9e2dc..a403a368c 100644
--- a/doc/admin/upgrading.html.textile.liquid
+++ b/doc/admin/upgrading.html.textile.liquid
@@ -41,17 +41,28 @@ table(table table-bordered table-condensed).
h3(#master). development master (as of 2019-08-12)
+h4. Delete "keep_services" records
+
+After all keepproxy and keepstore configurations have been migrated to the centralized configuration file (see below), all keep_services records you added manually during installation should be removed. System logs from keepstore and keepproxy at startup, as well as the output of @arvados-server config-check@, will remind you to do this.
+
+<notextile><pre><code>$ export ARVADOS_API_HOST=...
+$ export ARVADOS_API_TOKEN=...
+$ arv --format=uuid keep_service list | xargs -n1 arv keep_service delete --uuid
+</code></pre></notextile>
+
+Once these old records are removed, @arv keep_service list@ will instead return the services listed under Services/Keepstore/InternalURLs and Services/Keepproxy/ExternalURL in your centralized configuration file.
+
h4. Arv-git-httpd configuration migration
(feature "#14712":https://dev.arvados.org/issues/14712 ) The arv-git-httpd package can now be configured using the centralized configuration file at @/etc/arvados/config.yml at . Configuration via individual command line arguments is no longer available. Please see "arv-git-httpd's config migration guide":{{site.baseurl}}/admin/config-migration.html#arv-git-httpd for more details.
-h4. Keepstore and keep-web no longer support configuration via command line flags
+h4. Keepstore and keep-web configuration migration
-As we're migrating to a central cluster configuration file, the already deprecated way of getting configurations via environment variables and command line flags isn't valid anymore.
+keepstore and keep-web no longer support configuration via (previously deprecated) command line configuration flags and environment variables.
-Current keep-web supports both the now legacy @keep-web.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check "keep-web's install guide":{{site.baseurl}}/install/install-keep-web.html for more details.
+keep-web now supports the legacy @keep-web.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check "keep-web's install guide":{{site.baseurl}}/install/install-keep-web.html for more details.
-Current keepstore supports both the now legacy @keepstore.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check "keepstore's install guide":{{site.baseurl}}/install/install-keepstore.html for more details.
+keepstore now supports the legacy @keepstore.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check the "keepstore config migration notes":{{site.baseurl}}/admin/config-migration.html#keepstore and "keepstore install guide":{{site.baseurl}}/install/install-keepstore.html for more details.
h4. Jobs API is read-only
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list