[ARVADOS] updated: 2.1.0-1275-g2115f4609
Git user
git at public.arvados.org
Mon Aug 30 18:03:09 UTC 2021
Summary of changes:
build/run-build-packages-one-target.sh | 1 +
doc/api/methods.html.textile.liquid | 26 +-
doc/api/methods/collections.html.textile.liquid | 5 +-
lib/controller/fed_collections.go | 312 ---------------------
lib/controller/fed_containers.go | 123 --------
lib/controller/federation.go | 8 -
lib/controller/federation_test.go | 4 +-
lib/controller/localdb/collection.go | 102 +++++++
lib/controller/localdb/collection_test.go | 142 ++++++++++
lib/controller/server_test.go | 4 +
lib/costanalyzer/costanalyzer_test.go | 1 -
lib/recovercollection/cmd_test.go | 1 -
sdk/go/arvados/api.go | 1 +
sdk/go/arvadostest/run_servers.go | 88 ++----
sdk/go/dispatch/dispatch_test.go | 8 -
sdk/go/keepclient/keepclient_test.go | 2 -
sdk/python/arvados/collection.py | 4 +-
sdk/python/arvados/keep.py | 11 +-
sdk/python/tests/arvados_testutil.py | 8 +-
sdk/python/tests/test_keep_client.py | 43 ++-
.../arvados/v1/collections_controller.rb | 12 +-
services/api/app/models/collection.rb | 22 +-
services/api/lib/record_filters.rb | 34 ++-
.../arvados/v1/collections_controller_test.rb | 95 ++-----
.../api/test/functional/arvados/v1/filters_test.rb | 47 ++++
.../api/test/integration/collections_api_test.rb | 10 +-
.../api/test/unit/collection_performance_test.rb | 5 +-
services/api/test/unit/collection_test.rb | 12 +-
services/arv-git-httpd/auth_handler_test.go | 8 -
services/arv-git-httpd/integration_test.go | 8 -
.../crunch-dispatch-local_test.go | 5 -
.../crunch-dispatch-slurm_test.go | 5 +-
services/keep-balance/integration_test.go | 2 -
services/keep-web/server_test.go | 3 +-
services/keepproxy/keepproxy_test.go | 9 -
services/keepstore/pull_worker_integration_test.go | 2 -
tools/keep-block-check/keep-block-check_test.go | 5 -
tools/keep-rsync/keep-rsync_test.go | 5 -
38 files changed, 491 insertions(+), 692 deletions(-)
delete mode 100644 lib/controller/fed_collections.go
delete mode 100644 lib/controller/fed_containers.go
create mode 100644 lib/controller/localdb/collection.go
create mode 100644 lib/controller/localdb/collection_test.go
via 2115f4609adcc86156e8e0aa59ea38ba5808378e (commit)
via c113182fbbef72e390371bc40ce3e80b46975b14 (commit)
via d8e3a67d508e9a5f5c01884259c0e75a140f64e9 (commit)
via b1daec9a928eefbc71d8b7368b148fa7b04bf32d (commit)
via 0ea42bb254ce0df1e32816761b84d047cddffb06 (commit)
via d67a11c2ee42159dd0ecd8f6ef39af38b6380dfd (commit)
via 9b46e402be4366b591ce9c73e2afb24bef0a3dd5 (commit)
via 138e5417d2d11fcf404a63ca6e08815bfc0cca67 (commit)
via 2ab371465ac93bdf83d3fc423e361c50c54855d8 (commit)
via 168d3fe8640baf12c615d2a52b914836db8a8a27 (commit)
via be900941bb4ab286cbeb02f65509be938726d67e (commit)
via f1403a71fc2e2ecb254502510a515bb2c07dffa0 (commit)
via 5209c88ffd019d3d33a0640833875d92d2b21849 (commit)
via 0020ef5b7c44cfd0d65182f1eaf66a30d2907a19 (commit)
via a97f1b7ec7614b276a1c3f937fbcb330eb14ba6c (commit)
via 7c66aef044ec7bf48aad64cd82308f48ef00c143 (commit)
via 9ed0a21813117caf5c6bae73c09a0d725564743b (commit)
via 8e75f57bd693d0ceb1aab86ba0e84cb19b4d155a (commit)
via bdd74e4dff9d9c5c1a329cfbed2e2e08336c8c51 (commit)
via 402e69f6e55dce4e11d354c3ca708b8e536c124b (commit)
via 388713bd42fb20bae6f628c50f4ad1e3ba067b5b (commit)
via 902f8cd258a8dfec749a7f94d478a4027e319750 (commit)
from f68961a64f84de3987f99fe3dd79ca35abfab3aa (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 2115f4609adcc86156e8e0aa59ea38ba5808378e
Merge: c113182fb d8e3a67d5
Author: Tom Clegg <tom at curii.com>
Date: Mon Aug 30 14:02:46 2021 -0400
17995: Merge branch 'main'
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at curii.com>
diff --cc doc/api/methods.html.textile.liquid
index 914e9779f,e051ab66f..87ff621b8
--- a/doc/api/methods.html.textile.liquid
+++ b/doc/api/methods.html.textile.liquid
@@@ -105,22 -107,10 +107,24 @@@ table(table table-bordered table-conden
|@like@, @ilike@|string|SQL pattern match. Single character match is @_@ and wildcard is @%@. The @ilike@ operator is case-insensitive|@["script_version","like","d00220fb%"]@|
|@in@, @not in@|array of strings|Set membership|@["script_version","in",["main","d00220fb38d4b85ca8fc28a8151702a2b9d1dec5"]]@|
|@is_a@|string|Arvados object type|@["head_uuid","is_a","arvados#collection"]@|
- |@exists@|string|Test if a subproperty is present.|@["properties","exists","my_subproperty"]@|
+ |@exists@|string|Presence of subproperty|@["properties","exists","my_subproperty"]@|
+ |@contains@|string, array of strings|Presence of one or more keys or array elements|@["storage_classes_desired", "contains", ["foo", "bar"]]@ (matches both @["foo", "bar"]@ and @["foo", "bar", "baz"]@)
+ (note @[..., "contains", "foo"]@ is also accepted, and is equivalent to @[..., "contains", ["foo"]]@)|
+h4(#filterexpression). Filtering using boolean expressions
+
+In place of an attribute, the first element of the three-element filter array can be a boolean expression. The following restrictions apply:
+* The expression must contain exactly one operator.
+* The operator must be @=@, @<@, @<=@, @>@, or @>=@.
+* There must be exactly one pair of parentheses, surrounding the entire expression.
+* Each operand must be the name of a numeric attribute like @replication_desired@ (literal values like @3@ and non-numeric attributes like @uuid@ are not accepted).
+* The expression must not contain whitespace other than an ASCII space (newline and tab characters are not accepted).
+* The second and third elements of the filter array must be @"="@ and @true@ respectively.
+
+Examples:
+* @["(replication_desired < replication_confirmed)", "=", true]@
+* @["(replication_desired = replication_confirmed)", "=", true]@
+
h4(#substringsearchfilter). Filtering using substring search
Resources can also be filtered by searching for a substring in attributes of type @string@, @array of strings@, @text@, and @hash@, which are indexed in the database specifically for search. To use substring search, the filter must:
diff --cc services/api/lib/record_filters.rb
index 517a7fe28,409e48a6f..2f5b67074
--- a/services/api/lib/record_filters.rb
+++ b/services/api/lib/record_filters.rb
@@@ -141,25 -142,9 +142,26 @@@ module RecordFilter
cond_out << "jsonb_exists(#{attr_table_name}.#{attr}, ?)"
param_out << operand
+ elsif expr = /^ *\( *(\w+) *(<=?|>=?|=) *(\w+) *\) *$/.match(attr)
+ if operator != '=' || ![true,"true"].index(operand)
+ raise ArgumentError.new("Invalid expression filter '#{attr}': subsequent elements must be [\"=\", true]")
+ end
+ operator = expr[2]
+ attr1, attr2 = expr[1], expr[3]
+ allowed = attr_model_class.searchable_columns(operator)
+ [attr1, attr2].each do |tok|
+ if !allowed.index(tok)
+ raise ArgumentError.new("Invalid attribute in expression: '#{tok}'")
+ end
+ col = attr_model_class.columns.select { |c| c.name == tok }.first
+ if col.type != :integer
+ raise ArgumentError.new("Non-numeric attribute in expression: '#{tok}'")
+ end
+ end
+ cond_out << "#{attr1} #{operator} #{attr2}"
else
- if !attr_model_class.searchable_columns(operator).index attr
+ if !attr_model_class.searchable_columns(operator).index(attr) &&
+ !(col.andand.type == :jsonb && ['contains', '=', '<>', '!='].index(operator))
raise ArgumentError.new("Invalid attribute '#{attr}' in filter")
end
diff --cc services/api/test/functional/arvados/v1/collections_controller_test.rb
index 86a306731,6c923ff38..d9e909575
--- a/services/api/test/functional/arvados/v1/collections_controller_test.rb
+++ b/services/api/test/functional/arvados/v1/collections_controller_test.rb
@@@ -1456,47 -1404,17 +1404,62 @@@ EO
assert_equal col.version, json_response['version'], 'Trashing a collection should not create a new version'
end
+ [['<', :<],
+ ['<=', :<=],
+ ['>', :>],
+ ['>=', :>=],
+ ['=', :==]].each do |op, rubyop|
+ test "filter collections by replication_desired #{op} replication_confirmed" do
+ authorize_with(:active)
+ get :index, params: {
+ filters: [["(replication_desired #{op} replication_confirmed)", "=", true]],
+ }
+ assert_response :success
+ json_response["items"].each do |c|
+ assert_operator(c["replication_desired"], rubyop, c["replication_confirmed"])
+ end
+ end
+ end
+
+ ["(replication_desired < bogus)",
+ "replication_desired < replication_confirmed",
+ "(replication_desired < replication_confirmed",
+ "(replication_desired ! replication_confirmed)",
+ "(replication_desired <)",
+ "(replication_desired < manifest_text)",
+ "(manifest_text < manifest_text)", # currently only numeric attrs are supported
+ "(replication_desired < 2)", # currently only attrs are supported, not literals
+ "(1 < 2)",
+ ].each do |expr|
+ test "invalid filter expression #{expr}" do
+ authorize_with(:active)
+ get :index, params: {
+ filters: [[expr, "=", true]],
+ }
+ assert_response 422
+ end
+ end
+
+ test "invalid op/arg with filter expression" do
+ authorize_with(:active)
+ get :index, params: {
+ filters: [["replication_desired < replication_confirmed", "!=", false]],
+ }
+ assert_response 422
+ end
++
+ ["storage_classes_desired", "storage_classes_confirmed"].each do |attr|
+ test "filter collections by #{attr}" do
+ authorize_with(:active)
+ get :index, params: {
+ filters: [[attr, "=", '["default"]']]
+ }
+ assert_response :success
+ assert_not_equal 0, json_response["items"].length
+ json_response["items"].each do |c|
+ assert_equal ["default"], c[attr]
+ end
+ end
+ end
++>>>>>>> main
end
commit c113182fbbef72e390371bc40ce3e80b46975b14
Author: Tom Clegg <tom at curii.com>
Date: Mon Aug 30 14:02:03 2021 -0400
17995: Rephrase filter expression docs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at curii.com>
diff --git a/doc/api/methods.html.textile.liquid b/doc/api/methods.html.textile.liquid
index 09918830e..914e9779f 100644
--- a/doc/api/methods.html.textile.liquid
+++ b/doc/api/methods.html.textile.liquid
@@ -107,16 +107,20 @@ table(table table-bordered table-condensed).
|@is_a@|string|Arvados object type|@["head_uuid","is_a","arvados#collection"]@|
|@exists@|string|Test if a subproperty is present.|@["properties","exists","my_subproperty"]@|
-h4(#filterexpression). Filtering by comparing two attributes
+h4(#filterexpression). Filtering using boolean expressions
-If @a@ and @b@ are attributes that have numeric values, they can be compared in a filter condition using the form @["(a < b)", "=", true]@. Supported operators are @=@, @<@, @<=@, @>@, and @>=@. Examples:
+In place of an attribute, the first element of the three-element filter array can be a boolean expression. The following restrictions apply:
+* The expression must contain exactly one operator.
+* The operator must be @=@, @<@, @<=@, @>@, or @>=@.
+* There must be exactly one pair of parentheses, surrounding the entire expression.
+* Each operand must be the name of a numeric attribute like @replication_desired@ (literal values like @3@ and non-numeric attributes like @uuid@ are not accepted).
+* The expression must not contain whitespace other than an ASCII space (newline and tab characters are not accepted).
+* The second and third elements of the filter array must be @"="@ and @true@ respectively.
+
+Examples:
* @["(replication_desired < replication_confirmed)", "=", true]@
* @["(replication_desired = replication_confirmed)", "=", true]@
-Note that only a very specific subset of boolean expressions is supported. For example:
-* @["replication_desired < replication_confirmed", "=", true]@ is not accepted. Outer parentheses are mandatory.
-* @["(replication_desired < 1)", "=", true]@ is not accepted. Operands must be attribute names.
-
h4(#substringsearchfilter). Filtering using substring search
Resources can also be filtered by searching for a substring in attributes of type @string@, @array of strings@, @text@, and @hash@, which are indexed in the database specifically for search. To use substring search, the filter must:
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list