[arvados] updated: 2.5.0-263-gd9b8d0295
git repository hosting
git at public.arvados.org
Thu Mar 9 21:55:04 UTC 2023
Summary of changes:
.../_container_runtime_constraints.liquid | 3 +-
...controlling-container-reuse.html.textile.liquid | 2 +-
doc/admin/upgrading.html.textile.liquid | 12 +-
.../methods/container_requests.html.textile.liquid | 6 +-
doc/user/cwl/cwl-extensions.html.textile.liquid | 18 +++
lib/config/config.default.yml | 27 +++-
lib/config/export.go | 1 +
lib/controller/handler.go | 22 +++
lib/controller/handler_test.go | 56 ++++++++
lib/controller/integration_test.go | 2 +
lib/controller/localdb/container.go | 18 ++-
lib/dispatchcloud/dispatcher.go | 6 +-
lib/dispatchcloud/scheduler/run_queue.go | 8 ++
lib/dispatchcloud/scheduler/run_queue_test.go | 88 ++++++++++--
lib/dispatchcloud/scheduler/scheduler.go | 4 +-
lib/dispatchcloud/scheduler/sync_test.go | 4 +-
sdk/cwl/arvados_cwl/__init__.py | 1 +
sdk/cwl/arvados_cwl/arv-cwl-schema-v1.0.yml | 27 ++++
sdk/cwl/arvados_cwl/arv-cwl-schema-v1.1.yml | 27 ++++
sdk/cwl/arvados_cwl/arv-cwl-schema-v1.2.yml | 28 ++++
sdk/cwl/arvados_cwl/arvcontainer.py | 100 +++++++++++---
sdk/cwl/tests/arvados-tests.yml | 15 +++
sdk/cwl/tests/oom/19975-oom.cwl | 18 +++
sdk/cwl/tests/oom/19975-oom3.cwl | 19 +++
sdk/cwl/tests/oom/fakeoom.py | 13 ++
sdk/cwl/tests/{octo.yml => oom/fakeoom.yml} | 4 +-
sdk/cwl/tests/oom/fakeoom2.py | 13 ++
sdk/cwl/tests/{octo.yml => oom/fakeoom2.yml} | 4 +-
sdk/cwl/tests/oom/fakeoom3.py | 14 ++
sdk/cwl/tests/{octo.yml => oom/fakeoom3.yml} | 4 +-
sdk/go/arvados/config.go | 2 +
sdk/go/arvados/container.go | 1 +
services/api/app/models/arvados_model.rb | 1 +
services/api/app/models/container.rb | 110 +++++++++++----
services/api/app/models/container_request.rb | 13 +-
.../v1/container_requests_controller_test.rb | 19 ++-
services/api/test/unit/container_request_test.rb | 53 +++++---
services/api/test/unit/container_test.rb | 147 +++++++++++++++++++--
services/keep-web/handler.go | 2 +-
services/keep-web/handler_test.go | 2 +-
40 files changed, 811 insertions(+), 103 deletions(-)
create mode 100644 sdk/cwl/tests/oom/19975-oom.cwl
create mode 100644 sdk/cwl/tests/oom/19975-oom3.cwl
create mode 100644 sdk/cwl/tests/oom/fakeoom.py
copy sdk/cwl/tests/{octo.yml => oom/fakeoom.yml} (72%)
create mode 100644 sdk/cwl/tests/oom/fakeoom2.py
copy sdk/cwl/tests/{octo.yml => oom/fakeoom2.yml} (72%)
create mode 100644 sdk/cwl/tests/oom/fakeoom3.py
copy sdk/cwl/tests/{octo.yml => oom/fakeoom3.yml} (72%)
via d9b8d0295f1fa1c1690b9fe80ba38f6f665de4e9 (commit)
via 869497eff961e5d2951e84e8b46d81bbc1d3ec3b (commit)
via dcb3df6be978eef367e0d4fda95ccd71e0281749 (commit)
via 765604b096c464b242e60860ace75bd6645fd4e7 (commit)
via adb11a544a649b24196b38b6b614a95585728240 (commit)
via a7bb7c84d2e84eeb92810c319f54a8456011288a (commit)
via 1e55288da627c93d8fa01abaf4611f114a20525d (commit)
via 82de91b57218abf4f74a38f1bf53d0cc28cd8da9 (commit)
via efd62fdeffaef4d47710511f7e1ac37d5b4cc5a7 (commit)
via c75359c0116392ea13ebf578c74e495c9e158f62 (commit)
via b5a8acaf05b8fe8ef00855b3280f3b019539eda3 (commit)
via 3d65beeef57b2dd7a60171c8dce05785ef1ae383 (commit)
via 2b2707e4acb9ee7eed53f573cf5365af9189a8e3 (commit)
via 51b11d12c688dd9fae37f527d4c1ddf1e1a186f6 (commit)
via 9e76a12ff0b25322f86caf6d5ea70c09cbfd8829 (commit)
via db118cca358662e57a3dd0c1186dce1f0a62ca52 (commit)
via fe738b30d51dadb2b42ce7ae00b333d7ea157495 (commit)
via aede6055f332c1680f6fe955e03e3e6364f8eca8 (commit)
via 184d9fee6b72520be286b375ac569702246e224e (commit)
via 599e2e6798a2497ed56b6dfe917a09260dd74407 (commit)
via dc1837da3a36ab3b273fef7245975ecd92524072 (commit)
via 6392400ce074288657ba47ce98997b502538edda (commit)
via 7541b9c86e07a65139ee508cc2476fdc77ee4310 (commit)
via 5d73c8c825577b65d48b9dec05428f3db104efb2 (commit)
via e1c7395cf6e649876132030c2011434581d3a66a (commit)
via fe18471f4f1a04af7444d2162cab16eef2685677 (commit)
via 39fc2f223fae40dc4fb160758e76ca39304b44af (commit)
via 901f7a4f3723b982fd45562f74fb92c7900e0d47 (commit)
via 843fc048d7500fef8a461db4e8b6022d5c4b0e24 (commit)
via 1116589dd94d168b920b20340491472e2d0c8c2a (commit)
via b5874d1fae46eef00662d74fe25e6841bec68650 (commit)
via 921603e88993056676fc17939444c812dc302d27 (commit)
via afbea0d985d273232291f03f343baed727393108 (commit)
via 567ec845eb4d8b5a53b662ac56624395631f3637 (commit)
via 4eb591b3d541cc5ac035fb0eed7c39dc8e81dd02 (commit)
via 53a0cb93d935f02248f4e347ea0689fccf5e818e (commit)
via 4447b5a61f79edf2411ba94f4ad5d90e1ca7e220 (commit)
from cd171959303236ba9a85e6524e6c40310203325b (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit d9b8d0295f1fa1c1690b9fe80ba38f6f665de4e9
Author: Tom Clegg <tom at curii.com>
Date: Thu Mar 9 16:50:03 2023 -0500
20183: Update container priority in creation order.
When multiple containers in a tree need updates, this is effectively
top-down order, which is more efficient. The topmost update will
update the whole tree anyway, so updating subtrees first would be a
waste.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at curii.com>
diff --git a/lib/controller/localdb/container.go b/lib/controller/localdb/container.go
index ad1da6e13..52a3974aa 100644
--- a/lib/controller/localdb/container.go
+++ b/lib/controller/localdb/container.go
@@ -107,6 +107,7 @@ func (conn *Conn) containerPriorityUpdate(ctx context.Context, log logrus.FieldL
WHERE containers.state IN ('Queued', 'Locked', 'Running')
AND containers.priority = 0
AND (parent.uuid IS NULL OR parent.priority > 0)
+ ORDER BY containers.created_at
LIMIT 1`).Scan(&uuid)
if err == sql.ErrNoRows {
break
commit 869497eff961e5d2951e84e8b46d81bbc1d3ec3b
Author: Tom Clegg <tom at curii.com>
Date: Thu Mar 9 16:38:59 2023 -0500
20183: Clean up sql queries.
Remove "container_requests.uuid IS NOT NULL" (this is a tautology in
an inner join)
Use same "state=Committed" condition in Rails and controller
query. This is already implied by the "containers.state in (Queued,
Locked, Running)" condition since only a Committed request should
reference such a container. However, if the distinction ever comes
up (race? manual database update?) we would want to skip the
non-Committed requests, so this serves as cheap insurance.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at curii.com>
diff --git a/lib/controller/localdb/container.go b/lib/controller/localdb/container.go
index 86595e423..ad1da6e13 100644
--- a/lib/controller/localdb/container.go
+++ b/lib/controller/localdb/container.go
@@ -100,13 +100,12 @@ func (conn *Conn) containerPriorityUpdate(ctx context.Context, log logrus.FieldL
err := db.QueryRowxContext(ctx, `
SELECT containers.uuid from containers
JOIN container_requests
- ON container_requests.container_uuid=containers.uuid
+ ON container_requests.container_uuid = containers.uuid
AND container_requests.state = 'Committed' AND container_requests.priority > 0
LEFT JOIN containers parent
ON parent.uuid = container_requests.requesting_container_uuid
WHERE containers.state IN ('Queued', 'Locked', 'Running')
AND containers.priority = 0
- AND container_requests.uuid IS NOT NULL
AND (parent.uuid IS NULL OR parent.priority > 0)
LIMIT 1`).Scan(&uuid)
if err == sql.ErrNoRows {
diff --git a/services/api/app/models/container.rb b/services/api/app/models/container.rb
index 7837e0812..569a5841b 100644
--- a/services/api/app/models/container.rb
+++ b/services/api/app/models/container.rb
@@ -131,7 +131,7 @@ class Container < ArvadosModel
def update_priority!
return if ![Queued, Locked, Running].include?(state)
p = ContainerRequest.
- where('container_uuid=? and priority>0', uuid).
+ where('container_uuid=? and priority>0 and state=?', uuid, ContainerRequest::Committed).
includes(:requesting_container).
lock(true).
map do |cr|
commit dcb3df6be978eef367e0d4fda95ccd71e0281749
Author: Tom Clegg <tom at curii.com>
Date: Thu Mar 9 16:38:24 2023 -0500
20183: Explain priority update strategy in comments.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at curii.com>
diff --git a/lib/controller/localdb/container.go b/lib/controller/localdb/container.go
index 81f257181..86595e423 100644
--- a/lib/controller/localdb/container.go
+++ b/lib/controller/localdb/container.go
@@ -56,6 +56,10 @@ func (conn *Conn) containerPriorityUpdate(ctx context.Context, log logrus.FieldL
if err != nil {
return fmt.Errorf("getdb: %w", err)
}
+ // Stage 1: Fix containers that have priority>0 but should
+ // have priority=0 because there are no active
+ // container_requests (unfinished, priority>0) associated with
+ // them.
res, err := db.ExecContext(ctx, `
UPDATE containers
SET priority=0
@@ -73,6 +77,16 @@ func (conn *Conn) containerPriorityUpdate(ctx context.Context, log logrus.FieldL
} else if rows > 0 {
log.Infof("found %d containers with priority>0 and no active requests, updated to priority=0", rows)
}
+
+ // Stage 2: Fix containers that have priority=0 but should
+ // have priority>0 because there are active container_requests
+ // (priority>0, unfinished, and not children of cancelled
+ // containers).
+ //
+ // Fixing here means calling out to RailsAPI to compute the
+ // correct priority for the contianer and (if needed)
+ // propagate that change to child containers.
+
// In this loop we look for a single container that needs
// fixing, call out to Rails to fix it, and repeat until we
// don't find any more.
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list