[ARVADOS] updated: 2.1.0-1430-g4b40e894f
Git user
git at public.arvados.org
Wed Oct 6 13:00:00 UTC 2021
Summary of changes:
sdk/cli/arvados-cli.gemspec | 2 +-
sdk/ruby/arvados.gemspec | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
discards 647958facb785fb38cd81447da9db4757149d6ca (commit)
discards 481c0eb9d9b0adcf876c7e53ae4c8938b60c9b76 (commit)
discards 42df913507f45fad98b950e6aa7d0cf71ae92e67 (commit)
discards 23d74e8c4d1837d15381851d30ed241bf78b5e01 (commit)
discards e743140e6c9b2d7d82e18fa8327689dc73bf5cec (commit)
discards 735650984cac29cbc371bf8a11550a0f93ea92e2 (commit)
discards 1c95615a417b6435dc86f80e3f02563196449611 (commit)
discards c1872f15cdaf416ff9ccfc520085e8af1c5ed582 (commit)
discards 8cba1fc06c937f3c828a55703dad501c28915bda (commit)
discards e91291c2959c07c9929617e3d080077477a19b33 (commit)
discards 8d8ef7cca4001feb24eae52a9d458cafbf2de4fb (commit)
discards 04ea44bd67f6b0c2a700e6e9403629ad842e148d (commit)
discards 1b15db60a18e6bbb09a07e230b8f48630e05cdc1 (commit)
discards a3b5825a4fd71f2b8fbfdd288655f62af0816ca3 (commit)
discards 69b24ec1999603af6d8f36b083a9ce2fac9bd72b (commit)
discards 4b06eb378afc84b31e0986283fb7f807adb09a9b (commit)
discards ff05fbdf1fb4d68d519840d6f2007cf24064395f (commit)
discards a4c2309a118912e43469235c0dedef3a9f5dbfa7 (commit)
discards 9ce4b2c07e2b7809088e6333246501ad9418bbda (commit)
discards cfe9b58cc374794df00910b7708ffa245aaf03bf (commit)
discards 12932812e109e21e76a69b0a9f52533cdc76ea5d (commit)
discards 290695c429ea536bd4e4ee01ff76e0d770bc8021 (commit)
discards dcd77dd78f89a225cc9bcefc1930f8a54bc62791 (commit)
via 4b40e894f9606d0d1a4fa02db646eda77c99be61 (commit)
via 4c814e76fea378fe199f3a55b31445fa56fff02a (commit)
via 0516059abae641ed8a1e95581d070c71f3e943d3 (commit)
via c2a40d54421c4e5d221447419134fdca1520289f (commit)
via b4b8572bfbec0d5ce1008e36fc0cb21b476b5543 (commit)
via 718b62dc0b6710fff949aba5ea818d2a196c3a42 (commit)
via 3a817b3292c2af8d8d92d89df349d2e0321135f5 (commit)
via bc5301a4d4f3dc65847c2897a405dbbf3e56f0fd (commit)
via 1a9a02a3c7e711927a056d7cb222f261391822e5 (commit)
via f4cf648f18a8d82aaf674c474d385278c0527dab (commit)
via 63c6babe357292a234d6982bfcd67171b8259ec2 (commit)
via 63bd2a384496fe5e9d142b9354ec2f73911d17ec (commit)
via 2b79f4c95e82d9cc7840a59af52b5824259defc5 (commit)
via 6d8326a0f1521b0d7c49cd90dc775d409906bc49 (commit)
via 8fc8df8ca7e26ba757df0ac65cbd3a13bac3b01f (commit)
This update added new revisions after undoing existing revisions. That is
to say, the old revision is not a strict subset of the new revision. This
situation occurs when you --force push a change and generate a repository
containing something like this:
* -- * -- B -- O -- O -- O (647958facb785fb38cd81447da9db4757149d6ca)
\
N -- N -- N (4b40e894f9606d0d1a4fa02db646eda77c99be61)
When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 4b40e894f9606d0d1a4fa02db646eda77c99be61
Author: Ward Vandewege <ward at curii.com>
Date: Tue Oct 5 17:33:11 2021 -0400
Update the godoc path for keep-web in our documentation.
No issue #
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/install/install-keep-web.html.textile.liquid b/doc/install/install-keep-web.html.textile.liquid
index 9f63d1bcf..ea2ffb5e4 100644
--- a/doc/install/install-keep-web.html.textile.liquid
+++ b/doc/install/install-keep-web.html.textile.liquid
@@ -20,7 +20,7 @@ SPDX-License-Identifier: CC-BY-SA-3.0
h2(#introduction). Introduction
-The Keep-web server provides read/write access to files stored in Keep using WebDAV and S3 protocols. This makes it easy to access files in Keep from a browser, or mount Keep as a network folder using WebDAV support in various operating systems. It serves public data to unauthenticated clients, and serves private data to clients that supply Arvados API tokens. It can be installed anywhere with access to Keep services, typically behind a web proxy that provides TLS support. See the "godoc page":http://godoc.org/github.com/curoverse/arvados/services/keep-web for more detail.
+The Keep-web server provides read/write access to files stored in Keep using WebDAV and S3 protocols. This makes it easy to access files in Keep from a browser, or mount Keep as a network folder using WebDAV support in various operating systems. It serves public data to unauthenticated clients, and serves private data to clients that supply Arvados API tokens. It can be installed anywhere with access to Keep services, typically behind a web proxy that provides TLS support. See the "godoc page":https://pkg.go.dev/git.arvados.org/arvados.git/services/keep-web for more detail.
h2(#dns). Configure DNS
commit 4c814e76fea378fe199f3a55b31445fa56fff02a
Author: Ward Vandewege <ward at curii.com>
Date: Tue Oct 5 17:17:48 2021 -0400
18253: documentation updates around the Arvados GPG package signing key.
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/_includes/_install_redhat_key.liquid b/doc/_includes/_gpg_key_fingerprint.liquid
similarity index 82%
rename from doc/_includes/_install_redhat_key.liquid
rename to doc/_includes/_gpg_key_fingerprint.liquid
index c9430515f..a10fd8688 100644
--- a/doc/_includes/_install_redhat_key.liquid
+++ b/doc/_includes/_gpg_key_fingerprint.liquid
@@ -10,7 +10,6 @@ The Arvados signing key fingerprint is
<pre><code>pub rsa2048 2010-11-15 [SC]
B2DA 2991 656E B4A5 0314 CA2B 5716 5911 1078 ECD7
uid [ unknown] Arvados Automatic Signing Key <sysadmin at arvados.org>
-uid [ unknown] Curoverse, Inc Automatic Signing Key <sysadmin at curoverse.com>
sub rsa2048 2010-11-15 [E]
</code></pre>
</notextile>
diff --git a/doc/_includes/_install_debian_key.liquid b/doc/_includes/_install_debian_key.liquid
index adfdff873..1d5f73a4d 100644
--- a/doc/_includes/_install_debian_key.liquid
+++ b/doc/_includes/_install_debian_key.liquid
@@ -4,10 +4,14 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+<notextile>
+<pre><code># <span class="userinput">apt-get --no-install-recommends install curl gnupg2</span>
+# <span class="userinput">curl https://apt.arvados.org/pubkey.gpg -o /etc/apt/trusted.gpg.d/arvados.asc</span>
+</code></pre>
+</notextile>
+The Arvados package signing GPG key is also available via the keyservers, though they can be unreliable. To retrieve the signing key via keyserver.ubuntu.com:
<notextile>
-<pre><code># <span class="userinput">apt-get --no-install-recommends install gnupg</span>
-# <span class="userinput">/usr/bin/apt-key adv --keyserver pgp.mit.edu --recv 1078ECD7</span>
-</code></pre>
+<pre><code># <span class="userinput">/usr/bin/apt-key adv --keyserver keyserver.ubuntu.com --recv 1078ECD7</code></pre>
</notextile>
diff --git a/doc/install/packages.html.textile.liquid b/doc/install/packages.html.textile.liquid
index fb296ad5a..a111843a6 100644
--- a/doc/install/packages.html.textile.liquid
+++ b/doc/install/packages.html.textile.liquid
@@ -23,11 +23,11 @@ Packages are available for CentOS 7. To install them with yum, save this configu
name=Arvados
baseurl=http://rpm.arvados.org/CentOS/$releasever/os/$basearch/
gpgcheck=1
-gpgkey=http://rpm.arvados.org/CentOS/RPM-GPG-KEY-curoverse
+gpgkey=http://rpm.arvados.org/CentOS/RPM-GPG-KEY-arvados
</code></pre>
</notextile>
-{% include 'install_redhat_key' %}
+{% include 'gpg_key_fingerprint' %}
h3(#debian). Debian and Ubuntu
@@ -37,6 +37,8 @@ First, register the Arvados signing key in apt's database:
{% include 'install_debian_key' %}
+{% include 'gpg_key_fingerprint' %}
+
As root, add the Arvados package repository to your sources. This command depends on your OS vendor and version:
table(table table-bordered table-condensed).
commit 0516059abae641ed8a1e95581d070c71f3e943d3
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Oct 4 21:33:00 2021 -0400
Fix CWL tests refs #18238
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cwl/arvados_cwl/__init__.py b/sdk/cwl/arvados_cwl/__init__.py
index ee636be37..71ef742e3 100644
--- a/sdk/cwl/arvados_cwl/__init__.py
+++ b/sdk/cwl/arvados_cwl/__init__.py
@@ -301,7 +301,7 @@ def main(args, stdout, stderr, api_client=None, keep_client=None,
api_client.users().current().execute()
if keep_client is None:
keep_client = arvados.keep.KeepClient(api_client=api_client, num_retries=4)
- executor = ArvCwlExecutor(api_client, arvargs, keep_client=keep_client, num_retries=4)
+ executor = ArvCwlExecutor(api_client, arvargs, keep_client=keep_client, num_retries=4, stdout=stdout)
except WorkflowException as e:
logger.error(e, exc_info=(sys.exc_info()[1] if arvargs.debug else False))
return 1
diff --git a/sdk/cwl/arvados_cwl/arvcontainer.py b/sdk/cwl/arvados_cwl/arvcontainer.py
index 560e8979e..ae3c66889 100644
--- a/sdk/cwl/arvados_cwl/arvcontainer.py
+++ b/sdk/cwl/arvados_cwl/arvcontainer.py
@@ -520,10 +520,10 @@ class RunnerContainer(Runner):
if runtimeContext.debug:
command.append("--debug")
- if runtimeContext.storage_classes != "default":
+ if runtimeContext.storage_classes != "default" and runtimeContext.storage_classes:
command.append("--storage-classes=" + runtimeContext.storage_classes)
- if runtimeContext.intermediate_storage_classes != "default":
+ if runtimeContext.intermediate_storage_classes != "default" and runtimeContext.intermediate_storage_classes:
command.append("--intermediate-storage-classes=" + runtimeContext.intermediate_storage_classes)
if self.on_error:
diff --git a/sdk/cwl/arvados_cwl/executor.py b/sdk/cwl/arvados_cwl/executor.py
index db1db2816..aa19633d8 100644
--- a/sdk/cwl/arvados_cwl/executor.py
+++ b/sdk/cwl/arvados_cwl/executor.py
@@ -99,7 +99,8 @@ class ArvCwlExecutor(object):
arvargs=None,
keep_client=None,
num_retries=4,
- thread_count=4):
+ thread_count=4,
+ stdout=sys.stdout):
if arvargs is None:
arvargs = argparse.Namespace()
@@ -132,6 +133,7 @@ class ArvCwlExecutor(object):
self.should_estimate_cache_size = True
self.fs_access = None
self.secret_store = None
+ self.stdout = stdout
if keep_client is not None:
self.keep_client = keep_client
@@ -602,14 +604,15 @@ The 'jobs' API is no longer supported.
if existing_uuid or runtimeContext.create_workflow:
# Create a pipeline template or workflow record and exit.
if self.work_api == "containers":
- return (upload_workflow(self, tool, job_order,
+ uuid = upload_workflow(self, tool, job_order,
self.project_uuid,
uuid=existing_uuid,
submit_runner_ram=runtimeContext.submit_runner_ram,
name=runtimeContext.name,
merged_map=merged_map,
- submit_runner_image=runtimeContext.submit_runner_image),
- "success")
+ submit_runner_image=runtimeContext.submit_runner_image)
+ self.stdout.write(uuid + "\n")
+ return (None, "success")
self.apply_reqs(job_order, tool)
@@ -679,7 +682,8 @@ The 'jobs' API is no longer supported.
if runtimeContext.submit and not runtimeContext.wait:
runnerjob = next(jobiter)
runnerjob.run(runtimeContext)
- return (runnerjob.uuid, "success")
+ self.stdout.write(runnerjob.uuid+"\n")
+ return (None, "success")
current_container = arvados_cwl.util.get_current_container(self.api, self.num_retries, logger)
if current_container:
diff --git a/sdk/cwl/tests/test_container.py b/sdk/cwl/tests/test_container.py
index 8a380ff80..1a2bd112f 100644
--- a/sdk/cwl/tests/test_container.py
+++ b/sdk/cwl/tests/test_container.py
@@ -112,6 +112,7 @@ class TestContainer(unittest.TestCase):
runner.ignore_docker_for_reuse = False
runner.intermediate_output_ttl = 0
runner.secret_store = cwltool.secrets.SecretStore()
+ runner.api._rootDesc = {"revision": "20210628"}
keepdocker.return_value = [("zzzzz-4zz18-zzzzzzzzzzzzzz3", "")]
runner.api.collections().get().execute.return_value = {
@@ -175,6 +176,7 @@ class TestContainer(unittest.TestCase):
runner.ignore_docker_for_reuse = False
runner.intermediate_output_ttl = 3600
runner.secret_store = cwltool.secrets.SecretStore()
+ runner.api._rootDesc = {"revision": "20210628"}
keepdocker.return_value = [("zzzzz-4zz18-zzzzzzzzzzzzzz3", "")]
runner.api.collections().get().execute.return_value = {
@@ -270,6 +272,7 @@ class TestContainer(unittest.TestCase):
runner.ignore_docker_for_reuse = False
runner.intermediate_output_ttl = 0
runner.secret_store = cwltool.secrets.SecretStore()
+ runner.api._rootDesc = {"revision": "20210628"}
keepdocker.return_value = [("zzzzz-4zz18-zzzzzzzzzzzzzz3", "")]
runner.api.collections().get().execute.return_value = {
@@ -398,6 +401,7 @@ class TestContainer(unittest.TestCase):
runner.ignore_docker_for_reuse = False
runner.intermediate_output_ttl = 0
runner.secret_store = cwltool.secrets.SecretStore()
+ runner.api._rootDesc = {"revision": "20210628"}
keepdocker.return_value = [("zzzzz-4zz18-zzzzzzzzzzzzzz3", "")]
runner.api.collections().get().execute.return_value = {
@@ -622,6 +626,7 @@ class TestContainer(unittest.TestCase):
runner.ignore_docker_for_reuse = False
runner.intermediate_output_ttl = 0
runner.secret_store = cwltool.secrets.SecretStore()
+ runner.api._rootDesc = {"revision": "20210628"}
keepdocker.return_value = [("zzzzz-4zz18-zzzzzzzzzzzzzz3", "")]
runner.api.collections().get().execute.return_value = {
@@ -712,6 +717,7 @@ class TestContainer(unittest.TestCase):
runner.ignore_docker_for_reuse = False
runner.intermediate_output_ttl = 0
runner.secret_store = cwltool.secrets.SecretStore()
+ runner.api._rootDesc = {"revision": "20210628"}
keepdocker.return_value = [("zzzzz-4zz18-zzzzzzzzzzzzzz3", "")]
runner.api.collections().get().execute.return_value = {
@@ -808,6 +814,7 @@ class TestContainer(unittest.TestCase):
runner.ignore_docker_for_reuse = False
runner.intermediate_output_ttl = 0
runner.secret_store = cwltool.secrets.SecretStore()
+ runner.api._rootDesc = {"revision": "20210628"}
keepdocker.return_value = [("zzzzz-4zz18-zzzzzzzzzzzzzz3", "")]
runner.api.collections().get().execute.return_value = {
@@ -851,6 +858,7 @@ class TestContainer(unittest.TestCase):
runner.ignore_docker_for_reuse = False
runner.intermediate_output_ttl = 0
runner.secret_store = cwltool.secrets.SecretStore()
+ runner.api._rootDesc = {"revision": "20210628"}
keepdocker.return_value = [("zzzzz-4zz18-zzzzzzzzzzzzzz3", "")]
runner.api.collections().get().execute.return_value = {
@@ -924,6 +932,7 @@ class TestContainer(unittest.TestCase):
runner.ignore_docker_for_reuse = False
runner.intermediate_output_ttl = 0
runner.secret_store = cwltool.secrets.SecretStore()
+ runner.api._rootDesc = {"revision": "20210628"}
keepdocker.return_value = [("zzzzz-4zz18-zzzzzzzzzzzzzz3", "")]
runner.api.collections().get().execute.return_value = {
diff --git a/sdk/cwl/tests/test_submit.py b/sdk/cwl/tests/test_submit.py
index 8c0fcaf74..77f70851e 100644
--- a/sdk/cwl/tests/test_submit.py
+++ b/sdk/cwl/tests/test_submit.py
@@ -87,6 +87,7 @@ def stubs(func):
stubs.api = mock.MagicMock()
stubs.api._rootDesc = get_rootDesc()
stubs.api._rootDesc["uuidPrefix"] = "zzzzz"
+ stubs.api._rootDesc["revision"] = "20210628"
stubs.api.users().current().execute.return_value = {
"uuid": stubs.fake_user_uuid,
@@ -446,7 +447,7 @@ class TestSubmit(unittest.TestCase):
"enableReuse": False,
},
]
- expect_container["mounts"]["/var/lib/cwl/workflow.json"]["content"]["$graph"][0]["$namespaces"] = {
+ expect_container["mounts"]["/var/lib/cwl/workflow.json"]["content"]["$namespaces"] = {
"arv": "http://arvados.org/cwl#",
"cwltool": "http://commonwl.org/cwltool#"
}
@@ -572,6 +573,7 @@ class TestSubmit(unittest.TestCase):
def test_default_storage_classes_correctly_propagate_to_make_output_collection(self, stubs, make_output, job, tq):
final_output_c = arvados.collection.Collection()
make_output.return_value = ({},final_output_c)
+ stubs.api.config().get.return_value = {"default": {"Default": True}}
def set_final_output(job_order, output_callback, runtimeContext):
output_callback("zzzzz-4zz18-zzzzzzzzzzzzzzzz", "success")
@@ -1032,7 +1034,7 @@ class TestSubmit(unittest.TestCase):
"keep_cache": 512
}
]
- expect_container["mounts"]["/var/lib/cwl/workflow.json"]["content"]["$graph"][0]["$namespaces"] = {
+ expect_container["mounts"]["/var/lib/cwl/workflow.json"]["content"]["$namespaces"] = {
"arv": "http://arvados.org/cwl#",
}
expect_container['command'] = ['arvados-cwl-runner', '--local', '--api=containers',
@@ -1126,9 +1128,6 @@ class TestSubmit(unittest.TestCase):
"content": {
"$graph": [
{
- "$namespaces": {
- "cwltool": "http://commonwl.org/cwltool#"
- },
"arguments": [
"md5sum",
"example.conf"
@@ -1217,6 +1216,9 @@ class TestSubmit(unittest.TestCase):
]
}
],
+ "$namespaces": {
+ "cwltool": "http://commonwl.org/cwltool#"
+ },
"cwlVersion": "v1.0"
},
"kind": "json"
@@ -1445,7 +1447,7 @@ class TestSubmit(unittest.TestCase):
],
}
]
- expect_container["mounts"]["/var/lib/cwl/workflow.json"]["content"]["$graph"][0]["$namespaces"] = {
+ expect_container["mounts"]["/var/lib/cwl/workflow.json"]["content"]["$namespaces"] = {
"arv": "http://arvados.org/cwl#"
}
diff --git a/sdk/cwl/tests/wf/scatter2_subwf.cwl b/sdk/cwl/tests/wf/scatter2_subwf.cwl
index c54e1707f..218b0c501 100644
--- a/sdk/cwl/tests/wf/scatter2_subwf.cwl
+++ b/sdk/cwl/tests/wf/scatter2_subwf.cwl
@@ -86,5 +86,8 @@
]
}
],
+ "$namespaces": {
+ "arv": "http://arvados.org/cwl#"
+ },
"cwlVersion": "v1.0"
-}
\ No newline at end of file
+}
commit c2a40d54421c4e5d221447419134fdca1520289f
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Oct 4 17:40:40 2021 -0400
18238: Make sure to pull default image
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cwl/arvados_cwl/arvdocker.py b/sdk/cwl/arvados_cwl/arvdocker.py
index 3c8208271..26408317c 100644
--- a/sdk/cwl/arvados_cwl/arvdocker.py
+++ b/sdk/cwl/arvados_cwl/arvdocker.py
@@ -49,8 +49,10 @@ def arv_docker_get_image(api_client, dockerRequirement, pull_image, project_uuid
if not images:
# Fetch Docker image if necessary.
try:
- cwltool.docker.DockerCommandLineJob.get_image(dockerRequirement, pull_image,
+ result = cwltool.docker.DockerCommandLineJob.get_image(dockerRequirement, pull_image,
force_pull, tmp_outdir_prefix)
+ if not result:
+ raise WorkflowException("Docker image '%s' not available" % dockerRequirement["dockerImageId"])
except OSError as e:
raise WorkflowException("While trying to get Docker image '%s', failed to execute 'docker': %s" % (dockerRequirement["dockerImageId"], e))
diff --git a/sdk/cwl/arvados_cwl/arvtool.py b/sdk/cwl/arvados_cwl/arvtool.py
index 83648f46a..b66e8ad3a 100644
--- a/sdk/cwl/arvados_cwl/arvtool.py
+++ b/sdk/cwl/arvados_cwl/arvtool.py
@@ -62,7 +62,7 @@ class ArvadosCommandTool(CommandLineTool):
(docker_req, docker_is_req) = self.get_requirement("DockerRequirement")
if not docker_req:
self.hints.append({"class": "DockerRequirement",
- "dockerImageId": "arvados/jobs:"+__version__})
+ "dockerPull": "arvados/jobs:"+__version__})
self.arvrunner = arvrunner
commit b4b8572bfbec0d5ce1008e36fc0cb21b476b5543
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Oct 4 15:25:32 2021 -0400
18238: Update tests to use newer arvados/jobs image
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cwl/tests/wf/runin-reqs-wf.cwl b/sdk/cwl/tests/wf/runin-reqs-wf.cwl
index 3cc52936e..22cc82b7f 100644
--- a/sdk/cwl/tests/wf/runin-reqs-wf.cwl
+++ b/sdk/cwl/tests/wf/runin-reqs-wf.cwl
@@ -24,7 +24,7 @@ requirements:
StepInputExpressionRequirement: {}
hints:
DockerRequirement:
- dockerPull: arvados/jobs:1.4.0.20190604172024
+ dockerPull: arvados/jobs:2.2.2
steps:
substep:
in:
diff --git a/sdk/cwl/tests/wf/runin-reqs-wf2.cwl b/sdk/cwl/tests/wf/runin-reqs-wf2.cwl
index 7d06cb308..4bde6c562 100644
--- a/sdk/cwl/tests/wf/runin-reqs-wf2.cwl
+++ b/sdk/cwl/tests/wf/runin-reqs-wf2.cwl
@@ -24,7 +24,7 @@ requirements:
StepInputExpressionRequirement: {}
hints:
DockerRequirement:
- dockerPull: arvados/jobs:1.4.0.20190604172024
+ dockerPull: arvados/jobs:2.2.2
steps:
substep:
in:
diff --git a/sdk/cwl/tests/wf/runin-reqs-wf4.cwl b/sdk/cwl/tests/wf/runin-reqs-wf4.cwl
index 9a26d0113..d00ee8577 100644
--- a/sdk/cwl/tests/wf/runin-reqs-wf4.cwl
+++ b/sdk/cwl/tests/wf/runin-reqs-wf4.cwl
@@ -24,7 +24,7 @@ requirements:
StepInputExpressionRequirement: {}
hints:
DockerRequirement:
- dockerPull: arvados/jobs:1.4.0.20190604172024
+ dockerPull: arvados/jobs:2.2.2
steps:
substep:
in:
diff --git a/sdk/cwl/tests/wf/runin-reqs-wf5.cwl b/sdk/cwl/tests/wf/runin-reqs-wf5.cwl
index 34d7b2c39..647b07edf 100644
--- a/sdk/cwl/tests/wf/runin-reqs-wf5.cwl
+++ b/sdk/cwl/tests/wf/runin-reqs-wf5.cwl
@@ -24,7 +24,7 @@ requirements:
StepInputExpressionRequirement: {}
hints:
DockerRequirement:
- dockerPull: arvados/jobs:1.4.0.20190604172024
+ dockerPull: arvados/jobs:2.2.2
steps:
substep:
in:
diff --git a/sdk/cwl/tests/wf/runin-wf.cwl b/sdk/cwl/tests/wf/runin-wf.cwl
index 68a26a0d3..f819d0fe6 100644
--- a/sdk/cwl/tests/wf/runin-wf.cwl
+++ b/sdk/cwl/tests/wf/runin-wf.cwl
@@ -36,7 +36,7 @@ steps:
hints:
- class: arv:RunInSingleContainer
- class: DockerRequirement
- dockerPull: arvados/jobs:1.4.0.20190604172024
+ dockerPull: arvados/jobs:2.2.2
run:
class: Workflow
id: mysub
commit 718b62dc0b6710fff949aba5ea818d2a196c3a42
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Oct 1 16:31:06 2021 -0400
18238: Fix API revision check
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cwl/arvados_cwl/arvcontainer.py b/sdk/cwl/arvados_cwl/arvcontainer.py
index 165cbcf18..560e8979e 100644
--- a/sdk/cwl/arvados_cwl/arvcontainer.py
+++ b/sdk/cwl/arvados_cwl/arvcontainer.py
@@ -284,7 +284,7 @@ class ArvadosContainer(JobBase):
raise WorkflowException("Invalid value %d for output_ttl, cannot be less than zero" % container_request["output_ttl"])
- if api._rootDesc["revision"] >= "20210628":
+ if self.arvrunner.api._rootDesc["revision"] >= "20210628":
storage_class_req, _ = self.get_requirement("http://arvados.org/cwl#OutputStorageClass")
if storage_class_req and storage_class_req.get("intermediateStorageClass"):
container_request["output_storage_classes"] = aslist(storage_class_req["intermediateStorageClass"])
commit 3a817b3292c2af8d8d92d89df349d2e0321135f5
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Oct 1 10:26:42 2021 -0400
18238: Check API version before using output_storage_classes
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cwl/arvados_cwl/arvcontainer.py b/sdk/cwl/arvados_cwl/arvcontainer.py
index 1e79566f4..165cbcf18 100644
--- a/sdk/cwl/arvados_cwl/arvcontainer.py
+++ b/sdk/cwl/arvados_cwl/arvcontainer.py
@@ -283,11 +283,13 @@ class ArvadosContainer(JobBase):
if self.output_ttl < 0:
raise WorkflowException("Invalid value %d for output_ttl, cannot be less than zero" % container_request["output_ttl"])
- storage_class_req, _ = self.get_requirement("http://arvados.org/cwl#OutputStorageClass")
- if storage_class_req and storage_class_req.get("intermediateStorageClass"):
- container_request["output_storage_classes"] = aslist(storage_class_req["intermediateStorageClass"])
- else:
- container_request["output_storage_classes"] = runtimeContext.intermediate_storage_classes.strip().split(",")
+
+ if api._rootDesc["revision"] >= "20210628":
+ storage_class_req, _ = self.get_requirement("http://arvados.org/cwl#OutputStorageClass")
+ if storage_class_req and storage_class_req.get("intermediateStorageClass"):
+ container_request["output_storage_classes"] = aslist(storage_class_req["intermediateStorageClass"])
+ else:
+ container_request["output_storage_classes"] = runtimeContext.intermediate_storage_classes.strip().split(",")
if self.timelimit is not None and self.timelimit > 0:
scheduling_parameters["max_run_time"] = self.timelimit
diff --git a/services/api/app/controllers/arvados/v1/schema_controller.rb b/services/api/app/controllers/arvados/v1/schema_controller.rb
index 6c5472624..c1d4b74d6 100644
--- a/services/api/app/controllers/arvados/v1/schema_controller.rb
+++ b/services/api/app/controllers/arvados/v1/schema_controller.rb
@@ -37,7 +37,7 @@ class Arvados::V1::SchemaController < ApplicationController
# format is YYYYMMDD, must be fixed width (needs to be lexically
# sortable), updated manually, may be used by clients to
# determine availability of API server features.
- revision: "20210503",
+ revision: "20210628",
source_version: AppVersion.hash,
sourceVersion: AppVersion.hash, # source_version should be deprecated in the future
packageVersion: AppVersion.package_version,
commit bc5301a4d4f3dc65847c2897a405dbbf3e56f0fd
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Oct 1 10:02:40 2021 -0400
18238: Make backwards compatible when StorageClasses is not in config
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cwl/arvados_cwl/executor.py b/sdk/cwl/arvados_cwl/executor.py
index edb9d5b52..db1db2816 100644
--- a/sdk/cwl/arvados_cwl/executor.py
+++ b/sdk/cwl/arvados_cwl/executor.py
@@ -549,7 +549,7 @@ The 'jobs' API is no longer supported.
if runtimeContext.submit_request_uuid and self.work_api != "containers":
raise Exception("--submit-request-uuid requires containers API, but using '{}' api".format(self.work_api))
- default_storage_classes = ",".join([k for k,v in self.api.config()["StorageClasses"].items() if v.get("Default") is True])
+ default_storage_classes = ",".join([k for k,v in self.api.config().get("StorageClasses", {"default": {"Default": True}}).items() if v.get("Default") is True])
if runtimeContext.storage_classes == "default":
runtimeContext.storage_classes = default_storage_classes
if runtimeContext.intermediate_storage_classes == "default":
commit 1a9a02a3c7e711927a056d7cb222f261391822e5
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu Sep 30 16:13:28 2021 -0400
18238: Add required container_engine parameter to builder
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cwl/arvados_cwl/runner.py b/sdk/cwl/arvados_cwl/runner.py
index 66dff809e..126a2b511 100644
--- a/sdk/cwl/arvados_cwl/runner.py
+++ b/sdk/cwl/arvados_cwl/runner.py
@@ -105,7 +105,8 @@ def make_builder(joborder, hints, requirements, runtimeContext, metadata):
outdir="", # type: Text
tmpdir="", # type: Text
stagedir="", # type: Text
- cwlVersion=metadata.get("http://commonwl.org/cwltool#original_cwlVersion") or metadata.get("cwlVersion")
+ cwlVersion=metadata.get("http://commonwl.org/cwltool#original_cwlVersion") or metadata.get("cwlVersion"),
+ container_engine="docker"
)
def search_schemadef(name, reqs):
commit f4cf648f18a8d82aaf674c474d385278c0527dab
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu Sep 30 10:18:12 2021 -0400
18238: Update cwltool and schema-salad deps
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/sdk/cwl/setup.py b/sdk/cwl/setup.py
index 3f1f8a6be..0bde76e92 100644
--- a/sdk/cwl/setup.py
+++ b/sdk/cwl/setup.py
@@ -39,8 +39,8 @@ setup(name='arvados-cwl-runner',
# file to determine what version of cwltool and schema-salad to
# build.
install_requires=[
- 'cwltool==3.1.20210816212154',
- 'schema-salad==8.2.20210902094147',
+ 'cwltool==3.1.20210922203925',
+ 'schema-salad==8.2.20210918131710',
'arvados-python-client{}'.format(pysdk_dep),
'setuptools',
'ciso8601 >= 2.0.0',
commit 63c6babe357292a234d6982bfcd67171b8259ec2
Author: Ward Vandewege <ward at curii.com>
Date: Mon Oct 4 21:28:53 2021 -0400
18247: fix arvados gem dependency on the faraday gem
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/sdk/cli/arvados-cli.gemspec b/sdk/cli/arvados-cli.gemspec
index 08fcfe3a3..1ff841acd 100644
--- a/sdk/cli/arvados-cli.gemspec
+++ b/sdk/cli/arvados-cli.gemspec
@@ -41,8 +41,9 @@ Gem::Specification.new do |s|
s.required_ruby_version = '>= 2.1.0'
s.add_runtime_dependency 'arvados', '>= 1.4.1.20190320201707'
# Our google-api-client dependency used to be < 0.9, but that could be
- # satisfied by the buggy 0.9.pre*. https://dev.arvados.org/issues/9213
- s.add_runtime_dependency 'arvados-google-api-client', '~> 0.6', '>= 0.6.3', '<0.8.9'
+ # satisfied by the buggy 0.9.pre*, cf. https://dev.arvados.org/issues/9213
+ # We need at least version 0.8.7.3, cf. https://dev.arvados.org/issues/15673
+ s.add_runtime_dependency('arvados-google-api-client', '>= 0.8.7.3', '< 0.8.9')
s.add_runtime_dependency 'activesupport', '>= 3.2.13', '< 5.3'
s.add_runtime_dependency 'json', '>= 1.7.7', '<3'
s.add_runtime_dependency 'optimist', '~> 3.0'
@@ -51,8 +52,6 @@ Gem::Specification.new do |s|
s.add_runtime_dependency 'oj', '< 3.10.9'
s.add_runtime_dependency 'curb', '~> 0.8'
s.add_runtime_dependency 'launchy', '< 2.5'
- # arvados-google-api-client 0.8.7.2 is incompatible with faraday 0.16.2
- s.add_dependency('faraday', '< 0.16')
s.homepage =
'https://arvados.org'
end
diff --git a/sdk/ruby/arvados.gemspec b/sdk/ruby/arvados.gemspec
index 7cc2fd931..b196a1c33 100644
--- a/sdk/ruby/arvados.gemspec
+++ b/sdk/ruby/arvados.gemspec
@@ -41,13 +41,14 @@ Gem::Specification.new do |s|
s.add_dependency('activesupport', '>= 3')
s.add_dependency('andand', '~> 1.3', '>= 1.3.3')
# Our google-api-client dependency used to be < 0.9, but that could be
- # satisfied by the buggy 0.9.pre*. https://dev.arvados.org/issues/9213
- s.add_dependency('arvados-google-api-client', '>= 0.7', '< 0.8.9')
+ # satisfied by the buggy 0.9.pre*, cf. https://dev.arvados.org/issues/9213
+ # We need at least version 0.8.7.3, cf. https://dev.arvados.org/issues/15673
+ s.add_dependency('arvados-google-api-client', '>= 0.8.7.3', '< 0.8.9')
# work around undeclared dependency on i18n in some activesupport 3.x.x:
s.add_dependency('i18n', '~> 0')
s.add_dependency('json', '>= 1.7.7', '<3')
- # arvados-google-api-client 0.8.7.2 is incompatible with faraday 0.16.2
- s.add_dependency('faraday', '< 0.16')
+ # Avoid warning on Ruby 2.7, cf. https://dev.arvados.org/issues/18247
+ s.add_dependency('faraday', '>= 0.17.4')
s.add_runtime_dependency('jwt', '<2', '>= 0.1.5')
s.homepage =
'https://arvados.org'
commit 63bd2a384496fe5e9d142b9354ec2f73911d17ec
Author: Ward Vandewege <ward at curii.com>
Date: Fri Oct 1 16:25:39 2021 -0400
18245: Update the "Ruby and Bundler" installation page:
- installing the system bundler package is no longer needed
- the RVM installation method already installs bundler, no need to do
that explicitly
- update the references to the latest Ruby 2.5.x version
- remove reference to Ubuntu 16.04 which is no longer supported
- update the "install from source" documentation to make it work with
the Arvados rails packages
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/doc/_includes/_install_ruby_and_bundler.liquid b/doc/_includes/_install_ruby_and_bundler.liquid
index fe7714c62..ffaa1a158 100644
--- a/doc/_includes/_install_ruby_and_bundler.liquid
+++ b/doc/_includes/_install_ruby_and_bundler.liquid
@@ -22,10 +22,10 @@ The Ruby version shipped with Centos 7 is too old. Use "RVM":#rvm to install Ru
h3. Debian and Ubuntu
-Debian 10 (buster) and Ubuntu 18.04 (bionic) and later ship with Ruby 2.5, which is supported by Arvados.
+Debian 10 (buster) and Ubuntu 18.04 (bionic) and later ship with Ruby 2.5 or newer, which is sufficient for Arvados.
<notextile>
-<pre><code># <span class="userinput">apt-get --no-install-recommends install ruby ruby-dev bundler</span></code></pre>
+<pre><code># <span class="userinput">apt-get --no-install-recommends install ruby ruby-dev</span></code></pre>
</notextile>
h2(#rvm). Option 2: Install with RVM
@@ -44,34 +44,30 @@ h4. Debian and Ubuntu
apt-get --no-install-recommends install gpg curl
</pre>
-h3. Install RVM
+h3. Install RVM, Ruby and Bundler
<notextile>
<pre><code># <span class="userinput">gpg --keyserver pgp.mit.edu --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
\curl -sSL https://get.rvm.io | bash -s stable --ruby=2.5
</span></code></pre></notextile>
+This command installs the latest Ruby 2.5.x release, as well as the @gem@ and @bundle@ commands.
+
To use Ruby installed from RVM, load it in an open shell like this:
<notextile>
-<pre><code><span class="userinput">. /usr/local/rvm/scripts/rvm
+<pre><code><span class="userinput">source /usr/local/rvm/scripts/rvm
</span></code></pre></notextile>
Alternately you can use @rvm-exec@ (the first parameter is the ruby version to use, or "default"), for example:
<notextile>
-<pre><code><span class="userinput">rvm-exec default rails console
+<pre><code><span class="userinput">rvm-exec default ruby -v
</span></code></pre></notextile>
-Finally, install Bundler:
-
-<notextile>
-<pre><code>~$ <span class="userinput">gem install bundler</span>
-</code></pre></notextile>
-
h2(#fromsource). Option 3: Install from source
-Install prerequisites for Debian 10:
+Install prerequisites for Debian 10, Ubuntu 18.04 and Ubuntu 20.04:
<notextile>
<pre><code><span class="userinput">sudo apt-get install \
@@ -89,25 +85,20 @@ Install prerequisites for CentOS 7:
make automake libtool bison sqlite-devel tar
</span></code></pre></notextile>
-Install prerequisites for Ubuntu 16.04:
-
-<notextile>
-<pre><code><span class="userinput">sudo apt-get install \
- bison build-essential gettext libcurl3 \
- libcurl3-openssl-dev libpcre3-dev libreadline-dev \
- libssl-dev libxslt1.1 zlib1g-dev
-</span></code></pre></notextile>
-
Build and install Ruby:
<notextile>
<pre><code><span class="userinput">mkdir -p ~/src
cd ~/src
-curl -f http://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.5.tar.gz | tar xz
-cd ruby-2.5.5
+curl -f http://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.8.tar.gz | tar xz
+cd ruby-2.5.8
./configure --disable-install-rdoc
make
sudo make install
+# Make sure the post install script can find the gem and ruby executables
+sudo ln -s /usr/local/bin/gem /usr/bin/gem
+sudo ln -s /usr/local/bin/ruby /usr/bin/ruby
+# Install bundler
sudo -i gem install bundler</span>
</code></pre></notextile>
commit 2b79f4c95e82d9cc7840a59af52b5824259defc5
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Thu Sep 30 09:50:08 2021 -0400
Add Workbench to arvbox status, no issue #
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/tools/arvbox/bin/arvbox b/tools/arvbox/bin/arvbox
index 516ea3534..36a33376a 100755
--- a/tools/arvbox/bin/arvbox
+++ b/tools/arvbox/bin/arvbox
@@ -526,6 +526,7 @@ case "$subcmd" in
echo "Status: running"
echo "Container IP: $(getip)"
echo "Published host: $(gethost)"
+ echo "Workbench: https://$(gethost)"
else
echo "Status: not running"
fi
commit 6d8326a0f1521b0d7c49cd90dc775d409906bc49
Author: Tom Clegg <tom at curii.com>
Date: Wed Sep 29 10:17:11 2021 -0400
17830: Add test for logging request ID.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at curii.com>
diff --git a/services/api/test/integration/logging_test.rb b/services/api/test/integration/logging_test.rb
new file mode 100644
index 000000000..cbf9681d4
--- /dev/null
+++ b/services/api/test/integration/logging_test.rb
@@ -0,0 +1,28 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+require 'stringio'
+
+class LoggingTest < ActionDispatch::IntegrationTest
+ fixtures :collections
+
+ test "request_id" do
+ buf = StringIO.new
+ logcopy = ActiveSupport::Logger.new(buf)
+ logcopy.level = :info
+ begin
+ Rails.logger.extend(ActiveSupport::Logger.broadcast(logcopy))
+ get "/arvados/v1/collections/#{collections(:foo_file).uuid}",
+ params: {:format => :json},
+ headers: auth(:active).merge({ 'X-Request-Id' => 'req-aaaaaaaaaaaaaaaaaaaa' })
+ assert_response :success
+ assert_match /^{.*"request_id":"req-aaaaaaaaaaaaaaaaaaaa"/, buf.string
+ ensure
+ # We don't seem to have an "unbroadcast" option, so this is how
+ # we avoid filling buf with unlimited logs from subsequent
+ # tests.
+ logcopy.level = :fatal
+ end
+ end
+end
commit 8fc8df8ca7e26ba757df0ac65cbd3a13bac3b01f
Author: Ward Vandewege <ward at curii.com>
Date: Tue Sep 28 14:25:42 2021 -0400
The request_id passing was moved to Rails middleware as part of #17830.
Unfortunately, in that process the logging of the request_id parameter
in the rails logs was broken. This commit fixes that.
refs #17830
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward at curii.com>
diff --git a/services/api/app/controllers/application_controller.rb b/services/api/app/controllers/application_controller.rb
index ff4cb88ae..3750befb3 100644
--- a/services/api/app/controllers/application_controller.rb
+++ b/services/api/app/controllers/application_controller.rb
@@ -427,7 +427,7 @@ class ApplicationController < ActionController::Base
def append_info_to_payload(payload)
super
- payload[:request_id] = response.headers['X-Request-Id']
+ payload[:request_id] = request.request_id
payload[:client_ipaddr] = @remote_ip
payload[:client_auth] = current_api_client_authorization.andand.uuid || nil
end
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list