[ARVADOS] updated: 1.2.0-333-g02c2b3791
Git user
git at public.curoverse.com
Tue Nov 13 14:04:04 EST 2018
Summary of changes:
.../cases/{base-case.cwl => remote-case.cwl} | 2 +-
...-home-to-remote.cwl => twostep-both-remote.cwl} | 5 +-
.../federation/cases/twostep-home-to-remote.cwl | 3 -
.../federation/cases/twostep-remote-to-home.cwl | 8 +-
.../{base-case-input.txt => remote-case-input.txt} | 2 +-
...base-case-input.txt => twostep-both-remote.txt} | 2 +-
sdk/cwl/tests/federation/framework/prepare.py | 1 -
sdk/cwl/tests/federation/main.cwl | 142 ++++++++++++++++++++-
8 files changed, 149 insertions(+), 16 deletions(-)
copy sdk/cwl/tests/federation/cases/{base-case.cwl => remote-case.cwl} (88%)
copy sdk/cwl/tests/federation/cases/{twostep-home-to-remote.cwl => twostep-both-remote.cwl} (81%)
copy sdk/cwl/tests/federation/data/{base-case-input.txt => remote-case-input.txt} (93%)
copy sdk/cwl/tests/federation/data/{base-case-input.txt => twostep-both-remote.txt} (92%)
via 02c2b37917b15e2b93cdf0ea45bc4a8377a72c69 (commit)
via 51de99d5a9ea522d7068c54645d34f4fd54cf2d0 (commit)
via e40d1363c5266bf80d4e77b9f3bc01ca3062fabe (commit)
from 295907e5496eb5a3661ecfffd026c3f7aa6a053f (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 02c2b37917b15e2b93cdf0ea45bc4a8377a72c69
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Fri Nov 9 15:48:14 2018 -0500
14198: Add some basic doc comments.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/sdk/cwl/tests/federation/main.cwl b/sdk/cwl/tests/federation/main.cwl
index 4b95d8c03..732661220 100755
--- a/sdk/cwl/tests/federation/main.cwl
+++ b/sdk/cwl/tests/federation/main.cwl
@@ -54,6 +54,9 @@ outputs:
steps:
base-case:
+ doc: |
+ Base case (no federation), single step workflow with both the
+ runner and step on the same cluster.
in:
arvados_api_token: arvados_api_token
arvado_api_host_insecure: arvado_api_host_insecure
@@ -87,6 +90,9 @@ steps:
run: framework/testcase.cwl
runner-home-step-remote:
+ doc: |
+ Single step workflow with the runner on the home cluster and the
+ step on the remote cluster.
in:
arvados_api_token: arvados_api_token
arvado_api_host_insecure: arvado_api_host_insecure
@@ -121,6 +127,9 @@ steps:
run: framework/testcase.cwl
runner-remote-step-home:
+ doc: |
+ Single step workflow with the runner on the remote cluster and the
+ step on the home cluster.
in:
arvados_api_token: arvados_api_token
arvado_api_host_insecure: arvado_api_host_insecure
@@ -155,6 +164,9 @@ steps:
run: framework/testcase.cwl
remote-case:
+ doc: |
+ Single step workflow with both the runner and the step on the
+ remote cluster.
in:
arvados_api_token: arvados_api_token
arvado_api_host_insecure: arvado_api_host_insecure
@@ -189,6 +201,10 @@ steps:
run: framework/testcase.cwl
twostep-home-to-remote:
+ doc: |
+ Two step workflow. The runner is on the home cluster, the first
+ step is on the home cluster, the second step is on the remote
+ cluster.
in:
arvados_api_token: arvados_api_token
arvado_api_host_insecure: arvado_api_host_insecure
@@ -227,6 +243,10 @@ steps:
run: framework/testcase.cwl
twostep-remote-to-home:
+ doc: |
+ Two step workflow. The runner is on the home cluster, the first
+ step is on the remote cluster, the second step is on the home
+ cluster.
in:
arvados_api_token: arvados_api_token
arvado_api_host_insecure: arvado_api_host_insecure
@@ -265,6 +285,9 @@ steps:
run: framework/testcase.cwl
twostep-both-remote:
+ doc: |
+ Two step workflow. The runner is on the home cluster, both steps are
+ on the remote cluster.
in:
arvados_api_token: arvados_api_token
arvado_api_host_insecure: arvado_api_host_insecure
commit 51de99d5a9ea522d7068c54645d34f4fd54cf2d0
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Fri Nov 9 15:33:05 2018 -0500
14198: Add remote-case and twostep-both-remote
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/sdk/cwl/tests/federation/cases/remote-case.cwl b/sdk/cwl/tests/federation/cases/remote-case.cwl
new file mode 100644
index 000000000..02d1e953a
--- /dev/null
+++ b/sdk/cwl/tests/federation/cases/remote-case.cwl
@@ -0,0 +1,28 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+cwlVersion: v1.0
+class: Workflow
+$namespaces:
+ arv: "http://arvados.org/cwl#"
+requirements:
+ InlineJavascriptRequirement: {}
+ DockerRequirement:
+ dockerPull: arvados/fed-test:remote-case
+inputs:
+ inp:
+ type: File
+ inputBinding: {}
+ runOnCluster: string
+outputs:
+ hash:
+ type: File
+ outputSource: md5sum/hash
+steps:
+ md5sum:
+ in:
+ inp: inp
+ runOnCluster: runOnCluster
+ out: [hash]
+ run: md5sum.cwl
\ No newline at end of file
diff --git a/sdk/cwl/tests/federation/cases/twostep-both-remote.cwl b/sdk/cwl/tests/federation/cases/twostep-both-remote.cwl
new file mode 100644
index 000000000..c6cc811cb
--- /dev/null
+++ b/sdk/cwl/tests/federation/cases/twostep-both-remote.cwl
@@ -0,0 +1,35 @@
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+cwlVersion: v1.0
+class: Workflow
+$namespaces:
+ arv: "http://arvados.org/cwl#"
+requirements:
+ InlineJavascriptRequirement: {}
+ DockerRequirement:
+ dockerPull: arvados/fed-test:twostep-both-remote
+inputs:
+ inp:
+ type: File
+ inputBinding: {}
+ md5sumCluster: string
+ revCluster: string
+outputs:
+ hash:
+ type: File
+ outputSource: md5sum/hash
+steps:
+ md5sum:
+ in:
+ inp: inp
+ runOnCluster: md5sumCluster
+ out: [hash]
+ run: md5sum.cwl
+ rev:
+ in:
+ inp: md5sum/hash
+ runOnCluster: revCluster
+ out: [revhash]
+ run: rev.cwl
diff --git a/sdk/cwl/tests/federation/data/remote-case-input.txt b/sdk/cwl/tests/federation/data/remote-case-input.txt
new file mode 100644
index 000000000..21e87fb1d
--- /dev/null
+++ b/sdk/cwl/tests/federation/data/remote-case-input.txt
@@ -0,0 +1,16 @@
+Call me remote-case. Some years ago--never mind how long precisely--having
+little or no money in my purse, and nothing particular to interest me on
+shore, I thought I would sail about a little and see the watery part of
+the world. It is a way I have of driving off the spleen and regulating
+the circulation. Whenever I find myself growing grim about the mouth;
+whenever it is a damp, drizzly November in my soul; whenever I find
+myself involuntarily pausing before coffin warehouses, and bringing up
+the rear of every funeral I meet; and especially whenever my hypos get
+such an upper hand of me, that it requires a strong moral principle to
+prevent me from deliberately stepping into the street, and methodically
+knocking people's hats off--then, I account it high time to get to sea
+as soon as I can. This is my substitute for pistol and ball. With a
+philosophical flourish Cato throws himself upon his sword; I quietly
+take to the ship. There is nothing surprising in this. If they but knew
+it, almost all men in their degree, some time or other, cherish very
+nearly the same feelings towards the ocean with me.
diff --git a/sdk/cwl/tests/federation/data/twostep-both-remote.txt b/sdk/cwl/tests/federation/data/twostep-both-remote.txt
new file mode 100644
index 000000000..6218bb50e
--- /dev/null
+++ b/sdk/cwl/tests/federation/data/twostep-both-remote.txt
@@ -0,0 +1,16 @@
+Call me twostep-both-remote. Some years ago--never mind how long precisely--having
+little or no money in my purse, and nothing particular to interest me on
+shore, I thought I would sail about a little and see the watery part of
+the world. It is a way I have of driving off the spleen and regulating
+the circulation. Whenever I find myself growing grim about the mouth;
+whenever it is a damp, drizzly November in my soul; whenever I find
+myself involuntarily pausing before coffin warehouses, and bringing up
+the rear of every funeral I meet; and especially whenever my hypos get
+such an upper hand of me, that it requires a strong moral principle to
+prevent me from deliberately stepping into the street, and methodically
+knocking people's hats off--then, I account it high time to get to sea
+as soon as I can. This is my substitute for pistol and ball. With a
+philosophical flourish Cato throws himself upon his sword; I quietly
+take to the ship. There is nothing surprising in this. If they but knew
+it, almost all men in their degree, some time or other, cherish very
+nearly the same feelings towards the ocean with me.
diff --git a/sdk/cwl/tests/federation/main.cwl b/sdk/cwl/tests/federation/main.cwl
index 602cc2fcb..4b95d8c03 100755
--- a/sdk/cwl/tests/federation/main.cwl
+++ b/sdk/cwl/tests/federation/main.cwl
@@ -39,13 +39,16 @@ outputs:
runner-remote-step-home-success:
type: Any
outputSource: runner-remote-step-home/success
+ remote-case-success:
+ type: Any
+ outputSource: remote-case/success
twostep-home-to-remote-success:
type: Any
outputSource: twostep-home-to-remote/success
twostep-remote-to-home-success:
type: Any
outputSource: twostep-remote-to-home/success
- twostep-both-remote:
+ twostep-both-remote-success:
type: Any
outputSource: twostep-both-remote/success
@@ -151,6 +154,40 @@ steps:
out: [out, success]
run: framework/testcase.cwl
+ remote-case:
+ in:
+ arvados_api_token: arvados_api_token
+ arvado_api_host_insecure: arvado_api_host_insecure
+ arvados_api_hosts: arvados_api_hosts
+ arvados_cluster_ids: arvados_cluster_ids
+ acr: acr
+ wf:
+ default:
+ class: File
+ location: cases/remote-case.cwl
+ secondaryFiles:
+ - class: File
+ location: cases/md5sum.cwl
+ obj:
+ default:
+ inp:
+ class: File
+ location: data/remote-case-input.txt
+ valueFrom: |-
+ ${
+ self["runOnCluster"] = inputs.arvados_cluster_ids[1];
+ return self;
+ }
+ runner_cluster: { valueFrom: "$(inputs.arvados_cluster_ids[1])" }
+ scrub_image: {default: "arvados/fed-test:remote-case"}
+ scrub_collections:
+ default:
+ - 031a4ced0aa99de90fb630568afc6e9b+67 # input collection
+ - eb93a6718eb1a1a8ee9f66ee7d683472+51 # md5sum output collection
+ - f654d4048612135f4a5e7707ec0fcf3e+112 # final output json
+ out: [out, success]
+ run: framework/testcase.cwl
+
twostep-home-to-remote:
in:
arvados_api_token: arvados_api_token
@@ -264,5 +301,3 @@ steps:
- ddfa58a81953dad08436d571615dd584+112 # runner output json
out: [out, success]
run: framework/testcase.cwl
-
- # also: twostep-all-remote
commit e40d1363c5266bf80d4e77b9f3bc01ca3062fabe
Author: Peter Amstutz <pamstutz at veritasgenetics.com>
Date: Fri Nov 9 15:02:50 2018 -0500
14198: Add more twostep tests
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>
diff --git a/sdk/cwl/tests/federation/cases/twostep-home-to-remote.cwl b/sdk/cwl/tests/federation/cases/twostep-home-to-remote.cwl
index ffce6c3c4..6e8521b2b 100644
--- a/sdk/cwl/tests/federation/cases/twostep-home-to-remote.cwl
+++ b/sdk/cwl/tests/federation/cases/twostep-home-to-remote.cwl
@@ -33,6 +33,3 @@ steps:
runOnCluster: revCluster
out: [revhash]
run: rev.cwl
- requirements:
- arv:ClusterTarget:
- cluster_id: $(inputs.runOnCluster)
diff --git a/sdk/cwl/tests/federation/cases/twostep-remote-to-home.cwl b/sdk/cwl/tests/federation/cases/twostep-remote-to-home.cwl
index 4541ebba2..7804b2024 100644
--- a/sdk/cwl/tests/federation/cases/twostep-remote-to-home.cwl
+++ b/sdk/cwl/tests/federation/cases/twostep-remote-to-home.cwl
@@ -14,7 +14,8 @@ inputs:
inp:
type: File
inputBinding: {}
- runOnCluster: string
+ md5sumCluster: string
+ revCluster: string
outputs:
hash:
type: File
@@ -23,13 +24,12 @@ steps:
md5sum:
in:
inp: inp
+ runOnCluster: md5sumCluster
out: [hash]
run: md5sum.cwl
- requirements:
- arv:ClusterTarget:
- cluster_id: $(inputs.runOnCluster)
rev:
in:
inp: md5sum/hash
+ runOnCluster: revCluster
out: [revhash]
run: rev.cwl
diff --git a/sdk/cwl/tests/federation/framework/prepare.py b/sdk/cwl/tests/federation/framework/prepare.py
index 52c780a87..6fe90813e 100644
--- a/sdk/cwl/tests/federation/framework/prepare.py
+++ b/sdk/cwl/tests/federation/framework/prepare.py
@@ -36,7 +36,6 @@ for cluster_id in config["arvados_cluster_ids"]:
for cluster_id in config["arvados_cluster_ids"]:
matches = api.collections().list(filters=[["portable_data_hash", "in", list(scrub_collections)]],
select=["uuid", "portable_data_hash"], cluster_id=cluster_id).execute()
- print("matches from %s was %s" % (cluster_id, matches))
for m in matches["items"]:
api.collections().delete(uuid=m["uuid"]).execute()
print("Scrubbed %s (%s)" % (m["uuid"], m["portable_data_hash"]))
diff --git a/sdk/cwl/tests/federation/main.cwl b/sdk/cwl/tests/federation/main.cwl
index 346f97914..602cc2fcb 100755
--- a/sdk/cwl/tests/federation/main.cwl
+++ b/sdk/cwl/tests/federation/main.cwl
@@ -42,6 +42,12 @@ outputs:
twostep-home-to-remote-success:
type: Any
outputSource: twostep-home-to-remote/success
+ twostep-remote-to-home-success:
+ type: Any
+ outputSource: twostep-remote-to-home/success
+ twostep-both-remote:
+ type: Any
+ outputSource: twostep-both-remote/success
steps:
base-case:
@@ -183,4 +189,80 @@ steps:
out: [out, success]
run: framework/testcase.cwl
+ twostep-remote-to-home:
+ in:
+ arvados_api_token: arvados_api_token
+ arvado_api_host_insecure: arvado_api_host_insecure
+ arvados_api_hosts: arvados_api_hosts
+ arvados_cluster_ids: arvados_cluster_ids
+ acr: acr
+ wf:
+ default:
+ class: File
+ location: cases/twostep-remote-to-home.cwl
+ secondaryFiles:
+ - class: File
+ location: cases/md5sum.cwl
+ - class: File
+ location: cases/rev.cwl
+ obj:
+ default:
+ inp:
+ class: File
+ location: data/twostep-remote-to-home.txt
+ valueFrom: |-
+ ${
+ self["md5sumCluster"] = inputs.arvados_cluster_ids[1];
+ self["revCluster"] = inputs.arvados_cluster_ids[0];
+ return self;
+ }
+ runner_cluster: { valueFrom: "$(inputs.arvados_cluster_ids[0])" }
+ scrub_image: {default: "arvados/fed-test:twostep-remote-to-home"}
+ scrub_collections:
+ default:
+ - cce89b9f7b6e163978144051ce5f071a+74 # input collection
+ - 0c358c3af63644c6343766feff1b7238+51 # md5sum output collection
+ - 33fb7d512bf21f04847eca58cea46e74+51 # rev output collection
+ - 912e04aa3db04aba008cf5cd46c277b2+112 # runner output json
+ out: [out, success]
+ run: framework/testcase.cwl
+
+ twostep-both-remote:
+ in:
+ arvados_api_token: arvados_api_token
+ arvado_api_host_insecure: arvado_api_host_insecure
+ arvados_api_hosts: arvados_api_hosts
+ arvados_cluster_ids: arvados_cluster_ids
+ acr: acr
+ wf:
+ default:
+ class: File
+ location: cases/twostep-both-remote.cwl
+ secondaryFiles:
+ - class: File
+ location: cases/md5sum.cwl
+ - class: File
+ location: cases/rev.cwl
+ obj:
+ default:
+ inp:
+ class: File
+ location: data/twostep-both-remote.txt
+ valueFrom: |-
+ ${
+ self["md5sumCluster"] = inputs.arvados_cluster_ids[1];
+ self["revCluster"] = inputs.arvados_cluster_ids[1];
+ return self;
+ }
+ runner_cluster: { valueFrom: "$(inputs.arvados_cluster_ids[0])" }
+ scrub_image: {default: "arvados/fed-test:twostep-both-remote"}
+ scrub_collections:
+ default:
+ - 3c5e39939cf197d304ac1eac20841238+71 # input collection
+ - 3edb99aa607731593969cdab663d65b4+51 # md5sum output collection
+ - a91625b7139e60fe61a88cae42fbee13+51 # rev output collection
+ - ddfa58a81953dad08436d571615dd584+112 # runner output json
+ out: [out, success]
+ run: framework/testcase.cwl
+
# also: twostep-all-remote
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list