[ARVADOS] updated: 3610b230128153194c7e04cfab47d46a90489651

Git user git at public.curoverse.com
Mon Apr 25 17:06:31 EDT 2016


Summary of changes:
 build/package-build-dockerfiles/centos6/Dockerfile |  2 +-
 build/package-build-dockerfiles/debian7/Dockerfile |  2 +-
 build/package-build-dockerfiles/debian8/Dockerfile |  2 +-
 .../ubuntu1204/Dockerfile                          |  2 +-
 .../ubuntu1404/Dockerfile                          |  2 +-
 build/run-build-packages.sh                        |  9 ++-
 doc/install/install-keepstore.html.textile.liquid  |  2 +-
 sdk/cwl/setup.py                                   |  2 +-
 sdk/go/keepclient/perms.go                         | 14 ++--
 sdk/go/keepclient/perms_test.go                    | 29 ++++----
 sdk/go/manifest/manifest.go                        |  2 +-
 sdk/go/streamer/transfer.go                        |  4 +-
 services/api/app/models/blob.rb                    | 11 +--
 services/api/config/application.default.yml        | 17 +++--
 services/api/test/unit/blob_test.rb                | 21 +++++-
 services/crunch-run/crunchrun_test.go              |  2 +-
 services/datamanager/loggerutil/loggerutil.go      |  2 +-
 services/datamanager/summary/pull_list.go          |  2 +-
 services/keepproxy/keepproxy.go                    |  2 +-
 services/keepstore/keepstore.go                    |  2 +-
 services/keepstore/perms.go                        |  4 +-
 services/keepstore/perms_test.go                   |  7 +-
 services/keepstore/trash_worker_test.go            |  2 +-
 .../arvnodeman/computenode/dispatch/transitions.py |  6 +-
 .../tests/test_computenode_dispatch_slurm.py       | 25 +++----
 services/nodemanager/tests/test_daemon.py          |  8 +--
 services/nodemanager/tests/testutil.py             |  7 ++
 tools/arvbox/bin/arvbox                            |  3 +-
 tools/arvbox/lib/arvbox/docker/Dockerfile.base     |  2 +-
 .../{service/api/run-service => api-setup.sh}      | 18 +----
 tools/arvbox/lib/arvbox/docker/common.sh           |  1 +
 .../lib/arvbox/docker/service/api/run-service      | 78 +---------------------
 .../{workbench => websockets}/log/main/.gitstub    |  0
 .../service/{workbench => websockets}/log/run      |  0
 .../arvbox/docker/service/{sso => websockets}/run  |  0
 .../arvbox/docker/service/websockets/run-service   | 30 +++++++++
 tools/keep-block-check/keep-block-check.go         | 25 +++++--
 tools/keep-block-check/keep-block-check_test.go    | 19 +++---
 tools/keep-rsync/keep-rsync.go                     | 31 ++++++---
 tools/keep-rsync/keep-rsync_test.go                | 17 ++---
 40 files changed, 213 insertions(+), 201 deletions(-)
 copy tools/arvbox/lib/arvbox/docker/{service/api/run-service => api-setup.sh} (83%)
 copy tools/arvbox/lib/arvbox/docker/service/{workbench => websockets}/log/main/.gitstub (100%)
 copy tools/arvbox/lib/arvbox/docker/service/{workbench => websockets}/log/run (100%)
 copy tools/arvbox/lib/arvbox/docker/service/{sso => websockets}/run (100%)
 create mode 100755 tools/arvbox/lib/arvbox/docker/service/websockets/run-service

  discards  b2ad7e34cf7c99768a06db2ef9f4c6ce62a49253 (commit)
  discards  9f0c1ef69a99fd5f1497942e9fc73ee70bd91a91 (commit)
  discards  960e626794f8ea94759e5ef4d040df4f7eb68e4b (commit)
  discards  aa63face41bcc2028350249a3083eb2bc10896c1 (commit)
  discards  91ac0186b3f1c1bcebf5d9b4123fe6683ef7dc43 (commit)
  discards  3d43e0fbd074089f78568d65e332f75bfecdca81 (commit)
       via  3610b230128153194c7e04cfab47d46a90489651 (commit)
       via  fdb6acdccdfa1fb5410c9bb9a0709611185867be (commit)
       via  f7b2c6b75413d08da0546601552cecad1ebee6e8 (commit)
       via  3ee2d4442f3643f14b9324891d566698dbe3ae36 (commit)
       via  527686ab2232bf842567b8198b120e8add8cf7c0 (commit)
       via  fd71106f5be99eeda2c282611468a1262505bc78 (commit)
       via  d021d4a0549210dd513d4d28735ad81db85cafbf (commit)
       via  628f2f2e1bfabdb7221badab3d5189011aee5a54 (commit)
       via  08d8c1af187132692baf70a531f87c8082f72cdc (commit)
       via  3c88abd3cb33cbe80bb81a7cca779fe668036c9e (commit)
       via  469102b30008b4ee73c9f44b03a6bd82a2a91fb2 (commit)
       via  83332c0e546ad5e6332ce507224d9c6f64552ce6 (commit)
       via  2b152757ce494145c10971c36754c473d21dfcc7 (commit)
       via  0cacef6ec2b553052e3897fee3067951f65117cd (commit)
       via  899af276f73189b0581b65c6d77b19f5c041312a (commit)
       via  5784c9a32528a7795181a837660f7f21a7393cce (commit)
       via  4d9ece42c8482948e4eca9f587cdb36f62a00e2b (commit)
       via  aec99288fab5ceed8dc746e62efaeee33ff38a82 (commit)
       via  f2e0e32c7ab0f1de29e6e90a6af87c1969dfd640 (commit)
       via  b2cfb1a8873ee8c4128bd3bdd9fb32b9d9e8ee6e (commit)
       via  af754a64c48e30aec75bbcb1b741a768faacb94c (commit)
       via  6a202a2765bdbac5062fd34231804527edfb6a14 (commit)
       via  ecf4a87ac687b713d3eb4f191f4c5ead293fa046 (commit)

This update added new revisions after undoing existing revisions.  That is
to say, the old revision is not a strict subset of the new revision.  This
situation occurs when you --force push a change and generate a repository
containing something like this:

 * -- * -- B -- O -- O -- O (b2ad7e34cf7c99768a06db2ef9f4c6ce62a49253)
            \
             N -- N -- N (3610b230128153194c7e04cfab47d46a90489651)

When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.


commit 3610b230128153194c7e04cfab47d46a90489651
Author: Tom Clegg <tom at curoverse.com>
Date:   Mon Apr 25 16:47:50 2016 -0400

    8653: Translate pipeline parameter specs from CWL to Arvados.

diff --git a/sdk/cwl/arvados_cwl/__init__.py b/sdk/cwl/arvados_cwl/__init__.py
index d714a69..3f210f5 100644
--- a/sdk/cwl/arvados_cwl/__init__.py
+++ b/sdk/cwl/arvados_cwl/__init__.py
@@ -4,26 +4,27 @@
 
 import argparse
 import arvados
-import arvados.events
+import arvados.collection
 import arvados.commands.keepdocker
 import arvados.commands.run
-import arvados.collection
+import arvados.events
 import arvados.util
+import copy
+import cwltool.docker
 import cwltool.draft2tool
-import cwltool.workflow
+from cwltool.errors import WorkflowException
 import cwltool.main
 from cwltool.process import shortname
-from cwltool.errors import WorkflowException
-import threading
-import cwltool.docker
+import cwltool.workflow
 import fnmatch
-import logging
-import re
-import os
-import sys
 import functools
 import json
+import logging
+import os
 import pkg_resources  # part of setuptools
+import re
+import sys
+import threading
 
 from cwltool.process import get_feature, adjustFiles, scandeps
 from arvados.api import OrderedJsonModel
@@ -418,16 +419,81 @@ class RunnerJob(object):
 class RunnerTemplate(object):
     """An Arvados pipeline template that invokes a CWL workflow."""
 
+    type_to_dataclass = {
+        'boolean': 'boolean',
+        'File': 'File',
+        'float': 'number',
+        'int': 'number',
+        'string': 'text',
+    }
+
     def __init__(self, runner, tool, job_order, enable_reuse):
         self.runner = runner
+        self.tool = tool
         self.job = RunnerJob(
             runner=runner,
             tool=tool,
             job_order=job_order,
             enable_reuse=enable_reuse)
 
+    def pipeline_component_spec(self):
+        """Return a component that Workbench and a-r-p-i will understand.
+
+        Specifically, translate CWL input specs to Arvados pipeline
+        format, like {"dataclass":"File","value":"xyz"}.
+        """
+        spec = self.job.arvados_job_spec()
+        job_params = spec['script_parameters']
+        spec['script_parameters'] = {}
+
+        for param in self.tool.tool["inputs"]:
+            param = copy.deepcopy(param)
+
+            # Data type and "required" flag...
+            types = param['type']
+            if not isinstance(types, list):
+                types = [types]
+            param['required'] = "null" not in types
+            non_null_types = set(types) - set(["null"])
+            if len(non_null_types) == 1:
+                the_type = [c for c in non_null_types][0]
+                dataclass = self.type_to_dataclass.get(the_type)
+                if dataclass:
+                    param['dataclass'] = dataclass
+            # Note: If we didn't figure out a single appropriate
+            # dataclass, we just left that attribute out.  We leave
+            # the "type" attribute there in any case, which might help
+            # downstream.
+
+            # Title and description...
+            descr = param.get('description', '').rstrip('\n')
+            if descr:
+                if '\n' in descr:
+                    (param['title'],
+                     param['description']) = descr.split('\n', 1)
+                else:
+                    param['title'] = param['description']
+                    del param['description']
+
+            # Fill in the value from the current job order, if any.
+            param_id = param['id']
+            if '#' in param_id:
+                param_id = param_id.split('#', 1)[1]
+            value = job_params.get(param_id)
+            if value is None:
+                pass
+            elif not isinstance(value, dict):
+                param['value'] = value
+            elif param.get('dataclass') == 'File' and value.get('path'):
+                param['value'] = value['path']
+
+            del param['id']
+            spec['script_parameters'][param_id] = param
+        spec['script_parameters']['cwl:tool'] = job_params['cwl:tool']
+        return spec
+
     def save(self):
-        job_spec = self.job.arvados_job_spec()
+        job_spec = self.pipeline_component_spec()
         response = self.runner.api.pipeline_templates().create(body={
             "components": {
                 self.job.name: job_spec,
diff --git a/sdk/cwl/tests/test_submit.py b/sdk/cwl/tests/test_submit.py
index 3a5c38f..021394c 100644
--- a/sdk/cwl/tests/test_submit.py
+++ b/sdk/cwl/tests/test_submit.py
@@ -141,7 +141,13 @@ class TestCreateTemplate(unittest.TestCase):
         stubs.api.pipeline_instances().create.refute_called()
         stubs.api.jobs().create.refute_called()
 
-        expect_component = stubs.expect_job_spec
+        expect_component = copy.deepcopy(stubs.expect_job_spec)
+        expect_component['script_parameters']['x'] = {
+            'dataclass': 'File',
+            'required': True,
+            'type': 'File',
+            'value': '99999999999999999999999999999992+99/blorp.txt',
+        }
         expect_template = {
             "components": {
                 "submit_wf.cwl": expect_component,
@@ -150,11 +156,57 @@ class TestCreateTemplate(unittest.TestCase):
             "owner_uuid": project_uuid,
         }
         stubs.api.pipeline_templates().create.assert_called_with(
-            body=expect_template)
+            body=JsonDiffMatcher(expect_template))
 
         self.assertEqual(capture_stdout.getvalue(),
                          json.dumps(stubs.expect_pipeline_template_uuid))
 
+
+class TestTemplateInputs(unittest.TestCase):
+    expect_template = {
+        "components": {
+            "inputs_test.cwl": {
+                'runtime_constraints': {
+                    'docker_image': 'arvados/jobs',
+                },
+                'script_parameters': {
+                    'cwl:tool':
+                    '99999999999999999999999999999991+99/'
+                    'wf/inputs_test.cwl',
+                    'optionalFloatInput': None,
+                    'fileInput': {
+                        'type': 'File',
+                        'dataclass': 'File',
+                        'required': True,
+                        'title': "It's a file; we expect to find some characters in it.",
+                        'description': 'If there were anything further to say, it would be said here,\nor here.'
+                    },
+                    'floatInput': {
+                        'type': 'float',
+                        'dataclass': 'number',
+                        'required': True,
+                        'title': 'Floats like a duck',
+                    },
+                    'optionalFloatInput': {
+                        'type': ['null', 'float'],
+                        'dataclass': 'number',
+                        'required': False,
+                    },
+                    'boolInput': {
+                        'type': 'boolean',
+                        'dataclass': 'boolean',
+                        'required': True,
+                        'title': 'True or false?',
+                    },
+                },
+                'repository': 'arvados',
+                'script_version': 'master',
+                'script': 'cwl-runner',
+            },
+        },
+        "name": "inputs_test.cwl",
+    }
+
     @stubs
     def test_inputs_empty(self, stubs):
         exited = arvados_cwl.main(
@@ -163,42 +215,9 @@ class TestCreateTemplate(unittest.TestCase):
             cStringIO.StringIO(), sys.stderr, api_client=stubs.api)
         self.assertEqual(exited, 0)
 
-        expect_template = {
-            "owner_uuid": stubs.fake_user_uuid,
-            "components": {
-                "inputs_test.cwl": {
-                    'runtime_constraints': {
-                        'docker_image': 'arvados/jobs',
-                    },
-                    'script_parameters': {
-                        'cwl:tool':
-                        '99999999999999999999999999999991+99/'
-                        'wf/inputs_test.cwl',
-                        '#optionalFloatInput': None,
-                        '#fileInput': {
-                            'dataclass': 'File',
-                            'required': True,
-                        },
-                        '#floatInput': {
-                            'dataclass': 'number',
-                            'required': True,
-                        },
-                        '#optionalFloatInput': {
-                            'dataclass': 'number',
-                            'required': False,
-                        },
-                        '#boolInput': {
-                            'dataclass': 'Boolean',
-                            'required': True,
-                        },
-                    },
-                    'repository': 'arvados',
-                    'script_version': 'master',
-                    'script': 'cwl-runner',
-                },
-            },
-            "name": "inputs_test.cwl",
-        }
+        expect_template = copy.deepcopy(self.expect_template)
+        expect_template["owner_uuid"] = stubs.fake_user_uuid
+
         stubs.api.pipeline_templates().create.assert_called_with(
             body=JsonDiffMatcher(expect_template))
 
@@ -210,32 +229,15 @@ class TestCreateTemplate(unittest.TestCase):
             cStringIO.StringIO(), sys.stderr, api_client=stubs.api)
         self.assertEqual(exited, 0)
 
-        expect_template = {
-            "owner_uuid": stubs.fake_user_uuid,
-            "components": {
-                "inputs_test.cwl": {
-                    'runtime_constraints': {
-                        'docker_image': 'arvados/jobs',
-                    },
-                    'script_parameters': {
-                        'cwl:tool':
-                        '99999999999999999999999999999991+99/'
-                        'wf/inputs_test.cwl',
-                        '#optionalFloatInput': None,
-                        '#fileInput': {
-                            'path':
-                            '99999999999999999999999999999992+99/blorp.txt',
-                            'class': 'File',
-                        },
-                        '#floatInput': 1.234,
-                        '#boolInput': True,
-                    },
-                    'repository': 'arvados',
-                    'script_version': 'master',
-                    'script': 'cwl-runner',
-                },
-            },
-            "name": "inputs_test.cwl",
-        }
+        self.expect_template["owner_uuid"] = stubs.fake_user_uuid
+
+        expect_template = copy.deepcopy(self.expect_template)
+        expect_template["owner_uuid"] = stubs.fake_user_uuid
+        params = expect_template[
+            "components"]["inputs_test.cwl"]["script_parameters"]
+        params["fileInput"]["value"] = '99999999999999999999999999999992+99/blorp.txt'
+        params["floatInput"]["value"] = 1.234
+        params["boolInput"]["value"] = True
+
         stubs.api.pipeline_templates().create.assert_called_with(
             body=JsonDiffMatcher(expect_template))
diff --git a/sdk/cwl/tests/wf/inputs_test.cwl b/sdk/cwl/tests/wf/inputs_test.cwl
index 03d924c..61cb1c4 100644
--- a/sdk/cwl/tests/wf/inputs_test.cwl
+++ b/sdk/cwl/tests/wf/inputs_test.cwl
@@ -7,10 +7,14 @@ inputs:
     type: File
     description: |
       It's a file; we expect to find some characters in it.
+      If there were anything further to say, it would be said here,
+      or here.
   - id: "#boolInput"
     type: boolean
+    description: True or false?
   - id: "#floatInput"
     type: float
+    description: Floats like a duck
   - id: "#optionalFloatInput"
     type: ["null", float]
 outputs: []

commit fdb6acdccdfa1fb5410c9bb9a0709611185867be
Author: Tom Clegg <tom at curoverse.com>
Date:   Sun Apr 24 16:06:37 2016 -0400

    8653: Add test cases for input types.

diff --git a/sdk/cwl/tests/matcher.py b/sdk/cwl/tests/matcher.py
new file mode 100644
index 0000000..d3c9316
--- /dev/null
+++ b/sdk/cwl/tests/matcher.py
@@ -0,0 +1,23 @@
+import difflib
+import json
+
+
+class JsonDiffMatcher(object):
+    """Raise AssertionError with a readable JSON diff when not __eq__().
+
+    Used with assert_called_with() so it's possible for a human to see
+    the differences between expected and actual call arguments that
+    include non-trivial data structures.
+    """
+    def __init__(self, expected):
+        self.expected = expected
+
+    def __eq__(self, actual):
+        expected_json = json.dumps(self.expected, sort_keys=True, indent=2)
+        actual_json = json.dumps(actual, sort_keys=True, indent=2)
+        if expected_json != actual_json:
+            raise AssertionError("".join(difflib.context_diff(
+                expected_json.splitlines(1),
+                actual_json.splitlines(1),
+                fromfile="Expected", tofile="Actual")))
+        return True
diff --git a/sdk/cwl/tests/order/empty_order.json b/sdk/cwl/tests/order/empty_order.json
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/sdk/cwl/tests/order/empty_order.json
@@ -0,0 +1 @@
+{}
diff --git a/sdk/cwl/tests/order/inputs_test_order.json b/sdk/cwl/tests/order/inputs_test_order.json
new file mode 100644
index 0000000..8830523
--- /dev/null
+++ b/sdk/cwl/tests/order/inputs_test_order.json
@@ -0,0 +1,9 @@
+{
+    "fileInput": {
+        "class": "File",
+        "path": "../input/blorp.txt"
+    },
+    "boolInput": true,
+    "floatInput": 1.234,
+    "optionalFloatInput": null
+}
diff --git a/sdk/cwl/tests/test_submit.py b/sdk/cwl/tests/test_submit.py
index 750eeae..3a5c38f 100644
--- a/sdk/cwl/tests/test_submit.py
+++ b/sdk/cwl/tests/test_submit.py
@@ -11,6 +11,8 @@ import mock
 import sys
 import unittest
 
+from .matcher import JsonDiffMatcher
+
 
 def stubs(func):
     @functools.wraps(func)
@@ -124,7 +126,7 @@ class TestSubmit(unittest.TestCase):
 
 class TestCreateTemplate(unittest.TestCase):
     @stubs
-    def test_create_pipeline_template(self, stubs):
+    def test_create(self, stubs):
         project_uuid = 'zzzzz-j7d0g-zzzzzzzzzzzzzzz'
 
         capture_stdout = cStringIO.StringIO()
@@ -152,3 +154,88 @@ class TestCreateTemplate(unittest.TestCase):
 
         self.assertEqual(capture_stdout.getvalue(),
                          json.dumps(stubs.expect_pipeline_template_uuid))
+
+    @stubs
+    def test_inputs_empty(self, stubs):
+        exited = arvados_cwl.main(
+            ["--create-template", "--no-wait",
+             "tests/wf/inputs_test.cwl", "tests/order/empty_order.json"],
+            cStringIO.StringIO(), sys.stderr, api_client=stubs.api)
+        self.assertEqual(exited, 0)
+
+        expect_template = {
+            "owner_uuid": stubs.fake_user_uuid,
+            "components": {
+                "inputs_test.cwl": {
+                    'runtime_constraints': {
+                        'docker_image': 'arvados/jobs',
+                    },
+                    'script_parameters': {
+                        'cwl:tool':
+                        '99999999999999999999999999999991+99/'
+                        'wf/inputs_test.cwl',
+                        '#optionalFloatInput': None,
+                        '#fileInput': {
+                            'dataclass': 'File',
+                            'required': True,
+                        },
+                        '#floatInput': {
+                            'dataclass': 'number',
+                            'required': True,
+                        },
+                        '#optionalFloatInput': {
+                            'dataclass': 'number',
+                            'required': False,
+                        },
+                        '#boolInput': {
+                            'dataclass': 'Boolean',
+                            'required': True,
+                        },
+                    },
+                    'repository': 'arvados',
+                    'script_version': 'master',
+                    'script': 'cwl-runner',
+                },
+            },
+            "name": "inputs_test.cwl",
+        }
+        stubs.api.pipeline_templates().create.assert_called_with(
+            body=JsonDiffMatcher(expect_template))
+
+    @stubs
+    def test_inputs(self, stubs):
+        exited = arvados_cwl.main(
+            ["--create-template", "--no-wait",
+             "tests/wf/inputs_test.cwl", "tests/order/inputs_test_order.json"],
+            cStringIO.StringIO(), sys.stderr, api_client=stubs.api)
+        self.assertEqual(exited, 0)
+
+        expect_template = {
+            "owner_uuid": stubs.fake_user_uuid,
+            "components": {
+                "inputs_test.cwl": {
+                    'runtime_constraints': {
+                        'docker_image': 'arvados/jobs',
+                    },
+                    'script_parameters': {
+                        'cwl:tool':
+                        '99999999999999999999999999999991+99/'
+                        'wf/inputs_test.cwl',
+                        '#optionalFloatInput': None,
+                        '#fileInput': {
+                            'path':
+                            '99999999999999999999999999999992+99/blorp.txt',
+                            'class': 'File',
+                        },
+                        '#floatInput': 1.234,
+                        '#boolInput': True,
+                    },
+                    'repository': 'arvados',
+                    'script_version': 'master',
+                    'script': 'cwl-runner',
+                },
+            },
+            "name": "inputs_test.cwl",
+        }
+        stubs.api.pipeline_templates().create.assert_called_with(
+            body=JsonDiffMatcher(expect_template))
diff --git a/sdk/cwl/tests/wf/inputs_test.cwl b/sdk/cwl/tests/wf/inputs_test.cwl
new file mode 100644
index 0000000..03d924c
--- /dev/null
+++ b/sdk/cwl/tests/wf/inputs_test.cwl
@@ -0,0 +1,22 @@
+# Test case for arvados-cwl-runner. Used to test propagation of
+# various input types as script_parameters in pipeline templates.
+
+class: Workflow
+inputs:
+  - id: "#fileInput"
+    type: File
+    description: |
+      It's a file; we expect to find some characters in it.
+  - id: "#boolInput"
+    type: boolean
+  - id: "#floatInput"
+    type: float
+  - id: "#optionalFloatInput"
+    type: ["null", float]
+outputs: []
+steps:
+  - id: step1
+    inputs:
+      - { id: x, source: "#x" }
+    outputs: []
+    run: ../tool/submit_tool.cwl

commit f7b2c6b75413d08da0546601552cecad1ebee6e8
Author: Tom Clegg <tom at curoverse.com>
Date:   Thu Apr 21 15:08:01 2016 -0400

    8653: Add arvados-cwl-runner --create-template flag

diff --git a/sdk/cwl/arvados_cwl/__init__.py b/sdk/cwl/arvados_cwl/__init__.py
index 8f2102c..d714a69 100644
--- a/sdk/cwl/arvados_cwl/__init__.py
+++ b/sdk/cwl/arvados_cwl/__init__.py
@@ -322,7 +322,13 @@ class RunnerJob(object):
             for s in tool.steps:
                 self.upload_docker(s.embedded_tool)
 
-    def run(self, dry_run=False, pull_image=True, **kwargs):
+    def arvados_job_spec(self, dry_run=False, pull_image=True, **kwargs):
+        """Create an Arvados job specification for this workflow.
+
+        The returned dict can be used to create a job (i.e., passed as
+        the +body+ argument to jobs().create()), or as a component in
+        a pipeline template or pipeline instance.
+        """
         self.upload_docker(self.tool)
 
         workflowfiles = set()
@@ -364,9 +370,7 @@ class RunnerJob(object):
             del self.job_order["id"]
 
         self.job_order["cwl:tool"] = workflowmapper.mapper(self.tool.tool["id"])[1]
-
-        response = self.arvrunner.api.jobs().create(body={
-            "owner_uuid": self.arvrunner.project_uuid,
+        return {
             "script": "cwl-runner",
             "script_version": "master",
             "repository": "arvados",
@@ -374,7 +378,16 @@ class RunnerJob(object):
             "runtime_constraints": {
                 "docker_image": "arvados/jobs"
             }
-        }, find_or_create=self.enable_reuse).execute(num_retries=self.arvrunner.num_retries)
+        }
+
+    def run(self, *args, **kwargs):
+        job_spec = self.arvados_job_spec(*args, **kwargs)
+        job_spec.setdefault("owner_uuid", self.arvrunner.project_uuid)
+
+        response = self.arvrunner.api.jobs().create(
+            body=job_spec,
+            find_or_create=self.enable_reuse
+        ).execute(num_retries=self.arvrunner.num_retries)
 
         self.arvrunner.jobs[response["uuid"]] = self
 
@@ -401,6 +414,31 @@ class RunnerJob(object):
         finally:
             del self.arvrunner.jobs[record["uuid"]]
 
+
+class RunnerTemplate(object):
+    """An Arvados pipeline template that invokes a CWL workflow."""
+
+    def __init__(self, runner, tool, job_order, enable_reuse):
+        self.runner = runner
+        self.job = RunnerJob(
+            runner=runner,
+            tool=tool,
+            job_order=job_order,
+            enable_reuse=enable_reuse)
+
+    def save(self):
+        job_spec = self.job.arvados_job_spec()
+        response = self.runner.api.pipeline_templates().create(body={
+            "components": {
+                self.job.name: job_spec,
+            },
+            "name": self.job.name,
+            "owner_uuid": self.runner.project_uuid,
+        }).execute(num_retries=self.runner.num_retries)
+        self.uuid = response["uuid"]
+        logger.info("Created template %s", self.uuid)
+
+
 class ArvPathMapper(cwltool.pathmapper.PathMapper):
     """Convert container-local paths to and from Keep collection ids."""
 
@@ -502,7 +540,6 @@ class ArvCwlRunner(object):
                                                      body={"state": "Failed"}).execute(num_retries=self.num_retries)
         self.final_output = out
 
-
     def on_message(self, event):
         if "object_uuid" in event:
             if event["object_uuid"] in self.jobs and event["event_type"] == "update":
@@ -541,6 +578,13 @@ class ArvCwlRunner(object):
         self.project_uuid = args.project_uuid if args.project_uuid else useruuid
         self.pipeline = None
 
+        if args.create_template:
+            tmpl = RunnerTemplate(self, tool, job_order, args.enable_reuse)
+            tmpl.save()
+            # cwltool.main will print our return value (quoted as
+            # JSON, without a newline) on stdout.
+            return tmpl.uuid
+
         if args.submit:
             runnerjob = RunnerJob(self, tool, job_order, args.enable_reuse)
             if not args.wait:
@@ -655,6 +699,7 @@ def main(args, stdout, stderr, api_client=None):
                         default=True, dest="submit")
     exgroup.add_argument("--local", action="store_false", help="Run workflow on local host (submits jobs to Arvados).",
                         default=True, dest="submit")
+    exgroup.add_argument("--create-template", action="store_true", help="Create an Arvados pipeline template.")
 
     exgroup = parser.add_mutually_exclusive_group()
     exgroup.add_argument("--wait", action="store_true", help="After submitting workflow runner job, wait for completion.",
diff --git a/sdk/cwl/tests/test_submit.py b/sdk/cwl/tests/test_submit.py
index 5734821..750eeae 100644
--- a/sdk/cwl/tests/test_submit.py
+++ b/sdk/cwl/tests/test_submit.py
@@ -2,12 +2,16 @@ import arvados
 import arvados.keep
 import arvados.collection
 import arvados_cwl
+import copy
+import cStringIO
 import functools
+import json
 import hashlib
 import mock
 import sys
 import unittest
 
+
 def stubs(func):
     @functools.wraps(func)
     @mock.patch("arvados.commands.keepdocker.list_images_in_arv")
@@ -29,7 +33,9 @@ def stubs(func):
         stubs.fake_user_uuid = "zzzzz-tpzed-zzzzzzzzzzzzzzz"
 
         stubs.api = mock.MagicMock()
-        stubs.api.users().current().execute.return_value = {"uuid": stubs.fake_user_uuid}
+        stubs.api.users().current().execute.return_value = {
+            "uuid": stubs.fake_user_uuid,
+        }
         stubs.api.collections().list().execute.return_value = {"items": []}
         stubs.api.collections().create().execute.side_effect = ({
             "uuid": "zzzzz-4zz18-zzzzzzzzzzzzzz1",
@@ -42,8 +48,11 @@ def stubs(func):
             "uuid": "zzzzz-8i9sb-zzzzzzzzzzzzzzz",
             "state": "Queued",
         }
+        stubs.expect_pipeline_template_uuid = "zzzzz-d1hrv-zzzzzzzzzzzzzzz"
+        stubs.api.pipeline_templates().create().execute.return_value = {
+            "uuid": stubs.expect_pipeline_template_uuid,
+        }
         stubs.expect_job_spec = {
-            'owner_uuid': stubs.fake_user_uuid,
             'runtime_constraints': {
                 'docker_image': 'arvados/jobs'
             },
@@ -52,7 +61,8 @@ def stubs(func):
                     'path': '99999999999999999999999999999992+99/blorp.txt',
                     'class': 'File'
                 },
-                'cwl:tool': '99999999999999999999999999999991+99/wf/submit_wf.cwl'
+                'cwl:tool':
+                '99999999999999999999999999999991+99/wf/submit_wf.cwl'
             },
             'repository': 'arvados',
             'script_version': 'master',
@@ -66,27 +76,33 @@ class TestSubmit(unittest.TestCase):
     @stubs
     def test_submit(self, stubs):
         arvados_cwl.main(
-            ["--debug", "--submit", "--no-wait",
+            ["--submit", "--no-wait",
              "tests/wf/submit_wf.cwl", "tests/submit_test_job.json"],
             sys.stdout, sys.stderr, api_client=stubs.api)
 
         stubs.api.collections().create.assert_has_calls([
             mock.call(),
             mock.call(body={
-                'manifest_text': './tool 84ec4df683711de31b782505389a8843+429 0:16:blub.txt 16:413:submit_tool.cwl\n./wf 81d977a245a41b8e79859fbe00623fd0+344 0:344:submit_wf.cwl\n',
+                'manifest_text':
+                './tool 84ec4df683711de31b782505389a8843+429 '
+                '0:16:blub.txt 16:413:submit_tool.cwl\n./wf '
+                '81d977a245a41b8e79859fbe00623fd0+344 0:344:submit_wf.cwl\n',
                 'owner_uuid': 'zzzzz-tpzed-zzzzzzzzzzzzzzz',
                 'name': 'submit_wf.cwl',
             }, ensure_unique_name=True),
             mock.call().execute(),
             mock.call(body={
-                'manifest_text': '. 979af1245a12a1fed634d4222473bfdc+16 0:16:blorp.txt\n',
+                'manifest_text':
+                '. 979af1245a12a1fed634d4222473bfdc+16 0:16:blorp.txt\n',
                 'owner_uuid': 'zzzzz-tpzed-zzzzzzzzzzzzzzz',
                 'name': '#',
             }, ensure_unique_name=True),
             mock.call().execute()])
 
+        expect_job = copy.deepcopy(stubs.expect_job_spec)
+        expect_job["owner_uuid"] = stubs.fake_user_uuid
         stubs.api.jobs().create.assert_called_with(
-            body=stubs.expect_job_spec,
+            body=expect_job,
             find_or_create=True)
 
     @stubs
@@ -94,13 +110,45 @@ class TestSubmit(unittest.TestCase):
         project_uuid = 'zzzzz-j7d0g-zzzzzzzzzzzzzzz'
 
         arvados_cwl.main(
-            ["--debug", "--submit", "--no-wait",
+            ["--submit", "--no-wait",
              "--project-uuid", project_uuid,
              "tests/wf/submit_wf.cwl", "tests/submit_test_job.json"],
             sys.stdout, sys.stderr, api_client=stubs.api)
 
-        expect_body = stubs.expect_job_spec.copy()
+        expect_body = copy.deepcopy(stubs.expect_job_spec)
         expect_body["owner_uuid"] = project_uuid
         stubs.api.jobs().create.assert_called_with(
             body=expect_body,
             find_or_create=True)
+
+
+class TestCreateTemplate(unittest.TestCase):
+    @stubs
+    def test_create_pipeline_template(self, stubs):
+        project_uuid = 'zzzzz-j7d0g-zzzzzzzzzzzzzzz'
+
+        capture_stdout = cStringIO.StringIO()
+
+        exited = arvados_cwl.main(
+            ["--create-template", "--no-wait",
+             "--project-uuid", project_uuid,
+             "tests/wf/submit_wf.cwl", "tests/submit_test_job.json"],
+            capture_stdout, sys.stderr, api_client=stubs.api)
+        self.assertEqual(exited, 0)
+
+        stubs.api.pipeline_instances().create.refute_called()
+        stubs.api.jobs().create.refute_called()
+
+        expect_component = stubs.expect_job_spec
+        expect_template = {
+            "components": {
+                "submit_wf.cwl": expect_component,
+            },
+            "name": "submit_wf.cwl",
+            "owner_uuid": project_uuid,
+        }
+        stubs.api.pipeline_templates().create.assert_called_with(
+            body=expect_template)
+
+        self.assertEqual(capture_stdout.getvalue(),
+                         json.dumps(stubs.expect_pipeline_template_uuid))

commit 3ee2d4442f3643f14b9324891d566698dbe3ae36
Author: Tom Clegg <tom at curoverse.com>
Date:   Thu Apr 21 15:15:56 2016 -0400

    8653: DRY testing code.

diff --git a/sdk/cwl/tests/test_submit.py b/sdk/cwl/tests/test_submit.py
index 19745a0..5734821 100644
--- a/sdk/cwl/tests/test_submit.py
+++ b/sdk/cwl/tests/test_submit.py
@@ -1,103 +1,106 @@
-import unittest
-import mock
-import arvados_cwl
-import sys
 import arvados
 import arvados.keep
 import arvados.collection
+import arvados_cwl
+import functools
 import hashlib
+import mock
+import sys
+import unittest
 
-class TestSubmit(unittest.TestCase):
+def stubs(func):
+    @functools.wraps(func)
     @mock.patch("arvados.commands.keepdocker.list_images_in_arv")
     @mock.patch("arvados.collection.KeepClient")
     @mock.patch("arvados.events.subscribe")
-    def test_submit(self, events, keep, keepdocker):
-        api = mock.MagicMock()
+    def wrapped(self, events, KeepClient, keepdocker, *args, **kwargs):
+        class Stubs:
+            pass
+        stubs = Stubs()
+        stubs.events = events
+        stubs.KeepClient = KeepClient
+        stubs.keepdocker = keepdocker
+
         def putstub(p, **kwargs):
             return "%s+%i" % (hashlib.md5(p).hexdigest(), len(p))
-        keep().put.side_effect = putstub
-        keepdocker.return_value = True
-        user_uuid = "zzzzz-tpzed-zzzzzzzzzzzzzzz"
-        api.users().current().execute.return_value = {"uuid": user_uuid}
-        api.collections().list().execute.return_value = {"items": []}
-        api.collections().create().execute.side_effect = ({"uuid": "zzzzz-4zz18-zzzzzzzzzzzzzz1",
-                                                           "portable_data_hash": "99999999999999999999999999999991+99"},
-                                                          {"uuid": "zzzzz-4zz18-zzzzzzzzzzzzzz2",
-                                                           "portable_data_hash": "99999999999999999999999999999992+99"})
-        api.jobs().create().execute.return_value = {"uuid": "zzzzz-8i9sb-zzzzzzzzzzzzzzz", "state": "Queued"}
+        stubs.KeepClient().put.side_effect = putstub
+
+        stubs.keepdocker.return_value = True
+        stubs.fake_user_uuid = "zzzzz-tpzed-zzzzzzzzzzzzzzz"
+
+        stubs.api = mock.MagicMock()
+        stubs.api.users().current().execute.return_value = {"uuid": stubs.fake_user_uuid}
+        stubs.api.collections().list().execute.return_value = {"items": []}
+        stubs.api.collections().create().execute.side_effect = ({
+            "uuid": "zzzzz-4zz18-zzzzzzzzzzzzzz1",
+            "portable_data_hash": "99999999999999999999999999999991+99",
+        }, {
+            "uuid": "zzzzz-4zz18-zzzzzzzzzzzzzz2",
+            "portable_data_hash": "99999999999999999999999999999992+99",
+        })
+        stubs.api.jobs().create().execute.return_value = {
+            "uuid": "zzzzz-8i9sb-zzzzzzzzzzzzzzz",
+            "state": "Queued",
+        }
+        stubs.expect_job_spec = {
+            'owner_uuid': stubs.fake_user_uuid,
+            'runtime_constraints': {
+                'docker_image': 'arvados/jobs'
+            },
+            'script_parameters': {
+                'x': {
+                    'path': '99999999999999999999999999999992+99/blorp.txt',
+                    'class': 'File'
+                },
+                'cwl:tool': '99999999999999999999999999999991+99/wf/submit_wf.cwl'
+            },
+            'repository': 'arvados',
+            'script_version': 'master',
+            'script': 'cwl-runner'
+        }
+        return func(self, stubs, *args, **kwargs)
+    return wrapped
+
 
-        arvados_cwl.main(["--debug", "--submit", "--no-wait", "tests/wf/submit_wf.cwl", "tests/submit_test_job.json"],
-                         sys.stdout, sys.stderr, api_client=api)
+class TestSubmit(unittest.TestCase):
+    @stubs
+    def test_submit(self, stubs):
+        arvados_cwl.main(
+            ["--debug", "--submit", "--no-wait",
+             "tests/wf/submit_wf.cwl", "tests/submit_test_job.json"],
+            sys.stdout, sys.stderr, api_client=stubs.api)
 
-        api.collections().create.assert_has_calls([
+        stubs.api.collections().create.assert_has_calls([
             mock.call(),
-            mock.call(body={'manifest_text': './tool 84ec4df683711de31b782505389a8843+429 0:16:blub.txt 16:413:submit_tool.cwl\n./wf 81d977a245a41b8e79859fbe00623fd0+344 0:344:submit_wf.cwl\n',
-                            'owner_uuid': 'zzzzz-tpzed-zzzzzzzzzzzzzzz',
-                            'name': 'submit_wf.cwl'
-                        }, ensure_unique_name=True),
+            mock.call(body={
+                'manifest_text': './tool 84ec4df683711de31b782505389a8843+429 0:16:blub.txt 16:413:submit_tool.cwl\n./wf 81d977a245a41b8e79859fbe00623fd0+344 0:344:submit_wf.cwl\n',
+                'owner_uuid': 'zzzzz-tpzed-zzzzzzzzzzzzzzz',
+                'name': 'submit_wf.cwl',
+            }, ensure_unique_name=True),
             mock.call().execute(),
-            mock.call(body={'manifest_text': '. 979af1245a12a1fed634d4222473bfdc+16 0:16:blorp.txt\n',
-                            'owner_uuid': 'zzzzz-tpzed-zzzzzzzzzzzzzzz',
-                            'name': '#'
-                        }, ensure_unique_name=True),
+            mock.call(body={
+                'manifest_text': '. 979af1245a12a1fed634d4222473bfdc+16 0:16:blorp.txt\n',
+                'owner_uuid': 'zzzzz-tpzed-zzzzzzzzzzzzzzz',
+                'name': '#',
+            }, ensure_unique_name=True),
             mock.call().execute()])
 
-        api.jobs().create.assert_called_with(
-            body={
-                'owner_uuid': user_uuid,
-                'runtime_constraints': {
-                    'docker_image': 'arvados/jobs'
-                },
-                'script_parameters': {
-                    'x': {
-                        'path': '99999999999999999999999999999992+99/blorp.txt',
-                        'class': 'File'
-                    },
-                    'cwl:tool': '99999999999999999999999999999991+99/wf/submit_wf.cwl'
-                },
-                'repository': 'arvados',
-                'script_version': 'master',
-                'script': 'cwl-runner'
-            },
+        stubs.api.jobs().create.assert_called_with(
+            body=stubs.expect_job_spec,
             find_or_create=True)
 
-    @mock.patch("arvados.commands.keepdocker.list_images_in_arv")
-    @mock.patch("arvados.collection.KeepClient")
-    @mock.patch("arvados.events.subscribe")
-    def test_submit_with_project_uuid(self, events, keep, keepdocker):
-        api = mock.MagicMock()
-        def putstub(p, **kwargs):
-            return "%s+%i" % (hashlib.md5(p).hexdigest(), len(p))
-        keep().put.side_effect = putstub
-        keepdocker.return_value = True
-        api.users().current().execute.return_value = {"uuid": "zzzzz-tpzed-zzzzzzzzzzzzzzz"}
-        api.collections().list().execute.return_value = {"items": []}
-        api.collections().create().execute.side_effect = ({"uuid": "zzzzz-4zz18-zzzzzzzzzzzzzz1",
-                                                           "portable_data_hash": "99999999999999999999999999999991+99"},
-                                                          {"uuid": "zzzzz-4zz18-zzzzzzzzzzzzzz2",
-                                                           "portable_data_hash": "99999999999999999999999999999992+99"})
-        api.jobs().create().execute.return_value = {"uuid": "zzzzz-8i9sb-zzzzzzzzzzzzzzz", "state": "Queued"}
+    @stubs
+    def test_submit_with_project_uuid(self, stubs):
         project_uuid = 'zzzzz-j7d0g-zzzzzzzzzzzzzzz'
 
-        arvados_cwl.main(["--debug", "--submit", "--project-uuid", project_uuid,
-                          "--no-wait", "tests/wf/submit_wf.cwl", "tests/submit_test_job.json"],
-                         sys.stdout, sys.stderr, api_client=api)
+        arvados_cwl.main(
+            ["--debug", "--submit", "--no-wait",
+             "--project-uuid", project_uuid,
+             "tests/wf/submit_wf.cwl", "tests/submit_test_job.json"],
+            sys.stdout, sys.stderr, api_client=stubs.api)
 
-        api.jobs().create.assert_called_with(
-            body={
-                'owner_uuid': project_uuid,
-                'runtime_constraints': {
-                    'docker_image': 'arvados/jobs'
-                },
-                'script_parameters': {
-                    'x': {
-                        'path': '99999999999999999999999999999992+99/blorp.txt',
-                        'class': 'File'
-                    },
-                    'cwl:tool': '99999999999999999999999999999991+99/wf/submit_wf.cwl'
-                },
-                'repository': 'arvados',
-                'script_version': 'master',
-                'script': 'cwl-runner'
-            },
+        expect_body = stubs.expect_job_spec.copy()
+        expect_body["owner_uuid"] = project_uuid
+        stubs.api.jobs().create.assert_called_with(
+            body=expect_body,
             find_or_create=True)

commit 527686ab2232bf842567b8198b120e8add8cf7c0
Author: Tom Clegg <tom at curoverse.com>
Date:   Thu Apr 21 10:55:18 2016 -0400

    8653: Turn off debug messages / verbose logging in test suite.

diff --git a/sdk/cwl/tests/test_job.py b/sdk/cwl/tests/test_job.py
index e1ef605..e74e000 100644
--- a/sdk/cwl/tests/test_job.py
+++ b/sdk/cwl/tests/test_job.py
@@ -1,6 +1,13 @@
-import unittest
-import mock
 import arvados_cwl
+import logging
+import mock
+import unittest
+import os
+
+if not os.getenv('ARVADOS_DEBUG'):
+    logging.getLogger('arvados.cwl-runner').setLevel(logging.WARN)
+    logging.getLogger('arvados.arv-run').setLevel(logging.WARN)
+
 
 class TestJob(unittest.TestCase):
 

commit fd71106f5be99eeda2c282611468a1262505bc78
Author: Tom Clegg <tom at curoverse.com>
Date:   Thu Apr 21 10:54:11 2016 -0400

    8653: Fix whitespace.

diff --git a/sdk/cwl/tests/test_job.py b/sdk/cwl/tests/test_job.py
index bb15e60..e1ef605 100644
--- a/sdk/cwl/tests/test_job.py
+++ b/sdk/cwl/tests/test_job.py
@@ -28,9 +28,9 @@ class TestJob(unittest.TestCase):
                         'tasks': [{
                             'task.env': {'TMPDIR': '$(task.tmpdir)'},
                             'command': ['ls']
-                        }]
-            },
-            'script_version': 'master',
+                        }],
+                    },
+                    'script_version': 'master',
                     'minimum_script_version': '9e5b98e8f5f4727856b53447191f9c06e3da2ba6',
                     'repository': 'arvados',
                     'script': 'crunchrunner',
@@ -40,8 +40,8 @@ class TestJob(unittest.TestCase):
                         'min_ram_mb_per_node': 1024,
                         'min_scratch_mb_per_node': 2048 # tmpdirSize + outdirSize
                     }
-        },
-                                                    find_or_create=True,
+                },
+                find_or_create=True,
                 filters=[['repository', '=', 'arvados'],
                          ['script', '=', 'crunchrunner'],
                          ['script_version', 'in git', '9e5b98e8f5f4727856b53447191f9c06e3da2ba6'],
@@ -90,7 +90,7 @@ class TestJob(unittest.TestCase):
                     'min_ram_mb_per_node': 3000,
                     'min_scratch_mb_per_node': 5024 # tmpdirSize + outdirSize
                 }
-        },
+            },
             find_or_create=True,
             filters=[['repository', '=', 'arvados'],
                      ['script', '=', 'crunchrunner'],

-----------------------------------------------------------------------


hooks/post-receive
-- 




More information about the arvados-commits mailing list