[ARVADOS] updated: 80f048da2da46a41257bab5f9797b497d9978f21
git at public.curoverse.com
git at public.curoverse.com
Tue Jun 17 13:53:33 EDT 2014
Summary of changes:
doc/api/methods/jobs.html.textile.liquid | 31 +++++++++++++++-------
.../app/controllers/arvados/v1/jobs_controller.rb | 2 +-
.../arvados/v1/job_reuse_controller_test.rb | 28 +++++++++++++++++++
3 files changed, 50 insertions(+), 11 deletions(-)
via 80f048da2da46a41257bab5f9797b497d9978f21 (commit)
via a590d742cd531150ffdab97e48af510a532688fc (commit)
from 5c45c63e6a4acbbe6ec70f4e8deb7796df642ef5 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 80f048da2da46a41257bab5f9797b497d9978f21
Author: Brett Smith <brett at curoverse.com>
Date: Tue Jun 17 13:52:52 2014 -0400
2879: Document new Job API filters.
diff --git a/doc/api/methods/jobs.html.textile.liquid b/doc/api/methods/jobs.html.textile.liquid
index d75977a..879feb4 100644
--- a/doc/api/methods/jobs.html.textile.liquid
+++ b/doc/api/methods/jobs.html.textile.liquid
@@ -34,28 +34,37 @@ table(table table-bordered table-condensed).
|minimum_script_version |string |Git branch, tag, or commit hash specifying the minimum acceptable script version (earliest ancestor) to consider when deciding whether to re-use a past job.[1]|query|@"c3e86c9"@|
|exclude_script_versions|array of strings|Git commit branches, tags, or hashes to exclude when deciding whether to re-use a past job.|query|@["8f03c71","8f03c71"]@
@["badtag1","badtag2"]@|
+|filters|array|Conditions to find Jobs to reuse.|query||
|find_or_create |boolean |Before creating, look for an existing job that has identical script, script_version, and script_parameters to those in the present job, has nondeterministic=false, and did not fail (it could be queued, running, or completed). If such a job exists, respond with the existing job instead of submitting a new one.|query|@false@|
When a job is submitted to the queue using the **create** method, the @script_version@ attribute is updated to a full 40-character Git commit hash based on the current content of the specified repository. If @script_version@ cannot be resolved, the job submission is rejected.
fn1. See the "note about specifying Git commits on the Job resource page":{{site.baseurl}}/api/schema/Job.html#script_version for more detail.
+h3. Specialized filters
+
+Special filter operations are available for specific Job columns.
+
+* @script_version@ @in git@ @REFSPEC@<br>Resolve @REFSPEC@ to a list of git commits, and match jobs with a @script_version@ in that list. The create method will find commits between @REFSPEC@ and the submitted job's @script_version@; the list method will search between @REFSPEC@ and HEAD. This list may include parallel branches if there is more than one path between @REFSPEC@ and the end commit in the graph. Use @not in@ or @not in git@ filters (below) to blacklist specific commits.
+
+* @script_version@ @not in git@ @REFSPEC@<br>Resolve @REFSPEC@ to a list of git commits, and match jobs with a @script_version@ not in that list.
+
+* @docker_image_locator@ @in docker@ @SEARCH@<br>@SEARCH@ can be a Docker image hash, a repository name, or a repository name and tag separated by a colon (@:@). The server will find collections that contain a Docker image that match that search criteria, then match jobs with a @docker_image_locator@ in that list.
+
+* @docker_image_locator@ @not in docker@ @SEARCH@<br>Negate the @in docker@ filter.
+
h3. Reusing jobs
-Because Arvados records the exact version of the script, input parameters, and runtime environment [1] that was used to run the job, if the script is deterministic (meaning that the same code version is guaranteed to produce the same outputs from the same inputs) then it is possible to re-use the results of past jobs, and avoid re-running the computation to save time. Arvados uses the following algorithm to determine if a past job can be re-used:
+Because Arvados records the exact version of the script, input parameters, and runtime environment that was used to run the job, if the script is deterministic (meaning that the same code version is guaranteed to produce the same outputs from the same inputs) then it is possible to re-use the results of past jobs, and avoid re-running the computation to save time. Arvados uses the following algorithm to determine if a past job can be re-used:
notextile. <div class="spaced-out">
# If @find_or_create@ is false or omitted, create a new job and skip the rest of these steps.
-# Find a list of acceptable values for @script_version at . If @minimum_script_version@ is specified, this is the set of all revisions in the Git commit graph between @minimum_script_version@ and the @script_version@ in the submitted "job object":{{site.baseurl}}/api/schema/Job.html (inclusive)[2]. If @minimum_script_version@ is not specified, only @script_version@ is added to the list. If @exclude_script_versions@ is specified, the listed versions are excluded from the list.
-# Select jobs whose @script@ and @script_parameters@ attributes match those in the submitted "job object":{{site.baseurl}}/api/schema/Job.html, and whose @script_version@ attribute is in the list of acceptable versions. Exclude jobs that failed or set @nondeterministic@ to true.
-# If more than one of the candidate jobs has finished, check that all such jobs actually did produce the same output.
-# If existing jobs exist and do not disagree with one another about the correct output, return one of the selected past jobs instead of creating a new job. If there is more than one match, which job will be returned is undefined.
-# If an existing job could not be chosen this way, create a new job.
-
-fn1. As of this writing, versioning the runtime environment is still under development.
-
-fn2. This may include parallel branches if there is more than one path between @minimum_script_version@ and the submitted job's @script_version@ in the Git commit graph. Use @exclude_script_versions@ to blacklist specific commits.
+# If @filters@ are specified, find jobs that match those filters. Filters *must* be specified to limit the @repository@ and @script@ attributes. An error is returned if they are missing.
+# If @filters@ are not specified, find jobs with the same @repository@ and @script@, with a @script_version@ between @minimum_script_version@ and @script_version@ (excluding @excluded_script_versions@), and a @docker_image_locator@ with the latest Collection that matches the submitted job's @docker_image@ constraint.
+# If the found jobs include a completed job, and all found completed jobs have consistent output, return one of them. Which specific job is returned is undefined.
+# If the found jobs only include incomplete jobs, return one of them. Which specific job is returned is undefined.
+# If no job has been returned so far, create and return a new job.
</div>
@@ -159,6 +168,8 @@ table(table table-bordered table-condensed).
|order|string|Order in which to return matching jobs.|query||
|filters|array|Conditions for filtering jobs.|query||
+See the create method documentation for more information about Job-specific filters.
+
h2. log_tail_follow
log_tail_follow jobs
commit a590d742cd531150ffdab97e48af510a532688fc
Author: Brett Smith <brett at curoverse.com>
Date: Tue Jun 17 13:51:18 2014 -0400
2879: Support tag search in "in docker" Job filters.
The search uses the same `reponame:tagname` format used by
`docker tag`.
diff --git a/services/api/app/controllers/arvados/v1/jobs_controller.rb b/services/api/app/controllers/arvados/v1/jobs_controller.rb
index 20b809b..74bea41 100644
--- a/services/api/app/controllers/arvados/v1/jobs_controller.rb
+++ b/services/api/app/controllers/arvados/v1/jobs_controller.rb
@@ -190,7 +190,7 @@ class Arvados::V1::JobsController < ApplicationController
false
when ["docker_image_locator", "in docker"], ["docker_image_locator", "not in docker"]
filter[1].sub!(/ docker$/, '')
- filter[2] = Collection.uuids_for_docker_image(filter[2])
+ filter[2] = Collection.uuids_for_docker_image(*filter[2].split(':', 2))
true
else
true
diff --git a/services/api/test/functional/arvados/v1/job_reuse_controller_test.rb b/services/api/test/functional/arvados/v1/job_reuse_controller_test.rb
index 450093c..5cdf6c7 100644
--- a/services/api/test/functional/arvados/v1/job_reuse_controller_test.rb
+++ b/services/api/test/functional/arvados/v1/job_reuse_controller_test.rb
@@ -413,6 +413,34 @@ class Arvados::V1::JobReuseControllerTest < ActionController::TestCase
end
end
+ test "reuse Job with Docker image repo+tag" do
+ filter_h = BASE_FILTERS.
+ merge("script_version" =>
+ ["=", "4fe459abe02d9b365932b8f5dc419439ab4e2577"],
+ "docker_image_locator" =>
+ ["in docker", links(:docker_image_collection_tag2).name])
+ post(:create, {
+ job: {
+ script: "hash",
+ script_version: "4fe459abe02d9b365932b8f5dc419439ab4e2577",
+ repository: "foo",
+ script_parameters: {
+ input: 'fa7aeb5140e2848d39b416daeef4ffc5+45',
+ an_integer: '1'
+ },
+ },
+ filters: filters_from_hash(filter_h),
+ find_or_create: true,
+ })
+ assert_response :success
+ new_job = assigns(:object)
+ assert_not_nil new_job
+ target_job = jobs(:previous_docker_job_run)
+ [:uuid, :script_version, :docker_image_locator].each do |attr|
+ assert_equal(target_job.send(attr), new_job.send(attr))
+ end
+ end
+
test "new job with unknown Docker image filter" do
filter_h = BASE_FILTERS.
merge("docker_image_locator" => ["in docker", "_nonesuchname_"])
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list