[ARVADOS] updated: 1.3.0-1510-g8b873a9b3
Git user
git at public.curoverse.com
Mon Aug 12 20:48:58 UTC 2019
Summary of changes:
.../pipeline_instances_controller_test.rb | 103 -
.../test/controllers/projects_controller_test.rb | 26 -
.../test/helpers/repository_stub_helper.rb | 1 -
apps/workbench/test/integration/jobs_test.rb | 94 -
.../test/integration/pipeline_instances_test.rb | 369 ---
.../test/integration/pipeline_templates_test.rb | 29 -
apps/workbench/test/integration/projects_test.rb | 1 -
.../test/integration/repositories_browse_test.rb | 23 -
apps/workbench/test/integration/websockets_test.rb | 61 -
apps/workbench/test/integration/work_units_test.rb | 30 -
apps/workbench/test/unit/disabled_api_test.rb | 8 +-
build/run-build-packages.sh | 2 -
cmd/arvados-client/cmd.go | 16 +-
crunch_scripts/GATK2-VariantFiltration | 64 -
crunch_scripts/GATK2-bqsr | 103 -
crunch_scripts/GATK2-merge-call | 242 --
crunch_scripts/GATK2-realign | 163 --
crunch_scripts/arvados-bcbio-nextgen.py | 145 --
crunch_scripts/arvados_bwa.py | 115 -
crunch_scripts/arvados_gatk2.py | 52 -
crunch_scripts/arvados_ipc.py | 51 -
crunch_scripts/arvados_picard.py | 42 -
crunch_scripts/arvados_samtools.py | 110 -
crunch_scripts/bwa-aln | 127 -
crunch_scripts/bwa-index | 41 -
crunch_scripts/collection-merge | 49 -
crunch_scripts/crunchrunner | 10 -
crunch_scripts/crunchutil/__init__.py | 0
crunch_scripts/crunchutil/robust_put.py | 56 -
crunch_scripts/crunchutil/subst.py | 102 -
crunch_scripts/crunchutil/vwd.py | 107 -
crunch_scripts/cwl-runner | 117 -
crunch_scripts/decompress-all.py | 64 -
crunch_scripts/file-select | 18 -
crunch_scripts/grep | 24 -
crunch_scripts/hash | 37 -
crunch_scripts/pgp-survey-import | 119 -
crunch_scripts/pgp-survey-parse | 22 -
crunch_scripts/picard-gatk2-prep | 211 --
crunch_scripts/pyrtg.py | 75 -
crunch_scripts/rtg-fasta2sdf | 27 -
crunch_scripts/rtg-fastq2sdf | 45 -
crunch_scripts/rtg-map | 41 -
crunch_scripts/rtg-snp | 34 -
crunch_scripts/run-command | 458 ----
crunch_scripts/split-fastq.py | 70 -
crunch_scripts/test/task_output_dir | 19 -
doc/_config.yml | 22 +-
doc/_includes/_arv_run_redirection.liquid | 27 -
doc/admin/upgrading.html.textile.liquid | 10 +-
doc/api/crunch-scripts.html.textile.liquid | 2 +
doc/api/execution.html.textile.liquid | 7 -
doc/api/methods/humans.html.textile.liquid | 2 +
doc/api/methods/job_tasks.html.textile.liquid | 2 +
doc/api/methods/jobs.html.textile.liquid | 4 +-
.../methods/pipeline_instances.html.textile.liquid | 2 +
.../methods/pipeline_templates.html.textile.liquid | 2 +
doc/api/methods/specimens.html.textile.liquid | 2 +
doc/api/methods/traits.html.textile.liquid | 2 +
.../install-compute-node.html.textile.liquid | 112 -
.../install-crunch-dispatch.html.textile.liquid | 207 --
doc/sdk/cli/subcommands.html.textile.liquid | 95 -
.../crunch-utility-libraries.html.textile.liquid | 228 --
doc/user/cwl/cwl-run-options.html.textile.liquid | 26 +-
doc/user/cwl/cwl-versions.html.textile.liquid | 14 -
.../examples/crunch-examples.html.textile.liquid | 102 -
.../reference/job-pipeline-ref.html.textile.liquid | 14 -
doc/user/topics/arv-run.html.textile.liquid | 163 --
.../crunch-tools-overview.html.textile.liquid | 70 -
doc/user/topics/run-command.html.textile.liquid | 319 ---
...nning-pipeline-command-line.html.textile.liquid | 58 -
.../topics/tutorial-parallel.html.textile.liquid | 85 -
.../tutorial-trait-search.html.textile.liquid | 278 ---
.../running-external-program.html.textile.liquid | 85 -
.../tutorial-firstscript.html.textile.liquid | 112 -
.../tutorial-submit-job.html.textile.liquid | 95 -
lib/cli/external.go | 6 +-
lib/config/config.default.yml | 30 +-
lib/config/export.go | 5 -
lib/config/generated_config.go | 30 +-
lib/controller/fed_collections.go | 10 +-
lib/controller/federation/conn.go | 8 +-
lib/controller/federation_test.go | 17 +-
sdk/cli/arvados-cli.gemspec | 6 +-
sdk/cli/bin/arv | 10 -
sdk/cli/bin/arv-crunch-job | 6 -
sdk/cli/bin/arv-run-pipeline-instance | 781 ------
sdk/cli/bin/crunch-job | 2577 --------------------
sdk/cli/test/test_arv-run-pipeline-instance.rb | 37 -
sdk/cli/test/test_crunch-job.rb | 141 --
sdk/cwl/arvados_cwl/__init__.py | 35 +-
sdk/cwl/arvados_cwl/arvjob.py | 495 ----
sdk/cwl/arvados_cwl/arvtool.py | 12 -
sdk/cwl/arvados_cwl/crunch_script.py | 159 --
sdk/cwl/arvados_cwl/executor.py | 76 +-
sdk/cwl/tests/test_container.py | 2 -
sdk/cwl/tests/test_job.py | 554 -----
sdk/cwl/tests/test_submit.py | 431 ----
sdk/cwl/tests/wf/runin-reqs-wf.cwl | 4 +-
sdk/cwl/tests/wf/runin-reqs-wf2.cwl | 4 +-
sdk/cwl/tests/wf/runin-reqs-wf3.cwl | 4 +-
sdk/cwl/tests/wf/runin-reqs-wf4.cwl | 4 +-
sdk/go/arvados/config.go | 9 +-
sdk/go/arvadosclient/arvadosclient_test.go | 50 -
sdk/go/crunchrunner/crunchrunner.go | 439 ----
sdk/go/crunchrunner/crunchrunner_test.go | 478 ----
sdk/go/crunchrunner/upload.go | 241 --
sdk/go/crunchrunner/upload_test.go | 152 --
sdk/python/arvados/commands/run.py | 185 +-
sdk/python/bin/arv-run | 7 -
sdk/python/setup.py | 1 -
sdk/python/tests/test_arv_run.py | 28 -
sdk/python/tests/test_pipeline_template.py | 62 -
sdk/python/tests/test_retry_job_helpers.py | 31 -
.../controllers/arvados/v1/job_tasks_controller.rb | 5 +
.../app/controllers/arvados/v1/jobs_controller.rb | 104 +-
.../arvados/v1/pipeline_instances_controller.rb | 10 +-
.../arvados/v1/pipeline_templates_controller.rb | 5 +
.../api/app/helpers/commit_ancestors_helper.rb | 6 -
services/api/app/helpers/commits_helper.rb | 263 ++
services/api/app/models/commit.rb | 272 ---
services/api/app/models/commit_ancestor.rb | 44 -
services/api/app/models/job.rb | 214 +-
services/api/app/models/job_task.rb | 10 +
services/api/app/models/pipeline_instance.rb | 35 +-
services/api/app/models/pipeline_template.rb | 11 +
services/api/config/arvados_config.rb | 5 -
....rb => 20190808145904_drop_commit_ancestors.rb} | 4 +-
...s.rb => 20190809135453_remove_commits_table.rb} | 4 +-
services/api/db/structure.sql | 115 +-
services/api/lib/can_be_an_owner.rb | 2 +
services/api/lib/crunch_dispatch.rb | 981 --------
services/api/lib/enable_jobs_api.rb | 12 +
services/api/script/crunch-dispatch.rb | 20 +-
services/api/script/crunch_failure_report.py | 222 --
services/api/script/fail-jobs.rb | 21 -
.../arvados/v1/job_reuse_controller_test.rb | 702 ------
.../functional/arvados/v1/jobs_controller_test.rb | 307 +--
.../v1/pipeline_instances_controller_test.rb | 44 -
services/api/test/helpers/git_test_helper.rb | 8 +-
.../api/test/integration/crunch_dispatch_test.rb | 47 -
services/api/test/integration/jobs_api_test.rb | 83 -
services/api/test/integration/pipeline_test.rb | 36 -
.../test/integration/serialized_encoding_test.rb | 21 -
services/api/test/unit/commit_test.rb | 70 +-
services/api/test/unit/crunch_dispatch_test.rb | 218 --
services/api/test/unit/fail_jobs_test.rb | 83 -
services/api/test/unit/job_task_test.rb | 11 -
services/api/test/unit/job_test.rb | 406 +--
services/api/test/unit/pipeline_instance_test.rb | 106 -
services/fuse/arvados_fuse/__init__.py | 14 +-
tools/arvbox/lib/arvbox/docker/Dockerfile.base | 2 +-
tools/arvbox/lib/arvbox/docker/crunch-setup.sh | 40 -
.../service/crunch-dispatch0/log/main/.gitstub | 0
.../arvbox/docker/service/crunch-dispatch0/log/run | 1 -
.../lib/arvbox/docker/service/crunch-dispatch0/run | 1 -
.../docker/service/crunch-dispatch0/run-service | 6 -
.../service/crunch-dispatch1/log/main/.gitstub | 0
.../arvbox/docker/service/crunch-dispatch1/log/run | 1 -
.../lib/arvbox/docker/service/crunch-dispatch1/run | 1 -
.../docker/service/crunch-dispatch1/run-service | 7 -
161 files changed, 560 insertions(+), 17017 deletions(-)
delete mode 100755 crunch_scripts/GATK2-VariantFiltration
delete mode 100755 crunch_scripts/GATK2-bqsr
delete mode 100755 crunch_scripts/GATK2-merge-call
delete mode 100755 crunch_scripts/GATK2-realign
delete mode 100755 crunch_scripts/arvados-bcbio-nextgen.py
delete mode 100644 crunch_scripts/arvados_bwa.py
delete mode 100644 crunch_scripts/arvados_gatk2.py
delete mode 100644 crunch_scripts/arvados_ipc.py
delete mode 100644 crunch_scripts/arvados_picard.py
delete mode 100644 crunch_scripts/arvados_samtools.py
delete mode 100755 crunch_scripts/bwa-aln
delete mode 100755 crunch_scripts/bwa-index
delete mode 100755 crunch_scripts/collection-merge
delete mode 100755 crunch_scripts/crunchrunner
delete mode 100644 crunch_scripts/crunchutil/__init__.py
delete mode 100644 crunch_scripts/crunchutil/robust_put.py
delete mode 100644 crunch_scripts/crunchutil/subst.py
delete mode 100644 crunch_scripts/crunchutil/vwd.py
delete mode 100755 crunch_scripts/cwl-runner
delete mode 100755 crunch_scripts/decompress-all.py
delete mode 100755 crunch_scripts/file-select
delete mode 100755 crunch_scripts/grep
delete mode 100755 crunch_scripts/hash
delete mode 100755 crunch_scripts/pgp-survey-import
delete mode 100755 crunch_scripts/pgp-survey-parse
delete mode 100755 crunch_scripts/picard-gatk2-prep
delete mode 100644 crunch_scripts/pyrtg.py
delete mode 100755 crunch_scripts/rtg-fasta2sdf
delete mode 100755 crunch_scripts/rtg-fastq2sdf
delete mode 100755 crunch_scripts/rtg-map
delete mode 100755 crunch_scripts/rtg-snp
delete mode 100755 crunch_scripts/run-command
delete mode 100755 crunch_scripts/split-fastq.py
delete mode 100755 crunch_scripts/test/task_output_dir
delete mode 100644 doc/_includes/_arv_run_redirection.liquid
delete mode 100644 doc/install/install-compute-node.html.textile.liquid
delete mode 100644 doc/install/install-crunch-dispatch.html.textile.liquid
delete mode 100644 doc/sdk/python/crunch-utility-libraries.html.textile.liquid
delete mode 100644 doc/user/examples/crunch-examples.html.textile.liquid
delete mode 100644 doc/user/reference/job-pipeline-ref.html.textile.liquid
delete mode 100644 doc/user/topics/arv-run.html.textile.liquid
delete mode 100644 doc/user/topics/crunch-tools-overview.html.textile.liquid
delete mode 100644 doc/user/topics/run-command.html.textile.liquid
delete mode 100644 doc/user/topics/running-pipeline-command-line.html.textile.liquid
delete mode 100644 doc/user/topics/tutorial-parallel.html.textile.liquid
delete mode 100644 doc/user/topics/tutorial-trait-search.html.textile.liquid
delete mode 100644 doc/user/tutorials/running-external-program.html.textile.liquid
delete mode 100644 doc/user/tutorials/tutorial-firstscript.html.textile.liquid
delete mode 100644 doc/user/tutorials/tutorial-submit-job.html.textile.liquid
delete mode 100755 sdk/cli/bin/arv-crunch-job
delete mode 100755 sdk/cli/bin/arv-run-pipeline-instance
delete mode 100755 sdk/cli/bin/crunch-job
delete mode 100644 sdk/cli/test/test_arv-run-pipeline-instance.rb
delete mode 100644 sdk/cli/test/test_crunch-job.rb
delete mode 100644 sdk/cwl/arvados_cwl/arvjob.py
delete mode 100644 sdk/cwl/arvados_cwl/crunch_script.py
delete mode 100644 sdk/cwl/tests/test_job.py
delete mode 100644 sdk/go/crunchrunner/crunchrunner.go
delete mode 100644 sdk/go/crunchrunner/crunchrunner_test.go
delete mode 100644 sdk/go/crunchrunner/upload.go
delete mode 100644 sdk/go/crunchrunner/upload_test.go
delete mode 100755 sdk/python/bin/arv-run
delete mode 100644 sdk/python/tests/test_arv_run.py
delete mode 100644 sdk/python/tests/test_pipeline_template.py
delete mode 100644 services/api/app/helpers/commit_ancestors_helper.rb
delete mode 100644 services/api/app/models/commit.rb
delete mode 100644 services/api/app/models/commit_ancestor.rb
copy services/api/db/migrate/{20130315183626_add_log_to_jobs.rb => 20190808145904_drop_commit_ancestors.rb} (57%)
copy services/api/db/migrate/{20130315183626_add_log_to_jobs.rb => 20190809135453_remove_commits_table.rb} (58%)
delete mode 100644 services/api/lib/crunch_dispatch.rb
delete mode 100755 services/api/script/crunch_failure_report.py
delete mode 100755 services/api/script/fail-jobs.rb
delete mode 100644 services/api/test/integration/crunch_dispatch_test.rb
delete mode 100644 services/api/test/unit/crunch_dispatch_test.rb
delete mode 100644 services/api/test/unit/fail_jobs_test.rb
delete mode 100755 tools/arvbox/lib/arvbox/docker/crunch-setup.sh
delete mode 100644 tools/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/main/.gitstub
delete mode 120000 tools/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/run
delete mode 120000 tools/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run
delete mode 100755 tools/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run-service
delete mode 100644 tools/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/main/.gitstub
delete mode 120000 tools/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/run
delete mode 120000 tools/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run
delete mode 100755 tools/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run-service
via 8b873a9b3b8865a4d451263e48b49122b9c32759 (commit)
via 415910149bed2eef6ae818b7c059d413934df06e (commit)
via 35658af99f09f2f6768583d65246429f789fc5a2 (commit)
via d7a5e6e37e9e9251177f9fad98d867f477e69537 (commit)
via 22de1507bc49617b02aa219302cecb830694adb5 (commit)
via c72420e52c085c76dead3e700e05e6fd058dccb2 (commit)
via 52cfc5c943a591d4f8095794820b72b697ae6f12 (commit)
via 043185f6d16dd41135c255f0f36fb8a75203537b (commit)
via 021f8eb819919d4606d7b4c72a2497e842041526 (commit)
via 751745e032831e16c5b67cd4bd0d328cd1f11fd7 (commit)
via 6a13a5f663102b77c157a157c8c4089269d3c3b1 (commit)
via 14626b1a574924a23f18073b8bb6eff829d66e5a (commit)
via fec7796668a4f4a73bb52e16e13e4504f76649f5 (commit)
via 2384aa9a6179333daa03002c36e7b25b83aab903 (commit)
via e10aa8e0c8b9c45d69832e71480cfb3d6929834e (commit)
via a8aa509cf3913c8dc91640ddcad6b25e6609517a (commit)
via b96728249db70975be33bb5c5721ec5b8a0eb3ff (commit)
via 448f667a574b50da096051a0d062b9059ab3609f (commit)
via 9f6bfd78b88486d4b3384b4be05c1f5e4f546a9c (commit)
via 30e065c34db0ab9a0e824a77b1ac0a46412598e0 (commit)
via 9f912f75b3a0b75e8f3de94f4527d60f075f17fe (commit)
via bc1947e4aef52fe5f3aebc10dc2ea74cad86672d (commit)
via 4d56f9b913fcf41fbf89bf5016463b5353fa3a9f (commit)
via c49024ec99cc5717f7856d61f325c01c90f750a9 (commit)
via 07d92519438a592d531f2c7558cd51788da262ca (commit)
via 9a02d4117bfc1372ce5b6baedc9303f2500b3197 (commit)
via bdb3e297e65ed39f66b33a6fb4c3b3b0e17e6fd5 (commit)
via 40d4eb5c5a942d82d158ab9d4d3b3fb1b44fde83 (commit)
via b93cb0b828989f9e2651ba436fb8a267224e4ca8 (commit)
from c8f57c52224362d7621f1271774b0f2d60c55cac (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 8b873a9b3b8865a4d451263e48b49122b9c32759
Merge: c8f57c522 415910149
Author: Lucas Di Pentima <ldipentima at veritasgenetics.com>
Date: Mon Aug 12 17:47:26 2019 -0300
Merge branch 'master' into 14716-webdav-cluster-config
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima at veritasgenetics.com>
diff --cc doc/admin/upgrading.html.textile.liquid
index de7eeaca0,dad4884a2..8c2ca7657
--- a/doc/admin/upgrading.html.textile.liquid
+++ b/doc/admin/upgrading.html.textile.liquid
@@@ -39,15 -39,17 +39,21 @@@ table(table table-bordered table-conden
|"v1.1.4":#v1_1_4|"v1.1.3":#v1_1_3|"v1.1.2":#v1_1_2|"v1.1.1":#v1_1_1|"v1.1.0":#v1_1_0|
|\5. "older":#older|
- h3(#master). development master (as of 2019-08-05)
-h3(#master). development master (as of 2019-08-09)
++h3(#master). development master (as of 2019-08-12)
+
+h4. Keep-web dropped support on command line flags configuration
+
+As we're migrating to a central cluster configuration file, the already deprecated way of getting configurations via environment variables and command line flags isn't valid anymore. Current keep-web supports both the now legacy @keep-web.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check "keep-web's install guide":{{site.baseurl}}/install/install-keep-web.html for more details.
+ h4. Jobs API is read-only
+
+ (task "#15133":https://dev.arvados.org/issues/15133 ) The legacy 'jobs' API is now read-only. It has long been superceded by containers / container_requests (aka crunch v2). Arvados installations since the end of 2017 (v1.1.0) have probably only used containers, and are unaffected by this change.
+
+ So that older Arvados sites don't lose access to legacy records, the API has been converted to read-only. Creating and updating jobs (and related types job_task, pipeline_template and pipeline_instance) is disabled and much of the business logic related has been removed, along with various other code specific to the jobs API. Specifically, the following programs associated with the jobs API have been removed: @crunch-dispatch.rb@, @crunch-job@, @crunchrunner@, @arv-run-pipeline-instance@, @arv-run at .
+
h4. No longer stripping ':' from strings in serialized database columns
- (bug #15311) Strings read from serialized columns in the database with a leading ':' would have the ':' stripped after loading the record. This behavior existed due to legacy serialization behavior which stored Ruby symbols with a leading ':'. Unfortunately this corrupted fields where the leading ":" was intentional. This behavior has been removed.
+ (bug "#15311":https://dev.arvados.org/issues/15311 ) Strings read from serialized columns in the database with a leading ':' would have the ':' stripped after loading the record. This behavior existed due to legacy serialization behavior which stored Ruby symbols with a leading ':'. Unfortunately this corrupted fields where the leading ":" was intentional. This behavior has been removed.
You can test if any records in your database are affected by going to the API server directory and running @bundle exec rake symbols:check at . This will report which records contain fields with a leading ':' that would previously have been stripped. If there are records to be updated, you can update the database using @bundle exec rake symbols:stringify at .
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list