[ARVADOS] updated: 1.3.0-147-g2666c3b72
Git user
git at public.curoverse.com
Fri Feb 1 10:40:38 EST 2019
Summary of changes:
.licenseignore | 1 +
README.md | 10 +-
apps/workbench/Gemfile.lock | 76 +--
apps/workbench/app/helpers/application_helper.rb | 4 +
.../app/views/users/_virtual_machines.html.erb | 4 +-
apps/workbench/package-build.version | 1 +
apps/workbench/test/the.patch | 3 +
build/run-build-packages.sh | 3 +
build/run-tests.sh | 8 +
cmd/arvados-server/cmd.go | 4 +-
...oller.service => crunch-dispatch-cloud.service} | 4 +-
doc/README.textile | 2 +-
doc/Rakefile | 2 +-
doc/_config.yml | 1 +
doc/admin/upgrading.html.textile.liquid | 8 +
doc/index.html.liquid | 2 +-
doc/install/install-controller.html.textile.liquid | 2 +-
.../getting_started/community.html.textile.liquid | 4 +-
.../ssh-access-unix.html.textile.liquid | 19 +-
lib/cloud/interfaces.go | 179 ++++++
lib/cmd/cmd.go | 5 +-
lib/dispatchcloud/cmd.go | 19 +
lib/dispatchcloud/container/queue.go | 378 ++++++++++++
lib/dispatchcloud/dispatcher.go | 197 ++++++
lib/dispatchcloud/dispatcher_test.go | 269 ++++++++
lib/dispatchcloud/driver.go | 22 +
lib/dispatchcloud/instance_set_proxy.go | 25 +
lib/dispatchcloud/logger.go | 29 +
lib/dispatchcloud/node_size.go | 137 ++---
lib/dispatchcloud/node_size_test.go | 22 +
lib/dispatchcloud/readme.go | 70 +++
lib/dispatchcloud/scheduler/fix_stale_locks.go | 57 ++
lib/dispatchcloud/{ => scheduler}/gocheck_test.go | 2 +-
lib/dispatchcloud/scheduler/interfaces.go | 43 ++
lib/dispatchcloud/scheduler/run_queue.go | 165 +++++
lib/dispatchcloud/scheduler/run_queue_test.go | 318 ++++++++++
lib/dispatchcloud/scheduler/scheduler.go | 116 ++++
lib/dispatchcloud/scheduler/sync.go | 97 +++
lib/dispatchcloud/ssh_executor/executor.go | 190 ++++++
lib/dispatchcloud/ssh_executor/executor_test.go | 102 +++
lib/dispatchcloud/test/doc.go | 7 +
lib/dispatchcloud/test/fixtures.go | 28 +
lib/dispatchcloud/test/lame_instance_set.go | 118 ++++
lib/dispatchcloud/test/queue.go | 171 ++++++
lib/dispatchcloud/test/ssh_service.go | 169 +++++
lib/dispatchcloud/test/sshkey_dispatch | 27 +
lib/dispatchcloud/test/sshkey_dispatch.pub | 1 +
lib/dispatchcloud/test/sshkey_vm | 27 +
lib/dispatchcloud/test/sshkey_vm.pub | 1 +
lib/dispatchcloud/test/stub_driver.go | 318 ++++++++++
lib/dispatchcloud/{ => worker}/gocheck_test.go | 2 +-
lib/dispatchcloud/worker/pool.go | 684 +++++++++++++++++++++
lib/dispatchcloud/worker/pool_test.go | 135 ++++
lib/dispatchcloud/worker/worker.go | 320 ++++++++++
sdk/cwl/arvados_cwl/arvcontainer.py | 12 +-
sdk/cwl/arvados_cwl/arvjob.py | 20 +-
sdk/cwl/arvados_cwl/arvtool.py | 14 +-
sdk/cwl/arvados_cwl/arvworkflow.py | 3 +
sdk/cwl/arvados_cwl/crunch_script.py | 1 +
sdk/cwl/arvados_cwl/executor.py | 48 +-
sdk/cwl/arvados_cwl/pathmapper.py | 40 ++
sdk/cwl/arvados_cwl/runner.py | 28 +-
sdk/cwl/setup.py | 4 +-
sdk/cwl/test_with_arvbox.sh | 2 +-
..._test_job.json => submit_test_job_missing.json} | 9 -
sdk/cwl/tests/test_pathmapper.py | 129 ++++
sdk/cwl/tests/test_submit.py | 17 +
sdk/go/arvados/config.go | 105 +++-
sdk/go/arvados/container.go | 3 +-
sdk/go/arvados/contextgroup.go | 95 +++
sdk/go/arvados/fs_collection.go | 306 ++++++---
sdk/go/arvados/fs_collection_test.go | 50 +-
sdk/go/arvados/throttle.go | 17 +
sdk/go/health/aggregator_test.go | 38 +-
sdk/python/arvados/arvfile.py | 38 +-
sdk/python/arvados/collection.py | 39 +-
sdk/python/arvados/commands/put.py | 88 ++-
sdk/python/arvados/commands/run.py | 4 +-
sdk/python/arvados/keep.py | 13 +
sdk/python/tests/test_arv_put.py | 168 ++++-
sdk/python/tests/test_arvfile.py | 55 +-
sdk/python/tests/test_collections.py | 63 +-
services/api/Gemfile.lock | 76 +--
.../app/controllers/arvados/v1/links_controller.rb | 4 +-
services/api/app/models/arvados_model.rb | 2 +
services/api/app/models/collection.rb | 4 +-
services/api/app/models/container.rb | 8 -
...20181213183234_add_expression_index_to_links.rb | 11 +
services/api/db/structure.sql | 16 +
services/api/lib/arvados_model_updates.rb | 17 +
services/api/lib/record_filters.rb | 31 +-
services/api/test/fixtures/links.yml | 14 +
.../functional/arvados/v1/links_controller_test.rb | 17 +
services/api/test/unit/container_test.rb | 9 +-
.../crunch-dispatch-slurm/crunch-dispatch-slurm.go | 11 +-
.../crunch-dispatch-slurm_test.go | 9 +-
.../crunch-dispatch-slurm/node_type.go | 69 +--
services/crunch-run/background.go | 237 +++++++
services/crunch-run/crunchrun.go | 39 +-
services/crunch-run/crunchrun_test.go | 124 ++--
.../lib/arvbox/docker/service/controller/run | 8 +-
tools/sync-groups/sync-groups.go | 8 +-
vendor/vendor.json | 6 +
103 files changed, 6138 insertions(+), 599 deletions(-)
create mode 100644 apps/workbench/package-build.version
create mode 100644 apps/workbench/test/the.patch
copy cmd/arvados-server/{arvados-controller.service => crunch-dispatch-cloud.service} (89%)
create mode 100644 lib/cloud/interfaces.go
create mode 100644 lib/dispatchcloud/cmd.go
create mode 100644 lib/dispatchcloud/container/queue.go
create mode 100644 lib/dispatchcloud/dispatcher.go
create mode 100644 lib/dispatchcloud/dispatcher_test.go
create mode 100644 lib/dispatchcloud/driver.go
create mode 100644 lib/dispatchcloud/instance_set_proxy.go
create mode 100644 lib/dispatchcloud/logger.go
create mode 100644 lib/dispatchcloud/readme.go
create mode 100644 lib/dispatchcloud/scheduler/fix_stale_locks.go
copy lib/dispatchcloud/{ => scheduler}/gocheck_test.go (90%)
create mode 100644 lib/dispatchcloud/scheduler/interfaces.go
create mode 100644 lib/dispatchcloud/scheduler/run_queue.go
create mode 100644 lib/dispatchcloud/scheduler/run_queue_test.go
create mode 100644 lib/dispatchcloud/scheduler/scheduler.go
create mode 100644 lib/dispatchcloud/scheduler/sync.go
create mode 100644 lib/dispatchcloud/ssh_executor/executor.go
create mode 100644 lib/dispatchcloud/ssh_executor/executor_test.go
create mode 100644 lib/dispatchcloud/test/doc.go
create mode 100644 lib/dispatchcloud/test/fixtures.go
create mode 100644 lib/dispatchcloud/test/lame_instance_set.go
create mode 100644 lib/dispatchcloud/test/queue.go
create mode 100644 lib/dispatchcloud/test/ssh_service.go
create mode 100644 lib/dispatchcloud/test/sshkey_dispatch
create mode 100644 lib/dispatchcloud/test/sshkey_dispatch.pub
create mode 100644 lib/dispatchcloud/test/sshkey_vm
create mode 100644 lib/dispatchcloud/test/sshkey_vm.pub
create mode 100644 lib/dispatchcloud/test/stub_driver.go
copy lib/dispatchcloud/{ => worker}/gocheck_test.go (90%)
create mode 100644 lib/dispatchcloud/worker/pool.go
create mode 100644 lib/dispatchcloud/worker/pool_test.go
create mode 100644 lib/dispatchcloud/worker/worker.go
copy sdk/cwl/tests/{submit_test_job.json => submit_test_job_missing.json} (56%)
create mode 100644 sdk/go/arvados/contextgroup.go
create mode 100644 sdk/go/arvados/throttle.go
create mode 100644 services/api/db/migrate/20181213183234_add_expression_index_to_links.rb
copy lib/dispatchcloud/node_size.go => services/crunch-dispatch-slurm/node_type.go (53%)
create mode 100644 services/crunch-run/background.go
via 2666c3b726d190511f072db9e6606a1a99936968 (commit)
via 1b2d73e101e1fb27de201b33142c081e80bb7fc6 (commit)
via 76e3ea1d0d2f865272000a5e12f64266a90b3794 (commit)
via 9c013e7742d72254ae747948f96237698198aa85 (commit)
via 55da806a494b13e88ee1b0c1e6cad211786628f3 (commit)
via fe33cfe407912df46ae34b5419ebdc84650829f6 (commit)
via 7a047d8b6fb8f5a1e0a0749d45a9e99b34a5779c (commit)
via 7252f5a926ebcd8fdf0048a4c622d7ee95cbd3b7 (commit)
via c6b8556dfb4f71c5ccc6041b48061fdb1c00c3b2 (commit)
via ee1161113168d709f0cb0513f192631c1f35b3ab (commit)
via 7b45cf3217fbee46cdd706893ec948bfedaae24c (commit)
via e787539455a4a0bf1e4f299460258905a56352d4 (commit)
via f8da36c66e4109730f7ad6f1a29f4e8cb921198c (commit)
via b3dbac2705f2bcffbf346d0b7efb85e2566b11f8 (commit)
via eabe449b9a17e14eb2ba67f6e5cb8ce1dafc8005 (commit)
via 61379d58674c14d31beab12113cd731a2dc24fd8 (commit)
via d353ff6f67923d472c1623493c7d11b35aebea90 (commit)
via cecff7f38bb5894efd1d8e449310b8654938bc15 (commit)
via d5508647a2dfd23684c08383fc18c77de3d1c825 (commit)
via a4da61b088214fe7a33b7b1198029700d71b35bd (commit)
via 2c9a3ec2625342927179e4bc807c2caad3b0f1c9 (commit)
via cfdb528be32c0f9a4e093b941a4630e32f7a5a45 (commit)
via e8f99cfef7cfbfcf1a1485d69250f24ced3fd609 (commit)
via a758d03b74f1c62f7ed442f4a4e571e5bba29ddd (commit)
via 772fc9527716fea221ae88ea3200db94f754afde (commit)
via cd68182c48ec70804a2acbf95331ec25629a9e28 (commit)
via 06bcc58df3e531f4af2bbc3ccf13aacdac068623 (commit)
via 882727d49c58c519bb65eae92b1eaf3e37f7199c (commit)
via aab8398af23a6a01713be7c219d78bec36ffe5ad (commit)
via 79379f5d68466922964d71f03d9bbe1b918f6252 (commit)
via 32fc16adf894bb6299f94054b69ff66f133a66e7 (commit)
via 0bebb7ca32bb96b75d5996137b8e385b60f6476a (commit)
via 746deaa52d2fda99e9265fe20a9f265d8e6816f0 (commit)
via 69e1bb1544b601e103b5fce06f3ec4cbed03d272 (commit)
via d7d074d790366338a01736552e916c5e4b5cef69 (commit)
via 29fee810dd128be904354d439f8c85a3318f64f8 (commit)
via cec011b7718536de42ebd683aa96bee92cbca06c (commit)
via 6804c679b365a237621bef37eb11c40ba87c4f10 (commit)
via cd4bb0b5bde62d7fe32cc105d6311dd318ebe304 (commit)
via 55c142e4f5887810dc4c73b3498dad9e25f25347 (commit)
via cbf93e8d897448dbd52369afe89fef2392140ff1 (commit)
via 0c21d2bd2d0fd351a4e546002a249b0b748061eb (commit)
via 9085ebc570deb7387e17e64776d9694ed99b913a (commit)
via 96165e8fcea44632979ddf7db02c1b70cb95d0f4 (commit)
via 81dd4a91b279b3229fb359df6c5dbf07571083ac (commit)
via 0a6b4ed36bd06b37412f020f0622e0c89e4e1653 (commit)
via 4e32f0b140ec0ec7f96c1f9eaae00950c176ff03 (commit)
via 2aa58f31ac8fc696361214a05ab9ba75a5140b08 (commit)
via 8ff3fd06e165a275f53884d1d20287b68c1b32bd (commit)
via cd7a746df5e9bf8a5770d06410b3fe1908282a7b (commit)
via d1571f495b0e0e05c833d4666924bcb6a288b33d (commit)
via 4a093ba4a1e14275a9500f2c65dd48528bc1e095 (commit)
via 0731e1f9019fa841ef496e5e6e308e41deb7585a (commit)
via f09d4a342cced4915b76632fb996936b93f1cfc1 (commit)
via 3f76b6309bc82cb2d62feb2eab10b55aa8b257ce (commit)
via 4d27cabda122ebda4c8688580af9088c8732a074 (commit)
via 076d8a0d4885f7bfa22962a745a5aacb48fa9be1 (commit)
via 7317196f547cf8c5e0ec87d1526322c305faba30 (commit)
via f1dfa6a0288d9a0380a73349db6083a85bda15d6 (commit)
via df3485f29656a5c4e3f1d7c09ccfdf3c6274d312 (commit)
via 7ec3ab46727a04fb374debbad9e08b1f719f0ac2 (commit)
via 81b8420a261a095a269a46d965b2fc0ee6ecf793 (commit)
via f62eb24060bb89c89faaa2998206961923c7b0b4 (commit)
via c031d4145d8ab1a11463acf5b20ef4df1afe00a4 (commit)
via 471ac065323e087f2f844d21d1c39929191eba71 (commit)
via b205525d0b7c7b9042513fe77d2e8061534208ae (commit)
via a63b5a0397f8c65ab4777e8e91daf64c9098871b (commit)
via e6d59d90cfbac0da3a61bb48bbca8ad85ebef129 (commit)
via 734de92e6f0254d21a1b9d29c634804c44e77ca6 (commit)
via 934d880aa5d10ed3382f9924a9a9f5694b41f266 (commit)
via 18f8ceb73765a71a71449414f4b4f87ad620028a (commit)
via 5d05b79a21ee85fa6ec53f9bc88e6ddb54029f04 (commit)
via 262871175bbe08f6ab1b7051bf9c5dfdd74283ac (commit)
via cbc55ee5ac64597cb0856497d3dedeccec79f440 (commit)
via a3dd7b47276e3d509eecfb930b06309d0315ac0d (commit)
via c746a03b6b08865796b5ef722f33d4b744a9c694 (commit)
via fc53ce7d5a7bd7974e8dac163238073e26e8b5d6 (commit)
via d48d4acf9ac040aab9d961506e91f862d67850b7 (commit)
via 452d54d5167d23496ec92e5f8d641bbbc94a123e (commit)
via f5008c1fc3535d21817601dfefebf7a38023c69e (commit)
via 665379525fdf1a4a2ec89a45dcc3203a1ecfcf22 (commit)
via bfbf6e4853edc684171f8385505463f17a1a5306 (commit)
via cc952178056bf6d29471f6986306fb673dcf394a (commit)
via 10bf8762c5d882ab5e9ff4c6e316878caebc6bf8 (commit)
via 36cba96fabf2062a0ca567573cc6e9bce6414bf2 (commit)
via 0b2772025dd5132beaebeac33d3be2e56890766e (commit)
via f6e063ed8afcdd868fe1eec2fa86ac8f0f61e18b (commit)
via c2a086701c35ae71c1fa876221ff9cc3220ed22e (commit)
via 5dbc1ae3d451f904654a2a61e5df620808ac175d (commit)
via 6fc7bd0626e93dd20fc58167300186e9f8820638 (commit)
via fdf6033451b12862a8305f4e3a4b06fbbdd498e1 (commit)
via 32ea7036d2cb438f5c313cff73cfe17610a3b8b9 (commit)
via 5a4cb7afb56eeeceba5ba7e710e4a7ccc5f9ae36 (commit)
via 533dbbd1d3780fc17c50b62f5b9adb1c5c005a92 (commit)
via 4ad50255921b571d8e7748b4c6c098b53d803183 (commit)
via 1e02903b90dbaf1f0e9fac222f65e3969b5f0352 (commit)
via ea36c4ba2652d8b6996e2c68535c22a7dc1a3807 (commit)
via 127e57b8ea812b9d03b67c0118230a475b727158 (commit)
via a88f7ad9728ee6968367928c6d3d7613bbf290ec (commit)
via a45b256a3261fff8f168321c02e61b94ae9b4a64 (commit)
via 50f8d8487ad5156058087438b670d7c6f8a8d718 (commit)
via f929cc124fc1fde8f1a6a12679e327b34aa88bde (commit)
via 478fb1838aea03ebad17b66926ef03aac536707b (commit)
via e36e81150649a6457c9cbf0101130cfdb776336f (commit)
via 78c18757e42c40178d7a9eaf78f7b6d167bee926 (commit)
via 9b3d58d6fa3d7c300e006af16ce8072bb68eca30 (commit)
via 70d02ffd2919ffb4148cdfd8cb8566db7a01345c (commit)
via 80905cf49077ecf797f6d0ae3c375ca1a7f7df20 (commit)
via d9d145af78c61d900447434735cab7f3dc64fbdd (commit)
via df5c912a9eb5af7222e5446bc437ee97262542c8 (commit)
via 0393a615c6c795e66e021188a1a3c00c04ca68e8 (commit)
via 3877bdae211807bd4c8cc5d824ffbede288dd708 (commit)
via 0f65ce49f183f7936d694961ce0f5980a8abffcc (commit)
via ff8c3cfde1e73c1c0adccb670040e98d82120e99 (commit)
via 15b7d1d4fa484641d846c1750487e640bfe4be09 (commit)
via 93671366e7633cbf0ca3cab68395e211e3afc31c (commit)
via d6bf3b66c2c938ca2ce2bda3276c0c0f4e54f1b6 (commit)
via 49f46db107c8226d70ed6d489cd000bfba82ca2f (commit)
via 3cb857be0aad14d0ebd631305a7a5c0f847811d4 (commit)
via 3654f222637b00728158300e700d615c4c913b29 (commit)
via 1c1e245bc3163a7841859d5ff75d0cff59dfc90c (commit)
via f33f8084159f701b8a1435e8cf77b7ab7f3417ae (commit)
via e5e2dc5e6bf1bce9893517470632295d42bb5f1c (commit)
via 359dc0dba7e93744c583b431d428de62fb8e0c90 (commit)
via 17479bd75a29c52470abe0049cb447e114eb39e9 (commit)
via 67464732a18e0aac11d89a75987151543b704cf6 (commit)
via a5ef124795af5b28a97c98e69a89c4c6e99cf0b7 (commit)
via a5fe34a8cd05c4e55deaec599347f65e6d662d22 (commit)
via 8ef9e211fa1eb91deb1b6aa7245cd7e5b8dd4c33 (commit)
via 468a33e9931c3dbadc83332a3b85d7677c9e4dc3 (commit)
via 0f159e5eb304e08a0704156046762709bd6e1eba (commit)
via 3ebb991101ef62387268d594e3e4307a0f5c4e9e (commit)
via 968519edf4c9150ac65991aea5f977356551b4e4 (commit)
via 71460cf96c9b43c8ab0a38118c3745a4c0e6d7e9 (commit)
via 438ce6f86261376880b186775d7ba5fbade560a6 (commit)
via c11e1e9b7e04b819a8a1794acc3ffb0a816715b7 (commit)
via b10284a496cd2d953b865144a5f28265fc277938 (commit)
via 8dff5fe14f23f3db9efaf5d71a21668738a9f164 (commit)
via c81a653c1e800c40a0c6e1a5d94cddd6620b5e52 (commit)
via 8162f74b2611b686cc8bcec225c6724c71b8926d (commit)
via 1b47c040b802c09d8652b952704e2ad09f31aa27 (commit)
via 8c44ed01463d5e0f0d167b37474b0e78fe11bd8d (commit)
via c1b959750a78509a2f5182dbd17685d010826b64 (commit)
via c3c538444c15e68e96780f157935f2baa4ba0bc5 (commit)
from 7995d73085a0fd459edafa679e85c08c9c2b5605 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 2666c3b726d190511f072db9e6606a1a99936968
Merge: 7995d7308 1b2d73e10
Author: Ward Vandewege <wvandewege at veritasgenetics.com>
Date: Fri Feb 1 10:39:16 2019 -0500
9945: Merge branch 'master' into 9945-make-python-package-dependency-free
Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege at veritasgenetics.com>
diff --cc doc/admin/upgrading.html.textile.liquid
index 74e035753,6a3e000ca..fc4ccac5f
--- a/doc/admin/upgrading.html.textile.liquid
+++ b/doc/admin/upgrading.html.textile.liquid
@@@ -30,32 -30,14 +30,40 @@@ Note to developers: Add new items at th
TODO: extract this information based on git commit messages and generate changelogs / release notes automatically.
{% endcomment %}
+h3. current master branch
+
+h4. Python packaging change
+
+As part of story "#9945":https://dev.arvados.org/issues/9945, the distribution packaging (deb/rpm) of our Python packages has changed. These packages now include a built-in virtualenv to reduce dependencies on system packages. We have also stopped packaging and publishing backports for all the Python dependencies of our packages, as they are no longer needed.
+
+One practical consequence of this change is that the use of the Arvados Python SDK (aka "import arvados") will require a tweak if the SDK was installed from a distribution package. It now requires the loading of the virtualenv environment from our packages. The "Install documentation for the Arvados Python SDK":/sdk/python/sdk-python.html reflects this change. This does not affect the use of the command line tools (e.g. arv-get, etc.).
+
+Python scripts that rely on the distribution Arvados Python SDK packages to import the Arvados SDK will need to be tweaked to load the correct Python environment.
+
+This can be done by activating the virtualenv outside of the script:
+
+<notextile>
+<pre>~$ <code class="userinput">source /usr/share/python2.7/dist/python-arvados-python-client/bin/activate</code>
+(python-arvados-python-client) ~$ <code class="userinput">path-to-the-python-script</code>
+</pre>
+</notextile>
+
+Or alternatively, by updating the shebang line at the start of the script to:
+
+<notextile>
+<pre>
+#!/usr/share/python2.7/dist/python-arvados-python-client/bin/python
+</pre>
+</notextile>
+
+ h3. v1.3.0 (2018-12-05)
+
+ This release includes several database migrations, which will be executed automatically as part of the API server upgrade. On large Arvados installations, these migrations will take a while. We've seen the upgrade take 30 minutes or more on installations with a lot of collections.
+
+ The @arvados-controller@ component now requires the /etc/arvados/config.yml file to be present. See <a href="{{ site.baseurl }}/install/install-controller.html#configuration">the @arvados-controller@ installation instructions</a>.
+
+ Support for the deprecated "jobs" API is broken in this release. Users who rely on it should not upgrade. This will be fixed in an upcoming 1.3.1 patch release, however users are "encouraged to migrate":upgrade-crunch2.html as support for the "jobs" API will be dropped in an upcoming release. Users who are already using the "containers" API are not affected.
+
h3. v1.2.1 (2018-11-26)
There are no special upgrade notes for this release.
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list