[ARVADOS] updated: 1.3.0-2881-g33b27555b
Git user
git at public.arvados.org
Wed Aug 19 21:30:57 UTC 2020
Summary of changes:
.licenseignore | 4 +-
.../app/views/application/_show_sharing.html.erb | 2 +
... test-package-python3-arvados-python-client.sh} | 4 +-
...ge-rh-python36-python-arvados-python-client.sh} | 4 +-
build/run-build-packages-one-target.sh | 2 +
build/run-build-packages.sh | 10 +-
build/run-library.sh | 59 +-
cmd/arvados-client/cmd.go | 4 +-
doc/_config.yml | 9 +-
doc/admin/cloudtest.html.textile.liquid | 2 +-
doc/admin/config-migration.html.textile.liquid | 2 +-
doc/admin/link-accounts.html.textile.liquid | 48 ++
doc/admin/spot-instances.html.textile.liquid | 2 +-
doc/admin/upgrading.html.textile.liquid | 6 +-
doc/api/methods/users.html.textile.liquid | 18 +
...configure-s3-object-storage.html.textile.liquid | 9 +
.../install-compute-node.html.textile.liquid | 186 +++++
.../install-dispatch-cloud.html.textile.liquid | 106 ++-
.../install-compute-node.html.textile.liquid | 11 +-
.../install-dispatch.html.textile.liquid | 34 +-
.../install-prerequisites.html.textile.liquid | 2 +-
.../install-slurm.html.textile.liquid | 16 +-
.../crunch2-slurm/install-test.html.textile.liquid | 10 +-
doc/install/install-api-server.html.textile.liquid | 5 +-
.../install-arv-git-httpd.html.textile.liquid | 3 +-
doc/install/install-keep-web.html.textile.liquid | 3 +-
doc/install/install-keepproxy.html.textile.liquid | 4 +-
doc/install/install-keepstore.html.textile.liquid | 2 +
...nstall-manual-prerequisites.html.textile.liquid | 6 +-
.../install-nodemanager.html.textile.liquid | 12 +-
doc/install/install-webshell.html.textile.liquid | 183 +++++
.../install-workbench-app.html.textile.liquid | 3 +-
.../install-workbench2-app.html.textile.liquid | 3 +-
doc/install/install-ws.html.textile.liquid | 2 +-
doc/user/cwl/bwa-mem/bwa-mem-input-mixed.yml | 6 +-
doc/user/cwl/bwa-mem/bwa-mem-input-uuids.yml | 6 +-
doc/user/cwl/bwa-mem/bwa-mem.cwl | 6 +-
doc/user/cwl/cwl-run-options.html.textile.liquid | 22 +-
doc/user/cwl/cwl-runner.html.textile.liquid | 33 +-
doc/user/cwl/cwl-versions.html.textile.liquid | 8 +-
.../check-environment.html.textile.liquid | 10 +-
.../getting_started/workbench.html.textile.liquid | 2 +-
doc/user/index.html.textile.liquid | 2 +-
doc/user/topics/arv-copy.html.textile.liquid | 68 +-
doc/user/topics/arv-docker.html.textile.liquid | 2 +-
doc/user/topics/arv-web.html.textile.liquid | 106 ---
doc/user/topics/keep.html.textile.liquid | 2 +-
...rial-gatk-variantfiltration.html.textile.liquid | 173 ----
doc/user/topics/tutorial-job1.html.textile.liquid | 214 -----
...l-keep-collection-lifecycle.html.textile.liquid | 44 +-
.../tutorial-keep-get.html.textile.liquid | 64 +-
...torial-keep-mount-gnu-linux.html.textile.liquid | 4 +-
.../tutorial-keep-mount-os-x.html.textile.liquid | 4 +-
...tutorial-keep-mount-windows.html.textile.liquid | 2 +-
.../tutorials/tutorial-keep.html.textile.liquid | 61 +-
.../writing-cwl-workflow.html.textile.liquid | 34 +-
go.mod | 6 +-
go.sum | 24 +
lib/cloud/azure/azure.go | 8 +-
lib/config/config.default.yml | 20 +-
lib/config/deprecated_test.go | 1 +
lib/config/generated_config.go | 20 +-
lib/config/load_test.go | 1 +
lib/controller/api/routable.go | 17 +
lib/controller/federation.go | 4 +-
lib/controller/federation/federation_test.go | 2 +-
lib/controller/federation/generated.go | 3 +
lib/controller/federation/list.go | 5 +-
lib/controller/handler.go | 24 +-
lib/controller/integration_test.go | 1 +
lib/controller/localdb/conn.go | 4 +-
lib/controller/localdb/docker_test.go | 68 ++
lib/controller/localdb/login.go | 41 +-
lib/controller/localdb/login_ldap_docker_test.go | 10 +-
lib/controller/localdb/login_ldap_docker_test.sh | 5 +
lib/controller/localdb/login_ldap_test.go | 32 +-
lib/controller/localdb/login_oidc_test.go | 1 -
lib/controller/router/router.go | 109 +--
lib/controller/router/router_test.go | 8 +-
lib/controller/rpc/conn_test.go | 4 +
lib/ctrlctx/db.go | 122 +++
lib/ctrlctx/db_test.go | 87 ++
lib/deduplicationreport/command.go | 43 +
lib/deduplicationreport/report.go | 216 +++++
lib/deduplicationreport/report_test.go | 119 +++
lib/dispatchcloud/dispatcher_test.go | 4 +
lib/dispatchcloud/scheduler/run_queue_test.go | 4 +-
lib/dispatchcloud/scheduler/sync.go | 6 +-
lib/dispatchcloud/scheduler/sync_test.go | 62 ++
lib/dispatchcloud/worker/pool.go | 16 +-
lib/dispatchcloud/worker/pool_test.go | 57 +-
lib/dispatchcloud/worker/worker.go | 65 +-
lib/pam/.gitignore | 2 +
lib/pam/README | 18 +
lib/pam/docker_test.go | 173 ++++
.../arv-mount => lib/pam/fpm-info.sh | 2 +-
lib/pam/pam-configs-arvados | 19 +
lib/pam/pam_arvados.go | 185 +++++
lib/pam/pam_c.go | 24 +
lib/pam/testclient.go | 83 ++
lib/recovercollection/cmd_test.go | 1 +
lib/service/cmd_test.go | 1 +
sdk/R/install_deps.R | 6 +
sdk/cwl/arvados_cwl/__init__.py | 25 +-
...cwl-schema-v1.1.yml => arv-cwl-schema-v1.2.yml} | 0
sdk/cwl/arvados_cwl/arvcontainer.py | 28 +-
sdk/cwl/arvados_cwl/arvtool.py | 2 +-
sdk/cwl/arvados_cwl/arvworkflow.py | 5 +-
sdk/cwl/arvados_cwl/executor.py | 2 +-
sdk/cwl/arvados_cwl/fsaccess.py | 5 +
sdk/cwl/arvados_cwl/http.py | 5 +-
sdk/cwl/arvados_cwl/pathmapper.py | 1 +
sdk/cwl/arvados_cwl/runner.py | 18 +-
sdk/cwl/setup.py | 6 +-
sdk/cwl/tests/13976-keepref-wf.cwl | 17 +-
sdk/cwl/tests/16377-missing-default.cwl | 28 +
sdk/cwl/tests/arvados-tests.yml | 27 +-
sdk/cwl/tests/{octo.yml => hello.yml} | 4 +-
sdk/cwl/tests/test_http.py | 6 +-
sdk/cwl/tests/test_submit.py | 23 +-
sdk/cwl/tests/wf-defaults/default-dir4.cwl | 2 +-
.../{default-dir4.cwl => default-dir8.cwl} | 4 +-
sdk/cwl/tests/wf-defaults/wf4.cwl | 2 +-
sdk/cwl/tests/wf-defaults/{wf4.cwl => wf8.cwl} | 4 +-
sdk/go/arvados/api.go | 1 +
sdk/go/arvados/config.go | 3 +-
sdk/go/arvados/duration_test.go | 1 +
sdk/go/arvados/fs_project_test.go | 1 +
sdk/go/arvados/fs_site_test.go | 2 +
sdk/go/arvados/link.go | 17 +-
sdk/go/arvados/virtual_machine.go | 25 +
sdk/go/arvadosclient/arvadosclient_test.go | 5 +
sdk/go/arvadostest/db.go | 33 +
sdk/go/httpserver/logger_test.go | 4 +
sdk/go/keepclient/collectionreader_test.go | 10 +-
sdk/go/keepclient/keepclient_test.go | 23 +
sdk/python/arvados/commands/federation_migrate.py | 81 +-
sdk/python/tests/fed-migrate/check.py | 71 +-
sdk/python/tests/fed-migrate/create_users.py | 38 +-
services/api/app/models/arvados_model.rb | 25 +-
services/api/app/models/link.rb | 27 +
services/api/config/arvados_config.rb | 9 +-
services/api/test/unit/link_test.rb | 8 +
{sdk/python => services/dockercleaner}/fpm-info.sh | 5 +-
services/keep-balance/main_test.go | 1 +
services/keep/tools/traffic_test.py | 129 ---
services/keepstore/s3_volume.go | 4 +-
services/keepstore/s3aws_volume.go | 900 +++++++++++++++++++++
.../{s3_volume_test.go => s3aws_volume_test.go} | 254 ++++--
tools/arvbox/bin/arvbox | 2 +-
tools/compute-images/.gitignore | 3 +
.../compute-images/.licenseignore | 5 +-
.../compute-images/1078ECD7.asc | 0
tools/compute-images/arvados-images-aws.json | 80 ++
tools/compute-images/arvados-images-azure.json | 75 ++
tools/compute-images/build.sh | 277 +++++++
tools/compute-images/scripts/base.sh | 110 +++
...oud.cfg.d-07_compute_arvados_dispatch_cloud.cfg | 9 +
.../usr-local-bin-ensure-encrypted-partitions.sh | 159 ++++
tools/keep-exercise/keep-exercise.go | 303 +++++--
160 files changed, 4964 insertions(+), 1332 deletions(-)
copy build/package-testing/{test-package-python27-python-arvados-python-client.sh => test-package-python3-arvados-python-client.sh} (57%)
copy build/package-testing/{test-package-python27-python-arvados-python-client.sh => test-package-rh-python36-python-arvados-python-client.sh} (55%)
create mode 100644 doc/admin/link-accounts.html.textile.liquid
create mode 100644 doc/install/crunch2-cloud/install-compute-node.html.textile.liquid
rename doc/install/{ => crunch2-cloud}/install-dispatch-cloud.html.textile.liquid (75%)
create mode 100644 doc/install/install-webshell.html.textile.liquid
delete mode 100644 doc/user/topics/arv-web.html.textile.liquid
delete mode 100644 doc/user/topics/tutorial-gatk-variantfiltration.html.textile.liquid
delete mode 100644 doc/user/topics/tutorial-job1.html.textile.liquid
create mode 100644 lib/controller/api/routable.go
create mode 100644 lib/controller/localdb/docker_test.go
create mode 100644 lib/ctrlctx/db.go
create mode 100644 lib/ctrlctx/db_test.go
create mode 100644 lib/deduplicationreport/command.go
create mode 100644 lib/deduplicationreport/report.go
create mode 100644 lib/deduplicationreport/report_test.go
create mode 100644 lib/pam/.gitignore
create mode 100644 lib/pam/README
create mode 100644 lib/pam/docker_test.go
copy sdk/cli/test/binstub_arv-mount/arv-mount => lib/pam/fpm-info.sh (76%)
mode change 100755 => 100644
create mode 100644 lib/pam/pam-configs-arvados
create mode 100644 lib/pam/pam_arvados.go
create mode 100644 lib/pam/pam_c.go
create mode 100644 lib/pam/testclient.go
copy sdk/cwl/arvados_cwl/{arv-cwl-schema-v1.1.yml => arv-cwl-schema-v1.2.yml} (100%)
create mode 100644 sdk/cwl/tests/16377-missing-default.cwl
copy sdk/cwl/tests/{octo.yml => hello.yml} (61%)
copy sdk/cwl/tests/wf-defaults/{default-dir4.cwl => default-dir8.cwl} (87%)
copy sdk/cwl/tests/wf-defaults/{wf4.cwl => wf8.cwl} (86%)
create mode 100644 sdk/go/arvados/virtual_machine.go
create mode 100644 sdk/go/arvadostest/db.go
copy {sdk/python => services/dockercleaner}/fpm-info.sh (60%)
delete mode 100755 services/keep/tools/traffic_test.py
create mode 100644 services/keepstore/s3aws_volume.go
copy services/keepstore/{s3_volume_test.go => s3aws_volume_test.go} (70%)
create mode 100644 tools/compute-images/.gitignore
copy sdk/cli/test/binstub_docker_noop/docker.io => tools/compute-images/.licenseignore (83%)
mode change 100755 => 100644
copy docker/jobs/1078ECD7.key => tools/compute-images/1078ECD7.asc (100%)
create mode 100644 tools/compute-images/arvados-images-aws.json
create mode 100644 tools/compute-images/arvados-images-azure.json
create mode 100755 tools/compute-images/build.sh
create mode 100644 tools/compute-images/scripts/base.sh
create mode 100644 tools/compute-images/scripts/etc-cloud-cloud.cfg.d-07_compute_arvados_dispatch_cloud.cfg
create mode 100644 tools/compute-images/scripts/usr-local-bin-ensure-encrypted-partitions.sh
discards 649a637ca3546b775bef2d650ec60798f94b3d8e (commit)
discards d8dc80d74aeb66cb9e736a3cb9525f17c4b22a8e (commit)
discards 459ee5684c245b8cc960a4c0a0cc2825e85ec0e5 (commit)
discards 59af072670472944e6e377f67abca212da74bd32 (commit)
via 33b27555b25e70a0d37c5a8267450fddf3bd81b1 (commit)
via 409690acd4085f0c4ec764034f09c26a0b555f0c (commit)
via 7e47f5aaebcd7e4166e5e647f7b59af0914825bd (commit)
via dba9e46d852b50d2d72bd9619667f69637370fd7 (commit)
via 635ea4273130253cba3e00f4b0dca7d7734a741d (commit)
via 337b055122485e669cc601c91a78f8fdd2625b3e (commit)
via ddaa9596ebf94aa7b6ee720e06ceccd2d7dc43fc (commit)
via 2d046b9d2f27779f1e74e951fd7108dd88ba57c3 (commit)
via d4475f4c99f15161af3ececf3898d899bb3590c5 (commit)
via 530f7b503a5295ad7142b419735d3de7d83cda86 (commit)
via db507720c1863fca927625019d81d3a2d769f804 (commit)
via 970061ceccf362fe0d9236d7ff16a5535b0d01af (commit)
via 98b67ab5086ddd3c66d28548fa3b486b2d08e572 (commit)
via cf259a01bda2800cce674436661d3ca708f5d070 (commit)
via 084d8c9fedd6ee84d3f03f7f17b3a0f885b2d731 (commit)
via 615e280721c4be51556e2c31b4baffa69dd0d6d8 (commit)
via 7cc257b4e4f60cca1ca0354d9972cb36c59a77fe (commit)
via 9833a2b9be1a722bd9aad08baf48ca072bde7ca7 (commit)
via 04ab4d44425ff9cd5556c5978d85da3b6750e91c (commit)
via 4901a3c5f9442e8b884eb73b37da6272150c57a2 (commit)
via 78444f2fb480801787e486d4b65198d72ab4fe15 (commit)
via 2580743536835baedf91288dfb3599c0b11a3464 (commit)
via 877689fc26b0b69a94ca525a3bca1ed2236fb4b2 (commit)
via 70838209c214bfa57ef4bce289e1530a1cc2b081 (commit)
via 046863fce3eefdd8f2b4588855b2335dcb0215e1 (commit)
via 5fde4c5e8c464ec55d002735003a564a7802c720 (commit)
via de89bbf6824a39990e1605e4bd041b5d1ed464ea (commit)
via 2ba6cc7a5e4bfd05cd51e8ab22be2a99a883349d (commit)
via ab92008f810e7cdbea981ce903670621c54c082e (commit)
via 45c611003d84157370f7356b62ba8aa6972535a1 (commit)
via da283ea8958220d42431b43e05d40494365eee55 (commit)
via c6858cd333d073c972d6c050d2910f1c23f200d2 (commit)
via a44e0b163687a53bebf89472e4643e68cf1c567c (commit)
via 509084f45ffd17d740f7a26285210f8cde51f84c (commit)
via e1ed6272d0e8ecf96aeb6905328498d023d34e0e (commit)
via d186ff022ee427ded90307d795537c6429431f09 (commit)
via b2a1842d611a2739066a819badafaa2d2f14015d (commit)
via 3aa3fb78afa46e98c9be345045f4fea9fea0f08c (commit)
via a37291f5f992a082d88efb9cdf57cd92c710e883 (commit)
via 9c03bedaa6aa6a64b42dc61efcd6d46154fe6732 (commit)
via 4da1edec9c50c7adef8ab9fa500d5f3c40fb296f (commit)
via 4d3b8b299deaa4fff45102a26768e26129b17f10 (commit)
via 3757bbc7155d2a9c83d307925956fa2d524c92a9 (commit)
via 3e269516a7500d49ad90f8590d7dd9e81f135ad6 (commit)
via 5771cf273a4e09a5666122a7f67b4f088927e29d (commit)
via 63be17fdad6232fc9ff59738b6b358410953274c (commit)
via a79a27e215ce709455d1f5354b4ae5d045ed32dd (commit)
via ffeb31033857f36d26d06f8b7c2550a7950f941f (commit)
via 6036c55e1239281746152e85dfabbc9ed3cb6864 (commit)
via 414025952297852fff735664047ac14f6590522d (commit)
via daa50b3dc4488c705d2445fcadd2d35968787a68 (commit)
via 903d19948d3fd6c429ba99f68bb71fd989245f19 (commit)
via af106e9180ddc775de473c02a6269dbf3ae1b631 (commit)
via 7458c097e5ecdbb8836be941f03a5de4943f1eff (commit)
via aeb4a79c5f5eb2fd20603a16edffa0a1a0c9aef3 (commit)
via 43baa8f0cca640afc2934e6e65f51e5465e0ec88 (commit)
via 70c33e51905a84c3dcb61c88e5ec5c6479599c28 (commit)
via c9d7f6e5c734f58fa4b0574dfee40bbffbefdc8d (commit)
via dbd421c673da7199ceb3ed1d3398bd55d2707566 (commit)
via e5423707b3e05110b67de9c286b1fcfdcc65113f (commit)
via 2dd9827a1719f7deec9814e2aee89f4b17136925 (commit)
via 4d3ddc879fbc1dc39407c8b6e5b2f034d2d94e1e (commit)
via 8713dc0a0b40eb00228c229879183544e87be906 (commit)
via 715b5afbe744a85893eca159109658bb8cddc97f (commit)
via 6e5b24e817a0972ab30c9065cfc6a726821e7b66 (commit)
via 866decb73a9cf07b28e26c5028a1d42a5ef243a7 (commit)
via bfcd32beb41fcba2a5120ab899a080aef8305160 (commit)
via 7d7b9f9fe3dadcccffd76493f45a144d21c754ef (commit)
via e45d6feabba025220783beea6caed60cd1e7d782 (commit)
via 1bc7e16eb168274baea108092184925b7172ce02 (commit)
via 339da6b41fe014db93bd123cc9285cbeefe16936 (commit)
via 26a86a0239e5d626b1e02163aa417678c58e97ac (commit)
via 4d6383510afb08f90b631255e8da4db89770e1d9 (commit)
via cd4a811b896da640e5b8ddca7e515f19085932d4 (commit)
via 8a7644e8839181b8e13243c983c9c16b96591a95 (commit)
via 7407f41105f8000bb3908d41a31daaf3a30d9440 (commit)
via 29bd6c67e371a5ee6396e117956599eea572ac55 (commit)
via 81c2dc65f87ccc098695658d427270f7a3426f6d (commit)
via 7d9fc325e6d75d40e6f933a188d9a11fc016ed77 (commit)
via 0c2c8a174afde6a1a2472ee346eb5a3fe0bc6114 (commit)
via b7c46875f592b2f959f504e7f583979a594d528f (commit)
via 2b128f010976d8d4a55432e7b74349fd12362a93 (commit)
via 150a601dbcaf6b999e54bd17d5152f15b626c7f3 (commit)
via 5147b5515a59202940c55b5fb78eea270af78bb8 (commit)
via 163a40366283e336d8c8ea3fb9c0bd7322904865 (commit)
via bab30415501b92dbece93b636dda7f6914a123fa (commit)
via 5b8f2940dbb53ef170ab60f025bdbcc824d5d8c4 (commit)
via 67d3cbd984c8220d9bce0b7e36e49c9a55cf8bd2 (commit)
via 6563930ed602f47c1489b39b2eebfb54f9945e3c (commit)
via 28607876c98ae69dedb70e13a3ba5f12f9484608 (commit)
via c9e9f6ff4cadaf30657becdb70d0d828965397b0 (commit)
via 544d7aae0d58a25e8c761c638167c3564de06af5 (commit)
via aa7332d4cf8f44b4ce614293112417b32373cccf (commit)
via bfdc7fac1375e8fcb3d6d87c9a195b5606579c87 (commit)
via d7e3b180b3a2114fab340746791ecd913b69d795 (commit)
via 2e0648fb2b8a006664e6225826d78916f682eff5 (commit)
via 9a625dbcc8bca7f42a9043913006738a30a329d2 (commit)
via 34cb31d5191ce17c37ddd9d344f4d77125af64a8 (commit)
via 87c49a944230e400cb3529b30472e497d20912d6 (commit)
via 7be768ebfe665bcb30f4212b3f211c97b2fd65b9 (commit)
via 424910a551a2d564068693babb8058109203f4e5 (commit)
via 767f0799fb430b800920bad0ec83dfa210a51e01 (commit)
via ccefd86d022a00e89af37c6c7f9e71d4d68178ef (commit)
via d7b9e50e1ac6c3432f9e5c87bffd872b5f6fd572 (commit)
via 21d95b374fc204ae021948e70dd8cce264fda05c (commit)
via 44e035c4f9873171ae48025515102b513898a7c3 (commit)
via 5d5cc4f6b9272e3958721fc958453d649c919dcc (commit)
via 199d58a4df0dba1ee71c6816bc3a9d9d439cfd7e (commit)
via 9d977a7eae2fb2dd8556ae2391b21b22d28a2ff0 (commit)
via 63bcdd653f111d752ec11ccb3ab58ee5150478d5 (commit)
via a2b994f10fd73bdd882e691854239fc2d3b2e3a0 (commit)
via 4fa1cccccf752fd58dd2b10b1df4275a6de16ea2 (commit)
via 8d4010326400339c2e336caf85d2b827dd927972 (commit)
via 65b05a0f63dd137d680af4d98ae8d32dd2603158 (commit)
via d5b5f3d7c9a218c26fcb40c5e1d79136b2d75749 (commit)
via 9117766cd617a23ad5aa4ecd0aacfb2924db4b50 (commit)
via 183f8c6feeb8d3adbf36a1a4adf122607fb55617 (commit)
via 61d58bb6d4687c0794137700df1ba6aca418a191 (commit)
via e66f567ca467f2dfa576983c503deb98fd35028e (commit)
via 741b677dc5e85f60bc03ef130873e49ac0b75766 (commit)
via 85709a9266b0b70038c8c41a1e109670e1c47cd4 (commit)
via 0999dbe4a8bb91e316dd09ff25a00fcf20309ee4 (commit)
via 0dfee4fdb4671b398c63a7861bf9af1cd1b4794a (commit)
via c2ad835ffbc40e7e061526df7f9b9f6e0c1a83d3 (commit)
via dddfa30b07b2584353df378528f84945faa3ad7f (commit)
via 1dd02237ea4d135154743a1bf1d886949be2c5dd (commit)
via ff22ba71afe839832943099cc1fe273197c45ec7 (commit)
via 34316951a4e4f8439940a7eda5f1b044565f072b (commit)
via f95ac4c11b99daea342be0fba98e66c92d70e54c (commit)
via fe3fc9b1fab74cc494677ede67c8fa7444608364 (commit)
via f344802c8ec0d1e56151d69a25fa16da35788800 (commit)
via 8921a4561cf42d0d44a3676683a9be2b851aad2e (commit)
via 7a2b8cbedd6f97a044c10784a66e2e75f9010cb8 (commit)
via 24f90113a4e701f725229814ce5a36197306d059 (commit)
via 352b09c94caa1cf92972d28f7820276ed0ac84e4 (commit)
via c213ed2822ce1053e9c2f518a57c0126a912083e (commit)
via aeef1b8d7a84047ff93b04afc64ff78e511e21f8 (commit)
via 2c5417221843491727e4e5505012fc115e3bc7b0 (commit)
via ce3903121eb9645e99f6f6846de421b9af1bb23f (commit)
via 3a6559843024bda72c9a7b0baed3578d8698223d (commit)
via 876f66bbe9f30e48a45208a627dfd50e2a68ec22 (commit)
via b694b711c69fdfd46b6732a6d6c67e37951dab39 (commit)
via 7c9a94f70a895d5293ebdb5953518e417d173191 (commit)
via b578113d9c7f0d20d8d6f091b0dfc37e9b788ae8 (commit)
via a35ec27b40ce3ca0797cdcd8e0a79b2b8896af47 (commit)
via 9acfcfc07d459fb68cee5e75bf0cb128b87b44ec (commit)
via 4d1fa0d132e32d02cb87c1c6e3e4c2742bdec467 (commit)
This update added new revisions after undoing existing revisions. That is
to say, the old revision is not a strict subset of the new revision. This
situation occurs when you --force push a change and generate a repository
containing something like this:
* -- * -- B -- O -- O -- O (649a637ca3546b775bef2d650ec60798f94b3d8e)
\
N -- N -- N (33b27555b25e70a0d37c5a8267450fddf3bd81b1)
When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 33b27555b25e70a0d37c5a8267450fddf3bd81b1
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Wed Aug 19 17:29:19 2020 -0400
15964: Update and reorganize keep & CWL user guide docs
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/user/cwl/bwa-mem/bwa-mem.cwl b/doc/user/cwl/bwa-mem/bwa-mem.cwl
index a73329288..018867c83 100755
--- a/doc/user/cwl/bwa-mem/bwa-mem.cwl
+++ b/doc/user/cwl/bwa-mem/bwa-mem.cwl
@@ -10,11 +10,11 @@ hints:
DockerRequirement:
dockerPull: quay.io/biocontainers/bwa:0.7.17--ha92aebf_3
-baseCommand: [mem]
+baseCommand: [bwa, mem]
arguments:
- {prefix: "-t", valueFrom: $(runtime.cores)}
- - {prefix: "-R", valueFrom: "@RG\tID:$(inputs.group_id)\tPL:$(inputs.PL)\tSM:$(inputs.sample_id)"}
+ - {prefix: "-R", valueFrom: '@RG\\\tID:$(inputs.group_id)\\\tPL:$(inputs.PL)\\\tSM:$(inputs.sample_id)'}
inputs:
reference:
diff --git a/doc/user/getting_started/workbench.html.textile.liquid b/doc/user/getting_started/workbench.html.textile.liquid
index 52f1931c9..12b981148 100644
--- a/doc/user/getting_started/workbench.html.textile.liquid
+++ b/doc/user/getting_started/workbench.html.textile.liquid
@@ -23,6 +23,6 @@ If you are using the "playground" Arvados instance for this guide, you can Acces
You will be asked to log in. Arvados uses only your name and email address for identification, and will never access any personal information. If you are accessing Arvados for the first time, the Workbench may indicate your account status is *New / inactive*. If this is the case, contact the administrator of the Arvados instance to request activation of your account.
-Once your account is active, logging in to the Workbench will present you with the Dashboard. This gives a summary of your projects and recent activity in the Arvados instance. "You are now ready to run your first pipeline.":{{ site.baseurl }}/user/tutorials/tutorial-workflow-workbench.html
+Once your account is active, logging in to the Workbench will present you with the Dashboard. This gives a summary of your projects and recent activity in the Arvados instance. You are now ready to "upload data":{{ site.baseurl }}/user/tutorials/tutorial-keep.html or "run your first pipeline.":{{ site.baseurl }}/user/tutorials/tutorial-workflow-workbench.html
!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/workbench-dashboard.png!
diff --git a/doc/user/index.html.textile.liquid b/doc/user/index.html.textile.liquid
index 909394ef4..9749d1f28 100644
--- a/doc/user/index.html.textile.liquid
+++ b/doc/user/index.html.textile.liquid
@@ -9,7 +9,7 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-This guide provides a reference for using Arvados to solve scientific big data problems, including:
+Arvados is an open source platform for managing, processing, and sharing genomic and other large scientific and biomedical data. This guide provides a reference for using Arvados to solve scientific big data problems, including:
* Robust storage of very large files, such as whole genome sequences, using the "Arvados Keep":{{site.baseurl}}/user/tutorials/tutorial-keep.html content-addressable cluster file system.
* Running compute-intensive scientific analysis pipelines, such as genomic alignment and variant calls using the "Arvados Crunch":{{site.baseurl}}/user/tutorials/intro-crunch.html cluster compute engine.
diff --git a/doc/user/topics/arv-copy.html.textile.liquid b/doc/user/topics/arv-copy.html.textile.liquid
index e5e631d4e..d35df4fce 100644
--- a/doc/user/topics/arv-copy.html.textile.liquid
+++ b/doc/user/topics/arv-copy.html.textile.liquid
@@ -17,64 +17,66 @@ h2. arv-copy
@arv-copy@ allows users to copy collections and workflows from one cluster to another. By default, @arv-copy@ will recursively go through the workflow and copy all dependencies associated with the object.
-For example, let's copy from the <a href="https://playground.arvados.org/">Arvados playground</a>, also known as *pirca*, to *dstcl*. The names *pirca* and *dstcl* are interchangable with any cluster id. You can find the cluster name from the prefix of the uuid of the object you want to copy. For example, in *zzzzz*-4zz18-tci4vn4fa95w0zx, the cluster name is zzzzz.
+For example, let's copy from the <a href="https://playground.arvados.org/">Arvados playground</a>, also known as *pirca*, to *dstcl*. The names *pirca* and *dstcl* are interchangable with any cluster id. You can find the cluster name from the prefix of the uuid of the object you want to copy. For example, in *zzzzz*-4zz18-tci4vn4fa95w0zx, the cluster name is *zzzzz* .
-In order to communicate with both clusters, you must create custom configuration files for each cluster. In the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Current token*. Copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ in both of your clusters. Then, create two configuration files, one for each cluster. The names of the files must have the format of *ClusterID.conf*. In our example, let's make two files, one for *pirca* and one for *dstcl*. From your *Current token* page in *pirca* and *dstcl*, copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN at .
+In order to communicate with both clusters, you must create custom configuration files for each cluster. In the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Current token*. Copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ in both of your clusters. Then, create two configuration files in @~/.config/arvados@, one for each cluster. The names of the files must have the format of *ClusterID.conf*. Navigate to the *Current token* page on each of *pirca* and *dstcl* to get the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN at .
!{display: block;margin-left: 25px;margin-right: auto;}{{ site.baseurl }}/images/api-token-host.png!
-Copy your @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ into the config files as shown below in the shell account from which you are executing the commands. For example, the default shell you may have access to is shell.pirca. You can add these files in ~/.config/arvados/ in the pirca shell terminal.
+The config file consists of two lines, one for ARVADOS_API_HOST and one for ARVADOS_API_TOKEN:
-<notextile>
-<pre><code>~$ <span class="userinput">cd ~/.config/arvados</span>
-~$ <span class="userinput">echo "ARVADOS_API_HOST=pirca.arvadosapi.com" >> pirca.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_TOKEN=123456789abcdefghijkl" >> pirca.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_HOST=dstcl.arvadosapi.com" >> dstcl.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_TOKEN=987654321lkjihgfedcba" >> dstcl.conf</span>
-</code></pre>
-</notextile>
+<pre>
+ARVADOS_API_HOST=zzzzz.arvadosapi.com
+ARVADOS_API_TOKEN=v2/zzzzz-gj3su-xxxxxxxxxxxxxxx/123456789abcdefghijkl
+</pre>
+
+Copy your @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ into the config files as shown below in the shell account from which you are executing the commands. In our example, you need two files, @~/.config/arvados/pirca.conf@ and @~/.config/arvados/dstcl.conf at .
Now you're ready to copy between *pirca* and *dstcl*!
h3. How to copy a collection
-First, select the uuid of the collection you want to copy from the source cluster. The uuid can be found in the collection display page in the collection summary area (top left box), or from the URL bar (the part after @collections/...@)
+First, determine the uuid or portable data hash of the collection you want to copy from the source cluster. The uuid can be found in the collection display page in the collection summary area (top left box), or from the URL bar (the part after @collections/...@)
+
+Now copy the collection from *pirca* to *dstcl*. We will use the uuid @jutro-4zz18-tv416l321i4r01e@ as an example. You can find this collection on <a href="https://playground.arvados.org/collections/jutro-4zz18-tv416l321i4r01e">playground.arvados.org</a>.
+<notextile>
+<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl jutro-4zz18-tv416l321i4r01e</span>
+jutro-4zz18-tv416l321i4r01e: 6.1M / 6.1M 100.0%
+arvados.arv-copy[1234] INFO: Success: created copy with uuid dstcl-4zz18-xxxxxxxxxxxxxxx
+</code></pre>
+</notextile>
+
+You can also copy by content address:
-Now copy the collection from *pirca* to *dstcl*. We will use the uuid @pirca-4zz18-jyjuzyqq14qspl7@ as an example. You can find this collection in the <a href="https://playground.arvados.org/collections/pirca-4zz18-jyjuzyqq14qspl7">User guide resources project on playground.arvados.org</a>.
<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl pirca-4zz18-jyjuzyqq14qspl7</span>
-pirca-4zz18-tci4vn4fa95w0zx: 6.1M / 6.1M 100.0%
-arvados.arv-copy[1234] INFO: Success: created copy with uuid dstcl-4zz18-8765943210cdbae
+<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl 2463fa9efeb75e099685528b3b9071e0+438</span>
+2463fa9efeb75e099685528b3b9071e0+438: 6.1M / 6.1M 100.0%
+arvados.arv-copy[1234] INFO: Success: created copy with uuid dstcl-4zz18-xxxxxxxxxxxxxxx
</code></pre>
</notextile>
-The output of arv-copy displays the uuid of the collection generated in the destination cluster. By default, the output is placed in your home project in the destination cluster. If you want to place your collection in a pre-created project, you can specify the project you want it to be in using the tag @--project-uuid@ followed by the project uuid.
+The output of arv-copy displays the uuid of the collection generated in the destination cluster. By default, the output is placed in your home project in the destination cluster. If you want to place your collection in an existing project, you can specify the project you want it to be in using the tag @--project-uuid@ followed by the project uuid.
For example, this will copy the collection to project dstcl-j7d0g-a894213ukjhal12 in the destination cluster.
-<notextile> <pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl --project-uuid dstcl-j7d0g-a894213ukjhal12 pirca-4zz18-tci4vn4fa95w0zx</span>
+<notextile> <pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl --project-uuid dstcl-j7d0g-a894213ukjhal12 jutro-4zz18-tv416l321i4r01e
</code></pre>
</notextile>
h3. How to copy a workflow
-We will use the uuid @pirca-7fd4e-sampleworkflow1@ as an example workflow.
+We will use the uuid @jutro-7fd4e-mkmmq53m1ze6apx@ as an example workflow.
<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl --dst-git-repo $USER/tutorial pirca-7fd4e-sampleworkflow1</span>
-pirca-4zz18-jidprdejysravcr: 1143M / 1143M 100.0%
-2017-01-04 04:11:58 arvados.arv-copy[5906] INFO:
-2017-01-04 04:11:58 arvados.arv-copy[5906] INFO: Success: created copy with uuid dstcl-7fd4e-ojtgpne594ubkt7
+<pre><code>~$ <span class="userinput">arv-copy --src jutro --dst pirca --project-uuid pirca-j7d0g-ecak8knpefz8ere jutro-7fd4e-mkmmq53m1ze6apx</span>
+ae480c5099b81e17267b7445e35b4bc7+180: 23M / 23M 100.0%
+2463fa9efeb75e099685528b3b9071e0+438: 156M / 156M 100.0%
+jutro-4zz18-vvvqlops0a0kpdl: 94M / 94M 100.0%
+2020-08-19 17:04:13 arvados.arv-copy[4789] INFO:
+2020-08-19 17:04:13 arvados.arv-copy[4789] INFO: Success: created copy with uuid pirca-7fd4e-s0tw9rfbkpo2fmx
</code></pre>
</notextile>
-The name, description, and workflow definition from the original workflow will be used for the destination copy. In addition, any *locations* and *docker images* found in the src workflow definition will also be copied to the destination recursively.
+The name, description, and workflow definition from the original workflow will be used for the destination copy. In addition, any *collections* and *docker images* referenced in the source workflow definition will also be copied to the destination.
If you would like to copy the object without dependencies, you can use the @--no-recursive@ flag.
-
-For example, we can copy the same object non-recursively using the following:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl --dst-git-repo $USER/tutorial --no-recursive pirca-7fd4e-sampleworkflow1</span>
-</code></pre>
-</notextile>
diff --git a/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid b/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
index 180e08782..c4b9e31ec 100644
--- a/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
@@ -11,45 +11,42 @@ SPDX-License-Identifier: CC-BY-SA-3.0
During it's lifetime, a keep collection can be in various states. These states are *persisted*, *expiring*, *trashed* and *permanently deleted*.
+The nominal state is *persisted* which means the data can be can be accessed normally and will be retained indefinitely.
+
A collection is *expiring* when it has a *trash_at* time in the future. An expiring collection can be accessed as normal, but is scheduled to be trashed automatically at the *trash_at* time.
A collection is *trashed* when it has a *trash_at* time in the past. The *is_trashed* attribute will also be "true". The delete operation immediately puts the collection in the trash by setting the *trash_at* time to "now". Once trashed, the collection is no longer readable through normal data access APIs. The collection will have *delete_at* set to some time in the future. The trashed collection is recoverable until the delete_at time passes, at which point the collection is permanently deleted.
-# "*Collection lifecycle attributes*":#collection_attributes
# "*Deleting / trashing collections*":#delete-collection
# "*Recovering trashed collections*":#trash-recovery
+# "*Collection lifecycle attributes*":#collection_attributes
{% include 'tutorial_expectations' %}
-h2(#collection_attributes). Collection lifecycle attributes
-
-As listed above the attributes that are used to manage a collection lifecycle are it's *is_trashed*, *trash_at*, and *delete_at*. The table below lists the values of these attributes and how they influence the state of a collection and it's accessibility.
-
-table(table table-bordered table-condensed).
-|_. collection state|_. is_trashed|_. trash_at|_. delete_at|_. get|_. list|_. list?include_trash=true|_. can be modified|
-|persisted collection|false |null |null |yes |yes |yes |yes |
-|expiring collection|false |future |future |yes |yes |yes |yes |
-|trashed collection|true |past |future |no |no |yes |only is_trashed, trash_at and delete_at attribtues|
-|deleted collection|true|past |past |no |no |no |no |
-
h2(#delete-collection). Deleting / trashing collections
A collection can be deleted using either the arv command line tool or the workbench.
+h3. Trashing a collection using workbench
+
+To trash a collection using workbench, go to the Data collections tab in the project, and use the <i class="fa fa-fw fa-trash-o"></i> trash icon for this collection row.
+
h3. Trashing a collection using arv command line tool
<pre>
arv collection delete --uuid=zzzzz-4zz18-xxxxxxxxxxxxxxx
</pre>
-h3. Trashing a collection using workbench
-
-To trash a collection using workbench, go to the Data collections tab in the project, and use the trash icon for this collection row.
-
h2(#trash-recovery). Recovering trashed collections
A collection can be un-trashed / recovered using either the arv command line tool or the workbench.
+h3. Un-trashing a collection using workbench
+
+To untrash a collection using workbench, go to trash page on workbench by clicking on the "Trash" icon in the top navigation in workbench and use the recycle icon or selection dropdown option.
+
+!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/trash-button-topnav.png!
+
h3. Un-trashing a collection using arv command line tool
You can list the trashed collections using the list command.
@@ -64,8 +61,13 @@ You can then untrash a particular collection using arv using it's uuid.
arv collection untrash --uuid=zzzzz-4zz18-xxxxxxxxxxxxxxx
</pre>
-h3. Un-trashing a collection using workbench
+h2(#collection_attributes). Collection lifecycle attributes
-To untrash a collection using workbench, go to trash page on workbench by clicking on the "Trash" icon in the top navigation in workbench and use the recycle icon or selection dropdown option.
+As listed above the attributes that are used to manage a collection lifecycle are it's *is_trashed*, *trash_at*, and *delete_at*. The table below lists the values of these attributes and how they influence the state of a collection and it's accessibility.
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/trash-button-topnav.png!
+table(table table-bordered table-condensed).
+|_. collection state|_. is_trashed|_. trash_at|_. delete_at|_. get|_. list|_. list?include_trash=true|_. can be modified|
+|persisted collection|false |null |null |yes |yes |yes |yes |
+|expiring collection|false |future |future |yes |yes |yes |yes |
+|trashed collection|true |past |future |no |no |yes |only is_trashed, trash_at and delete_at attribtues|
+|deleted collection|true|past |past |no |no |no |no |
diff --git a/doc/user/tutorials/tutorial-keep-get.html.textile.liquid b/doc/user/tutorials/tutorial-keep-get.html.textile.liquid
index c73675bf1..acf4b33c1 100644
--- a/doc/user/tutorials/tutorial-keep-get.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep-get.html.textile.liquid
@@ -11,11 +11,39 @@ SPDX-License-Identifier: CC-BY-SA-3.0
Arvados Data collections can be downloaded using either the arv commands or using Workbench.
-# "*Downloading using arv commands*":#download-using-arv
-# "*Downloading using Workbench*":#download-using-workbench
-# "*Downloading a shared collection using Workbench*":#download-shared-collection
+# "*Download using Workbench*":#download-using-workbench
+# "*Sharing collections*":#download-shared-collection
+# "*Download using command line tools*":#download-using-arv
-h2(#download-using-arv). Downloading using arv commands
+h2(#download-using-workbench). Download using Workbench
+
+You can also download Arvados data collections using the Workbench.
+
+Visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu, select your *Home* project. You will see the *Data collections* tab, which lists the collections in this project.
+
+You can access the contents of a collection by clicking on the *<i class="fa fa-fw fa-archive"></i> Show* button next to the collection. This will take you to the collection's page. Using this page you can see the collection's contents, and download individual files.
+
+You can now download the collection files by clicking on the <span class="btn btn-sm btn-info"><i class="fa fa-download"></i></span> button(s).
+
+h2(#download-shared-collection). Sharing collections
+
+h3. Sharing with other Arvados users
+
+Collections can be shared with other users on the Arvados cluster by sharing the parent project. Navigate to the parent project using the "breadcrumbs" bar, then click on the *Sharing* tab. From the sharing tab, you can choose which users or groups to share with, and their level of access.
+
+h3. Creating a special download URL
+
+To share a collection with users that do not have an account on your Arvados cluster, visit the collection page using Workbench as described in the above section. Once on this page, click on the <span class="btn btn-sm btn-primary" >Create sharing link</span> button.
+
+This will create a sharing link for the collection as shown below. You can copy the sharing link in this page and share it with other users.
+
+!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/shared-collection.png!
+
+A user with this url can download this collection by simply accessing this url using browser. It will present a downloadable version of the collection as shown below.
+
+!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/download-shared-collection.png!
+
+h2(#download-using-arv). Download using command line tools
{% include 'tutorial_expectations' %}
@@ -71,27 +99,3 @@ You may also request a collection by UUID. In this case, it will contact the cl
<pre><code>~$ <span class="userinput">arv-get zzzzz-4zz18-fw6dnjxtkvzdewt/ .</span>
</code></pre>
</notextile>
-
-h2(#download-using-workbench). Downloading using Workbench
-
-You can also download Arvados data collections using the Workbench.
-
-Visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu, select your *Home* project. You will see the *Data collections* tab, which lists the collections in this project.
-
-You can access the contents of a collection by clicking on the *<i class="fa fa-fw fa-archive"></i> Show* button next to the collection. This will take you to the collection's page. Using this page you can see the collection's contents, download individual files, and set sharing options.
-
-You can now download the collection files by clicking on the <span class="btn btn-sm btn-info"><i class="fa fa-download"></i></span> button(s).
-
-h2(#download-shared-collection). Downloading a shared collection using Workbench
-
-Collections can be shared to allow downloads by anonymous users.
-
-To share a collection with anonymous users, visit the collection page using Workbench as described in the above section. Once on this page, click on the <span class="btn btn-sm btn-primary" >Create sharing link</span> button.
-
-This will create a sharing link for the collection as shown below. You can copy the sharing link in this page and share it with other users.
-
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/shared-collection.png!
-
-A user with this url can download this collection by simply accessing this url using browser. It will present a downloadable version of the collection as shown below.
-
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/download-shared-collection.png!
diff --git a/doc/user/tutorials/tutorial-keep-mount-gnu-linux.html.textile.liquid b/doc/user/tutorials/tutorial-keep-mount-gnu-linux.html.textile.liquid
index e17602199..eb2a91b75 100644
--- a/doc/user/tutorials/tutorial-keep-mount-gnu-linux.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep-mount-gnu-linux.html.textile.liquid
@@ -1,7 +1,7 @@
---
layout: default
navsection: userguide
-title: "Accessing Keep from GNU/Linux"
+title: "Keep network mount on GNU/Linux"
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
@@ -9,7 +9,7 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-This tutoral describes how to access Arvados collections on GNU/Linux using traditional filesystem tools by mounting Keep as a file system using @arv-mount at .
+GNU/Linux users can use @arv-mount@ to mount Keep as a file system in order to access Arvados collections using traditional filesystem tools.
{% include 'tutorial_expectations' %}
diff --git a/doc/user/tutorials/tutorial-keep-mount-os-x.html.textile.liquid b/doc/user/tutorials/tutorial-keep-mount-os-x.html.textile.liquid
index 9397d61e0..736952b5c 100644
--- a/doc/user/tutorials/tutorial-keep-mount-os-x.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep-mount-os-x.html.textile.liquid
@@ -1,7 +1,7 @@
---
layout: default
navsection: userguide
-title: "Accessing Keep from OS X"
+title: "Keep network mount on OS X"
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
@@ -19,6 +19,6 @@ This mount is read-only. Write support for the @/users/@ directory is planned fo
h3. Accessing a specific collection in Keep (read-write)
-In Finder, use "Connect to Server..." under the "Go" menu and enter @https://collections.ClusterID.example.com/@ in popup dialog. When prompted for credentials, put a valid Arvados token in the @Password@ field and anything in the Name field (it will be ignored by Arvados).
+In Finder, use "Connect to Server..." under the "Go" menu and enter @https://collections.ClusterID.example.com/c=your-collection-uuid@ in popup dialog. When prompted for credentials, put a valid Arvados token in the @Password@ field and anything in the Name field (it will be ignored by Arvados).
This collection is now accessible read/write.
diff --git a/doc/user/tutorials/tutorial-keep-mount-windows.html.textile.liquid b/doc/user/tutorials/tutorial-keep-mount-windows.html.textile.liquid
index 29b28fff9..0818f632f 100644
--- a/doc/user/tutorials/tutorial-keep-mount-windows.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep-mount-windows.html.textile.liquid
@@ -1,7 +1,7 @@
---
layout: default
navsection: userguide
-title: "Accessing Keep from Windows"
+title: "Keep network mount on Windows"
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
diff --git a/doc/user/tutorials/tutorial-keep.html.textile.liquid b/doc/user/tutorials/tutorial-keep.html.textile.liquid
index 5dd87c263..21efc475c 100644
--- a/doc/user/tutorials/tutorial-keep.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep.html.textile.liquid
@@ -9,13 +9,44 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Arvados Data collections can be uploaded using either the @arv-put@ command line tool or using Workbench.
+Arvados Data collections can be uploaded using either Workbench or the @arv-put@ command line tool.
-# "*Upload using command line tool*":#upload-using-command
# "*Upload using Workbench*":#upload-using-workbench
+# "*Creating projects*":#creating-projects
+# "*Upload using command line tool*":#upload-using-command
+
+h2(#upload-using-workbench). Upload using Workbench
+
+To upload using Workbench, visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu and select your *Home* project or any other project of your choosing. You will see the *Data collections* tab for this project, which lists the collections in this project.
+
+To upload files into a new collection, click on *Add data*<span class="caret"></span> dropdown menu and select *Upload files from my computer*.
+
+!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/upload-using-workbench.png!
+
+<br/>This will create a new empty collection in your chosen project and will take you to the *Upload* tab for that collection.
+
+!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/upload-tab-in-new-collection.png!
+
+Click on the *Browse...* button and select the files you would like to upload. Selected files will be added to a list of files to be uploaded. After you are done selecting files to upload, click on the *<i class="fa fa-fw fa-play"></i> Start* button to start upload. This will start uploading files to Arvados and Workbench will show you the progress bar. When upload is completed, you will see an indication to that effect.
+
+!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/files-uploaded.png!
+
+*Note:* If you leave the collection page during the upload, the upload process will be aborted and you will need to upload the files again.
+
+*Note:* You can also use the Upload tab to add additional files to an existing collection.
notextile. <div class="spaced-out">
+h2(#creating-projects). Creating projects
+
+Files are organized into Collections, and Collections are organized by Projects.
+
+Click on *Projects*<span class="caret"></span> <span class="rarr">→</span> <i class="fa fa-fw fa-plus"></i>*Add a new project* to add a top level project.
+
+To create a subproject, navigate to the parent project, and click on <i class="fa fa-fw fa-plus"></i>*Add a subproject*.
+
+See "Sharing collections":tutorial-keep-get.html#download-shared-collection for information about sharing projects and collections with other users.
+
h2(#upload-using-command). Upload using command line tool
{% include 'tutorial_expectations' %}
@@ -63,23 +94,3 @@ To move the collection to a different project, check the box at the left of the
Click on the *<i class="fa fa-fw fa-archive"></i> Show* button next to the collection's listing on a project page to go to the Workbench page for your collection. On this page, you can see the collection's contents, download individual files, and set sharing options.
notextile. </div>
-
-h2(#upload-using-workbench). Upload using Workbench
-
-To upload using Workbench, visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu and select your *Home* project or any other project of your choosing. You will see the *Data collections* tab for this project, which lists the collections in this project.
-
-To upload files into a new collection, click on *Add data*<span class="caret"></span> dropdown menu and select *Upload files from my computer*.
-
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/upload-using-workbench.png!
-
-<br/>This will create a new empty collection in your chosen project and will take you to the *Upload* tab for that collection.
-
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/upload-tab-in-new-collection.png!
-
-Click on the *Browse...* button and select the files you would like to upload. Selected files will be added to a list of files to be uploaded. After you are done selecting files to upload, click on the *<i class="fa fa-fw fa-play"></i> Start* button to start upload. This will start uploading files to Arvados and Workbench will show you the progress bar. When upload is completed, you will see an indication to that effect.
-
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/files-uploaded.png!
-
-*Note:* If you leave the collection page during the upload, the upload process will be aborted and you will need to upload the files again.
-
-*Note:* You can also use the Upload tab to add additional files to an existing collection.
commit 409690acd4085f0c4ec764034f09c26a0b555f0c
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Wed Aug 19 15:34:31 2020 -0400
15964: Fix link
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/user/cwl/cwl-runner.html.textile.liquid b/doc/user/cwl/cwl-runner.html.textile.liquid
index 3ed26a007..ed481334c 100644
--- a/doc/user/cwl/cwl-runner.html.textile.liquid
+++ b/doc/user/cwl/cwl-runner.html.textile.liquid
@@ -19,7 +19,7 @@ h2. Running arvados-cwl-runner
h3. Get the example files
-The tutorial files are located in the documentation section of the Arvados source repository, which can be found on "git.arvados.org": https://git.arvados.org/arvados.git/tree/HEAD:/doc/user/cwl/bwa-mem or "github":https://github.com/arvados/arvados/tree/master/doc/user/cwl/bwa-mem
+The tutorial files are located in the documentation section of the Arvados source repository, which can be found on "git.arvados.org":https://git.arvados.org/arvados.git/tree/HEAD:/doc/user/cwl/bwa-mem or "github":https://github.com/arvados/arvados/tree/master/doc/user/cwl/bwa-mem
<notextile>
<pre><code>~$ <span class="userinput">git clone https://git.arvados.org/arvados.git</span>
commit 7e47f5aaebcd7e4166e5e647f7b59af0914825bd
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Wed Aug 19 14:29:14 2020 -0400
15964: Change uuids to jutro, which is federated with pirca
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/user/cwl/bwa-mem/bwa-mem-input-mixed.yml b/doc/user/cwl/bwa-mem/bwa-mem-input-mixed.yml
index 4920bb29e..73dd65c46 100755
--- a/doc/user/cwl/bwa-mem/bwa-mem-input-mixed.yml
+++ b/doc/user/cwl/bwa-mem/bwa-mem-input-mixed.yml
@@ -15,15 +15,15 @@ cwl:tool: bwa-mem.cwl
reference:
class: File
location: keep:2463fa9efeb75e099685528b3b9071e0+438/19.fasta.bwt
- arv:collectionUUID: pirca-4zz18-jyjuzyqq14qspl7
+ arv:collectionUUID: jutro-4zz18-tv416l321i4r01e
read_p1:
class: File
location: keep:ae480c5099b81e17267b7445e35b4bc7+180/HWI-ST1027_129_D0THKACXX.1_1.fastq
- arv:collectionUUID: pirca-4zz18-fhbq692jyemnknq
+ arv:collectionUUID: jutro-4zz18-8k5hsvee0izv2g3
read_p2:
class: File
location: keep:ae480c5099b81e17267b7445e35b4bc7+180/HWI-ST1027_129_D0THKACXX.1_2.fastq
- arv:collectionUUID: pirca-4zz18-fhbq692jyemnknq
+ arv:collectionUUID: jutro-4zz18-8k5hsvee0izv2g3
group_id: arvados_tutorial
sample_id: HWI-ST1027_129
PL: illumina
diff --git a/doc/user/cwl/bwa-mem/bwa-mem-input-uuids.yml b/doc/user/cwl/bwa-mem/bwa-mem-input-uuids.yml
index 736a58beb..e76aa7817 100755
--- a/doc/user/cwl/bwa-mem/bwa-mem-input-uuids.yml
+++ b/doc/user/cwl/bwa-mem/bwa-mem-input-uuids.yml
@@ -9,13 +9,13 @@
cwl:tool: bwa-mem.cwl
reference:
class: File
- location: keep:pirca-4zz18-jyjuzyqq14qspl7/19.fasta.bwt
+ location: keep:jutro-4zz18-tv416l321i4r01e/19.fasta.bwt
read_p1:
class: File
- location: keep:pirca-4zz18-fhbq692jyemnknq/HWI-ST1027_129_D0THKACXX.1_1.fastq
+ location: keep:jutro-4zz18-8k5hsvee0izv2g3/HWI-ST1027_129_D0THKACXX.1_1.fastq
read_p2:
class: File
- location: keep:pirca-4zz18-fhbq692jyemnknq/HWI-ST1027_129_D0THKACXX.1_2.fastq
+ location: keep:jutro-4zz18-8k5hsvee0izv2g3/HWI-ST1027_129_D0THKACXX.1_2.fastq
group_id: arvados_tutorial
sample_id: HWI-ST1027_129
PL: illumina
commit dba9e46d852b50d2d72bd9619667f69637370fd7
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Tue Aug 18 14:12:43 2020 -0400
15964: Batch change qr1hi->zzzzz
Change some uuids to point to new collections on pirca
Delete several unlinked, obsolete crunch1 documentation pages.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/user/cwl/bwa-mem/bwa-mem-input-mixed.yml b/doc/user/cwl/bwa-mem/bwa-mem-input-mixed.yml
index 73bd9f599..4920bb29e 100755
--- a/doc/user/cwl/bwa-mem/bwa-mem-input-mixed.yml
+++ b/doc/user/cwl/bwa-mem/bwa-mem-input-mixed.yml
@@ -15,15 +15,15 @@ cwl:tool: bwa-mem.cwl
reference:
class: File
location: keep:2463fa9efeb75e099685528b3b9071e0+438/19.fasta.bwt
- arv:collectionUUID: qr1hi-4zz18-pwid4w22a40jp8l
+ arv:collectionUUID: pirca-4zz18-jyjuzyqq14qspl7
read_p1:
class: File
location: keep:ae480c5099b81e17267b7445e35b4bc7+180/HWI-ST1027_129_D0THKACXX.1_1.fastq
- arv:collectionUUID: qr1hi-4zz18-h615rgfmqt3wje0
+ arv:collectionUUID: pirca-4zz18-fhbq692jyemnknq
read_p2:
class: File
location: keep:ae480c5099b81e17267b7445e35b4bc7+180/HWI-ST1027_129_D0THKACXX.1_2.fastq
- arv:collectionUUID: qr1hi-4zz18-h615rgfmqt3wje0
+ arv:collectionUUID: pirca-4zz18-fhbq692jyemnknq
group_id: arvados_tutorial
sample_id: HWI-ST1027_129
PL: illumina
diff --git a/doc/user/cwl/bwa-mem/bwa-mem-input-uuids.yml b/doc/user/cwl/bwa-mem/bwa-mem-input-uuids.yml
index 7e71e959e..736a58beb 100755
--- a/doc/user/cwl/bwa-mem/bwa-mem-input-uuids.yml
+++ b/doc/user/cwl/bwa-mem/bwa-mem-input-uuids.yml
@@ -9,13 +9,13 @@
cwl:tool: bwa-mem.cwl
reference:
class: File
- location: keep:qr1hi-4zz18-pwid4w22a40jp8l/19.fasta.bwt
+ location: keep:pirca-4zz18-jyjuzyqq14qspl7/19.fasta.bwt
read_p1:
class: File
- location: keep:qr1hi-4zz18-h615rgfmqt3wje0/HWI-ST1027_129_D0THKACXX.1_1.fastq
+ location: keep:pirca-4zz18-fhbq692jyemnknq/HWI-ST1027_129_D0THKACXX.1_1.fastq
read_p2:
class: File
- location: keep:qr1hi-4zz18-h615rgfmqt3wje0/HWI-ST1027_129_D0THKACXX.1_2.fastq
+ location: keep:pirca-4zz18-fhbq692jyemnknq/HWI-ST1027_129_D0THKACXX.1_2.fastq
group_id: arvados_tutorial
sample_id: HWI-ST1027_129
PL: illumina
diff --git a/doc/user/cwl/bwa-mem/bwa-mem.cwl b/doc/user/cwl/bwa-mem/bwa-mem.cwl
index 200197126..a73329288 100755
--- a/doc/user/cwl/bwa-mem/bwa-mem.cwl
+++ b/doc/user/cwl/bwa-mem/bwa-mem.cwl
@@ -8,7 +8,7 @@ class: CommandLineTool
hints:
DockerRequirement:
- dockerPull: lh3lh3/bwa
+ dockerPull: quay.io/biocontainers/bwa:0.7.17--ha92aebf_3
baseCommand: [mem]
diff --git a/doc/user/cwl/cwl-run-options.html.textile.liquid b/doc/user/cwl/cwl-run-options.html.textile.liquid
index 725528f44..0c7de5cb3 100644
--- a/doc/user/cwl/cwl-run-options.html.textile.liquid
+++ b/doc/user/cwl/cwl-run-options.html.textile.liquid
@@ -74,10 +74,10 @@ Use the @--name@ and @--output-name@ options to specify the name of the workflow
<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --name "Example bwa run" --output-name "Example bwa output" bwa-mem.cwl bwa-mem-input.yml</span>
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to qr1hi-4zz18-h7ljh5u76760ww2
-2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Running
-2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Complete
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to zzzzz-4zz18-h7ljh5u76760ww2
+2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job zzzzz-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Running
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Complete
2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Overall process status is success
{
"aligned_sam": {
@@ -98,9 +98,9 @@ To submit a workflow and exit immediately, use the @--no-wait@ option. This wil
<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --no-wait bwa-mem.cwl bwa-mem-input.yml</span>
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Uploaded to qr1hi-4zz18-eqnfwrow8aysa9q
-2016-06-30 15:07:52 arvados.cwl-runner[12480] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-qr1hi-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Uploaded to zzzzz-4zz18-eqnfwrow8aysa9q
+2016-06-30 15:07:52 arvados.cwl-runner[12480] INFO: Submitted job zzzzz-8i9sb-fm2n3b1w0l6bskg
+zzzzz-8i9sb-fm2n3b1w0l6bskg
</code></pre>
</notextile>
@@ -111,10 +111,10 @@ To run a workflow with local control, use @--local at . This means that the host w
<notextile>
<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --local bwa-mem.cwl bwa-mem-input.yml</span>
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
-2016-07-01 10:05:19 arvados.cwl-runner[16290] INFO: Pipeline instance qr1hi-d1hrv-92wcu6ldtio74r4
-2016-07-01 10:05:28 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Queued
-2016-07-01 10:05:29 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Running
-2016-07-01 10:05:45 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Complete
+2016-07-01 10:05:19 arvados.cwl-runner[16290] INFO: Pipeline instance zzzzz-d1hrv-92wcu6ldtio74r4
+2016-07-01 10:05:28 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-2nzzfbuf9zjrj4g) is Queued
+2016-07-01 10:05:29 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-2nzzfbuf9zjrj4g) is Running
+2016-07-01 10:05:45 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-2nzzfbuf9zjrj4g) is Complete
2016-07-01 10:05:46 arvados.cwl-runner[16290] INFO: Overall process status is success
{
"aligned_sam": {
diff --git a/doc/user/cwl/cwl-runner.html.textile.liquid b/doc/user/cwl/cwl-runner.html.textile.liquid
index 8af1c574f..3ed26a007 100644
--- a/doc/user/cwl/cwl-runner.html.textile.liquid
+++ b/doc/user/cwl/cwl-runner.html.textile.liquid
@@ -19,30 +19,27 @@ h2. Running arvados-cwl-runner
h3. Get the example files
-The tutorial files are located in the "documentation section of the Arvados source repository:":https://github.com/arvados/arvados/tree/master/doc/user/cwl/bwa-mem
+The tutorial files are located in the documentation section of the Arvados source repository, which can be found on "git.arvados.org": https://git.arvados.org/arvados.git/tree/HEAD:/doc/user/cwl/bwa-mem or "github":https://github.com/arvados/arvados/tree/master/doc/user/cwl/bwa-mem
<notextile>
-<pre><code>~$ <span class="userinput">git clone https://github.com/arvados/arvados</span>
+<pre><code>~$ <span class="userinput">git clone https://git.arvados.org/arvados.git</span>
~$ <span class="userinput">cd arvados/doc/user/cwl/bwa-mem</span>
</code></pre>
</notextile>
-The tutorial data is hosted on "https://playground.arvados.org":https://playground.arvados.org (also referred to by the identifier *qr1hi*). If you are using a different Arvados instance, you may need to copy the data to your own instance. The easiest way to do this is with "arv-copy":{{site.baseurl}}/user/topics/arv-copy.html (this requires signing up for a free playground.arvados.org account).
+The tutorial data is hosted on "https://playground.arvados.org":https://playground.arvados.org (also referred to by the identifier *pirca*). If you are using a different Arvados instance, you may need to copy the data to your own instance. One way to do this is with "arv-copy":{{site.baseurl}}/user/topics/arv-copy.html (this requires signing up for a free playground.arvados.org account).
<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src qr1hi --dst settings 2463fa9efeb75e099685528b3b9071e0+438</span>
-~$ <span class="userinput">arv-copy --src qr1hi --dst settings ae480c5099b81e17267b7445e35b4bc7+180</span>
-~$ <span class="userinput">arv-copy --src qr1hi --dst settings 655c6cd07550151b210961ed1d3852cf+57</span>
+<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst settings 2463fa9efeb75e099685528b3b9071e0+438</span>
+~$ <span class="userinput">arv-copy --src pirca --dst settings ae480c5099b81e17267b7445e35b4bc7+180</span>
</code></pre>
</notextile>
If you do not wish to create an account on "https://playground.arvados.org":https://playground.arvados.org, you may download the files anonymously and upload them to your local Arvados instance:
-"https://playground.arvados.org/collections/2463fa9efeb75e099685528b3b9071e0+438":https://playground.arvados.org/collections/2463fa9efeb75e099685528b3b9071e0+438
+"https://collections.pirca.arvadosapi.com/c=2463fa9efeb75e099685528b3b9071e0+438/":https://collections.pirca.arvadosapi.com/c=2463fa9efeb75e099685528b3b9071e0+438/
-"https://playground.arvados.org/collections/ae480c5099b81e17267b7445e35b4bc7+180":https://playground.arvados.org/collections/ae480c5099b81e17267b7445e35b4bc7+180
-
-"https://playground.arvados.org/collections/655c6cd07550151b210961ed1d3852cf+57":https://playground.arvados.org/collections/655c6cd07550151b210961ed1d3852cf+57
+"https://collections.pirca.arvadosapi.com/c=ae480c5099b81e17267b7445e35b4bc7+180/":https://collections.pirca.arvadosapi.com/c=ae480c5099b81e17267b7445e35b4bc7+180/
h2. Submitting a workflow to an Arvados cluster
@@ -56,10 +53,10 @@ Use @arvados-cwl-runner@ to submit CWL workflows to Arvados. After submitting t
<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner bwa-mem.cwl bwa-mem-input.yml</span>
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to qr1hi-4zz18-h7ljh5u76760ww2
-2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Running
-2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Complete
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to zzzzz-4zz18-h7ljh5u76760ww2
+2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job zzzzz-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Running
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Complete
2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Overall process status is success
{
"aligned_sam": {
@@ -87,10 +84,10 @@ You can also execute CWL files directly from Keep:
<notextile>
<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner keep:655c6cd07550151b210961ed1d3852cf+57/bwa-mem.cwl bwa-mem-input.yml</span>
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
-2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to qr1hi-4zz18-h7ljh5u76760ww2
-2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Running
-2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Complete
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to zzzzz-4zz18-h7ljh5u76760ww2
+2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job zzzzz-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Running
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Complete
2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Overall process status is success
{
"aligned_sam": {
diff --git a/doc/user/getting_started/check-environment.html.textile.liquid b/doc/user/getting_started/check-environment.html.textile.liquid
index b707891a1..1097e4e9d 100644
--- a/doc/user/getting_started/check-environment.html.textile.liquid
+++ b/doc/user/getting_started/check-environment.html.textile.liquid
@@ -16,14 +16,14 @@ Check that you are able to access the Arvados API server using @arv user current
<notextile>
<pre><code>$ <span class="userinput">arv user current</span>
{
- "href":"https://qr1hi.arvadosapi.com/arvados/v1/users/qr1hi-xioed-9z2p3pn12yqdaem",
+ "href":"https://zzzzz.arvadosapi.com/arvados/v1/users/zzzzz-xioed-9z2p3pn12yqdaem",
"kind":"arvados#user",
"etag":"8u0xwb9f3otb2xx9hto4wyo03",
- "uuid":"qr1hi-tpzed-92d3kxnimy3d4e8",
- "owner_uuid":"qr1hi-tpqed-23iddeohxta2r59",
+ "uuid":"zzzzz-tpzed-92d3kxnimy3d4e8",
+ "owner_uuid":"zzzzz-tpqed-23iddeohxta2r59",
"created_at":"2013-12-02T17:05:47Z",
- "modified_by_client_uuid":"qr1hi-xxfg8-owxa2oa2s33jyej",
- "modified_by_user_uuid":"qr1hi-tpqed-23iddeohxta2r59",
+ "modified_by_client_uuid":"zzzzz-xxfg8-owxa2oa2s33jyej",
+ "modified_by_user_uuid":"zzzzz-tpqed-23iddeohxta2r59",
"modified_at":"2013-12-02T17:07:08Z",
"updated_at":"2013-12-05T19:51:08Z",
"email":"you at example.com",
diff --git a/doc/user/topics/arv-copy.html.textile.liquid b/doc/user/topics/arv-copy.html.textile.liquid
index 805d71b2d..e5e631d4e 100644
--- a/doc/user/topics/arv-copy.html.textile.liquid
+++ b/doc/user/topics/arv-copy.html.textile.liquid
@@ -17,54 +17,54 @@ h2. arv-copy
@arv-copy@ allows users to copy collections and workflows from one cluster to another. By default, @arv-copy@ will recursively go through the workflow and copy all dependencies associated with the object.
-For example, let's copy from the <a href="https://playground.arvados.org/">Arvados playground</a>, also known as *qr1hi*, to *dst_cluster*. The names *qr1hi* and *dst_cluster* are interchangable with any cluster name. You can find the cluster name from the prefix of the uuid of the object you want to copy. For example, in *qr1hi*-4zz18-tci4vn4fa95w0zx, the cluster name is qr1hi.
+For example, let's copy from the <a href="https://playground.arvados.org/">Arvados playground</a>, also known as *pirca*, to *dstcl*. The names *pirca* and *dstcl* are interchangable with any cluster id. You can find the cluster name from the prefix of the uuid of the object you want to copy. For example, in *zzzzz*-4zz18-tci4vn4fa95w0zx, the cluster name is zzzzz.
-In order to communicate with both clusters, you must create custom configuration files for each cluster. In the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Current token*. Copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ in both of your clusters. Then, create two configuration files, one for each cluster. The names of the files must have the format of *ClusterID.conf*. In our example, let's make two files, one for *qr1hi* and one for *dst_cluster*. From your *Current token* page in *qr1hi* and *dst_cluster*, copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN at .
+In order to communicate with both clusters, you must create custom configuration files for each cluster. In the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Current token*. Copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ in both of your clusters. Then, create two configuration files, one for each cluster. The names of the files must have the format of *ClusterID.conf*. In our example, let's make two files, one for *pirca* and one for *dstcl*. From your *Current token* page in *pirca* and *dstcl*, copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN at .
!{display: block;margin-left: 25px;margin-right: auto;}{{ site.baseurl }}/images/api-token-host.png!
-Copy your @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ into the config files as shown below in the shell account from which you are executing the commands. For example, the default shell you may have access to is shell.qr1hi. You can add these files in ~/.config/arvados/ in the qr1hi shell terminal.
+Copy your @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ into the config files as shown below in the shell account from which you are executing the commands. For example, the default shell you may have access to is shell.pirca. You can add these files in ~/.config/arvados/ in the pirca shell terminal.
<notextile>
<pre><code>~$ <span class="userinput">cd ~/.config/arvados</span>
-~$ <span class="userinput">echo "ARVADOS_API_HOST=qr1hi.arvadosapi.com" >> qr1hi.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_TOKEN=123456789abcdefghijkl" >> qr1hi.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_HOST=dst_cluster.arvadosapi.com" >> dst_cluster.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_TOKEN=987654321lkjihgfedcba" >> dst_cluster.conf</span>
+~$ <span class="userinput">echo "ARVADOS_API_HOST=pirca.arvadosapi.com" >> pirca.conf</span>
+~$ <span class="userinput">echo "ARVADOS_API_TOKEN=123456789abcdefghijkl" >> pirca.conf</span>
+~$ <span class="userinput">echo "ARVADOS_API_HOST=dstcl.arvadosapi.com" >> dstcl.conf</span>
+~$ <span class="userinput">echo "ARVADOS_API_TOKEN=987654321lkjihgfedcba" >> dstcl.conf</span>
</code></pre>
</notextile>
-Now you're ready to copy between *qr1hi* and *dst_cluster*!
+Now you're ready to copy between *pirca* and *dstcl*!
h3. How to copy a collection
First, select the uuid of the collection you want to copy from the source cluster. The uuid can be found in the collection display page in the collection summary area (top left box), or from the URL bar (the part after @collections/...@)
-Now copy the collection from *qr1hi* to *dst_cluster*. We will use the uuid @qr1hi-4zz18-tci4vn4fa95w0zx@ as an example. You can find this collection in the <a href="https://playground.arvados.org/collections/qr1hi-4zz18-tci4vn4fa95w0zx">lobSTR v.3 project on playground.arvados.org</a>.
+Now copy the collection from *pirca* to *dstcl*. We will use the uuid @pirca-4zz18-jyjuzyqq14qspl7@ as an example. You can find this collection in the <a href="https://playground.arvados.org/collections/pirca-4zz18-jyjuzyqq14qspl7">User guide resources project on playground.arvados.org</a>.
<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src qr1hi --dst dst_cluster qr1hi-4zz18-tci4vn4fa95w0zx</span>
-qr1hi-4zz18-tci4vn4fa95w0zx: 6.1M / 6.1M 100.0%
-arvados.arv-copy[1234] INFO: Success: created copy with uuid dst_cluster-4zz18-8765943210cdbae
+<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl pirca-4zz18-jyjuzyqq14qspl7</span>
+pirca-4zz18-tci4vn4fa95w0zx: 6.1M / 6.1M 100.0%
+arvados.arv-copy[1234] INFO: Success: created copy with uuid dstcl-4zz18-8765943210cdbae
</code></pre>
</notextile>
The output of arv-copy displays the uuid of the collection generated in the destination cluster. By default, the output is placed in your home project in the destination cluster. If you want to place your collection in a pre-created project, you can specify the project you want it to be in using the tag @--project-uuid@ followed by the project uuid.
-For example, this will copy the collection to project dst_cluster-j7d0g-a894213ukjhal12 in the destination cluster.
+For example, this will copy the collection to project dstcl-j7d0g-a894213ukjhal12 in the destination cluster.
-<notextile> <pre><code>~$ <span class="userinput">arv-copy --src qr1hi --dst dst_cluster --project-uuid dst_cluster-j7d0g-a894213ukjhal12 qr1hi-4zz18-tci4vn4fa95w0zx</span>
+<notextile> <pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl --project-uuid dstcl-j7d0g-a894213ukjhal12 pirca-4zz18-tci4vn4fa95w0zx</span>
</code></pre>
</notextile>
h3. How to copy a workflow
-We will use the uuid @zzzzz-7fd4e-sampleworkflow1@ as an example workflow.
+We will use the uuid @pirca-7fd4e-sampleworkflow1@ as an example workflow.
<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src zzzzz --dst dst_cluster --dst-git-repo $USER/tutorial zzzzz-7fd4e-sampleworkflow1</span>
-zzzzz-4zz18-jidprdejysravcr: 1143M / 1143M 100.0%
+<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl --dst-git-repo $USER/tutorial pirca-7fd4e-sampleworkflow1</span>
+pirca-4zz18-jidprdejysravcr: 1143M / 1143M 100.0%
2017-01-04 04:11:58 arvados.arv-copy[5906] INFO:
-2017-01-04 04:11:58 arvados.arv-copy[5906] INFO: Success: created copy with uuid dst_cluster-7fd4e-ojtgpne594ubkt7
+2017-01-04 04:11:58 arvados.arv-copy[5906] INFO: Success: created copy with uuid dstcl-7fd4e-ojtgpne594ubkt7
</code></pre>
</notextile>
@@ -75,6 +75,6 @@ If you would like to copy the object without dependencies, you can use the @--no
For example, we can copy the same object non-recursively using the following:
<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src zzzzz --dst dst_cluster --dst-git-repo $USER/tutorial --no-recursive zzzzz-7fd4e-sampleworkflow1</span>
+<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl --dst-git-repo $USER/tutorial --no-recursive pirca-7fd4e-sampleworkflow1</span>
</code></pre>
</notextile>
diff --git a/doc/user/topics/arv-docker.html.textile.liquid b/doc/user/topics/arv-docker.html.textile.liquid
index d371400f5..b03ef0bdf 100644
--- a/doc/user/topics/arv-docker.html.textile.liquid
+++ b/doc/user/topics/arv-docker.html.textile.liquid
@@ -103,7 +103,7 @@ h3. Uploading Docker images to a shared project
Docker images are subject to normal Arvados permissions. If wish to share your Docker image with others you should use @arv-keepdocker@ with the @--project-uuid@ option to add the image to a shared project and ensure that metadata is set correctly.
<notextile>
-<pre><code>$ <span class="userinput">arv-keepdocker docker-example-r-base --project-uuid qr1hi-j7d0g-xxxxxxxxxxxxxxx</span>
+<pre><code>$ <span class="userinput">arv-keepdocker docker-example-r-base --project-uuid zzzzz-j7d0g-xxxxxxxxxxxxxxx</span>
</code></pre>
</notextile>
diff --git a/doc/user/topics/arv-web.html.textile.liquid b/doc/user/topics/arv-web.html.textile.liquid
deleted file mode 100644
index 9671e9709..000000000
--- a/doc/user/topics/arv-web.html.textile.liquid
+++ /dev/null
@@ -1,106 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Using arv-web"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
- at arv-web@ enables you to run a custom web service from the contents of an Arvados collection.
-
-{% include 'tutorial_expectations_workstation' %}
-
-h2. Usage
-
- at arv-web@ enables you to set up a web service based on the most recent collection in a project. An arv-web application is a reproducible, immutable application bundle where the web app is packaged with both the code to run and the data to serve. Because Arvados Collections can be updated with minimum duplication, it is efficient to produce a new application bundle when the code or data needs to be updated; retaining old application bundles makes it easy to go back and run older versions of your web app.
-
-<pre>
-$ cd $HOME/arvados/services/arv-web
-usage: arv-web.py [-h] --project-uuid PROJECT_UUID [--port PORT]
- [--image IMAGE]
-
-optional arguments:
- -h, --help show this help message and exit
- --project-uuid PROJECT_UUID
- Project uuid to watch
- --port PORT Host port to listen on (default 8080)
- --image IMAGE Docker image to run
-</pre>
-
-At startup, @arv-web@ queries an Arvados project and mounts the most recently modified collection into a temporary directory. It then runs a Docker image with the collection bound to @/mnt@ inside the container. When a new collection is added to the project, or an existing project is updated, it will stop the running Docker container, unmount the old collection, mount the new most recently modified collection, and restart the Docker container with the new mount.
-
-h2. Docker container
-
-The @Dockerfile@ in @arvados/docker/arv-web@ builds a Docker image that runs Apache with @/mnt@ as the DocumentRoot. It is configured to run web applications which use Python WSGI, Ruby Rack, or CGI; to serve static HTML; or browse the contents of the @public@ subdirectory of the collection using default Apache index pages.
-
-To build the Docker image:
-
-<notextile>
-<pre><code>~$ <span class="userinput">cd arvados/docker</span>
-~/arvados/docker$ <span class="userinput">docker build -t arvados/arv-web arv-web</span>
-</code></pre>
-</notextile>
-
-h2. Running sample applications
-
-First, in Arvados Workbench, create a new project. Copy the project UUID from the URL bar (this is the part of the URL after @projects/...@).
-
-Now upload a collection containing a "Python WSGI web app:":http://wsgi.readthedocs.org/en/latest/
-
-<notextile>
-<pre><code>~$ <span class="userinput">cd arvados/services/arv-web</span>
-~/arvados/services/arv-web$ <span class="userinput">arv-put --project [zzzzz-j7d0g-yourprojectuuid] --name sample-wsgi-app sample-wsgi-app</span>
-0M / 0M 100.0%
-Collection saved as 'sample-wsgi-app'
-zzzzz-4zz18-ebohzfbzh82qmqy
-~/arvados/services/arv-web$ <span class="userinput">./arv-web.py --project [zzzzz-j7d0g-yourprojectuuid] --port 8888</span>
-2015-01-30 11:21:00 arvados.arv-web[4897] INFO: Mounting zzzzz-4zz18-ebohzfbzh82qmqy
-2015-01-30 11:21:01 arvados.arv-web[4897] INFO: Starting Docker container arvados/arv-web
-2015-01-30 11:21:02 arvados.arv-web[4897] INFO: Container id e79e70558d585a3e038e4bfbc97e5c511f21b6101443b29a8017bdf3d84689a3
-2015-01-30 11:21:03 arvados.arv-web[4897] INFO: Waiting for events
-</code></pre>
-</notextile>
-
-The sample application will be available at @http://localhost:8888@.
-
-h3. Updating the application
-
-If you upload a new collection to the same project, arv-web will restart the web service and serve the new collection. For example, uploading a collection containing a "Ruby Rack web app:":https://github.com/rack/rack/wiki
-
-<notextile>
-<pre><code>~$ <span class="userinput">cd arvados/services/arv-web</span>
-~/arvados/services/arv-web$ <span class="userinput">arv-put --project [zzzzz-j7d0g-yourprojectuuid] --name sample-rack-app sample-rack-app</span>
-0M / 0M 100.0%
-Collection saved as 'sample-rack-app'
-zzzzz-4zz18-dhhm0ay8k8cqkvg
-</code></pre>
-</notextile>
-
- at arv-web@ will automatically notice the change, load a new container, and send an update signal (SIGHUP) to the service:
-
-<pre>
-2015-01-30 11:21:03 arvados.arv-web[4897] INFO:Waiting for events
-2015-01-30 11:21:04 arvados.arv-web[4897] INFO:create zzzzz-4zz18-dhhm0ay8k8cqkvg
-2015-01-30 11:21:05 arvados.arv-web[4897] INFO:Mounting zzzzz-4zz18-dhhm0ay8k8cqkvg
-2015-01-30 11:21:06 arvados.arv-web[4897] INFO:Sending refresh signal to container
-2015-01-30 11:21:07 arvados.arv-web[4897] INFO:Waiting for events
-</pre>
-
-h2. Writing your own applications
-
-The @arvados/arv-web@ image serves Python and Ruby applications using Phusion Passenger and Apache @mod_passenger at . See "Phusion Passenger users guide for Apache":https://www.phusionpassenger.com/documentation/Users%20guide%20Apache.html for details, and look at the sample apps @arvados/services/arv-web/sample-wsgi-app@ and @arvados/services/arv-web/sample-rack-app at .
-
-You can serve CGI applications using standard Apache CGI support. See "Apache Tutorial: Dynamic Content with CGI":https://httpd.apache.org/docs/current/howto/cgi.html for details, and look at the sample app @arvados/services/arv-web/sample-cgi-app at .
-
-You can also serve static content from the @public@ directory of the collection. Look at @arvados/services/arv-web/sample-static-page@ for an example. If no @index.html@ is found in @public/@, it will render default Apache index pages, permitting simple browsing of the collection contents.
-
-h3. Custom images
-
-You can provide your own Docker image. The Docker image that will be used create the web application container is specified in the @docker_image@ file in the root of the collection. You can also specify @--image@ on the command @arv-web@ line to choose the docker image (this will override the contents of @docker_image@).
-
-h3. Reloading the web service
-
-Stopping the Docker container and starting it again can result in a small amount of downtime. When the collection containing a new or updated web application uses the same Docker image as the currently running web application, it is possible to avoid this downtime by keeping the existing container and only reloading the web server. This is accomplished by providing a file called @reload@ in the root of the collection, which should contain the commands necessary to reload the web server inside the container.
diff --git a/doc/user/topics/keep.html.textile.liquid b/doc/user/topics/keep.html.textile.liquid
index 68b6a87d0..c415cebbd 100644
--- a/doc/user/topics/keep.html.textile.liquid
+++ b/doc/user/topics/keep.html.textile.liquid
@@ -38,7 +38,7 @@ notextile. <pre><code>~$ <span class="userinput">cd /scratch/<b>you</b></span>
When you run this command, you may get this API warning:
-notextile. <pre><code>WARNING:root:API lookup failed for collection 204e43b8a1185621ca55a94839582e6f+67108864 (<class 'apiclient.errors.HttpError'>: <HttpError 404 when requesting https://qr1hi.arvadosapi.com/arvados/v1/collections/204e43b8a1185621ca55a94839582e6f%2B67108864?alt=json returned "Not Found">)</code></pre>
+notextile. <pre><code>WARNING:root:API lookup failed for collection 204e43b8a1185621ca55a94839582e6f+67108864 (<class 'apiclient.errors.HttpError'>: <HttpError 404 when requesting https://zzzzz.arvadosapi.com/arvados/v1/collections/204e43b8a1185621ca55a94839582e6f%2B67108864?alt=json returned "Not Found">)</code></pre>
This happens because @arv-get@ tries to find a collection with this identifier. When that fails, it emits this warning, then looks for a datablock instead, which succeeds.
diff --git a/doc/user/topics/tutorial-gatk-variantfiltration.html.textile.liquid b/doc/user/topics/tutorial-gatk-variantfiltration.html.textile.liquid
deleted file mode 100644
index 544ccbd35..000000000
--- a/doc/user/topics/tutorial-gatk-variantfiltration.html.textile.liquid
+++ /dev/null
@@ -1,173 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Using GATK with Arvados"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-This tutorial demonstrates how to use the Genome Analysis Toolkit (GATK) with Arvados. In this example we will install GATK and then create a VariantFiltration job to assign pass/fail scores to variants in a VCF file.
-
-{% include 'tutorial_expectations' %}
-
-h2. Installing GATK
-
-Download the GATK binary tarball[1] -- e.g., @GenomeAnalysisTK-2.6-4.tar.bz2@ -- and "copy it to your Arvados VM":{{site.baseurl}}/user/tutorials/tutorial-keep.html.
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv-put GenomeAnalysisTK-2.6-4.tar.bz2</span>
-c905c8d8443a9c44274d98b7c6cfaa32+94
-</code></pre>
-</notextile>
-
-Next, you need the GATK Resource Bundle[2]. This may already be available in Arvados. If not, you will need to download the files listed below and put them into Keep.
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv keep ls -s d237a90bae3870b3b033aea1e99de4a9+10820</span>
- 50342 1000G_omni2.5.b37.vcf.gz
- 1 1000G_omni2.5.b37.vcf.gz.md5
- 464 1000G_omni2.5.b37.vcf.idx.gz
- 1 1000G_omni2.5.b37.vcf.idx.gz.md5
- 43981 1000G_phase1.indels.b37.vcf.gz
- 1 1000G_phase1.indels.b37.vcf.gz.md5
- 326 1000G_phase1.indels.b37.vcf.idx.gz
- 1 1000G_phase1.indels.b37.vcf.idx.gz.md5
- 537210 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.gz
- 1 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.gz.md5
- 3473 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.idx.gz
- 1 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.idx.gz.md5
- 19403 Mills_and_1000G_gold_standard.indels.b37.vcf.gz
- 1 Mills_and_1000G_gold_standard.indels.b37.vcf.gz.md5
- 536 Mills_and_1000G_gold_standard.indels.b37.vcf.idx.gz
- 1 Mills_and_1000G_gold_standard.indels.b37.vcf.idx.gz.md5
- 29291 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.gz
- 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.gz.md5
- 565 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.idx.gz
- 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.idx.gz.md5
- 37930 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.gz
- 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.gz.md5
- 592 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.idx.gz
- 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.idx.gz.md5
-5898484 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam
- 112 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam.bai.gz
- 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam.bai.gz.md5
- 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam.md5
- 3837 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.gz
- 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.gz.md5
- 65 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.idx.gz
- 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.idx.gz.md5
- 275757 dbsnp_137.b37.excluding_sites_after_129.vcf.gz
- 1 dbsnp_137.b37.excluding_sites_after_129.vcf.gz.md5
- 3735 dbsnp_137.b37.excluding_sites_after_129.vcf.idx.gz
- 1 dbsnp_137.b37.excluding_sites_after_129.vcf.idx.gz.md5
- 998153 dbsnp_137.b37.vcf.gz
- 1 dbsnp_137.b37.vcf.gz.md5
- 3890 dbsnp_137.b37.vcf.idx.gz
- 1 dbsnp_137.b37.vcf.idx.gz.md5
- 58418 hapmap_3.3.b37.vcf.gz
- 1 hapmap_3.3.b37.vcf.gz.md5
- 999 hapmap_3.3.b37.vcf.idx.gz
- 1 hapmap_3.3.b37.vcf.idx.gz.md5
- 3 human_g1k_v37.dict.gz
- 1 human_g1k_v37.dict.gz.md5
- 2 human_g1k_v37.fasta.fai.gz
- 1 human_g1k_v37.fasta.fai.gz.md5
- 849537 human_g1k_v37.fasta.gz
- 1 human_g1k_v37.fasta.gz.md5
- 1 human_g1k_v37.stats.gz
- 1 human_g1k_v37.stats.gz.md5
- 3 human_g1k_v37_decoy.dict.gz
- 1 human_g1k_v37_decoy.dict.gz.md5
- 2 human_g1k_v37_decoy.fasta.fai.gz
- 1 human_g1k_v37_decoy.fasta.fai.gz.md5
- 858592 human_g1k_v37_decoy.fasta.gz
- 1 human_g1k_v37_decoy.fasta.gz.md5
- 1 human_g1k_v37_decoy.stats.gz
- 1 human_g1k_v37_decoy.stats.gz.md5
-</code></pre>
-</notextile>
-
-h2. Submit a GATK job
-
-The Arvados distribution includes an example crunch script ("crunch_scripts/GATK2-VariantFiltration":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-VariantFiltration) that runs the GATK VariantFiltration tool with some default settings.
-
-<notextile>
-<pre><code>~$ <span class="userinput">src_version=76588bfc57f33ea1b36b82ca7187f465b73b4ca4</span>
-~$ <span class="userinput">vcf_input=5ee633fe2569d2a42dd81b07490d5d13+82</span>
-~$ <span class="userinput">gatk_binary=c905c8d8443a9c44274d98b7c6cfaa32+94</span>
-~$ <span class="userinput">gatk_bundle=d237a90bae3870b3b033aea1e99de4a9+10820</span>
-~$ <span class="userinput">cat >the_job <<EOF
-{
- "script":"GATK2-VariantFiltration",
- "repository":"arvados",
- "script_version":"$src_version",
- "script_parameters":
- {
- "input":"$vcf_input",
- "gatk_binary_tarball":"$gatk_binary",
- "gatk_bundle":"$gatk_bundle"
- }
-}
-EOF</span>
-</code></pre>
-</notextile>
-
-* @"input"@ is collection containing the source VCF data. Here we are using an exome report from PGP participant hu34D5B9.
-* @"gatk_binary_tarball"@ is a Keep collection containing the GATK 2 binary distribution[1] tar file.
-* @"gatk_bundle"@ is a Keep collection containing the GATK resource bundle[2].
-
-Now start a job:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv job create --job "$(cat the_job)"</span>
-{
- "href":"https://qr1hi.arvadosapi.com/arvados/v1/jobs/qr1hi-8i9sb-n9k7qyp7bs5b9d4",
- "kind":"arvados#job",
- "etag":"9j99n1feoxw3az448f8ises12",
- "uuid":"qr1hi-8i9sb-n9k7qyp7bs5b9d4",
- "owner_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
- "created_at":"2013-12-17T19:02:15Z",
- "modified_by_client_uuid":"qr1hi-ozdt8-obw7foaks3qjyej",
- "modified_by_user_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
- "modified_at":"2013-12-17T19:02:15Z",
- "updated_at":"2013-12-17T19:02:15Z",
- "submit_id":null,
- "priority":null,
- "script":"GATK2-VariantFiltration",
- "script_parameters":{
- "input":"5ee633fe2569d2a42dd81b07490d5d13+82",
- "gatk_binary_tarball":"c905c8d8443a9c44274d98b7c6cfaa32+94",
- "gatk_bundle":"d237a90bae3870b3b033aea1e99de4a9+10820"
- },
- "script_version":"76588bfc57f33ea1b36b82ca7187f465b73b4ca4",
- "cancelled_at":null,
- "cancelled_by_client_uuid":null,
- "cancelled_by_user_uuid":null,
- "started_at":null,
- "finished_at":null,
- "output":null,
- "success":null,
- "running":null,
- "is_locked_by_uuid":null,
- "log":null,
- "runtime_constraints":{},
- "tasks_summary":{}
-}
-</code></pre>
-</notextile>
-
-Once the job completes, the output can be found in hu34D5B9-exome-filtered.vcf:
-
-<notextile><pre><code>~$ <span class="userinput">arv keep ls bedd6ff56b3ae9f90d873b1fcb72f9a3+91</span>
-hu34D5B9-exome-filtered.vcf
-</code></pre>
-</notextile>
-
-h2. Notes
-
-fn1. "Download the GATK tools":http://www.broadinstitute.org/gatk/download
-
-fn2. "Information about the GATK resource bundle":http://gatkforums.broadinstitute.org/discussion/1213/whats-in-the-resource-bundle-and-how-can-i-get-it and "direct download link":ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/2.5/b37/ (if prompted, submit an empty password)
diff --git a/doc/user/topics/tutorial-job1.html.textile.liquid b/doc/user/topics/tutorial-job1.html.textile.liquid
deleted file mode 100644
index f7a206010..000000000
--- a/doc/user/topics/tutorial-job1.html.textile.liquid
+++ /dev/null
@@ -1,214 +0,0 @@
----
-layout: default
-navsection: userguide
-title: "Running a Crunch job on the command line"
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-This tutorial introduces how to run individual Crunch jobs using the @arv@ command line tool.
-
-{% include 'tutorial_expectations' %}
-
-You will create a job to run the "hash" Crunch script. The "hash" script computes the MD5 hash of each file in a collection.
-
-h2. Jobs
-
-Crunch pipelines consist of one or more jobs. A "job" is a single run of a specific version of a Crunch script with a specific input. You can also run jobs individually.
-
-A request to run a Crunch job are is described using a JSON object. For example:
-
-<notextile>
-<pre><code>~$ <span class="userinput">cat >~/the_job <<EOF
-{
- "script": "hash",
- "repository": "arvados",
- "script_version": "master",
- "script_parameters": {
- "input": "c1bad4b39ca5a924e481008009d94e32+210"
- },
- "no_reuse": "true"
-}
-EOF
-</code></pre>
-</notextile>
-
-* @cat@ is a standard Unix utility that writes a sequence of input to standard output.
-* @<<EOF@ tells the shell to direct the following lines into the standard input for @cat@ up until it sees the line @EOF at .
-* @>~/the_job@ redirects standard output to a file called @~/the_job at .
-* @"repository"@ is the name of a Git repository to search for the script version. You can access a list of available git repositories on the Arvados Workbench under "*Code repositories*":{{site.arvados_workbench_host}}/repositories.
-* @"script_version"@ specifies the version of the script that you wish to run. This can be in the form of an explicit Git revision hash, a tag, or a branch. Arvados logs the script version that was used in the run, enabling you to go back and re-run any past job with the guarantee that the exact same code will be used as was used in the previous run.
-* @"script"@ specifies the name of the script to run. The script must be given relative to the @crunch_scripts/@ subdirectory of the Git repository.
-* @"script_parameters"@ are provided to the script. In this case, the input is the PGP data Collection that we "put in Keep earlier":{{site.baseurl}}/user/tutorials/tutorial-keep.html.
-* Setting the @"no_reuse"@ flag tells Crunch not to reuse work from past jobs. This helps ensure that you can watch a new Job process for the rest of this tutorial, without reusing output from a past run that you made, or somebody else marked as public. (If you want to experiment, after the first run below finishes, feel free to edit this job to remove the @"no_reuse"@ line and resubmit it. See what happens!)
-
-Use @arv job create@ to actually submit the job. It should print out a JSON object which describes the newly created job:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv job create --job "$(cat ~/the_job)"</span>
-{
- "href":"https://qr1hi.arvadosapi.com/arvados/v1/jobs/qr1hi-8i9sb-1pm1t02dezhupss",
- "kind":"arvados#job",
- "etag":"ax3cn7w9whq2hdh983yxvq09p",
- "uuid":"qr1hi-8i9sb-1pm1t02dezhupss",
- "owner_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
- "created_at":"2013-12-16T20:44:32Z",
- "modified_by_client_uuid":"qr1hi-ozdt8-obw7foaks3qjyej",
- "modified_by_user_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
- "modified_at":"2013-12-16T20:44:32Z",
- "updated_at":"2013-12-16T20:44:33Z",
- "submit_id":null,
- "priority":null,
- "script":"hash",
- "script_parameters":{
- "input":"c1bad4b39ca5a924e481008009d94e32+210"
- },
- "script_version":"d9cd657b733d578ac0d2167dd75967aa4f22e0ac",
- "cancelled_at":null,
- "cancelled_by_client_uuid":null,
- "cancelled_by_user_uuid":null,
- "started_at":null,
- "finished_at":null,
- "output":null,
- "success":null,
- "running":null,
- "is_locked_by_uuid":null,
- "log":null,
- "runtime_constraints":{},
- "tasks_summary":{}
-}
-</code></pre>
-</notextile>
-
-The job is now queued and will start running as soon as it reaches the front of the queue. Fields to pay attention to include:
-
- * @"uuid"@ is the unique identifier for this specific job.
- * @"script_version"@ is the actual revision of the script used. This is useful if the version was described using the "repository:branch" format.
-
-h2. Monitor job progress
-
-Go to "*Recent jobs*":{{site.arvados_workbench_host}}/jobs in Workbench. Your job should be near the top of the table. This table refreshes automatically. When the job has completed successfully, it will show <span class="label label-success">finished</span> in the *Status* column.
-
-h2. Inspect the job output
-
-On the "Workbench Dashboard":{{site.arvados_workbench_host}}, look for the *Output* column of the *Recent jobs* table. Click on the link under *Output* for your job to go to the files page with the job output. The files page lists all the files that were output by the job. Click on the link under the *file* column to view a file, or click on the download button <span class="glyphicon glyphicon-download-alt"></span> to download the output file.
-
-On the command line, you can use @arv job get@ to access a JSON object describing the output:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv job get --uuid qr1hi-8i9sb-xxxxxxxxxxxxxxx</span>
-{
- "href":"https://qr1hi.arvadosapi.com/arvados/v1/jobs/qr1hi-8i9sb-1pm1t02dezhupss",
- "kind":"arvados#job",
- "etag":"1bk98tdj0qipjy0rvrj03ta5r",
- "uuid":"qr1hi-8i9sb-1pm1t02dezhupss",
- "owner_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
- "created_at":"2013-12-16T20:44:32Z",
- "modified_by_client_uuid":null,
- "modified_by_user_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
- "modified_at":"2013-12-16T20:44:55Z",
- "updated_at":"2013-12-16T20:44:55Z",
- "submit_id":null,
- "priority":null,
- "script":"hash",
- "script_parameters":{
- "input":"c1bad4b39ca5a924e481008009d94e32+210"
- },
- "script_version":"d9cd657b733d578ac0d2167dd75967aa4f22e0ac",
- "cancelled_at":null,
- "cancelled_by_client_uuid":null,
- "cancelled_by_user_uuid":null,
- "started_at":"2013-12-16T20:44:36Z",
- "finished_at":"2013-12-16T20:44:53Z",
- "output":"dd755dbc8d49a67f4fe7dc843e4f10a6+54",
- "success":true,
- "running":false,
- "is_locked_by_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
- "log":"2afdc6c8b67372ffd22d8ce89d35411f+91",
- "runtime_constraints":{},
- "tasks_summary":{
- "done":2,
- "running":0,
- "failed":0,
- "todo":0
- }
-}
-</code></pre>
-</notextile>
-
-* @"output"@ is the unique identifier for this specific job's output. This is a Keep collection. Because the output of Arvados jobs should be deterministic, the known expected output is <code>dd755dbc8d49a67f4fe7dc843e4f10a6+54</code>.
-
-Now you can list the files in the collection:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv keep ls dd755dbc8d49a67f4fe7dc843e4f10a6+54</span>
-./md5sum.txt
-</code></pre>
-</notextile>
-
-This collection consists of the @md5sum.txt@ file. Use @arv-get@ to show the contents of the @md5sum.txt@ file:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv-get dd755dbc8d49a67f4fe7dc843e4f10a6+54/md5sum.txt</span>
-44b8ae3fde7a8a88d2f7ebd237625b4f ./var-GS000016015-ASM.tsv.bz2
-</code></pre>
-</notextile>
-
-This MD5 hash matches the MD5 hash which we "computed earlier":{{site.baseurl}}/user/tutorials/tutorial-keep.html.
-
-h2. The job log
-
-When the job completes, you can access the job log. On the Workbench, visit "*Recent jobs*":{{site.arvados_workbench_host}}/jobs %(rarr)→% your job's UUID under the *uuid* column %(rarr)→% the collection link on the *log* row.
-
-On the command line, the Keep identifier listed in the @"log"@ field from @arv job get@ specifies a collection. You can list the files in the collection:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv keep ls xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+91</span>
-./qr1hi-8i9sb-xxxxxxxxxxxxxxx.log.txt
-</code></pre>
-</notextile>
-
-The log collection consists of one log file named with the job's UUID. You can access it using @arv-get@:
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv-get xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+91/qr1hi-8i9sb-xxxxxxxxxxxxxxx.log.txt</span>
-2013-12-16_20:44:35 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 check slurm allocation
-2013-12-16_20:44:35 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 node compute13 - 8 slots
-2013-12-16_20:44:36 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 start
-2013-12-16_20:44:36 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 Install revision d9cd657b733d578ac0d2167dd75967aa4f22e0ac
-2013-12-16_20:44:37 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 Clean-work-dir exited 0
-2013-12-16_20:44:37 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 Install exited 0
-2013-12-16_20:44:37 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 script hash
-2013-12-16_20:44:37 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 script_version d9cd657b733d578ac0d2167dd75967aa4f22e0ac
-2013-12-16_20:44:37 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 script_parameters {"input":"c1bad4b39ca5a924e481008009d94e32+210"}
-2013-12-16_20:44:37 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 runtime_constraints {"max_tasks_per_node":0}
-2013-12-16_20:44:37 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 start level 0
-2013-12-16_20:44:37 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 status: 0 done, 0 running, 1 todo
-2013-12-16_20:44:38 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 0 job_task qr1hi-ot0gb-23c1k3kwrf8da62
-2013-12-16_20:44:38 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 0 child 7681 started on compute13.1
-2013-12-16_20:44:38 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 status: 0 done, 1 running, 0 todo
-2013-12-16_20:44:39 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 0 child 7681 on compute13.1 exit 0 signal 0 success=true
-2013-12-16_20:44:39 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 0 success in 1 seconds
-2013-12-16_20:44:39 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 0 output
-2013-12-16_20:44:39 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 wait for last 0 children to finish
-2013-12-16_20:44:39 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 status: 1 done, 0 running, 1 todo
-2013-12-16_20:44:39 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 start level 1
-2013-12-16_20:44:39 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 status: 1 done, 0 running, 1 todo
-2013-12-16_20:44:39 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 1 job_task qr1hi-ot0gb-iwr0o3unqothg28
-2013-12-16_20:44:39 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 1 child 7716 started on compute13.1
-2013-12-16_20:44:39 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 status: 1 done, 1 running, 0 todo
-2013-12-16_20:44:52 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 1 child 7716 on compute13.1 exit 0 signal 0 success=true
-2013-12-16_20:44:52 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 1 success in 13 seconds
-2013-12-16_20:44:52 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 1 output dd755dbc8d49a67f4fe7dc843e4f10a6+54
-2013-12-16_20:44:52 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 wait for last 0 children to finish
-2013-12-16_20:44:52 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 status: 2 done, 0 running, 0 todo
-2013-12-16_20:44:52 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 release job allocation
-2013-12-16_20:44:52 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 Freeze not implemented
-2013-12-16_20:44:52 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 collate
-2013-12-16_20:44:53 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 output dd755dbc8d49a67f4fe7dc843e4f10a6+54+K at qr1hi
-2013-12-16_20:44:53 qr1hi-8i9sb-xxxxxxxxxxxxxxx 7575 finish
-</code></pre>
-</notextile>
diff --git a/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid b/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
index 2375e8b3d..180e08782 100644
--- a/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
@@ -39,7 +39,7 @@ A collection can be deleted using either the arv command line tool or the workbe
h3. Trashing a collection using arv command line tool
<pre>
-arv collection delete --uuid=qr1hi-4zz18-xxxxxxxxxxxxxxx
+arv collection delete --uuid=zzzzz-4zz18-xxxxxxxxxxxxxxx
</pre>
h3. Trashing a collection using workbench
@@ -61,7 +61,7 @@ arv collection list --include-trash=true --filters '[["is_trashed", "=", "true"]
You can then untrash a particular collection using arv using it's uuid.
<pre>
-arv collection untrash --uuid=qr1hi-4zz18-xxxxxxxxxxxxxxx
+arv collection untrash --uuid=zzzzz-4zz18-xxxxxxxxxxxxxxx
</pre>
h3. Un-trashing a collection using workbench
diff --git a/doc/user/tutorials/tutorial-keep-get.html.textile.liquid b/doc/user/tutorials/tutorial-keep-get.html.textile.liquid
index f206d302d..c73675bf1 100644
--- a/doc/user/tutorials/tutorial-keep-get.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep-get.html.textile.liquid
@@ -65,10 +65,10 @@ If your cluster is "configured to be part of a federation":{{site.baseurl}}/admi
If you request a collection by portable data hash, it will first search the home cluster, then search federated clusters.
-You may also request a collection by UUID. In this case, it will contact the cluster named in the UUID prefix (in this example, @qr1hi@).
+You may also request a collection by UUID. In this case, it will contact the cluster named in the UUID prefix (in this example, @zzzzz@).
<notextile>
-<pre><code>~$ <span class="userinput">arv-get qr1hi-4zz18-fw6dnjxtkvzdewt/ .</span>
+<pre><code>~$ <span class="userinput">arv-get zzzzz-4zz18-fw6dnjxtkvzdewt/ .</span>
</code></pre>
</notextile>
diff --git a/doc/user/tutorials/tutorial-keep.html.textile.liquid b/doc/user/tutorials/tutorial-keep.html.textile.liquid
index ec7086db9..5dd87c263 100644
--- a/doc/user/tutorials/tutorial-keep.html.textile.liquid
+++ b/doc/user/tutorials/tutorial-keep.html.textile.liquid
@@ -25,12 +25,12 @@ To upload a file to Keep using @arv-put@:
<pre><code>~$ <span class="userinput">arv-put var-GS000016015-ASM.tsv.bz2</span>
216M / 216M 100.0%
Collection saved as ...
-qr1hi-4zz18-xxxxxxxxxxxxxxx
+zzzzz-4zz18-xxxxxxxxxxxxxxx
</code></pre>
</notextile>
-The output value @qr1hi-4zz18-xxxxxxxxxxxxxxx@ is the uuid of the Arvados collection created.
+The output value @zzzzz-4zz18-xxxxxxxxxxxxxxx@ is the uuid of the Arvados collection created.
Note: The file used in this example is a freely available TSV file containing variant annotations from the "Personal Genome Project (PGP)":http://www.pgp-hms.org participant "hu599905":https://my.pgp-hms.org/profile/hu599905), downloadable "here":https://warehouse.pgp-hms.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2. Alternatively, you can replace @var-GS000016015-ASM.tsv.bz2@ with the name of any file you have locally, or you could get the TSV file by "downloading it from Keep.":{{site.baseurl}}/user/tutorials/tutorial-keep-get.html
@@ -44,7 +44,7 @@ Note: The file used in this example is a freely available TSV file containing va
~$ <span class="userinput">arv-put tmp</span>
0M / 0M 100.0%
Collection saved as ...
-qr1hi-4zz18-yyyyyyyyyyyyyyy
+zzzzz-4zz18-yyyyyyyyyyyyyyy
</code></pre>
</notextile>
diff --git a/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid b/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid
index dd537c46a..4e10eff01 100644
--- a/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid
+++ b/doc/user/tutorials/writing-cwl-workflow.html.textile.liquid
@@ -35,9 +35,9 @@ Use @--create-workflow@ to register a CWL workflow with Arvados. This enables y
<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --create-workflow bwa-mem.cwl</span>
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-07-01 12:21:01 arvados.arv-run[15796] INFO: Upload local files: "bwa-mem.cwl"
-2016-07-01 12:21:01 arvados.arv-run[15796] INFO: Uploaded to qr1hi-4zz18-7e0hedrmkuyoei3
-2016-07-01 12:21:01 arvados.cwl-runner[15796] INFO: Created template qr1hi-p5p6p-rjleou1dwr167v5
-qr1hi-p5p6p-rjleou1dwr167v5
+2016-07-01 12:21:01 arvados.arv-run[15796] INFO: Uploaded to zzzzz-4zz18-7e0hedrmkuyoei3
+2016-07-01 12:21:01 arvados.cwl-runner[15796] INFO: Created template zzzzz-p5p6p-rjleou1dwr167v5
+zzzzz-p5p6p-rjleou1dwr167v5
</code></pre>
</notextile>
@@ -47,9 +47,9 @@ You can provide a partial input file to set default values for the workflow inpu
<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --name "My workflow with defaults" --create-workflow bwa-mem.cwl bwa-mem-template.yml</span>
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-07-01 14:09:50 arvados.arv-run[3730] INFO: Upload local files: "bwa-mem.cwl"
-2016-07-01 14:09:50 arvados.arv-run[3730] INFO: Uploaded to qr1hi-4zz18-0f91qkovk4ml18o
-2016-07-01 14:09:50 arvados.cwl-runner[3730] INFO: Created template qr1hi-p5p6p-0deqe6nuuyqns2i
-qr1hi-p5p6p-zuniv58hn8d0qd8
+2016-07-01 14:09:50 arvados.arv-run[3730] INFO: Uploaded to zzzzz-4zz18-0f91qkovk4ml18o
+2016-07-01 14:09:50 arvados.cwl-runner[3730] INFO: Created template zzzzz-p5p6p-0deqe6nuuyqns2i
+zzzzz-p5p6p-zuniv58hn8d0qd8
</code></pre>
</notextile>
@@ -58,10 +58,10 @@ h3. Running registered workflows at the command line
You can run a registered workflow at the command line by its UUID:
<notextile>
-<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner qr1hi-p5p6p-zuniv58hn8d0qd8 --help</span>
+<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner zzzzz-p5p6p-zuniv58hn8d0qd8 --help</span>
/home/peter/work/scripts/venv/bin/arvados-cwl-runner 0d62edcb9d25bf4dcdb20d8872ea7b438e12fc59 1.0.20161209192028, arvados-python-client 0.1.20161212125425, cwltool 1.0.20161207161158
-Resolved 'qr1hi-p5p6p-zuniv58hn8d0qd8' to 'keep:655c6cd07550151b210961ed1d3852cf+57/bwa-mem.cwl'
-usage: qr1hi-p5p6p-zuniv58hn8d0qd8 [-h] [--PL PL] --group_id GROUP_ID
+Resolved 'zzzzz-p5p6p-zuniv58hn8d0qd8' to 'keep:655c6cd07550151b210961ed1d3852cf+57/bwa-mem.cwl'
+usage: zzzzz-p5p6p-zuniv58hn8d0qd8 [-h] [--PL PL] --group_id GROUP_ID
--read_p1 READ_P1 [--read_p2 READ_P2]
[--reference REFERENCE] --sample_id
SAMPLE_ID
@@ -164,10 +164,10 @@ You can make a workflow file directly executable (@cwl-runner@ should be an alia
<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">./bwa-mem.cwl bwa-mem-input.yml</span>
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to qr1hi-4zz18-h7ljh5u76760ww2
-2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Running
-2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Complete
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to zzzzz-4zz18-h7ljh5u76760ww2
+2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job zzzzz-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Running
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Complete
2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Overall process status is success
{
"aligned_sam": {
@@ -192,10 +192,10 @@ cwl:tool: <span class="userinput">bwa-mem.cwl</span>
<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">./bwa-mem-input.yml</span>
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to qr1hi-4zz18-h7ljh5u76760ww2
-2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Running
-2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Complete
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to zzzzz-4zz18-h7ljh5u76760ww2
+2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job zzzzz-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Running
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Complete
2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Overall process status is success
{
"aligned_sam": {
commit 635ea4273130253cba3e00f4b0dca7d7734a741d
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Jun 29 13:58:16 2020 -0400
15964: Add a line about storing Dockerfiles in source control
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/user/topics/arv-docker.html.textile.liquid b/doc/user/topics/arv-docker.html.textile.liquid
index f039a6c3e..d371400f5 100644
--- a/doc/user/topics/arv-docker.html.textile.liquid
+++ b/doc/user/topics/arv-docker.html.textile.liquid
@@ -45,6 +45,8 @@ The "RUN" command is executed inside the container and can be any shell command
You can also visit the "Docker tutorial":https://docs.docker.com/get-started/part2/ for more information and examples.
+You should add your Dockerfiles to the same source control repository as the Workflows that use them.
+
h3. Create a new image
We're now ready to create a new Docker image. Use @docker build@ to create a new image from the Dockerfile.
commit 337b055122485e669cc601c91a78f8fdd2625b3e
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Mon Jun 29 13:56:11 2020 -0400
15964: Rewrite docker tutorial to reflect current best practices
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/user/topics/arv-docker.html.textile.liquid b/doc/user/topics/arv-docker.html.textile.liquid
index e9e845026..f039a6c3e 100644
--- a/doc/user/topics/arv-docker.html.textile.liquid
+++ b/doc/user/topics/arv-docker.html.textile.liquid
@@ -1,7 +1,7 @@
---
layout: default
navsection: userguide
-title: "Customizing Crunch environment using Docker"
+title: "Working with Docker images"
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
@@ -9,130 +9,64 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-This page describes how to customize the runtime environment (e.g., the programs, libraries, and other dependencies needed to run a job) that a crunch script will be run in using "Docker.":https://www.docker.com/ Docker is a tool for building and running containers that isolate applications from other applications running on the same node. For detailed information about Docker, see the "Docker User Guide.":https://docs.docker.com/userguide/
+This page describes how to set up the runtime environment (e.g., the programs, libraries, and other dependencies needed to run a job) that a workflow step will be run in using "Docker.":https://www.docker.com/ Docker is a tool for building and running containers that isolate applications from other applications running on the same node. For detailed information about Docker, see the "Docker User Guide.":https://docs.docker.com/userguide/
-This page will demonstrate how to:
+This page will describe:
-# Fetch the arvados/jobs Docker image
-# Manually install additional software into the container
-# Create a new custom image
-# Upload that image to Arvados for use by Crunch jobs
-# Share your image with others
+# Create a custom image using a Dockerfile
+# Uploading an image to Arvados
+# Sources of pre-built bioinformatics Docker images
{% include 'tutorial_expectations_workstation' %}
You also need ensure that "Docker is installed,":https://docs.docker.com/installation/ the Docker daemon is running, and you have permission to access Docker. You can test this by running @docker version at . If you receive a permission denied error, your user account may need to be added to the @docker@ group. If you have root access, you can add yourself to the @docker@ group using @$ sudo addgroup $USER docker@ then log out and log back in again; otherwise consult your local sysadmin.
-h2. Fetch a starting image
+h2. Create a custom image using a Dockerfile
-The easiest way to begin is to start from the "arvados/jobs" image which already has the Arvados SDK installed along with other configuration required for use with Crunch.
+This example shows how to create a Docker image and add the R package.
-Download the latest "arvados/jobs" image from the Docker registry:
+First, create new directory called @docker-example@, in that directory create a file called @Dockerfile at .
<notextile>
-<pre><code>$ <span class="userinput">docker pull arvados/jobs:latest</span>
-Pulling repository arvados/jobs
-3132168f2acb: Download complete
-a42b7f2c59b6: Download complete
-e5afdf26a7ae: Download complete
-5cae48636278: Download complete
-7a4f91b70558: Download complete
-a04a275c1fd6: Download complete
-c433ff206a22: Download complete
-b2e539b45f96: Download complete
-073b2581c6be: Download complete
-593915af19dc: Download complete
-32260b35005e: Download complete
-6e5b860c1cde: Download complete
-95f0bfb43d4d: Download complete
-c7fd77eedb96: Download complete
-0d7685aafd00: Download complete
+<pre><code>
+$ mkdir docker-example-r-base
+$ cd docker-example-r-base
</code></pre>
</notextile>
-h2. Install new packages
-
-Next, enter the container using @docker run@, providing the arvados/jobs image and the program you want to run (in this case the bash shell).
-
<notextile>
-<pre><code>$ <span class="userinput">docker run --interactive --tty --user root arvados/jobs /bin/bash</span>
-root at fbf1d0f529d5:/#
+<pre><code>
+FROM ubuntu:bionic
+RUN apt-get update && apt-get -yq --no-install-recommends install r-base-core
</code></pre>
</notextile>
-Next, update the package list using @apt-get update at .
+The "RUN" command is executed inside the container and can be any shell command line. You are not limited to installing Debian packages. You may compile programs or libraries from source and install them, edit systemwide configuration files, use other package managers such as @pip@ or @gem@, and perform any other customization necessary to run your program.
-<notextile>
-<pre><code>root at fbf1d0f529d5:/# apt-get update
-Get:2 http://apt.arvados.org stretch-dev InRelease [3260 B]
-Get:1 http://security-cdn.debian.org/debian-security stretch/updates InRelease [94.3 kB]
-Ign:3 http://cdn-fastly.deb.debian.org/debian stretch InRelease
-Get:4 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB]
-Get:5 http://apt.arvados.org stretch-dev/main amd64 Packages [208 kB]
-Get:6 http://cdn-fastly.deb.debian.org/debian stretch Release [118 kB]
-Get:7 http://security-cdn.debian.org/debian-security stretch/updates/main amd64 Packages [499 kB]
-Get:8 http://cdn-fastly.deb.debian.org/debian stretch Release.gpg [2434 B]
-Get:9 http://cdn-fastly.deb.debian.org/debian stretch-updates/main amd64 Packages.diff/Index [10.6 kB]
-Get:10 http://cdn-fastly.deb.debian.org/debian stretch-updates/main amd64 Packages 2019-07-08-0821.07.pdiff [445 B]
-Get:10 http://cdn-fastly.deb.debian.org/debian stretch-updates/main amd64 Packages 2019-07-08-0821.07.pdiff [445 B]
-Fetched 1026 kB in 0s (1384 kB/s)
-Reading package lists... Done
-</code></pre>
-</notextile>
+You can also visit the "Docker tutorial":https://docs.docker.com/get-started/part2/ for more information and examples.
+
+h3. Create a new image
-In this example, we will install the "R" statistical language Debian package "r-base-core". Use @apt-get install@:
+We're now ready to create a new Docker image. Use @docker build@ to create a new image from the Dockerfile.
<notextile>
-<pre><code>root at fbf1d0f529d5:/# <span class="userinput">apt-get install r-base-core</span>
-Reading package lists... Done
-Building dependency tree
-Reading state information... Done
-The following additional packages will be installed:
-[...]
-done.
+<pre><code>
+docker-example-r-base$ docker build -t docker-example-r-base .
</code></pre>
</notextile>
+h3. Verify image
+
Now we can verify that "R" is installed:
<notextile>
-<pre><code>root at fbf1d0f529d5:/# <span class="userinput">R</span>
+<pre><code>$ docker run -ti docker-example-r-base
+root at 57ec8f8b2663:/# R
-R version 3.3.3 (2017-03-06) -- "Another Canoe"
-Copyright (C) 2017 The R Foundation for Statistical Computing
+R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
+Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
-R is free software and comes with ABSOLUTELY NO WARRANTY.
-You are welcome to redistribute it under certain conditions.
-Type 'license()' or 'licence()' for distribution details.
-
-R is a collaborative project with many contributors.
-Type 'contributors()' for more information and
-'citation()' on how to cite R or R packages in publications.
-
-Type 'demo()' for some demos, 'help()' for on-line help, or
-'help.start()' for an HTML browser interface to help.
-Type 'q()' to quit R.
-
->
-</code></pre>
-</notextile>
-
-Note that you are not limited to installing Debian packages. You may compile programs or libraries from source and install them, edit systemwide configuration files, use other package managers such as @pip@ or @gem@, and perform any other customization necessary to run your program.
-
-h2. Create a new image
-
-We're now ready to create a new Docker image. First, quit the container, then use @docker commit@ to create a new image from the stopped container. The container id can be found in the default hostname of the container displayed in the prompt, in this case @fbf1d0f529d5@:
-
-<notextile>
-<pre><code>root at fbf1d0f529d5:/# <span class="userinput">exit</span>
-$ <span class="userinput">docker commit fbf1d0f529d5 arvados/jobs-with-r</span>
-sha256:2818853ff9f9af5d7f77979803baac9c4710790ad2b84c1a754b02728fdff205
-$ <span class="userinput">docker images</span>
-$ docker images |head
-REPOSITORY TAG IMAGE ID CREATED SIZE
-arvados/jobs-with-r latest 2818853ff9f9 9 seconds ago 703.1 MB
-arvados/jobs latest 12b9f859d48c 4 days ago 362 MB
</code></pre>
</notextile>
@@ -141,13 +75,16 @@ h2. Upload your image
Finally, we are ready to upload the new Docker image to Arvados. Use @arv-keepdocker@ with the image repository name to upload the image. Without arguments, @arv-keepdocker@ will print out the list of Docker images in Arvados that are available to you.
<notextile>
-<pre><code>$ <span class="userinput">arv-keepdocker arvados/jobs-with-r</span>
-703M / 703M 100.0%
-Collection saved as 'Docker image arvados/jobs-with-r:latest 2818853ff9f9'
-qr1hi-4zz18-abcdefghijklmno
-$ <span class="userinput">arv-keepdocker</span>
+<pre><code>$ <span class="userinput">arv-keepdocker docker-example-r-base</span>
+2020-06-29 13:48:19 arvados.arv_put[769] INFO: Creating new cache file at /home/peter/.cache/arvados/arv-put/39ddb51ebf6c5fcb3d713b5969466967
+206M / 206M 100.0% 2020-06-29 13:48:21 arvados.arv_put[769] INFO:
+
+2020-06-29 13:48:21 arvados.arv_put[769] INFO: Collection saved as 'Docker image docker-example-r-base:latest sha256:edd10'
+x20vs-4zz18-0tayximqcyb6uf8
+
+$ <span class="userinput">arv-keepdocker images</span>
REPOSITORY TAG IMAGE ID COLLECTION CREATED
-arvados/jobs-with-r latest 2818853ff9f9 qr1hi-4zz18-abcdefghijklmno Tue Jan 17 20:35:53 2017
+docker-example-r-base latest sha256:edd10 x20vs-4zz18-0tayximqcyb6uf8 Mon Jun 29 17:46:16 2020
</code></pre>
</notextile>
@@ -156,14 +93,24 @@ You are now able to specify the runtime environment for your program using @Dock
<pre>
hints:
DockerRequirement:
- dockerPull: arvados/jobs-with-r
+ dockerPull: docker-example-r-base
</pre>
-h2. Share Docker images
+h3. Uploading Docker images to a shared project
-Docker images are subject to normal Arvados permissions. If wish to share your Docker image with others (or wish to share a pipeline template that uses your Docker image) you will need to use @arv-keepdocker@ with the @--project-uuid@ option to upload the image to a shared project.
+Docker images are subject to normal Arvados permissions. If wish to share your Docker image with others you should use @arv-keepdocker@ with the @--project-uuid@ option to add the image to a shared project and ensure that metadata is set correctly.
<notextile>
-<pre><code>$ <span class="userinput">arv-keepdocker arvados/jobs-with-r --project-uuid qr1hi-j7d0g-xxxxxxxxxxxxxxx</span>
+<pre><code>$ <span class="userinput">arv-keepdocker docker-example-r-base --project-uuid qr1hi-j7d0g-xxxxxxxxxxxxxxx</span>
</code></pre>
</notextile>
+
+h2. Sources of pre-built images
+
+In addition to creating your own contianers, there are a number of resources where you can find bioinformatics tools already wrapped in container images:
+
+"BioContainers":https://biocontainers.pro/
+
+"Dockstore":https://dockstore.org/
+
+"Docker Hub":https://hub.docker.com/
commit ddaa9596ebf94aa7b6ee720e06ceccd2d7dc43fc
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Jun 26 17:41:48 2020 -0400
15964: Removing mentions of qr1hi from docs. Cleaning up user guide.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/_config.yml b/doc/_config.yml
index bbab3f307..8d31f7e4b 100644
--- a/doc/_config.yml
+++ b/doc/_config.yml
@@ -28,26 +28,25 @@ navbar:
- user/getting_started/workbench.html.textile.liquid
- user/tutorials/tutorial-workflow-workbench.html.textile.liquid
- user/composer/composer.html.textile.liquid
+ - Working at the Command Line:
+ - user/getting_started/setup-cli.html.textile.liquid
+ - user/reference/api-tokens.html.textile.liquid
+ - user/getting_started/check-environment.html.textile.liquid
- Access an Arvados virtual machine:
- user/getting_started/vm-login-with-webshell.html.textile.liquid
- user/getting_started/ssh-access-unix.html.textile.liquid
- user/getting_started/ssh-access-windows.html.textile.liquid
- - user/getting_started/check-environment.html.textile.liquid
- - user/reference/api-tokens.html.textile.liquid
- Working with data sets:
- user/tutorials/tutorial-keep.html.textile.liquid
- user/tutorials/tutorial-keep-get.html.textile.liquid
- user/tutorials/tutorial-keep-mount-gnu-linux.html.textile.liquid
- user/tutorials/tutorial-keep-mount-os-x.html.textile.liquid
- user/tutorials/tutorial-keep-mount-windows.html.textile.liquid
- - user/topics/keep.html.textile.liquid
- user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
- user/topics/arv-copy.html.textile.liquid
- - user/topics/storage-classes.html.textile.liquid
- user/topics/collection-versioning.html.textile.liquid
- - Working with git repositories:
- - user/tutorials/add-new-repository.html.textile.liquid
- - user/tutorials/git-arvados-guide.html.textile.liquid
+ - user/topics/storage-classes.html.textile.liquid
+ - user/topics/keep.html.textile.liquid
- Running workflows at the command line:
- user/cwl/cwl-runner.html.textile.liquid
- user/cwl/cwl-run-options.html.textile.liquid
@@ -59,6 +58,9 @@ navbar:
- user/cwl/cwl-extensions.html.textile.liquid
- user/cwl/cwl-versions.html.textile.liquid
- user/topics/arv-docker.html.textile.liquid
+ - Working with git repositories:
+ - user/tutorials/add-new-repository.html.textile.liquid
+ - user/tutorials/git-arvados-guide.html.textile.liquid
- Reference:
- user/topics/link-accounts.html.textile.liquid
- user/reference/cookbook.html.textile.liquid
diff --git a/doc/_includes/_ssh_addkey.liquid b/doc/_includes/_ssh_addkey.liquid
index 7a8a992b6..de0da6a76 100644
--- a/doc/_includes/_ssh_addkey.liquid
+++ b/doc/_includes/_ssh_addkey.liquid
@@ -18,6 +18,10 @@ Paste your public key into the text area labeled *Public Key*, and click on the
h1(#login). Using SSH to log into an Arvados VM
-To see a list of virtual machines that you have access to and determine the name and login information, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu and click on the menu item *Virtual machines* to go to the Virtual machines page. This page lists the virtual machines you can access. The *Host name* column lists the name of each available VM. The *Login name* column will have a list of comma separated values of the form @you at . In this guide the hostname will be *_shell_* and the login will be *_you_*. Replace these with your hostname and login name as appropriate.
+To see a list of virtual machines that you have access to, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, then click on the menu item *Virtual machines* to go to the Virtual machines page.
+This page lists the virtual machines you can access. The *Host name* column lists the name of each available VM. The *Login name* column lists your login name on that VM. The *Command line* column provides a sample @ssh@ command line.
+At the bottom of the page there may be additional instructions for connecting your specific Arvados instance. If so, follow your site-specific instructions. If there are no site-specific instructions, you can probably connect directly with @ssh at .
+
+The following are generic instructions. In the examples the login will be *_you_* and the hostname will be *_shell.ClusterID.example.com_* and . Replace these with your login name and hostname as appropriate.
diff --git a/doc/_includes/_tutorial_cluster_name.liquid b/doc/_includes/_tutorial_cluster_name.liquid
index 3833abbf2..1de5fa2fa 100644
--- a/doc/_includes/_tutorial_cluster_name.liquid
+++ b/doc/_includes/_tutorial_cluster_name.liquid
@@ -5,5 +5,5 @@ SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
{% include 'notebox_begin' %}
-This tutorial assumes you are using the playground Arvados instance, @qr1hi at . If you are using a different instance, replace @qr1hi@ with your instance. See "Accessing Arvados Workbench":{{site.baseurl}}/user/getting_started/workbench.html for more details.
+This tutorial assumes you are using the playground Arvados instance, @pirca at . If you are using a different instance, replace @pirca@ with your instance. See "Accessing Arvados Workbench":{{site.baseurl}}/user/getting_started/workbench.html for more details.
{% include 'notebox_end' %}
diff --git a/doc/_includes/_tutorial_expectations.liquid b/doc/_includes/_tutorial_expectations.liquid
index 6c4fbeb1f..09b18f0d4 100644
--- a/doc/_includes/_tutorial_expectations.liquid
+++ b/doc/_includes/_tutorial_expectations.liquid
@@ -5,5 +5,5 @@ SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
{% include 'notebox_begin' %}
-This tutorial assumes that you are logged into an Arvados VM instance (instructions for "Webshell":{{site.baseurl}}/user/getting_started/vm-login-with-webshell.html or "Unix":{{site.baseurl}}/user/getting_started/ssh-access-unix.html#login or "Windows":{{site.baseurl}}/user/getting_started/ssh-access-windows.html#login) or you have installed the Arvados "FUSE Driver":{{site.baseurl}}/sdk/python/arvados-fuse.html and "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html on your workstation and have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html
+This tutorial assumes that you have access to the "Arvados command line tools":/user/getting_started/setup-cli.html and have set the "API token":{{site.baseurl}}/user/reference/api-tokens.html and confirmed a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html .
{% include 'notebox_end' %}
diff --git a/doc/_includes/_tutorial_hello_cwl.liquid b/doc/_includes/_tutorial_hello_cwl.liquid
new file mode 100644
index 000000000..ae1ec80ab
--- /dev/null
+++ b/doc/_includes/_tutorial_hello_cwl.liquid
@@ -0,0 +1,11 @@
+#!/usr/bin/env cwl-runner
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+cwlVersion: v1.0
+class: CommandLineTool
+inputs: []
+outputs: []
+arguments: ["echo", "hello world!"]
diff --git a/doc/architecture/federation.html.textile.liquid b/doc/architecture/federation.html.textile.liquid
index e2b80de70..2ec2c93fb 100644
--- a/doc/architecture/federation.html.textile.liquid
+++ b/doc/architecture/federation.html.textile.liquid
@@ -26,7 +26,7 @@ Clusters are identified by a five-digit alphanumeric id (numbers and lowercase l
Cluster identifiers are mapped API server hosts one of two ways:
-* Through DNS resolution, under the @arvadosapi.com@ domain. For example, the API server for the cluster @qr1hi@ can be found at @qr1hi.arvadosapi.com at . To register a cluster id for free under @arvadosapi.com@, contact "info at curii.com":mailto:info at curii.com
+* Through DNS resolution, under the @arvadosapi.com@ domain. For example, the API server for the cluster @pirca@ can be found at @pirca.arvadosapi.com at . To register a cluster id for free under @arvadosapi.com@, contact "info at curii.com":mailto:info at curii.com
* Through explicit configuration:
The @RemoteClusters@ section of @/etc/arvados/config.yml@ (for arvados-controller)
diff --git a/doc/examples/pipeline_templates/gatk-exome-fq-snp.json b/doc/examples/pipeline_templates/gatk-exome-fq-snp.json
deleted file mode 100644
index 481dda38c..000000000
--- a/doc/examples/pipeline_templates/gatk-exome-fq-snp.json
+++ /dev/null
@@ -1,175 +0,0 @@
-{
- "name":"GATK / exome PE fastq to snp",
- "components":{
- "extract-reference":{
- "repository":"arvados",
- "script_version":"e820bd1c6890f93ea1a84ffd5730bbf0e3d8e153",
- "script":"file-select",
- "script_parameters":{
- "names":[
- "human_g1k_v37.fasta.gz",
- "human_g1k_v37.fasta.fai.gz",
- "human_g1k_v37.dict.gz"
- ],
- "input":"d237a90bae3870b3b033aea1e99de4a9+10820+K at qr1hi"
- },
- "output_name":false
- },
- "bwa-index":{
- "repository":"arvados",
- "script_version":"e820bd1c6890f93ea1a84ffd5730bbf0e3d8e153",
- "script":"bwa-index",
- "script_parameters":{
- "input":{
- "output_of":"extract-reference"
- },
- "bwa_tbz":{
- "value":"8b6e2c4916133e1d859c9e812861ce13+70",
- "required":true
- }
- },
- "output_name":false
- },
- "bwa-aln":{
- "repository":"arvados",
- "script_version":"e820bd1c6890f93ea1a84ffd5730bbf0e3d8e153",
- "script":"bwa-aln",
- "script_parameters":{
- "input":{
- "dataclass":"Collection",
- "required":"true"
- },
- "reference_index":{
- "output_of":"bwa-index"
- },
- "samtools_tgz":{
- "value":"c777e23cf13e5d5906abfdc08d84bfdb+74",
- "required":true
- },
- "bwa_tbz":{
- "value":"8b6e2c4916133e1d859c9e812861ce13+70",
- "required":true
- }
- },
- "runtime_constraints":{
- "max_tasks_per_node":1
- },
- "output_name":false
- },
- "picard-gatk2-prep":{
- "repository":"arvados",
- "script_version":"e820bd1c6890f93ea1a84ffd5730bbf0e3d8e153",
- "script":"picard-gatk2-prep",
- "script_parameters":{
- "input":{
- "output_of":"bwa-aln"
- },
- "reference":{
- "output_of":"extract-reference"
- },
- "picard_zip":{
- "value":"687f74675c6a0e925dec619cc2bec25f+77",
- "required":true
- }
- },
- "runtime_constraints":{
- "max_tasks_per_node":1
- },
- "output_name":false
- },
- "GATK2-realign":{
- "repository":"arvados",
- "script_version":"e820bd1c6890f93ea1a84ffd5730bbf0e3d8e153",
- "script":"GATK2-realign",
- "script_parameters":{
- "input":{
- "output_of":"picard-gatk2-prep"
- },
- "gatk_bundle":{
- "value":"d237a90bae3870b3b033aea1e99de4a9+10820+K at qr1hi",
- "required":true
- },
- "picard_zip":{
- "value":"687f74675c6a0e925dec619cc2bec25f+77",
- "required":true
- },
- "gatk_tbz":{
- "value":"7e0a277d6d2353678a11f56bab3b13f2+87",
- "required":true
- },
- "regions":{
- "value":"13b53dbe1ec032dfc495fd974aa5dd4a+87/S02972011_Covered_sort_merged.bed"
- },
- "region_padding":{
- "value":10
- }
- },
- "runtime_constraints":{
- "max_tasks_per_node":2
- },
- "output_name":false
- },
- "GATK2-bqsr":{
- "repository":"arvados",
- "script_version":"e820bd1c6890f93ea1a84ffd5730bbf0e3d8e153",
- "script":"GATK2-bqsr",
- "script_parameters":{
- "input":{
- "output_of":"GATK2-realign"
- },
- "gatk_bundle":{
- "value":"d237a90bae3870b3b033aea1e99de4a9+10820+K at qr1hi",
- "required":true
- },
- "picard_zip":{
- "value":"687f74675c6a0e925dec619cc2bec25f+77",
- "required":true
- },
- "gatk_tbz":{
- "value":"7e0a277d6d2353678a11f56bab3b13f2+87",
- "required":true
- }
- },
- "output_name":false
- },
- "GATK2-merge-call":{
- "repository":"arvados",
- "script_version":"e820bd1c6890f93ea1a84ffd5730bbf0e3d8e153",
- "script":"GATK2-merge-call",
- "script_parameters":{
- "input":{
- "output_of":"GATK2-bqsr"
- },
- "gatk_bundle":{
- "value":"d237a90bae3870b3b033aea1e99de4a9+10820+K at qr1hi",
- "required":true
- },
- "picard_zip":{
- "value":"687f74675c6a0e925dec619cc2bec25f+77",
- "required":true
- },
- "gatk_tbz":{
- "value":"7e0a277d6d2353678a11f56bab3b13f2+87",
- "required":true
- },
- "regions":{
- "value":"13b53dbe1ec032dfc495fd974aa5dd4a+87/S02972011_Covered_sort_merged.bed"
- },
- "region_padding":{
- "value":10
- },
- "GATK2_UnifiedGenotyper_args":{
- "default":[
- "-stand_call_conf",
- "30.0",
- "-stand_emit_conf",
- "30.0",
- "-dcov",
- "200"
- ]
- }
- },
- "output_name":"Variant calls from UnifiedGenotyper"
- }
- }
-}
diff --git a/doc/examples/pipeline_templates/rtg-fq-snp.json b/doc/examples/pipeline_templates/rtg-fq-snp.json
deleted file mode 100644
index c951c4c43..000000000
--- a/doc/examples/pipeline_templates/rtg-fq-snp.json
+++ /dev/null
@@ -1,76 +0,0 @@
-{
- "name":"Real Time Genomics / PE fastq to snp",
- "components":{
- "extract_reference":{
- "script":"file-select",
- "script_parameters":{
- "names":[
- "human_g1k_v37.fasta.gz"
- ],
- "input":"d237a90bae3870b3b033aea1e99de4a9+10820+K at qr1hi"
- },
- "script_version":"4c1f8cd1431ece2ef11c130d48bb2edfd2f00ec2"
- },
- "reformat_reference":{
- "script_version":"4c1f8cd1431ece2ef11c130d48bb2edfd2f00ec2",
- "script":"rtg-fasta2sdf",
- "script_parameters":{
- "input":{
- "output_of":"extract_reference"
- },
- "rtg_binary_zip":"5d33618193f763b7dc3a3fdfa11d452e+95+K at qr1hi",
- "rtg_license":{
- "optional":false
- }
- }
- },
- "reformat_reads":{
- "script_version":"4c1f8cd1431ece2ef11c130d48bb2edfd2f00ec2",
- "script":"rtg-fastq2sdf",
- "script_parameters":{
- "input":{
- "optional":false
- },
- "rtg_binary_zip":"5d33618193f763b7dc3a3fdfa11d452e+95+K at qr1hi",
- "rtg_license":{
- "optional":false
- }
- }
- },
- "map_reads":{
- "script_version":"4c1f8cd1431ece2ef11c130d48bb2edfd2f00ec2",
- "script":"rtg-map",
- "script_parameters":{
- "input":{
- "output_of":"reformat_reads"
- },
- "reference":{
- "output_of":"reformat_reference"
- },
- "rtg_binary_zip":"5d33618193f763b7dc3a3fdfa11d452e+95+K at qr1hi",
- "rtg_license":{
- "optional":false
- }
- },
- "runtime_constraints":{
- "max_tasks_per_node":1
- }
- },
- "report_snp":{
- "script_version":"4c1f8cd1431ece2ef11c130d48bb2edfd2f00ec2",
- "script":"rtg-snp",
- "script_parameters":{
- "input":{
- "output_of":"map_reads"
- },
- "reference":{
- "output_of":"reformat_reference"
- },
- "rtg_binary_zip":"5d33618193f763b7dc3a3fdfa11d452e+95+K at qr1hi",
- "rtg_license":{
- "optional":false
- }
- }
- }
- }
-}
diff --git a/doc/examples/ruby/list-active-nodes.rb b/doc/examples/ruby/list-active-nodes.rb
deleted file mode 100755
index a3eb20540..000000000
--- a/doc/examples/ruby/list-active-nodes.rb
+++ /dev/null
@@ -1,16 +0,0 @@
-#!/usr/bin/env ruby
-# Copyright (C) The Arvados Authors. All rights reserved.
-#
-# SPDX-License-Identifier: CC-BY-SA-3.0
-
-abort 'Error: Ruby >= 1.9.3 required.' if RUBY_VERSION < '1.9.3'
-
-require 'arvados'
-
-arv = Arvados.new(api_version: 'v1')
-arv.node.list[:items].each do |node|
- if node[:crunch_worker_state] != 'down'
- ping_age = (Time.now - Time.parse(node[:last_ping_at])).to_i rescue -1
- puts "#{node[:uuid]} #{node[:crunch_worker_state]} #{ping_age}"
- end
-end
diff --git a/doc/sdk/java-v2/example.html.textile.liquid b/doc/sdk/java-v2/example.html.textile.liquid
index e73f968c8..8d2fc2f4a 100644
--- a/doc/sdk/java-v2/example.html.textile.liquid
+++ b/doc/sdk/java-v2/example.html.textile.liquid
@@ -28,7 +28,7 @@ public class CollectionExample {
public static void main(String[] argv) {
ConfigProvider conf = ExternalConfigProvider.builder().
apiProtocol("https").
- apiHost("qr1hi.arvadosapi.com").
+ apiHost("zzzzz.arvadosapi.com").
apiPort(443).
apiToken("...").
build();
diff --git a/doc/sdk/python/cookbook.html.textile.liquid b/doc/sdk/python/cookbook.html.textile.liquid
index bd7f64b33..ff8b8052e 100644
--- a/doc/sdk/python/cookbook.html.textile.liquid
+++ b/doc/sdk/python/cookbook.html.textile.liquid
@@ -47,7 +47,7 @@ h2. Get input of a CWL workflow
{% codeblock as python %}
import arvados
api = arvados.api()
-container_request_uuid="qr1hi-xvhdp-zzzzzzzzzzzzzzz"
+container_request_uuid="zzzzz-xvhdp-zzzzzzzzzzzzzzz"
container_request = api.container_requests().get(uuid=container_request_uuid).execute()
print(container_request["mounts"]["/var/lib/cwl/cwl.input.json"])
{% endcodeblock %}
@@ -58,7 +58,7 @@ h2. Get output of a CWL workflow
import arvados
import arvados.collection
api = arvados.api()
-container_request_uuid="qr1hi-xvhdp-zzzzzzzzzzzzzzz"
+container_request_uuid="zzzzz-xvhdp-zzzzzzzzzzzzzzz"
container_request = api.container_requests().get(uuid=container_request_uuid).execute()
collection = arvados.collection.CollectionReader(container_request["output_uuid"])
print(collection.open("cwl.output.json").read())
@@ -89,7 +89,7 @@ def get_cr_state(cr_uuid):
elif c['runtime_status'].get('warning', None):
return 'Warning'
return c['state']
-container_request_uuid = 'qr1hi-xvhdp-zzzzzzzzzzzzzzz'
+container_request_uuid = 'zzzzz-xvhdp-zzzzzzzzzzzzzzz'
print(get_cr_state(container_request_uuid))
{% endcodeblock %}
@@ -98,7 +98,7 @@ h2. List input of child requests
{% codeblock as python %}
import arvados
api = arvados.api()
-parent_request_uuid = "qr1hi-xvhdp-zzzzzzzzzzzzzzz"
+parent_request_uuid = "zzzzz-xvhdp-zzzzzzzzzzzzzzz"
namefilter = "bwa%" # the "like" filter uses SQL pattern match syntax
container_request = api.container_requests().get(uuid=parent_request_uuid).execute()
parent_container_uuid = container_request["container_uuid"]
@@ -117,7 +117,7 @@ h2. List output of child requests
{% codeblock as python %}
import arvados
api = arvados.api()
-parent_request_uuid = "qr1hi-xvhdp-zzzzzzzzzzzzzzz"
+parent_request_uuid = "zzzzz-xvhdp-zzzzzzzzzzzzzzz"
namefilter = "bwa%" # the "like" filter uses SQL pattern match syntax
container_request = api.container_requests().get(uuid=parent_request_uuid).execute()
parent_container_uuid = container_request["container_uuid"]
@@ -136,7 +136,7 @@ h2. List failed child requests
{% codeblock as python %}
import arvados
api = arvados.api()
-parent_request_uuid = "qr1hi-xvhdp-zzzzzzzzzzzzzzz"
+parent_request_uuid = "zzzzz-xvhdp-zzzzzzzzzzzzzzz"
container_request = api.container_requests().get(uuid=parent_request_uuid).execute()
parent_container_uuid = container_request["container_uuid"]
child_requests = api.container_requests().list(filters=[
@@ -155,7 +155,7 @@ h2. Get log of a child request
import arvados
import arvados.collection
api = arvados.api()
-container_request_uuid = "qr1hi-xvhdp-zzzzzzzzzzzzzzz"
+container_request_uuid = "zzzzz-xvhdp-zzzzzzzzzzzzzzz"
container_request = api.container_requests().get(uuid=container_request_uuid).execute()
collection = arvados.collection.CollectionReader(container_request["log_uuid"])
for c in collection:
@@ -168,7 +168,7 @@ h2(#sharing_link). Create a collection sharing link
import arvados
api = arvados.api()
download="https://your.download.server"
-collection_uuid="qr1hi-4zz18-zzzzzzzzzzzzzzz"
+collection_uuid="zzzzz-4zz18-zzzzzzzzzzzzzzz"
token = api.api_client_authorizations().create(body={"api_client_authorization":{"scopes": [
"GET /arvados/v1/collections/%s" % collection_uuid,
"GET /arvados/v1/collections/%s/" % collection_uuid,
@@ -184,8 +184,8 @@ Note, if two collections have files of the same name, the contents will be conca
import arvados
import arvados.collection
api = arvados.api()
-project_uuid = "qr1hi-tpzed-zzzzzzzzzzzzzzz"
-collection_uuids = ["qr1hi-4zz18-aaaaaaaaaaaaaaa", "qr1hi-4zz18-bbbbbbbbbbbbbbb"]
+project_uuid = "zzzzz-tpzed-zzzzzzzzzzzzzzz"
+collection_uuids = ["zzzzz-4zz18-aaaaaaaaaaaaaaa", "zzzzz-4zz18-bbbbbbbbbbbbbbb"]
combined_manifest = ""
for u in collection_uuids:
c = api.collections().get(uuid=u).execute()
@@ -200,7 +200,7 @@ h2. Upload a file into a new collection
import arvados
import arvados.collection
-project_uuid = "qr1hi-j7d0g-zzzzzzzzzzzzzzz"
+project_uuid = "zzzzz-j7d0g-zzzzzzzzzzzzzzz"
collection_name = "My collection"
filename = "file1.txt"
@@ -222,7 +222,7 @@ h2. Download a file from a collection
import arvados
import arvados.collection
-collection_uuid = "qr1hi-4zz18-zzzzzzzzzzzzzzz"
+collection_uuid = "zzzzz-4zz18-zzzzzzzzzzzzzzz"
filename = "file1.txt"
api = arvados.api()
diff --git a/doc/sdk/ruby/example.html.textile.liquid b/doc/sdk/ruby/example.html.textile.liquid
index b8c0dcbb8..f2ea1c09d 100644
--- a/doc/sdk/ruby/example.html.textile.liquid
+++ b/doc/sdk/ruby/example.html.textile.liquid
@@ -55,7 +55,7 @@ first_repo = repos[:items][0]
puts "UUID of first repo returned is #{first_repo[:uuid]}"</code>
{% endcodeblock %}
-UUID of first repo returned is qr1hi-s0uqq-b1bnybpx3u5temz
+UUID of first repo returned is zzzzz-s0uqq-b1bnybpx3u5temz
h2. update
diff --git a/doc/user/cwl/cwl-runner.html.textile.liquid b/doc/user/cwl/cwl-runner.html.textile.liquid
index 2be803b52..8af1c574f 100644
--- a/doc/user/cwl/cwl-runner.html.textile.liquid
+++ b/doc/user/cwl/cwl-runner.html.textile.liquid
@@ -13,12 +13,6 @@ SPDX-License-Identifier: CC-BY-SA-3.0
{% include 'tutorial_expectations' %}
-{% include 'notebox_begin' %}
-
-By default, the @arvados-cwl-runner@ is installed on Arvados shell nodes. If you want to submit jobs from somewhere else, such as your workstation, you may install "arvados-cwl-runner.":#setup
-
-{% include 'notebox_end' %}
-
This tutorial will demonstrate how to submit a workflow at the command line using @arvados-cwl-runner at .
h2. Running arvados-cwl-runner
diff --git a/doc/user/getting_started/ssh-access-unix.html.textile.liquid b/doc/user/getting_started/ssh-access-unix.html.textile.liquid
index 284d0a1f0..c2d5da29b 100644
--- a/doc/user/getting_started/ssh-access-unix.html.textile.liquid
+++ b/doc/user/getting_started/ssh-access-unix.html.textile.liquid
@@ -49,7 +49,7 @@ ssh-rsa AAAAB3NzaC1ycEDoNotUseExampleKeyDoNotUseExampleKeyDoNotUseExampleKeyDoNo
Now you can set up @ssh-agent@ (next) or proceed with "adding your key to the Arvados Workbench.":#workbench
-h3. Set up ssh-agent (recommended)
+h3. Set up ssh-agent (optional)
If you find you are entering your passphrase frequently, you can use @ssh-agent@ to manage your credentials. Use @ssh-add -l@ to test if you already have ssh-agent running:
@@ -80,11 +80,21 @@ When everything is set up, @ssh-add -l@ should yield output that looks something
{% include 'ssh_addkey' %}
-h3. Connecting to the virtual machine
+h3. Connecting directly
-Use the following command to connect to the _shell_ VM instance as _you_. Replace *<code>you at shell</code>* at the end of the following command with your *login* and *hostname* from Workbench:
+If the VM is available on the public Internet (or you are on the same private network as the VM) you can connect directly with @ssh at . You can probably copy-and-paste the text from *Command line* column directly into a terminal.
-notextile. <pre><code>$ <span class="userinput">ssh -o "ProxyCommand ssh -p2222 turnout at switchyard.{{ site.arvados_api_host }} -x -a <b>shell</b>" -x <b>you at shell</b></span></code></pre>
+Use the following example command to connect as _you_ to the _shell.ClusterID.example.com_ VM instance. Replace *<code>you at shell.ClusterID.example.com</code>* at the end of the following command with your *login* and *hostname* from Workbench.
+
+notextile. <pre><code>$ <span class="userinput">ssh <b>you at shell.ClusterID.example.com</b></span></code></pre>
+
+h3. Connecting through switchyard
+
+Some Arvados installations use "switchyard" to isolate shell VMs from the public Internet.
+
+Use the following example command to connect to the _shell_ VM instance as _you_. Replace *<code>you at shell</code>* at the end of the following command with your *login* and *hostname* from Workbench:
+
+notextile. <pre><code>$ <span class="userinput">ssh -o "ProxyCommand ssh -p2222 turnout at switchyard.ClusterID.example.com -x -a <b>shell</b>" -x <b>you at shell</b></span></code></pre>
This command does several things at once. You usually cannot log in directly to virtual machines over the public Internet. Instead, you log into a "switchyard" server and then tell the switchyard which virtual machine you want to connect to.
@@ -99,7 +109,7 @@ This command does several things at once. You usually cannot log in directly to
You should now be able to log into the Arvados VM and "check your environment.":check-environment.html
-h3. Configuration (recommended)
+h4. Configuration (recommended)
The command line above is cumbersome, but you can configure SSH to remember many of these settings. Add this text to the file @.ssh/config@ in your home directory (create a new file if @.ssh/config@ doesn't exist):
diff --git a/doc/user/getting_started/ssh-access-windows.html.textile.liquid b/doc/user/getting_started/ssh-access-windows.html.textile.liquid
index 0406e7c03..ee4ab4e92 100644
--- a/doc/user/getting_started/ssh-access-windows.html.textile.liquid
+++ b/doc/user/getting_started/ssh-access-windows.html.textile.liquid
@@ -9,13 +9,13 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-This document is for accessing an Arvados VM using SSH keys in Windows environments. If you would like to use to access VM through your browser, please visit the "Accessing an Arvados VM with Webshell":vm-login-with-webshell.html page. If you are using a Unix environment (Linux, OS X, Cygwin), please visit the "Accessing an Arvados VM with SSH - Unix Environments":ssh-access-unix.html page.
+This document is for accessing an Arvados VM using SSH keys in Windows environments using PuTTY. If you would like to use to access VM through your browser, please visit the "Accessing an Arvados VM with Webshell":vm-login-with-webshell.html page. If you are using a Unix environment (Linux, OS X, Cygwin, or Windows Subsystem for Linux), please visit the "Accessing an Arvados VM with SSH - Unix Environments":ssh-access-unix.html page.
{% include 'ssh_intro' %}
h1(#gettingkey). Getting your SSH key
-(Note: if you are using the SSH client that comes with "Cygwin":http://cygwin.com, please use instructions found in the "Accessing an Arvados VM with SSH - Unix Environments":ssh-access-unix.html page.)
+(Note: If you are using the SSH client that comes with "Cygwin":http://cygwin.com or Windows Subsystem for Linux (WSL) please use instructions found in the "Accessing an Arvados VM with SSH - Unix Environments":ssh-access-unix.html page.)
We will be using PuTTY to connect to Arvados. "PuTTY":http://www.chiark.greenend.org.uk/~sgtatham/putty/ is a free (MIT-licensed) Win32 Telnet and SSH client. PuTTY includes all the tools a Windows user needs to create private keys and make SSH connections to your virtual machines in the Arvados Cloud.
@@ -57,6 +57,16 @@ Pageant is a PuTTY utility that manages your private keys so is not necessary to
h3. Initial configuration
+h4. Connecting directly
+
+# Open PuTTY from the Start Menu.
+# On the Session screen set the Host Name (or IP address) to “shell.ClusterID.example.com”, which is the hostname listed in the _Virtual Machines_ page.
+# On the Session screen set the Port to “22”.
+# On the Connection %(rarr)→% Data screen set the Auto-login username to the username listed in the *Login name* column on the Arvados Workbench Virtual machines_ page.
+# Return to the Session screen. In the Saved Sessions box, enter a name for this configuration and click Save.
+
+h4. Connecting through switchyard
+
# Open PuTTY from the Start Menu.
# On the Session screen set the Host Name (or IP address) to “shell”, which is the hostname listed in the _Virtual Machines_ page.
# On the Session screen set the Port to “22”.
diff --git a/doc/user/getting_started/vm-login-with-webshell.html.textile.liquid b/doc/user/getting_started/vm-login-with-webshell.html.textile.liquid
index 551002e55..2aa494ae9 100644
--- a/doc/user/getting_started/vm-login-with-webshell.html.textile.liquid
+++ b/doc/user/getting_started/vm-login-with-webshell.html.textile.liquid
@@ -15,6 +15,10 @@ h2(#webshell). Access VM using webshell
Webshell gives you access to an arvados virtual machine from your browser with no additional setup.
+{% include 'notebox_begin' %}
+Some Arvados clusters may not have webshell set up. If you do not see a "Log in" button or "web shell" column, you will have to follow the "Unix":ssh-access-unix.html or "Windows":ssh-access-windows.html @ssh@ instructions.
+{% include 'notebox_end' %}
+
In the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Virtual machines* to see the list of virtual machines you can access. If you do not have access to any virtual machines, please click on <span class="btn btn-sm btn-primary">Send request for shell access</span> or send an email to "support at curoverse.com":mailto:support at curoverse.com.
Each row in the Virtual Machines panel lists the hostname of the VM, along with a <code>Log in as *you*</code> button under the column "Web shell". Clicking on this button will open up a webshell terminal for you in a new browser tab and log you in.
diff --git a/doc/user/getting_started/workbench.html.textile.liquid b/doc/user/getting_started/workbench.html.textile.liquid
index fc704227e..52f1931c9 100644
--- a/doc/user/getting_started/workbench.html.textile.liquid
+++ b/doc/user/getting_started/workbench.html.textile.liquid
@@ -9,13 +9,19 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-If you are using the default Arvados instance for this guide, you can Access Arvados Workbench using this link:
+{% include 'notebox_begin' %}
+This guide covers the classic Arvados Workbench web application, sometimes referred to as "Workbench 1". There is also a new Workbench web application under development called "Workbench 2". Sites which have both Workbench applications installed will have a dropdown menu option "Switch to Workbench 2" to switch between versions.
+
+This guide will be updated to cover "Workbench 2" in the future.
+{% include 'notebox_end' %}
+
+If you are using the "playground" Arvados instance for this guide, you can Access Arvados Workbench using this link:
<a href="{{site.arvados_workbench_host}}/" target="_blank">{{site.arvados_workbench_host}}/</a>
(If you are using a different Arvados instance than the default for this guide, replace *{{ site.arvados_workbench_host }}* with your private instance in all of the examples in this guide.)
-You may be asked to log in using a Google account. Arvados uses only your name and email address from Google services for identification, and will never access any personal information. If you are accessing Arvados for the first time, the Workbench may indicate your account status is *New / inactive*. If this is the case, contact the administrator of the Arvados instance to request activation of your account.
+You will be asked to log in. Arvados uses only your name and email address for identification, and will never access any personal information. If you are accessing Arvados for the first time, the Workbench may indicate your account status is *New / inactive*. If this is the case, contact the administrator of the Arvados instance to request activation of your account.
Once your account is active, logging in to the Workbench will present you with the Dashboard. This gives a summary of your projects and recent activity in the Arvados instance. "You are now ready to run your first pipeline.":{{ site.baseurl }}/user/tutorials/tutorial-workflow-workbench.html
diff --git a/doc/user/topics/arv-copy.html.textile.liquid b/doc/user/topics/arv-copy.html.textile.liquid
index 0f0e40be9..805d71b2d 100644
--- a/doc/user/topics/arv-copy.html.textile.liquid
+++ b/doc/user/topics/arv-copy.html.textile.liquid
@@ -9,17 +9,13 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-{% include 'crunch1only_begin' %}
-On those sites, the "copy a pipeline template" feature described below is not available. However, "copy a workflow" feature is not yet implemented.
-{% include 'crunch1only_end' %}
-
This tutorial describes how to copy Arvados objects from one cluster to another by using @arv-copy at .
{% include 'tutorial_expectations' %}
h2. arv-copy
- at arv-copy@ allows users to copy collections and pipeline templates from one cluster to another. By default, @arv-copy@ will recursively go through a template and copy all dependencies associated with the object.
+ at arv-copy@ allows users to copy collections and workflows from one cluster to another. By default, @arv-copy@ will recursively go through the workflow and copy all dependencies associated with the object.
For example, let's copy from the <a href="https://playground.arvados.org/">Arvados playground</a>, also known as *qr1hi*, to *dst_cluster*. The names *qr1hi* and *dst_cluster* are interchangable with any cluster name. You can find the cluster name from the prefix of the uuid of the object you want to copy. For example, in *qr1hi*-4zz18-tci4vn4fa95w0zx, the cluster name is qr1hi.
@@ -60,33 +56,6 @@ For example, this will copy the collection to project dst_cluster-j7d0g-a894213u
</code></pre>
</notextile>
-h3. How to copy a pipeline template
-
-{% include 'arv_copy_expectations' %}
-
-We will use the uuid @qr1hi-p5p6p-9pkaxt6qjnkxhhu@ as an example pipeline template.
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src qr1hi --dst dst_cluster --dst-git-repo $USER/tutorial qr1hi-p5p6p-9pkaxt6qjnkxhhu</span>
-To git at git.dst_cluster.arvadosapi.com:$USER/tutorial.git
- * [new branch] git_git_qr1hi_arvadosapi_com_arvados_git_ac21f0d45a76294aaca0c0c0fdf06eb72d03368d -> git_git_qr1hi_arvadosapi_com_arvados_git_ac21f0d45a76294aaca0c0c0fdf06eb72d03368d
-arvados.arv-copy[19694] INFO: Success: created copy with uuid dst_cluster-p5p6p-rym2h5ub9m8ofwj
-</code></pre>
-</notextile>
-
-New branches in the destination git repo will be created for each branch used in the pipeline template. For example, if your source branch was named ac21f0d45a76294aaca0c0c0fdf06eb72d03368d, your new branch will be named @git_git_qr1hi_arvadosapi_com_reponame_git_ac21f0d45a76294aaca0c0c0fdf06eb72d03368d at .
-
-By default, if you copy a pipeline template recursively, you will find that the template as well as all the dependencies are in your home project.
-
-If you would like to copy the object without dependencies, you can use the @--no-recursive@ tag.
-
-For example, we can copy the same object using this tag.
-
-<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src qr1hi --dst dst_cluster --dst-git-repo $USER/tutorial --no-recursive qr1hi-p5p6p-9pkaxt6qjnkxhhu</span>
-</code></pre>
-</notextile>
-
h3. How to copy a workflow
We will use the uuid @zzzzz-7fd4e-sampleworkflow1@ as an example workflow.
diff --git a/doc/user/tutorials/add-new-repository.html.textile.liquid b/doc/user/tutorials/add-new-repository.html.textile.liquid
index 9d8e768a7..cb69d460a 100644
--- a/doc/user/tutorials/add-new-repository.html.textile.liquid
+++ b/doc/user/tutorials/add-new-repository.html.textile.liquid
@@ -9,7 +9,7 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Arvados repositories are managed through the Git revision control system. You can use these repositories to store your crunch scripts and run them in the arvados cluster.
+Arvados supports managing git repositories. You can acess these repositories using your Arvados credentials and share them with other Arvados users.
{% include 'tutorial_expectations' %}
diff --git a/doc/user/tutorials/git-arvados-guide.html.textile.liquid b/doc/user/tutorials/git-arvados-guide.html.textile.liquid
index 2e255219d..ad719a66e 100644
--- a/doc/user/tutorials/git-arvados-guide.html.textile.liquid
+++ b/doc/user/tutorials/git-arvados-guide.html.textile.liquid
@@ -9,20 +9,13 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-This tutorial describes how to work with a new Arvados git repository. Working with an Arvados git repository is analogous to working with other public git repositories. It will show you how to upload custom scripts to a remote Arvados repository, so you can use it in Arvados pipelines.
+This tutorial describes how to work with an Arvados-managed git repository. Working with an Arvados git repository is very similar to working with other public git repositories.
{% include 'tutorial_expectations' %}
{% include 'tutorial_git_repo_expectations' %}
-{% include 'notebox_begin' %}
-For more information about using Git, try
-<notextile>
-<pre><code>$ <span class="userinput">man gittutorial</span></code></pre>
-</notextile> or *"search Google for Git tutorials":http://google.com/#q=git+tutorial*.
-{% include 'notebox_end' %}
-
-h2. Cloning an Arvados repository
+h2. Cloning a git repository
Before you start using Git, you should do some basic configuration (you only need to do this the first time):
@@ -65,33 +58,22 @@ Create a git branch named *tutorial_branch* in the *tutorial* Arvados git reposi
h2. Adding scripts to an Arvados repository
-Arvados crunch scripts need to be added in a *crunch_scripts* subdirectory in the repository. If this subdirectory does not exist, first create it in the local repository and change to that directory:
-
-<notextile>
-<pre><code>~/tutorial$ <span class="userinput">mkdir crunch_scripts</span>
-~/tutorial$ <span class="userinput">cd crunch_scripts</span></code></pre>
-</notextile>
-
-Next, using @nano@ or your favorite Unix text editor, create a new file called @hash.py@ in the @crunch_scripts@ directory.
-
-notextile. <pre>~/tutorial/crunch_scripts$ <code class="userinput">nano hash.py</code></pre>
-
-Add the following code to compute the MD5 hash of each file in a collection
+A git repository is a good place to store the CWL workflows that you run on Arvados.
-<notextile> {% code 'tutorial_hash_script_py' as python %} </notextile>
+First, create a simple CWL CommandLineTool:
-Make the file executable:
+notextile. <pre>~/tutorials$ <code class="userinput">nano hello.cwl</code></pre>
-notextile. <pre><code>~/tutorial/crunch_scripts$ <span class="userinput">chmod +x hash.py</span></code></pre>
+<notextile> {% code 'tutorial_hello_cwl' as yaml %} </notextile>
Next, add the file to the git repository. This tells @git@ that the file should be included on the next commit.
-notextile. <pre><code>~/tutorial/crunch_scripts$ <span class="userinput">git add hash.py</span></code></pre>
+notextile. <pre><code>~/tutorial$ <span class="userinput">git add hello.cwl</span></code></pre>
Next, commit your changes. All staged changes are recorded into the local git repository:
<notextile>
-<pre><code>~/tutorial/crunch_scripts$ <span class="userinput">git commit -m "my first script"</span>
+<pre><code>~/tutorial$ <span class="userinput">git commit -m "my first script"</span>
</code></pre>
</notextile>
@@ -102,4 +84,4 @@ Finally, upload your changes to the remote repository:
</code></pre>
</notextile>
-Although this tutorial shows how to add a python script to Arvados, the same steps can be used to add any of your custom bash, R, or python scripts to an Arvados repository.
+The same steps can be used to add any of your custom bash, R, or python scripts to an Arvados repository.
commit 2d046b9d2f27779f1e74e951fd7108dd88ba57c3
Author: Peter Amstutz <peter.amstutz at curii.com>
Date: Fri Jan 3 10:10:47 2020 -0500
15964: Delete long out of date (and unused) 'getting started' section
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>
diff --git a/doc/_includes/_tutorial_cluster_name.liquid b/doc/_includes/_tutorial_cluster_name.liquid
index 22fbc463e..3833abbf2 100644
--- a/doc/_includes/_tutorial_cluster_name.liquid
+++ b/doc/_includes/_tutorial_cluster_name.liquid
@@ -5,5 +5,5 @@ SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
{% include 'notebox_begin' %}
-This tutorial assumes you are using the default Arvados instance, @qr1hi at . If you are using a different instance, replace @qr1hi@ with your instance. See "Accessing Arvados Workbench":{{site.baseurl}}/user/getting_started/workbench.html for more details.
+This tutorial assumes you are using the playground Arvados instance, @qr1hi at . If you are using a different instance, replace @qr1hi@ with your instance. See "Accessing Arvados Workbench":{{site.baseurl}}/user/getting_started/workbench.html for more details.
{% include 'notebox_end' %}
diff --git a/doc/install/copy_pipeline_from_curoverse.html.textile.liquid b/doc/install/copy_pipeline_from_curoverse.html.textile.liquid
deleted file mode 100644
index 2c2b3c466..000000000
--- a/doc/install/copy_pipeline_from_curoverse.html.textile.liquid
+++ /dev/null
@@ -1,68 +0,0 @@
----
-layout: default
-navsection: installguide
-title: Copy pipeline from the Arvados Playground
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-This tutorial describes how to find and copy a publicly shared pipeline from the Arvados Playground. Please note that you can use similar steps to copy any template you can access from the Arvados Playground to your cluster.
-
-h3. Access a public pipeline in the Arvados Playground using Workbench
-
-the Arvados Playground provides access to some public data, which can be used to experience Arvados in action. Let's access a public pipeline and copy it to your cluster, so that you can run it in your environment.
-
-Start by visiting the "*Arvados Playground public projects page*":https://playground.arvados.org/projects/public. This page lists all the publicly accessible projects in this arvados installation. Click on one of these projects to open it. We will use "*lobSTR v.3 (Public)*":https://playground.arvados.org/projects/qr1hi-j7d0g-up6qgpqz5ie2vfq as the example in this tutorial.
-
-Once in the "*lobSTR v.3 (Public)*":https://playground.arvados.org/projects/qr1hi-j7d0g-up6qgpqz5ie2vfq project, click on the *Pipeline templates* tab. In the pipeline templates tab, you will see a template named *lobSTR v.3*. Click on the <span class="fa fa-lg fa-gears"></span> *Show* button to the left of this name. This will take to you to the "*lobSTR v.3*":https://playground.arvados.org/pipeline_templates/qr1hi-p5p6p-9pkaxt6qjnkxhhu template page.
-
-Once in this page, you can take the *uuid* of this template from the address bar, which is *qr1hi-p5p6p-9pkaxt6qjnkxhhu*. Next, we will copy this template to your Arvados instance.
-
-h3. Copying a pipeline template from the Arvados Playground to your cluster
-
-As described above, navigate to the publicly shared pipeline template "*lobSTR v.3*":https://playground.arvados.org/pipeline_templates/qr1hi-p5p6p-9pkaxt6qjnkxhhu on the Arvados Playground. We will now copy this template with uuid *qr1hi-p5p6p-9pkaxt6qjnkxhhu* to your cluster.
-
-{% include 'tutorial_expectations' %}
-
-We will use the Arvados *arv-copy* command to copy this template to your cluster. In order to use arv-copy, first you need to setup the source and destination cluster configuration files. Here, *qr1hi* would be the source cluster and your Arvados instance would be the *dst_cluster*.
-
-During this setup, if you have an account in the Arvados Playground, you can use "your access token":#using-your-token to create the source configuration file. If you do not have an account in the Arvados Playground, you can use the "anonymous access token":#using-anonymous-token for the source cluster configuration.
-
-h4(#using-anonymous-token). *Configuring source and destination setup files using anonymous access token*
-
-Configure the source and destination clusters as described in the "*Using arv-copy*":http://doc.arvados.org/user/topics/arv-copy.html tutorial in user guide, while using *5vqmz9mik2ou2k9objb8pnyce8t97p6vocyaouqo3qalvpmjs5* as the API token for source configuration.
-
-<notextile>
-<pre><code>~$ <span class="userinput">cd ~/.config/arvados</span>
-~$ <span class="userinput">echo "ARVADOS_API_HOST=qr1hi.arvadosapi.com" >> qr1hi.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_TOKEN=5vqmz9mik2ou2k9objb8pnyce8t97p6vocyaouqo3qalvpmjs5" >> qr1hi.conf</span>
-</code></pre>
-</notextile>
-
-You can now copy the pipeline template from *qr1hi* to *your cluster*. Replace *dst_cluster* with the *ClusterID* of your cluster.
-
-<notextile>
-<pre><code>~$ <span class="userinput"> arv-copy --no-recursive --src qr1hi --dst dst_cluster qr1hi-p5p6p-9pkaxt6qjnkxhhu</span>
-</code></pre>
-</notextile>
-
-*Note:* When you are using anonymous access token to copy the template, you will not be able to do a recursive copy since you will not be able to provide the dst-git-repo parameter. In order to perform a recursive copy of the template, you would need to use the Arvados API token from your account as explained in the "using your token":#using-your-token section below.
-
-h4(#using-your-token). *Configuring source and destination setup files using personal access token*
-
-If you already have an account in the Arvados Playground, you can follow the instructions in the "*Using arv-copy*":http://doc.arvados.org/user/topics/arv-copy.html user guide to get your *Current token* for source and destination clusters, and use them to create the source *qr1hi.conf* and dst_cluster.conf configuration files.
-
-You can now copy the pipeline template from *qr1hi* to *your cluster* with or without recursion. Replace *dst_cluster* with the *ClusterID* of your cluster.
-
-*Non-recursive copy:*
-<notextile>
-<pre><code>~$ <span class="userinput"> arv-copy --no-recursive --src qr1hi --dst dst_cluster qr1hi-p5p6p-9pkaxt6qjnkxhhu</span></code></pre>
-</notextile>
-
-*Recursive copy:*
-<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src qr1hi --dst dst_cluster --dst-git-repo $USER/tutorial qr1hi-p5p6p-9pkaxt6qjnkxhhu</span></code></pre>
-</notextile>
diff --git a/doc/start/getting_started/firstpipeline.html.textile.liquid b/doc/start/getting_started/firstpipeline.html.textile.liquid
deleted file mode 100644
index 43369a3bb..000000000
--- a/doc/start/getting_started/firstpipeline.html.textile.liquid
+++ /dev/null
@@ -1,94 +0,0 @@
----
-layout: default
-navsection: start
-title: Run your first pipeline in minutes
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-h2. LobSTR v3
-
-In this quickstart guide, we'll run an existing pipeline with pre-existing data. Step-by-step instructions are shown below. You can follow along using your own local install or by using the <a href="https://playground.arvados.org/">Arvados Playground</a> (any Google account can be used to log in).
-
-(For more information about this pipeline, see our <a href="https://dev.arvados.org/projects/arvados/wiki/LobSTR_tutorial">detailed lobSTR guide</a>).
-
-<div id="carousel-firstpipe" class="carousel slide" data-interval="false">
- <!-- Indicators -->
- <ol class="carousel-indicators">
- <li data-target="#carousel-firstpipe" data-slide-to="0" class="active"></li>
- <li data-target="#carousel-firstpipe" data-slide-to="1"></li>
- <li data-target="#carousel-firstpipe" data-slide-to="2"></li>
- <li data-target="#carousel-firstpipe" data-slide-to="3"></li>
- <li data-target="#carousel-firstpipe" data-slide-to="4"></li>
- <li data-target="#carousel-firstpipe" data-slide-to="5"></li>
- <li data-target="#carousel-firstpipe" data-slide-to="6"></li>
- </ol>
-
- <!-- Wrapper for slides -->
- <div class="carousel-inner" role="listbox">
- <div class="item active">
- <img src="{{ site.baseurl }}/images/quickstart/1.png" alt="Step 1. At the dashboard, click 'Run a pipeline...'.">
- <div class="carousel-caption">
- Step 1. At the dashboard, click 'Run a pipeline...'.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/quickstart/2.png" alt="Choose 'lobstr v.3' and hit 'Next'.">
- <div class="carousel-caption">
- Choose 'lobstr v.3' and hit 'Next'.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/quickstart/3.png" alt="Rename the pipeline instance, then click 'Run'. Click 'Choose' to change the default inputs.">
- <div class="carousel-caption">
- Rename the pipeline instance, then click 'Run'. Click 'Choose' to change the default inputs.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/quickstart/4.png" alt="Here we search for and choose new inputs.">
- <div class="carousel-caption">
- Here we search for and choose new inputs.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/quickstart/5.png" alt="After the job completes, you can re-run it with one click.">
- <div class="carousel-caption">
- After the job completes, you can re-run it with one click.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/quickstart/6.png" alt="You can inspect details about the pipeline which are automatically logged.">
- <div class="carousel-caption">
- You can inspect automatically-logged details about the pipeline.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/quickstart/7.png" alt="Click 'Create sharing link' to share the output files with people outside Arvados. [END]">
- <div class="carousel-caption">
- Click 'Create sharing link' to share the output files with people outside Arvados. [END]
- </div>
- </div>
-
- </div>
-
- <!-- Controls -->
- <a class="left carousel-control" href="#carousel-firstpipe" role="button" data-slide="prev">
- <span class="glyphicon glyphicon-chevron-left" aria-hidden="true"></span>
- <span class="sr-only">Previous</span>
- </a>
- <a class="right carousel-control" href="#carousel-firstpipe" role="button" data-slide="next">
- <span class="glyphicon glyphicon-chevron-right" aria-hidden="true"></span>
- <span class="sr-only">Next</span>
- </a>
-</div>
-
-Tip: You may need to make your browser window bigger to see full-size images in the gallery above.
diff --git a/doc/start/getting_started/nextsteps.html.textile.liquid b/doc/start/getting_started/nextsteps.html.textile.liquid
deleted file mode 100644
index dd059ea8d..000000000
--- a/doc/start/getting_started/nextsteps.html.textile.liquid
+++ /dev/null
@@ -1,12 +0,0 @@
----
-layout: default
-navsection: start
-title: Check out the User Guide
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-Now that you've finished the Getting Started guide, check out the "User Guide":{{site.baseurl}}/user/index.html. The User Guide goes into more depth than the Getting Started guide, covers how to develop your own pipelines in addition to using pre-existing pipelines, covers the Arvados command line tools in addition to the Workbench graphical interface to Arvados, and can be referenced in any order.
diff --git a/doc/start/getting_started/publicproject.html.textile.liquid b/doc/start/getting_started/publicproject.html.textile.liquid
deleted file mode 100644
index 0fabad7aa..000000000
--- a/doc/start/getting_started/publicproject.html.textile.liquid
+++ /dev/null
@@ -1,133 +0,0 @@
----
-layout: default
-navsection: start
-title: Visit an Arvados Public Project
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-h2. <a href="https://workbench.qr1hi.arvadosapi.com/projects/qr1hi-j7d0g-662ij1pcw6bj8uj">Mason Lab - Pathomap / Ancestry Mapper (Public)</a>
-
-You can see Arvados in action by accessing the <a href="https://workbench.qr1hi.arvadosapi.com/projects/qr1hi-j7d0g-662ij1pcw6bj8uj">Mason Lab - Pathomap / Ancestry Mapper (Public) project</a>. By visiting this project, you can see what an Arvados project is, access data collections in this project, and click through a pipeline instance's contents.
-
-You will be accessing this project in read-only mode and will not be able to make any modifications such as running a new pipeline instance.
-
-<div id="carousel-publicproject" class="carousel slide" data-interval="false">
- <!-- Indicators -->
- <ol class="carousel-indicators">
- <li data-target="#carousel-publicproject" data-slide-to="0" class="active"></li>
- <li data-target="#carousel-publicproject" data-slide-to="1"></li>
- <li data-target="#carousel-publicproject" data-slide-to="2"></li>
- <li data-target="#carousel-publicproject" data-slide-to="3"></li>
- <li data-target="#carousel-publicproject" data-slide-to="4"></li>
- <li data-target="#carousel-publicproject" data-slide-to="5"></li>
- <li data-target="#carousel-publicproject" data-slide-to="6"></li>
- <li data-target="#carousel-publicproject" data-slide-to="7"></li>
- <li data-target="#carousel-publicproject" data-slide-to="8"></li>
- <li data-target="#carousel-publicproject" data-slide-to="9"></li>
- <li data-target="#carousel-publicproject" data-slide-to="10"></li>
- <li data-target="#carousel-publicproject" data-slide-to="11"></li>
- </ol>
-
- <!-- Wrapper for slides -->
- <div class="carousel-inner" role="listbox">
- <div class="item active">
- <img src="{{ site.baseurl }}/images/publicproject/description.png" alt="Step 1. The project's first tab, *Description*, describes what this project is all about.">
- <div class="carousel-caption">
- Step 1. The project's first tab, *Description*, describes what this project is all about.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/collections.png" alt="The *Data collections* tab contains the various pipeline inputs, logs, and outputs.">
- <div class="carousel-caption">
- The *Data collections* tab contains the various pipeline inputs, logs, and outputs.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/instances.png" alt="You can see the jobs and pipelines in this project by accessing the *Jobs and pipelines* tab.">
- <div class="carousel-caption">
- You can see the jobs and pipelines in this project by accessing the *Jobs and pipelines* tab.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/collection-show.png" alt="In the *Data collections* tab, click on the *Show* icon to the left of a collection to see the collection contents.">
- <div class="carousel-caption">
- In the *Data collections* tab, click on the *Show* icon to the left of a collection to see the collection contents.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/collection-files.png" alt="The collection page lists the details about it. The *Files* tab can be used to view and download individual files in it.">
- <div class="carousel-caption">
- The collection page lists the details about it. The *Files* tab can be used to view and download individual files in it.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/collection-graph.png" alt="The collection *Provenance graph* tab gives a visual representation of this collection's provenance.">
- <div class="carousel-caption">
- The collection *Provenance graph* tab gives a visual representation of this collection's provenance.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/instance-show.png" alt="In the project *Jobs and pipelines* tab, click on the *Show* icon to the left of a pipeline to access the pipeline contents.">
- <div class="carousel-caption">
- In the project *Jobs and pipelines* tab, click on the *Show* icon to the left of a pipeline to access the pipeline contents.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/instance-components.png" alt="The pipeline *Components* tab details the various jobs in it and how long it took to run it.">
- <div class="carousel-caption">
- The pipeline *Components* tab details the various jobs in it and how long it took to run it.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/instance-job.png" alt="Click on the down arrow in one of the job rows to see the job details. You can also click on the job's output.">
- <div class="carousel-caption">
- Click on the down arrow <i class="fa fa-lg fa-fw fa-caret-down"></i> in one of the job rows to see the job details. You can also click on the job's output.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/instance-log.png" alt="The *Log* tab can be used to see the log for the pipeline instance.">
- <div class="carousel-caption">
- The *Log* tab can be used to see the log for the pipeline instance.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/instance-graph.png" alt="The *Graph* tab provides a visual representation of the pipeline run.">
- <div class="carousel-caption">
- The *Graph* tab provides a visual representation of the pipeline run.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/publicproject/instance-advanced.png" alt="The *Advanced* tab can be used to access metadata about the pipeline. [END]">
- <div class="carousel-caption">
- The *Advanced* tab can be used to access metadata about the pipeline. [END]
- </div>
- </div>
- </div>
-
- <!-- Controls -->
- <a class="left carousel-control" href="#carousel-publicproject" role="button" data-slide="prev">
- <span class="glyphicon glyphicon-chevron-left" aria-hidden="true"></span>
- <span class="sr-only">Previous</span>
- </a>
- <a class="right carousel-control" href="#carousel-publicproject" role="button" data-slide="next">
- <span class="glyphicon glyphicon-chevron-right" aria-hidden="true"></span>
- <span class="sr-only">Next</span>
- </a>
-</div>
-
-Tip: You may need to make your browser window bigger to see full-size images in the gallery above.
diff --git a/doc/start/getting_started/sharedata.html.textile.liquid b/doc/start/getting_started/sharedata.html.textile.liquid
deleted file mode 100644
index 02e0b7032..000000000
--- a/doc/start/getting_started/sharedata.html.textile.liquid
+++ /dev/null
@@ -1,102 +0,0 @@
----
-layout: default
-navsection: start
-title: Sharing Data
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-You can easily share data entirely through Workbench, the web interface to Arvados.
-
-h2. Upload and share your existing data
-
-Step-by-step instructions are shown below.
-
-<div id="carousel-sharedata" class="carousel slide" data-interval="false">
- <!-- Indicators -->
- <ol class="carousel-indicators">
- <li data-target="#carousel-sharedata" data-slide-to="0" class="active"></li>
- <li data-target="#carousel-sharedata" data-slide-to="1"></li>
- <li data-target="#carousel-sharedata" data-slide-to="2"></li>
- <li data-target="#carousel-sharedata" data-slide-to="3"></li>
- <li data-target="#carousel-sharedata" data-slide-to="4"></li>
- <li data-target="#carousel-sharedata" data-slide-to="5"></li>
- <li data-target="#carousel-sharedata" data-slide-to="6"></li>
- <li data-target="#carousel-sharedata" data-slide-to="7"></li>
- </ol>
-
- <!-- Wrapper for slides -->
- <div class="carousel-inner" role="listbox">
- <div class="item active">
- <img src="{{ site.baseurl }}/images/uses/gotohome.png" alt="Step 1. From the dashboard, go to your Home project.">
- <div class="carousel-caption">
- Step 1. From the dashboard, go to your Home project.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/uses/uploaddata.png" alt="Click 'Add data' → 'Upload files'.">
- <div class="carousel-caption">
- Click 'Add data' → 'Upload files'.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/uses/choosefiles.png" alt="A new collection is created automatically. Choose files to upload and hit Start.">
- <div class="carousel-caption">
- A new collection is created automatically. Choose files to upload and hit Start.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/uses/uploading.png" alt="Files will upload and stay uploaded even if the browser is closed.">
- <div class="carousel-caption">
- Files will upload and stay uploaded even if the browser is closed.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/uses/rename.png" alt="Rename the collection appropriately.">
- <div class="carousel-caption">
- Rename the collection appropriately.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/uses/sharing.png" alt="Click 'Create sharing link'. You can click 'unshare' at any later point.">
- <div class="carousel-caption">
- Click 'Create sharing link'. You can click 'Unshare' at any later point.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/uses/shared.png" alt="Now just share this link with anyone you want.">
- <div class="carousel-caption">
- Now just share this link with anyone you want.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/uses/sharedsubdirs.png" alt="Here's a more complex collection. [END]">
- <div class="carousel-caption">
- Here's a more complex collection. [END]
- </div>
- </div>
-
- </div>
-
- <!-- Controls -->
- <a class="left carousel-control" href="#carousel-sharedata" role="button" data-slide="prev">
- <span class="glyphicon glyphicon-chevron-left" aria-hidden="true"></span>
- <span class="sr-only">Previous</span>
- </a>
- <a class="right carousel-control" href="#carousel-sharedata" role="button" data-slide="next">
- <span class="glyphicon glyphicon-chevron-right" aria-hidden="true"></span>
- <span class="sr-only">Next</span>
- </a>
-</div>
-
-Tip: You may need to make your browser window bigger to see full-size images in the gallery above.
diff --git a/doc/start/index.html.textile.liquid b/doc/start/index.html.textile.liquid
deleted file mode 100644
index cddfb8e44..000000000
--- a/doc/start/index.html.textile.liquid
+++ /dev/null
@@ -1,133 +0,0 @@
----
-layout: default
-navsection: start
-title: Welcome to Arvados!
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-This guide provides an introduction to using Arvados to solve big data bioinformatics problems.
-
-h2. What is Arvados?
-
-Arvados is a free and open source bioinformatics platform for genomic and biomedical data.
-
-We address the needs of IT directors, lab principals, and bioinformaticians.
-
-h2. Why use Arvados?
-
-Arvados enables you to quickly begin using cloud computing resources in your bioinformatics work. It allows you to track your methods and datasets, share them securely, and easily re-run analyses.
-
-h3. Take a look (Screenshots gallery)
-
-<div id="carousel-keyfeatures" class="carousel slide" data-interval="false">
- <!-- Indicators -->
- <ol class="carousel-indicators">
- <li data-target="#carousel-keyfeatures" data-slide-to="0" class="active"></li>
- <li data-target="#carousel-keyfeatures" data-slide-to="1"></li>
- <li data-target="#carousel-keyfeatures" data-slide-to="2"></li>
- <li data-target="#carousel-keyfeatures" data-slide-to="3"></li>
- <li data-target="#carousel-keyfeatures" data-slide-to="4"></li>
- <li data-target="#carousel-keyfeatures" data-slide-to="5"></li>
- <li data-target="#carousel-keyfeatures" data-slide-to="6"></li>
- <li data-target="#carousel-keyfeatures" data-slide-to="7"></li>
- <li data-target="#carousel-keyfeatures" data-slide-to="8"></li>
- </ol>
-
- <!-- Wrapper for slides -->
- <div class="carousel-inner" role="listbox">
- <div class="item active">
- <img src="{{ site.baseurl }}/images/keyfeatures/dashboard2.png" alt="[START] After logging in, you will see Workbench's dashboard.">
- <div class="carousel-caption">
- [START] After logging in, you will see Workbench's dashboard.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/keyfeatures/running2.png" alt="Pipelines describe a set of computational tasks (jobs).">
- <div class="carousel-caption">
- Pipelines describe a set of computational tasks (jobs).
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/keyfeatures/log.png" alt="The output of all jobs is logged and stored automatically.">
- <div class="carousel-caption">
- The output of all jobs is logged and stored automatically.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/keyfeatures/graph.png" alt="Pipelines can also be viewed in auto-generated graph form.">
- <div class="carousel-caption">
- Pipelines can also be viewed in auto-generated graph form.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/keyfeatures/rerun.png" alt="Pipelines can easily be re-run exactly as before, or...">
- <div class="carousel-caption">
- Pipelines can easily be re-run exactly as before, or...
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/keyfeatures/chooseinputs.png" alt="...you can change parameters or pick new datasets.">
- <div class="carousel-caption">
- ...you can change parameters or pick new datasets.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/keyfeatures/webupload.png" alt="With web upload, data can be uploaded right in Workbench.">
- <div class="carousel-caption">
- With web upload, data can be uploaded right in Workbench.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/keyfeatures/collectionpage.png" alt="Collections allow sharing datasets and job outputs easily. 'Create sharing link' with one click.">
- <div class="carousel-caption">
- Collections allow sharing datasets and job outputs easily. 'Create sharing link' with one click.
- </div>
- </div>
-
- <div class="item">
- <img src="{{ site.baseurl }}/images/keyfeatures/provenance.png" alt="Data provenance is tracked automatically. [END]">
- <div class="carousel-caption">
- Data provenance is tracked automatically. [END]
- </div>
- </div>
-
-
- </div>
-
- <!-- Controls -->
- <a class="left carousel-control" href="#carousel-keyfeatures" role="button" data-slide="prev">
- <span class="glyphicon glyphicon-chevron-left" aria-hidden="true"></span>
- <span class="sr-only">Previous</span>
- </a>
- <a class="right carousel-control" href="#carousel-keyfeatures" role="button" data-slide="next">
- <span class="glyphicon glyphicon-chevron-right" aria-hidden="true"></span>
- <span class="sr-only">Next</span>
- </a>
-</div>
-
-Note: Workbench is the web interface to Arvados.
-Tip: You may need to make your browser window bigger to see full-size images in the gallery above.
-
-h3. Key Features
-
-<ul>
-<li><strong>Track your methods</strong><br/>
-We log every compute job: software versions, machine images, input and output data hashes. Rely on a computer, not your memory and your note-taking skills.<br/><br/></li>
-<li><strong>Share your methods</strong><br/>
-Show other people what you did. Let them use your workflow on their own data. Publish a permalink to your methods and data, so others can reproduce and build on them easily.<br/><br/></li>
-<li><strong>Track data origin</strong><br/>
-Did you really only use fully consented public data in this analysis?<br/><br/></li>
-<li><strong>Get results sooner</strong><br/>
-Run your compute jobs faster by using multi-nodes and multi-cores, even if your programs are single-threaded.<br/><br/></li>
-</ul>
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list