[ARVADOS] created: 1.4.0
Git user
git at public.curoverse.com
Wed Jun 5 20:16:03 UTC 2019
at 1b6ce6f86b4dbf79049f29d70ba8f72b3cf9b916 (tag)
tagging a8fe18b2b941f31896033bb58b9056e7731581b7 (commit)
replaces 1.3.0
tagged by Fernando Monserrat
on Wed Jun 5 17:15:00 2019 -0300
closes #15283 . Release 1.4.0
Arvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat at veritasgenetics.com>
Eric Biagiotti (119):
13306: Changes to arvados-cwl-runner code after running futurize --stage2
13306: Fixed formatting for newly introduced imports
13306: Fix for sdk/cwl tests.test_job.TestJob.test_run
13306: Fix for test_job.TestJob.test_done and test_done_use_existing_collection
13306: Fixed test_submit.TestCreateTemplate encoding
13306: Added py2 and 3 compatible 'next' for test_job.TestWorkflow.test_run
13306: Updated use of built in 'sorted' and a test_util bytestring
13306: Adding py2 and py3 compatible byte objects for literals
13306: test_submit fixes for py2 unicode support
13306: test_submit additional fixes for py2 unicode support
13306: Adding py2/3 unicode support to the test_job tests
13306: Updates ArvCwlExecutor to properly convert objects to JSON in unicode
13306: Updating depreciated warn function
13306: Updating depreciated assertEquals
13306: Updating get_current_container to raise when not a 404
13306: Removing unneeded try/except statements in the acr submit tests
13306: Prevents multiple RunTimeStatusLoggingHandlers from being added
13306: Removes a log handler added during a test, breaking subsequent tests.
13306: Adds test for when get_current_container results in a 404
13306: Updating dict value iteration to be more efficient with py2
13306: Updating dict keys as a list to ensure py2/3 compatibility
13306: Updating dict values as a list to improve py2 performance
13306: Fixing previous commit to use viewvalues instead of listvalues
13306: Updating dict items as lists to improve py2 performance
13306: Removing list wrapping added by futurize to PathMapper objects
13306: Removing old_div to simplify division operations
13306: Unicode fixes for the conformance tests
13306: Unicode fix for docker image generation in py3 environment
13306: Adding support for building the dev docker jobs image with py3
13306: Removing debug echo
Merge branch 'master' into 13306-arvados-cwl-runner-py3-support
13306: Fixes futurize mistakes regarding PathMapper items()
13306: Additional dict item iteration improvements
13306: Improved py2/3 str compatibility
13306: fixes test_with_arvados.sh pip executable with python2
13306: Adds declaration of py2/3 support to arvados cwl and python sdks
Merge branch '13306-arvados-cwl-runner-py3-support'
13306: Removing unneeded import statements
Merge branch 'master' into 13306-arvados-cwl-runner-py3-support
Merge branch '13306-arvados-cwl-runner-py3-support'
14745: Makes the azure driver its own package
14745: Adds gocheck hook to go test for azure dirver
14745: Updates config keys, documentation, and addresses golint suggestions
14745: Makes DeleteDanglingResourcesAfter an arvados.Duration
14745: Un-exports identifiers
14745: Updates drivers to use json.RawMessage instead of map[string]interface
14745: Updates StubDriver to match driver interface and updates vendor.json
14745: Replaces debug println with proper return and doc clean up
14745: Adds back param loading to StubDriver InstanceSet
14745: Removes unneeded file and debug code and improves Duration comments
14745: Improves azure driver exported var comment
14745: Adds support for passing nil params to StubDriver InstanceSet
14745: Removes unnecessary test parameter
Merge branch '14745-azure-cloud-driver-fixups'
14853: Adds os.name check for subprocess conditional import
Merge branch '14853-chapmanb-subprocess-merge'
9865: Simplifies ACR exception handling
9865: Removes handling of exceptions derived from BaseException
9865: Executor exception improvement
9865: Adds stack trace to API server state check exception
14885: Updates sdk/cwl and services/fuse ciso8601 dependency
Merge branch 'master' into 14885-ciso-and-conda-packaging-pr
14885: Removes as_naive datetime parsing distinction
14885: Adds python2 conditional pytz import
9865: Removes trailing whitespace
Merge branch '9865-cwl-fix-ignored-exceptions'
Merge branch '14885-ciso-and-conda-packaging-pr'
14886: Adds a test that reproduces the error
14886: Simplifies test mocking
14886: Adds flag to stop infinite logs
14886: Updates test to fail for the more specific case
14886: Fixes mock issue between tests
14886: Fixes typo
Merge branch '14886-acr-recursion-runtime-error'
14885: Reverts back to parse_datetime_as_naive
Merge branch '14885-ciso-and-conda-packaging-pr'
14645: Adds FUSE ops timing using prometheus
14645: Adds FUSE ops time to crunchstat output
Merge remote-tracking branch 'origin/master' into 14645-fuse-operations-reporting
14645: Adds prometheus_client dependency
14645: Adds individual ops time output
14645: Adds op counts to reporting
14645: Updates crunchstat format for fuse ops
Merge remote-tracking branch 'origin/master' into 14645-fuse-operations-reporting
Merge branch '14645-fuse-operations-reporting'
14484: Adds migration file for collection file count and size
14484: Updates the db structure after db:migrate
14484: Adds functionality and test for pdh grouping in the container model
14484: Fixes bug in pdh transaction grouping
14484: Adds migration for collection file count and total size
14484: Changes file token to match the one used in collection.rb model
14484: Fixes grouping bug by simplifying nested pdhs arrays
14484: Adds file count and size callback to the collection model
14484: Updates the migration to use the sdk for manifest parsing
14484: Makes quotation marks consistent
14484: Simplifies pdh transaction grouping, keeps SQL in the migration
14484: Adds test for pdh grouping functionality in the container model
14484: Fixes tests that intentionally pass in invalid manifests
14484: Fixes performance test by changing file_size_total to bigint
14484: Updates collection documentation to include the new file stats
14484: Adds tests for Collection model and makes file attributes readonly
14484: Moves pdh grouping into a lib module
14484: Moves test for grouping pdhs to its own file
14484: Moves the file count and total size callback to after_validation
14844: Adds collections controller tests and updates collection model tests
14484: Update collection model to overwrite file stats if a change is attempted
14484: Improves test coverage and attribute setting performance
14484: Adds test for updating a collection with file stats and manifest changes
Merge remote-tracking branch 'origin/master' into 14484-collection-record-update
Merge branch '14484-collection-record-update'
14723: Mapping each target destination to an input file WIP
14723: Fixes literal mapping and name conflict resolution
14723: Adds tests for multiple file targets and literal name resolution
14723: Fixes trailing whitespace
15002: Adds admin feature for cancelling completed containers
Merge branch '15002-cancel-completed-container'
14723: Adds comments for the changes to StagingPathMapper
Merge branch 'master' into 14723-cwl-multiple-file-targets
Merge branch '14723-cwl-multiple-file-targets'
Fernando Monserrat (15):
no issue # Fix and update rvm GPG Key signature issue
no issue # Fix and update rvm GPG Key signature issue on test dockerfiles
issue #14891 upload rcs to testing repo
issue #14891 upload rcs . Add verbose
issue #14891 Update to receive param rc to build for release candidates
issue #14891 remove -x
issue #14891 fix spaces and initialize RC
refs #14891 .Merge branch '14891-upload-to-testing-repo'
no issue # Remove jessie-updates from sources.list as it is no longer supported
issue #15041 Updates adding versioning for java sdk and gradle task to build and publish it. gradle publish include push to sonatype \nArvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat at veritasgenetics.com>
Merge branch '14670-new-java-sdk-docs' of git.curoverse.com:arvados into 14670-new-java-sdk-docs
issue #15041 Updates adding versioning for java sdk and gradle task to build and publish it. gradle publish include push to sonatype \nArvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat at veritasgenetics.com>
issue #15041 Updates adding groupID org.arvados to the settings and install to script to deploy to staging repo \nArvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat at veritasgenetics.com>
issue #15041 Fix gradle command \nArvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat at veritasgenetics.com>
issue #15041 Split test and upload. uploadArchives passed as parameter to publish artifact \nArvados-DCO-1.1-Signed-off-by: Fernando Monserrat <fmonserrat at veritasgenetics.com>
Lucas Di Pentima (208):
14012: Early cached manifest validation by doing HEAD req on first block.
Merge branch 'master' into 14012-arvput-check-cache
14012: Do HEAD request on the oldest non-expired signature for cache validation
14012: Rephrases error message.
14012: Fix test
14012: Split cache file name generation for readability
14012: Enhance test coverage on _cached_manifest_valid() method
13006: Test exposing the bug
13006: Changes is_a filter to use a substring query to support remote UUIDs.
13006: Adds expression index on links.[tail|head]_uuid
Merge branch '14012-arvput-check-cache'
13006: Fail when using is_a filter on uuid attr and prefix_uuid doesn't match.
13006: Uses 'is_a' filter again because it's being fixed on the API server.
Merge branch '13006-api-is_a-filter' into 13006-sync-groups-is_a-filter
Merge branch '13006-api-is_a-filter'
Merge branch '13006-sync-groups-is_a-filter'
14640: Upgrades API to rails 4.2.11
14640: Upgrades WB rails to 4.2.11
Merge branch '14640-api-wb-activejob-upgrade'
14539: Persists empty directories by adding a '\056' empty file.
14539: Tests that '\056' cannot be used as a file or directory name.
14539: Fixes fuse tests.
14539: Escape backslash & space chars on file/dir names, with tests.
14482: Enhance code readability
14482: Don't list empty dir placeholder files on collections listings.
14482: Handle escaped chars on manifest validation.
14482: Fix Manifest::files_count()
Merge branch 'master' into 14539-pysdk-empty-dir
14538: Update comment.
14539: Encode \t & \n chars on file/dir names on collection's manifest.
Merge branch 'master' into 14482-rubysdk-empty-dir
14482: Adds failing test about partially unescaped file tokens.
14482: Allow unescaped " " on stream and file token regexes (WIP)
14539: Fixes literal backslash escaping.
14539: Fixes and simplifies the escape() function. Updates test.
14539: Removes pointless code comment.
14539: Simplifies a bit further the regexp replacement on escape() function.
Merge branch '14539-pysdk-empty-dir'
14482: Manifest validation including 8-bit escaped chars; with tests.
14482: Stricter token regexes.
Merge branch '14482-rubysdk-empty-dir'
14770: Fixes parameter on acr call.
14718: Issue salted token when passing 'remote' param to login.
14718: Changes the way remote param is packed into return_to.
14718: Validates remote cluster id parameter on login endpoint & callback.
14718: Updates controller test to support josh_id's return_to new format.
Merge branch '14718-api-login-salted-token'
14738: Fixes the double-slash usage on requests to prevent 301 from controller
14738: Enhances code readability when concatenating strings.
14738: Replaces double with single quotes for code style.
Merge branch '14738-tag-editor-fix-es5'
Merge branch '14770-wb-acr-bad-param'
13937: Export stats as prometheus metrics. (WIP)
13937: Refactors approach to pass volume metrics as curried vecs (WIP)
13937: Moves statsTicker metrics init to its own func.
13937: Adds facility for drivers to register their own counters.
14860: Exposes the bug with a new test.
14860: Fixes bug by assigning the signed manifest text to the snapshot being saved.
Merge branch '14860-fix-collection-versioning'
Merge branch 'master' into 13937-keepstore-prometheus
13937: Simplified volume specific metric handling (WIP)
14804: Exposes the bug with a test.
14804: Returns 503 status (instead of 404) on GET when a volume does the same.
14794: Adds Workbench2 menu item.
14804: Adds test scenario comment
Merge branch '14804-keepstore-transient-backend-errors'
14794: Adds configuration to point to workbench2 URL explicitly
14794: Adds test
14794: Adds workbench2 url to workbench's config on arvbox
14794: Validates workbench2_url config at startup.
14794: Moved config validator to lib/ so that it can be testable.
Merge branch '14794-link-wb1-to-wb2'
14884: Adds unicode-named file upload test.
13937: Don't try to update stats if they aren't initialized (useful for tests)
13937: Fixes file naming inconsistency between storage drivers.
13937: Adds counters to S3 driver.
13937: Adds counters to azure driver.
14916: Adds upgrading note with script example for collection checking.
14916: Rephrases upgrade notes. Enhances example script with success message.
Merge branch '14916-rubysdk-upgrade-note'
Merge branch 'master' into 13937-keepstore-prometheus
13937: Simplifies arg passing on Start() funcs
13937: Fixes metrics naming
13937: Simplifies volume counter stats assignments. Fixes test.
13937: Adds tests for operation & I/O counters.
13593: Adds 'async' parameter to groups create & update calls for perms updates.
13593: Updates API documentation.
13937: Removes unnecessary data passing on tests.
13593: Changes default async permission update interval. Clears cache.
13593: Cleans up old experimental code.
13937: Removes the 'any' operation type for a better way of testing driver labels
13593: Moves & enhances group creation with async=true test.
Merge branch '13937-keepstore-prometheus'
Merge branch '13593-async-perm-graph-update'
14966: Fixes test.
14966: Implements cleanup on test case teardown.
Merge branch '14966-api-test-hang-fix'
14873: Updates arvados & arvados-cli gems dependency on google-api-client.
14873: Removes protected_attributes gem dependency.
14873: Removes activerecord-deprecated_finders gem dependency.
Merge branch '14873-google-api-client-update'
14873: Rails 5 upgrade. Some tests still failing. (WIP)
14873: Replaces *_filter with *_action
14873: Fixes all deprecation notes on unit tests
14873: Adds missing copyright notices
14873: Fixes deprecation notes on functional tests.
14783: Fixes more deprecation warnings from functional tests.
14873: Adds new abstract class ApplicationRecord.
14873: Removes legacy setting as all controllers were migrated.
14873: Upgrades 'pg' gem & solves an issue related to a new internal table.
14873: Fixes deprecation warnings on integration tests.
14873: Fixes integration tests.
14873: Fixes parameter handling (WIP)
14873: Marks past migrations with version 4.2
14873: Skips indexes which 'columns' attribute isn't an Array.
14873: Gets rid of yet another deprecation warning.
14873: Adds headers to controller tests' requests.
Merge branch 'master' into 14873-api-rails5-upgrade
14873: Fixes unit tests.
14873: Removes debug data on fixture.
14873: Reverts previous fixture changes.
14873: Updates test for fix deprecation warning.
14873: Fixes nested params behavior on controller tests.
14873: Custom JSON attributes that default to [] or {} when nil.
14873: Fixes JSON custom types.
14873: Removes byebug debug call on test.
14873: Adds JSONB columns to discovery doc.
14873: Modifies failing tests for debugging purposes.
14873: Migrated functional tests to be integration tests.
14873: Changes perftest with benchmark because of unidentified errors.
14873: Fixes permission benchmark test
Merge branch 'master' into 14873-api-rails5-upgrade
14873: Adjusts latest API Server changes to be rails5 worthy.
14873: Fixes typo on schema file.
14873: Upgrades missing config settings.
14873: Updates dependency on our themes_for_rails fork.
14873: Fixes performance test.
Merge branch '14873-api-rails5-upgrade'
14988: Initial Gemfile update. Replaces EOLed sass gem.
14988: Updates assets server config.
14988: Updates *_filter callbacks to use the new *_action form.
14988: Allows units & functionals tests to run.
14988: File updates & additions from 'rake rails:update'
14988: Removes deprecated warning about mini_test. Adds byebug for tests.
15109: Adds admin doc section on how to manage the api server's log table.
15109: Adds missing doc file.
Merge branch '15109-logs-table-admin-docs'
14988: Uses wiselinks gem fork to fix some deprecation warnings.
14988: Fixes deprecation warnings on controller tests (WIP)
14988: Removes almost all pending deprecated warnings on functional tests.
14988: More deprecation warning fixes on functional tests.
Set up "test" environment before dropping the rails database.
Adjust the environment setting command to be consistent with other calls.
Merge branch 'rails5-schema-env-data'
14988: Refactors ArvadosBase class to not subclass ActiveRecord.
14988: Makes test runner defer the test failures summary.
Fixes API Server's test runner to print failures summary at the end of the run.
Merge branch 'api-tests-summary-fix'
15167: Adds mention about unlogged_attributes setting on Admin docs section.
Fixes test running call on API server to be able to run single test files.
15161: Re-adds API's config:check rake task (as a noop)
Merge branch '15161-config-check-rake-fix'
14988: Fixes functionals tests (WIP)
14988: Fixes unit test by adding 'x-python' type to the preview-allowed list.
14988: Fixes CSS syntax issue.
14988: Fixes CSS syntax error on Jobs controller tests.
14988: Adds missing create!() method to ArvadosBase.
14988: Fixes anonymous config & token usage issues on tests.
14988: Removes deprecation warning about rendering nothing.
14988: Disables wiselinks gem. Testing if it's really needed.
14988: Fixes exception raising when logging out.
14988: Fixes .dup method behavior on ArvadosBase.
14988: Adds empty subdir to test/ as it's required by rails.
14988: Fixes exception raising due to a change in rails5 behavior.
Merge branch 'master' into 14988-wb-rails5-upgrade
14988: Fixes merge conflict
Merge branch 'master' into 14988-wb-rails5-upgrade
14988: Fixes assert_equal param ordering for proper error messages.
14988: Fixes pipeline instance controller's param handling.
14988: Upgrades bootstrap-sass to deal with CVE-2019-8331.
14988: Fixes bootstrap import issue
Merge branch '15167-unlogged-attrs-api-docs'
15227: Adds test exposing bug.
Syncs up generated config.
15227: More tests to expose the bug.
15227: Fixes type checking on json attributes.
14988: Fixes destroyed? method to return false on new records.
14988: Restores debug message when accessing non-loaded attributes.
Merge branch '15227-apiserver-properties-bugfix'
Merge branch '14988-wb-rails5-upgrade'
15258: Adds /lib to the eager_load paths.
15258: Removes /lib from eager load paths because of failing integration tests.
15258: Explicitly require libs instead of relying on autoloading.
Merge branch '15258-wb-report-issue-bugfix'
15725: Adds test exposing 2 issues when trashing collections with versioning
15275: Saves snapshot as system user instead of assigning a signed manifest to it
15275: Avoids running manage_versioning when not needed.
15275: Moves comment to its proper place.
15275: Adds another test exposing the jsonb changed field bug.
15275: Don't create snapshot when trashing collection.
15275: Don't create snapshots whenever is_trashed is true.
15275: Simplifies method code.
Merge branch '15275-collection-delete-bugfix'. Refs #15275
15275: Re-adds the test exposing this branch's particular bug.
Merge branch 'master' into 15275-attribute-dirtyness-bugfix
15275: Checks for jsonb types changes by comparing deserialized values.
15275: Adds test for different jsonb properties comparisons cases.
Merge branch '15275-attribute-dirtyness-bugfix'
Nico Cesar (1):
Typo in one of the parameters
Peter Amstutz (234):
14574: Ensure output callbacks take workflow lock
Merge branch '14574-expression-fix' refs #14574
14593: Fix a-c-r when running on the "jobs" API
14593: Add note about broken crunch1 to upgrade docs.
14593: Fix typo.
Merge branch '14593-acr-crunch1' refs #14593
14476: When creating a Runner, check input is valid before submitting
14476: Fix RunnerTemplate
14476: Add test that submitting with missing parameters correctly gets error.
14495: Add dispatchcloud.EstimateScratchSpace()
14476: Remove redundant falsey check
Merge branch '14476-cwl-missing-input' refs #14476
14594: Update to latest cwltool
Merge branch '14594-update-cwltool' refs #14594
14495: Add warnings when estimateDockerImageSize gets bad input
Merge branch '14495-crunch-docker-space' refs #14495
14595: Add leave_modified_at_alone
14595: Fix comment.
Merge branch '14595-leave-modified-at' refs #14595
Set DBName correctly for arvbox demo, hopefully fix ci.commonwl.org
Update README and community doc page. refs #14617
Update cwltool refs #14510
14327: Don't create new collection if source has expected secondaryFiles
Fix test_with_arvbox.sh to use https://versions.arvados.org no issue #
14327: Add comments
Merge branch '14327-cwl-unnecessary-collections' refs #14327
14660: Add workbench2 to arvbox. Improve SSL support in arvbox.
14660: Proxy keep-web for https. Install root cert inside the container.
14660: Enable workbench2 to connect to development server via websockets.
14660: make "sv restart" work for workbench2, prevent fail loop
Merge branch '14660-arvbox-workbench2' refs #14660
Configure UTF-8 locale inside arvbox, no issue #
arvbox uses 'down: ready' as ready condition no issue #
13964: Azure provider WIP
13964: Can successfully create a VM, list existing VMs
13964: Full lifecycle for NICs/VHDs/VMs
13964: Return Arvados instance type
13964: Detect and report rate limit and quota exceeded errors
13964: Don't panic running tests on stubs
13964: VM names are namespaced. Can set/get tags.
13964: InstanceTags type
13964: SSH access WIP
13964: Tweak test
13964: Set and check node-token WIP
13964: ssh key verification wip
13964: ssh key checking
13964: Trim space
13964: Fix string pointer capture
13964: Forget the node token once the key fingerprint is known
13964: Fail on missing node-token tag
14324: Azure driver WIP
14324: Azure driver for crunch-dispatch-cloud
14324: Remove context from interface, use Stop() instead
14324: Use logrus in Azure driver. Fix Sirupsen->sirupsen in imports
14324: Fix tests
Fix arvbox demo for workbench2 no issue #
14324: Fix API throttling header fallbacks. Add gocheck_test
14324: Embed test public key instead of reading from file
Merge branch '14324-cdc-azure' refs #14324
13306: Fix 12213-keepref test
arvbox certificates can have DNS in subjectAltName
13306: test_with_arvbox.sh can test either python2 or python3
13306: select between "conformance" and "integration" in test_with_arvbox.sh
13306: test_with_arvbox can select api
13306: fix crunchv1 bug
arvbox support for python3 refs #13306
14725: Don't give a cluster_id when updating container request record
14826: API server looks in CA cert path that works on CentOS
14725: Fix tests.
Merge branch '14725-wes' refs #14725
14826: Fix copy and paste error
Merge branch '14826-cert-path' refs #14826
11561: Add lock_count to containers, cancel container on unlock exceeded
Merge branch '11561-limit-container-locks' refs #11561
14291: EC2 driver WIP
14291: Working on instance creation
14291: Add github.com/aws/aws-sdk-go to vendor
14291: Add individual packages in github.com/aws/aws-sdk-go to vendor
14291: Still trying to nail down aws go sdk vendor
14291: Vendoring
14291: Support EBS attached storage and preemptible instances
14291: Change instance type to a supported one, add SetTags test
14291: Fix tag prefix typo
14291: minimal stub so that tests don't panic when run without live config
14291: Adjust ExtraScratch (int) to AttachScratch (bool)
14844: Azure fixes
Merge branch '14844-cdc-azure-fixes' closes #14844
14870: Ruby SDK fix for copying collection into root of an empty collection
Merge branch '14870-ruby-sdk-cp-r' refs #14870
14870: Saving previous retry logs into collection WIP
14870: Update Arvados Ruby SDK gem dependency for bugfix
14870: Copy previous container logs to "container uuid" subdir
14870: Add tests
14870: Fix another test
14291: Tweak symbol naming
14870: Remove \x01 from edge cases
14291: Introduce "AddedScratch" and "IncludedScratch" to InstanceType
14870: Adjust edge case some more, a file named '\' isn't legal either
14870: Oops need to fix edge case assertion, too
14291: Add missing packages to vendor
14291: Add more missing packages to vendor
14291: Add more missing packages to vendor
14291: Another update to vendor.json
14291: Try again with vendor.json
14291: Report errors from ImportKeyPair
Merge branch '14870-retry-logs' refs #14870
14291: Tweak Scratch/AddedScratch/IncludedScratch behavior
14291: Add EbsVolumeType, also test fixes
14291: Generate key fingerprints that work with AWS. Filter out terminated instances
14291: Exclude both "shutting-down" and "terminated" instances.
14880: Fix argument parsing issues
Add test script. Add licensing information
Tweak test script no issue #
14291: EbsVolumeType → EBSVolumeType
Merge branch '14291-cdc-aws' refs #14291
14880: Fix help message typo.
Merge branch '14880-keepdocker-argparse' refs #14880
Add 'sdk/java-v2/' from commit '55f103e336ca9fb8bf1720d2ef4ee8dd4e221118'
14669: Add java v2 SDK to run-tests
14670: Add java sdk v2 to docs
14670: Tweak install instructions
Arvbox updates, no issue #
Fix a-c-r printing logs on error refs #14870
14322: Convert uuids on input to portable data hashes
14322: Propagate collection UUID to container mount
14322: Tests for edge cases
14322: CollectionFsAccess accepts UUIDs
14322: Tweak documentation to mention uuid keep URI form
14322: Fix tests
14322: Update workbench to use new collectionUUID
14322: Tweak workbench support from manual testing
14322: Add additional example input files
14322: Update comment on bwa-mem-input-mixed.yml
14322: Workbench links collection uuids in workflow inputs
14322: Add missing test files
14322: Fix noisy unicode logging error in tests, add comments
14322: Tweak JSON_KEEP_LOCATOR_REGEXP and comment
Merge branch '14322-cwl-uuid-input' refs #14322
Move back arvbox version of nodejs to fix composer compile issue
arvbox nginx redirects http to https
Merge branch 'master' into 14669-java-sdk-v2
Merge branch '14669-java-sdk-v2' into 14670-new-java-sdk-docs
test mode creates certificate so that setting up SSO can succeed
run prepare.py in fed tests in docker container refs #15040
prepare.py in fed tests uses default python refs #15040
Fix assigning APIHost in NewClientFromConfig refs #15045
15050: Limit number of containers that crunch-dispatch-local runs at a time
15050: Need to make sure to raise the semaphore when we're done
15050: After waking up, check if the container was cancelled
Merge branch '15050-limit-cdl-containers' closes #15050
Update arvbox docs to describe how to install root cert
Merge branch 'arvbox-cert-docs' no issue #
Pin networkx because 2.2 is the last version to support Python 2.7 and 3.4
Run check-exist.py in Docker container refs #15040
check-exist uses default python (oops) refs #15040
13996: Config migration WIP
13996: Migrating defaults to new config structure
13996: Updating API server to use new config object WIP
13996: More config updates
13996: Migrate majority of defaults to config.defaults.yml
13996: Parsing durations wip
13996: More config migrations, refactor some code into config_loader.rb
13996: Migrate tests to new config
13996: Fixing test WIP
13996: Unit tests pass
13996: Don't force check for GitInternalDir, breaks tests
13996: Explicitly get string representation of URLs
13996: Fix discovery document remoteHosts
13996: Add rake config:diff
13996: Update comments
13996: Adjust config:dump to dump active config
13996: Refactor, create ConfigLoader class
13996: Can now use database info from config.yml
13996: Fix tests.
13996: Re-add symlink config.default.yml
13996: Fix legacy_jobs_api check
13996: Move configuration load earlier, before database setup
13996: Copy config into OrderedOptions for nicer access
13996: Update tests for cleaner config access
13996: Don't override name of test db if one was provided in database.yml
13996: Update rest of code for cleaner config access
13996: Tweak rake tasks
13996: Documentation for config migration
13996: Enable ERB for application.yml and database.yml
13996: Improve docs a bit
Merge branch 'master' into 14670-new-java-sdk-docs
Add gradle to arvbox, refs #14670
Update arvbox documentation for devenv config, no issue #
14670: Fix jdk and gradle links
Merge branch '13996-new-api-config' refs #13996
14670: Fix run-tests. Skip failing test.
14670: Ignore failing test
14670: Skip test (fixed)
Set secrets.secret_key_base refs #13996
Merge, don't replace RemoteClusters refs #13996
14670: Add an example page
15119: Fixes for Workbench1.ExternalURL and Controller.ExternalURL
Merge branch '15119-api-config-fixes'
14670: Update documentation with example gradle file
Handle unrecognized uuid_prefix of remote host refs #13996
Merge branch '14670-new-java-sdk-docs' refs #14670
Add gradle.properties with empty ossrhUsername/Password refs #14670
15164: Add locking between container completion and reuse
15112: Script to re-run containers
15112: Don't submit anything, just produce a csv file with results
15112: Rename since it doesn't rerun anything
15112: log root user. log progress.
Merge branch '15164-cr-finalize-lock' closes #15164
Fix mistake setting container_count_max from wrong config parameter
Merge branch '15112-rerun-containers' refs #15112
arvbox crunch-run and certificate fixes
Pin pykka < 2 for nodemanager no issue #
15061: Correctly filter out "*" from RemoteClusters
15061: API support for linking local account to remote user
15061: Implement admin variant of user merge by uuid
15061: Add missing file
15061: add federation-migrate script
15061: Move federation migrate script into sdk
15061: Add documentation for using arv-federation-migrate
Merge branch '15061-fed-login' refs #15061
Migrate arvbox api application.yml settings to config.yml
Add workbench2Url to discovery document refs #15064
15061: Check well-connectedness. Fix typos.
15061: Add warning about clusters that are not in tokens file.
15061: Code cleanup, improve validity checking & error handling
Merge branch '15061-fed-migrate' refs #15061
Add user email and cancel button to federated user redirect page
Update the note about supported Python versions for Arv SDK.
15241: Fix input bare collection as directory
Merge branch '15241-collection-input' refs #15241
Fix cwl tests refs #15241
Increase client_max_body_size in arvbox to support file uploads.
arvbox use lowercase keys for database connection params refs #15273
Fix reporting logs on container requests refs #15028
Tom Clegg (304):
14360: Initial version of dispatch-cloud.
14360: Fix tests.
14360: Add explicit Start method to dispatcher.
14360: Update readme.
14360: Clarify sync loop timing.
14360: Close SSH connections when no longer needed.
14360: Use consts for tag keys.
14360: Split up long run func.
14360: Cut excess time.Now() calls.
14360: Just notify once per sync.
14360: Move shutdown-if-broken check to its own func.
14360: Move monitoring API test from lame driver to stub driver.
14360: Update busy/lastUUID/running together. Clarify race comment.
14360: Rename keeplocal -> dontupdate, add explanation.
14360: Rename View to InstanceView.
14360: Add missing mutex lock.
14360: Improve comments.
14360: Log level debug->warning for unexpected condition.
14360: Cancel containers asynchronously in Sync cleanup.
14360: Encapsulate scheduler object.
14360: Fix mismatched flag name, -nodetach -> -detached.
14360: Ensure signals don't pass to child in startup race.
14360: Remove redundant wkr.booted flag.
14360: Remove redundant wp.unallocated field.
14360: Fail Create calls fast during quota quarantine.
14360: Move slurm-only code to crunch-dispatch-slurm.
14360: Fix concurrent map access in tests.
14360: Add "idle" state.
14360: Note race in docs, fix flaky test.
14360: Move single-worker Pool methods to worker type.
14360: Don't block scheduling while locking containers.
14360: Fix false negative in test suite.
14360: Retry shutdown after configured timeout.
14360: Move lockfiles from /var/run to /var/lock.
14360: Merge branch 'master'
14538: Use concurrent writers to sync a directory.
14538: Use concurrent writers to sync multiple streams.
14538: Move cancel-on-first-error logic to contextGroup.
14538: Use concurrent writers to flush chunks while writing.
14538: Check write concurrency limit.
14538: Merge branch 'master' into 14538-async-write
14538: Fix test.
14538: Remove redundant condition.
14538: Reduce max writeahead to 1.
14538: Split inline func from sync() into commitBlock().
14538: Fix racy tests.
11419: Support text modes in Collection.open().
14360: Use nonblocking mode when checking for stale lock.
14360: Fix container UUID missing in proc info file.
14360: Notify pool on worker shutdown.
14360: Merge branch 'master' into 14360-dispatch-cloud
14360: Rename Workers -> CountWorkers.
14360: Disable management APIs if config has no ManagementToken.
14360: Rename -detached -> -no-detach.
14360: Add missing argument to -kill flag.
14360: Kill containers whose priority drops to 0 while running.
14360: Merge branch 'master' into 14360-dispatch-cloud
11419: Merge branch 'master' into 11419-text-mode
14538: Merge branch 'master' into 14538-async-write
Merge branch '14538-async-write'
Merge branch '11419-text-mode'
14360: Fix race between per-container lock and garbage collection.
14596: Use sbatch --no-requeue to avoid bogus container re-runs.
Merge branch '14596-no-requeue'
14596: crunch-run: don't do anything unless container is Locked.
Merge branch '14596-check-container-locked'
11419: Read JSON files in binary mode.
14360: Comment TestDispatchToStubDriver.
14360: Fix log spam on normal race condition.
14360: Locking comment.
14360: Comment stubServer.
14360: Avoid overreporting instances during Create/List race.
14360: Clean up stub driver.
14360: Shutdown between tests to eliminate leaking logs.
Merge branch 'master' into 14360-dispatch-cloud
14360: Remove log spam.
14360: Call ChooseType just once per container.
14360: Replace OnComplete/OnCancel with ExecuteContainer callback.
14360: Shutdown pool between tests to eliminate leaking logs.
14360: Debug log when acquiring lock.
14360: Fix panic in test when last container is run twice.
14360: Fix instance creation limiting.
14360: Fix test case.
14360: Test for errant Create calls after failure.
14360: Fix error log level on first probe after boot.
14360: Merge branch 'master' into 14360-dispatch-cloud
Merge branch '14360-dispatch-cloud'
14325: Propagate API env vars to crunch-run.
14325: Add worker state diagram.
14325: Add IdleBehavior to worker pool.
14325: Add management API endpoints for drain/hold/run behavior.
14325: More testing for hold/drain states.
14325: Configurable SSH target port for cloud VMs.
14325: Add total houly price metric.
14325: Test that ssh_executor obeys a negative VerifyHostKey result.
14325: Fix dropped request params when body not specified by caller.
14325: Test dispatch container queue against controller.
14325: Cancel containers with unsatisfiable runtime constraints.
14325: Use snake_case keys in management API responses.
14325: Include instance ID in log messages from stub VM.
14325: Shutdown unknown-state worker after boot timeout.
14325: Log when a worker's first probe succeeds.
14325: Move magic number to const.
14325: Obey EarliestRetry specified by cloud rate-limiting errors.
14325: Wake up scheduler when quota timeout expires.
14325: Don't shutdown busy VMs even if boot probe fails.
14325: Document Running() return value.
14325: Clean up unsafe concurrency in tests.
14325: Remove obsolete test stubs.
14325: Test resuming worker pool state after restart.
14325: Point to API details in interface doc comments.
14325: Build crunch-dispatch-cloud package.
14325: Don't count busy workers with state=Unknown as Unallocated.
14325: Change remaining LameInstanceSet uses to StubDriver.
14325: Clean up test suite logging.
14325: Fix test.
14325: Note assumption concurrent dispatchers share a VM size menu.
14325: Improve identifiers/comments.
14325: Rephrase confusing conditions and add comments.
14325: Merge branch 'master'
14325: Log stderr from last boot-probe when giving up on boot.
14806: Handle escaped chars when parsing docker image manifest.
14806: Fix test suite to work even if PERL5LIB is set.
14806: Merge branch 'master'
Merge branch '14806-crunch1-unescape-manifest'
14325: Merge branch 'master'
Merge branch '14325-dispatch-cloud'
14325: Rename crunch-dispatch-cloud → arvados-dispatch-cloud.
Merge branch '14325-dispatch-cloud'
14325: Fix PrivateKey config type.
14325: Start up immediately if there are no stale locks.
14828: Fix double-cancel of unsatisfiable container.
14845: Fix inconsistent timeout error message.
Merge branch '14828-fix-flaky-test'
14325: Start up immediately if there are no stale locks.
14807: Match systemd description to component name.
14807: Pass env vars on stdin instead of using SSH feature.
14807: When ProviderType is unspecified, default to Arvados type.
14807: Allow driver to specify SSH username.
14807: Pass SSH public key to driver.
14807: Accept .../instances/_/drain?instance_id=X.
14807: Fix SSH target address.
14807: Expose instance IP addresses in logs and management API.
14807: Log full instance ID.
14807: Always set node-token tag.
14807: Fix crunch-run --list output when /var/lock is a symlink.
14807: Include more detail in errors.
14807: Wait at least 1 second between retries on initial queue poll.
14807: Load API host/token directly from stdin without shell hack.
14807: Don't delete existing tags when updating.
14807: Move secret-tag host key verify mechanism out of Azure driver.
14807: Fix up azure log message.
14807: Split instance count/size/cost metrics by idle/hold status.
14807: Log when a container is added/removed from the queue.
14807: Use context to pass a suitable logger to all service commands.
14807: Cancel or requeue container when priority drops to zero.
14807: Log idle time in seconds instead of nanoseconds.
14807: Update API endpoints: instance_id is always a query param.
14807: Remove errant rm.
Merge branch '14845-fix-flaky-test'
14807: Send detached crunch-run logs to journal via systemd-cat.
14807: Fix admin permissions for containers.
14807: Fix logger type mismatch.
14807: Fix missing import.
14807: Merge branch 'master'
14807: Avoid doing concurrent update requests per container.
14807: Force test SSH server to listen on IPv4.
Merge branch '14807-dispatch-cloud-fixes'
14911: Fix initialization order.
Merge branch '14911-azure-panic'
14807: Start reporting metrics immediately.
Merge branch '14807-zero-metrics'
14920: Fix state==unknown during create/list race.
14920: Improve comment.
14920: Update metrics when worker state changes idle->running.
Merge branch '14920-unknown-booting-race'
14938: Fix unreliable test.
Merge branch '14938-unreliable-test'
Fix error message.
14947: run-tests: add --interactive, skip deps on --skip-install.
14977: Drop containers when the controller says they don't exist.
14977: Kill containers that don't exist according to controller.
14977: Don't try fixStaleLocks until worker pool state is loaded.
14977: Re-order statements for clarity.
Merge branch '14977-kill-if-not-in-queue'
14807: Fix unreliable test.
14807: Fail test if a container gets run twice.
14807: Escalate sigterm->sigkill->drain instance.
14807: Lengthen kill delays in stub driver to test term-kill-drain.
14807: Update tests to new worker/pool interface.
14947: Suggest "install deps" as first interactive command.
14807: Add comments to "error indicates success" code.
14807: Don't send SIGKILL to crunch-run processes.
14947: Install controller (cmd/arvados-server) for integration tests.
14947: Accept test suite options in interactive mode.
14947: Fix controller install in deps.
14947: Ignore --only / --skip args in interactive mode.
Merge branch '14947-run-tests-interactive'
14807: Tweak log message.
14807: Merge branch 'master'
Merge branch '14807-escalate-sigterm'
14807: Drain instances that crunch-run reports broken.
14807: Configurable rate limit for cloud provider API calls.
14807: Add "kill instance" management API.
14807: Report queue metrics.
14807: Give up if initial keep services list isn't loaded in 60s.
14807: Allow some fake nodes to remain unbroken.
Merge branch '14807-prod-blockers'
15007: Fix crash when Azure nic has IP config but no IP address.
Merge branch '15007-azure-panic'
14807: Load API host/token from cluster config if present.
14807: Refuse to create instance with AddedScratch>0.
14807: Pass token to service handlers explicitly, not via context.
14807: Fix go vet warnings about %s args.
14807: Add lib/service test.
14807: Move "fallback to ARVADOS_*" logic out to lib/service.
Merge branch '14807-prod-blockers'
Accept :py3 modifier for any Python suite in interactive mode.
run-tests.sh: don't prepopulate cmd if running in an emacs buffer.
14998: Report total_* (hierarchy) stats if available.
Merge branch '14998-hoststat-total'
15044: Override unusable nginx defaults. Test nginx startup success.
15044: Fix interactive command handling.
15044: Fix error messages.
15044: Always use the en_US.UTF-8 locale. Add sanity check.
15044: Test gofmt as a separate suite, instead of refusing to run Go tests.
15044: Listen on localhost instead of default dual-stack [::].
15044: Improve error reporting in test case.
15044: Fix workbench test args in interactive mode.
15044: Fix unnecessarily unsafe yaml load.
15044: Specify test database collation.
15044: Don't kill run-tests's keepproxy server in sdk/python tests.
15044: Fix sensitivity to group existence on the testing host.
Add another nil check to Azure driver.
14758: Fix panic on error reading request body.
Merge branch '14758-keepproxy-panic'
15044: Require newer mocha.
15044: Sanity-check test services when bringing them up.
Merge branch '15044-test-fixes'
Remove dependency on jq.
Give up if test services don't come up.
14796: Add arvados-dispatch-cloud (experimental) to install docs.
14796: Add Amazon EC2 configuration example.
14796: Tweak example boot probe command.
Merge branch '14796-dispatchcloud-install-docs'
Fix stale cached container state after successful Cancel.
Fix "failed" status leaking to subsequent attempt.
Fix "failed" status leaking to subsequent attempt.
15002: Fix container update permissions.
15002: Test invalid state changes, too.
Merge branch '15002-container-permissions'
15112: Report volumes without replicas as "none" in -dump output.
15112: Use FieldLogger interface so tests can use ctxlog.TestLogger.
15112: Fix mount IDs in test case.
15112: Test multiple volumes with same device IDs.
15112: Show #collections and class->#desired map in dump.
15112: Fix log check in test case.
15112: Error out right away on unusable manifest.
15112: Fix duplicate default.
15112: Stop processing collections on first error.
15112: Refuse to run if modified_at column has any nulls.
15112: Add test.
15112: Note rationale for non-obvious paging approach.
Merge branch '15112-dont-trash-needed-replica'
15112: Fill in missing modified_at values from before #14595.
Merge branch '15112-migration'
Update structure.sql for last migration.
15003: Generate config.DefaultYAML from source file.
15003: Load config over defaults.
15003: Add 'arvados-server dump-config' command.
15003: Use config lib for arvados-server.
15003: Move keys from RequestLimits to API section.
15003: Split out deprecated config stuff.
15003: Check DB conn in controller health check. Check test services.
15003: Add missed error checks.
15003: Error out if PostgreSQL connection map is ambiguous.
15003: Add config-check command.
15003: Update service names.
15003: Warn about unknown keys and bad value types.
15112: Add config option to save list of lost blocks to a file.
15112: Fix fd leak on error.
15112: Remove tempfile only if still locked (not renamed).
Merge branch '15112-save-lost-blocks-file'
14399: Retry Azure ListBlobs call after 503 errors.
14399: Don't insert error text in the middle of an index response.
Merge branch '14399-azure-list-retry'
15003: Merge branch 'master' into 15003-preprocess-config
15148: Include list of affected PDHs in LostBlocksFile.
Merge branch '15148-lost-collection-pdh'
15003: Merge branch 'master' into 15003-preprocess-config
Remove extra \n from log.Printf formats
Log if client hangs up while waiting for unix volume Serialize lock.
15003: Merge branch 'master' into 15003-preprocess-config
Merge branch '15003-preprocess-config'
15232: Fix type of Collections.BlobSigningTTL config key.
Merge branch '15232-blobttl-suffix'
15003: Temporarily disable config warnings.
Merge branch '15003-disable-config-warnings'
15273: Use lowercase keys for database connection params.
15273: Fix key-conflict test.
Merge branch '15273-db-conn-keys'
14878: Skip flock() when testing priority updater.
14878: Add comment.
Merge branch '14878-priority-race'
Tom Morris (26):
Don't add trailing slash
Merge branch '14383-java-sdk-double-slash'. Fixes #14383.
14884: Allow non-ASCII filenames in manifests.
14884: Fix some additional formats where Unicode can be used
Merge branch '14884-unicode-name-support'
Update crunchstat-summary to Python 3. Refs #14939
Fix test Python 3 integer divide difference.
Add crunchstat-summary to Python 3 tests. refs #14939
Fix resource leak warned by Python 3
Resolve deprecation warnings
Merge branch '14939-crunchstat-summary-python3'
Spell MB consistently.
Fix Keep cache recommendation calculation.
Don't recommend RAM or CPU changes based on zero usage. refs #10570
Use default of 1 if no cores requested. refs #12026
Improve wording with direction of recommendation change.
Merge branch '14451-crunchstat-summary-cache-ram'
Add missing Javadoc for overloaded methods.
Reorganize, clean up, and copy edit docs.
Remove Gradle from source repo. Update docs & build to reflect change.
Add crunchstat-summary Python 3 to all tests.
Fix iteration order dependency.
Merge branch '14939-crunchstat-summary-python3-fixup'
Fix elapsed time calculation.
Merge branch '14627-crunchstat-summary-elapsed'
Fix debug setup accidentally committed.
Tomasz Marciniak (1):
Arvados SDK Java - release 2.0.0
Ward Vandewege (57):
Add a few upgrade notes for the 1.3.0 release.
We've moved the chat/community support channel from IRC to gitter.im.
9945: first set of changes.
14691: documentation update
14691: workbench update, bring in line with documentation
Merge branch '14691-suggested-ssh-config-update'
9945: there is a new rvm gpg key
9945: run-library changes
9945: use virtualenv for our python packages
9945: address a bunch of review comments.
9945: address more review comments.
9945: More changes based on review comments.
9945: Merge branch 'master' into 9945-make-python-package-dependency-free
9945: Backwards compatibility fix for libpam-arvados
9945: Merge branch 'master' into 9945-make-python-package-dependency-free
Merge branch '9945-make-python-package-dependency-free'
The libpam-arvados package on Centos7 has a dependency on the python-pam package.
Do not try to test packages that were not built.
We have reset the package iterations to -1, make the
Make the python-arvados-cwl-runner package expose the cwltool executable, make
In the arvados/jobs image, use the Python executable from the
Make sure to bump the dependency on the arvados gems to the 1.3 series.
14853: we need a newer version of setuptools to build on centos7, debian8 and ubuntu1404.
Remove a few files that were checked in erroneously, update the .gitignore
14853: we need a newer version of setuptools to build on centos7, debian8 and ubuntu1404 (part 2).
Some build script housekeeping: make the CentOS builds less noisy, and remove
Change argument order for arv-keepdocker to the one that works, as it is
Ubuntu 18.04 ("bionic") packages are now confirmed working, add it to the list
14819: upgrade the arvados/jobs image to Debian Stretch. Also add argument to
Fix typo.
14819: use the python executable from the python-arvados-cwl-runner package
14819: Merge branch 'master' into 14819-arvados-jobs-on-stretch
Merge branch '14819-arvados-jobs-on-stretch'
Explicitly print the version numbers of virtualenv, pip, setuptools and wheel
Build the cwltest package, which lives out of tree.
Some build script housekeeping: fix the detection logic for the existence of
Switch to stretch-slim as the base image for arvados/jobs, this results in a
13996: rename config.defaults.yml to config.default.yml, to remain consistent
15042: handle config.default.yml in our packaging.
Mention the credential expiry date option in the instructions to create Azure
Make the doc Rakefile a little more helpful when it's missing javadoc
Fix typo.
Fix unit error in the node manager documentation for EC2.
Fix the api server package postinstall script to use `rake config:dump` to get a
Bump the rails package iteration to 9, for the packaging fix in commit
When specified in the api configuration,
Merge branch '15200-fix-blobsigning-config-migration'
Reset the rails package iteration to 1
Our Python SDK now also supports version 3.4+, update the fuse drive doc page
While our Python SDK now supports Python 3.4+, our FUSE driver does not yet. On
Make sure that the libpam-arvados package puts its various files in the place
Move the population of the new columns on the collections table to a standalone
Address review comments.
Merge branch '15093-move-collection-update-out-of-band' into 1.4-dev
Update the 'upgrading' documentation to reflect the v1.4.0 release, and warn
Do not blow up when Rails.configuration.Users.UserProfileNotificationAddress is
Crunch-job needs to call the python executable from our sdk package for
chapmanb (2):
Python 3: only import subprocess32 on py27
Python 3: move to recent ciso8601 2.x branch
ebiagiotti (2):
13306: Adding a py3 variant of the sdk/cwl tests to run-tests.sh
13306: Changes to arvados-cwl-runner code after running futurize --stage1
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list