From tfmorris at veritasgenetics.com Fri Dec 7 13:06:10 2018 From: tfmorris at veritasgenetics.com (Tom Morris) Date: Fri, 7 Dec 2018 13:06:10 -0500 Subject: [arvados] Arvados 1.3 release & 2019 planning Message-ID: The Arvados team is happy to announce the immediate availability of our latest quarterly release, Arvados 1.3 . It contains a total of 392 commits by 9 different authors. Two major new features in this release are: - Distributed workflows across multiple clusters in a federation - Collection versions but there are a number of other features and enhancements as well as a bunch of bug fixes, which you can find enumerated in the release notes below. As we come to the end of a successful 2018 and look forward to the new year, we're interested in to hear your input on what the core team and the Arvados community in general on should focus on for 2019. We'll be collecting feedback over the next 4-6 weeks and using it to extend the Arvados roadmap and adjust priorities. Let us know what's on your mind and what features are important to you. Best wishes for the holidays and New Year! The Arvados Team ---------- Arvados 1.3 Release Notes The Arvados 1.3 release includes two major new features: 1) distributed workflows across a federation of Arvados clusters and 2) Collection versioning. There are also a number of smaller features and improvements, as well as a bunch of bug fixes. Please review the Upgrading Arvados page for any special upgrade notes. Source code: https://github.com/curoverse/arvados/releases Packages for supported distributions: https://apt.arvados.org and https://rpm.arvados.org New Features in Arvados 1.3 Below is a description of the new features which are included in this release. Distributed Workflows Arvados was designed to support the federation of multiple clusters from Day 1 and now the fruits of all that design and implementation work are becoming visible. This release includes the ability to run workflows which are distributed across multiple clusters. In use cases where it?s impossible or impractical to centralize the data, it is now possible to do the computations in a distributed fashion, co-located with the data, and then aggregate the results. You can read more about building distributed workflows here . Federated Identity A user can now have the same identity across all the clusters in a federation (as defined by the system administrator), simplifying access control, sharing, etc. This is one of the key supporting pieces for the distributed workflow feature in this release. You can find information on migrating users to federated identity here . Collection Versions On a configurable basis, collections can now be versioned so that you can easily inspect the changes which have been made to them over time. Arvados has always stored detailed audit logs of every change made to objects in the system, but those aren?t conveniently accessible. This new feature allows the system administrator to turn on collection versioning and configure which old versions are preserved automatically, and allows users and API clients to easily browse those versions. Collection versioning is described in more detail here . New Workflow Manager (Containers API / Crunch2) We implemented an entirely new workflow manager several releases ago and have been running it internally for over a year as well as at a number of customer sites. We?ve put together a document describing the migration process to guide customers. If you haven?t already migrated, we recommend that you plan to migrate as soon as convenient. The new subsystem offers substantial improvements in scalability and reliability, as well as a number of nice new features. Improved WebDAV support The WebDAV support has been enhanced with the ability to write as well as read and improved compatibility for Windows and Mac OS X clients. ?Failing? Status for Containers Workflows which have child jobs that have failed, and thus are doomed to fail themselves eventually, are now tagged with a special status so that users and monitoring software can have early insight into problem jobs. Crunchstat-summary We made a number of small improvements to this performance analysis tool, especially around support for the Containers API and the enhanced logging that Crunch 2 supports. See tickets 12690 , 12748 , 13759 in the list below for more details. . Full list of tickets for Arvados 1.3 10181 Crunch job output logging improvement stories 10865 [Crunch2] [Crunch1] Disable crunch1 APIs by default on new installs 13493 Federated record retrieval 13561 [API] Store, and add APIs to retrieve, previous versions of collection objects 13619 [Controller] Federated multi object retrieval 13773 "Will fail" status for failing (but not yet failed) containers 13993 [API] Fetch remote-hosted collection by UUID 13994 [Keepstore] Fetch blocks from federated clusters 14087 [controller] Fetch remote-hosted collections by PDH 14196 [API] issues v2 format auth tokens 14197 [controller] Federated container requests 14198 [CWL] run steps on remote clusters 14199 [keepstore] copy block from remote keepstore to local keepstore 14201 [API] Accept "exclude_home_project" flag in groups#contents 14259 [SDK] Python collection class uses copy remote block to local keepstore 14260 [API] Add "runtime_token" field to container_requests 14262 [Controller] Specify runtime_token when creating container requests on a remote cluster 14285 [keep-balance] metrics endpoint 14323 [API] Accept container mounts that specify both uuid and portable_data_hash 14406 [SDK] Go collection uses copy remote block to local keepstore 14086 [keep-web] Serve previous collection versions 14242 [controller] Verify hash on manifest text when querying by PDH 14299 [keep-balance] Ensure blocks referenced by old collection versions are safe from garbage collection 14366 Crunch 1 to Crunch 2 migration documentation 14440 Write user documentation for writing and running distributed workflows 14483 Update ruaml version for a-c-r 12690 crunchstat-summary --container-request does not report on keep i/o 12748 [crunch2] crunchstat-summary doesn't report elapsed time for crunch2 jobs 13759 crunchstat-summary Error parsing value 14236 [WebDAV] Can't delete the last file in a collection 14274 "FactoryGirl" deprecation warning in API tests 14377 [API] 401 on 4xphq with generated token 14398 [keep-balance] deadlock on index retrieval error 14407 [Workbench] tests fail on current version of firefox 14441 [sso] merge account feature issues 14458 [controller] collection federation panic send on closed channel 14475 [Controller] POST .../collections/$uuid seems to have broken Workbench integration tests 14507 [controller] database ping failed 14516 [api] merged account usernames should be taken into account when calculating the next available username 14519 [Workbench] Textile table markup [webdav] Windows/Mac compatibility -- empty folders and locking CR Priority does not propagate to children [c-d-s] validation needs to be done if outdirMin is a number with decimals [keepstore] S3 support for AWS4-HMAC-SHA256 required for new AWS data centers Container fails when initial working directory and output share a directory [1.2.0] search performance regression [keep-balance] reporting or replication accounting bug [keepstore] while deleting a lot of blocks: fatal error: concurrent map writes [keepstore] Misleading log messages say Deleted but mean "marked as trash" The tickets without numbers are private tickets which contain customer information or other private data. -------------- next part -------------- An HTML attachment was scrubbed... URL: