[arvados] Arvados 1.3 release & 2019 planning

Tom Morris tfmorris at veritasgenetics.com
Fri Dec 7 13:06:10 EST 2018 

The Arvados team is happy to announce the immediate availability of our
latest quarterly release, Arvados 1.3
<https://dev.arvados.org/rb/release/14>. It contains a total of 392 commits
by 9 different authors.

Two major new features in this release are:

   - Distributed workflows across multiple clusters in a federation
   - Collection versions

but there are a number of other features and enhancements as well as a
bunch of bug fixes, which you can find enumerated in the release notes
below.

As we come to the end of a successful 2018 and look forward to the new
year, we're interested in to hear your input on what the core team and the
Arvados community in general on should focus on for 2019. We'll be
collecting feedback over the next 4-6 weeks and using it to extend the
Arvados roadmap and adjust priorities.

Let us know what's on your mind and what features are important to you.

Best wishes for the holidays and New Year!

The Arvados Team

----------

Arvados 1.3 Release Notes

The Arvados 1.3 release includes two major new features: 1) distributed
workflows across a federation of Arvados clusters and 2) Collection
versioning. There are also a number of smaller features and improvements,
as well as a bunch of bug fixes.

Please review the Upgrading Arvados page
<https://doc.arvados.org/admin/upgrading.html> for any special upgrade
notes.

Source code:  https://github.com/curoverse/arvados/releases

Packages for supported distributions: https://apt.arvados.org and
https://rpm.arvados.org
New Features in Arvados 1.3

Below is a description of the new features which are included in this
release.
Distributed Workflows

Arvados was designed to support the federation of multiple clusters from
Day 1 and now the fruits of all that design and implementation work are
becoming visible. This release includes the ability to run workflows which
are distributed across multiple clusters. In use cases where it’s
impossible or impractical to centralize the data, it is now possible to do
the computations in a distributed fashion, co-located with the data, and
then aggregate the results.

You can read more about building distributed workflows here
<https://doc.arvados.org/user/cwl/federated-workflows.html>.
Federated Identity

A user can now have the same identity across all the clusters in a
federation (as defined by the system administrator), simplifying access
control, sharing, etc. This is one of the key supporting pieces for the
distributed workflow feature in this release. You can find information on
migrating users to federated identity here
<https://doc.arvados.org/admin/merge-remote-account.html>.
Collection Versions

On a configurable basis, collections can now be versioned so that you can
easily inspect the changes which have been made to them over time. Arvados
has always stored detailed audit logs of every change made to objects in
the system, but those aren’t conveniently accessible. This new feature
allows the system administrator to turn on collection versioning and
configure which old versions are preserved automatically, and allows users
and API clients to easily browse those versions. Collection versioning is
described in more detail here
<https://doc.arvados.org/user/topics/collection-versioning.html>.

New Workflow Manager (Containers API / Crunch2)

We implemented an entirely new workflow manager several releases ago and
have been running it internally for over a year as well as at a number of
customer sites. We’ve put together a document
<https://doc.arvados.org/admin/upgrade-crunch2.html> describing the
migration process to guide customers. If you haven’t already migrated, we
recommend that you plan to migrate as soon as convenient. The new subsystem
offers substantial improvements in scalability and reliability, as well as
a number of nice new features.
Improved WebDAV support

The WebDAV support has been enhanced with the ability to write as well as
read and improved compatibility for Windows
<https://doc.arvados.org/user/tutorials/tutorial-keep-mount-windows.html>
and Mac OS X
<https://doc.arvados.org/user/tutorials/tutorial-keep-mount-os-x.html>
clients.
“Failing” Status for Containers

Workflows which have child jobs that have failed, and thus are doomed to
fail themselves eventually, are now tagged with a special status so that
users and monitoring software can have early insight into problem jobs.
Crunchstat-summary

We made a number of small improvements to this performance analysis tool,
especially around support for the Containers API and the enhanced logging
that Crunch 2 supports. See tickets 12690
<https://dev.arvados.org/issues/12690>, 12748
<https://dev.arvados.org/issues/12748>, 13759
<https://dev.arvados.org/issues/13759> in the list below for more details.

.

Full list of tickets for Arvados 1.3 <https://dev.arvados.org/rb/release/14>

10181 <https://dev.arvados.org/issues/10181>

Crunch job output logging improvement stories

10865 <https://dev.arvados.org/issues/10865>

[Crunch2] [Crunch1] Disable crunch1 APIs by default on new installs

13493 <https://dev.arvados.org/issues/13493>

Federated record retrieval

13561 <https://dev.arvados.org/issues/13561>

[API] Store, and add APIs to retrieve, previous versions of collection
objects

13619 <https://dev.arvados.org/issues/13619>

[Controller] Federated multi object retrieval

13773 <https://dev.arvados.org/issues/13773>

"Will fail" status for failing (but not yet failed) containers

13993 <https://dev.arvados.org/issues/13993>

[API] Fetch remote-hosted collection by UUID

13994 <https://dev.arvados.org/issues/13994>

[Keepstore] Fetch blocks from federated clusters

14087 <https://dev.arvados.org/issues/14087>

[controller] Fetch remote-hosted collections by PDH

14196 <https://dev.arvados.org/issues/14196>

[API] issues v2 format auth tokens

14197 <https://dev.arvados.org/issues/14197>

[controller] Federated container requests

14198 <https://dev.arvados.org/issues/14198>

[CWL] run steps on remote clusters

14199 <https://dev.arvados.org/issues/14199>

[keepstore] copy block from remote keepstore to local keepstore

14201 <https://dev.arvados.org/issues/14201>

[API] Accept "exclude_home_project" flag in groups#contents

14259 <https://dev.arvados.org/issues/14259>

[SDK] Python collection class uses copy remote block to local keepstore

14260 <https://dev.arvados.org/issues/14260>

[API] Add "runtime_token" field to container_requests

14262 <https://dev.arvados.org/issues/14262>

[Controller] Specify runtime_token when creating container requests on a
remote cluster

14285 <https://dev.arvados.org/issues/14285>

[keep-balance] metrics endpoint

14323 <https://dev.arvados.org/issues/14323>

[API] Accept container mounts that specify both uuid and portable_data_hash

14406 <https://dev.arvados.org/issues/14406>

[SDK] Go collection uses copy remote block to local keepstore

14086 <https://dev.arvados.org/issues/14086>

[keep-web] Serve previous collection versions

14242 <https://dev.arvados.org/issues/14242>

[controller] Verify hash on manifest text when querying by PDH

14299 <https://dev.arvados.org/issues/14299>

[keep-balance] Ensure blocks referenced by old collection versions are safe
from garbage collection

14366 <https://dev.arvados.org/issues/14366>

Crunch 1 to Crunch 2 migration documentation

14440 <https://dev.arvados.org/issues/14440>

Write user documentation for writing and running distributed workflows

14483 <https://dev.arvados.org/issues/14483>

Update ruaml version for a-c-r

12690 <https://dev.arvados.org/issues/12690>

crunchstat-summary --container-request does not report on keep i/o

12748 <https://dev.arvados.org/issues/12748>

[crunch2] crunchstat-summary doesn't report elapsed time for crunch2 jobs

13759 <https://dev.arvados.org/issues/13759>

crunchstat-summary Error parsing value

14236 <https://dev.arvados.org/issues/14236>

[WebDAV] Can't delete the last file in a collection

14274 <https://dev.arvados.org/issues/14274>

"FactoryGirl" deprecation warning in API tests

14377 <https://dev.arvados.org/issues/14377>

[API] 401 on 4xphq with generated token

14398 <https://dev.arvados.org/issues/14398>

[keep-balance] deadlock on index retrieval error

14407 <https://dev.arvados.org/issues/14407>

[Workbench] tests fail on current version of firefox

14441 <https://dev.arvados.org/issues/14441>

[sso] merge account feature issues

14458 <https://dev.arvados.org/issues/14458>

[controller] collection federation panic send on closed channel

14475 <https://dev.arvados.org/issues/14475>

[Controller] POST .../collections/$uuid seems to have broken Workbench
integration tests

14507 <https://dev.arvados.org/issues/14507>

[controller] database ping failed

14516 <https://dev.arvados.org/issues/14516>

[api] merged account usernames should be taken into account when
calculating the next available username

14519 <https://dev.arvados.org/issues/14519>

[Workbench] Textile table markup

[webdav] Windows/Mac compatibility -- empty folders and locking

CR Priority does not propagate to children

[c-d-s] validation needs to be done if outdirMin is a number with decimals

[keepstore] S3 support for AWS4-HMAC-SHA256 required for new AWS data
centers

Container fails when initial working directory and output share a directory

[1.2.0] search performance regression

[keep-balance] reporting or replication accounting bug

[keepstore] while deleting a lot of blocks: fatal error: concurrent map
writes

[keepstore] Misleading log messages say Deleted but mean "marked as trash"

The tickets without numbers are private tickets which contain customer
information or other private data.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.arvados.org/pipermail/arvados/attachments/20181207/50234313/attachment-0001.html>

More information about the arvados mailing list