[arvados] Happy New Year from Curoverse!

[Tim Pierce]

Happy New Year, everyone!  We're poking our heads up through the snow here
in New England to bring you our latest engineering updates!

*Collection API*

One of the new features to come out of our most recent sprint is a revised
Collection API <http://arvados.org/issues/4823> that more closely resembles
the classic POSIX filesystem API. The Arvados storage system, Keep
<http://arvados.org/projects/arvados/wiki/Keep>, is a content-addressed
storage system that doesn't offer a POSIX interface for accessing data.  A few
sprints ago
<https://arvados.org/projects/arvados/repository/revisions/7bf8f6c701e28e574c137b0c942522e8f8ee4d8c>,
we released an interface that presents Arvados data streams as "file-like
objects", but the overall collection API still just isn't very similar to
the POSIX calls that Unix and Linux programmers are so familiar with.

Our new release brings us much closer to that goal, offering users a single
API for both reading and writing collections, with familiar methods for
addressing files like open(), rename(), remove() and so on. We anticipate
that new users will find it much easier to get into the flow of using
Arvados with these patterns, and existing users will find it more
convenient to port pipelines to Arvados.

*Crunch failure reporting*

Another useful new tool is a Crunch job failure report
<https://arvados.org/issues/4598>. At present, while Crunch reports every
job failure and success into Arvados, identifying the underlying causes of
failed jobs can be a little tedious.  Moreover, a breakdown report that
gives visibility into why jobs have failed can be a huge boon to debugging.

With this tool, an administrator can now set up a nightly report breaking
down job failures over the last 24 hours -- or get a report over any time
period at all.  We've already used this tool in development to help quickly
debug knotty job failures, and expect that Arvados administrators
everywhere will find it a great help.

*New User Interfaces*

We're also experimenting with newer ways of writing more responsive
interfaces.  Our primary interface to Arvados, Workbench
<http://doc.arvados.org/user/getting_started/workbench.html>, is written as
a Rails <http://rubyonrails.org/> application that uses the Arvados API
instead of a local database. This makes it easy for Rails developers to
work on, but also means that every request to Arvados goes through two
software layers, which doubles the latency for every action a user takes.

Since December we've been working with newer JavaScript frameworks like
Mithril <http://lhorie.github.io/mithril/> and Angular
<https://angularjs.org/> to build a pure JavaScript interface to Arvados
that eliminates the need for a Workbench middle layer entirely.  Our
recently-launched collection uploader tool is written in Angular, and we've
used Mithril to build an experimental administrative interface we're
calling "Backstage."  The improved responsiveness we're getting from these
tools is really showing a lot of promise, and we hope to have lots more to
show you soon!

Thanks to everyone for your ongoing input. Please give us a shout if you
have any questions or idea for us!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.arvados.org/pipermail/arvados/attachments/20150112/c766b356/attachment.html>