[GET-dev] trait-o-matic's mysql queries seem unnecessarily slow

Alexander Wait Zaranek awaitz at post.harvard.edu
Thu Sep 16 15:02:29 EDT 2010


On Thu, Sep 16, 2010 at 2:55 PM, Madeleine Price Ball <meprice at gmail.com> wrote:
> I suspect all the other steps in Trait-o-matic processing can be improved in
> the same way: simultaneously moving through pre-sorted files rather than
> loading one into MySQL and then querying MySQL.
I wrote code for another project that does exactly this.  In that
project, UNIX sort / IO / compression was the bottleneck for
data-processing.

We might be able to do better than pre-sorting but, clearly, it's an
improvement over the existing method.

Sasha




More information about the Arvados mailing list