<div dir="ltr"><div style class="markdown-here-wrapper"><p style="margin:1.2em 0px!important">Hi everyone,</p>
<p style="margin:1.2em 0px!important">I recently merged code to resolve #3720, where Workbench would time out before it could render large collections.  For being “just” a performance issue, we had to make surprising number of changes to code organization to get this fixed, so I wanted to outline those for everyone.</p>
<p style="margin:1.2em 0px!important">The crux of the issue was simple: Collections had a <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">files</code> attribute, an array that lists each file in the Collection along with its location and size.  When the manifest contains many files, it took too long to have the API server parse the entire manifest, and then encode this “alternate view” of the manifest to JSON.</p>
<h2 style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.4em;border-bottom:1px solid rgb(238,238,238)" id="collections-api-changes">Collections API changes</h2>
<p style="margin:1.2em 0px!important">The fix is equally simple to explain: the API server no longer prepares or provides the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">files</code> attribute, for list or get requests.  The related <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">data_size</code> attribute is also gone.  Clients are responsible for their own manifest parsing.  This is still a performance win, because it avoids a JSON encode/decode cycle, and many consumers don’t want or need to parse the entire manifest; they only care about the first few files.</p>
<h2 style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.4em;border-bottom:1px solid rgb(238,238,238)" id="ruby-sdk-changes">Ruby SDK changes</h2>
<p style="margin:1.2em 0px!important">The API server and Workbench both use this file information.  To avoid duplicating code between them, I added a Keep::Manifest class to our Ruby SDK.  This handles common parsing tasks, and both the API server and Workbench use it now.  The API server’s old Locator class became Keep::Locator and moved here too.</p>
<p style="margin:1.2em 0px!important">It sounds a little funny to say that our API server depends on our SDK, but it makes more sense when you remember it’s using the Keep client part of the SDK.</p>
<p style="margin:1.2em 0px!important">Keep::Manifest yields results incrementally as they’re parsed out.  This way, if you only need information from the early part of the manifest, you can get that without the expense of parsing the entire text.</p>
<h2 style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.4em;border-bottom:1px solid rgb(238,238,238)" id="build-changes">Build changes</h2>
<p style="margin:1.2em 0px!important">Jenkins now tests our Ruby SDK, builds the gem, and uses that gem when testing other components like the API server.  This means it takes a little care to manage the servers’ version dependency on the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">arvados</code> Gem, just like we’ve had with the API server and <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">arvados-cli</code> for a while.  I’ve updated our <a href="https://arvados.org/projects/arvados/wiki/Hacking_Ruby_SDK">Ruby SDK wiki page</a> with more information about how you can test your changes before pushing them, and arrange a consistent build with Jenkins.</p>
<p style="margin:1.2em 0px!important">Let me know if you have questions about any of this.</p>
<div title="MDH:SGkgZXZlcnlvbmUsPGJyPjxicj5JIHJlY2VudGx5IG1lcmdlZCBjb2RlIHRvIHJlc29sdmUgIzM3
MjAsIHdoZXJlIFdvcmtiZW5jaCB3b3VsZCB0aW1lIG91dCBiZWZvcmUgaXQgY291bGQgcmVuZGVy
IGxhcmdlIGNvbGxlY3Rpb25zLsKgIEZvciBiZWluZyAianVzdCIgYSBwZXJmb3JtYW5jZSBpc3N1
ZSwgd2UgaGFkIHRvIG1ha2Ugc3VycHJpc2luZyBudW1iZXIgb2YgY2hhbmdlcyB0byBjb2RlIG9y
Z2FuaXphdGlvbiB0byBnZXQgdGhpcyBmaXhlZCwgc28gSSB3YW50ZWQgdG8gb3V0bGluZSB0aG9z
ZSBmb3IgZXZlcnlvbmUuPGJyPjxicj5UaGUgY3J1eCBvZiB0aGUgaXNzdWUgd2FzIHNpbXBsZTog
Q29sbGVjdGlvbnMgaGFkIGEgYGZpbGVzYCBhdHRyaWJ1dGUsIGFuIGFycmF5IHRoYXQgbGlzdHMg
ZWFjaCBmaWxlIGluIHRoZSBDb2xsZWN0aW9uIGFsb25nIHdpdGggaXRzIGxvY2F0aW9uIGFuZCBz
aXplLsKgIFdoZW4gdGhlIG1hbmlmZXN0IGNvbnRhaW5zIG1hbnkgZmlsZXMsIGl0IHRvb2sgdG9v
IGxvbmcgdG8gaGF2ZSB0aGUgQVBJIHNlcnZlciBwYXJzZSB0aGUgZW50aXJlIG1hbmlmZXN0LCBh
bmQgdGhlbiBlbmNvZGUgdGhpcyAiYWx0ZXJuYXRlIHZpZXciIG9mIHRoZSBtYW5pZmVzdCB0byBK
U09OLjxicj48YnI+IyMgQ29sbGVjdGlvbnMgQVBJIGNoYW5nZXM8YnI+PGJyPlRoZSBmaXggaXMg
ZXF1YWxseSBzaW1wbGUgdG8gZXhwbGFpbjogdGhlIEFQSSBzZXJ2ZXIgbm8gbG9uZ2VyIHByZXBh
cmVzIG9yIHByb3ZpZGVzIHRoZSBgZmlsZXNgIGF0dHJpYnV0ZSwgZm9yIGxpc3Qgb3IgZ2V0IHJl
cXVlc3RzLsKgIFRoZSByZWxhdGVkIGBkYXRhX3NpemVgIGF0dHJpYnV0ZSBpcyBhbHNvIGdvbmUu
wqAgQ2xpZW50cyBhcmUgcmVzcG9uc2libGUgZm9yIHRoZWlyIG93biBtYW5pZmVzdCBwYXJzaW5n
LsKgIFRoaXMgaXMgc3RpbGwgYSBwZXJmb3JtYW5jZSB3aW4sIGJlY2F1c2UgaXQgYXZvaWRzIGEg
SlNPTiBlbmNvZGUvZGVjb2RlIGN5Y2xlLCBhbmQgbWFueSBjb25zdW1lcnMgZG9uJ3Qgd2FudCBv
ciBuZWVkIHRvIHBhcnNlIHRoZSBlbnRpcmUgbWFuaWZlc3Q7IHRoZXkgb25seSBjYXJlIGFib3V0
IHRoZSBmaXJzdCBmZXcgZmlsZXMuPGJyPjxicj4jIyBSdWJ5IFNESyBjaGFuZ2VzPGJyPjxicj5U
aGUgQVBJIHNlcnZlciBhbmQgV29ya2JlbmNoIGJvdGggdXNlIHRoaXMgZmlsZSBpbmZvcm1hdGlv
bi7CoCBUbyBhdm9pZCBkdXBsaWNhdGluZyBjb2RlIGJldHdlZW4gdGhlbSwgSSBhZGRlZCBhIEtl
ZXA6Ok1hbmlmZXN0IGNsYXNzIHRvIG91ciBSdWJ5IFNESy7CoCBUaGlzIGhhbmRsZXMgY29tbW9u
IHBhcnNpbmcgdGFza3MsIGFuZCBib3RoIHRoZSBBUEkgc2VydmVyIGFuZCBXb3JrYmVuY2ggdXNl
IGl0IG5vdy7CoCBUaGUgQVBJIHNlcnZlcidzIG9sZCBMb2NhdG9yIGNsYXNzIGJlY2FtZSBLZWVw
OjpMb2NhdG9yIGFuZCBtb3ZlZCBoZXJlIHRvby48YnI+PGJyPkl0IHNvdW5kcyBhIGxpdHRsZSBm
dW5ueSB0byBzYXkgdGhhdCBvdXIgQVBJIHNlcnZlciBkZXBlbmRzIG9uIG91ciBTREssIGJ1dCBp
dCBtYWtlcyBtb3JlIHNlbnNlIHdoZW4geW91IHJlbWVtYmVyIGl0J3MgdXNpbmcgdGhlIEtlZXAg
Y2xpZW50IHBhcnQgb2YgdGhlIFNESy48YnI+PGJyPktlZXA6Ok1hbmlmZXN0IHlpZWxkcyByZXN1
bHRzIGluY3JlbWVudGFsbHkgYXMgdGhleSdyZSBwYXJzZWQgb3V0LsKgIFRoaXMgd2F5LCBpZiB5
b3Ugb25seSBuZWVkIGluZm9ybWF0aW9uIGZyb20gdGhlIGVhcmx5IHBhcnQgb2YgdGhlIG1hbmlm
ZXN0LCB5b3UgY2FuIGdldCB0aGF0IHdpdGhvdXQgdGhlIGV4cGVuc2Ugb2YgcGFyc2luZyB0aGUg
ZW50aXJlIHRleHQuPGJyPjxicj4jIyBCdWlsZCBjaGFuZ2VzPGJyPjxicj5KZW5raW5zIG5vdyB0
ZXN0cyBvdXIgUnVieSBTREssIGJ1aWxkcyB0aGUgZ2VtLCBhbmQgdXNlcyB0aGF0IGdlbSB3aGVu
IHRlc3Rpbmcgb3RoZXIgY29tcG9uZW50cyBsaWtlIHRoZSBBUEkgc2VydmVyLsKgIFRoaXMgbWVh
bnMgaXQgdGFrZXMgYSBsaXR0bGUgY2FyZSB0byBtYW5hZ2UgdGhlIHNlcnZlcnMnIHZlcnNpb24g
ZGVwZW5kZW5jeSBvbiB0aGUgYGFydmFkb3NgIEdlbSwganVzdCBsaWtlIHdlJ3ZlIGhhZCB3aXRo
IHRoZSBBUEkgc2VydmVyIGFuZCBgYXJ2YWRvcy1jbGlgIGZvciBhIHdoaWxlLsKgIEkndmUgdXBk
YXRlZCBvdXIgW1J1YnkgU0RLIHdpa2kgcGFnZV0oaHR0cHM6Ly9hcnZhZG9zLm9yZy9wcm9qZWN0
cy9hcnZhZG9zL3dpa2kvSGFja2luZ19SdWJ5X1NESykgd2l0aCBtb3JlIGluZm9ybWF0aW9uIGFi
b3V0IGhvdyB5b3UgY2FuIHRlc3QgeW91ciBjaGFuZ2VzIGJlZm9yZSBwdXNoaW5nIHRoZW0sIGFu
ZCBhcnJhbmdlIGEgY29uc2lzdGVudCBidWlsZCB3aXRoIEplbmtpbnMuPGJyPjxicj5MZXQgbWUg
a25vdyBpZiB5b3UgaGF2ZSBxdWVzdGlvbnMgYWJvdXQgYW55IG9mIHRoaXMuPGJyPjxicj4=" style="height:0;font-size:0em;padding:0;margin:0">​</div></div></div>