<div dir="ltr"><b style="font-weight:normal" id="gmail-docs-internal-guid-0aa70bf5-7fff-eff0-c423-1e725a4ce18d"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(34,34,34);background-color:rgb(255,255,255);font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We recently discovered an issue which was introduced in the Arvados v1.3 release which can cause certain collections to not be tallied correctly during block garbage collection accounting. This can, in turn, make it look like blocks are not being referenced and are candidates for deletion. This only affects collections created as output from container, not those created by the Arvados Workbench or arv-put. This only affects installations where the keep-balance service is running with the -commit-trash command line flag (which is the default) and the keepstore(s) are running with EnableDelete enabled (which is not the default).</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(34,34,34);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We are testing a fix for this which will be included an Arvados v1.3.2 hotfix release as soon as it's ready, but in the mean time, we strongly recommend that customers turn off block garbage collection. The Arvados DevOps team has already done this for all customers with support contracts, so no action is necessary on your part.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(34,34,34);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">You can minimize impact by preserving data that has been incorrectly trashed but not yet deleted. On each keepstore server node, update the config file /etc/arvados/keepstore/keepstore.yml, setting the TrashCheckInterval to a very long time. Then restart keepstore.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(34,34,34);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">    TrashCheckInterval: 87600h</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(34,34,34);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Also stop the keep-balance daemon on whatever machine it's running on.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(34,34,34);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">    $ systemctl --now disable keep-balance</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(34,34,34);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We will send a followup note shortly when we have a patched version of the software available, as well as recommendations on data recovery.</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(34,34,34);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(34,34,34);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Best regards,</span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(34,34,34);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Tom Morris</span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font face="Arial"><span style="font-size:14.6667px;white-space:pre-wrap">Director, Product Management</span></font></p><br></b><br></div>