<div dir="ltr">I tried running the keep-balance and it does not seem to delete the actual blocks. I am using the superuser token created during the installation, i.e. via sudo -u apache RAILS_ENV=production bundle exec ./script/create_superuser_token.rb<div><br></div><div>When I set keep-balance.yml to the following:<br></div><div><br></div><div><div>Client:</div><div> APIHost: "<REMOVED>"</div><div> AuthToken: "<REMOVED>"</div><div> Insecure: false</div><div>RunPeriod: 1h</div><div>KeepServiceTypes:</div><div> - disk</div><div>CollectionBatchSize: 100000</div><div>CollectionBuffers: 1000</div></div><div><br></div><div>I recieve the following from keep-balance</div><div><br></div><div><div>[root@arvados-dev ~]# keep-balance -once -commit-trash</div><div>2017/03/02 17:38:17 Run: start</div><div>2017/03/02 17:38:17 skipping cridv-bi6l4-jq4hc7zhh8ipkvt with service type "proxy"</div><div>2017/03/02 17:38:17 clearing existing trash lists, in case the new rendezvous order differs from previous run</div><div>2017/03/02 17:38:17 cridv-bi6l4-pb945ioz1yrqkq9 (<a href="http://172.17.1.117:25107">172.17.1.117:25107</a>, disk): send trash list: start</div><div>2017/03/02 17:38:17 cridv-bi6l4-42p7ovknp27lsj1 (<a href="http://172.17.1.121:25107">172.17.1.121:25107</a>, disk): send trash list: start</div><div>2017/03/02 17:38:17 cridv-bi6l4-42p7ovknp27lsj1 (<a href="http://172.17.1.121:25107">172.17.1.121:25107</a>, disk): send trash list: took 2.153563ms</div><div>2017/03/02 17:38:17 cridv-bi6l4-pb945ioz1yrqkq9 (<a href="http://172.17.1.117:25107">172.17.1.117:25107</a>, disk): send trash list: took 2.200265ms</div><div>2017/03/02 17:38:17 cridv-bi6l4-42p7ovknp27lsj1 (<a href="http://172.17.1.121:25107">172.17.1.121:25107</a>, disk): send trash list: request failed: {http <nil> <a href="http://172.17.1.121:25107">172.17.1.121:25107</a> /trash %!s(bool=false) }: 401 Unauthorized</div><div>2017/03/02 17:38:17 cridv-bi6l4-pb945ioz1yrqkq9 (<a href="http://172.17.1.117:25107">172.17.1.117:25107</a>, disk): send trash list: request failed: {http <nil> <a href="http://172.17.1.117:25107">172.17.1.117:25107</a> /trash %!s(bool=false) }: 401 Unauthorized</div><div>2017/03/02 17:38:17 Run: took 123.919044ms</div><div>2017/03/02 17:38:17 cridv-bi6l4-pb945ioz1yrqkq9 (<a href="http://172.17.1.117:25107">172.17.1.117:25107</a>, disk): send trash list: request failed: {http <nil> <a href="http://172.17.1.117:25107">172.17.1.117:25107</a> /trash %!s(bool=false) }: 401 Unauthorized</div></div><div><br></div><div><div>When I set the keep-balance.yml to the following:</div><div><br></div><div><div>Client:</div><div> APIHost: "<REMOVED>"</div><div> AuthToken: "<REMOVED>"</div><div> Insecure: false</div><div>RunPeriod: 1h</div><div>KeepServiceTypes: null</div><div>CollectionBatchSize: 100000<br></div><div>CollectionBuffers: 1000</div></div><div><br></div><div>I receive the following from keep-balance</div><div><br></div><div><div>[root@arvados-dev ~]# keep-balance -once -commit-trash</div><div>2017/03/02 17:40:54 Run: start</div><div>2017/03/02 17:40:54 skipping cridv-bi6l4-42p7ovknp27lsj1 with service type "disk"</div><div>2017/03/02 17:40:54 skipping cridv-bi6l4-jq4hc7zhh8ipkvt with service type "proxy"</div><div>2017/03/02 17:40:54 skipping cridv-bi6l4-pb945ioz1yrqkq9 with service type "disk"</div><div>2017/03/02 17:40:54 GetCurrentState: start</div><div>2017/03/02 17:40:54 collections: 0/21</div><div>2017/03/02 17:40:54 collections: 21/21</div><div>2017/03/02 17:40:54 collections: 21/21</div><div>2017/03/02 17:40:54 GetCurrentState: took 208.558523ms</div><div>2017/03/02 17:40:54 ComputeChangeSets: start</div><div>2017/03/02 17:40:54 ComputeChangeSets: took 515.354µs</div><div>2017/03/02 17:40:54 ===</div><div>2017/03/02 17:40:54 1118 replicas (559 blocks, 70393357406 bytes) lost (0=have<want)</div><div>2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) underreplicated (0<have<want)</div><div>2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) just right (have=want)</div><div>2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) overreplicated (have>want>0)</div><div>2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) unreferenced (have>want=0, new)</div><div>2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) garbage (have>want=0, old)</div><div>2017/03/02 17:40:54 ===</div><div>2017/03/02 17:40:54 1118 replicas (559 blocks, 70393357406 bytes) total commitment (excluding unreferenced)</div><div>2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) total usage</div><div>2017/03/02 17:40:54 ===</div><div>2017/03/02 17:40:54 ===</div><div>2017/03/02 17:40:54 Replication level distribution (counting N replicas on a single server as N):</div><div>2017/03/02 17:40:54 0: 559 ##########################################################</div><div>2017/03/02 17:40:54 1: 0</div><div>2017/03/02 17:40:54 ===</div><div>2017/03/02 17:40:54 Run: took 332.677865ms</div></div><div><br></div><div><br></div><div>Any help would be greatly appreciated.</div><div><br></div><div>Thanks!</div><div><br></div></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><br>George Chlipala, Ph.D.<br>Senior Research Specialist<br>Research Resources Center<br>University of Illinois at Chicago<br><br>phone: 312-413-1700<br>email: <a href="mailto:gchlip2@uic.edu" target="_blank">gchlip2@uic.edu</a></div></div>
<br><div class="gmail_quote">On Mon, Feb 13, 2017 at 4:08 PM, George Chlipala <span dir="ltr"><<a href="mailto:gchlip2@uic.edu" target="_blank">gchlip2@uic.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Peter -<div><br></div><div>Thanks for the information!</div><div><br></div><div>Is there a recommended user to use for the API token?</div><div><br></div></div><div class="gmail_extra"><span class=""><br clear="all"><div><div class="m_8583623672189757067gmail_signature" data-smartmail="gmail_signature"><br>George Chlipala, Ph.D.<br>Senior Research Specialist<br>Research Resources Center<br>University of Illinois at Chicago<br><br>phone: <a href="tel:(312)%20413-1700" value="+13124131700" target="_blank">312-413-1700</a><br>email: <a href="mailto:gchlip2@uic.edu" target="_blank">gchlip2@uic.edu</a></div></div>
<br></span><div><div class="h5"><div class="gmail_quote">On Mon, Feb 13, 2017 at 1:45 PM, Peter Amstutz <span dir="ltr"><<a href="mailto:peter.amstutz@curoverse.com" target="_blank">peter.amstutz@curoverse.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi George,<br>
<br>
The garbage collection service is called "keep-balance" and is located<br>
at arvados/services/keep-balance.<wbr> It is responsible for both deleting<br>
unreferenced blocks and rebalancing blocks among storage servers.<br>
<br>
Unfortunately it doesn't seems to be properly documented. It replaces<br>
an older implementation called "data manager".<br>
<br>
Thanks,<br>
Peter<br>
<div><div class="m_8583623672189757067h5"><br>
<br>
On Mon, Feb 13, 2017 at 2:35 PM, George Chlipala <<a href="mailto:gchlip2@uic.edu" target="_blank">gchlip2@uic.edu</a>> wrote:<br>
> Is there any information about tuning/configuring garbage collection for the<br>
> keep data manager? Or is garbage collection not yet implemented?<br>
><br>
><br>
> George Chlipala, Ph.D.<br>
> Senior Research Specialist<br>
> Research Resources Center<br>
> University of Illinois at Chicago<br>
><br>
> phone: <a href="tel:312-413-1700" value="+13124131700" target="_blank">312-413-1700</a><br>
> email: <a href="mailto:gchlip2@uic.edu" target="_blank">gchlip2@uic.edu</a><br>
><br>
> On Thu, Jan 26, 2017 at 11:57 AM, George Chlipala <<a href="mailto:gchlip2@uic.edu" target="_blank">gchlip2@uic.edu</a>> wrote:<br>
>><br>
>> Is a way to trigger the keep data manager to perform garbage collection or<br>
>> if it is possible to know/modify the garbage collection cycle?<br>
>><br>
>> We were performing a deletion test from within arvados to see if the keep<br>
>> would delete blocks that are no longer used. We had uploaded new data and<br>
>> saw that the disk usage for the keep store had increased by 8GB. However,<br>
>> when we deleted the collection from the arvados workbench, we did not see<br>
>> the disk usage decrease by the same amount. Between the adding and deletion<br>
>> we did not upload any additional data that might reference the new blocks as<br>
>> well, so I would expect that the keep store should release that space.<br>
>><br>
>> Any help in this matter would be greatly appreciated.<br>
>><br>
>> Thanks!<br>
>><br>
>> George Chlipala, Ph.D.<br>
>> Senior Research Specialist<br>
>> Research Resources Center<br>
>> University of Illinois at Chicago<br>
>><br>
>> phone: <a href="tel:312-413-1700" value="+13124131700" target="_blank">312-413-1700</a><br>
>> email: <a href="mailto:gchlip2@uic.edu" target="_blank">gchlip2@uic.edu</a><br>
><br>
><br>
><br>
</div></div>> ______________________________<wbr>_________________<br>
> arvados mailing list<br>
> <a href="mailto:arvados@arvados.org" target="_blank">arvados@arvados.org</a><br>
> <a href="http://lists.arvados.org/mailman/listinfo/arvados" rel="noreferrer" target="_blank">http://lists.arvados.org/mailm<wbr>an/listinfo/arvados</a><br>
><br>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>