[arvados] Triggering Keep garbage collection

Thu Mar 2 18:50:44 EST 2017

I tried running the keep-balance and it does not seem to delete the actual
blocks.  I am using the superuser token created during the installation,
i.e. via sudo -u apache RAILS_ENV=production bundle exec
./script/create_superuser_token.rb

When I set keep-balance.yml to the following:

Client:
  APIHost: "<REMOVED>"
  AuthToken: "<REMOVED>"
  Insecure: false
RunPeriod: 1h
KeepServiceTypes:
    - disk
CollectionBatchSize: 100000
CollectionBuffers: 1000

I recieve the following from keep-balance

[root at arvados-dev ~]# keep-balance -once -commit-trash
2017/03/02 17:38:17 Run: start
2017/03/02 17:38:17 skipping cridv-bi6l4-jq4hc7zhh8ipkvt with service type
"proxy"
2017/03/02 17:38:17 clearing existing trash lists, in case the new
rendezvous order differs from previous run
2017/03/02 17:38:17 cridv-bi6l4-pb945ioz1yrqkq9 (172.17.1.117:25107, disk):
send trash list: start
2017/03/02 17:38:17 cridv-bi6l4-42p7ovknp27lsj1 (172.17.1.121:25107, disk):
send trash list: start
2017/03/02 17:38:17 cridv-bi6l4-42p7ovknp27lsj1 (172.17.1.121:25107, disk):
send trash list: took 2.153563ms
2017/03/02 17:38:17 cridv-bi6l4-pb945ioz1yrqkq9 (172.17.1.117:25107, disk):
send trash list: took 2.200265ms
2017/03/02 17:38:17 cridv-bi6l4-42p7ovknp27lsj1 (172.17.1.121:25107, disk):
send trash list: request failed: {http  <nil> 172.17.1.121:25107 /trash
 %!s(bool=false)  }: 401 Unauthorized
2017/03/02 17:38:17 cridv-bi6l4-pb945ioz1yrqkq9 (172.17.1.117:25107, disk):
send trash list: request failed: {http  <nil> 172.17.1.117:25107 /trash
 %!s(bool=false)  }: 401 Unauthorized
2017/03/02 17:38:17 Run: took 123.919044ms
2017/03/02 17:38:17 cridv-bi6l4-pb945ioz1yrqkq9 (172.17.1.117:25107, disk):
send trash list: request failed: {http  <nil> 172.17.1.117:25107 /trash
 %!s(bool=false)  }: 401 Unauthorized

When I set the keep-balance.yml to the following:

Client:
  APIHost: "<REMOVED>"
  AuthToken: "<REMOVED>"
  Insecure: false
RunPeriod: 1h
KeepServiceTypes: null
CollectionBatchSize: 100000
CollectionBuffers: 1000

I receive the following from keep-balance

[root at arvados-dev ~]# keep-balance -once -commit-trash
2017/03/02 17:40:54 Run: start
2017/03/02 17:40:54 skipping cridv-bi6l4-42p7ovknp27lsj1 with service type
"disk"
2017/03/02 17:40:54 skipping cridv-bi6l4-jq4hc7zhh8ipkvt with service type
"proxy"
2017/03/02 17:40:54 skipping cridv-bi6l4-pb945ioz1yrqkq9 with service type
"disk"
2017/03/02 17:40:54 GetCurrentState: start
2017/03/02 17:40:54 collections: 0/21
2017/03/02 17:40:54 collections: 21/21
2017/03/02 17:40:54 collections: 21/21
2017/03/02 17:40:54 GetCurrentState: took 208.558523ms
2017/03/02 17:40:54 ComputeChangeSets: start
2017/03/02 17:40:54 ComputeChangeSets: took 515.354µs
2017/03/02 17:40:54 ===
2017/03/02 17:40:54 1118 replicas (559 blocks, 70393357406 bytes) lost
(0=have<want)
2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) underreplicated
(0<have<want)
2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) just right (have=want)
2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) overreplicated
(have>want>0)
2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) unreferenced
(have>want=0, new)
2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) garbage (have>want=0,
old)
2017/03/02 17:40:54 ===
2017/03/02 17:40:54 1118 replicas (559 blocks, 70393357406 bytes) total
commitment (excluding unreferenced)
2017/03/02 17:40:54 0 replicas (0 blocks, 0 bytes) total usage
2017/03/02 17:40:54 ===
2017/03/02 17:40:54 ===
2017/03/02 17:40:54 Replication level distribution (counting N replicas on
a single server as N):
2017/03/02 17:40:54  0: 559
##########################################################
2017/03/02 17:40:54  1:   0
2017/03/02 17:40:54 ===
2017/03/02 17:40:54 Run: took 332.677865ms

Any help would be greatly appreciated.

Thanks!

George Chlipala, Ph.D.
Senior Research Specialist
Research Resources Center
University of Illinois at Chicago

phone: 312-413-1700
email: gchlip2 at uic.edu

On Mon, Feb 13, 2017 at 4:08 PM, George Chlipala <gchlip2 at uic.edu> wrote:

> Peter -
>
> Thanks for the information!
>
> Is there a recommended user to use for the API token?
>
>
>
> George Chlipala, Ph.D.
> Senior Research Specialist
> Research Resources Center
> University of Illinois at Chicago
>
> phone: 312-413-1700 <(312)%20413-1700>
> email: gchlip2 at uic.edu
>
> On Mon, Feb 13, 2017 at 1:45 PM, Peter Amstutz <
> peter.amstutz at curoverse.com> wrote:
>
>> Hi George,
>>
>> The garbage collection service is called "keep-balance" and is located
>> at arvados/services/keep-balance.  It is responsible for both deleting
>> unreferenced blocks and rebalancing blocks among storage servers.
>>
>> Unfortunately it doesn't seems to be properly documented.  It replaces
>> an older implementation called "data manager".
>>
>> Thanks,
>> Peter
>>
>>
>> On Mon, Feb 13, 2017 at 2:35 PM, George Chlipala <gchlip2 at uic.edu> wrote:
>> > Is there any information about tuning/configuring garbage collection
>> for the
>> > keep data manager?  Or is garbage collection not yet implemented?
>> >
>> >
>> > George Chlipala, Ph.D.
>> > Senior Research Specialist
>> > Research Resources Center
>> > University of Illinois at Chicago
>> >
>> > phone: 312-413-1700
>> > email: gchlip2 at uic.edu
>> >
>> > On Thu, Jan 26, 2017 at 11:57 AM, George Chlipala <gchlip2 at uic.edu>
>> wrote:
>> >>
>> >> Is a way to trigger the keep data manager to perform garbage
>> collection or
>> >> if it is possible to know/modify the garbage collection cycle?
>> >>
>> >> We were performing a deletion test from within arvados to see if the
>> keep
>> >> would delete blocks that are no longer used.  We had uploaded new data
>> and
>> >> saw that the disk usage for the keep store had increased by 8GB.
>> However,
>> >> when we deleted the collection from the arvados workbench, we did not
>> see
>> >> the disk usage decrease by the same amount.  Between the adding and
>> deletion
>> >> we did not upload any additional data that might reference the new
>> blocks as
>> >> well, so I would expect that the keep store should release that space.
>> >>
>> >> Any help in this matter would be greatly appreciated.
>> >>
>> >> Thanks!
>> >>
>> >> George Chlipala, Ph.D.
>> >> Senior Research Specialist
>> >> Research Resources Center
>> >> University of Illinois at Chicago
>> >>
>> >> phone: 312-413-1700
>> >> email: gchlip2 at uic.edu
>> >
>> >
>> >
>> > _______________________________________________
>> > arvados mailing list
>> > arvados at arvados.org
>> > http://lists.arvados.org/mailman/listinfo/arvados
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.arvados.org/pipermail/arvados/attachments/20170302/46e9f49d/attachment.html>