[ARVADOS] created: c718d729795f423e4c1254b781a881da7506fe2f

[git at public.curoverse.com]

        at  c718d729795f423e4c1254b781a881da7506fe2f (commit)


commit c718d729795f423e4c1254b781a881da7506fe2f
Author: mishaz <misha at curoverse.com>
Date:   Sat Jan 24 02:22:01 2015 +0000

    A bunch of changes, most in response to Peter's review.
    
    Logger:
    Edit() and Record() have been replaced with the single Update() method which takes a function as input (suggested by Tom).
    lastWrite replaced by nextWriteAllowed, for cleaner logic
    Added writeScheduled to reduce the number of writes scheduled and attempted, thereby reducing lock contention
    Added sanity-checking of params
    A bunch of overdue cleanup
    Update documentation to reflect the above changes
    
    Manifest:
    Renamed ManifestLine to ManifestStream
    
    Util:
    Deleted a lot of crap that proved less useful than I thought.
    Moved collection.NumberCollectionsAvailable() to util.NumberItemsAvailable() and made it more generic.
    
    collection:
    Just cleanup in response to changes in above packages.
    
    keep:
    Switched Mtime from int to int64 to avoid y2038 problems.
    Switched approach for avoiding keep proxy from using "accessible" to filtering on service_type = disk.
    Cleanup in response to changes in above packages.
    
    loggerutil:
    Cleanup in response to changes in logger.

commit fc696c2f99353a1de472f4620f51d345988d7d27
Author: mishaz <misha at curoverse.com>
Date:   Wed Jan 21 01:34:29 2015 +0000

    Added comment, ran gofmt.

commit f48f9250f4bf1c7326c1379bc7c6cf490ad87d0d
Author: mishaz <misha at curoverse.com>
Date:   Wed Jan 21 01:31:17 2015 +0000

    Added Logger.MutateLog() on Tom's suggestion. Tried it out in one instance to make sure it works.

commit f1491aab5b94ac37410d71eeb40a9479c8a5d952
Author: mishaz <misha at curoverse.com>
Date:   Wed Jan 14 01:40:50 2015 +0000

    Finished adding logging to keep.GetServerContents but have not tested fully yet.

commit 0856c5303ed068c25403eae359c985d5601b6866
Author: mishaz <misha at curoverse.com>
Date:   Wed Jan 14 00:50:21 2015 +0000

    ran gofmt

commit 70a227b4ebc27c64bb64ecb596c1ae2959dd4135
Author: mishaz <misha at curoverse.com>
Date:   Wed Jan 14 00:49:31 2015 +0000

    Broke keep.GetServerContents() into smaller functions.

commit a31092d05a4c5b72f1a7ca2218a3f45a1fc780a0
Author: mishaz <misha at curoverse.com>
Date:   Tue Jan 13 23:56:05 2015 +0000

    Removed channel awareness from keep.GetServerContents().

commit 439a4e1b1a1736083151d10f0365c1b16aea4d0a
Author: mishaz <misha at curoverse.com>
Date:   Tue Jan 13 23:27:05 2015 +0000

    gofmt'd all my source code. No other changes.

commit 821f806abae252f5d5762416e0766747126bff14
Author: mishaz <misha at curoverse.com>
Date:   Tue Jan 13 01:15:34 2015 +0000

    Started logging in keep.go. More work to be done.

commit e4b8d0b51358c88f527e52e55dae9b83a60d9e35
Author: mishaz <misha at curoverse.com>
Date:   Tue Jan 13 00:49:53 2015 +0000

    Created loggerutil to hold common datamanager logger code. Moved FatalWithMessage to it.

commit e921c415ea3c8eb9a2fafa8ca9c9e5cceb53b2c4
Author: mishaz <misha at curoverse.com>
Date:   Tue Jan 13 00:23:17 2015 +0000

    Moved some logic from datamanager.go to keep.go.

commit bc50ca6971dc13c84dc518b40887ab30ebe5d941
Author: mishaz <misha at curoverse.com>
Date:   Mon Jan 12 23:20:50 2015 +0000

    Started reading collections and keep data in parallel. Moved some logic from datamanager.go to collections.go. Added logging to end of run.

commit aef8ae1e9a50ec6a39141aa91c2b4e8e2da7697c
Merge: 89ecce9 61fdce2
Author: mishaz <misha at curoverse.com>
Date:   Sun Jan 11 18:29:13 2015 +0000

    Merge branch 'master' of git.curoverse.com:arvados into 3408-production-datamanager
    
    Conflicts resolved:
    	services/api/Gemfile
    	services/api/Gemfile.lock

commit 89ecce95bc435c32027db8ee9f77d705c089719b
Author: mishaz <misha at curoverse.com>
Date:   Fri Jan 9 04:53:42 2015 +0000

    Added recording of fatal errors to logger.

commit ee278406810bc2522802cd33dcf94d93fb1fc913
Author: mishaz <misha at curoverse.com>
Date:   Fri Jan 9 04:11:29 2015 +0000

    Added ForceRecord() method to enable writing remaining log changes before exiting.

commit bd001d4c8be314d3905f0b7c7087ba4d82b94e79
Author: mishaz <misha at curoverse.com>
Date:   Fri Jan 9 04:00:40 2015 +0000

    Switched Logger edit hooks to write hooks so they'll trigger less often.

commit dbeccd1b8575ba96506cb74d04c56f71c127f776
Author: mishaz <misha at curoverse.com>
Date:   Thu Jan 8 23:22:09 2015 +0000

    Switched batch size to 50. Added logging of batch size.

commit fb76b0b8e1890ec27cce3ec5cac42fff07a97294
Author: mishaz <misha at curoverse.com>
Date:   Thu Jan 8 23:17:52 2015 +0000

    Added memory alloc in use to stats exported to log. Also added EditHooks to Logger, enabling users to add functions to get called on each Edit() call.

commit eb5d5d99b6694423852ec229935fac8376403712
Author: mishaz <misha at curoverse.com>
Date:   Thu Jan 8 22:35:37 2015 +0000

    Added structure to data manager log entries, grouping similar fields.

commit 0c9832f8d1af07c85ab84b6e42b61aa094543dcf
Author: mishaz <misha at curoverse.com>
Date:   Thu Jan 8 22:24:00 2015 +0000

    Added ability to turn off logging by passing an empty string as the event type.

commit e94c6eb56982e236daeffc19b9def869f14fbab4
Author: mishaz <misha at curoverse.com>
Date:   Thu Jan 8 22:16:49 2015 +0000

    Started using Logger in data manager and collections.

commit 9f6c09436da853d56db4922ee1c6e54b3c4b36d6
Author: mishaz <misha at curoverse.com>
Date:   Thu Jan 8 21:06:49 2015 +0000

    Added support for MinimumWriteInterval.

commit a50b8ad5773b13db45070205af0754b88ed2ce25
Author: mishaz <misha at curoverse.com>
Date:   Thu Jan 8 20:14:10 2015 +0000

    Fixed bugs in logger, changed interface some, added documentation.
    Still need to add support for MinimunWriteInterval.

commit 10fda8ea470a9e83bd1d349b33b43d1f684a35cb
Author: mishaz <misha at curoverse.com>
Date:   Thu Jan 8 01:47:51 2015 +0000

    Added logger to write log messages that grow over time. Not working yet.

commit 1dd5156a83dc05d959ccffe1567271889954d5a1
Author: mishaz <misha at curoverse.com>
Date:   Wed Jan 7 04:16:40 2015 +0000

    Started focusing on Keep Server responses again. Switched to using blockdigest instead of strings. Added per block info so that we can track block replication across servers.

commit df69d7a558943f25b9cb2fb215e3fad665c8d242
Author: mishaz <misha at curoverse.com>
Date:   Wed Jan 7 01:45:55 2015 +0000

    Fixed heap profile writing so that we overwrite previous heap profiles rather than adding to them. Minor cleanup too.

commit 9e327b69799585a69c3e6dc997a3d870337e1e5b
Author: mishaz <misha at curoverse.com>
Date:   Wed Dec 24 20:26:38 2014 +0000

    Added string copying to try to reduce memory usage, didn't seem to work. Cleaned up logging (and logging logic) so that we only see one line per batch.

commit 1e7f49ed49f458ab3f9b62145760a14bb86d1bb5
Author: mishaz <misha at curoverse.com>
Date:   Wed Dec 24 19:29:08 2014 +0000

    Started parsing modification date as a timestamp instead of leaving it as a string.

commit d35d6189a3dc06ec86ef19927860986a2ef97ac3
Author: mishaz <misha at curoverse.com>
Date:   Wed Dec 24 01:36:43 2014 +0000

    Switched from strings to BlockDigests to hold block digests more efficiently. Started clearing out manifest text once we finished with it. Made profiling conitional on flag (before it crashed if not provided). Added final heap profile once collections were finished.
    
    Runs to completion!

commit 30412881e854e9255c237f3ca3c4686e48e8e7a3
Author: mishaz <misha at curoverse.com>
Date:   Wed Dec 24 00:24:07 2014 +0000

    Changes to manifest that I forgot to add to previous checking.

commit 8a2cf36f8254bee55141155a4b9054f4178f5cae
Author: mishaz <misha at curoverse.com>
Date:   Tue Dec 23 23:55:12 2014 +0000

    Added blockdigest class to store digests more efficiently. This has the nice side effect of reducing how many string slices we use from the SDK, so the large string can get garbage collected once we remove other usages.

commit fecfe4c45fb498b22ada515ff4d01544e8dbeb94
Author: mishaz <misha at curoverse.com>
Date:   Tue Dec 23 19:33:07 2014 +0000

    Long overdue checkin of data manager. Current code runs, but uses way too much memory and eventually crashes. This checkin includes heap profiling to track down memory usage.

commit 4b59ecbe561ebd0f41d3c44697cb527fbf993720
Author: mishaz <misha at curoverse.com>
Date:   Sat Nov 22 00:57:40 2014 +0000

    Added reporting of disk usage. This is the Collection Storage of each user as described here: https://arvados.org/projects/arvados/wiki/Data_Manager_Design_Doc#Reports-Produced
    But it does not include the size of projects owned by the user (projects and subprojects are each reported as their own users)
    
    Report is just logged to screen for now.

commit 509065a0f0156cb3c402a37efe20943b4a1fc50e
Author: mishaz <misha at curoverse.com>
Date:   Thu Oct 16 20:57:06 2014 +0000

    Started reading index from keep servers.
    Added lots of code to handle unexpected results from keep server.

commit f048f2dda540064a5c4f0aff16e0e726bf621ef4
Author: mishaz <misha at curoverse.com>
Date:   Wed Oct 15 20:53:53 2014 +0000

    Started reading response from keep server.

commit 7136493014aea9787497464269d9aaddf1830e97
Merge: 74dba22 df54f41
Author: mishaz <misha at curoverse.com>
Date:   Tue Oct 14 19:47:32 2014 +0000

    Merge branch 'master' of git.curoverse.com:arvados into 3408-production-datamanager

commit 74dba22bf46c375d6d8daa8e5afff3e55df993ae
Author: mishaz <misha at curoverse.com>
Date:   Tue Oct 14 19:46:12 2014 +0000

    Added flags to read data manager token from a file.
    Started trying to get index from keep servers but it's not working yet.

commit df54f4111b0afaaf073c7387dff5960bf19a53e2
Author: Brett Smith <brett at curoverse.com>
Date:   Tue Oct 14 14:49:49 2014 -0400

    4126: API server uses fixed a-r-p-i.
    
    Refs #4126.

commit 593316198c1647b24f66a64e4a0f7a34e75b54ab
Merge: debedb2 b8fdb90
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Tue Oct 14 11:41:02 2014 -0400

    Merge branch '3692-event-bus-fix-and' closes #3692

commit debedb222cfea4bfc4519032cad8475858e04034
Merge: 26d74dc 9b50fd8
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Tue Oct 14 11:33:31 2014 -0400

    Merge branch '3656-arv-create' closes #3656

commit 26d74dc0524c87c5dcc0c76040ce413a4848b57a
Author: Tim Pierce <twp at curoverse.com>
Date:   Tue Oct 14 10:58:18 2014 -0400

    Adding Keep slides from BarCamp Boston 9.
    
    No issue #.

commit 1a0303d4cb6e1870e958f0121c26a79e0116af64
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 10 20:59:55 2014 +0000

    Deleted unused type.

commit 8370f2aee294e270c1f89379af813510fb06079e
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 10 20:27:36 2014 +0000

    Added tests to check that we're iterating on manifest lines correctly and handling blank lines in manifests.
    LineIter will now handle long manifest lines properly and added test to check that we continue to do so.

commit ded3f79df5f4079f29ad8a4d1a1d0dd3438aaba3
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 10 15:28:09 2014 +0000

    Added test to show that our code fails on long manifests.

commit efaf24041034cbc5de09d80283bd98f3e6434854
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 3 22:30:30 2014 +0000

    Reported scanner errors, which I forgot to do before (and was hiding real problems).

commit 512e652c33bb3a03b46ec630521fb9d6df6585b7
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 3 21:57:06 2014 +0000

    Switched to using structs rather than maps to hold SDK responses and code is much easier, much less noisy parsing code.
    Added warning to SDK documentation about field name capitalization.
    Switched from using keep_disks to keep_services, since keep_services actually has all the server addresses.

commit d44a5c508cfa664134daad806d7be9a7cb0bd6ee
Merge: 828f692 60998a3
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 3 19:49:24 2014 +0000

    Merge branch 'master' of git.curoverse.com:arvados into 3408-production-datamanager

commit 828f692c3ada2691d5c86dff79d91e7e4e2cdda0
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 3 02:17:01 2014 +0000

    Moved Keep code to its own package.
    Added lots of keep code (not currently working because API server is returning hosts with empty service_host and service_port).

commit 023798117b11e4c3f3b68003999dd6d27c9d131e
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 3 00:51:33 2014 +0000

    Created sdk util package and moved helpful methods there.

commit 3dcb5f41ca5e20004ca3d042958246bd341101bb
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 3 00:42:09 2014 +0000

    Added some code to deal with keep disks.

commit 4d670eff4c565d07b5d61d43b9fba8fd121281e5
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 3 00:39:57 2014 +0000

    Created helper method to deal with iterating through list returned by SDK.

commit a9034a1baae24357fa49e9dc4ef25922c1ec6a90
Author: mishaz <misha at curoverse.com>
Date:   Fri Oct 3 00:38:39 2014 +0000

    Fixed bug where error was dropped on the floor!

commit ffb5d12c1670a1dd8f86eca7e80283b0c37a8656
Author: mishaz <misha at curoverse.com>
Date:   Thu Oct 2 22:54:49 2014 +0000

    Added params struct for GetCollections
    Added periodic logging of collections processed
    Factored out logic for checking whether all available items are returned

commit 29fe4fbeb9eb0a2bd748fa6fcaac803682172e48
Author: mishaz <misha at curoverse.com>
Date:   Mon Sep 29 17:39:41 2014 +0000

    Moved manifest package from services/datamanager to sdk/go

commit 043e2d7fa61b4954f0d84c6e4281e37a4f6f865f
Author: mishaz <misha at curoverse.com>
Date:   Thu Sep 25 21:12:37 2014 +0000

    Responded to Tim's comments.
    Switched to using bufio.scanner, some other cleanup and added some tests.

commit 49064ef7e89a6b4e9010199d79314a98a9c90132
Author: mishaz <misha at curoverse.com>
Date:   Thu Sep 25 01:34:44 2014 +0000

    Updated code in response to Tim's Review.

commit fb0cbc2cf28f7dea6efa112685ee6074cef0d629
Author: mishaz <misha at curoverse.com>
Date:   Tue Sep 23 01:41:57 2014 +0000

    Committing just to get some input on my Go Style.
    
    Everything compiles.
    manifest_test runs and passes.
    datamanger runs successfully.

-----------------------------------------------------------------------


hooks/post-receive
--