[ARVADOS] updated: 2.1.0-1552-gf2788dd5f

[Git user]

Summary of changes:
 doc/admin/upgrading.html.textile.liquid |  11 ++
 lib/config/config.default.yml           |  36 ++++++
 lib/config/export.go                    |   2 +
 lib/config/generated_config.go          |  36 ++++++
 lib/config/load.go                      |  10 ++
 lib/crunchrun/background.go             |  54 +++++---
 lib/crunchrun/bufthenwrite.go           |  34 +++++
 lib/crunchrun/crunchrun.go              | 216 +++++++++++++++++++++++++++-----
 lib/crunchrun/integration_test.go       |  82 +++++++++++-
 lib/crunchrun/logging.go                |  51 ++++++++
 lib/crunchrun/logging_test.go           |  34 +++++
 lib/dispatchcloud/node_size.go          |   1 +
 lib/dispatchcloud/test/stub_driver.go   |  11 +-
 lib/dispatchcloud/worker/pool.go        |   2 +
 lib/dispatchcloud/worker/pool_test.go   |  70 ++++++-----
 lib/dispatchcloud/worker/runner.go      |  20 +--
 lib/dispatchcloud/worker/worker_test.go |  21 +++-
 sdk/go/arvados/config.go                |   2 +
 services/keep-web/handler_test.go       |  32 ++---
 services/keepstore/azure_blob_volume.go |   3 +
 services/keepstore/handler_test.go      |  31 ++---
 services/keepstore/handlers.go          |  31 +----
 22 files changed, 631 insertions(+), 159 deletions(-)
 create mode 100644 lib/crunchrun/bufthenwrite.go

       via  f2788dd5fc3ef725969d6c0fcc4ddee3754829fc (commit)
       via  54836b787450bf23abcf7be291831799093a17b0 (commit)
       via  39a723673e92b842233b1da5fde27aa595fcc59f (commit)
       via  83c996d75698093446fbfff89ea4abeb36cbc8c4 (commit)
       via  623ffe7320d0a159ba9a5912fc1948cac12ad505 (commit)
       via  68259bcde57277cb709296fc24e86826d9c131d5 (commit)
       via  b7aa839c669f1c2055ade415b99378e5c9ba0baa (commit)
       via  a0080bdf353bcc107eb80a5755958c6c276d49eb (commit)
       via  53b2e5895715c73febffb563ebc89153339e02ab (commit)
       via  c8d252f51c23484484e4aa023fcd1f86ee961eab (commit)
       via  bda9093be4d24d45a6fff29148fbb5438e283897 (commit)
       via  6b39f96311c832de21bcacc3f17a611682d522a9 (commit)
       via  5f95bfc2c5c7706c7961aeca3aabd90ea5661f0a (commit)
       via  e04586707b923f857c61a17ad38a9ce795e9af14 (commit)
       via  f968e4a8eeb562964f265d7555eaf295f8bac7f6 (commit)
       via  146c1e2c5d12f478e4dbf35c5a675f05a793f5b7 (commit)
       via  c6e6a2531ef791e7fdae10b440c778244dd62019 (commit)
       via  9a615740476541185073f98d8bc6e69e06ecf340 (commit)
       via  6b3a880d607ee3e3dd273f019981fd6cae62373c (commit)
      from  e1e1a215d954c3337a969019da7f7becc8d14012 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.


commit f2788dd5fc3ef725969d6c0fcc4ddee3754829fc
Merge: e1e1a215d 54836b787
Author: Tom Clegg <tom at curii.com>
Date:   Thu Oct 28 10:14:09 2021 -0400

    Merge branch '16347-container-local-keepstore'
    
    closes #16347
    
    Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom at curii.com>

diff --cc doc/admin/upgrading.html.textile.liquid
index 304eedb1d,170e2c51b..aee2b581b
--- a/doc/admin/upgrading.html.textile.liquid
+++ b/doc/admin/upgrading.html.textile.liquid
@@@ -35,9 -35,20 +35,20 @@@ TODO: extract this information based o
  <div class="releasenotes">
  </notextile>
  
+ h2(#main). development main (as of 2021-10-27)
+ 
 -"Upgrading from 2.3.0":#v2_3_0
++"previous: Upgrading from 2.3.0":#v2_3_0
+ 
+ h3. Dedicated keepstore process for each container
+ 
+ When Arvados runs a container via @arvados-dispatch-cloud@, the @crunch-run@ supervisor process now brings up its own keepstore server to handle I/O for mounted collections, outputs, and logs. With the default configuration, the keepstore process allocates one 64 MiB block buffer per VCPU requested by the container. For most workloads this will increase throughput, reduce total network traffic, and make it possible to run more containers at once without provisioning additional keepstore nodes to handle the I/O load.
+ * If you have containers that can effectively handle multiple I/O threads per VCPU, consider increasing the @Containers.LocalKeepBlobBuffersPerVCPU@ value.
+ * If you already have a robust permanent keepstore infrastructure, you can set @Containers.LocalKeepBlobBuffersPerVCPU@ to 0 to disable this feature and preserve the previous behavior of sending container I/O traffic to your separately provisioned keepstore servers.
+ * This feature is enabled only if no volumes use @AccessViaHosts@, and no volumes have underlying @Replication@ less than @Collections.DefaultReplication at . If the feature is configured but cannot be enabled due to an incompatible volume configuration, this will be noted in the @crunch-run.txt@ file in the container log.
+ 
  h2(#v2_3_0). v2.3.0 (2021-10-27)
  
 -"Upgrading from 2.2.0":#v2_2_0
 +"previous: Upgrading from 2.2.0":#v2_2_0
  
  h3. Ubuntu 18.04 packages for arvados-api-server and arvados-workbench now conflict with ruby-bundler
  

-----------------------------------------------------------------------


hooks/post-receive
--