[arvados] created: 2.6.0-619-g56add69a08

Fri Sep 15 20:48:15 UTC 2023

at  56add69a087e8c94f3e8c99fb04501f7f570aaeb (commit)


commit 56add69a087e8c94f3e8c99fb04501f7f570aaeb
Author: Peter Amstutz <peter.amstutz at curii.com>
Date:   Fri Sep 15 16:47:12 2023 -0400

    20955: Update documentation about the new and legacy container logging systems
    
    Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>

diff --git a/doc/admin/logs-table-management.html.textile.liquid b/doc/admin/logs-table-management.html.textile.liquid
index ef794054a7..0f127831c0 100644
--- a/doc/admin/logs-table-management.html.textile.liquid
+++ b/doc/admin/logs-table-management.html.textile.liquid
@@ -14,10 +14,14 @@ This page aims to provide insight about managing the ever growing API Server's l
 
 h3. Logs table purpose & behavior
 
-This database table currently serves three purposes:
-* It's an audit log, permitting admins and users to look up the time and details of past changes to Arvados objects via @arvados.v1.logs.*@ endpoints.
-* It's a mechanism for passing cache-invalidation events, used by websocket servers, the Python SDK "events" library, and @arvados-cwl-runner@ to detect when an object has changed.
-* It's a staging area for stdout/stderr text coming from users' containers, permitting users to see what their containers are doing while they are still running (i.e., before those text files are written to Keep).
+This database table is accessed viad the via "the @logs@ endpoint.":../api/methods/logs.html
+
+This table currently serves several purposes:
+
+* Audit logging, permitting admins and users to look up the time and details of past changes to Arvados objects.
+* Logging other system events, specifically "file uploads and downloads from keep-web.":restricting-upload-download.html#audit_logs
+* The source for cache-invalidation events, published through websockets to Workbench to refresh the view.  It can also be monitored by the Python SDK "events module.":../sdk/python/events.html
+* Prior to Arvados 2.7, it was used a staging area for stdout/stderr text coming from users' containers, permitting users to see what their containers are doing while they are still running (i.e., before those text files are written to Keep).  Starting with Arvados 2.7, this is superceded by a more efficient mechanism and so these logs are disabled by default.  See "2.7.0 upgrade notes":upgrading.html#v2_7_0 for details.
 
 As a result, this table grows indefinitely, even on sites where policy does not require an audit log; making backups, migrations, and upgrades unnecessarily slow and painful.
 
diff --git a/doc/admin/restricting-upload-download.html.textile.liquid b/doc/admin/restricting-upload-download.html.textile.liquid
index 44a0467cf4..b6b2be4372 100644
--- a/doc/admin/restricting-upload-download.html.textile.liquid
+++ b/doc/admin/restricting-upload-download.html.textile.liquid
@@ -148,7 +148,7 @@ This policy is suitable for an installation where data is being shared with a gr
 </pre>
 
 
-h2. Accessing the audit log
+h2(#audit_log). Accessing the audit log
 
 When @WebDAVLogEvents@ is enabled, uploads and downloads of files are logged in the Arvados audit log. These events are included in the "User Activity Report":user-activity.html. The audit log can also be accessed via the API, SDKs or command line. For example, to show the 100 most recent file downloads:
 
@@ -166,4 +166,4 @@ File upload in Workbench 1 and the @arv-get@ and @arv-put@ tools use @Keepproxy@
 Jul 20 15:03:38 workbench.xxxx1.arvadosapi.com keepproxy[63828]: {"level":"info","locator":"abcdefghijklmnopqrstuvwxyz012345+53251584","msg":"Block download","time":"2021-07-20T15:03:38.458792300Z","user_full_name":"Albert User","user_uuid":"ce8i5-tpzed-abcdefghijklmno"}
 </pre>
 
-It is possible to do a reverse lookup from the locator to find all matching collections: the @manifest_text@ field of a collection lists all the block locators that are part of the collection. The @manifest_text@ field also provides the relevant filename in the collection. Because this lookup is rather involved and there is no automated tool to do it, we recommend disabling @KeepproxyPermission/User/Download@ and @KeepproxyPermission/User/Upload@ for sites where the audit log is important and @arv-get@ and @arv-put@ are not essential.
+It is possible to do a reverse lookup from the locator to find all matching collections: the @manifest_text@ field of a collection lists all the block locators that are part of the collection. The @manifest_text@ field also provides the relevant filename in the collection. Because this lookup is rather involved and there is no automated tool to do it, we recommend disabling @KeepproxyPermission.User.Download@ and @KeepproxyPermission.User.Upload@ for sites where the audit log is important and @arv-get@ and @arv-put@ are not essential.
diff --git a/doc/admin/upgrading.html.textile.liquid b/doc/admin/upgrading.html.textile.liquid
index d51e39eafc..28a1db1fd6 100644
--- a/doc/admin/upgrading.html.textile.liquid
+++ b/doc/admin/upgrading.html.textile.liquid
@@ -36,10 +36,20 @@ h2(#v2_7_0). v2.7.0 (2023-09-??)
 
 "previous: Upgrading to 2.6.3":#v2_6_3
 
+h3. New system for live container logs
+
+Starting with Arvados 2.7, a new system for fetching live container logs is in place.  This system features significantly reduced database load compared to previous releases.  When Workbench or another application need to access the logs of a process (running or completed), they should use the "log endpoint of container_requests.":https://doc.arvados.org/main/api/methods/container_requests.html which forwards requests to the running container.  This supercedes the previous system where compute processes would send all of their logs to the database, which produced significant database load.
+
+The legacy logging system is now disabled by default for all installations with the setting @Containers.Logging.LimitLogBytesForJob: 0 at .  If you have an existing Arvados installation where you have customized this value and do not need the legacy container logging system, we recommend removing @LimitLogBytesForJob@ from your configuration.
+
+If you need to re-enable the legacy logging system, set @Containers.Logging.LimitLogBytesForJob@ to a positive value (the previous default was @Containers.Logging.LimitLogBytesForJob: 67108864@).
+
 h3. Workbench 1 deprecated
 
 The original Arvados Workbench application (referred to as "Workbench 1") is deprecated and will be removed in a future major version of Arvados.  Users are advised to migrate to "Workbench 2".  Starting with this release, new installations of Arvados will only set up Workbench 2 and no longer include Workbench 1 by default.
 
+It is also important to note that Workbench 1 only supports the legacy logging system, which is now disabled by default.  If you need to re-enable the legacy logging system, see above.
+
 h3. Multi-node installer's domain name configuration changes
 
 The @domain_name@ variable at @terraform/vpc/terraform.tfvars@ and @DOMAIN@ variable at @local.params@ changed their meaning. In previous versions they were used in combination with @cluster_name@ and @CLUSTER@ to build the cluster's domain name (e.g.: @cluster_name at .@domain_name@). To allow the use of any arbitrary cluster domain, now we don't enforce using the cluster prefix as part of the domain, so @domain_name@ and @DOMAIN@ need to hold the entire domain for the given cluster.

commit 2cf2121b848c11f5753c54d89415dae67a6f037d
Author: Peter Amstutz <peter.amstutz at curii.com>
Date:   Fri Sep 15 13:47:51 2023 -0400

    20955: Reorder upgrade notes in rough order of disruptiveness
    
    Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz at curii.com>

diff --git a/doc/admin/upgrading.html.textile.liquid b/doc/admin/upgrading.html.textile.liquid
index 9b61b30fb6..d51e39eafc 100644
--- a/doc/admin/upgrading.html.textile.liquid
+++ b/doc/admin/upgrading.html.textile.liquid
@@ -28,20 +28,27 @@ TODO: extract this information based on git commit messages and generate changel
 <div class="releasenotes">
 </notextile>
 
+h2(#main). development main (as of 2023-09-??)
 
-h2(#main). development main (as of 2023-09-03)
+"previous: Upgrading to 2.7.0":#v2_7_0
+
+h2(#v2_7_0). v2.7.0 (2023-09-??)
 
 "previous: Upgrading to 2.6.3":#v2_6_3
 
-h3. Crunchstat log format change
+h3. Workbench 1 deprecated
 
-The reported number of CPUs available in a container is now formatted in @crunchstat.txt@ log files and @crunchstat-summary@ text reports as a floating-point number rather than an integer (@2.00 cpus@ rather than @2 cpus@). Programs that parse these files may need to be updated accordingly.
+The original Arvados Workbench application (referred to as "Workbench 1") is deprecated and will be removed in a future major version of Arvados.  Users are advised to migrate to "Workbench 2".  Starting with this release, new installations of Arvados will only set up Workbench 2 and no longer include Workbench 1 by default.
 
 h3. Multi-node installer's domain name configuration changes
 
 The @domain_name@ variable at @terraform/vpc/terraform.tfvars@ and @DOMAIN@ variable at @local.params@ changed their meaning. In previous versions they were used in combination with @cluster_name@ and @CLUSTER@ to build the cluster's domain name (e.g.: @cluster_name at .@domain_name@). To allow the use of any arbitrary cluster domain, now we don't enforce using the cluster prefix as part of the domain, so @domain_name@ and @DOMAIN@ need to hold the entire domain for the given cluster.
 For example, if @cluster_name@ is set to @"xarv1"@ and @domain_name@ was previously set to @"example.com"@, it should now be set to @"xarv1.example.com"@ to keep using the same cluster domain.
 
+h3. Crunchstat log format change
+
+The reported number of CPUs available in a container is now formatted in @crunchstat.txt@ log files and @crunchstat-summary@ text reports as a floating-point number rather than an integer (@2.00 cpus@ rather than @2 cpus@). Programs that parse these files may need to be updated accordingly.
+
 h3. arvados-login-sync configuration changes, including ignored groups
 
 In the @Users@ section of your cluster configuration, there are now several options to control what system resources are or are not managed by @arvados-login-sync at . These options all have names that begin with @Sync at .
@@ -52,19 +59,15 @@ h3. API clients can always retrieve their current token, regardless of scopes
 
 We have introduced a small exception to the previous behavior of "Arvados API token scopes":{{ site.baseurl }}/admin/scoped-tokens.html in this release. A valid token is now always allowed to issue a request to "@GET /arvados/v1/api_client_authorizations/current@":{{ site.baseurl }}/api/methods/api_client_authorizations.html#current regardless of its scopes. This allows clients to reliably determine whether a request failed because a token is invalid, or because the token is not permitted to perform a particular request. The API server itself needs to be able to do this to validate tokens issued by other clusters in a federation.
 
-h3. UseAWSS3v2Driver option removed
-
-The old "v1" S3 driver for keepstore has been removed. The new "v2" implementation, which has been the default since Arvados 2.5.0, is always used. The @Volumes.*.DriverParameters.UseAWSS3v2Driver@ configuration key is no longer recognized. If your config file uses it, remove it to avoid warning messages at startup.
-
 h3. Deprecated/legacy APIs slated for removal
 
 The legacy APIs "humans":../api/methods/humans.html, "specimens":../api/methods/specimens.html, "traits":../api/methods/traits.html, "jobs":../api/methods/jobs.html, "job_tasks":../api/methods/job_tasks.html, "pipeline_instances":../api/methods/pipeline_instances.html, "pipeline_templates":../api/methods/pipeline_templates.html, "nodes":../api/methods/nodes.html, "repositories":../api/methods/repositories.html, and "keep_disks":../api/methods/keep_disks.html are deprecated and will be removed in a future major version of Arvados.
 
 In addition, the @default_owner_uuid@, @api_client_id@, and @user_id@ fields of "api_client_authorizations":../api/methods/api_client_authorizations.html are deprecated and will be removed from @api_client_authorization@ responses in a future major version of Arvados.  This should not affect clients as  @default_owner_uuid@ was never implemented, and @api_client_id@ and @user_id@ returned internal ids that were not meaningful or usable with any other API call.
 
-h3. Workbench 1 deprecated
+h3. UseAWSS3v2Driver option removed
 
-The original Arvados Workbench application (referred to as "Workbench 1") is deprecated and will be removed in a future major version of Arvados.  Users are advised to migrate to "Workbench 2".  Starting with this release, new installations of Arvados will only set up Workbench 2 and no longer include Workbench 1 by default.
+The old "v1" S3 driver for keepstore has been removed. The new "v2" implementation, which has been the default since Arvados 2.5.0, is always used. The @Volumes.*.DriverParameters.UseAWSS3v2Driver@ configuration key is no longer recognized. If your config file uses it, remove it to avoid warning messages at startup.
 
 h2(#v2_6_3). v2.6.3 (2023-06-06)
 

-----------------------------------------------------------------------


hooks/post-receive
-- 


