[ARVADOS] updated: 1.3.0-1442-g63756e0e8
Git user
git at public.curoverse.com
Mon Aug 5 18:46:07 UTC 2019
Summary of changes:
doc/install/install-keep-web.html.textile.liquid | 82 ++++++++++++++------
sdk/python/tests/run_test_server.py | 3 -
services/keep-web/doc.go | 68 ++++++++++------
services/keep-web/main.go | 35 ++++++---
services/keep-web/server_test.go | 62 +++++++++++++++
services/keep-web/usage.go | 99 ------------------------
6 files changed, 190 insertions(+), 159 deletions(-)
delete mode 100644 services/keep-web/usage.go
via 63756e0e8a1188e01d0c238394db71af278863cd (commit)
via 752c2ff97b3670bcc87b08816560f5a0be0b8d06 (commit)
via 5d82dc390a8e158b71c3c96cb1d0f301f3f12dbb (commit)
from 01fd373a99ab6e3ee13bf2c8ec38e48ee6f42ba3 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 63756e0e8a1188e01d0c238394db71af278863cd
Author: Lucas Di Pentima <ldipentima at veritasgenetics.com>
Date: Mon Aug 5 15:44:34 2019 -0300
14716: Updates documentation, add test check.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima at veritasgenetics.com>
diff --git a/doc/install/install-keep-web.html.textile.liquid b/doc/install/install-keep-web.html.textile.liquid
index 2991d7b0d..08dc186e8 100644
--- a/doc/install/install-keep-web.html.textile.liquid
+++ b/doc/install/install-keep-web.html.textile.liquid
@@ -45,14 +45,22 @@ Verify that @Keep-web@ is functional:
<notextile>
<pre><code>~$ <span class="userinput">keep-web -h</span>
Usage of keep-web:
- -allow-anonymous
- Serve public data to anonymous clients. Try the token supplied in the ARVADOS_API_TOKEN environment variable when none of the tokens provided in an HTTP request succeed in reading the desired collection. (default false)
- -attachment-only-host string
- Accept credentials, and add "Content-Disposition: attachment" response headers, for requests at this hostname:port. Prohibiting inline display makes it possible to serve untrusted and non-public content from a single origin, i.e., without wildcard DNS or TLS.
- -listen string
- Address to listen on: "host:port", or ":port" to listen on all interfaces. (default ":80")
- -trust-all-content
- Serve non-public content from a single origin. Dangerous: read docs before using!
+ -config file
+ Site configuration file (default may be overridden by setting an ARVADOS_CONFIG environment variable) (default "/etc/arvados/config.yml")
+ -dump-config
+ write current configuration to stdout and exit
+ -legacy-crunch-dispatch-slurm-config file
+ Legacy crunch-dispatch-slurm configuration file (default "/etc/arvados/crunch-dispatch-slurm/crunch-dispatch-slurm.yml")
+ -legacy-keepstore-config file
+ Legacy keepstore configuration file (default "/etc/arvados/keepstore/keepstore.yml")
+ -legacy-keepweb-config file
+ Legacy keep-web configuration file (default "/etc/arvados/keep-web/keep-web.yml")
+ -legacy-ws-config file
+ Legacy arvados-ws configuration file (default "/etc/arvados/ws/ws.yml")
+ -skip-legacy
+ Don't load legacy config files
+ -version
+ print version information and exit.
</code></pre>
</notextile>
@@ -62,22 +70,40 @@ If you intend to use Keep-web to serve public data to anonymous clients, configu
Install runit to supervise the Keep-web daemon. {% include 'install_runit' %}
+Set the cluster config file like the following:
+
+<notextile>
+<pre><code>Clusters:
+ <span class="userinput">uuid_prefix</span>:
+ SystemRootToken: "{{railsout}}"
+ Services:
+ Controller:
+ ExternalURL: "https://<span class="userinput">uuid_prefix</span>.your.domain"
+ Insecure: false
+ WebDAV:
+ InternalURLs:
+ "http://:9002/": {}
+ WebDAVDownload:
+ InternalURLs:
+ "http://:9002/": {}
+ ExternalURL: "https://download.<span class="userinput">uuid_prefix</span>.your.domain/"
+ Users:
+ AnonymousUserToken: "xxxxxxxxxxxxxxxxxxxx"
+ Collections:
+ TrustAllContent: false
+</code></pre>
+</notextile>
+
The basic command to start Keep-web in the service run script is:
<notextile>
-<pre><code>export ARVADOS_API_HOST=<span class="userinput">uuid_prefix</span>.your.domain
-export ARVADOS_API_TOKEN="<span class="userinput">{{railsout}}</span>"
-exec sudo -u nobody keep-web \
- -listen=<span class="userinput">:9002</span> \
- -attachment-only-host=<span class="userinput">download.uuid_prefix.your.domain</span> \
- -allow-anonymous \
- 2>&1
+<pre><code>exec sudo -u nobody keep-web -config=/path/to/arvados.yml
</code></pre>
</notextile>
-Omit the @-allow-anonymous@ argument if you do not want to serve public data.
+Set @Users.AnonymousUserToken: ""@ (empty string) if you do not want to serve public data.
-Set @ARVADOS_API_HOST_INSECURE=1@ if your API server's TLS certificate is not signed by a recognized CA.
+Set @Services.Controller.Insecure: true@ if your API server's TLS certificate is not signed by a recognized CA.
h3. Set up a reverse proxy with TLS support
@@ -134,24 +160,32 @@ Configure your DNS servers so the following names resolve to your Nginx proxy's
If neither of the above wildcard options is feasible, you have two choices:
# Serve web content at @collections.uuid_prefix.your.domain@, but only for unauthenticated requests (public data and collection sharing links). Authenticated requests will always result in file downloads, using the @download@ name. For example, the Workbench "preview" button and the "view entire log file" link will invoke file downloads instead of displaying content in the browser window.
-# In the special case where you know you are immune to XSS exploits, you can enable the "trust all content" mode in Keep-web (with the @-trust-all-content@ command line flag) and Workbench (with the @trust_all_content@ item in @application.yml@). With both of these enabled, inline web content can be served from a single @collections@ host name; no wildcard DNS or certificate is needed. Do not do this without understanding the security implications described in the "Keep-web documentation":http://godoc.org/github.com/curoverse/arvados/services/keep-web.
+# In the special case where you know you are immune to XSS exploits, you can enable the "trust all content" mode in Keep-web and Workbench (setting @Collections.TrustAllContent: true@ on the config file). With this enabled, inline web content can be served from a single @collections@ host name; no wildcard DNS or certificate is needed. Do not do this without understanding the security implications described in the "Keep-web documentation":http://godoc.org/github.com/curoverse/arvados/services/keep-web.
h3. Tell Workbench about the Keep-web service
Workbench has features like "download file from collection" and "show image" which work better if the content is served by Keep-web rather than Workbench itself. We recommend using the two different hostnames ("download" and "collections" above) for file downloads and inline content respectively.
-Add the following entry to your Workbench configuration file (@/etc/arvados/workbench/application.yml@). This URL will be used for file downloads.
+Add the following entry to your Workbench cluster configuration file (@/etc/arvados/config.yml@). This URL will be used for file downloads.
<notextile>
-<pre><code>keep_web_download_url: https://download.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}
+<pre><code>Clusters:
+ zzzzz:
+ Services:
+ WebDAVDownload:
+ ExternalURL: "https://download.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}"
</code></pre>
</notextile>
-Additionally, add *one* of the following entries to your Workbench configuration file, depending on your DNS setup. This URL will be used to serve user content that can be displayed in the browser, like image previews and static HTML pages.
+Additionally, add *one* of the following entries to your Workbench cluster configuration file, depending on your DNS setup. This URL will be used to serve user content that can be displayed in the browser, like image previews and static HTML pages.
<notextile>
-<pre><code>keep_web_url: https://%{uuid_or_pdh}--collections.<span class="userinput">uuid_prefix</span>.your.domain
-keep_web_url: https://%{uuid_or_pdh}.collections.<span class="userinput">uuid_prefix</span>.your.domain
-keep_web_url: https://collections.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}
+<pre><code>Clusters:
+ zzzzz:
+ Services:
+ WebDAV:
+ ExternalURL: "https://%{uuid_or_pdh}--collections.<span class="userinput">uuid_prefix</span>.your.domain"
+ ExternalURL: "https://%{uuid_or_pdh}.collections.<span class="userinput">uuid_prefix</span>.your.domain"
+ ExternalURL: "https://collections.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}"
</code></pre>
</notextile>
diff --git a/services/keep-web/doc.go b/services/keep-web/doc.go
index d65156f98..8682eac2d 100644
--- a/services/keep-web/doc.go
+++ b/services/keep-web/doc.go
@@ -12,31 +12,40 @@
//
// Configuration
//
-// The default configuration file location is
-// /etc/arvados/keep-web/keep-web.yml.
+// The default cluster configuration file location is
+// /etc/arvados/config.yml.
//
// Example configuration file
//
-// Client:
-// APIHost: "zzzzz.arvadosapi.com:443"
-// AuthToken: ""
-// Insecure: false
-// Listen: :1234
-// AnonymousTokens:
-// - xxxxxxxxxxxxxxxxxxxx
-// AttachmentOnlyHost: ""
-// TrustAllContent: false
+// Clusters:
+// zzzzz:
+// SystemRootToken: ""
+// Services:
+// Controller:
+// ExternalURL: "https://example.com"
+// Insecure: false
+// WebDAV:
+// InternalURLs:
+// "http://:1234/": {}
+// WebDAVDownload:
+// InternalURLs:
+// "http://:1234/": {}
+// ExternalURL: "https://download.example.com/"
+// Users:
+// AnonymousUserToken: "xxxxxxxxxxxxxxxxxxxx"
+// Collections:
+// TrustAllContent: false
//
// Starting the server
//
// Start a server using the default config file
-// /etc/arvados/keep-web/keep-web.yml:
+// /etc/arvados/config.yml:
//
// keep-web
//
-// Start a server using the config file /path/to/keep-web.yml:
+// Start a server using the config file /path/to/config.yml:
//
-// keep-web -config /path/to/keep-web.yml
+// keep-web -config /path/to/config.yml
//
// Proxy configuration
//
@@ -68,12 +77,15 @@
//
// Anonymous downloads
//
-// The "AnonymousTokens" configuration entry is an array of tokens to
-// use when processing anonymous requests, i.e., whenever a web client
+// The "Users.AnonymousUserToken" configuration entry used when
+// when processing anonymous requests, i.e., whenever a web client
// does not supply its own Arvados API token via path, query string,
// cookie, or request header.
//
-// "AnonymousTokens":["xxxxxxxxxxxxxxxxxxxxxxx"]
+// Clusters:
+// zzzzz:
+// Users:
+// AnonymousUserToken: "xxxxxxxxxxxxxxxxxxxxxxx"
//
// See http://doc.arvados.org/install/install-keep-web.html for examples.
//
@@ -246,7 +258,11 @@
// only when the designated origin matches exactly the Host header
// provided by the client or downstream proxy.
//
-// "AttachmentOnlyHost":"domain.example:9999"
+// Clusters:
+// zzzzz:
+// Services:
+// WebDAVDownload:
+// ExternalURL: "https://domain.example:9999"
//
// Trust All Content mode
//
@@ -259,14 +275,22 @@
//
// In such cases you can enable trust-all-content mode.
//
-// "TrustAllContent":true
+// Clusters:
+// zzzzz:
+// Collections:
+// TrustAllContent: true
//
// When TrustAllContent is enabled, the only effect of the
-// AttachmentOnlyHost flag is to add a "Content-Disposition:
+// Attachment-Only host setting is to add a "Content-Disposition:
// attachment" header.
//
-// "AttachmentOnlyHost":"domain.example:9999",
-// "TrustAllContent":true
+// Clusters:
+// zzzzz:
+// Services:
+// WebDAVDownload:
+// ExternalURL: "https://domain.example:9999"
+// Collections:
+// TrustAllContent: true
//
// Depending on your site configuration, you might also want to enable
// the "trust all content" setting in Workbench. Normally, Workbench
diff --git a/services/keep-web/server_test.go b/services/keep-web/server_test.go
index ab9ff60b5..534ed9b30 100644
--- a/services/keep-web/server_test.go
+++ b/services/keep-web/server_test.go
@@ -435,7 +435,7 @@ func (s *UnitSuite) TestLegacyConfig(c *check.C) {
"anonusertoken"
],
"AttachmentOnlyHost": "download.example.com",
- "TrustAllContent": false,
+ "TrustAllContent": true,
"Cache": {
"TTL": "1m",
"UUIDTTL": "1s",
@@ -477,6 +477,7 @@ func (s *UnitSuite) TestLegacyConfig(c *check.C) {
c.Check(cfg.cluster.Services.WebDAVDownload.InternalURLs[arvados.URL{Host: ":80"}], check.NotNil)
c.Check(cfg.cluster.Services.WebDAV.InternalURLs[arvados.URL{Host: ":80"}], check.NotNil)
+ c.Check(cfg.cluster.Collections.TrustAllContent, check.Equals, true)
c.Check(cfg.cluster.Users.AnonymousUserToken, check.Equals, "anonusertoken")
c.Check(cfg.cluster.ManagementToken, check.Equals, "xyzzy")
}
commit 752c2ff97b3670bcc87b08816560f5a0be0b8d06
Author: Lucas Di Pentima <ldipentima at veritasgenetics.com>
Date: Mon Aug 5 13:57:18 2019 -0300
14716: Adds legacy config test.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima at veritasgenetics.com>
diff --git a/services/keep-web/main.go b/services/keep-web/main.go
index 289f413f5..9dae50a8a 100644
--- a/services/keep-web/main.go
+++ b/services/keep-web/main.go
@@ -11,9 +11,9 @@ import (
"git.curoverse.com/arvados.git/lib/config"
"git.curoverse.com/arvados.git/sdk/go/arvados"
- sdkConfig "git.curoverse.com/arvados.git/sdk/go/config"
"github.com/coreos/go-systemd/daemon"
log "github.com/sirupsen/logrus"
+ yaml "gopkg.in/yaml.v2"
)
var (
@@ -55,12 +55,8 @@ func init() {
})
}
-func main() {
- prog := os.Args[0]
- args := os.Args[1:]
- logger := log.New()
-
- flags := flag.NewFlagSet(prog, flag.ExitOnError)
+func configure(logger log.FieldLogger, args []string) *Config {
+ flags := flag.NewFlagSet(args[0], flag.ExitOnError)
loader := config.NewLoader(os.Stdin, logger)
loader.SetupFlags(flags)
@@ -70,13 +66,13 @@ func main() {
getVersion := flags.Bool("version", false,
"print version information and exit.")
- args = loader.MungeLegacyConfigArgs(logger, args, "-legacy-keepweb-config")
+ args = loader.MungeLegacyConfigArgs(logger, args[1:], "-legacy-keepweb-config")
flags.Parse(args)
// Print version information if requested
if *getVersion {
fmt.Printf("keep-web %s\n", version)
- return
+ return nil
}
arvCfg, err := loader.Load()
@@ -86,7 +82,25 @@ func main() {
cfg := DefaultConfig(arvCfg)
if *dumpConfig {
- log.Fatal(sdkConfig.DumpAndExit(cfg.cluster))
+ out, err := yaml.Marshal(cfg)
+ if err != nil {
+ log.Fatal(err)
+ }
+ _, err = os.Stdout.Write(out)
+ if err != nil {
+ log.Fatal(err)
+ }
+ return nil
+ }
+ return cfg
+}
+
+func main() {
+ logger := log.New()
+
+ cfg := configure(logger, os.Args)
+ if cfg == nil {
+ return
}
log.Printf("keep-web %s started", version)
diff --git a/services/keep-web/server_test.go b/services/keep-web/server_test.go
index 7391ffb55..ab9ff60b5 100644
--- a/services/keep-web/server_test.go
+++ b/services/keep-web/server_test.go
@@ -16,12 +16,14 @@ import (
"os/exec"
"strings"
"testing"
+ "time"
"git.curoverse.com/arvados.git/lib/config"
"git.curoverse.com/arvados.git/sdk/go/arvados"
"git.curoverse.com/arvados.git/sdk/go/arvadosclient"
"git.curoverse.com/arvados.git/sdk/go/arvadostest"
"git.curoverse.com/arvados.git/sdk/go/keepclient"
+ log "github.com/sirupsen/logrus"
check "gopkg.in/check.v1"
)
@@ -420,6 +422,65 @@ func (s *IntegrationSuite) SetUpSuite(c *check.C) {
kc.PutB([]byte("waz"))
}
+func (s *UnitSuite) TestLegacyConfig(c *check.C) {
+ content := []byte(`
+{
+ "Client": {
+ "Scheme": "",
+ "APIHost": "example.com",
+ "AuthToken": "abcdefg",
+ },
+ "Listen": ":80",
+ "AnonymousTokens": [
+ "anonusertoken"
+ ],
+ "AttachmentOnlyHost": "download.example.com",
+ "TrustAllContent": false,
+ "Cache": {
+ "TTL": "1m",
+ "UUIDTTL": "1s",
+ "MaxCollectionEntries": 42,
+ "MaxCollectionBytes": 1234567890,
+ "MaxPermissionEntries": 100,
+ "MaxUUIDEntries": 100
+ },
+ "ManagementToken": "xyzzy"
+}
+`)
+ tmpfile, err := ioutil.TempFile("", "example")
+ if err != nil {
+ c.Error(err)
+ }
+ defer os.Remove(tmpfile.Name())
+
+ if _, err := tmpfile.Write(content); err != nil {
+ c.Error(err)
+ }
+ if err := tmpfile.Close(); err != nil {
+ c.Error(err)
+ }
+ cfg := configure(log.New(), []string{"keep-web", "-config", tmpfile.Name()})
+ c.Check(cfg, check.NotNil)
+ c.Check(cfg.cluster, check.NotNil)
+
+ c.Check(cfg.cluster.Services.Controller.ExternalURL, check.Equals, arvados.URL{Scheme: "https", Host: "example.com"})
+ c.Check(cfg.cluster.SystemRootToken, check.Equals, "abcdefg")
+
+ c.Check(cfg.cluster.Collections.WebDAVCache.TTL, check.Equals, arvados.Duration(60*time.Second))
+ c.Check(cfg.cluster.Collections.WebDAVCache.UUIDTTL, check.Equals, arvados.Duration(time.Second))
+ c.Check(cfg.cluster.Collections.WebDAVCache.MaxCollectionEntries, check.Equals, 42)
+ c.Check(cfg.cluster.Collections.WebDAVCache.MaxCollectionBytes, check.Equals, int64(1234567890))
+ c.Check(cfg.cluster.Collections.WebDAVCache.MaxPermissionEntries, check.Equals, 100)
+ c.Check(cfg.cluster.Collections.WebDAVCache.MaxUUIDEntries, check.Equals, 100)
+
+ c.Check(cfg.cluster.Services.WebDAVDownload.ExternalURL, check.Equals, arvados.URL{Host: "download.example.com"})
+ c.Check(cfg.cluster.Services.WebDAVDownload.InternalURLs[arvados.URL{Host: ":80"}], check.NotNil)
+ c.Check(cfg.cluster.Services.WebDAV.InternalURLs[arvados.URL{Host: ":80"}], check.NotNil)
+
+ c.Check(cfg.cluster.Users.AnonymousUserToken, check.Equals, "anonusertoken")
+ c.Check(cfg.cluster.ManagementToken, check.Equals, "xyzzy")
+}
+
func (s *IntegrationSuite) TearDownSuite(c *check.C) {
arvadostest.StopKeep(2)
arvadostest.StopAPI()
commit 5d82dc390a8e158b71c3c96cb1d0f301f3f12dbb
Author: Lucas Di Pentima <ldipentima at veritasgenetics.com>
Date: Mon Aug 5 11:18:36 2019 -0300
14716: Removes old usage message.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima at veritasgenetics.com>
diff --git a/sdk/python/tests/run_test_server.py b/sdk/python/tests/run_test_server.py
index 80227e6cd..679f04b98 100644
--- a/sdk/python/tests/run_test_server.py
+++ b/sdk/python/tests/run_test_server.py
@@ -743,9 +743,6 @@ def setup_config():
"Services": services,
"Users": {
"AnonymousUserToken": auth_token('anonymous')
- },
- "Collections": {
- "TrustAllContent": True
}
}
}
diff --git a/services/keep-web/main.go b/services/keep-web/main.go
index b8b272509..289f413f5 100644
--- a/services/keep-web/main.go
+++ b/services/keep-web/main.go
@@ -61,7 +61,6 @@ func main() {
logger := log.New()
flags := flag.NewFlagSet(prog, flag.ExitOnError)
- flags.Usage = usage
loader := config.NewLoader(os.Stdin, logger)
loader.SetupFlags(flags)
diff --git a/services/keep-web/usage.go b/services/keep-web/usage.go
deleted file mode 100644
index fda030dba..000000000
--- a/services/keep-web/usage.go
+++ /dev/null
@@ -1,99 +0,0 @@
-// Copyright (C) The Arvados Authors. All rights reserved.
-//
-// SPDX-License-Identifier: AGPL-3.0
-
-package main
-
-import (
- "encoding/json"
- "flag"
- "fmt"
- "os"
-)
-
-func usage() {
- c := DefaultConfig(nil)
- c.cluster.Users.AnonymousUserToken = "xxxxxxxxxxxxxxxxxxxxxxx"
- c.Client.APIHost = "zzzzz.arvadosapi.com:443"
- exampleConfigFile, err := json.MarshalIndent(c, " ", " ")
- if err != nil {
- panic(err)
- }
- fmt.Fprintf(os.Stderr, `
-
-Keep-web provides read-only HTTP access to files stored in Keep; see
-https://godoc.org/github.com/curoverse/arvados/services/keep-web and
-http://doc.arvados.org/install/install-keep-web.html
-
-Usage: keep-web -config path/to/keep-web.yml
-
-Options:
-`)
- flag.PrintDefaults()
- fmt.Fprintf(os.Stderr, `
-Example config file:
- %s
-
-Client.APIHost:
-
- Address (or address:port) of the Arvados API endpoint.
-
-Client.AuthToken:
-
- Unused. Normally empty, or omitted entirely.
-
-Client.Insecure:
-
- True if your Arvados API endpoint uses an unverifiable SSL/TLS
- certificate.
-
-Listen:
-
- Local port to listen on. Can be "address", "address:port", or
- ":port", where "address" is a host IP address or name and "port"
- is a port number or name.
-
-AnonymousTokens:
-
- Array of tokens to try when a client does not provide a token.
-
-AttachmentOnlyHost:
-
- Accept credentials, and add "Content-Disposition: attachment"
- response headers, for requests at this hostname:port.
-
- This prohibits inline display, which makes it possible to serve
- untrusted and non-public content from a single origin, i.e.,
- without wildcard DNS or SSL.
-
-TrustAllContent:
-
- Serve non-public content from a single origin. Dangerous: read
- docs before using!
-
-Cache.TTL:
-
- Maximum time to cache manifests and permission checks.
-
-Cache.UUIDTTL:
-
- Maximum time to cache collection state.
-
-Cache.MaxCollectionEntries:
-
- Maximum number of collection cache entries.
-
-Cache.MaxCollectionBytes:
-
- Approximate memory limit for collection cache.
-
-Cache.MaxPermissionEntries:
-
- Maximum number of permission cache entries.
-
-Cache.MaxUUIDEntries:
-
- Maximum number of UUID cache entries.
-
-`, exampleConfigFile)
-}
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list