[ARVADOS] updated: 970fab5b5bdfa9e2ece74032b7e6950aa90ffc72
Git user
git at public.curoverse.com
Tue May 31 11:26:47 EDT 2016
Summary of changes:
backports/python-gflags/fpm-info.sh | 1 -
build/package-build-dockerfiles/centos6/Dockerfile | 5 +-
build/package-build-dockerfiles/debian7/Dockerfile | 3 +-
build/package-build-dockerfiles/debian8/Dockerfile | 3 +-
.../ubuntu1204/Dockerfile | 3 +-
.../ubuntu1404/Dockerfile | 3 +-
build/package-test-dockerfiles/centos6/Dockerfile | 7 +-
build/run-build-packages-one-target.sh | 4 +-
build/run-build-packages.sh | 21 ++--
doc/_includes/_note_python27_sc.liquid | 2 +-
sdk/{ruby => cwl}/LICENSE-2.0.txt | 0
sdk/{python => cwl}/MANIFEST.in | 0
sdk/cwl/setup.py | 3 +
services/crunch-run/crunchrun_test.go | 33 ++++-
services/crunch-run/logging_test.go | 23 ++--
services/keepstore/azure_blob_volume.go | 137 +++++++++++++++++++--
services/keepstore/azure_blob_volume_test.go | 33 ++++-
services/keepstore/s3_volume.go | 4 +-
services/keepstore/volume_generic_test.go | 62 +++++++++-
services/keepstore/volume_unix.go | 4 +-
services/nodemanager/arvnodeman/baseactor.py | 20 ++-
.../arvnodeman/computenode/dispatch/__init__.py | 2 +-
services/nodemanager/arvnodeman/daemon.py | 7 +-
tools/arvbox/lib/arvbox/docker/Dockerfile.base | 9 +-
24 files changed, 318 insertions(+), 71 deletions(-)
delete mode 100644 backports/python-gflags/fpm-info.sh
copy sdk/{ruby => cwl}/LICENSE-2.0.txt (100%)
copy sdk/{python => cwl}/MANIFEST.in (100%)
discards 47adcb535cff8f6121f8f55b2121973cbc7b0480 (commit)
discards 6e03a31b8d4bc8990fd61d25e64a143b5fd18dcc (commit)
via 970fab5b5bdfa9e2ece74032b7e6950aa90ffc72 (commit)
via 65d88f48d190ca7a7bfa43215a32df0a622bbdad (commit)
via 63e7b750c32f4450f3279558210025b905b5e183 (commit)
via 6cb51851834989ff540c5e2a1fe6b88f1c26a089 (commit)
via 94c78c8792df002805d05cf349094c7833716e65 (commit)
via 44ceaa474a330f12dd9e00115af107d7258044f2 (commit)
via 287c0b37e5ed6787c6b5f28c193b7295388e8c04 (commit)
via 5d5375b28437c59f9625d8653ecef4afec1c2120 (commit)
via 3297c7fb97510aeea14df586d81caf65fd6c4aa6 (commit)
via bde564d8ad6f00326c16d7b85ce6dbeec9d338d3 (commit)
via a5260a0bd87f83b17744445476aaf7f107ca24e4 (commit)
via 5039497244e9cf973d40b377b62889a9356686d1 (commit)
via a1d4ab66c56e0150eb89f0deaafe2cb2a997b61b (commit)
via 0c5e684422234d62870b3c06b9e1014988878099 (commit)
via a6b15a15c6edb39d17ce79d71ec7b0816d7dcc0f (commit)
via 045d2a52055334c7f90725e587925851f7bc326e (commit)
via 22fb8112e4a66280669cc84a44550172b5a438e7 (commit)
via 1eb5f8fe7b290813e2c40a8d248280d488fc37fb (commit)
via d3bd05e221e5b7bf516bb1043bee96ab363c0af8 (commit)
via fedd51d0c04895951920d75147e3640f706653f6 (commit)
via aa1d6f9c5e1e21ceedf855bdb2f6db9154f26669 (commit)
via 25be2534e8746475ddc799042fefa08bd0548e9d (commit)
This update added new revisions after undoing existing revisions. That is
to say, the old revision is not a strict subset of the new revision. This
situation occurs when you --force push a change and generate a repository
containing something like this:
* -- * -- B -- O -- O -- O (47adcb535cff8f6121f8f55b2121973cbc7b0480)
\
N -- N -- N (970fab5b5bdfa9e2ece74032b7e6950aa90ffc72)
When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
commit 970fab5b5bdfa9e2ece74032b7e6950aa90ffc72
Author: Brett Smith <brett at curoverse.com>
Date: Tue May 31 11:13:41 2016 -0400
9242: Python packages install libraries to the distro path.
This avoids breaking dependent packages that expect to find files in
the same place.
diff --git a/backports/python-ciso8601/fpm-info.sh b/backports/python-ciso8601/fpm-info.sh
index 42f31e3..c9665be 100644
--- a/backports/python-ciso8601/fpm-info.sh
+++ b/backports/python-ciso8601/fpm-info.sh
@@ -8,4 +8,4 @@ case "$TARGET" in
esac
# FIXME: Remove this line after #6885 is done.
-fpm_args+=(--iteration 2)
+fpm_args+=(--iteration 3)
diff --git a/backports/python-llfuse/fpm-info.sh b/backports/python-llfuse/fpm-info.sh
index a7d9398..9fc0098 100644
--- a/backports/python-llfuse/fpm-info.sh
+++ b/backports/python-llfuse/fpm-info.sh
@@ -10,4 +10,4 @@ case "$TARGET" in
esac
# FIXME: Remove this line after #6885 is done.
-fpm_args+=(--iteration 2)
+fpm_args+=(--iteration 3)
diff --git a/backports/python-pycrypto/fpm-info.sh b/backports/python-pycrypto/fpm-info.sh
index 52df465..65a83a0 100644
--- a/backports/python-pycrypto/fpm-info.sh
+++ b/backports/python-pycrypto/fpm-info.sh
@@ -11,4 +11,4 @@ case "$TARGET" in
esac
# FIXME: Remove this line after #6885 is done.
-fpm_args+=(--iteration 2)
+fpm_args+=(--iteration 3)
diff --git a/backports/python-pycurl/fpm-info.sh b/backports/python-pycurl/fpm-info.sh
index e4fbf59..5bfb010 100644
--- a/backports/python-pycurl/fpm-info.sh
+++ b/backports/python-pycurl/fpm-info.sh
@@ -1,5 +1,5 @@
# FIXME: Remove this line after #6885 is done.
-fpm_args+=(--iteration 2)
+fpm_args+=(--iteration 3)
case "$TARGET" in
centos6)
diff --git a/build/run-build-packages.sh b/build/run-build-packages.sh
index cf12b65..90793e9 100755
--- a/build/run-build-packages.sh
+++ b/build/run-build-packages.sh
@@ -86,9 +86,13 @@ PYTHON3_VERSION=$(python3 -c 'import sys; print("{v.major}.{v.minor}".format(v=s
# You can customize them as needed in distro sections below.
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON2_PKG_PREFIX=python
+PYTHON2_PREFIX=/usr
+PYTHON2_INSTALL_LIB=lib/python$PYTHON2_VERSION/dist-packages
PYTHON3_PACKAGE=python$PYTHON3_VERSION
PYTHON3_PKG_PREFIX=python3
+PYTHON3_PREFIX=/usr
+PYTHON3_INSTALL_LIB=lib/python$PYTHON3_VERSION/dist-packages
## End Debian Python defaults.
case "$TARGET" in
@@ -134,8 +138,12 @@ case "$TARGET" in
FORMAT=rpm
PYTHON2_PACKAGE=$(rpm -qf "$(which python$PYTHON2_VERSION)" --queryformat '%{NAME}\n')
PYTHON2_PKG_PREFIX=$PYTHON2_PACKAGE
+ PYTHON2_PREFIX=/opt/rh/python27/root/usr
+ PYTHON2_INSTALL_LIB=lib64/python$PYTHON2_VERSION
PYTHON3_PACKAGE=$(rpm -qf "$(which python$PYTHON3_VERSION)" --queryformat '%{NAME}\n')
PYTHON3_PKG_PREFIX=$PYTHON3_PACKAGE
+ PYTHON3_PREFIX=/opt/rh/python33/root/usr
+ PYTHON3_INSTALL_LIB=lib64/python$PYTHON3_VERSION
PYTHON_BACKPORTS=(python-gflags==2.0 google-api-python-client==1.4.2 \
oauth2client==1.5.2 pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
@@ -405,7 +413,7 @@ fpm_build $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'C
# cwl-runner
cd $WORKSPACE/packages/$TARGET
rm -rf "$WORKSPACE/sdk/cwl/build"
-fpm_build $WORKSPACE/sdk/cwl "${PYTHON2_PKG_PREFIX}-arvados-cwl-runner" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/cwl/arvados_cwl_runner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados CWL runner" --iteration 2
+fpm_build $WORKSPACE/sdk/cwl "${PYTHON2_PKG_PREFIX}-arvados-cwl-runner" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/cwl/arvados_cwl_runner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados CWL runner" --iteration 3
# schema_salad. This is a python dependency of arvados-cwl-runner,
# but we can't use the usual PYTHONPACKAGES way to build this package due to the
@@ -422,17 +430,17 @@ fpm_build $WORKSPACE/sdk/cwl "${PYTHON2_PKG_PREFIX}-arvados-cwl-runner" 'Curover
# So we build this thing separately.
#
# Ward, 2016-03-17
-fpm --maintainer='Ward Vandewege <ward at curoverse.com>' -s python -t $FORMAT --exclude=*/dist-packages/tests/* --exclude=*/site-packages/tests/* --deb-ignore-iteration-in-dependencies -n "${PYTHON2_PKG_PREFIX}-schema-salad" --iteration 1 --python-bin python2.7 --python-easyinstall "$EASY_INSTALL2" --python-package-name-prefix "$PYTHON2_PKG_PREFIX" --depends "$PYTHON2_PACKAGE" -v 1.11.20160506154702 schema_salad
+fpm_build schema_salad schema_salad "" python 1.7.20160316203940
-# And schema_salad now depends on ruamel-yaml, which apparently has a braindead setup.py that requires special arguments to build (otherwise, it aborts with 'error: you have to install with "pip install ."'). Sigh.
+# And schema_salad now depends on ruamel-yaml, which apparently has a braindead setup.py that requires special arguments to build (otherwise, it aborts with 'error: you have to install with "pip install ."'). Sigh.
# Ward, 2016-05-26
-fpm --maintainer='Ward Vandewege <ward at curoverse.com>' -s python -t $FORMAT --exclude=*/dist-packages/tests/* --exclude=*/site-packages/tests/* --deb-ignore-iteration-in-dependencies --iteration 1 --python-bin python2.7 --python-easyinstall "$EASY_INSTALL2" --python-package-name-prefix "$PYTHON2_PKG_PREFIX" --depends "$PYTHON2_PACKAGE" --python-setup-py-arguments "--single-version-externally-managed" ruamel.yaml
+fpm_build ruamel.yaml ruamel.yaml "" python "" --python-setup-py-arguments "--single-version-externally-managed"
# And for cwltool we have the same problem as for schema_salad. Ward, 2016-03-17
-fpm --maintainer='Ward Vandewege <ward at curoverse.com>' -s python -t $FORMAT --exclude=*/dist-packages/tests/* --exclude=*/site-packages/tests/* --deb-ignore-iteration-in-dependencies -n "${PYTHON2_PKG_PREFIX}-cwltool" --iteration 1 --python-bin python2.7 --python-easyinstall "$EASY_INSTALL2" --python-package-name-prefix "$PYTHON2_PKG_PREFIX" --depends "$PYTHON2_PACKAGE" -v 1.0.20160519182434 cwltool
+fpm_build cwltool cwltool "" python 1.0.20160427142240
# FPM eats the trailing .0 in the python-rdflib-jsonld package when built with 'rdflib-jsonld>=0.3.0'. Force the version. Ward, 2016-03-25
-fpm --maintainer='Ward Vandewege <ward at curoverse.com>' -s python -t $FORMAT --exclude=*/dist-packages/tests/* --exclude=*/site-packages/tests/* --deb-ignore-iteration-in-dependencies --verbose --log info -n "${PYTHON2_PKG_PREFIX}-rdflib-jsonld" --iteration 1 --python-bin python2.7 --python-easyinstall "$EASY_INSTALL2" --python-package-name-prefix "$PYTHON2_PKG_PREFIX" --depends "$PYTHON2_PACKAGE" -v 0.3.0 rdflib-jsonld
+fpm_build rdflib-jsonld rdflib-jsonld "" python 0.3.0 --verbose --log info
# The PAM module
if [[ $TARGET =~ debian|ubuntu ]]; then
@@ -506,9 +514,7 @@ for deppkg in "${PYTHON_BACKPORTS[@]}"; do
"python$PYTHON2_VERSION" setup.py $DASHQ_UNLESS_DEBUG egg_info build
chmod -R go+rX .
set +e
- # --iteration 2 provides an upgrade for previously built
- # buggy packages.
- fpm_build . "$outname" "" python "" --iteration 2
+ fpm_build . "$outname" "" python "" --iteration 3
# The upload step uses the package timestamp to determine
# whether it's new. --no-clobber plays nice with that.
mv --no-clobber "$outname"*.$FORMAT "$WORKSPACE/packages/$TARGET"
diff --git a/build/run-library.sh b/build/run-library.sh
index 8d97ada..c7bc9538 100755
--- a/build/run-library.sh
+++ b/build/run-library.sh
@@ -192,6 +192,8 @@ fpm_build () {
VERSION=$1
shift
+ local default_iteration_value="$(default_iteration "$PACKAGE" "$VERSION")"
+
case "$PACKAGE_TYPE" in
python)
# All Arvados Python2 packages depend on Python 2.7.
@@ -199,7 +201,12 @@ fpm_build () {
set -- "$@" --python-bin python2.7 \
--python-easyinstall "$EASY_INSTALL2" \
--python-package-name-prefix "$PYTHON2_PKG_PREFIX" \
+ --prefix "$PYTHON2_PREFIX" \
+ --python-install-lib "$PYTHON2_INSTALL_LIB" \
+ --exclude "${PYTHON2_INSTALL_LIB#/}/tests" \
--depends "$PYTHON2_PACKAGE"
+ # Fix --iteration for #9242.
+ default_iteration_value=$(($default_iteration_value + 1))
;;
python3)
# fpm does not actually support a python3 package type. Instead
@@ -210,19 +217,21 @@ fpm_build () {
set -- "$@" --python-bin python3 \
--python-easyinstall "$EASY_INSTALL3" \
--python-package-name-prefix "$PYTHON3_PKG_PREFIX" \
+ --prefix "$PYTHON3_PREFIX" \
+ --python-install-lib "$PYTHON3_INSTALL_LIB" \
+ --exclude "${PYTHON3_INSTALL_LIB#/}/tests" \
--depends "$PYTHON3_PACKAGE"
+ # Fix --iteration for #9242.
+ default_iteration_value=$(($default_iteration_value + 1))
;;
esac
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "$FORMAT")
- if [ python = "$PACKAGE_TYPE" ]; then
- COMMAND_ARR+=(--exclude=\*/{dist,site}-packages/tests/\*)
- if [ deb = "$FORMAT" ]; then
- # Dependencies are built from setup.py. Since setup.py will never
- # refer to Debian package iterations, it doesn't make sense to
- # enforce those in the .deb dependencies.
- COMMAND_ARR+=(--deb-ignore-iteration-in-dependencies)
- fi
+ if [ python = "$PACKAGE_TYPE" ] && [ deb = "$FORMAT" ]; then
+ # Dependencies are built from setup.py. Since setup.py will never
+ # refer to Debian package iterations, it doesn't make sense to
+ # enforce those in the .deb dependencies.
+ COMMAND_ARR+=(--deb-ignore-iteration-in-dependencies)
fi
if [[ "${DEBUG:-0}" != "0" ]]; then
@@ -242,7 +251,7 @@ fpm_build () {
fi
# We can always add an --iteration here. If another one is specified in $@,
# that will take precedence, as desired.
- COMMAND_ARR+=(--iteration "$(default_iteration "$PACKAGE" "$VERSION")")
+ COMMAND_ARR+=(--iteration "$default_iteration_value")
# Append --depends X and other arguments specified by fpm-info.sh in
# the package source dir. These are added last so they can override
commit 65d88f48d190ca7a7bfa43215a32df0a622bbdad
Author: Brett Smith <brett at curoverse.com>
Date: Thu May 19 15:41:16 2016 -0400
9242: Refactor Python constant definitions in r-b-p.
There are about to be more of them, which will make this a real space
savings.
diff --git a/build/run-build-packages.sh b/build/run-build-packages.sh
index 87d396a..cf12b65 100755
--- a/build/run-build-packages.sh
+++ b/build/run-build-packages.sh
@@ -82,13 +82,18 @@ declare -a PYTHON_BACKPORTS PYTHON3_BACKPORTS
PYTHON2_VERSION=2.7
PYTHON3_VERSION=$(python3 -c 'import sys; print("{v.major}.{v.minor}".format(v=sys.version_info))')
+## These defaults are suitable for any Debian-based distribution.
+# You can customize them as needed in distro sections below.
+PYTHON2_PACKAGE=python$PYTHON2_VERSION
+PYTHON2_PKG_PREFIX=python
+
+PYTHON3_PACKAGE=python$PYTHON3_VERSION
+PYTHON3_PKG_PREFIX=python3
+## End Debian Python defaults.
+
case "$TARGET" in
debian7)
FORMAT=deb
- PYTHON2_PACKAGE=python$PYTHON2_VERSION
- PYTHON2_PKG_PREFIX=python
- PYTHON3_PACKAGE=python$PYTHON3_VERSION
- PYTHON3_PKG_PREFIX=python3
PYTHON_BACKPORTS=(python-gflags==2.0 google-api-python-client==1.4.2 \
oauth2client==1.5.2 pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
@@ -99,10 +104,6 @@ case "$TARGET" in
;;
debian8)
FORMAT=deb
- PYTHON2_PACKAGE=python$PYTHON2_VERSION
- PYTHON2_PKG_PREFIX=python
- PYTHON3_PACKAGE=python$PYTHON3_VERSION
- PYTHON3_PKG_PREFIX=python3
PYTHON_BACKPORTS=(python-gflags==2.0 google-api-python-client==1.4.2 \
oauth2client==1.5.2 pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
@@ -113,10 +114,6 @@ case "$TARGET" in
;;
ubuntu1204)
FORMAT=deb
- PYTHON2_PACKAGE=python$PYTHON2_VERSION
- PYTHON2_PKG_PREFIX=python
- PYTHON3_PACKAGE=python$PYTHON3_VERSION
- PYTHON3_PKG_PREFIX=python3
PYTHON_BACKPORTS=(python-gflags==2.0 google-api-python-client==1.4.2 \
oauth2client==1.5.2 pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
@@ -127,10 +124,6 @@ case "$TARGET" in
;;
ubuntu1404)
FORMAT=deb
- PYTHON2_PACKAGE=python$PYTHON2_VERSION
- PYTHON2_PKG_PREFIX=python
- PYTHON3_PACKAGE=python$PYTHON3_VERSION
- PYTHON3_PKG_PREFIX=python3
PYTHON_BACKPORTS=(pyasn1==0.1.7 pyasn1-modules==0.0.5 llfuse==0.41.1 ciso8601 \
google-api-python-client==1.4.2 six uritemplate oauth2client==1.5.2 httplib2 \
rsa 'pycurl<7.21.5' backports.ssl_match_hostname pyyaml 'rdflib>=4.2.0' \
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list