[ARVADOS] created: 41d9a0a4bb961c1c2d0b4a7786cfe76e4efa0083
Git user
git at public.curoverse.com
Thu Feb 18 14:44:52 EST 2016
at 41d9a0a4bb961c1c2d0b4a7786cfe76e4efa0083 (commit)
commit 41d9a0a4bb961c1c2d0b4a7786cfe76e4efa0083
Merge: 03245f8 8568240
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Feb 18 14:43:53 2016 -0500
Add 'build/' from commit '8568240aaa6b9dba2af675dc0d6e09fe1001abfd'
git-subtree-dir: build
git-subtree-mainline: 03245f8fb2e143864966dc151bf12368d2bd78fa
git-subtree-split: 8568240aaa6b9dba2af675dc0d6e09fe1001abfd
diff --cc build/COPYING
index 0000000,af63e41..af63e41
mode 000000,100644..100644
--- a/build/COPYING
+++ b/build/COPYING
diff --cc build/README
index 0000000,b076f0b..b076f0b
mode 000000,100644..100644
--- a/build/README
+++ b/build/README
diff --cc build/agpl-3.0.txt
index 0000000,0000000..dba13ed
new file mode 100644
--- /dev/null
+++ b/build/agpl-3.0.txt
@@@ -1,0 -1,0 +1,661 @@@
++ GNU AFFERO GENERAL PUBLIC LICENSE
++ Version 3, 19 November 2007
++
++ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
++ Everyone is permitted to copy and distribute verbatim copies
++ of this license document, but changing it is not allowed.
++
++ Preamble
++
++ The GNU Affero General Public License is a free, copyleft license for
++software and other kinds of works, specifically designed to ensure
++cooperation with the community in the case of network server software.
++
++ The licenses for most software and other practical works are designed
++to take away your freedom to share and change the works. By contrast,
++our General Public Licenses are intended to guarantee your freedom to
++share and change all versions of a program--to make sure it remains free
++software for all its users.
++
++ When we speak of free software, we are referring to freedom, not
++price. Our General Public Licenses are designed to make sure that you
++have the freedom to distribute copies of free software (and charge for
++them if you wish), that you receive source code or can get it if you
++want it, that you can change the software or use pieces of it in new
++free programs, and that you know you can do these things.
++
++ Developers that use our General Public Licenses protect your rights
++with two steps: (1) assert copyright on the software, and (2) offer
++you this License which gives you legal permission to copy, distribute
++and/or modify the software.
++
++ A secondary benefit of defending all users' freedom is that
++improvements made in alternate versions of the program, if they
++receive widespread use, become available for other developers to
++incorporate. Many developers of free software are heartened and
++encouraged by the resulting cooperation. However, in the case of
++software used on network servers, this result may fail to come about.
++The GNU General Public License permits making a modified version and
++letting the public access it on a server without ever releasing its
++source code to the public.
++
++ The GNU Affero General Public License is designed specifically to
++ensure that, in such cases, the modified source code becomes available
++to the community. It requires the operator of a network server to
++provide the source code of the modified version running there to the
++users of that server. Therefore, public use of a modified version, on
++a publicly accessible server, gives the public access to the source
++code of the modified version.
++
++ An older license, called the Affero General Public License and
++published by Affero, was designed to accomplish similar goals. This is
++a different license, not a version of the Affero GPL, but Affero has
++released a new version of the Affero GPL which permits relicensing under
++this license.
++
++ The precise terms and conditions for copying, distribution and
++modification follow.
++
++ TERMS AND CONDITIONS
++
++ 0. Definitions.
++
++ "This License" refers to version 3 of the GNU Affero General Public License.
++
++ "Copyright" also means copyright-like laws that apply to other kinds of
++works, such as semiconductor masks.
++
++ "The Program" refers to any copyrightable work licensed under this
++License. Each licensee is addressed as "you". "Licensees" and
++"recipients" may be individuals or organizations.
++
++ To "modify" a work means to copy from or adapt all or part of the work
++in a fashion requiring copyright permission, other than the making of an
++exact copy. The resulting work is called a "modified version" of the
++earlier work or a work "based on" the earlier work.
++
++ A "covered work" means either the unmodified Program or a work based
++on the Program.
++
++ To "propagate" a work means to do anything with it that, without
++permission, would make you directly or secondarily liable for
++infringement under applicable copyright law, except executing it on a
++computer or modifying a private copy. Propagation includes copying,
++distribution (with or without modification), making available to the
++public, and in some countries other activities as well.
++
++ To "convey" a work means any kind of propagation that enables other
++parties to make or receive copies. Mere interaction with a user through
++a computer network, with no transfer of a copy, is not conveying.
++
++ An interactive user interface displays "Appropriate Legal Notices"
++to the extent that it includes a convenient and prominently visible
++feature that (1) displays an appropriate copyright notice, and (2)
++tells the user that there is no warranty for the work (except to the
++extent that warranties are provided), that licensees may convey the
++work under this License, and how to view a copy of this License. If
++the interface presents a list of user commands or options, such as a
++menu, a prominent item in the list meets this criterion.
++
++ 1. Source Code.
++
++ The "source code" for a work means the preferred form of the work
++for making modifications to it. "Object code" means any non-source
++form of a work.
++
++ A "Standard Interface" means an interface that either is an official
++standard defined by a recognized standards body, or, in the case of
++interfaces specified for a particular programming language, one that
++is widely used among developers working in that language.
++
++ The "System Libraries" of an executable work include anything, other
++than the work as a whole, that (a) is included in the normal form of
++packaging a Major Component, but which is not part of that Major
++Component, and (b) serves only to enable use of the work with that
++Major Component, or to implement a Standard Interface for which an
++implementation is available to the public in source code form. A
++"Major Component", in this context, means a major essential component
++(kernel, window system, and so on) of the specific operating system
++(if any) on which the executable work runs, or a compiler used to
++produce the work, or an object code interpreter used to run it.
++
++ The "Corresponding Source" for a work in object code form means all
++the source code needed to generate, install, and (for an executable
++work) run the object code and to modify the work, including scripts to
++control those activities. However, it does not include the work's
++System Libraries, or general-purpose tools or generally available free
++programs which are used unmodified in performing those activities but
++which are not part of the work. For example, Corresponding Source
++includes interface definition files associated with source files for
++the work, and the source code for shared libraries and dynamically
++linked subprograms that the work is specifically designed to require,
++such as by intimate data communication or control flow between those
++subprograms and other parts of the work.
++
++ The Corresponding Source need not include anything that users
++can regenerate automatically from other parts of the Corresponding
++Source.
++
++ The Corresponding Source for a work in source code form is that
++same work.
++
++ 2. Basic Permissions.
++
++ All rights granted under this License are granted for the term of
++copyright on the Program, and are irrevocable provided the stated
++conditions are met. This License explicitly affirms your unlimited
++permission to run the unmodified Program. The output from running a
++covered work is covered by this License only if the output, given its
++content, constitutes a covered work. This License acknowledges your
++rights of fair use or other equivalent, as provided by copyright law.
++
++ You may make, run and propagate covered works that you do not
++convey, without conditions so long as your license otherwise remains
++in force. You may convey covered works to others for the sole purpose
++of having them make modifications exclusively for you, or provide you
++with facilities for running those works, provided that you comply with
++the terms of this License in conveying all material for which you do
++not control copyright. Those thus making or running the covered works
++for you must do so exclusively on your behalf, under your direction
++and control, on terms that prohibit them from making any copies of
++your copyrighted material outside their relationship with you.
++
++ Conveying under any other circumstances is permitted solely under
++the conditions stated below. Sublicensing is not allowed; section 10
++makes it unnecessary.
++
++ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
++
++ No covered work shall be deemed part of an effective technological
++measure under any applicable law fulfilling obligations under article
++11 of the WIPO copyright treaty adopted on 20 December 1996, or
++similar laws prohibiting or restricting circumvention of such
++measures.
++
++ When you convey a covered work, you waive any legal power to forbid
++circumvention of technological measures to the extent such circumvention
++is effected by exercising rights under this License with respect to
++the covered work, and you disclaim any intention to limit operation or
++modification of the work as a means of enforcing, against the work's
++users, your or third parties' legal rights to forbid circumvention of
++technological measures.
++
++ 4. Conveying Verbatim Copies.
++
++ You may convey verbatim copies of the Program's source code as you
++receive it, in any medium, provided that you conspicuously and
++appropriately publish on each copy an appropriate copyright notice;
++keep intact all notices stating that this License and any
++non-permissive terms added in accord with section 7 apply to the code;
++keep intact all notices of the absence of any warranty; and give all
++recipients a copy of this License along with the Program.
++
++ You may charge any price or no price for each copy that you convey,
++and you may offer support or warranty protection for a fee.
++
++ 5. Conveying Modified Source Versions.
++
++ You may convey a work based on the Program, or the modifications to
++produce it from the Program, in the form of source code under the
++terms of section 4, provided that you also meet all of these conditions:
++
++ a) The work must carry prominent notices stating that you modified
++ it, and giving a relevant date.
++
++ b) The work must carry prominent notices stating that it is
++ released under this License and any conditions added under section
++ 7. This requirement modifies the requirement in section 4 to
++ "keep intact all notices".
++
++ c) You must license the entire work, as a whole, under this
++ License to anyone who comes into possession of a copy. This
++ License will therefore apply, along with any applicable section 7
++ additional terms, to the whole of the work, and all its parts,
++ regardless of how they are packaged. This License gives no
++ permission to license the work in any other way, but it does not
++ invalidate such permission if you have separately received it.
++
++ d) If the work has interactive user interfaces, each must display
++ Appropriate Legal Notices; however, if the Program has interactive
++ interfaces that do not display Appropriate Legal Notices, your
++ work need not make them do so.
++
++ A compilation of a covered work with other separate and independent
++works, which are not by their nature extensions of the covered work,
++and which are not combined with it such as to form a larger program,
++in or on a volume of a storage or distribution medium, is called an
++"aggregate" if the compilation and its resulting copyright are not
++used to limit the access or legal rights of the compilation's users
++beyond what the individual works permit. Inclusion of a covered work
++in an aggregate does not cause this License to apply to the other
++parts of the aggregate.
++
++ 6. Conveying Non-Source Forms.
++
++ You may convey a covered work in object code form under the terms
++of sections 4 and 5, provided that you also convey the
++machine-readable Corresponding Source under the terms of this License,
++in one of these ways:
++
++ a) Convey the object code in, or embodied in, a physical product
++ (including a physical distribution medium), accompanied by the
++ Corresponding Source fixed on a durable physical medium
++ customarily used for software interchange.
++
++ b) Convey the object code in, or embodied in, a physical product
++ (including a physical distribution medium), accompanied by a
++ written offer, valid for at least three years and valid for as
++ long as you offer spare parts or customer support for that product
++ model, to give anyone who possesses the object code either (1) a
++ copy of the Corresponding Source for all the software in the
++ product that is covered by this License, on a durable physical
++ medium customarily used for software interchange, for a price no
++ more than your reasonable cost of physically performing this
++ conveying of source, or (2) access to copy the
++ Corresponding Source from a network server at no charge.
++
++ c) Convey individual copies of the object code with a copy of the
++ written offer to provide the Corresponding Source. This
++ alternative is allowed only occasionally and noncommercially, and
++ only if you received the object code with such an offer, in accord
++ with subsection 6b.
++
++ d) Convey the object code by offering access from a designated
++ place (gratis or for a charge), and offer equivalent access to the
++ Corresponding Source in the same way through the same place at no
++ further charge. You need not require recipients to copy the
++ Corresponding Source along with the object code. If the place to
++ copy the object code is a network server, the Corresponding Source
++ may be on a different server (operated by you or a third party)
++ that supports equivalent copying facilities, provided you maintain
++ clear directions next to the object code saying where to find the
++ Corresponding Source. Regardless of what server hosts the
++ Corresponding Source, you remain obligated to ensure that it is
++ available for as long as needed to satisfy these requirements.
++
++ e) Convey the object code using peer-to-peer transmission, provided
++ you inform other peers where the object code and Corresponding
++ Source of the work are being offered to the general public at no
++ charge under subsection 6d.
++
++ A separable portion of the object code, whose source code is excluded
++from the Corresponding Source as a System Library, need not be
++included in conveying the object code work.
++
++ A "User Product" is either (1) a "consumer product", which means any
++tangible personal property which is normally used for personal, family,
++or household purposes, or (2) anything designed or sold for incorporation
++into a dwelling. In determining whether a product is a consumer product,
++doubtful cases shall be resolved in favor of coverage. For a particular
++product received by a particular user, "normally used" refers to a
++typical or common use of that class of product, regardless of the status
++of the particular user or of the way in which the particular user
++actually uses, or expects or is expected to use, the product. A product
++is a consumer product regardless of whether the product has substantial
++commercial, industrial or non-consumer uses, unless such uses represent
++the only significant mode of use of the product.
++
++ "Installation Information" for a User Product means any methods,
++procedures, authorization keys, or other information required to install
++and execute modified versions of a covered work in that User Product from
++a modified version of its Corresponding Source. The information must
++suffice to ensure that the continued functioning of the modified object
++code is in no case prevented or interfered with solely because
++modification has been made.
++
++ If you convey an object code work under this section in, or with, or
++specifically for use in, a User Product, and the conveying occurs as
++part of a transaction in which the right of possession and use of the
++User Product is transferred to the recipient in perpetuity or for a
++fixed term (regardless of how the transaction is characterized), the
++Corresponding Source conveyed under this section must be accompanied
++by the Installation Information. But this requirement does not apply
++if neither you nor any third party retains the ability to install
++modified object code on the User Product (for example, the work has
++been installed in ROM).
++
++ The requirement to provide Installation Information does not include a
++requirement to continue to provide support service, warranty, or updates
++for a work that has been modified or installed by the recipient, or for
++the User Product in which it has been modified or installed. Access to a
++network may be denied when the modification itself materially and
++adversely affects the operation of the network or violates the rules and
++protocols for communication across the network.
++
++ Corresponding Source conveyed, and Installation Information provided,
++in accord with this section must be in a format that is publicly
++documented (and with an implementation available to the public in
++source code form), and must require no special password or key for
++unpacking, reading or copying.
++
++ 7. Additional Terms.
++
++ "Additional permissions" are terms that supplement the terms of this
++License by making exceptions from one or more of its conditions.
++Additional permissions that are applicable to the entire Program shall
++be treated as though they were included in this License, to the extent
++that they are valid under applicable law. If additional permissions
++apply only to part of the Program, that part may be used separately
++under those permissions, but the entire Program remains governed by
++this License without regard to the additional permissions.
++
++ When you convey a copy of a covered work, you may at your option
++remove any additional permissions from that copy, or from any part of
++it. (Additional permissions may be written to require their own
++removal in certain cases when you modify the work.) You may place
++additional permissions on material, added by you to a covered work,
++for which you have or can give appropriate copyright permission.
++
++ Notwithstanding any other provision of this License, for material you
++add to a covered work, you may (if authorized by the copyright holders of
++that material) supplement the terms of this License with terms:
++
++ a) Disclaiming warranty or limiting liability differently from the
++ terms of sections 15 and 16 of this License; or
++
++ b) Requiring preservation of specified reasonable legal notices or
++ author attributions in that material or in the Appropriate Legal
++ Notices displayed by works containing it; or
++
++ c) Prohibiting misrepresentation of the origin of that material, or
++ requiring that modified versions of such material be marked in
++ reasonable ways as different from the original version; or
++
++ d) Limiting the use for publicity purposes of names of licensors or
++ authors of the material; or
++
++ e) Declining to grant rights under trademark law for use of some
++ trade names, trademarks, or service marks; or
++
++ f) Requiring indemnification of licensors and authors of that
++ material by anyone who conveys the material (or modified versions of
++ it) with contractual assumptions of liability to the recipient, for
++ any liability that these contractual assumptions directly impose on
++ those licensors and authors.
++
++ All other non-permissive additional terms are considered "further
++restrictions" within the meaning of section 10. If the Program as you
++received it, or any part of it, contains a notice stating that it is
++governed by this License along with a term that is a further
++restriction, you may remove that term. If a license document contains
++a further restriction but permits relicensing or conveying under this
++License, you may add to a covered work material governed by the terms
++of that license document, provided that the further restriction does
++not survive such relicensing or conveying.
++
++ If you add terms to a covered work in accord with this section, you
++must place, in the relevant source files, a statement of the
++additional terms that apply to those files, or a notice indicating
++where to find the applicable terms.
++
++ Additional terms, permissive or non-permissive, may be stated in the
++form of a separately written license, or stated as exceptions;
++the above requirements apply either way.
++
++ 8. Termination.
++
++ You may not propagate or modify a covered work except as expressly
++provided under this License. Any attempt otherwise to propagate or
++modify it is void, and will automatically terminate your rights under
++this License (including any patent licenses granted under the third
++paragraph of section 11).
++
++ However, if you cease all violation of this License, then your
++license from a particular copyright holder is reinstated (a)
++provisionally, unless and until the copyright holder explicitly and
++finally terminates your license, and (b) permanently, if the copyright
++holder fails to notify you of the violation by some reasonable means
++prior to 60 days after the cessation.
++
++ Moreover, your license from a particular copyright holder is
++reinstated permanently if the copyright holder notifies you of the
++violation by some reasonable means, this is the first time you have
++received notice of violation of this License (for any work) from that
++copyright holder, and you cure the violation prior to 30 days after
++your receipt of the notice.
++
++ Termination of your rights under this section does not terminate the
++licenses of parties who have received copies or rights from you under
++this License. If your rights have been terminated and not permanently
++reinstated, you do not qualify to receive new licenses for the same
++material under section 10.
++
++ 9. Acceptance Not Required for Having Copies.
++
++ You are not required to accept this License in order to receive or
++run a copy of the Program. Ancillary propagation of a covered work
++occurring solely as a consequence of using peer-to-peer transmission
++to receive a copy likewise does not require acceptance. However,
++nothing other than this License grants you permission to propagate or
++modify any covered work. These actions infringe copyright if you do
++not accept this License. Therefore, by modifying or propagating a
++covered work, you indicate your acceptance of this License to do so.
++
++ 10. Automatic Licensing of Downstream Recipients.
++
++ Each time you convey a covered work, the recipient automatically
++receives a license from the original licensors, to run, modify and
++propagate that work, subject to this License. You are not responsible
++for enforcing compliance by third parties with this License.
++
++ An "entity transaction" is a transaction transferring control of an
++organization, or substantially all assets of one, or subdividing an
++organization, or merging organizations. If propagation of a covered
++work results from an entity transaction, each party to that
++transaction who receives a copy of the work also receives whatever
++licenses to the work the party's predecessor in interest had or could
++give under the previous paragraph, plus a right to possession of the
++Corresponding Source of the work from the predecessor in interest, if
++the predecessor has it or can get it with reasonable efforts.
++
++ You may not impose any further restrictions on the exercise of the
++rights granted or affirmed under this License. For example, you may
++not impose a license fee, royalty, or other charge for exercise of
++rights granted under this License, and you may not initiate litigation
++(including a cross-claim or counterclaim in a lawsuit) alleging that
++any patent claim is infringed by making, using, selling, offering for
++sale, or importing the Program or any portion of it.
++
++ 11. Patents.
++
++ A "contributor" is a copyright holder who authorizes use under this
++License of the Program or a work on which the Program is based. The
++work thus licensed is called the contributor's "contributor version".
++
++ A contributor's "essential patent claims" are all patent claims
++owned or controlled by the contributor, whether already acquired or
++hereafter acquired, that would be infringed by some manner, permitted
++by this License, of making, using, or selling its contributor version,
++but do not include claims that would be infringed only as a
++consequence of further modification of the contributor version. For
++purposes of this definition, "control" includes the right to grant
++patent sublicenses in a manner consistent with the requirements of
++this License.
++
++ Each contributor grants you a non-exclusive, worldwide, royalty-free
++patent license under the contributor's essential patent claims, to
++make, use, sell, offer for sale, import and otherwise run, modify and
++propagate the contents of its contributor version.
++
++ In the following three paragraphs, a "patent license" is any express
++agreement or commitment, however denominated, not to enforce a patent
++(such as an express permission to practice a patent or covenant not to
++sue for patent infringement). To "grant" such a patent license to a
++party means to make such an agreement or commitment not to enforce a
++patent against the party.
++
++ If you convey a covered work, knowingly relying on a patent license,
++and the Corresponding Source of the work is not available for anyone
++to copy, free of charge and under the terms of this License, through a
++publicly available network server or other readily accessible means,
++then you must either (1) cause the Corresponding Source to be so
++available, or (2) arrange to deprive yourself of the benefit of the
++patent license for this particular work, or (3) arrange, in a manner
++consistent with the requirements of this License, to extend the patent
++license to downstream recipients. "Knowingly relying" means you have
++actual knowledge that, but for the patent license, your conveying the
++covered work in a country, or your recipient's use of the covered work
++in a country, would infringe one or more identifiable patents in that
++country that you have reason to believe are valid.
++
++ If, pursuant to or in connection with a single transaction or
++arrangement, you convey, or propagate by procuring conveyance of, a
++covered work, and grant a patent license to some of the parties
++receiving the covered work authorizing them to use, propagate, modify
++or convey a specific copy of the covered work, then the patent license
++you grant is automatically extended to all recipients of the covered
++work and works based on it.
++
++ A patent license is "discriminatory" if it does not include within
++the scope of its coverage, prohibits the exercise of, or is
++conditioned on the non-exercise of one or more of the rights that are
++specifically granted under this License. You may not convey a covered
++work if you are a party to an arrangement with a third party that is
++in the business of distributing software, under which you make payment
++to the third party based on the extent of your activity of conveying
++the work, and under which the third party grants, to any of the
++parties who would receive the covered work from you, a discriminatory
++patent license (a) in connection with copies of the covered work
++conveyed by you (or copies made from those copies), or (b) primarily
++for and in connection with specific products or compilations that
++contain the covered work, unless you entered into that arrangement,
++or that patent license was granted, prior to 28 March 2007.
++
++ Nothing in this License shall be construed as excluding or limiting
++any implied license or other defenses to infringement that may
++otherwise be available to you under applicable patent law.
++
++ 12. No Surrender of Others' Freedom.
++
++ If conditions are imposed on you (whether by court order, agreement or
++otherwise) that contradict the conditions of this License, they do not
++excuse you from the conditions of this License. If you cannot convey a
++covered work so as to satisfy simultaneously your obligations under this
++License and any other pertinent obligations, then as a consequence you may
++not convey it at all. For example, if you agree to terms that obligate you
++to collect a royalty for further conveying from those to whom you convey
++the Program, the only way you could satisfy both those terms and this
++License would be to refrain entirely from conveying the Program.
++
++ 13. Remote Network Interaction; Use with the GNU General Public License.
++
++ Notwithstanding any other provision of this License, if you modify the
++Program, your modified version must prominently offer all users
++interacting with it remotely through a computer network (if your version
++supports such interaction) an opportunity to receive the Corresponding
++Source of your version by providing access to the Corresponding Source
++from a network server at no charge, through some standard or customary
++means of facilitating copying of software. This Corresponding Source
++shall include the Corresponding Source for any work covered by version 3
++of the GNU General Public License that is incorporated pursuant to the
++following paragraph.
++
++ Notwithstanding any other provision of this License, you have
++permission to link or combine any covered work with a work licensed
++under version 3 of the GNU General Public License into a single
++combined work, and to convey the resulting work. The terms of this
++License will continue to apply to the part which is the covered work,
++but the work with which it is combined will remain governed by version
++3 of the GNU General Public License.
++
++ 14. Revised Versions of this License.
++
++ The Free Software Foundation may publish revised and/or new versions of
++the GNU Affero General Public License from time to time. Such new versions
++will be similar in spirit to the present version, but may differ in detail to
++address new problems or concerns.
++
++ Each version is given a distinguishing version number. If the
++Program specifies that a certain numbered version of the GNU Affero General
++Public License "or any later version" applies to it, you have the
++option of following the terms and conditions either of that numbered
++version or of any later version published by the Free Software
++Foundation. If the Program does not specify a version number of the
++GNU Affero General Public License, you may choose any version ever published
++by the Free Software Foundation.
++
++ If the Program specifies that a proxy can decide which future
++versions of the GNU Affero General Public License can be used, that proxy's
++public statement of acceptance of a version permanently authorizes you
++to choose that version for the Program.
++
++ Later license versions may give you additional or different
++permissions. However, no additional obligations are imposed on any
++author or copyright holder as a result of your choosing to follow a
++later version.
++
++ 15. Disclaimer of Warranty.
++
++ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
++APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
++HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
++OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
++THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
++PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
++IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
++ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
++
++ 16. Limitation of Liability.
++
++ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
++WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
++THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
++GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
++USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
++DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
++PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
++EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
++SUCH DAMAGES.
++
++ 17. Interpretation of Sections 15 and 16.
++
++ If the disclaimer of warranty and limitation of liability provided
++above cannot be given local legal effect according to their terms,
++reviewing courts shall apply local law that most closely approximates
++an absolute waiver of all civil liability in connection with the
++Program, unless a warranty or assumption of liability accompanies a
++copy of the Program in return for a fee.
++
++ END OF TERMS AND CONDITIONS
++
++ How to Apply These Terms to Your New Programs
++
++ If you develop a new program, and you want it to be of the greatest
++possible use to the public, the best way to achieve this is to make it
++free software which everyone can redistribute and change under these terms.
++
++ To do so, attach the following notices to the program. It is safest
++to attach them to the start of each source file to most effectively
++state the exclusion of warranty; and each file should have at least
++the "copyright" line and a pointer to where the full notice is found.
++
++ <one line to give the program's name and a brief idea of what it does.>
++ Copyright (C) <year> <name of author>
++
++ This program is free software: you can redistribute it and/or modify
++ it under the terms of the GNU Affero General Public License as published by
++ the Free Software Foundation, either version 3 of the License, or
++ (at your option) any later version.
++
++ This program is distributed in the hope that it will be useful,
++ but WITHOUT ANY WARRANTY; without even the implied warranty of
++ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
++ GNU Affero General Public License for more details.
++
++ You should have received a copy of the GNU Affero General Public License
++ along with this program. If not, see <http://www.gnu.org/licenses/>.
++
++Also add information on how to contact you by electronic and paper mail.
++
++ If your software can interact with users remotely through a computer
++network, you should also make sure that it provides a way for users to
++get its source. For example, if your program is a web application, its
++interface could display a "Source" link that leads users to an archive
++of the code. There are many ways you could offer source, and different
++solutions will be better for different programs; see section 13 for the
++specific requirements.
++
++ You should also get your employer (if you work as a programmer) or school,
++if any, to sign a "copyright disclaimer" for the program, if necessary.
++For more information on this, and how to apply and follow the GNU AGPL, see
++<http://www.gnu.org/licenses/>.
diff --cc build/arvbox/README.md
index 0000000,8098686..8098686
mode 000000,100644..100644
--- a/build/arvbox/README.md
+++ b/build/arvbox/README.md
diff --cc build/arvbox/bin/arvbox
index 0000000,7be361b..7be361b
mode 000000,100755..100755
--- a/build/arvbox/bin/arvbox
+++ b/build/arvbox/bin/arvbox
diff --cc build/arvbox/lib/arvbox/docker/Dockerfile.base
index 0000000,160afee..160afee
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/Dockerfile.base
+++ b/build/arvbox/lib/arvbox/docker/Dockerfile.base
diff --cc build/arvbox/lib/arvbox/docker/Dockerfile.demo
index 0000000,d3e36c6..d3e36c6
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/Dockerfile.demo
+++ b/build/arvbox/lib/arvbox/docker/Dockerfile.demo
diff --cc build/arvbox/lib/arvbox/docker/Dockerfile.dev
index 0000000,5ec73bb..5ec73bb
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/Dockerfile.dev
+++ b/build/arvbox/lib/arvbox/docker/Dockerfile.dev
diff --cc build/arvbox/lib/arvbox/docker/application_yml_override.py
index 0000000,98a8e48..98a8e48
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/application_yml_override.py
+++ b/build/arvbox/lib/arvbox/docker/application_yml_override.py
diff --cc build/arvbox/lib/arvbox/docker/common.sh
index 0000000,4c2de47..4c2de47
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/common.sh
+++ b/build/arvbox/lib/arvbox/docker/common.sh
diff --cc build/arvbox/lib/arvbox/docker/createusers.sh
index 0000000,b77c9c2..b77c9c2
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/createusers.sh
+++ b/build/arvbox/lib/arvbox/docker/createusers.sh
diff --cc build/arvbox/lib/arvbox/docker/crunch-setup.sh
index 0000000,178fec1..178fec1
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/crunch-setup.sh
+++ b/build/arvbox/lib/arvbox/docker/crunch-setup.sh
diff --cc build/arvbox/lib/arvbox/docker/fuse.conf
index 0000000,a439ab8..a439ab8
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/fuse.conf
+++ b/build/arvbox/lib/arvbox/docker/fuse.conf
diff --cc build/arvbox/lib/arvbox/docker/gitolite.rc
index 0000000,03c4b29..03c4b29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/gitolite.rc
+++ b/build/arvbox/lib/arvbox/docker/gitolite.rc
diff --cc build/arvbox/lib/arvbox/docker/keep-setup.sh
index 0000000,b66463f..b66463f
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/keep-setup.sh
+++ b/build/arvbox/lib/arvbox/docker/keep-setup.sh
diff --cc build/arvbox/lib/arvbox/docker/logger
index 0000000,a79a518..a79a518
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/logger
+++ b/build/arvbox/lib/arvbox/docker/logger
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/.gitignore
index 0000000,bbf313b..bbf313b
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/.gitignore
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/.gitignore
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/LICENSE
index 0000000,d158667..d158667
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/LICENSE
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/LICENSE
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/Makefile
index 0000000,9a28963..9a28963
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/Makefile
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/Makefile
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/README.md
index 0000000,1bcb8cc..1bcb8cc
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/README.md
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/README.md
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/debian/changelog
index 0000000,7d8689f..7d8689f
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/debian/changelog
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/debian/changelog
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/debian/compat
index 0000000,ec63514..ec63514
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/debian/compat
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/debian/compat
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/debian/control
index 0000000,4060915..4060915
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/debian/control
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/debian/control
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/debian/copyright
index 0000000,8679a6a..8679a6a
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/debian/copyright
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/debian/copyright
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/debian/docs
index 0000000,b43bf86..b43bf86
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/debian/docs
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/debian/docs
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/debian/rules
index 0000000,ce15cce..ce15cce
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/runit-docker/debian/rules
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/debian/rules
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/debian/source/format
index 0000000,163aaf8..163aaf8
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/debian/source/format
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/debian/source/format
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/runit-docker
index 0000000,fdbaad5..fdbaad5
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/runit-docker/runit-docker
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/runit-docker
diff --cc build/arvbox/lib/arvbox/docker/runit-docker/runit-docker.c
index 0000000,825a35f..825a35f
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/runit-docker/runit-docker.c
+++ b/build/arvbox/lib/arvbox/docker/runit-docker/runit-docker.c
diff --cc build/arvbox/lib/arvbox/docker/runsu.sh
index 0000000,1557d09..1557d09
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/runsu.sh
+++ b/build/arvbox/lib/arvbox/docker/runsu.sh
diff --cc build/arvbox/lib/arvbox/docker/service/api/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/api/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/api/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/api/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/api/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/api/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/api/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/api/run
+++ b/build/arvbox/lib/arvbox/docker/service/api/run
diff --cc build/arvbox/lib/arvbox/docker/service/api/run-service
index 0000000,058939c..058939c
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/api/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/api/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/arv-git-httpd/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/arv-git-httpd/run
+++ b/build/arvbox/lib/arvbox/docker/service/arv-git-httpd/run
diff --cc build/arvbox/lib/arvbox/docker/service/arv-git-httpd/run-service
index 0000000,854464e..854464e
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/arv-git-httpd/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/arv-git-httpd/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run-service
index 0000000,211b438..211b438
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run-service
index 0000000,fa3a73a..fa3a73a
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run
diff --cc build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run-service
index 0000000,6430e9c..6430e9c
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/doc/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/doc/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/doc/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/doc/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/doc/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/doc/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/doc/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/doc/run
+++ b/build/arvbox/lib/arvbox/docker/service/doc/run
diff --cc build/arvbox/lib/arvbox/docker/service/doc/run-service
index 0000000,acbe21c..acbe21c
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/doc/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/doc/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/docker/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/docker/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/docker/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/docker/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/docker/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/docker/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/docker/run
index 0000000,1ecdc16..1ecdc16
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/docker/run
+++ b/build/arvbox/lib/arvbox/docker/service/docker/run
diff --cc build/arvbox/lib/arvbox/docker/service/gitolite/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/gitolite/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/gitolite/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/gitolite/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/gitolite/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/gitolite/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/gitolite/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/gitolite/run
+++ b/build/arvbox/lib/arvbox/docker/service/gitolite/run
diff --cc build/arvbox/lib/arvbox/docker/service/gitolite/run-service
index 0000000,e0e8771..e0e8771
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/gitolite/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/gitolite/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/keep-web/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/keep-web/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/keep-web/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/keep-web/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/keep-web/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/keep-web/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/keep-web/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/keep-web/run
+++ b/build/arvbox/lib/arvbox/docker/service/keep-web/run
diff --cc build/arvbox/lib/arvbox/docker/service/keep-web/run-service
index 0000000,a2c6aa1..a2c6aa1
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/keep-web/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/keep-web/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/keepproxy/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/keepproxy/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/keepproxy/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/keepproxy/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/keepproxy/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/keepproxy/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/keepproxy/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/keepproxy/run
+++ b/build/arvbox/lib/arvbox/docker/service/keepproxy/run
diff --cc build/arvbox/lib/arvbox/docker/service/keepproxy/run-service
index 0000000,413a67e..413a67e
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/keepproxy/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/keepproxy/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/keepstore0/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/keepstore0/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/keepstore0/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/keepstore0/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/keepstore0/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/keepstore0/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/keepstore0/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/keepstore0/run
+++ b/build/arvbox/lib/arvbox/docker/service/keepstore0/run
diff --cc build/arvbox/lib/arvbox/docker/service/keepstore0/run-service
index 0000000,cf411e4..cf411e4
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/keepstore0/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/keepstore0/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/keepstore1/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/keepstore1/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/keepstore1/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/keepstore1/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/keepstore1/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/keepstore1/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/keepstore1/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/keepstore1/run
+++ b/build/arvbox/lib/arvbox/docker/service/keepstore1/run
diff --cc build/arvbox/lib/arvbox/docker/service/keepstore1/run-service
index 0000000,8d34d06..8d34d06
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/keepstore1/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/keepstore1/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/postgres/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/postgres/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/postgres/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/postgres/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/postgres/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/postgres/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/postgres/run
index 0000000,4918bd7..4918bd7
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/postgres/run
+++ b/build/arvbox/lib/arvbox/docker/service/postgres/run
diff --cc build/arvbox/lib/arvbox/docker/service/postgres/run-service
index 0000000,a05be62..a05be62
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/postgres/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/postgres/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/ready/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/ready/run
+++ b/build/arvbox/lib/arvbox/docker/service/ready/run
diff --cc build/arvbox/lib/arvbox/docker/service/ready/run-service
index 0000000,f560de0..f560de0
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/ready/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/ready/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/sdk/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/sdk/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/sdk/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/sdk/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/sdk/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/sdk/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/sdk/run
index 0000000,816b166..816b166
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/sdk/run
+++ b/build/arvbox/lib/arvbox/docker/service/sdk/run
diff --cc build/arvbox/lib/arvbox/docker/service/sdk/run-service
index 0000000,b51f0fc..b51f0fc
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/sdk/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/sdk/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/ssh/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/ssh/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/ssh/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/ssh/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/ssh/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/ssh/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/ssh/run
index 0000000,0f23542..0f23542
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/ssh/run
+++ b/build/arvbox/lib/arvbox/docker/service/ssh/run
diff --cc build/arvbox/lib/arvbox/docker/service/sso/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/sso/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/sso/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/sso/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/sso/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/sso/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/sso/run
index 0000000,a388c8b..a388c8b
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/sso/run
+++ b/build/arvbox/lib/arvbox/docker/service/sso/run
diff --cc build/arvbox/lib/arvbox/docker/service/sso/run-service
index 0000000,da413e0..da413e0
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/sso/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/sso/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/vm/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/vm/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/vm/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/vm/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/vm/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/vm/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/vm/run
index 0000000,b7fb9cc..b7fb9cc
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/vm/run
+++ b/build/arvbox/lib/arvbox/docker/service/vm/run
diff --cc build/arvbox/lib/arvbox/docker/service/vm/run-service
index 0000000,5bb6825..5bb6825
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/vm/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/vm/run-service
diff --cc build/arvbox/lib/arvbox/docker/service/workbench/log/main/.gitstub
index 0000000,e69de29..e69de29
mode 000000,100644..100644
--- a/build/arvbox/lib/arvbox/docker/service/workbench/log/main/.gitstub
+++ b/build/arvbox/lib/arvbox/docker/service/workbench/log/main/.gitstub
diff --cc build/arvbox/lib/arvbox/docker/service/workbench/log/run
index 0000000,d6aef4a..d6aef4a
mode 000000,120000..120000
--- a/build/arvbox/lib/arvbox/docker/service/workbench/log/run
+++ b/build/arvbox/lib/arvbox/docker/service/workbench/log/run
diff --cc build/arvbox/lib/arvbox/docker/service/workbench/run
index 0000000,6ac0476..6ac0476
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/workbench/run
+++ b/build/arvbox/lib/arvbox/docker/service/workbench/run
diff --cc build/arvbox/lib/arvbox/docker/service/workbench/run-service
index 0000000,850022a..850022a
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/service/workbench/run-service
+++ b/build/arvbox/lib/arvbox/docker/service/workbench/run-service
diff --cc build/arvbox/lib/arvbox/docker/waitforpostgres.sh
index 0000000,84d9904..84d9904
mode 000000,100755..100755
--- a/build/arvbox/lib/arvbox/docker/waitforpostgres.sh
+++ b/build/arvbox/lib/arvbox/docker/waitforpostgres.sh
diff --cc build/git/hooks/coding-standards.sh
index 0000000,d4e4c71..d4e4c71
mode 000000,100755..100755
--- a/build/git/hooks/coding-standards.sh
+++ b/build/git/hooks/coding-standards.sh
diff --cc build/install/easy-docker-install.sh
index 0000000,fe6e186..fe6e186
mode 000000,100755..100755
--- a/build/install/easy-docker-install.sh
+++ b/build/install/easy-docker-install.sh
diff --cc build/jenkins/create-plot-data-from-log.sh
index 0000000,ce3bfed..ce3bfed
mode 000000,100755..100755
--- a/build/jenkins/create-plot-data-from-log.sh
+++ b/build/jenkins/create-plot-data-from-log.sh
diff --cc build/jenkins/libcloud-pin
index 0000000,3fa07e6..3fa07e6
mode 000000,100644..100644
--- a/build/jenkins/libcloud-pin
+++ b/build/jenkins/libcloud-pin
diff --cc build/jenkins/package-build-dockerfiles/.gitignore
index 0000000,ceee9fa..ceee9fa
mode 000000,100644..100644
--- a/build/jenkins/package-build-dockerfiles/.gitignore
+++ b/build/jenkins/package-build-dockerfiles/.gitignore
diff --cc build/jenkins/package-build-dockerfiles/Makefile
index 0000000,70fbf28..70fbf28
mode 000000,100644..100644
--- a/build/jenkins/package-build-dockerfiles/Makefile
+++ b/build/jenkins/package-build-dockerfiles/Makefile
diff --cc build/jenkins/package-build-dockerfiles/README
index 0000000,0dfab94..0dfab94
mode 000000,100644..100644
--- a/build/jenkins/package-build-dockerfiles/README
+++ b/build/jenkins/package-build-dockerfiles/README
diff --cc build/jenkins/package-build-dockerfiles/build-all-build-containers.sh
index 0000000,34ffcce..34ffcce
mode 000000,100755..100755
--- a/build/jenkins/package-build-dockerfiles/build-all-build-containers.sh
+++ b/build/jenkins/package-build-dockerfiles/build-all-build-containers.sh
diff --cc build/jenkins/package-build-dockerfiles/centos6/Dockerfile
index 0000000,2bbec2e..2bbec2e
mode 000000,100644..100644
--- a/build/jenkins/package-build-dockerfiles/centos6/Dockerfile
+++ b/build/jenkins/package-build-dockerfiles/centos6/Dockerfile
diff --cc build/jenkins/package-build-dockerfiles/debian7/Dockerfile
index 0000000,21b554f..21b554f
mode 000000,100644..100644
--- a/build/jenkins/package-build-dockerfiles/debian7/Dockerfile
+++ b/build/jenkins/package-build-dockerfiles/debian7/Dockerfile
diff --cc build/jenkins/package-build-dockerfiles/debian8/Dockerfile
index 0000000,6fb622a..6fb622a
mode 000000,100644..100644
--- a/build/jenkins/package-build-dockerfiles/debian8/Dockerfile
+++ b/build/jenkins/package-build-dockerfiles/debian8/Dockerfile
diff --cc build/jenkins/package-build-dockerfiles/ubuntu1204/Dockerfile
index 0000000,4984d37..4984d37
mode 000000,100644..100644
--- a/build/jenkins/package-build-dockerfiles/ubuntu1204/Dockerfile
+++ b/build/jenkins/package-build-dockerfiles/ubuntu1204/Dockerfile
diff --cc build/jenkins/package-build-dockerfiles/ubuntu1404/Dockerfile
index 0000000,a3d6c8d..a3d6c8d
mode 000000,100644..100644
--- a/build/jenkins/package-build-dockerfiles/ubuntu1404/Dockerfile
+++ b/build/jenkins/package-build-dockerfiles/ubuntu1404/Dockerfile
diff --cc build/jenkins/package-test-dockerfiles/centos6/Dockerfile
index 0000000,69927a1..69927a1
mode 000000,100644..100644
--- a/build/jenkins/package-test-dockerfiles/centos6/Dockerfile
+++ b/build/jenkins/package-test-dockerfiles/centos6/Dockerfile
diff --cc build/jenkins/package-test-dockerfiles/centos6/localrepo.repo
index 0000000,ac6b898..ac6b898
mode 000000,100644..100644
--- a/build/jenkins/package-test-dockerfiles/centos6/localrepo.repo
+++ b/build/jenkins/package-test-dockerfiles/centos6/localrepo.repo
diff --cc build/jenkins/package-test-dockerfiles/debian7/Dockerfile
index 0000000,c9a2fdc..c9a2fdc
mode 000000,100644..100644
--- a/build/jenkins/package-test-dockerfiles/debian7/Dockerfile
+++ b/build/jenkins/package-test-dockerfiles/debian7/Dockerfile
diff --cc build/jenkins/package-test-dockerfiles/debian8/Dockerfile
index 0000000,cde1847..cde1847
mode 000000,100644..100644
--- a/build/jenkins/package-test-dockerfiles/debian8/Dockerfile
+++ b/build/jenkins/package-test-dockerfiles/debian8/Dockerfile
diff --cc build/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
index 0000000,0cb77c8..0cb77c8
mode 000000,100644..100644
--- a/build/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
+++ b/build/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
diff --cc build/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
index 0000000,6c4d0e9..6c4d0e9
mode 000000,100644..100644
--- a/build/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
+++ b/build/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
diff --cc build/jenkins/package-testing/common-test-packages.sh
index 0000000,2dc67ab..2dc67ab
mode 000000,100755..100755
--- a/build/jenkins/package-testing/common-test-packages.sh
+++ b/build/jenkins/package-testing/common-test-packages.sh
diff --cc build/jenkins/package-testing/deb-common-test-packages.sh
index 0000000,5f32a60..5f32a60
mode 000000,100755..100755
--- a/build/jenkins/package-testing/deb-common-test-packages.sh
+++ b/build/jenkins/package-testing/deb-common-test-packages.sh
diff --cc build/jenkins/package-testing/test-package-arvados-api-server.sh
index 0000000,e975448..e975448
mode 000000,100755..100755
--- a/build/jenkins/package-testing/test-package-arvados-api-server.sh
+++ b/build/jenkins/package-testing/test-package-arvados-api-server.sh
diff --cc build/jenkins/package-testing/test-package-arvados-node-manager.sh
index 0000000,2f416d1..2f416d1
mode 000000,100755..100755
--- a/build/jenkins/package-testing/test-package-arvados-node-manager.sh
+++ b/build/jenkins/package-testing/test-package-arvados-node-manager.sh
diff --cc build/jenkins/package-testing/test-package-arvados-sso-server.sh
index 0000000,c1a377e..c1a377e
mode 000000,100755..100755
--- a/build/jenkins/package-testing/test-package-arvados-sso-server.sh
+++ b/build/jenkins/package-testing/test-package-arvados-sso-server.sh
diff --cc build/jenkins/package-testing/test-package-arvados-workbench.sh
index 0000000,1be4dea..1be4dea
mode 000000,100755..100755
--- a/build/jenkins/package-testing/test-package-arvados-workbench.sh
+++ b/build/jenkins/package-testing/test-package-arvados-workbench.sh
diff --cc build/jenkins/package-testing/test-package-python27-python-arvados-fuse.sh
index 0000000,1654be9..1654be9
mode 000000,100755..100755
--- a/build/jenkins/package-testing/test-package-python27-python-arvados-fuse.sh
+++ b/build/jenkins/package-testing/test-package-python27-python-arvados-fuse.sh
diff --cc build/jenkins/package-testing/test-package-python27-python-arvados-python-client.sh
index 0000000,0772fbf..0772fbf
mode 000000,100755..100755
--- a/build/jenkins/package-testing/test-package-python27-python-arvados-python-client.sh
+++ b/build/jenkins/package-testing/test-package-python27-python-arvados-python-client.sh
diff --cc build/jenkins/package-testing/test-packages-centos6.sh
index 0000000,4e05364..4e05364
mode 000000,100755..100755
--- a/build/jenkins/package-testing/test-packages-centos6.sh
+++ b/build/jenkins/package-testing/test-packages-centos6.sh
diff --cc build/jenkins/package-testing/test-packages-debian7.sh
index 0000000,54ce94c..54ce94c
mode 000000,120000..120000
--- a/build/jenkins/package-testing/test-packages-debian7.sh
+++ b/build/jenkins/package-testing/test-packages-debian7.sh
diff --cc build/jenkins/package-testing/test-packages-debian8.sh
index 0000000,54ce94c..54ce94c
mode 000000,120000..120000
--- a/build/jenkins/package-testing/test-packages-debian8.sh
+++ b/build/jenkins/package-testing/test-packages-debian8.sh
diff --cc build/jenkins/package-testing/test-packages-ubuntu1204.sh
index 0000000,54ce94c..54ce94c
mode 000000,120000..120000
--- a/build/jenkins/package-testing/test-packages-ubuntu1204.sh
+++ b/build/jenkins/package-testing/test-packages-ubuntu1204.sh
diff --cc build/jenkins/package-testing/test-packages-ubuntu1404.sh
index 0000000,54ce94c..54ce94c
mode 000000,120000..120000
--- a/build/jenkins/package-testing/test-packages-ubuntu1404.sh
+++ b/build/jenkins/package-testing/test-packages-ubuntu1404.sh
diff --cc build/jenkins/rails-package-scripts/README.md
index 0000000,3a93c31..3a93c31
mode 000000,100644..100644
--- a/build/jenkins/rails-package-scripts/README.md
+++ b/build/jenkins/rails-package-scripts/README.md
diff --cc build/jenkins/rails-package-scripts/arvados-api-server.sh
index 0000000,c2b99f0..c2b99f0
mode 000000,100644..100644
--- a/build/jenkins/rails-package-scripts/arvados-api-server.sh
+++ b/build/jenkins/rails-package-scripts/arvados-api-server.sh
diff --cc build/jenkins/rails-package-scripts/arvados-sso-server.sh
index 0000000,10b2ee2..10b2ee2
mode 000000,100644..100644
--- a/build/jenkins/rails-package-scripts/arvados-sso-server.sh
+++ b/build/jenkins/rails-package-scripts/arvados-sso-server.sh
diff --cc build/jenkins/rails-package-scripts/arvados-workbench.sh
index 0000000,f2b8a56..f2b8a56
mode 000000,100644..100644
--- a/build/jenkins/rails-package-scripts/arvados-workbench.sh
+++ b/build/jenkins/rails-package-scripts/arvados-workbench.sh
diff --cc build/jenkins/rails-package-scripts/postinst.sh
index 0000000,5ff2a9b..5ff2a9b
mode 000000,100644..100644
--- a/build/jenkins/rails-package-scripts/postinst.sh
+++ b/build/jenkins/rails-package-scripts/postinst.sh
diff --cc build/jenkins/rails-package-scripts/postrm.sh
index 0000000,8c45d2f..8c45d2f
mode 000000,100644..100644
--- a/build/jenkins/rails-package-scripts/postrm.sh
+++ b/build/jenkins/rails-package-scripts/postrm.sh
diff --cc build/jenkins/rails-package-scripts/prerm.sh
index 0000000,4ef5904..4ef5904
mode 000000,100644..100644
--- a/build/jenkins/rails-package-scripts/prerm.sh
+++ b/build/jenkins/rails-package-scripts/prerm.sh
diff --cc build/jenkins/rails-package-scripts/step2.sh
index 0000000,816b906..816b906
mode 000000,100644..100644
--- a/build/jenkins/rails-package-scripts/step2.sh
+++ b/build/jenkins/rails-package-scripts/step2.sh
diff --cc build/jenkins/run-build-docker-images.sh
index 0000000,71910c2..71910c2
mode 000000,100755..100755
--- a/build/jenkins/run-build-docker-images.sh
+++ b/build/jenkins/run-build-docker-images.sh
diff --cc build/jenkins/run-build-docker-jobs-image.sh
index 0000000,fcf849b..fcf849b
mode 000000,100755..100755
--- a/build/jenkins/run-build-docker-jobs-image.sh
+++ b/build/jenkins/run-build-docker-jobs-image.sh
diff --cc build/jenkins/run-build-packages-all-targets.sh
index 0000000,f1a1e1c..f1a1e1c
mode 000000,100755..100755
--- a/build/jenkins/run-build-packages-all-targets.sh
+++ b/build/jenkins/run-build-packages-all-targets.sh
diff --cc build/jenkins/run-build-packages-one-target.sh
index 0000000,9597703..9597703
mode 000000,100755..100755
--- a/build/jenkins/run-build-packages-one-target.sh
+++ b/build/jenkins/run-build-packages-one-target.sh
diff --cc build/jenkins/run-build-packages-sso.sh
index 0000000,cc673a6..cc673a6
mode 000000,100755..100755
--- a/build/jenkins/run-build-packages-sso.sh
+++ b/build/jenkins/run-build-packages-sso.sh
diff --cc build/jenkins/run-build-packages.sh
index 0000000,97251c5..97251c5
mode 000000,100755..100755
--- a/build/jenkins/run-build-packages.sh
+++ b/build/jenkins/run-build-packages.sh
diff --cc build/jenkins/run-cwl-tests.sh
index 0000000,53c0538..53c0538
mode 000000,100755..100755
--- a/build/jenkins/run-cwl-tests.sh
+++ b/build/jenkins/run-cwl-tests.sh
diff --cc build/jenkins/run-deploy.sh
index 0000000,1b06c65..1b06c65
mode 000000,100755..100755
--- a/build/jenkins/run-deploy.sh
+++ b/build/jenkins/run-deploy.sh
diff --cc build/jenkins/run-diagnostics-suite.sh
index 0000000,015a053..015a053
mode 000000,100755..100755
--- a/build/jenkins/run-diagnostics-suite.sh
+++ b/build/jenkins/run-diagnostics-suite.sh
diff --cc build/jenkins/run-library.sh
index 0000000,23ed383..23ed383
mode 000000,100755..100755
--- a/build/jenkins/run-library.sh
+++ b/build/jenkins/run-library.sh
diff --cc build/jenkins/run-performance-suite.sh
index 0000000,2944bda..2944bda
mode 000000,100755..100755
--- a/build/jenkins/run-performance-suite.sh
+++ b/build/jenkins/run-performance-suite.sh
diff --cc build/jenkins/run-tapestry-tests.sh
index 0000000,851a81d..851a81d
mode 000000,100755..100755
--- a/build/jenkins/run-tapestry-tests.sh
+++ b/build/jenkins/run-tapestry-tests.sh
diff --cc build/jenkins/run-tests.sh
index 0000000,1289095..1289095
mode 000000,100755..100755
--- a/build/jenkins/run-tests.sh
+++ b/build/jenkins/run-tests.sh
diff --cc build/jenkins/run_upload_packages.py
index 0000000,04e6c80..04e6c80
mode 000000,100755..100755
--- a/build/jenkins/run_upload_packages.py
+++ b/build/jenkins/run_upload_packages.py
commit 8568240aaa6b9dba2af675dc0d6e09fe1001abfd
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Tue Feb 16 10:16:55 2016 -0500
Move services to /var/lib/arvbox/services because they include logs and logs
don't go in /usr. no issue #
diff --git a/arvbox/bin/arvbox b/arvbox/bin/arvbox
index 10fac0d..7be361b 100755
--- a/arvbox/bin/arvbox
+++ b/arvbox/bin/arvbox
@@ -181,12 +181,12 @@ run() {
docker exec -ti \
$ARVBOX_CONTAINER \
/usr/local/lib/arvbox/runsu.sh \
- /usr/local/lib/arvbox/service/sso/run-service --only-setup
+ /var/lib/arvbox/service/sso/run-service --only-setup
docker exec -ti \
$ARVBOX_CONTAINER \
/usr/local/lib/arvbox/runsu.sh \
- /usr/local/lib/arvbox/service/api/run-service --only-setup
+ /var/lib/arvbox/service/api/run-service --only-setup
docker exec -ti \
$ARVBOX_CONTAINER \
diff --git a/arvbox/lib/arvbox/docker/Dockerfile.demo b/arvbox/lib/arvbox/docker/Dockerfile.demo
index 1085abc..d3e36c6 100644
--- a/arvbox/lib/arvbox/docker/Dockerfile.demo
+++ b/arvbox/lib/arvbox/docker/Dockerfile.demo
@@ -6,9 +6,9 @@ RUN cd /usr/src && \
RUN chown -R 1000:1000 /usr/src && /usr/local/lib/arvbox/createusers.sh
-RUN sudo -u arvbox /usr/local/lib/arvbox/service/sso/run-service --only-deps
-RUN sudo -u arvbox /usr/local/lib/arvbox/service/api/run-service --only-deps
-RUN sudo -u arvbox /usr/local/lib/arvbox/service/workbench/run-service --only-deps
-RUN sudo -u arvbox /usr/local/lib/arvbox/service/doc/run-service --only-deps
-RUN sudo -u arvbox /usr/local/lib/arvbox/service/vm/run-service --only-deps
-RUN sudo -u arvbox /usr/local/lib/arvbox/service/sdk/run-service
+RUN sudo -u arvbox /var/lib/arvbox/service/sso/run-service --only-deps
+RUN sudo -u arvbox /var/lib/arvbox/service/api/run-service --only-deps
+RUN sudo -u arvbox /var/lib/arvbox/service/workbench/run-service --only-deps
+RUN sudo -u arvbox /var/lib/arvbox/service/doc/run-service --only-deps
+RUN sudo -u arvbox /var/lib/arvbox/service/vm/run-service --only-deps
+RUN sudo -u arvbox /var/lib/arvbox/service/sdk/run-service
diff --git a/arvbox/lib/arvbox/docker/Dockerfile.dev b/arvbox/lib/arvbox/docker/Dockerfile.dev
index 408e2de..5ec73bb 100644
--- a/arvbox/lib/arvbox/docker/Dockerfile.dev
+++ b/arvbox/lib/arvbox/docker/Dockerfile.dev
@@ -10,4 +10,4 @@ RUN set -e && \
tar -C /usr/local -xjf /tmp/$PJS.tar.bz2 && \
ln -s ../$PJS/bin/phantomjs /usr/local/bin/
-RUN mkdir /etc/test-service && ln -sf /usr/local/lib/arvbox/service/postgres /etc/test-service
+RUN mkdir /etc/test-service && ln -sf /var/lib/arvbox/service/postgres /etc/test-service
diff --git a/arvbox/lib/arvbox/docker/tests-service/docker b/arvbox/lib/arvbox/docker/tests-service/docker
deleted file mode 120000
index 9374f25..0000000
--- a/arvbox/lib/arvbox/docker/tests-service/docker
+++ /dev/null
@@ -1 +0,0 @@
-../service/docker
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/tests-service/logger b/arvbox/lib/arvbox/docker/tests-service/logger
deleted file mode 120000
index b30194b..0000000
--- a/arvbox/lib/arvbox/docker/tests-service/logger
+++ /dev/null
@@ -1 +0,0 @@
-../service/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/tests-service/postgres b/arvbox/lib/arvbox/docker/tests-service/postgres
deleted file mode 120000
index 9b2d8a0..0000000
--- a/arvbox/lib/arvbox/docker/tests-service/postgres
+++ /dev/null
@@ -1 +0,0 @@
-../service/postgres
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/tests-service/runsu.sh b/arvbox/lib/arvbox/docker/tests-service/runsu.sh
deleted file mode 120000
index 22030b0..0000000
--- a/arvbox/lib/arvbox/docker/tests-service/runsu.sh
+++ /dev/null
@@ -1 +0,0 @@
-../service/runsu.sh
\ No newline at end of file
commit aa65f4dc68b750a635c857c0480df3f5a2dca7a4
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Feb 15 16:37:00 2016 -0500
Explicitly install Docker 1.9.1 instead of using get.dockerproject.org (which gets the latest). no issue #
diff --git a/arvbox/lib/arvbox/docker/Dockerfile.base b/arvbox/lib/arvbox/docker/Dockerfile.base
index c0dc767..160afee 100644
--- a/arvbox/lib/arvbox/docker/Dockerfile.base
+++ b/arvbox/lib/arvbox/docker/Dockerfile.base
@@ -9,13 +9,21 @@ RUN apt-get update && \
libpython-dev fuse libfuse-dev python-pip python-yaml \
pkg-config libattr1-dev python-llfuse python-pycurl \
libwww-perl libio-socket-ssl-perl libcrypt-ssleay-perl \
- libjson-perl nginx gitolite3 lsof python-epydoc graphviz
+ libjson-perl nginx gitolite3 lsof python-epydoc graphviz \
+ apt-transport-https ca-certificates
-RUN curl -sSL https://get.docker.com/ | sh
VOLUME /var/lib/docker
VOLUME /var/log/nginx
VOLUME /etc/ssl/private
+RUN apt-key adv --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D || \
+ apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
+
+RUN mkdir -p /etc/apt/sources.list.d && \
+ echo deb https://apt.dockerproject.org/repo debian-jessie main > /etc/apt/sources.list.d/docker.list && \
+ apt-get update && \
+ DEBIAN_FRONTEND=noninteractive apt-get -yq install docker-engine=1.9.1-0~jessie
+
RUN rm -rf /var/lib/postgresql && mkdir -p /var/lib/postgresql
RUN cd /root && \
@@ -30,8 +38,8 @@ ADD crunch-setup.sh gitolite.rc \
application_yml_override.py \
/usr/local/lib/arvbox/
-ADD service/ /usr/local/lib/arvbox/service
-RUN rmdir /etc/service && ln -sf /usr/local/lib/arvbox/service /etc
+ADD service/ /var/lib/arvbox/service
+RUN rmdir /etc/service && ln -sf /var/lib/arvbox/service /etc
# Start the supervisor.
CMD ["/usr/local/bin/runsvinit"]
commit 819ffb0ad38eb1f86ab6b6e3c4ae7c860b465636
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Sat Feb 13 23:18:27 2016 -0500
Add crunch-dispatch-local to arvbox. no issue #
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/run b/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run b/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run-service b/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run-service
new file mode 100755
index 0000000..211b438
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/crunch-dispatch-local/run-service
@@ -0,0 +1,22 @@
+#!/bin/bash
+
+exec 2>&1
+set -eux -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+mkdir -p /var/lib/arvados/gostuff
+cd /var/lib/arvados/gostuff
+
+export GOPATH=$PWD
+mkdir -p "$GOPATH/src/git.curoverse.com"
+ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/crunch-run"
+flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/crunch-dispatch-local"
+install bin/crunch-run bin/crunch-dispatch-local /usr/local/bin
+
+export ARVADOS_API_HOST=$localip:${services[api]}
+export ARVADOS_API_HOST_INSECURE=1
+export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+
+exec /usr/local/bin/crunch-dispatch-local -crunch-run-command=/usr/local/bin/crunch-run
commit 6f19089ca96e485f457932dce44256e46f1e3711
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Feb 12 15:14:01 2016 -0500
Add arvbox support for overriding settings in application.yml
no issue #
diff --git a/arvbox/README.md b/arvbox/README.md
index 9ced1d1..8098686 100644
--- a/arvbox/README.md
+++ b/arvbox/README.md
@@ -46,6 +46,11 @@ are bind mounted from the host file system for easy access and persistence
across container rebuilds. Services are bound to the Docker container's
network IP address and can only be accessed on the local host.
+In "dev" mode, you can override the default autogenerated settings of Rails
+projects by adding "application.yml.override" to any Rails project (sso, api,
+workbench). This can be used to test out API server settings or point
+Workbench at an alternate API server.
+
### localdemo
Demo configuration. Boots a complete Arvados environment inside the container.
Unlike the development configuration, code directories are included in the demo
diff --git a/arvbox/bin/arvbox b/arvbox/bin/arvbox
index d60c354..10fac0d 100755
--- a/arvbox/bin/arvbox
+++ b/arvbox/bin/arvbox
@@ -196,7 +196,7 @@ run() {
WORKSPACE=/usr/src/arvados \
GEM_HOME=/var/lib/gems \
"$@"
- else
+ elif echo "$1" | grep 'dev$' ; then
docker run \
--detach \
--name=$ARVBOX_CONTAINER \
@@ -212,6 +212,8 @@ run() {
updateconf
wait_for_arvbox
echo "The Arvados source code is checked out at: $ARVADOS_ROOT"
+ else
+ echo "Unknown configuration '$1'"
fi
fi
}
@@ -348,15 +350,18 @@ case "$subcmd" in
fi
;;
- log|svrestart)
+ log)
if test -n "$1" ; then
- if test "$subcmd" = log ; then
- docker exec -ti $ARVBOX_CONTAINER /usr/bin/env TERM=$TERM less --follow-name +GF "/etc/service/$1/log/main/current"
- fi
- if test "$subcmd" = svrestart ; then
- docker exec -ti $ARVBOX_CONTAINER sv restart "$1"
- docker exec -ti $ARVBOX_CONTAINER sv restart ready
- fi
+ docker exec -ti $ARVBOX_CONTAINER /usr/bin/env TERM=$TERM less --follow-name +GF "/etc/service/$1/log/main/current"
+ else
+ docker exec -ti $ARVBOX_CONTAINER /usr/bin/env TERM=$TERM tail $(docker exec -ti $ARVBOX_CONTAINER find -L /etc -path '/etc/service/*/log/main/current' -printf " %p")
+ fi
+ ;;
+
+ svrestart)
+ if test -n "$1" ; then
+ docker exec -ti $ARVBOX_CONTAINER sv restart "$1"
+ docker exec -ti $ARVBOX_CONTAINER sv restart ready
else
echo "Usage: $0 $subcmd <service>"
echo "Available services:"
diff --git a/arvbox/lib/arvbox/docker/Dockerfile.base b/arvbox/lib/arvbox/docker/Dockerfile.base
index 108ed53..c0dc767 100644
--- a/arvbox/lib/arvbox/docker/Dockerfile.base
+++ b/arvbox/lib/arvbox/docker/Dockerfile.base
@@ -6,7 +6,7 @@ RUN apt-get update && \
ruby rake bundler curl libpq-dev \
libcurl4-openssl-dev libssl-dev zlib1g-dev libpcre3-dev \
openssh-server python-setuptools netcat-traditional \
- libpython-dev fuse libfuse-dev python-pip \
+ libpython-dev fuse libfuse-dev python-pip python-yaml \
pkg-config libattr1-dev python-llfuse python-pycurl \
libwww-perl libio-socket-ssl-perl libcrypt-ssleay-perl \
libjson-perl nginx gitolite3 lsof python-epydoc graphviz
@@ -27,7 +27,9 @@ ADD fuse.conf /etc/
ADD crunch-setup.sh gitolite.rc \
keep-setup.sh common.sh createusers.sh \
logger runsu.sh waitforpostgres.sh \
+ application_yml_override.py \
/usr/local/lib/arvbox/
+
ADD service/ /usr/local/lib/arvbox/service
RUN rmdir /etc/service && ln -sf /usr/local/lib/arvbox/service /etc
diff --git a/arvbox/lib/arvbox/docker/application_yml_override.py b/arvbox/lib/arvbox/docker/application_yml_override.py
new file mode 100755
index 0000000..98a8e48
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/application_yml_override.py
@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+import yaml
+
+try:
+ with open("application.yml.override") as f:
+ b = yaml.load(f)
+except IOError:
+ exit()
+
+with open("application.yml") as f:
+ a = yaml.load(f)
+
+def recursiveMerge(a, b):
+ if isinstance(a, dict) and isinstance(b, dict):
+ for k in b:
+ print k
+ a[k] = recursiveMerge(a.get(k), b[k])
+ return a
+ else:
+ return b
+
+with open("application.yml", "w") as f:
+ yaml.dump(recursiveMerge(a, b), f)
diff --git a/arvbox/lib/arvbox/docker/service/api/run-service b/arvbox/lib/arvbox/docker/service/api/run-service
index 530a039..058939c 100755
--- a/arvbox/lib/arvbox/docker/service/api/run-service
+++ b/arvbox/lib/arvbox/docker/service/api/run-service
@@ -64,6 +64,8 @@ development:
default_collection_replication: 1
EOF
+(cd config && /usr/local/lib/arvbox/application_yml_override.py)
+
if ! test -f /var/lib/arvados/api_database_pw ; then
ruby -e 'puts rand(2**128).to_s(36)' > /var/lib/arvados/api_database_pw
fi
diff --git a/arvbox/lib/arvbox/docker/service/sso/run-service b/arvbox/lib/arvbox/docker/service/sso/run-service
index 2f501a4..da413e0 100755
--- a/arvbox/lib/arvbox/docker/service/sso/run-service
+++ b/arvbox/lib/arvbox/docker/service/sso/run-service
@@ -39,6 +39,8 @@ development:
allow_account_registration: true
EOF
+(cd config && /usr/local/lib/arvbox/application_yml_override.py)
+
if ! test -f /var/lib/arvados/sso_database_pw ; then
ruby -e 'puts rand(2**128).to_s(36)' > /var/lib/arvados/sso_database_pw
fi
diff --git a/arvbox/lib/arvbox/docker/service/workbench/run-service b/arvbox/lib/arvbox/docker/service/workbench/run-service
index 7f0542f..850022a 100755
--- a/arvbox/lib/arvbox/docker/service/workbench/run-service
+++ b/arvbox/lib/arvbox/docker/service/workbench/run-service
@@ -36,3 +36,5 @@ development:
keep_web_url: http://$localip:${services[keep-web]}/c=%{uuid_or_pdh}
arvados_docsite: http://$localip:${services[doc]}/
EOF
+
+(cd config && /usr/local/lib/arvbox/application_yml_override.py)
commit bb26448a7551a9165e8674ade83e32f4d17adc9a
Author: Brett Smith <brett at curoverse.com>
Date: Wed Feb 3 14:24:07 2016 -0500
Inhibit udev startup in Debian test Docker images.
The FUSE driver package depends on fuse, which depends on udev.
udev can't start its daemon from postinst in a container.
It has code to try to detect running in a container and skip the
daemon, but it fails intermittently.
e.g., <https://ci.curoverse.com/job/build-packages/1699/consoleFull>
It also skips starting the daemon when /etc/udev/disabled exists, so
create that.
diff --git a/jenkins/package-test-dockerfiles/debian7/Dockerfile b/jenkins/package-test-dockerfiles/debian7/Dockerfile
index 3277e60..c9a2fdc 100644
--- a/jenkins/package-test-dockerfiles/debian7/Dockerfile
+++ b/jenkins/package-test-dockerfiles/debian7/Dockerfile
@@ -8,4 +8,7 @@ RUN apt-get update && apt-get -y install curl procps && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1
+# udev daemon can't start in a container, so don't try.
+RUN mkdir -p /etc/udev/disabled
+
RUN echo "deb file:///arvados/packages/debian7/ /" >>/etc/apt/sources.list
diff --git a/jenkins/package-test-dockerfiles/debian8/Dockerfile b/jenkins/package-test-dockerfiles/debian8/Dockerfile
index 01abbc9..cde1847 100644
--- a/jenkins/package-test-dockerfiles/debian8/Dockerfile
+++ b/jenkins/package-test-dockerfiles/debian8/Dockerfile
@@ -8,4 +8,7 @@ RUN apt-get update && apt-get -y install curl && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1
+# udev daemon can't start in a container, so don't try.
+RUN mkdir -p /etc/udev/disabled
+
RUN echo "deb file:///arvados/packages/debian8/ /" >>/etc/apt/sources.list
diff --git a/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile b/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
index 80573bf..0cb77c8 100644
--- a/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
+++ b/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
@@ -8,4 +8,7 @@ RUN apt-get update && apt-get -y install curl && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1
+# udev daemon can't start in a container, so don't try.
+RUN mkdir -p /etc/udev/disabled
+
RUN echo "deb file:///arvados/packages/ubuntu1204/ /" >>/etc/apt/sources.list
\ No newline at end of file
diff --git a/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile b/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
index 2d8ebe9..6c4d0e9 100644
--- a/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
+++ b/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
@@ -8,4 +8,7 @@ RUN apt-get update && apt-get -y install curl && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1
+# udev daemon can't start in a container, so don't try.
+RUN mkdir -p /etc/udev/disabled
+
RUN echo "deb file:///arvados/packages/ubuntu1404/ /" >>/etc/apt/sources.list
\ No newline at end of file
commit 48e6cb38b2e0c023589d4083192190bdc30c4762
Author: Brett Smith <brett at curoverse.com>
Date: Wed Feb 3 08:19:34 2016 -0500
.deb dependencies from Python packages ignore iteration.
See comment for rationale.
Inspired by <https://github.com/curoverse/arvados/pull/35>.
No issue #.
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index 83e0ea0..23ed383 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -218,6 +218,12 @@ fpm_build () {
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "$FORMAT")
if [ python = "$PACKAGE_TYPE" ]; then
COMMAND_ARR+=(--exclude=\*/{dist,site}-packages/tests/\*)
+ if [ deb = "$FORMAT" ]; then
+ # Dependencies are built from setup.py. Since setup.py will never
+ # refer to Debian package iterations, it doesn't make sense to
+ # enforce those in the .deb dependencies.
+ COMMAND_ARR+=(--deb-ignore-iteration-in-dependencies)
+ fi
fi
if [[ "$PACKAGE_NAME" != "$PACKAGE" ]]; then
commit 512193d7a3befd4172b5aa4685c01da84a690430
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Feb 1 09:16:52 2016 -0500
Pin llfuse to 0.41.1 because 0.42 came out and broke things. no issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 445b3a0..97251c5 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -92,7 +92,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname llfuse \
+ ciso8601 pycrypto backports.ssl_match_hostname llfuse==0.41.1 \
'pycurl<7.21.5')
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -105,7 +105,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname llfuse \
+ ciso8601 pycrypto backports.ssl_match_hostname llfuse==0.41.1 \
'pycurl<7.21.5')
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -118,7 +118,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname llfuse \
+ ciso8601 pycrypto backports.ssl_match_hostname llfuse==0.41.1 \
'pycurl<7.21.5')
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -128,7 +128,7 @@ case "$TARGET" in
PYTHON2_PKG_PREFIX=python
PYTHON3_PACKAGE=python$PYTHON3_VERSION
PYTHON3_PKG_PREFIX=python3
- PYTHON_BACKPORTS=(pyasn1==0.1.7 pyvcf pyasn1-modules==0.0.5 llfuse ciso8601 \
+ PYTHON_BACKPORTS=(pyasn1==0.1.7 pyvcf pyasn1-modules==0.0.5 llfuse==0.41.1 ciso8601 \
google-api-python-client six uritemplate oauth2client httplib2 \
rsa 'pycurl<7.21.5' backports.ssl_match_hostname)
PYTHON3_BACKPORTS=(docker-py requests websocket-client)
@@ -143,7 +143,7 @@ case "$TARGET" in
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname 'pycurl<7.21.5' \
- python-daemon lockfile llfuse 'pbr<1.0')
+ python-daemon lockfile llfuse==0.41.1 'pbr<1.0')
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
export PYCURL_SSL_LIBRARY=nss
;;
commit 17d02ba1aa5c25c101ae1a887f18e9dc5d2bb139
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Jan 29 16:15:41 2016 -0500
Add distinct fpm_exclude so it shows up in the right place on the fpm command
line. refs #8008
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index eb2384a..83e0ea0 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -244,6 +244,7 @@ fpm_build () {
# the arguments added by this script.
declare -a fpm_args=()
declare -a fpm_depends=()
+ declare -a fpm_exclude=()
if [[ -d "$PACKAGE_DIR" ]]; then
FPM_INFO="$PACKAGE_DIR/fpm-info.sh"
else
@@ -256,6 +257,9 @@ fpm_build () {
for i in "${fpm_depends[@]}"; do
COMMAND_ARR+=('--depends' "$i")
done
+ for i in "${fpm_exclude[@]}"; do
+ COMMAND_ARR+=('--exclude' "$i")
+ done
# Append remaining function arguments directly to fpm's command line.
for i; do
commit a1ca4978b7e4e6d917c50c64c3bf5e527690a7e4
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jan 28 14:55:29 2016 -0500
Use volumes for /var/log/nginx and /etc/ssl/private so that changing ownership
sticks. refs #8080
diff --git a/arvbox/lib/arvbox/docker/Dockerfile.base b/arvbox/lib/arvbox/docker/Dockerfile.base
index cdbface..108ed53 100644
--- a/arvbox/lib/arvbox/docker/Dockerfile.base
+++ b/arvbox/lib/arvbox/docker/Dockerfile.base
@@ -13,6 +13,8 @@ RUN apt-get update && \
RUN curl -sSL https://get.docker.com/ | sh
VOLUME /var/lib/docker
+VOLUME /var/log/nginx
+VOLUME /etc/ssl/private
RUN rm -rf /var/lib/postgresql && mkdir -p /var/lib/postgresql
diff --git a/arvbox/lib/arvbox/docker/createusers.sh b/arvbox/lib/arvbox/docker/createusers.sh
index 4a8abfd..b77c9c2 100755
--- a/arvbox/lib/arvbox/docker/createusers.sh
+++ b/arvbox/lib/arvbox/docker/createusers.sh
@@ -21,21 +21,12 @@ if ! grep "^arvbox:" /etc/passwd >/dev/null 2>/dev/null ; then
useradd --groups docker,fuse crunch
chown arvbox:arvbox -R /usr/local /var/lib/arvados /var/lib/gems \
- /var/lib/passenger /var/lib/postgresql /etc/ssl/private
+ /var/lib/passenger /var/lib/postgresql \
+ /var/lib/nginx /var/log/nginx /etc/ssl/private
mkdir -p /var/lib/gems/ruby/2.1.0
chown arvbox:arvbox -R /var/lib/gems/ruby/2.1.0
- chown arvbox:arvbox -R /var/lib/nginx
-
- # There's something weird about /var/log/nginx that prevents a non-root
- # arvbox user from writing to it, even after the ownership has been
- # changed. As a workaround, delete it and recreate it.
-
- rm -r /var/log/nginx
- mkdir -p /var/log/nginx
- chown arvbox:arvbox -R /var/log/nginx
-
mkdir -p /tmp/crunch0 /tmp/crunch1
chown crunch:crunch -R /tmp/crunch0 /tmp/crunch1
commit a86550746f8343128d9bc38eb115d9495decc58f
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jan 28 14:22:23 2016 -0500
Arvbox fixes: use 'postgres' database when testing if a user exists. Copy
gitolite.rc to the right place. Print out what is being deleted when using
"reset" or "destroy". refs #8080
diff --git a/arvbox/bin/arvbox b/arvbox/bin/arvbox
index 4f1d94a..d60c354 100755
--- a/arvbox/bin/arvbox
+++ b/arvbox/bin/arvbox
@@ -325,6 +325,7 @@ case "$subcmd" in
echo "Use destroy -f if you really mean it."
exit 1
fi
+ set -x
rm -rf "$ARVBOX_DATA"
else
if test "$1" != -f ; then
@@ -333,6 +334,7 @@ case "$subcmd" in
echo "Use reset -f if you really mean it."
exit 1
fi
+ set -x
rm -rf "$ARVBOX_DATA/postgres"
rm -rf "$ARVBOX_DATA/var"
fi
@@ -341,6 +343,7 @@ case "$subcmd" in
echo "WARNING! This will delete your data container $ARVBOX_CONTAINER-data. Use -f if you really mean it."
exit 1
fi
+ set -x
docker rm "$ARVBOX_CONTAINER-data"
fi
;;
diff --git a/arvbox/lib/arvbox/docker/service/api/run-service b/arvbox/lib/arvbox/docker/service/api/run-service
index 31f90d1..530a039 100755
--- a/arvbox/lib/arvbox/docker/service/api/run-service
+++ b/arvbox/lib/arvbox/docker/service/api/run-service
@@ -69,9 +69,9 @@ if ! test -f /var/lib/arvados/api_database_pw ; then
fi
database_pw=$(cat /var/lib/arvados/api_database_pw)
-if ! (psql -c "\du" | grep "^ arvados ") >/dev/null ; then
- psql -c "create user arvados with password '$database_pw'"
- psql -c "ALTER USER arvados CREATEDB;"
+if ! (psql postgres -c "\du" | grep "^ arvados ") >/dev/null ; then
+ psql postgres -c "create user arvados with password '$database_pw'"
+ psql postgres -c "ALTER USER arvados CREATEDB;"
fi
sed "s/password:.*/password: $database_pw/" <config/database.yml.example >config/database.yml
diff --git a/arvbox/lib/arvbox/docker/service/gitolite/run-service b/arvbox/lib/arvbox/docker/service/gitolite/run-service
index 8e6cb0e..e0e8771 100755
--- a/arvbox/lib/arvbox/docker/service/gitolite/run-service
+++ b/arvbox/lib/arvbox/docker/service/gitolite/run-service
@@ -40,7 +40,7 @@ if ! test -f /var/lib/arvados/gitolite-setup ; then
ssh -o stricthostkeychecking=no git at localhost true
rm .ssh/authorized_keys
- cp -r /usr/local/lib/arvbox/gitolite.rc .
+ cp /usr/local/lib/arvbox/gitolite.rc .gitolite.rc
gitolite setup -pk .ssh/id_rsa.pub
diff --git a/arvbox/lib/arvbox/docker/service/postgres/run-service b/arvbox/lib/arvbox/docker/service/postgres/run-service
index ecb0aa6..a05be62 100755
--- a/arvbox/lib/arvbox/docker/service/postgres/run-service
+++ b/arvbox/lib/arvbox/docker/service/postgres/run-service
@@ -5,7 +5,7 @@ set -eux -o pipefail
if ! test -d /var/lib/postgresql/9.4/main ; then
/usr/lib/postgresql/9.4/bin/initdb -D /var/lib/postgresql/9.4/main
- sh -c "while ! (psql -c'\du' | grep '^ arvbox ') >/dev/null ; do createuser -s arvbox ; sleep 1 ; done" &
+ sh -c "while ! (psql postgres -c'\du' | grep '^ arvbox ') >/dev/null ; do createuser -s arvbox ; sleep 1 ; done" &
fi
mkdir -p /var/run/postgresql/9.4-main.pg_stat_tmp
diff --git a/arvbox/lib/arvbox/docker/service/sso/run-service b/arvbox/lib/arvbox/docker/service/sso/run-service
index 8a96d47..2f501a4 100755
--- a/arvbox/lib/arvbox/docker/service/sso/run-service
+++ b/arvbox/lib/arvbox/docker/service/sso/run-service
@@ -44,9 +44,9 @@ if ! test -f /var/lib/arvados/sso_database_pw ; then
fi
database_pw=$(cat /var/lib/arvados/sso_database_pw)
-if ! (psql -c "\du" | grep "^ arvados_sso ") >/dev/null ; then
- psql -c "create user arvados_sso with password '$database_pw'"
- psql -c "ALTER USER arvados_sso CREATEDB;"
+if ! (psql postgres -c "\du" | grep "^ arvados_sso ") >/dev/null ; then
+ psql postgres -c "create user arvados_sso with password '$database_pw'"
+ psql postgres -c "ALTER USER arvados_sso CREATEDB;"
fi
sed "s/password:.*/password: $database_pw/" <config/database.yml.example >config/database.yml
diff --git a/arvbox/lib/arvbox/docker/waitforpostgres.sh b/arvbox/lib/arvbox/docker/waitforpostgres.sh
index a07fa8c..84d9904 100755
--- a/arvbox/lib/arvbox/docker/waitforpostgres.sh
+++ b/arvbox/lib/arvbox/docker/waitforpostgres.sh
@@ -1,4 +1,4 @@
#!/bin/sh
-while ! psql -c\\du >/dev/null 2>/dev/null ; do
+while ! psql postgres -c\\du >/dev/null 2>/dev/null ; do
sleep 1
done
commit 42f2e4db0a40db50617d8631a2ff7e5add6539e4
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 27 16:40:53 2016 -0500
chown /etc/ssl/private to arvbox refs #8080
diff --git a/arvbox/lib/arvbox/docker/createusers.sh b/arvbox/lib/arvbox/docker/createusers.sh
index 1e6c95d..4a8abfd 100755
--- a/arvbox/lib/arvbox/docker/createusers.sh
+++ b/arvbox/lib/arvbox/docker/createusers.sh
@@ -20,7 +20,8 @@ if ! grep "^arvbox:" /etc/passwd >/dev/null 2>/dev/null ; then
useradd --home-dir /var/lib/arvados/git --uid $HOSTUID --gid $HOSTGID --non-unique git
useradd --groups docker,fuse crunch
- chown arvbox:arvbox -R /usr/local /var/lib/arvados /var/lib/gems /var/lib/passenger /var/lib/postgresql
+ chown arvbox:arvbox -R /usr/local /var/lib/arvados /var/lib/gems \
+ /var/lib/passenger /var/lib/postgresql /etc/ssl/private
mkdir -p /var/lib/gems/ruby/2.1.0
chown arvbox:arvbox -R /var/lib/gems/ruby/2.1.0
commit 5dacdfe33d9d72e2445fff52069c2afd604271d3
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 27 15:39:38 2016 -0500
Bump iteration of rails packages to -4 refs #8008
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index 7666837..eb2384a 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -149,7 +149,8 @@ handle_rails_package() {
"$(cat "$version_file")")
local license_arg="$license_path=$railsdir/$(basename "$license_path")"
# --iteration=3 accommodates the package scripts change from #8014.
- local -a switches=(--iteration=3
+ # --iteration=4 accommodates the packaging changes (inclusion of vendor/cache) from #8008.
+ local -a switches=(--iteration=4
--after-install "$scripts_dir/postinst"
--before-remove "$scripts_dir/prerm"
--after-remove "$scripts_dir/postrm")
commit aa204e80a18f8827405fbb5a67de2ef8fa7074fc
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 27 14:34:06 2016 -0500
Fix selecting most recent package for package testing, refs #8008.
diff --git a/jenkins/package-testing/deb-common-test-packages.sh b/jenkins/package-testing/deb-common-test-packages.sh
index 3928ba2..5f32a60 100755
--- a/jenkins/package-testing/deb-common-test-packages.sh
+++ b/jenkins/package-testing/deb-common-test-packages.sh
@@ -24,7 +24,7 @@ cd /tmp/opts
export ARV_PACKAGES_DIR="/arvados/packages/$target"
-dpkg-deb -x $ARV_PACKAGES_DIR/"$1"_*.deb .
+dpkg-deb -x $(ls -t "$ARV_PACKAGES_DIR/$1"_*.deb | head -n1) .
while read so && [ -n "$so" ]; do
echo
diff --git a/jenkins/package-testing/test-packages-centos6.sh b/jenkins/package-testing/test-packages-centos6.sh
index 1960944..4e05364 100755
--- a/jenkins/package-testing/test-packages-centos6.sh
+++ b/jenkins/package-testing/test-packages-centos6.sh
@@ -25,7 +25,7 @@ fi
mkdir -p /tmp/opts
cd /tmp/opts
-rpm2cpio "$ARV_PACKAGES_DIR/$1"-*.rpm | cpio -idm 2>/dev/null
+rpm2cpio $(ls -t "$ARV_PACKAGES_DIR/$1"-*.rpm | head -n1) | cpio -idm 2>/dev/null
shared=$(find -name '*.so')
if test -n "$shared" ; then
commit baa66db7b5b05032c408a60180fa81b98d079ee4
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jan 25 18:15:12 2016 -0500
8008: Make dependencies for build-dockerfiles.
We've apparently never had code to do this before. I guess all
deployments had the dependencies generated by hand before running.
This makes them automated to fix failures like
<https://ci.curoverse.com/job/build-packages/1668/console>.
Refs #8008.
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index e800897..9597703 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -111,6 +111,7 @@ if [[ -n "$test_packages" ]]; then
pushd "$JENKINS_DIR/package-test-dockerfiles"
else
pushd "$JENKINS_DIR/package-build-dockerfiles"
+ make "$TARGET/generated"
fi
echo $TARGET
commit 0a18c5fb313d91930f4ff0c5cf05aabfe5e1a748
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Jan 25 12:14:44 2016 -0500
Fix markdown for arvbox README.md refs #8080
diff --git a/arvbox/README.md b/arvbox/README.md
index cdb3914..9ced1d1 100644
--- a/arvbox/README.md
+++ b/arvbox/README.md
@@ -4,10 +4,13 @@ Self-contained development, demonstration and testing environment for Arvados.
## Quick start
+```
$ bin/arvbox reboot localdemo
+```
## Usage
+```
Arvados-in-a-box
arvbox (build|start|run|open|shell|ip|stop|reboot|reset|destroy|log|svrestart)
@@ -25,12 +28,13 @@ destroy delete all arvbox code and data (be careful!)
log <service> tail log of specified service
svrestart <service> restart specified service inside arvbox
clone <from> <to> clone an arvbox
+```
## Requirements
-Linux 3.x+ and Docker 1.9+
-Minimum of 3 GiB of RAM + additional memory to run jobs
-Minimum of 3 GiB of disk + storage for actual data
+* Linux 3.x+ and Docker 1.9+
+* Minimum of 3 GiB of RAM + additional memory to run jobs
+* Minimum of 3 GiB of disk + storage for actual data
## Configs
commit 738bac6d373ec54b4bf75b2da71b9c65203ecc76
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Jan 25 10:34:02 2016 -0500
Add "status" command, refs #8080
diff --git a/arvbox/bin/arvbox b/arvbox/bin/arvbox
index 284f0ef..4f1d94a 100755
--- a/arvbox/bin/arvbox
+++ b/arvbox/bin/arvbox
@@ -299,6 +299,23 @@ case "$subcmd" in
fi
;;
+ status)
+ echo "Selected: $ARVBOX_CONTAINER"
+ if docker ps -a --filter "status=running" | grep -E "$ARVBOX_CONTAINER$" -q ; then
+ echo "Status: running"
+ echo "IP: $(getip)"
+ else
+ echo "Status: not running"
+ fi
+ if test -d "$ARVBOX_DATA" ; then
+ echo "Data: $ARVBOX_DATA"
+ elif docker ps -a | grep -E "$ARVBOX_CONTAINER-data$" -q ; then
+ echo "Data: $ARVBOX_CONTAINER-data"
+ else
+ echo "Data: none"
+ fi
+ ;;
+
reset|destroy)
stop
if test -d "$ARVBOX_DATA" ; then
@@ -365,6 +382,7 @@ case "$subcmd" in
echo "open open arvbox workbench in a web browser"
echo "shell enter arvbox shell"
echo "ip print arvbox ip address"
+ echo "status print some information about current arvbox"
echo "stop stop arvbox container"
echo "restart <config> stop, then run again"
echo "reboot <config> stop, build arvbox Docker image, run"
commit 9a0aa9fcfdaec36bfa0cec543d8502f4006df2e8
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Jan 25 10:21:22 2016 -0500
Just create arvbox superuser instead of creating a whole useless database.
Check directly whether creating the arvbox is required. refs #8080
diff --git a/arvbox/lib/arvbox/docker/service/postgres/run-service b/arvbox/lib/arvbox/docker/service/postgres/run-service
index dd2eb1a..ecb0aa6 100755
--- a/arvbox/lib/arvbox/docker/service/postgres/run-service
+++ b/arvbox/lib/arvbox/docker/service/postgres/run-service
@@ -5,7 +5,7 @@ set -eux -o pipefail
if ! test -d /var/lib/postgresql/9.4/main ; then
/usr/lib/postgresql/9.4/bin/initdb -D /var/lib/postgresql/9.4/main
- sh -c "while ! psql -c'\du' >/dev/null 2>/dev/null ; do createdb ; sleep 1 ; done" &
+ sh -c "while ! (psql -c'\du' | grep '^ arvbox ') >/dev/null ; do createuser -s arvbox ; sleep 1 ; done" &
fi
mkdir -p /var/run/postgresql/9.4-main.pg_stat_tmp
commit ff110d616ea28bffb9677f91702f716992e7d130
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jan 22 17:44:55 2016 -0500
Make sure not to accidentally include config/application.yml and
config/database.yml in our rails packages, take 2: make sure that
config/database.yml *is* present in our workbench packages, where we
want to include it.
No issue #
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index aed260e..7666837 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -157,10 +157,13 @@ handle_rails_package() {
local exclude_root="${railsdir#/}"
# .git and packages are for the SSO server, which is built from its
# repository root.
- for exclude in .git packages tmp log coverage \
- Capfile\* config/deploy\* \
- config/application.yml \
- config/database.yml; do
+ local -a exclude_list=(.git packages tmp log coverage Capfile\* \
+ config/deploy\* config/application.yml)
+ # for arvados-workbench, we need to have the (dummy) config/database.yml in the package
+ if [[ "$pkgname" != "arvados-workbench" ]]; then
+ exclude_list+=('config/database.yml')
+ fi
+ for exclude in ${exclude_list[@]}; do
switches+=(-x "$exclude_root/$exclude")
done
fpm_build "${pos_args[@]}" "${switches[@]}" \
commit 680bebbec151616f1582e6c1e85b7515907670f8
Merge: 43b16e4 487aeb0
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jan 22 16:40:26 2016 -0500
Merge branch '8066-git-init-from-postinst'
closes #8066
commit 487aeb0d3e2263edc777f3f5451fa05f5f071285
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jan 22 16:39:23 2016 -0500
Some review fixes.
closes #8066
diff --git a/jenkins/rails-package-scripts/README.md b/jenkins/rails-package-scripts/README.md
index 6779fb4..3a93c31 100644
--- a/jenkins/rails-package-scripts/README.md
+++ b/jenkins/rails-package-scripts/README.md
@@ -9,5 +9,6 @@ Since our build process is a tower of shell scripts, concatenating files seemed
postinst.sh lets the early parts define a few hooks to control behavior:
* After it installs the core configuration files (database.yml, application.yml, and production.rb) to /etc/arvados/server, it calls setup_extra_conffiles. By default this is a noop function (in step2.sh). API server defines this to set up the old omniauth.rb conffile.
+* Before it restarts nginx, it calls setup_before_nginx_restart. By default this is a noop function (in step2.sh). API server defines this to set up the internal git repository, if necessary.
* $RAILSPKG_DATABASE_LOAD_TASK defines the Rake task to load the database. API server uses db:structure:load. SSO server uses db:schema:load. Workbench doesn't set this, which causes the postinst to skip all database work.
* If $RAILSPKG_SUPPORTS_CONFIG_CHECK != 1, it won't run the config:check rake task. SSO clears this flag (it doesn't have that task code).
diff --git a/jenkins/rails-package-scripts/arvados-api-server.sh b/jenkins/rails-package-scripts/arvados-api-server.sh
index 4df87eb..c2b99f0 100644
--- a/jenkins/rails-package-scripts/arvados-api-server.sh
+++ b/jenkins/rails-package-scripts/arvados-api-server.sh
@@ -10,3 +10,23 @@ RAILSPKG_DATABASE_LOAD_TASK=db:structure:load
setup_extra_conffiles() {
setup_conffile initializers/omniauth.rb
}
+
+setup_before_nginx_restart() {
+ # initialize git_internal_dir
+ # usually /var/lib/arvados/internal.git (set in application.default.yml )
+ if [ "$APPLICATION_READY" = "1" ]; then
+ GIT_INTERNAL_DIR=$($COMMAND_PREFIX bundle exec rake config:check 2>&1 | grep git_internal_dir | awk '{ print $2 }')
+ if [ ! -e "$GIT_INTERNAL_DIR" ]; then
+ run_and_report "Creating git_internal_dir '$GIT_INTERNAL_DIR'" \
+ mkdir -p "$GIT_INTERNAL_DIR"
+ run_and_report "Initializing git_internal_dir '$GIT_INTERNAL_DIR'" \
+ git init --quiet --bare $GIT_INTERNAL_DIR
+ else
+ echo "Initializing git_internal_dir $GIT_INTERNAL_DIR: directory exists, skipped."
+ fi
+ run_and_report "Making sure '$GIT_INTERNAL_DIR' has the right permission" \
+ chown -R "$WWW_OWNER:" "$GIT_INTERNAL_DIR"
+ else
+ echo "Initializing git_internal_dir... skipped."
+ fi
+}
diff --git a/jenkins/rails-package-scripts/postinst.sh b/jenkins/rails-package-scripts/postinst.sh
index 69d6316..5ff2a9b 100644
--- a/jenkins/rails-package-scripts/postinst.sh
+++ b/jenkins/rails-package-scripts/postinst.sh
@@ -220,23 +220,6 @@ configure_version() {
$COMMAND_PREFIX bundle exec rake config:check || APPLICATION_READY=0
fi
- # initialize git_internal_dir
- # usually /var/lib/arvados/internal.git (set in application.default.yml )
- if [ "$APPLICATION_READY" = "1" ]; then
- GIT_INTERNAL_DIR=$($COMMAND_PREFIX bundle exec rake config:check 2>&1 | grep git_internal_dir | awk '{ print $2 }')
- if [ -e "$GIT_INTERNAL_DIR" ]
- then
- run_and_report "Creating and initializing git_internal_dir '$GIT_INTERNAL_DIR'" \
- mkdir -p "$GIT_INTERNAL_DIR" && \
- chown "$WWW_OWNER:" "$GIT_INTERNAL_DIR" && \
- su -c "git init --bare $GIT_INTERNAL_DIR" "$WWW_OWNER:"
- fi
- run_and_report "Making sure '$GIT_INTERNAL_DIR' has the right permission" \
- chown -R "$WWW_OWNER:" "$GIT_INTERNAL_DIR"
- else
- echo "Initializing git_internal_dir... skipped."
- fi
-
# precompile assets; thankfully this does not take long
if [ "$APPLICATION_READY" = "1" ]; then
run_and_report "Precompiling assets" \
@@ -247,6 +230,8 @@ configure_version() {
fi
chown -R "$WWW_OWNER:" $RELEASE_PATH/tmp
+ setup_before_nginx_restart
+
if [ ! -z "$WEB_SERVICE" ]; then
service "$WEB_SERVICE" restart
fi
diff --git a/jenkins/rails-package-scripts/step2.sh b/jenkins/rails-package-scripts/step2.sh
index 6678d49..816b906 100644
--- a/jenkins/rails-package-scripts/step2.sh
+++ b/jenkins/rails-package-scripts/step2.sh
@@ -23,3 +23,6 @@ RAILSPKG_SUPPORTS_CONFIG_CHECK=${RAILSPKG_SUPPORTS_CONFIG_CHECK:-1}
if ! type setup_extra_conffiles >/dev/null 2>&1; then
setup_extra_conffiles() { return; }
fi
+if ! type setup_before_nginx_restart >/dev/null 2>&1; then
+ setup_before_nginx_restart() { return; }
+fi
commit 18fb92535f505e30b86c50e1c58eaf93c0179412
Merge: 5cc977d 43b16e4
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jan 22 16:21:23 2016 -0500
Merge branch 'master' into 8066-git-init-from-postinst
commit 43b16e43f7f2558bc619f3611ee4b96fb74b5e05
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jan 22 10:27:59 2016 -0500
Make sure not to accidentally include config/application.yml and
config/database.yml in our rails packages.
No issue #
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index 3c62d7b..aed260e 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -158,7 +158,9 @@ handle_rails_package() {
# .git and packages are for the SSO server, which is built from its
# repository root.
for exclude in .git packages tmp log coverage \
- Capfile\* config/deploy\*; do
+ Capfile\* config/deploy\* \
+ config/application.yml \
+ config/database.yml; do
switches+=(-x "$exclude_root/$exclude")
done
fpm_build "${pos_args[@]}" "${switches[@]}" \
commit e9043f6d8de8ca8c928b0097e39b1695a26ffb2b
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Jan 22 10:26:14 2016 -0500
Bump libcloud package version pin to 0.20.2dev1 refs #8235
diff --git a/jenkins/libcloud-pin b/jenkins/libcloud-pin
index c399227..3fa07e6 100644
--- a/jenkins/libcloud-pin
+++ b/jenkins/libcloud-pin
@@ -1 +1 @@
-LIBCLOUD_PIN=0.18.1.dev4
\ No newline at end of file
+LIBCLOUD_PIN=0.20.2.dev1
\ No newline at end of file
commit 575d7f1fd77f66448df49c341a59baef6b3b5ef9
Merge: 7f05c28 a8c09e0
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jan 21 14:47:40 2016 -0500
Merge branch '8008-test-all-packages' closes #8008
commit a8c09e00f06c51887497b94b98178b2815e149e2
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jan 21 14:30:03 2016 -0500
8008: Tweak error code reporting. Remove debugging line.
diff --git a/jenkins/run-build-packages-all-targets.sh b/jenkins/run-build-packages-all-targets.sh
index 64c1262..f1a1e1c 100755
--- a/jenkins/run-build-packages-all-targets.sh
+++ b/jenkins/run-build-packages-all-targets.sh
@@ -84,11 +84,10 @@ cd $(dirname $0)
FINAL_EXITCODE=0
for dockerfile_path in $(find -name Dockerfile); do
- ./run-build-packages-one-target.sh --target "$(basename $(dirname "$dockerfile_path"))" --command "$COMMAND" $DEBUG $TEST_PACKAGES $ONLY_TEST
-
- CODE=$?
- if test $CODE != 0 ; then
- FINAL_EXITCODE=$CODE
+ if ./run-build-packages-one-target.sh --target "$(basename $(dirname "$dockerfile_path"))" --command "$COMMAND" $DEBUG $TEST_PACKAGES $ONLY_TEST ; then
+ true
+ else
+ FINAL_EXITCODE=$?
fi
done
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index ebfc0be..e800897 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -149,30 +149,27 @@ FINAL_EXITCODE=0
package_fails=""
-set +e
if [[ -n "$test_packages" ]]; then
for p in $packages ; do
- docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
+ if docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
--env ARVADOS_DEBUG=1 \
--env "TARGET=$TARGET" \
--env "WORKSPACE=/arvados" \
- "$IMAGE" $COMMAND $p
- CODE=$?
- if test $CODE != 0 ; then
- FINAL_EXITCODE=$CODE
+ "$IMAGE" $COMMAND $p ; then
+ true
+ else
+ FINAL_EXITCODE=$?
package_fails="$package_fails $p"
echo "ERROR: $tag test failed with exit status $FINAL_EXITCODE." >&2
fi
done
else
- docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
- --env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND
- CODE=$?
- if test $CODE = 0 ; then
+ if docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
+ --env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND ; then
echo
echo "Build packages for $TARGET succeeded." >&2
else
- FINAL_EXITCODE=$CODE
+ FINAL_EXITCODE=$?
echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
fi
fi
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 30ba4cd..445b3a0 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -285,7 +285,6 @@ handle_python_package
fpm_build $SRC_BUILD_DIR/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
rm -rf "$SRC_BUILD_DIR"
- echo $?
)
# Go binaries
commit 7512b31d3c7e3ae890a37b2222f113c9b351d86a
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jan 20 18:52:36 2016 -0500
8008: Quote more variables in package test scripts.
diff --git a/jenkins/package-testing/deb-common-test-packages.sh b/jenkins/package-testing/deb-common-test-packages.sh
index 0d15436..3928ba2 100755
--- a/jenkins/package-testing/deb-common-test-packages.sh
+++ b/jenkins/package-testing/deb-common-test-packages.sh
@@ -6,12 +6,12 @@ set -eu
# from the invocation path.
target=$(echo $0 | sed 's/.*test-packages-\([^.]*\)\.sh.*/\1/')
-export ARV_PACKAGES_DIR=/arvados/packages/$target
+export ARV_PACKAGES_DIR="/arvados/packages/$target"
dpkg-query --show > "$ARV_PACKAGES_DIR/$1.before"
apt-get -qq update
-apt-get --assume-yes --force-yes install $1
+apt-get --assume-yes --force-yes install "$1"
dpkg-query --show > "$ARV_PACKAGES_DIR/$1.after"
@@ -22,7 +22,7 @@ set -e
mkdir -p /tmp/opts
cd /tmp/opts
-export ARV_PACKAGES_DIR=/arvados/packages/$target
+export ARV_PACKAGES_DIR="/arvados/packages/$target"
dpkg-deb -x $ARV_PACKAGES_DIR/"$1"_*.deb .
@@ -34,4 +34,4 @@ done <<EOF
$(find -name '*.so')
EOF
-exec /jenkins/package-testing/common-test-packages.sh $1
+exec /jenkins/package-testing/common-test-packages.sh "$1"
diff --git a/jenkins/package-testing/test-package-arvados-api-server.sh b/jenkins/package-testing/test-package-arvados-api-server.sh
index 64494aa..e975448 100755
--- a/jenkins/package-testing/test-package-arvados-api-server.sh
+++ b/jenkins/package-testing/test-package-arvados-api-server.sh
@@ -17,4 +17,4 @@ case "$TARGET" in
;;
esac
-/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arados-api-server.gems
+/usr/local/rvm/bin/rvm-exec default bundle list >"$ARV_PACKAGES_DIR/arvados-api-server.gems"
diff --git a/jenkins/package-testing/test-package-arvados-sso-server.sh b/jenkins/package-testing/test-package-arvados-sso-server.sh
index 28c5618..c1a377e 100755
--- a/jenkins/package-testing/test-package-arvados-sso-server.sh
+++ b/jenkins/package-testing/test-package-arvados-sso-server.sh
@@ -104,7 +104,7 @@ if [[ "$FORMAT" == "deb" ]]; then
dpkg-reconfigure arvados-sso-server || EXITCODE=3
cd /var/www/arvados-sso/current/
- /usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arvados-sso-server.gems
+ /usr/local/rvm/bin/rvm-exec default bundle list >"$ARV_PACKAGES_DIR/arvados-sso-server.gems"
# Test 3: the package should remove cleanly
apt-get remove arvados-sso-server --yes || EXITCODE=3
diff --git a/jenkins/package-testing/test-package-arvados-workbench.sh b/jenkins/package-testing/test-package-arvados-workbench.sh
index e255786..1be4dea 100755
--- a/jenkins/package-testing/test-package-arvados-workbench.sh
+++ b/jenkins/package-testing/test-package-arvados-workbench.sh
@@ -17,4 +17,4 @@ case "$TARGET" in
;;
esac
-/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arvados-workbench.gems
+/usr/local/rvm/bin/rvm-exec default bundle list >"$ARV_PACKAGES_DIR/arvados-workbench.gems"
diff --git a/jenkins/package-testing/test-packages-centos6.sh b/jenkins/package-testing/test-packages-centos6.sh
index ad80564..1960944 100755
--- a/jenkins/package-testing/test-packages-centos6.sh
+++ b/jenkins/package-testing/test-packages-centos6.sh
@@ -7,14 +7,14 @@ touch /var/lib/rpm/*
export ARV_PACKAGES_DIR=/arvados/packages/centos6
-rpm -qa | sort > $ARV_PACKAGES_DIR/$1.before
+rpm -qa | sort > "$ARV_PACKAGES_DIR/$1.before"
yum install --assumeyes $1
-rpm -qa | sort > $ARV_PACKAGES_DIR/$1.after
+rpm -qa | sort > "$ARV_PACKAGES_DIR/$1.after"
set +e
-diff $ARV_PACKAGES_DIR/$1.before $ARV_PACKAGES_DIR/$1.after > $ARV_PACKAGES_DIR/$1.diff
+diff "$ARV_PACKAGES_DIR/$1.before" "$ARV_PACKAGES_DIR/$1.after" >"$ARV_PACKAGES_DIR/$1.diff"
set -e
SCL=""
@@ -25,7 +25,7 @@ fi
mkdir -p /tmp/opts
cd /tmp/opts
-rpm2cpio $ARV_PACKAGES_DIR/$1-*.rpm | cpio -idm 2>/dev/null
+rpm2cpio "$ARV_PACKAGES_DIR/$1"-*.rpm | cpio -idm 2>/dev/null
shared=$(find -name '*.so')
if test -n "$shared" ; then
@@ -38,7 +38,7 @@ if test -n "$shared" ; then
fi
if test -n "$SCL" ; then
- exec $SCL "/jenkins/package-testing/common-test-packages.sh $1"
+ exec $SCL "/jenkins/package-testing/common-test-packages.sh '$1'"
else
- exec /jenkins/package-testing/common-test-packages.sh $1
+ exec /jenkins/package-testing/common-test-packages.sh "$1"
fi
commit 7f05c28c4ac0ddffdf9e13bfc104bc60f7195174
Merge: 7b2fd8f 06b0b75
Author: radhika <radhika at curoverse.com>
Date: Wed Jan 20 12:44:02 2016 -0500
refs #8028
Merge branch '8028-crunch-dispatch-local'
commit 06b0b75584e41e90aa635b6d833936343075c132
Merge: c9e5a80 7b2fd8f
Author: radhika <radhika at curoverse.com>
Date: Wed Jan 20 12:43:32 2016 -0500
Merge branch 'master' into 8028-crunch-dispatch-local
Conflicts:
jenkins/run-tests.sh
diff --cc jenkins/run-tests.sh
index 152319c,40cf907..1289095
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@@ -721,7 -720,7 +721,8 @@@ gostuff=
services/datamanager/collection
services/datamanager/keep
services/datamanager
+ services/crunch-dispatch-local
+ services/crunch-run
tools/keep-rsync
)
for g in "${gostuff[@]}"
commit 7b2fd8f0c85a013df886e881d5d97aebf4890169
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 20 12:10:26 2016 -0500
Document some arvbox hw/sw requirements refs #8080
diff --git a/arvbox/README.md b/arvbox/README.md
index b4a9c25..cdb3914 100644
--- a/arvbox/README.md
+++ b/arvbox/README.md
@@ -26,6 +26,11 @@ log <service> tail log of specified service
svrestart <service> restart specified service inside arvbox
clone <from> <to> clone an arvbox
+## Requirements
+
+Linux 3.x+ and Docker 1.9+
+Minimum of 3 GiB of RAM + additional memory to run jobs
+Minimum of 3 GiB of disk + storage for actual data
## Configs
commit 026d70487631662d1b0ccf8f66fbef71b9be40d6
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 20 11:43:10 2016 -0500
Enable arvbox user to sudo to crunch user. refs #8080
diff --git a/arvbox/lib/arvbox/docker/createusers.sh b/arvbox/lib/arvbox/docker/createusers.sh
index cc7e11f..1e6c95d 100755
--- a/arvbox/lib/arvbox/docker/createusers.sh
+++ b/arvbox/lib/arvbox/docker/createusers.sh
@@ -37,4 +37,6 @@ if ! grep "^arvbox:" /etc/passwd >/dev/null 2>/dev/null ; then
mkdir -p /tmp/crunch0 /tmp/crunch1
chown crunch:crunch -R /tmp/crunch0 /tmp/crunch1
+
+ echo "arvbox ALL=(crunch) NOPASSWD: ALL" >> /etc/sudoers
fi
commit 22e538ac7ece05cbfc32b2dfb463f3fb955d7f61
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 20 11:40:09 2016 -0500
Add crunch-run to run-tests refs #7816
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index bdf541a..40cf907 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -720,6 +720,7 @@ gostuff=(
services/datamanager/collection
services/datamanager/keep
services/datamanager
+ services/crunch-run
tools/keep-rsync
)
for g in "${gostuff[@]}"
commit fe5164555b117d5c9c538905c495192adad0b815
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 20 10:38:43 2016 -0500
Fix gitolite-shell path refs #8080
diff --git a/arvbox/lib/arvbox/docker/service/arv-git-httpd/run-service b/arvbox/lib/arvbox/docker/service/arv-git-httpd/run-service
index 7bd6a7c..854464e 100755
--- a/arvbox/lib/arvbox/docker/service/arv-git-httpd/run-service
+++ b/arvbox/lib/arvbox/docker/service/arv-git-httpd/run-service
@@ -23,5 +23,5 @@ cd ~git
exec /usr/local/bin/arv-git-httpd \
-address=:${services[arv-git-httpd]} \
- -git-command=/var/lib/arvados/git/gitolite/src/gitolite-shell \
+ -git-command=/usr/share/gitolite3/gitolite-shell \
-repo-root=/var/lib/arvados/git/repositories
commit 5cc977da78b718596faa9ad0352b7c14402f90fa
Author: Nico Cesar <nico at curoverse.com>
Date: Tue Jan 19 15:17:22 2016 -0500
added git_internal_dir init in postinst.sh
diff --git a/jenkins/rails-package-scripts/postinst.sh b/jenkins/rails-package-scripts/postinst.sh
index 1032d14..dfc58c5 100644
--- a/jenkins/rails-package-scripts/postinst.sh
+++ b/jenkins/rails-package-scripts/postinst.sh
@@ -220,6 +220,23 @@ configure_version() {
$COMMAND_PREFIX bundle exec rake config:check || APPLICATION_READY=0
fi
+ # initialize git_internal_dir
+ # usually /var/lib/arvados/internal.git (set in application.default.yml )
+ if [ "$APPLICATION_READY" = "1" ]; then
+ GIT_INTERNAL_DIR=$($COMMAND_PREFIX bundle exec rake config:check 2>&1 | grep git_internal_dir | awk '{ print $2 }')
+ if [ -e "$GIT_INTERNAL_DIR" ]
+ then
+ run_and_report "Creating and initializing git_internal_dir '$GIT_INTERNAL_DIR'" \
+ mkdir -p "$GIT_INTERNAL_DIR" && \
+ chown "$WWW_OWNER:" "$GIT_INTERNAL_DIR" && \
+ su -c "git init --bare $GIT_INTERNAL_DIR" "$WWW_OWNER:"
+ fi
+ run_and_report "Making sure '$GIT_INTERNAL_DIR' has the right permission" \
+ chown -R "$WWW_OWNER:" "$GIT_INTERNAL_DIR"
+ else
+ echo "Initializing git_internal_dir... skipped."
+ fi
+
# precompile assets; thankfully this does not take long
if [ "$APPLICATION_READY" = "1" ]; then
run_and_report "Precompiling assets" \
commit b408ce71a9caddea144fd7d90aa166c9cb16e715
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Tue Jan 19 13:09:31 2016 -0500
Merge branch '8080-arvbox' closes #8080
diff --git a/arvbox/README.md b/arvbox/README.md
new file mode 100644
index 0000000..b4a9c25
--- /dev/null
+++ b/arvbox/README.md
@@ -0,0 +1,107 @@
+# Arvados-in-a-box
+
+Self-contained development, demonstration and testing environment for Arvados.
+
+## Quick start
+
+$ bin/arvbox reboot localdemo
+
+## Usage
+
+Arvados-in-a-box
+
+arvbox (build|start|run|open|shell|ip|stop|reboot|reset|destroy|log|svrestart)
+
+build <config> build arvbox Docker image
+start|run <config> start arvbox container
+open open arvbox workbench in a web browser
+shell enter arvbox shell
+ip print arvbox ip address
+stop stop arvbox container
+restart <config> stop, then run again
+reboot <config> stop, build arvbox Docker image, run
+reset delete arvbox arvados data (be careful!)
+destroy delete all arvbox code and data (be careful!)
+log <service> tail log of specified service
+svrestart <service> restart specified service inside arvbox
+clone <from> <to> clone an arvbox
+
+
+## Configs
+
+### dev
+Development configuration. Boots a complete Arvados environment inside the
+container. The "arvados", "arvado-dev" and "sso-devise-omniauth-provider" code
+directories along data directories "postgres", "var", "passenger" and "gems"
+are bind mounted from the host file system for easy access and persistence
+across container rebuilds. Services are bound to the Docker container's
+network IP address and can only be accessed on the local host.
+
+### localdemo
+Demo configuration. Boots a complete Arvados environment inside the container.
+Unlike the development configuration, code directories are included in the demo
+image, and data directories are stored in a separate data volume container.
+Services are bound to the Docker container's network IP address and can only be
+accessed on the local host.
+
+### test
+Run the test suite.
+
+### publicdev
+Publicly accessible development configuration. Similar to 'dev' except that
+service ports are published to the host's IP address and can accessed by anyone
+who can connect to the host system. WARNING! The public arvbox configuration
+is NOT SECURE and must not be placed on a public IP address or used for
+production work.
+
+### publicdemo
+Publicly accessible development configuration. Similar to 'localdemo' except
+that service ports are published to the host's IP address and can accessed by
+anyone who can connect to the host system. WARNING! The public arvbox configuration
+is NOT SECURE and must not be placed on a public IP address or used for
+production work.
+
+## Environment variables
+
+### ARVBOX_DOCKER
+The location of Dockerfile.base and associated files used by "arvbox build".
+default: result of $(readlink -f $(dirname $0)/../lib/arvbox/docker)
+
+### ARVBOX_CONTAINER
+The name of the Docker container to manipulate.
+default: arvbox
+
+### ARVBOX_BASE
+The base directory to store persistent data for arvbox containers.
+default: $HOME/.arvbox
+
+### ARVBOX_DATA
+The base directory to store persistent data for the current container.
+default: $ARVBOX_BASE/$ARVBOX_CONTAINER
+
+### ARVADOS_ROOT
+The root directory of the Arvados source tree
+default: $ARVBOX_DATA/arvados
+
+### ARVADOS_DEV_ROOT
+The root directory of the Arvados-dev source tree
+default: $ARVBOX_DATA/arvados-dev
+
+### SSO_ROOT
+The root directory of the SSO source tree
+default: $ARVBOX_DATA/sso-devise-omniauth-provider
+
+### ARVBOX_PUBLISH_IP
+The IP address on which to publish services when running in public
+configuration. Overrides default detection of the host's IP address.
+
+## Notes
+
+Services are designed to install and auto-configure on start or restart. For
+example, the service script for keepstore always compiles keepstore from source
+and registers the daemon with the API server.
+
+Services are run with process supervision, so a service which exits will be
+restarted. Dependencies between services are handled by repeatedly trying and
+failing the service script until dependencies are fulfilled (by other service
+scripts) enabling the service script to complete.
diff --git a/arvbox/bin/arvbox b/arvbox/bin/arvbox
new file mode 100755
index 0000000..284f0ef
--- /dev/null
+++ b/arvbox/bin/arvbox
@@ -0,0 +1,377 @@
+#!/bin/sh
+
+set -e
+
+if ! test -d /sys/fs/cgroup ; then
+ echo "Arvbox requires cgroups to be mounted at /sys/fs/cgroup in order to use"
+ echo "Docker-in-Docker. Older operating systems that put cgroups in other"
+ echo "places (such as /cgroup) are not supported."
+ exit 1
+fi
+
+if ! which docker >/dev/null 2>/dev/null ; then
+ echo "Arvbox requires Docker. To install, run the following command as root:"
+ echo "curl -sSL https://get.docker.com/ | sh"
+ exit 1
+fi
+
+if test -z "$ARVBOX_DOCKER" ; then
+ if which greadlink >/dev/null 2>/dev/null ; then
+ ARVBOX_DOCKER=$(greadlink -f $(dirname $0)/../lib/arvbox/docker)
+ else
+ ARVBOX_DOCKER=$(readlink -f $(dirname $0)/../lib/arvbox/docker)
+ fi
+fi
+
+if test -z "$ARVBOX_CONTAINER" ; then
+ ARVBOX_CONTAINER=arvbox
+fi
+
+if test -z "$ARVBOX_BASE" ; then
+ ARVBOX_BASE="$HOME/.arvbox"
+fi
+
+if test -z "$ARVBOX_DATA" ; then
+ ARVBOX_DATA="$ARVBOX_BASE/$ARVBOX_CONTAINER"
+fi
+
+if test -z "$ARVADOS_ROOT" ; then
+ ARVADOS_ROOT="$ARVBOX_DATA/arvados"
+fi
+
+if test -z "$ARVADOS_DEV_ROOT" ; then
+ ARVADOS_DEV_ROOT="$ARVBOX_DATA/arvados-dev"
+fi
+
+if test -z "$SSO_ROOT" ; then
+ SSO_ROOT="$ARVBOX_DATA/sso-devise-omniauth-provider"
+fi
+
+PG_DATA="$ARVBOX_DATA/postgres"
+VAR_DATA="$ARVBOX_DATA/var"
+PASSENGER="$ARVBOX_DATA/passenger"
+GEMS="$ARVBOX_DATA/gems"
+
+getip() {
+ docker inspect $ARVBOX_CONTAINER | grep \"IPAddress\" | head -n1 | tr -d ' ":,\n' | cut -c10-
+}
+
+updateconf() {
+ if test -f ~/.config/arvados/$ARVBOX_CONTAINER.conf ; then
+ sed "s/ARVADOS_API_HOST=.*/ARVADOS_API_HOST=$(getip):8000/" <$HOME/.config/arvados/$ARVBOX_CONTAINER.conf >$HOME/.config/arvados/$ARVBOX_CONTAINER.conf.tmp
+ mv ~/.config/arvados/$ARVBOX_CONTAINER.conf.tmp ~/.config/arvados/$ARVBOX_CONTAINER.conf
+ else
+ mkdir -p $HOME/.config/arvados
+ cat >$HOME/.config/arvados/$ARVBOX_CONTAINER.conf <<EOF
+ARVADOS_API_HOST=$(getip):8000
+ARVADOS_API_TOKEN=
+ARVADOS_API_HOST_INSECURE=true
+EOF
+ fi
+}
+
+wait_for_arvbox() {
+ FF=/tmp/arvbox-fifo-$$
+ mkfifo $FF
+ docker logs -f $ARVBOX_CONTAINER > $FF &
+ LOGPID=$!
+ while read line ; do
+ echo $line
+ if echo $line | grep "Workbench is running at" >/dev/null ; then
+ kill $LOGPID
+ fi
+ done < $FF
+ rm $FF
+ echo
+ if test -n "$localip" ; then
+ echo "export ARVADOS_API_HOST=$localip:8000"
+ else
+ echo "export ARVADOS_API_HOST=$(getip):8000"
+ fi
+}
+
+run() {
+ if docker ps -a | grep -E "$ARVBOX_CONTAINER$" -q ; then
+ echo "Container $ARVBOX_CONTAINER is already running, use stop, restart or reboot"
+ exit 0
+ fi
+
+ if echo "$1" | grep '^public' ; then
+ if test -n "$ARVBOX_PUBLISH_IP" ; then
+ localip=$ARVBOX_PUBLISH_IP
+ else
+ defaultdev=$(/sbin/ip route|awk '/default/ { print $5 }')
+ localip=$(ip addr show $defaultdev | grep 'inet ' | sed 's/ *inet \(.*\)\/.*/\1/')
+ fi
+ iptemp=$(tempfile)
+ echo $localip > $iptemp
+ chmod og+r $iptemp
+ PUBLIC="--volume=$iptemp:/var/run/localip_override
+ --publish=80:80
+ --publish=8000:8000
+ --publish=8900:8900
+ --publish=9001:9001
+ --publish=9002:9002
+ --publish=25100:25100
+ --publish=25107:25107
+ --publish=25108:25108
+ --publish=8001:8001"
+ else
+ PUBLIC=""
+ fi
+
+ if echo "$1" | grep 'demo$' ; then
+ if test -d "$ARVBOX_DATA" ; then
+ echo "It looks like you already have a development container named $ARVBOX_CONTAINER."
+ echo "Set ARVBOX_CONTAINER to set a different name for your demo container"
+ exit 1
+ fi
+
+ if ! (docker ps -a | grep -E "$ARVBOX_CONTAINER-data$" -q) ; then
+ docker create -v /var/lib/postgresql -v /var/lib/arvados --name $ARVBOX_CONTAINER-data arvados/arvbox-demo /bin/true
+ fi
+
+ docker run \
+ --detach \
+ --name=$ARVBOX_CONTAINER \
+ --privileged \
+ --volumes-from $ARVBOX_CONTAINER-data \
+ $PUBLIC \
+ arvados/arvbox-demo
+ updateconf
+ wait_for_arvbox
+ else
+ mkdir -p "$PG_DATA" "$VAR_DATA" "$PASSENGER" "$GEMS"
+
+ if ! test -d "$ARVADOS_ROOT" ; then
+ git clone https://github.com/curoverse/arvados.git "$ARVADOS_ROOT"
+ fi
+ if ! test -d "$SSO_ROOT" ; then
+ git clone https://github.com/curoverse/sso-devise-omniauth-provider.git "$SSO_ROOT"
+ fi
+
+ if test "$1" = test ; then
+ shift
+
+ if ! test -d "$ARVADOS_DEV_ROOT" ; then
+ git clone https://github.com/curoverse/arvados-dev.git "$ARVADOS_DEV_ROOT"
+ fi
+
+ mkdir -p $VAR_DATA/test
+
+ docker run \
+ --detach \
+ --name=$ARVBOX_CONTAINER \
+ --privileged \
+ "--volume=$ARVADOS_ROOT:/usr/src/arvados:rw" \
+ "--volume=$ARVADOS_DEV_ROOT:/usr/src/arvados-dev:rw" \
+ "--volume=$SSO_ROOT:/usr/src/sso:rw" \
+ "--volume=$PG_DATA:/var/lib/postgresql:rw" \
+ "--volume=$VAR_DATA:/var/lib/arvados:rw" \
+ "--volume=$PASSENGER:/var/lib/passenger:rw" \
+ "--volume=$GEMS:/var/lib/gems:rw" \
+ arvados/arvbox-dev \
+ /usr/local/bin/runsvinit -svdir=/etc/test-service
+
+ docker exec -ti \
+ $ARVBOX_CONTAINER \
+ /usr/local/lib/arvbox/runsu.sh \
+ /usr/local/lib/arvbox/waitforpostgres.sh
+
+ docker exec -ti \
+ $ARVBOX_CONTAINER \
+ /usr/local/lib/arvbox/runsu.sh \
+ /usr/local/lib/arvbox/service/sso/run-service --only-setup
+
+ docker exec -ti \
+ $ARVBOX_CONTAINER \
+ /usr/local/lib/arvbox/runsu.sh \
+ /usr/local/lib/arvbox/service/api/run-service --only-setup
+
+ docker exec -ti \
+ $ARVBOX_CONTAINER \
+ /usr/local/lib/arvbox/runsu.sh \
+ /usr/src/arvados-dev/jenkins/run-tests.sh \
+ --temp /var/lib/arvados/test \
+ WORKSPACE=/usr/src/arvados \
+ GEM_HOME=/var/lib/gems \
+ "$@"
+ else
+ docker run \
+ --detach \
+ --name=$ARVBOX_CONTAINER \
+ --privileged \
+ "--volume=$ARVADOS_ROOT:/usr/src/arvados:rw" \
+ "--volume=$SSO_ROOT:/usr/src/sso:rw" \
+ "--volume=$PG_DATA:/var/lib/postgresql:rw" \
+ "--volume=$VAR_DATA:/var/lib/arvados:rw" \
+ "--volume=$PASSENGER:/var/lib/passenger:rw" \
+ "--volume=$GEMS:/var/lib/gems:rw" \
+ $PUBLIC \
+ arvados/arvbox-dev
+ updateconf
+ wait_for_arvbox
+ echo "The Arvados source code is checked out at: $ARVADOS_ROOT"
+ fi
+ fi
+}
+
+stop() {
+ if docker ps -a --filter "status=running" | grep -E "$ARVBOX_CONTAINER$" -q ; then
+ docker stop $ARVBOX_CONTAINER
+ fi
+
+ VOLUMES=--volumes=true
+ if docker ps -a --filter "status=created" | grep -E "$ARVBOX_CONTAINER$" -q ; then
+ docker rm $VOLUMES $ARVBOX_CONTAINER
+ fi
+ if docker ps -a --filter "status=exited" | grep -E "$ARVBOX_CONTAINER$" -q ; then
+ docker rm $VOLUMES $ARVBOX_CONTAINER
+ fi
+}
+
+build() {
+ if ! test -f "$ARVBOX_DOCKER/Dockerfile.base" ; then
+ echo "Could not find Dockerfile ($ARVBOX_DOCKER/Dockerfile.base)"
+ exit 1
+ fi
+ docker build -t arvados/arvbox-base -f "$ARVBOX_DOCKER/Dockerfile.base" "$ARVBOX_DOCKER"
+ if test "$1" = localdemo -o "$1" = publicdemo ; then
+ docker build -t arvados/arvbox-demo -f "$ARVBOX_DOCKER/Dockerfile.demo" "$ARVBOX_DOCKER"
+ else
+ docker build -t arvados/arvbox-dev -f "$ARVBOX_DOCKER/Dockerfile.dev" "$ARVBOX_DOCKER"
+ fi
+}
+
+check() {
+ case "$1" in
+ localdemo|publicdemo|dev|publicdev|test)
+ true
+ ;;
+ *)
+ echo "Argument to $subcmd must be one of localdemo, publicdemo, dev, publicdev, test"
+ exit 1
+ ;;
+ esac
+}
+
+subcmd="$1"
+if test -n "$subcmd" ; then
+ shift
+fi
+case "$subcmd" in
+ build)
+ check $@
+ build $@
+ ;;
+
+ start|run)
+ check $@
+ run $@
+ ;;
+
+ sh*)
+ docker exec -ti $ARVBOX_CONTAINER /usr/bin/env TERM=$TERM GEM_HOME=/var/lib/gems /bin/bash
+ ;;
+
+ stop)
+ stop
+ ;;
+
+ restart)
+ check $@
+ stop
+ run $@
+ ;;
+
+ reboot)
+ check $@
+ stop
+ build $@
+ run $@
+ ;;
+
+ ip|open)
+ if test "$subcmd" = 'ip' ; then
+ echo $(getip)
+ else
+ xdg-open http://$(getip)
+ fi
+ ;;
+
+ reset|destroy)
+ stop
+ if test -d "$ARVBOX_DATA" ; then
+ if test "$subcmd" = destroy ; then
+ if test "$1" != -f ; then
+ echo "WARNING! This will delete your entire arvbox ($ARVBOX_DATA)."
+ echo "Use destroy -f if you really mean it."
+ exit 1
+ fi
+ rm -rf "$ARVBOX_DATA"
+ else
+ if test "$1" != -f ; then
+ echo "WARNING! This will delete your arvbox data ($ARVBOX_DATA)."
+ echo "Code and downloaded packages will be preserved."
+ echo "Use reset -f if you really mean it."
+ exit 1
+ fi
+ rm -rf "$ARVBOX_DATA/postgres"
+ rm -rf "$ARVBOX_DATA/var"
+ fi
+ else
+ if test "$1" != -f ; then
+ echo "WARNING! This will delete your data container $ARVBOX_CONTAINER-data. Use -f if you really mean it."
+ exit 1
+ fi
+ docker rm "$ARVBOX_CONTAINER-data"
+ fi
+ ;;
+
+ log|svrestart)
+ if test -n "$1" ; then
+ if test "$subcmd" = log ; then
+ docker exec -ti $ARVBOX_CONTAINER /usr/bin/env TERM=$TERM less --follow-name +GF "/etc/service/$1/log/main/current"
+ fi
+ if test "$subcmd" = svrestart ; then
+ docker exec -ti $ARVBOX_CONTAINER sv restart "$1"
+ docker exec -ti $ARVBOX_CONTAINER sv restart ready
+ fi
+ else
+ echo "Usage: $0 $subcmd <service>"
+ echo "Available services:"
+ docker exec -ti $ARVBOX_CONTAINER ls /etc/service
+ fi
+ ;;
+
+ clone)
+ if test -n "$2" ; then
+ cp -r "$ARVBOX_BASE/$1" "$ARVBOX_BASE/$2"
+ echo "Created new arvbox $2"
+ echo "export ARVBOX_CONTAINER=$2"
+ else
+ echo "clone <from> <to> clone an arvbox"
+ echo "available arvboxes: $(ls $ARVBOX_BASE)"
+ fi
+ ;;
+
+ *)
+ echo "Arvados-in-a-box"
+ echo
+ echo "$(basename $0) (build|start|run|open|shell|ip|stop|reboot|reset|destroy|log|svrestart)"
+ echo
+ echo "build <config> build arvbox Docker image"
+ echo "start|run <config> start $ARVBOX_CONTAINER container"
+ echo "open open arvbox workbench in a web browser"
+ echo "shell enter arvbox shell"
+ echo "ip print arvbox ip address"
+ echo "stop stop arvbox container"
+ echo "restart <config> stop, then run again"
+ echo "reboot <config> stop, build arvbox Docker image, run"
+ echo "reset delete arvbox arvados data (be careful!)"
+ echo "destroy delete all arvbox code and data (be careful!)"
+ echo "log <service> tail log of specified service"
+ echo "svrestart <service> restart specified service inside arvbox"
+ echo "clone <from> <to> clone an arvbox"
+ ;;
+esac
diff --git a/arvbox/lib/arvbox/docker/Dockerfile.base b/arvbox/lib/arvbox/docker/Dockerfile.base
new file mode 100644
index 0000000..cdbface
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/Dockerfile.base
@@ -0,0 +1,33 @@
+FROM debian:8
+
+RUN apt-get update && \
+ DEBIAN_FRONTEND=noninteractive apt-get -yq install \
+ postgresql-9.4 git gcc golang-go runit \
+ ruby rake bundler curl libpq-dev \
+ libcurl4-openssl-dev libssl-dev zlib1g-dev libpcre3-dev \
+ openssh-server python-setuptools netcat-traditional \
+ libpython-dev fuse libfuse-dev python-pip \
+ pkg-config libattr1-dev python-llfuse python-pycurl \
+ libwww-perl libio-socket-ssl-perl libcrypt-ssleay-perl \
+ libjson-perl nginx gitolite3 lsof python-epydoc graphviz
+
+RUN curl -sSL https://get.docker.com/ | sh
+VOLUME /var/lib/docker
+
+RUN rm -rf /var/lib/postgresql && mkdir -p /var/lib/postgresql
+
+RUN cd /root && \
+ GOPATH=$PWD go get github.com/curoverse/runsvinit && \
+ install bin/runsvinit /usr/local/bin
+
+ADD fuse.conf /etc/
+
+ADD crunch-setup.sh gitolite.rc \
+ keep-setup.sh common.sh createusers.sh \
+ logger runsu.sh waitforpostgres.sh \
+ /usr/local/lib/arvbox/
+ADD service/ /usr/local/lib/arvbox/service
+RUN rmdir /etc/service && ln -sf /usr/local/lib/arvbox/service /etc
+
+# Start the supervisor.
+CMD ["/usr/local/bin/runsvinit"]
diff --git a/arvbox/lib/arvbox/docker/Dockerfile.demo b/arvbox/lib/arvbox/docker/Dockerfile.demo
new file mode 100644
index 0000000..1085abc
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/Dockerfile.demo
@@ -0,0 +1,14 @@
+FROM arvados/arvbox-base
+
+RUN cd /usr/src && \
+ git clone https://github.com/curoverse/arvados.git && \
+ git clone https://github.com/curoverse/sso-devise-omniauth-provider.git sso
+
+RUN chown -R 1000:1000 /usr/src && /usr/local/lib/arvbox/createusers.sh
+
+RUN sudo -u arvbox /usr/local/lib/arvbox/service/sso/run-service --only-deps
+RUN sudo -u arvbox /usr/local/lib/arvbox/service/api/run-service --only-deps
+RUN sudo -u arvbox /usr/local/lib/arvbox/service/workbench/run-service --only-deps
+RUN sudo -u arvbox /usr/local/lib/arvbox/service/doc/run-service --only-deps
+RUN sudo -u arvbox /usr/local/lib/arvbox/service/vm/run-service --only-deps
+RUN sudo -u arvbox /usr/local/lib/arvbox/service/sdk/run-service
diff --git a/arvbox/lib/arvbox/docker/Dockerfile.dev b/arvbox/lib/arvbox/docker/Dockerfile.dev
new file mode 100644
index 0000000..408e2de
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/Dockerfile.dev
@@ -0,0 +1,13 @@
+FROM arvados/arvbox-base
+
+RUN apt-get update && \
+ DEBIAN_FRONTEND=noninteractive apt-get -yq install \
+ python-virtualenv python3-virtualenv linkchecker xvfb iceweasel
+
+RUN set -e && \
+ PJS=phantomjs-1.9.7-linux-x86_64 && \
+ curl -L -o/tmp/$PJS.tar.bz2 https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2 && \
+ tar -C /usr/local -xjf /tmp/$PJS.tar.bz2 && \
+ ln -s ../$PJS/bin/phantomjs /usr/local/bin/
+
+RUN mkdir /etc/test-service && ln -sf /usr/local/lib/arvbox/service/postgres /etc/test-service
diff --git a/arvbox/lib/arvbox/docker/common.sh b/arvbox/lib/arvbox/docker/common.sh
new file mode 100644
index 0000000..4c2de47
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/common.sh
@@ -0,0 +1,62 @@
+
+if test -s /var/run/localip_override ; then
+ localip=$(cat /var/run/localip_override)
+else
+ defaultdev=$(/sbin/ip route|awk '/default/ { print $5 }')
+ localip=$(ip addr show $defaultdev | grep 'inet ' | sed 's/ *inet \(.*\)\/.*/\1/')
+fi
+
+export GEM_HOME=/var/lib/gems
+export GEM_PATH=/var/lib/gems
+
+declare -A services
+services=(
+ [workbench]=80
+ [api]=8000
+ [sso]=8900
+ [arv-git-httpd]=9001
+ [keep-web]=9002
+ [keepproxy]=25100
+ [keepstore0]=25107
+ [keepstore1]=25108
+ [ssh]=22
+ [doc]=8001
+)
+
+if test "$(id arvbox -u 2>/dev/null)" = 0 ; then
+ PGUSER=postgres
+ PGGROUP=postgres
+else
+ PGUSER=arvbox
+ PGGROUP=arvbox
+fi
+
+run_bundler() {
+ if test -f Gemfile.lock ; then
+ frozen=--frozen
+ else
+ frozen=""
+ fi
+ if ! flock /var/lib/arvados/gems.lock bundle install --path $GEM_HOME --local --no-deployment $frozen "$@" ; then
+ flock /var/lib/arvados/gems.lock bundle install --path $GEM_HOME --no-deployment $frozen "$@"
+ fi
+}
+
+pip_install() {
+ pushd /var/lib/arvados/pip
+ for p in $(ls http*.tar.gz) ; do
+ if test -f $p ; then
+ ln -sf $p $(echo $p | sed 's/.*%2F\(.*\)/\1/')
+ fi
+ done
+ for p in $(ls http*.whl) ; do
+ if test -f $p ; then
+ ln -sf $p $(echo $p | sed 's/.*%2F\(.*\)/\1/')
+ fi
+ done
+ popd
+
+ if ! pip install --no-index --find-links /var/lib/arvados/pip $1 ; then
+ pip install $1
+ fi
+}
diff --git a/arvbox/lib/arvbox/docker/createusers.sh b/arvbox/lib/arvbox/docker/createusers.sh
new file mode 100755
index 0000000..cc7e11f
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/createusers.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+
+set -e -o pipefail
+
+if ! grep "^arvbox:" /etc/passwd >/dev/null 2>/dev/null ; then
+ HOSTUID=$(ls -nd /usr/src/arvados | sed 's/ */ /' | cut -d' ' -f4)
+ HOSTGID=$(ls -nd /usr/src/arvados | sed 's/ */ /' | cut -d' ' -f5)
+ FUSEGID=$(ls -nd /dev/fuse | sed 's/ */ /' | cut -d' ' -f5)
+
+ mkdir -p /var/lib/arvados/git /var/lib/gems /var/lib/passenger
+
+ groupadd --gid $HOSTGID --non-unique arvbox
+ groupadd --gid $FUSEGID --non-unique fuse
+ groupadd --gid $HOSTGID --non-unique git
+ useradd --home-dir /var/lib/arvados \
+ --uid $HOSTUID --gid $HOSTGID \
+ --non-unique \
+ --groups docker,fuse \
+ arvbox
+ useradd --home-dir /var/lib/arvados/git --uid $HOSTUID --gid $HOSTGID --non-unique git
+ useradd --groups docker,fuse crunch
+
+ chown arvbox:arvbox -R /usr/local /var/lib/arvados /var/lib/gems /var/lib/passenger /var/lib/postgresql
+
+ mkdir -p /var/lib/gems/ruby/2.1.0
+ chown arvbox:arvbox -R /var/lib/gems/ruby/2.1.0
+
+ chown arvbox:arvbox -R /var/lib/nginx
+
+ # There's something weird about /var/log/nginx that prevents a non-root
+ # arvbox user from writing to it, even after the ownership has been
+ # changed. As a workaround, delete it and recreate it.
+
+ rm -r /var/log/nginx
+ mkdir -p /var/log/nginx
+ chown arvbox:arvbox -R /var/log/nginx
+
+ mkdir -p /tmp/crunch0 /tmp/crunch1
+ chown crunch:crunch -R /tmp/crunch0 /tmp/crunch1
+fi
diff --git a/arvbox/lib/arvbox/docker/crunch-setup.sh b/arvbox/lib/arvbox/docker/crunch-setup.sh
new file mode 100755
index 0000000..178fec1
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/crunch-setup.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+
+exec 2>&1
+set -eux -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+mkdir -p /var/lib/arvados/gostuff
+cd /var/lib/arvados/gostuff
+
+export GOPATH=$PWD
+mkdir -p "$GOPATH/src/git.curoverse.com"
+ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/crunchstat"
+install bin/crunchstat /usr/local/bin
+
+export ARVADOS_API_HOST=$localip:${services[api]}
+export ARVADOS_API_HOST_INSECURE=1
+export ARVADOS_API_TOKEN=$(cat /usr/src/arvados/services/api/superuser_token)
+export CRUNCH_JOB_BIN=/usr/src/arvados/sdk/cli/bin/crunch-job
+export PERLLIB=/usr/src/arvados/sdk/perl/lib
+export CRUNCH_TMP=/tmp/$1
+export CRUNCH_DISPATCH_LOCKFILE=/var/lock/$1-dispatch
+export CRUNCH_JOB_DOCKER_BIN=docker
+export HOME=/tmp/$1
+
+cd /usr/src/arvados/services/api
+exec bundle exec ./script/crunch-dispatch.rb development
diff --git a/arvbox/lib/arvbox/docker/fuse.conf b/arvbox/lib/arvbox/docker/fuse.conf
new file mode 100644
index 0000000..a439ab8
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/fuse.conf
@@ -0,0 +1 @@
+user_allow_other
diff --git a/arvbox/lib/arvbox/docker/gitolite.rc b/arvbox/lib/arvbox/docker/gitolite.rc
new file mode 100644
index 0000000..03c4b29
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/gitolite.rc
@@ -0,0 +1,213 @@
+# This is based on the default Gitolite configuration file with the following
+# changes applied as described here:
+# http://doc.arvados.org/install/install-arv-git-httpd.html
+
+# configuration variables for gitolite
+
+# This file is in perl syntax. But you do NOT need to know perl to edit it --
+# just mind the commas, use single quotes unless you know what you're doing,
+# and make sure the brackets and braces stay matched up!
+
+# (Tip: perl allows a comma after the last item in a list also!)
+
+# HELP for commands can be had by running the command with "-h".
+
+# HELP for all the other FEATURES can be found in the documentation (look for
+# "list of non-core programs shipped with gitolite" in the master index) or
+# directly in the corresponding source file.
+
+my $repo_aliases;
+my $aliases_src = "$ENV{HOME}/.gitolite/arvadosaliases.pl";
+if ($ENV{HOME} && (-e $aliases_src)) {
+ $repo_aliases = do $aliases_src;
+}
+$repo_aliases ||= {};
+
+%RC = (
+
+ REPO_ALIASES => $repo_aliases,
+
+ # ------------------------------------------------------------------
+
+ # default umask gives you perms of '0700'; see the rc file docs for
+ # how/why you might change this
+ UMASK => 0022,
+
+ # look for "git-config" in the documentation
+ GIT_CONFIG_KEYS => '',
+
+ # comment out if you don't need all the extra detail in the logfile
+ LOG_EXTRA => 1,
+ # logging options
+ # 1. leave this section as is for 'normal' gitolite logging (default)
+ # 2. uncomment this line to log ONLY to syslog:
+ # LOG_DEST => 'syslog',
+ # 3. uncomment this line to log to syslog and the normal gitolite log:
+ # LOG_DEST => 'syslog,normal',
+ # 4. prefixing "repo-log," to any of the above will **also** log just the
+ # update records to "gl-log" in the bare repo directory:
+ # LOG_DEST => 'repo-log,normal',
+ # LOG_DEST => 'repo-log,syslog',
+ # LOG_DEST => 'repo-log,syslog,normal',
+
+ # roles. add more roles (like MANAGER, TESTER, ...) here.
+ # WARNING: if you make changes to this hash, you MUST run 'gitolite
+ # compile' afterward, and possibly also 'gitolite trigger POST_COMPILE'
+ ROLES => {
+ READERS => 1,
+ WRITERS => 1,
+ },
+
+ # enable caching (currently only Redis). PLEASE RTFM BEFORE USING!!!
+ # CACHE => 'Redis',
+
+ # ------------------------------------------------------------------
+
+ # rc variables used by various features
+
+ # the 'info' command prints this as additional info, if it is set
+ # SITE_INFO => 'Please see http://blahblah/gitolite for more help',
+
+ # the CpuTime feature uses these
+ # display user, system, and elapsed times to user after each git operation
+ # DISPLAY_CPU_TIME => 1,
+ # display a warning if total CPU times (u, s, cu, cs) crosses this limit
+ # CPU_TIME_WARN_LIMIT => 0.1,
+
+ # the Mirroring feature needs this
+ # HOSTNAME => "foo",
+
+ # TTL for redis cache; PLEASE SEE DOCUMENTATION BEFORE UNCOMMENTING!
+ # CACHE_TTL => 600,
+
+ # ------------------------------------------------------------------
+
+ # suggested locations for site-local gitolite code (see cust.html)
+
+ # this one is managed directly on the server
+ # LOCAL_CODE => "$ENV{HOME}/local",
+
+ # or you can use this, which lets you put everything in a subdirectory
+ # called "local" in your gitolite-admin repo. For a SECURITY WARNING
+ # on this, see http://gitolite.com/gitolite/non-core.html#pushcode
+ # LOCAL_CODE => "$rc{GL_ADMIN_BASE}/local",
+
+ # ------------------------------------------------------------------
+
+ # List of commands and features to enable
+
+ ENABLE => [
+
+ # COMMANDS
+
+ # These are the commands enabled by default
+ 'help',
+ 'desc',
+ 'info',
+ 'perms',
+ 'writable',
+
+ # Uncomment or add new commands here.
+ # 'create',
+ # 'fork',
+ # 'mirror',
+ # 'readme',
+ # 'sskm',
+ # 'D',
+
+ # These FEATURES are enabled by default.
+
+ # essential (unless you're using smart-http mode)
+ 'ssh-authkeys',
+
+ # creates git-config enties from gitolite.conf file entries like 'config foo.bar = baz'
+ 'git-config',
+
+ # creates git-daemon-export-ok files; if you don't use git-daemon, comment this out
+ 'daemon',
+
+ # creates projects.list file; if you don't use gitweb, comment this out
+ 'gitweb',
+
+ # These FEATURES are disabled by default; uncomment to enable. If you
+ # need to add new ones, ask on the mailing list :-)
+
+ # user-visible behaviour
+
+ # prevent wild repos auto-create on fetch/clone
+ # 'no-create-on-read',
+ # no auto-create at all (don't forget to enable the 'create' command!)
+ # 'no-auto-create',
+
+ # access a repo by another (possibly legacy) name
+ 'Alias',
+
+ # give some users direct shell access. See documentation in
+ # sts.html for details on the following two choices.
+ # "Shell $ENV{HOME}/.gitolite.shell-users",
+ # 'Shell alice bob',
+
+ # set default roles from lines like 'option default.roles-1 = ...', etc.
+ # 'set-default-roles',
+
+ # show more detailed messages on deny
+ # 'expand-deny-messages',
+
+ # show a message of the day
+ # 'Motd',
+
+ # system admin stuff
+
+ # enable mirroring (don't forget to set the HOSTNAME too!)
+ # 'Mirroring',
+
+ # allow people to submit pub files with more than one key in them
+ # 'ssh-authkeys-split',
+
+ # selective read control hack
+ # 'partial-copy',
+
+ # manage local, gitolite-controlled, copies of read-only upstream repos
+ # 'upstream',
+
+ # updates 'description' file instead of 'gitweb.description' config item
+ # 'cgit',
+
+ # allow repo-specific hooks to be added
+ # 'repo-specific-hooks',
+
+ # performance, logging, monitoring...
+
+ # be nice
+ # 'renice 10',
+
+ # log CPU times (user, system, cumulative user, cumulative system)
+ # 'CpuTime',
+
+ # syntactic_sugar for gitolite.conf and included files
+
+ # allow backslash-escaped continuation lines in gitolite.conf
+ # 'continuation-lines',
+
+ # create implicit user groups from directory names in keydir/
+ # 'keysubdirs-as-groups',
+
+ # allow simple line-oriented macros
+ # 'macros',
+
+ # Kindergarten mode
+
+ # disallow various things that sensible people shouldn't be doing anyway
+ # 'Kindergarten',
+ ],
+
+);
+
+# ------------------------------------------------------------------------------
+# per perl rules, this should be the last line in such a file:
+1;
+
+# Local variables:
+# mode: perl
+# End:
+# vim: set syn=perl:
diff --git a/arvbox/lib/arvbox/docker/keep-setup.sh b/arvbox/lib/arvbox/docker/keep-setup.sh
new file mode 100755
index 0000000..b66463f
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/keep-setup.sh
@@ -0,0 +1,51 @@
+#!/bin/bash
+
+exec 2>&1
+sleep 2
+set -eux -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+mkdir -p /var/lib/arvados/gostuff
+cd /var/lib/arvados/gostuff
+
+export GOPATH=$PWD
+mkdir -p "$GOPATH/src/git.curoverse.com"
+ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/keepstore"
+install bin/keepstore /usr/local/bin
+
+mkdir -p /var/lib/arvados/$1
+
+export ARVADOS_API_HOST=$localip:${services[api]}
+export ARVADOS_API_HOST_INSECURE=1
+export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+
+set +e
+read -rd $'\000' keepservice <<EOF
+{
+ "service_host":"$localip",
+ "service_port":$2,
+ "service_ssl_flag":false,
+ "service_type":"disk"
+}
+EOF
+set -e
+
+if test -s /var/lib/arvados/$1-uuid ; then
+ keep_uuid=$(cat /var/lib/arvados/$1-uuid)
+ arv keep_service update --uuid $keep_uuid --keep-service "$keepservice"
+else
+ UUID=$(arv --format=uuid keep_service create --keep-service "$keepservice")
+ echo $UUID > /var/lib/arvados/$1-uuid
+fi
+
+set +e
+killall -HUP keepproxy
+
+exec /usr/local/bin/keepstore \
+ -listen=:$2 \
+ -enforce-permissions=true \
+ -blob-signing-key-file=/var/lib/arvados/blob_signing_key \
+ -max-buffers=20 \
+ -volume=/var/lib/arvados/$1
diff --git a/arvbox/lib/arvbox/docker/logger b/arvbox/lib/arvbox/docker/logger
new file mode 100755
index 0000000..a79a518
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/logger
@@ -0,0 +1,2 @@
+#!/bin/sh
+exec svlogd -tt ./main
diff --git a/arvbox/lib/arvbox/docker/runit-docker/.gitignore b/arvbox/lib/arvbox/docker/runit-docker/.gitignore
new file mode 100644
index 0000000..bbf313b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/.gitignore
@@ -0,0 +1,32 @@
+# Object files
+*.o
+*.ko
+*.obj
+*.elf
+
+# Precompiled Headers
+*.gch
+*.pch
+
+# Libraries
+*.lib
+*.a
+*.la
+*.lo
+
+# Shared objects (inc. Windows DLLs)
+*.dll
+*.so
+*.so.*
+*.dylib
+
+# Executables
+*.exe
+*.out
+*.app
+*.i*86
+*.x86_64
+*.hex
+
+# Debug files
+*.dSYM/
diff --git a/arvbox/lib/arvbox/docker/runit-docker/LICENSE b/arvbox/lib/arvbox/docker/runit-docker/LICENSE
new file mode 100644
index 0000000..d158667
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/LICENSE
@@ -0,0 +1,28 @@
+Copyright (c) 2015, Kosma Moczek
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+* Neither the name of runit-docker nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
diff --git a/arvbox/lib/arvbox/docker/runit-docker/Makefile b/arvbox/lib/arvbox/docker/runit-docker/Makefile
new file mode 100644
index 0000000..9a28963
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/Makefile
@@ -0,0 +1,18 @@
+CFLAGS=-std=c99 -Wall -O2 -fPIC -D_POSIX_SOURCE -D_GNU_SOURCE
+LDLIBS=-ldl
+
+PROGNAME=runit-docker
+
+all: $(PROGNAME).so
+
+%.so: %.c
+ gcc -shared $(CFLAGS) $(LDLIBS) -o $@ $^
+
+install: runit-docker.so
+ mkdir -p $(DESTDIR)/sbin
+ mkdir -p $(DESTDIR)/lib
+ install -m 755 $(PROGNAME) $(DESTDIR)/sbin/
+ install -m 755 $(PROGNAME).so $(DESTDIR)/lib/
+
+clean:
+ $(RM) $(PROGNAME).so
diff --git a/arvbox/lib/arvbox/docker/runit-docker/README.md b/arvbox/lib/arvbox/docker/runit-docker/README.md
new file mode 100644
index 0000000..1bcb8cc
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/README.md
@@ -0,0 +1,24 @@
+# runit-docker
+
+Docker and `runsvdir` don't quite agree on what each signal means, causing
+TONS of frustration when attempting to use `runsvdir` as init under Docker.
+`runit-docker` is a plug'n'play adapter library which does signal translation
+without the overhead and nuisance of running a nanny process.
+
+## Features
+
+* Pressing Ctrl-C does a clean shutdown.
+* `docker stop` does a clean shutdown.
+
+Under the hood, `runit-docker` translates `SIGTERM` and `SIGINT` to `SIGHUP`.
+
+## Usage
+
+* Build with `make`, install with `make install`.
+* Add `CMD ["/sbin/runit-docker"]` to your `Dockerfile`.
+* Run `debian/rules clean build binary` to build a Debian package.
+
+## Author
+
+runit-docker was written by Kosma Moczek <kosma.moczek at pixers.pl> during a single Scrum
+planning meeting. Damn meetings.
diff --git a/arvbox/lib/arvbox/docker/runit-docker/debian/changelog b/arvbox/lib/arvbox/docker/runit-docker/debian/changelog
new file mode 100644
index 0000000..7d8689f
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/debian/changelog
@@ -0,0 +1,12 @@
+runit-docker (1.1) unstable; urgency=low
+
+ * Simplify logic.
+ * Install for SIGINT as well.
+
+ -- Kosma Moczek <kosma at kosma.pl> Mon, 11 May 2015 12:23:59 +0000
+
+runit-docker (1.0) unstable; urgency=low
+
+ * Initial release
+
+ -- Kosma Moczek <kosma at kosma.pl> Mon, 11 May 2015 12:23:59 +0000
diff --git a/arvbox/lib/arvbox/docker/runit-docker/debian/compat b/arvbox/lib/arvbox/docker/runit-docker/debian/compat
new file mode 100644
index 0000000..ec63514
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/debian/compat
@@ -0,0 +1 @@
+9
diff --git a/arvbox/lib/arvbox/docker/runit-docker/debian/control b/arvbox/lib/arvbox/docker/runit-docker/debian/control
new file mode 100644
index 0000000..4060915
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/debian/control
@@ -0,0 +1,14 @@
+Source: runit-docker
+Section: contrib/admin
+Priority: optional
+Maintainer: Kosma Moczek <kosma at kosma.pl>
+Build-Depends: debhelper (>= 9)
+Standards-Version: 3.9.5
+Homepage: https://github.com/kosma/runit-docker
+#Vcs-Git: git://anonscm.debian.org/collab-maint/runit-docker.git
+#Vcs-Browser: http://anonscm.debian.org/?p=collab-maint/runit-docker.git;a=summary
+
+Package: runit-docker
+Architecture: any
+Depends: ${shlibs:Depends}, ${misc:Depends}
+Description: painlessly use runit in Docker containers
diff --git a/arvbox/lib/arvbox/docker/runit-docker/debian/copyright b/arvbox/lib/arvbox/docker/runit-docker/debian/copyright
new file mode 100644
index 0000000..8679a6a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/debian/copyright
@@ -0,0 +1,31 @@
+Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Upstream-Name: runit-docker
+Source: https://github.com/kosma/runit-docker
+
+Files: *
+Copyright: 2015 Kosma Moczek <kosma at kosma.pl>
+License: MIT
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+ * Neither the name of runit-docker nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+ FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/arvbox/lib/arvbox/docker/runit-docker/debian/docs b/arvbox/lib/arvbox/docker/runit-docker/debian/docs
new file mode 100644
index 0000000..b43bf86
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/debian/docs
@@ -0,0 +1 @@
+README.md
diff --git a/arvbox/lib/arvbox/docker/runit-docker/debian/rules b/arvbox/lib/arvbox/docker/runit-docker/debian/rules
new file mode 100755
index 0000000..ce15cce
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/debian/rules
@@ -0,0 +1,32 @@
+#!/usr/bin/make -f
+# See debhelper(7) (uncomment to enable)
+# output every command that modifies files on the build system.
+#DH_VERBOSE = 1
+
+# see EXAMPLES in dpkg-buildflags(1) and read /usr/share/dpkg/*
+DPKG_EXPORT_BUILDFLAGS = 1
+include /usr/share/dpkg/default.mk
+
+# see FEATURE AREAS in dpkg-buildflags(1)
+#export DEB_BUILD_MAINT_OPTIONS = hardening=+all
+
+# see ENVIRONMENT in dpkg-buildflags(1)
+# package maintainers to append CFLAGS
+#export DEB_CFLAGS_MAINT_APPEND = -Wall -pedantic
+# package maintainers to append LDFLAGS
+#export DEB_LDFLAGS_MAINT_APPEND = -Wl,--as-needed
+
+
+# main packaging script based on dh7 syntax
+%:
+ dh $@
+
+# debmake generated override targets
+# This is example for Cmake (See http://bugs.debian.org/641051 )
+#override_dh_auto_configure:
+# dh_auto_configure -- \
+# -DCMAKE_LIBRARY_PATH=$(DEB_HOST_MULTIARCH)
+
+
+
+
diff --git a/arvbox/lib/arvbox/docker/runit-docker/debian/source/format b/arvbox/lib/arvbox/docker/runit-docker/debian/source/format
new file mode 100644
index 0000000..163aaf8
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/debian/source/format
@@ -0,0 +1 @@
+3.0 (quilt)
diff --git a/arvbox/lib/arvbox/docker/runit-docker/runit-docker b/arvbox/lib/arvbox/docker/runit-docker/runit-docker
new file mode 100755
index 0000000..fdbaad5
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/runit-docker
@@ -0,0 +1,4 @@
+#!/bin/sh
+
+export LD_PRELOAD=/lib/runit-docker.so
+exec runsvdir /etc/service
diff --git a/arvbox/lib/arvbox/docker/runit-docker/runit-docker.c b/arvbox/lib/arvbox/docker/runit-docker/runit-docker.c
new file mode 100644
index 0000000..825a35f
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runit-docker/runit-docker.c
@@ -0,0 +1,32 @@
+#include <signal.h>
+#include <dlfcn.h>
+#include <stdlib.h>
+
+
+int sigaction(int signum, const struct sigaction *act, struct sigaction *oldact)
+{
+ static int (*real_sigaction)(int signum, const struct sigaction *act, struct sigaction *oldact) = NULL;
+
+ // Retrieve the real sigaction we just shadowed.
+ if (real_sigaction == NULL) {
+ real_sigaction = (void *) dlsym(RTLD_NEXT, "sigaction");
+ // Prevent further shadowing in children.
+ unsetenv("LD_PRELOAD");
+ }
+
+ if (signum == SIGTERM) {
+ // Skip this handler, it doesn't do what we want.
+ return 0;
+ }
+
+ if (signum == SIGHUP) {
+ // Install this handler for others as well.
+ real_sigaction(SIGTERM, act, oldact);
+ real_sigaction(SIGINT, act, oldact);
+ }
+
+ // Forward the call the the real sigaction.
+ return real_sigaction(signum, act, oldact);
+}
+
+// vim: ts=2 sw=2 et
diff --git a/arvbox/lib/arvbox/docker/runsu.sh b/arvbox/lib/arvbox/docker/runsu.sh
new file mode 100755
index 0000000..1557d09
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/runsu.sh
@@ -0,0 +1,14 @@
+#!/bin/sh
+
+HOSTUID=$(ls -nd /usr/src/arvados | sed 's/ */ /' | cut -d' ' -f4)
+HOSTGID=$(ls -nd /usr/src/arvados | sed 's/ */ /' | cut -d' ' -f5)
+
+flock /var/lib/arvados/createusers.lock /usr/local/lib/arvbox/createusers.sh
+
+export HOME=/var/lib/arvados
+
+if test -z "$1" ; then
+ exec chpst -u arvbox:arvbox:docker $0-service
+else
+ exec chpst -u arvbox:arvbox:docker $@
+fi
diff --git a/arvbox/lib/arvbox/docker/service/api/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/api/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/api/log/run b/arvbox/lib/arvbox/docker/service/api/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/api/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/api/run b/arvbox/lib/arvbox/docker/service/api/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/api/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/api/run-service b/arvbox/lib/arvbox/docker/service/api/run-service
new file mode 100755
index 0000000..31f90d1
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/api/run-service
@@ -0,0 +1,100 @@
+#!/bin/bash
+
+exec 2>&1
+set -ex -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+cd /usr/src/arvados/services/api
+export RAILS_ENV=development
+
+run_bundler --without=development
+bundle exec passenger start --runtime-check-only --runtime-dir=/var/lib/passenger
+
+if test "$1" = "--only-deps" ; then
+ exit
+fi
+
+set -u
+
+if ! test -s /var/lib/arvados/api_uuid_prefix ; then
+ ruby -e 'puts "#{rand(2**64).to_s(36)[0,5]}"' > /var/lib/arvados/api_uuid_prefix
+fi
+uuid_prefix=$(cat /var/lib/arvados/api_uuid_prefix)
+
+if ! test -s /var/lib/arvados/api_secret_token ; then
+ ruby -e 'puts rand(2**400).to_s(36)' > /var/lib/arvados/api_secret_token
+fi
+secret_token=$(cat /var/lib/arvados/api_secret_token)
+
+if ! test -s /var/lib/arvados/blob_signing_key ; then
+ ruby -e 'puts rand(2**400).to_s(36)' > /var/lib/arvados/blob_signing_key
+fi
+blob_signing_key=$(cat /var/lib/arvados/blob_signing_key)
+
+# self signed key will be created by SSO server script.
+test -s /var/lib/arvados/self-signed.key
+
+sso_app_secret=$(cat /var/lib/arvados/sso_app_secret)
+
+if test -s /var/lib/arvados/vm-uuid ; then
+ vm_uuid=$(cat /var/lib/arvados/vm-uuid)
+else
+ vm_uuid=$uuid_prefix-2x53u-$(ruby -e 'puts rand(2**400).to_s(36)[0,15]')
+ echo $vm_uuid > /var/lib/arvados/vm-uuid
+fi
+
+cat >config/application.yml <<EOF
+development:
+ uuid_prefix: $uuid_prefix
+ secret_token: $secret_token
+ blob_signing_key: $blob_signing_key
+ sso_app_secret: $sso_app_secret
+ sso_app_id: arvados-server
+ sso_provider_url: "https://$localip:${services[sso]}"
+ sso_insecure: true
+ workbench_address: "http://$localip/"
+ git_repo_ssh_base: "git@$localip:"
+ git_repo_https_base: "http://$localip:${services[arv-git-httpd]}/"
+ new_users_are_active: true
+ auto_admin_first_user: true
+ auto_setup_new_users: true
+ auto_setup_new_users_with_vm_uuid: $vm_uuid
+ auto_setup_new_users_with_repository: true
+ default_collection_replication: 1
+EOF
+
+if ! test -f /var/lib/arvados/api_database_pw ; then
+ ruby -e 'puts rand(2**128).to_s(36)' > /var/lib/arvados/api_database_pw
+fi
+database_pw=$(cat /var/lib/arvados/api_database_pw)
+
+if ! (psql -c "\du" | grep "^ arvados ") >/dev/null ; then
+ psql -c "create user arvados with password '$database_pw'"
+ psql -c "ALTER USER arvados CREATEDB;"
+fi
+
+sed "s/password:.*/password: $database_pw/" <config/database.yml.example >config/database.yml
+
+if ! test -f /var/lib/arvados/api_database_setup ; then
+ bundle exec rake db:setup
+ touch /var/lib/arvados/api_database_setup
+fi
+
+if ! test -s /var/lib/arvados/superuser_token ; then
+ bundle exec ./script/create_superuser_token.rb > /var/lib/arvados/superuser_token
+fi
+
+rm -rf tmp
+
+bundle exec rake db:migrate
+
+set +u
+if test "$1" = "--only-setup" ; then
+ exit
+fi
+
+ARVADOS_WEBSOCKETS=1 exec bundle exec passenger start --port=${services[api]} \
+ --runtime-dir=/var/lib/passenger \
+ --ssl --ssl-certificate=/var/lib/arvados/self-signed.pem \
+ --ssl-certificate-key=/var/lib/arvados/self-signed.key
diff --git a/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/run b/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/arv-git-httpd/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/arv-git-httpd/run b/arvbox/lib/arvbox/docker/service/arv-git-httpd/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/arv-git-httpd/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/arv-git-httpd/run-service b/arvbox/lib/arvbox/docker/service/arv-git-httpd/run-service
new file mode 100755
index 0000000..7bd6a7c
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/arv-git-httpd/run-service
@@ -0,0 +1,27 @@
+#!/bin/bash
+
+exec 2>&1
+set -eux -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+mkdir -p /var/lib/arvados/gostuff
+cd /var/lib/arvados/gostuff
+
+export GOPATH=$PWD
+mkdir -p "$GOPATH/src/git.curoverse.com"
+ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/arv-git-httpd"
+install bin/arv-git-httpd /usr/local/bin
+
+export ARVADOS_API_HOST=$localip:${services[api]}
+export ARVADOS_API_HOST_INSECURE=1
+export GITOLITE_HTTP_HOME=/var/lib/arvados/git
+export GL_BYPASS_ACCESS_CHECKS=1
+export PATH="$PATH:/var/lib/arvados/git/bin"
+cd ~git
+
+exec /usr/local/bin/arv-git-httpd \
+ -address=:${services[arv-git-httpd]} \
+ -git-command=/var/lib/arvados/git/gitolite/src/gitolite-shell \
+ -repo-root=/var/lib/arvados/git/repositories
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/run b/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/crunch-dispatch0/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run b/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run-service b/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run-service
new file mode 100755
index 0000000..fa3a73a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/crunch-dispatch0/run-service
@@ -0,0 +1,2 @@
+#!/bin/sh
+exec /usr/local/lib/arvbox/crunch-setup.sh crunch0
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/run b/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/crunch-dispatch1/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run b/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run-service b/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run-service
new file mode 100755
index 0000000..6430e9c
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/crunch-dispatch1/run-service
@@ -0,0 +1,3 @@
+#!/bin/sh
+sleep 1
+exec /usr/local/lib/arvbox/crunch-setup.sh crunch1
diff --git a/arvbox/lib/arvbox/docker/service/doc/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/doc/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/doc/log/run b/arvbox/lib/arvbox/docker/service/doc/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/doc/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/doc/run b/arvbox/lib/arvbox/docker/service/doc/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/doc/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/doc/run-service b/arvbox/lib/arvbox/docker/service/doc/run-service
new file mode 100755
index 0000000..acbe21c
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/doc/run-service
@@ -0,0 +1,41 @@
+#!/bin/bash
+
+exec 2>&1
+set -ex -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+cd /usr/src/arvados/doc
+run_bundler --without=development
+
+if test "$1" = "--only-deps" ; then
+ exit
+fi
+
+set -u
+
+cat <<EOF >/var/lib/arvados/doc-nginx.conf
+worker_processes auto;
+pid /var/lib/arvados/doc-nginx.pid;
+daemon off;
+
+events {
+ worker_connections 64;
+}
+
+http {
+ include /etc/nginx/mime.types;
+ default_type application/octet-stream;
+ server {
+ listen ${services[doc]} default_server;
+ listen [::]:${services[doc]} default_server;
+ root /usr/src/arvados/doc/.site;
+ index index.html;
+ server_name _;
+ }
+}
+EOF
+
+bundle exec rake generate baseurl=http://$localip:${services[doc]} arvados_api_host=$localip:${services[api]} arvados_workbench_host=http://$localip
+
+exec nginx -c /var/lib/arvados/doc-nginx.conf
diff --git a/arvbox/lib/arvbox/docker/service/docker/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/docker/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/docker/log/run b/arvbox/lib/arvbox/docker/service/docker/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/docker/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/docker/run b/arvbox/lib/arvbox/docker/service/docker/run
new file mode 100755
index 0000000..1ecdc16
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/docker/run
@@ -0,0 +1,102 @@
+#!/bin/bash
+
+# Taken from https://github.com/jpetazzo/dind
+
+exec 2>&1
+
+# Ensure that all nodes in /dev/mapper correspond to mapped devices currently loaded by the device-mapper kernel driver
+dmsetup mknodes
+
+: {LOG:=stdio}
+
+# First, make sure that cgroups are mounted correctly.
+CGROUP=/sys/fs/cgroup
+[ -d $CGROUP ] || mkdir $CGROUP
+
+if mountpoint -q $CGROUP ; then
+ break
+else
+ mount -n -t tmpfs -o uid=0,gid=0,mode=0755 cgroup $CGROUP
+fi
+
+if ! mountpoint -q $CGROUP ; then
+ echo "Could not find or mount cgroups. Tried /sys/fs/cgroup and /cgroup. Did you use --privileged?"
+ exit 1
+fi
+
+if [ -d /sys/kernel/security ] && ! mountpoint -q /sys/kernel/security
+then
+ mount -t securityfs none /sys/kernel/security || {
+ echo "Could not mount /sys/kernel/security."
+ echo "AppArmor detection and --privileged mode might break."
+ }
+fi
+
+# Mount the cgroup hierarchies exactly as they are in the parent system.
+for SUBSYS in $(cut -d: -f2 /proc/1/cgroup)
+do
+ [ -d $CGROUP/$SUBSYS ] || mkdir $CGROUP/$SUBSYS
+ mountpoint -q $CGROUP/$SUBSYS ||
+ mount -n -t cgroup -o $SUBSYS cgroup $CGROUP/$SUBSYS
+
+ # The two following sections address a bug which manifests itself
+ # by a cryptic "lxc-start: no ns_cgroup option specified" when
+ # trying to start containers withina container.
+ # The bug seems to appear when the cgroup hierarchies are not
+ # mounted on the exact same directories in the host, and in the
+ # container.
+
+ # Named, control-less cgroups are mounted with "-o name=foo"
+ # (and appear as such under /proc/<pid>/cgroup) but are usually
+ # mounted on a directory named "foo" (without the "name=" prefix).
+ # Systemd and OpenRC (and possibly others) both create such a
+ # cgroup. To avoid the aforementioned bug, we symlink "foo" to
+ # "name=foo". This shouldn't have any adverse effect.
+ echo $SUBSYS | grep -q ^name= && {
+ NAME=$(echo $SUBSYS | sed s/^name=//)
+ ln -s $SUBSYS $CGROUP/$NAME
+ }
+
+ # Likewise, on at least one system, it has been reported that
+ # systemd would mount the CPU and CPU accounting controllers
+ # (respectively "cpu" and "cpuacct") with "-o cpuacct,cpu"
+ # but on a directory called "cpu,cpuacct" (note the inversion
+ # in the order of the groups). This tries to work around it.
+ [ $SUBSYS = cpuacct,cpu ] && ln -s $SUBSYS $CGROUP/cpu,cpuacct
+done
+
+# Note: as I write those lines, the LXC userland tools cannot setup
+# a "sub-container" properly if the "devices" cgroup is not in its
+# own hierarchy. Let's detect this and issue a warning.
+grep -q :devices: /proc/1/cgroup ||
+ echo "WARNING: the 'devices' cgroup should be in its own hierarchy."
+grep -qw devices /proc/1/cgroup ||
+ echo "WARNING: it looks like the 'devices' cgroup is not mounted."
+
+# Now, close extraneous file descriptors.
+pushd /proc/self/fd >/dev/null
+for FD in *
+do
+ case "$FD" in
+ # Keep stdin/stdout/stderr
+ [012])
+ ;;
+ # Nuke everything else
+ *)
+ eval exec "$FD>&-"
+ ;;
+ esac
+done
+popd >/dev/null
+
+
+# If a pidfile is still around (for example after a container restart),
+# delete it so that docker can start.
+rm -rf /var/run/docker.pid
+
+read pid cmd state ppid pgrp session tty_nr tpgid rest < /proc/self/stat
+trap "kill -TERM -$pgrp; exit" EXIT TERM KILL SIGKILL SIGTERM SIGQUIT
+
+if ! docker daemon --storage-driver=overlay $DOCKER_DAEMON_ARGS ; then
+ docker daemon $DOCKER_DAEMON_ARGS
+fi
diff --git a/arvbox/lib/arvbox/docker/service/gitolite/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/gitolite/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/gitolite/log/run b/arvbox/lib/arvbox/docker/service/gitolite/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/gitolite/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/gitolite/run b/arvbox/lib/arvbox/docker/service/gitolite/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/gitolite/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/gitolite/run-service b/arvbox/lib/arvbox/docker/service/gitolite/run-service
new file mode 100755
index 0000000..8e6cb0e
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/gitolite/run-service
@@ -0,0 +1,116 @@
+#!/bin/bash
+
+exec 2>&1
+set -eux -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+mkdir -p /var/lib/arvados/git
+
+export ARVADOS_API_HOST=$localip:${services[api]}
+export ARVADOS_API_HOST_INSECURE=1
+export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+
+export USER=git
+export USERNAME=git
+export LOGNAME=git
+export HOME=/var/lib/arvados/git
+
+cd ~arvbox
+
+mkdir -p ~arvbox/.ssh ~git/.ssh
+chmod 0700 ~arvbox/.ssh ~git/.ssh
+
+if ! test -s ~arvbox/.ssh/id_rsa ; then
+ ssh-keygen -t rsa -P '' -f .ssh/id_rsa
+ cp ~arvbox/.ssh/id_rsa ~arvbox/.ssh/id_rsa.pub ~git/.ssh
+fi
+
+if test -s ~arvbox/.ssh/known_hosts ; then
+ ssh-keygen -f ".ssh/known_hosts" -R localhost
+fi
+
+if ! test -f /var/lib/arvados/gitolite-setup ; then
+ cd ~git
+
+ # Do a no-op login to populate known_hosts
+ # with the hostkey, so it won't try to ask
+ # about it later.
+ cp .ssh/id_rsa.pub .ssh/authorized_keys
+ ssh -o stricthostkeychecking=no git at localhost true
+ rm .ssh/authorized_keys
+
+ cp -r /usr/local/lib/arvbox/gitolite.rc .
+
+ gitolite setup -pk .ssh/id_rsa.pub
+
+ if ! test -d gitolite-admin ; then
+ git clone git at localhost:gitolite-admin
+ fi
+
+ cd gitolite-admin
+ git config user.email arvados
+ git config user.name arvados
+ git config push.default simple
+ git push
+
+ touch /var/lib/arvados/gitolite-setup
+else
+ # Do a no-op login to populate known_hosts
+ # with the hostkey, so it won't try to ask
+ # about it later. Don't run anything,
+ # get the default gitolite behavior.
+ ssh -o stricthostkeychecking=no git at localhost
+fi
+
+prefix=$(arv --format=uuid user current | cut -d- -f1)
+
+if ! test -s /var/lib/arvados/arvados-git-uuid ; then
+ repo_uuid=$(arv --format=uuid repository create --repository "{\"owner_uuid\":\"$prefix-tpzed-000000000000000\", \"name\":\"arvados\"}")
+ echo $repo_uuid > /var/lib/arvados/arvados-git-uuid
+fi
+
+repo_uuid=$(cat /var/lib/arvados/arvados-git-uuid)
+
+if ! test -s /var/lib/arvados/arvados-git-link-uuid ; then
+ all_users_group_uuid="$prefix-j7d0g-fffffffffffffff"
+
+ set +e
+ read -rd $'\000' newlink <<EOF
+{
+ "tail_uuid":"$all_users_group_uuid",
+ "head_uuid":"$repo_uuid",
+ "link_class":"permission",
+ "name":"can_read"
+}
+EOF
+ set -e
+ link_uuid=$(arv --format=uuid link create --link "$newlink")
+ echo $link_uuid > /var/lib/arvados/arvados-git-link-uuid
+fi
+
+if ! test -d /var/lib/arvados/git/repositories/$repo_uuid.git ; then
+ git clone --bare /usr/src/arvados /var/lib/arvados/git/repositories/$repo_uuid.git
+else
+ git --git-dir=/var/lib/arvados/git/repositories/$repo_uuid.git fetch -f /usr/src/arvados master:master
+fi
+
+cd /usr/src/arvados/services/api
+export RAILS_ENV=development
+
+git_user_key=$(cat ~git/.ssh/id_rsa.pub)
+
+cat > config/arvados-clients.yml <<EOF
+development:
+ gitolite_url: /var/lib/arvados/git/repositories/gitolite-admin.git
+ gitolite_tmp: /var/lib/arvados/git
+ arvados_api_host: $localip:${services[api]}
+ arvados_api_token: "$ARVADOS_API_TOKEN"
+ arvados_api_host_insecure: true
+ gitolite_arvados_git_user_key: "$git_user_key"
+EOF
+
+while true ; do
+ bundle exec script/arvados-git-sync.rb development
+ sleep 120
+done
diff --git a/arvbox/lib/arvbox/docker/service/keep-web/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/keep-web/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/keep-web/log/run b/arvbox/lib/arvbox/docker/service/keep-web/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keep-web/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/keep-web/run b/arvbox/lib/arvbox/docker/service/keep-web/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keep-web/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/keep-web/run-service b/arvbox/lib/arvbox/docker/service/keep-web/run-service
new file mode 100755
index 0000000..a2c6aa1
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keep-web/run-service
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+exec 2>&1
+set -eux -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+mkdir -p /var/lib/arvados/gostuff
+cd /var/lib/arvados/gostuff
+
+export GOPATH=$PWD
+mkdir -p "$GOPATH/src/git.curoverse.com"
+ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/keep-web"
+install bin/keep-web /usr/local/bin
+
+export ARVADOS_API_HOST=$localip:${services[api]}
+export ARVADOS_API_HOST_INSECURE=1
+export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+
+exec /usr/local/bin/keep-web -trust-all-content -listen=:${services[keep-web]}
diff --git a/arvbox/lib/arvbox/docker/service/keepproxy/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/keepproxy/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/keepproxy/log/run b/arvbox/lib/arvbox/docker/service/keepproxy/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keepproxy/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/keepproxy/run b/arvbox/lib/arvbox/docker/service/keepproxy/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keepproxy/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/keepproxy/run-service b/arvbox/lib/arvbox/docker/service/keepproxy/run-service
new file mode 100755
index 0000000..413a67e
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keepproxy/run-service
@@ -0,0 +1,41 @@
+#!/bin/bash
+
+exec 2>&1
+sleep 2
+set -eux -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+mkdir -p /var/lib/arvados/gostuff
+cd /var/lib/arvados/gostuff
+
+export GOPATH=$PWD
+mkdir -p "$GOPATH/src/git.curoverse.com"
+ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/keepproxy"
+install bin/keepproxy /usr/local/bin
+
+export ARVADOS_API_HOST=$localip:${services[api]}
+export ARVADOS_API_HOST_INSECURE=1
+export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+
+set +e
+read -rd $'\000' keepservice <<EOF
+{
+ "service_host":"$localip",
+ "service_port":${services[keepproxy]},
+ "service_ssl_flag":false,
+ "service_type":"proxy"
+}
+EOF
+set -e
+
+if test -s /var/lib/arvados/keepproxy-uuid ; then
+ keep_uuid=$(cat /var/lib/arvados/keepproxy-uuid)
+ arv keep_service update --uuid $keep_uuid --keep-service "$keepservice"
+else
+ UUID=$(arv --format=uuid keep_service create --keep-service "$keepservice")
+ echo $UUID > /var/lib/arvados/keepproxy-uuid
+fi
+
+exec /usr/local/bin/keepproxy -listen=:${services[keepproxy]}
diff --git a/arvbox/lib/arvbox/docker/service/keepstore0/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/keepstore0/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/keepstore0/log/run b/arvbox/lib/arvbox/docker/service/keepstore0/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keepstore0/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/keepstore0/run b/arvbox/lib/arvbox/docker/service/keepstore0/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keepstore0/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/keepstore0/run-service b/arvbox/lib/arvbox/docker/service/keepstore0/run-service
new file mode 100755
index 0000000..cf411e4
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keepstore0/run-service
@@ -0,0 +1,3 @@
+#!/bin/bash
+. /usr/local/lib/arvbox/common.sh
+exec /usr/local/lib/arvbox/keep-setup.sh keep0 ${services[keepstore0]}
diff --git a/arvbox/lib/arvbox/docker/service/keepstore1/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/keepstore1/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/keepstore1/log/run b/arvbox/lib/arvbox/docker/service/keepstore1/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keepstore1/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/keepstore1/run b/arvbox/lib/arvbox/docker/service/keepstore1/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keepstore1/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/keepstore1/run-service b/arvbox/lib/arvbox/docker/service/keepstore1/run-service
new file mode 100755
index 0000000..8d34d06
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/keepstore1/run-service
@@ -0,0 +1,3 @@
+#!/bin/bash
+. /usr/local/lib/arvbox/common.sh
+exec /usr/local/lib/arvbox/keep-setup.sh keep1 ${services[keepstore1]}
diff --git a/arvbox/lib/arvbox/docker/service/postgres/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/postgres/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/postgres/log/run b/arvbox/lib/arvbox/docker/service/postgres/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/postgres/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/postgres/run b/arvbox/lib/arvbox/docker/service/postgres/run
new file mode 100755
index 0000000..4918bd7
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/postgres/run
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+flock /var/lib/arvados/createusers.lock /usr/local/lib/arvbox/createusers.sh
+
+. /usr/local/lib/arvbox/common.sh
+
+chown -R $PGUSER:$PGGROUP /var/lib/postgresql
+chown -R $PGUSER:$PGGROUP /var/run/postgresql
+chown -R $PGUSER:$PGGROUP /etc/postgresql
+chown -R $PGUSER:$PGGROUP /etc/ssl/private
+
+exec chpst -u $PGUSER:$PGGROUP $0-service
diff --git a/arvbox/lib/arvbox/docker/service/postgres/run-service b/arvbox/lib/arvbox/docker/service/postgres/run-service
new file mode 100755
index 0000000..dd2eb1a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/postgres/run-service
@@ -0,0 +1,14 @@
+#!/bin/bash
+
+exec 2>&1
+set -eux -o pipefail
+
+if ! test -d /var/lib/postgresql/9.4/main ; then
+ /usr/lib/postgresql/9.4/bin/initdb -D /var/lib/postgresql/9.4/main
+ sh -c "while ! psql -c'\du' >/dev/null 2>/dev/null ; do createdb ; sleep 1 ; done" &
+fi
+mkdir -p /var/run/postgresql/9.4-main.pg_stat_tmp
+
+rm -f /var/lib/postgresql/9.4/main/postmaster.pid
+
+exec /usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/9.4/main -c config_file=/etc/postgresql/9.4/main/postgresql.conf
diff --git a/arvbox/lib/arvbox/docker/service/ready/run b/arvbox/lib/arvbox/docker/service/ready/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/ready/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/ready/run-service b/arvbox/lib/arvbox/docker/service/ready/run-service
new file mode 100755
index 0000000..f560de0
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/ready/run-service
@@ -0,0 +1,96 @@
+#!/bin/bash
+
+. /usr/local/lib/arvbox/common.sh
+
+set -eu -o pipefail
+
+if ! [[ -d /tmp/arvbox-ready ]] ; then
+ echo
+ echo "Arvados-in-a-box starting"
+ echo
+ echo "Note: if this is a fresh arvbox installation, it may take 10-15 minutes (or longer) to download and"
+ echo "install dependencies. Use \"arvbox log\" to monitor the progress of specific services."
+ echo
+ mkdir -p /tmp/arvbox-ready
+fi
+
+sleep 3
+
+waiting=""
+
+for s in "${!services[@]}"
+do
+ if ! [[ -f /tmp/arvbox-ready/$s ]] ; then
+ if nc -z localhost ${services[$s]} ; then
+ echo "$s is ready at $localip:${services[$s]}"
+ touch /tmp/arvbox-ready/$s
+ else
+ waiting="$waiting $s"
+ fi
+ fi
+done
+
+if ! docker version >/dev/null 2>/dev/null ; then
+ waiting="$waiting docker"
+fi
+
+if ! which arv >/dev/null ; then
+ waiting="$waiting sdk"
+elif ! which arv-get >/dev/null ; then
+ waiting="$waiting sdk"
+fi
+
+if ! (ps x | grep -v grep | grep "crunch-dispatch") > /dev/null ; then
+ waiting="$waiting crunch-dispatch"
+fi
+
+export ARVADOS_API_HOST=$localip:${services[api]}
+export ARVADOS_API_HOST_INSECURE=1
+
+vm_ok=0
+if test -s /var/lib/arvados/vm-uuid -a -s /var/lib/arvados/superuser_token; then
+ vm_uuid=$(cat /var/lib/arvados/vm-uuid)
+ export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+ if (which arv && arv virtual_machine get --uuid $vm_uuid) >/dev/null 2>/dev/null ; then
+ vm_ok=1
+ fi
+fi
+
+if test $vm_ok = 0 ; then
+ waiting="$waiting vm"
+fi
+
+if ! [[ -z "$waiting" ]] ; then
+ if ps x | grep -v grep | grep "bundle install" > /dev/null; then
+ gemcount=$(ls /var/lib/gems/ruby/2.1.0/gems 2>/dev/null | wc -l)
+
+ gemlockcount=0
+ for l in /usr/src/arvados/services/api/Gemfile.lock \
+ /usr/src/arvados/apps/workbench/Gemfile.lock \
+ /usr/src/sso/Gemfile.lock ; do
+ gc=$(cat $l \
+ | grep -vE "(GEM|PLATFORMS|DEPENDENCIES|$^|remote:|specs:)" \
+ | sed 's/^ *//' | sed 's/(.*)//' | sed 's/ *$//' | sort | uniq | wc -l)
+ gemlockcount=$(($gemlockcount + $gc))
+ done
+ waiting="$waiting (installing ruby gems $gemcount/$gemlockcount)"
+ fi
+
+ if ps x | grep -v grep | grep "c++.*/var/lib/passenger" > /dev/null ; then
+ waiting="$waiting (compiling passenger)"
+ fi
+
+ if ps x | grep -v grep | grep "pip install" > /dev/null; then
+ waiting="$waiting (installing python packages)"
+ fi
+ echo " Waiting for$waiting ..."
+ exit 1
+fi
+
+echo
+echo "Your Arvados-in-a-box is ready!"
+echo "Workbench is running at http://$localip"
+
+rm -r /tmp/arvbox-ready
+
+sv stop ready >/dev/null
diff --git a/arvbox/lib/arvbox/docker/service/sdk/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/sdk/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/sdk/log/run b/arvbox/lib/arvbox/docker/service/sdk/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/sdk/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/sdk/run b/arvbox/lib/arvbox/docker/service/sdk/run
new file mode 100755
index 0000000..816b166
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/sdk/run
@@ -0,0 +1,5 @@
+#!/bin/sh
+set -e
+
+/usr/local/lib/arvbox/runsu.sh $0-service
+sv stop sdk
diff --git a/arvbox/lib/arvbox/docker/service/sdk/run-service b/arvbox/lib/arvbox/docker/service/sdk/run-service
new file mode 100755
index 0000000..b51f0fc
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/sdk/run-service
@@ -0,0 +1,24 @@
+#!/bin/bash
+
+exec 2>&1
+set -eux -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+mkdir -p ~/.pip /var/lib/arvados/pip
+cat > ~/.pip/pip.conf <<EOF
+[global]
+download_cache = /var/lib/arvados/pip
+EOF
+
+cd /usr/src/arvados/sdk/cli
+run_bundler --binstubs=$PWD/binstubs
+ln -sf /usr/src/arvados/sdk/cli/binstubs/arv /usr/local/bin/arv
+
+cd /usr/src/arvados/sdk/python
+python setup.py sdist
+pip_install $(ls dist/arvados-python-client-*.tar.gz | tail -n1)
+
+cd /usr/src/arvados/services/fuse
+python setup.py sdist
+pip_install $(ls dist/arvados_fuse-*.tar.gz | tail -n1)
diff --git a/arvbox/lib/arvbox/docker/service/ssh/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/ssh/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/ssh/log/run b/arvbox/lib/arvbox/docker/service/ssh/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/ssh/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/ssh/run b/arvbox/lib/arvbox/docker/service/ssh/run
new file mode 100755
index 0000000..0f23542
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/ssh/run
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+exec 2>&1
+set -eux -o pipefail
+
+if ! test -d /var/run/sshd ; then
+ mkdir /var/run/sshd
+ chmod 0755 /var/run/sshd
+fi
+/usr/sbin/sshd -D
diff --git a/arvbox/lib/arvbox/docker/service/sso/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/sso/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/sso/log/run b/arvbox/lib/arvbox/docker/service/sso/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/sso/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/sso/run b/arvbox/lib/arvbox/docker/service/sso/run
new file mode 120000
index 0000000..a388c8b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/sso/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/sso/run-service b/arvbox/lib/arvbox/docker/service/sso/run-service
new file mode 100755
index 0000000..8a96d47
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/sso/run-service
@@ -0,0 +1,85 @@
+#!/bin/bash
+
+exec 2>&1
+set -ex -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+cd /usr/src/sso
+export RAILS_ENV=development
+
+run_bundler --without=development
+bundle exec passenger start --runtime-check-only --runtime-dir=/var/lib/passenger
+
+if test "$1" = "--only-deps" ; then
+ exit
+fi
+
+set -u
+
+if ! test -s /var/lib/arvados/sso_uuid_prefix ; then
+ ruby -e 'puts "#{rand(2**64).to_s(36)[0,5]}"' > /var/lib/arvados/sso_uuid_prefix
+fi
+uuid_prefix=$(cat /var/lib/arvados/sso_uuid_prefix)
+
+if ! test -s /var/lib/arvados/sso_secret_token ; then
+ ruby -e 'puts rand(2**400).to_s(36)' > /var/lib/arvados/sso_secret_token
+fi
+secret_token=$(cat /var/lib/arvados/sso_secret_token)
+
+if ! test -s /var/lib/arvados/self-signed.key ; then
+ openssl req -new -x509 -nodes -out /var/lib/arvados/self-signed.pem -keyout /var/lib/arvados/self-signed.key -days 365 -subj '/CN=localhost'
+fi
+
+cat >config/application.yml <<EOF
+development:
+ uuid_prefix: $uuid_prefix
+ secret_token: $secret_token
+ default_link_url: "http://$localip"
+ allow_account_registration: true
+EOF
+
+if ! test -f /var/lib/arvados/sso_database_pw ; then
+ ruby -e 'puts rand(2**128).to_s(36)' > /var/lib/arvados/sso_database_pw
+fi
+database_pw=$(cat /var/lib/arvados/sso_database_pw)
+
+if ! (psql -c "\du" | grep "^ arvados_sso ") >/dev/null ; then
+ psql -c "create user arvados_sso with password '$database_pw'"
+ psql -c "ALTER USER arvados_sso CREATEDB;"
+fi
+
+sed "s/password:.*/password: $database_pw/" <config/database.yml.example >config/database.yml
+
+if ! test -f /var/lib/arvados/sso_database_setup ; then
+ bundle exec rake db:setup
+
+ if ! test -s /var/lib/arvados/sso_app_secret ; then
+ ruby -e 'puts rand(2**400).to_s(36)' > /var/lib/arvados/sso_app_secret
+ fi
+ app_secret=$(cat /var/lib/arvados/sso_app_secret)
+
+ bundle exec rails console <<EOF
+c = Client.new
+c.name = "joshid"
+c.app_id = "arvados-server"
+c.app_secret = "$app_secret"
+c.save!
+EOF
+
+ touch /var/lib/arvados/sso_database_setup
+fi
+
+rm -rf tmp
+
+bundle exec rake db:migrate
+
+set +u
+if test "$1" = "--only-setup" ; then
+ exit
+fi
+
+exec bundle exec passenger start --port=${services[sso]} \
+ --runtime-dir=/var/lib/passenger \
+ --ssl --ssl-certificate=/var/lib/arvados/self-signed.pem \
+ --ssl-certificate-key=/var/lib/arvados/self-signed.key
diff --git a/arvbox/lib/arvbox/docker/service/vm/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/vm/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/vm/log/run b/arvbox/lib/arvbox/docker/service/vm/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/vm/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/vm/run b/arvbox/lib/arvbox/docker/service/vm/run
new file mode 100755
index 0000000..b7fb9cc
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/vm/run
@@ -0,0 +1,9 @@
+#!/bin/bash
+set -e
+
+. /usr/local/lib/arvbox/common.sh
+
+git config --system "credential.http://$localip:${services[arv-git-httpd]}/.username" none
+git config --system "credential.http://$localip:${services[arv-git-httpd]}/.helper" '!cred(){ cat >/dev/null; if [ "$1" = get ]; then echo password=$ARVADOS_API_TOKEN; fi; };cred'
+
+exec /usr/local/lib/arvbox/runsu.sh $0-service
diff --git a/arvbox/lib/arvbox/docker/service/vm/run-service b/arvbox/lib/arvbox/docker/service/vm/run-service
new file mode 100755
index 0000000..5bb6825
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/vm/run-service
@@ -0,0 +1,41 @@
+#!/bin/bash
+
+exec 2>&1
+sleep 2
+set -ex -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+cd /usr/src/arvados/services/login-sync
+run_bundler
+
+if test "$1" = "--only-deps" ; then
+ exit
+fi
+
+set -u
+
+export ARVADOS_API_HOST=$localip:${services[api]}
+export ARVADOS_API_HOST_INSECURE=1
+export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+export ARVADOS_VIRTUAL_MACHINE_UUID=$(cat /var/lib/arvados/vm-uuid)
+
+set +e
+read -rd $'\000' vm <<EOF
+{
+ "uuid": "$ARVADOS_VIRTUAL_MACHINE_UUID",
+ "hostname":"$localip"
+}
+EOF
+set -e
+
+if arv virtual_machine get --uuid $ARVADOS_VIRTUAL_MACHINE_UUID ; then
+ arv virtual_machine update --uuid $ARVADOS_VIRTUAL_MACHINE_UUID --virtual-machine "$vm"
+else
+ arv virtual_machine create --virtual-machine "$vm"
+fi
+
+while true ; do
+ bundle exec arvados-login-sync
+ sleep 120
+done
diff --git a/arvbox/lib/arvbox/docker/service/workbench/log/main/.gitstub b/arvbox/lib/arvbox/docker/service/workbench/log/main/.gitstub
new file mode 100644
index 0000000..e69de29
diff --git a/arvbox/lib/arvbox/docker/service/workbench/log/run b/arvbox/lib/arvbox/docker/service/workbench/log/run
new file mode 120000
index 0000000..d6aef4a
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/workbench/log/run
@@ -0,0 +1 @@
+/usr/local/lib/arvbox/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/service/workbench/run b/arvbox/lib/arvbox/docker/service/workbench/run
new file mode 100755
index 0000000..6ac0476
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/workbench/run
@@ -0,0 +1,15 @@
+#!/bin/sh
+set -e
+
+/usr/local/lib/arvbox/runsu.sh $0-service $1
+
+cd /usr/src/arvados/apps/workbench
+
+rm -rf tmp
+mkdir tmp
+chown arvbox:arvbox tmp
+
+if test "$1" != "--only-deps" ; then
+ exec bundle exec passenger start --port 80 \
+ --user arvbox --runtime-dir=/var/lib/passenger
+fi
diff --git a/arvbox/lib/arvbox/docker/service/workbench/run-service b/arvbox/lib/arvbox/docker/service/workbench/run-service
new file mode 100755
index 0000000..7f0542f
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/service/workbench/run-service
@@ -0,0 +1,38 @@
+#!/bin/bash
+
+exec 2>&1
+set -ex -o pipefail
+
+. /usr/local/lib/arvbox/common.sh
+
+cd /usr/src/arvados/apps/workbench
+export RAILS_ENV=development
+
+run_bundler --without=development
+bundle exec passenger start --runtime-check-only --runtime-dir=/var/lib/passenger
+
+if test "$1" = "--only-deps" ; then
+ exit
+fi
+
+set -u
+
+if ! test -s /var/lib/arvados/workbench_secret_token ; then
+ ruby -e 'puts rand(2**400).to_s(36)' > /var/lib/arvados/workbench_secret_token
+fi
+secret_token=$(cat /var/lib/arvados/workbench_secret_token)
+
+if ! test -s self-signed.key ; then
+ openssl req -new -x509 -nodes -out self-signed.pem -keyout self-signed.key -days 365 -subj '/CN=localhost'
+fi
+
+cat >config/application.yml <<EOF
+development:
+ secret_token: $secret_token
+ arvados_login_base: https://$localip:${services[api]}/login
+ arvados_v1_base: https://$localip:${services[api]}/arvados/v1
+ arvados_insecure_https: true
+ keep_web_download_url: http://$localip:${services[keep-web]}/c=%{uuid_or_pdh}
+ keep_web_url: http://$localip:${services[keep-web]}/c=%{uuid_or_pdh}
+ arvados_docsite: http://$localip:${services[doc]}/
+EOF
diff --git a/arvbox/lib/arvbox/docker/tests-service/docker b/arvbox/lib/arvbox/docker/tests-service/docker
new file mode 120000
index 0000000..9374f25
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/tests-service/docker
@@ -0,0 +1 @@
+../service/docker
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/tests-service/logger b/arvbox/lib/arvbox/docker/tests-service/logger
new file mode 120000
index 0000000..b30194b
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/tests-service/logger
@@ -0,0 +1 @@
+../service/logger
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/tests-service/postgres b/arvbox/lib/arvbox/docker/tests-service/postgres
new file mode 120000
index 0000000..9b2d8a0
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/tests-service/postgres
@@ -0,0 +1 @@
+../service/postgres
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/tests-service/runsu.sh b/arvbox/lib/arvbox/docker/tests-service/runsu.sh
new file mode 120000
index 0000000..22030b0
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/tests-service/runsu.sh
@@ -0,0 +1 @@
+../service/runsu.sh
\ No newline at end of file
diff --git a/arvbox/lib/arvbox/docker/waitforpostgres.sh b/arvbox/lib/arvbox/docker/waitforpostgres.sh
new file mode 100755
index 0000000..a07fa8c
--- /dev/null
+++ b/arvbox/lib/arvbox/docker/waitforpostgres.sh
@@ -0,0 +1,4 @@
+#!/bin/sh
+while ! psql -c\\du >/dev/null 2>/dev/null ; do
+ sleep 1
+done
commit 846a9311a328b9cbadc8b70d34134b72b0b6b332
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Jan 18 15:33:03 2016 -0500
8080: fpm_depends comes before run-build-package commands; fpm_args comes after
run-build package commands. Improve quoting. Use sed -i.
diff --git a/jenkins/package-build-dockerfiles/centos6/Dockerfile b/jenkins/package-build-dockerfiles/centos6/Dockerfile
index 51bad92..2bbec2e 100644
--- a/jenkins/package-build-dockerfiles/centos6/Dockerfile
+++ b/jenkins/package-build-dockerfiles/centos6/Dockerfile
@@ -23,8 +23,7 @@ RUN scl enable python33 "easy_install-3.3 pip" && scl enable python27 "easy_inst
RUN cd /tmp && \
curl -OL 'http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm' && \
rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm && \
- cat /etc/yum.repos.d/rpmforge.repo | sed 's/enabled = 0/enabled = 1/' > /tmp/rpmforge.repo && \
- mv /tmp/rpmforge.repo /etc/yum.repos.d
+ sed -i 's/enabled = 0/enabled = 1/' /etc/yum.repos.d/rpmforge.repo
RUN touch /var/lib/rpm/* && yum install --assumeyes git
diff --git a/jenkins/package-test-dockerfiles/centos6/Dockerfile b/jenkins/package-test-dockerfiles/centos6/Dockerfile
index 95d8157..69927a1 100644
--- a/jenkins/package-test-dockerfiles/centos6/Dockerfile
+++ b/jenkins/package-test-dockerfiles/centos6/Dockerfile
@@ -15,7 +15,6 @@ RUN touch /var/lib/rpm/* && \
RUN cd /tmp && \
curl -OL 'http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm' && \
rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm && \
- cat /etc/yum.repos.d/rpmforge.repo | sed 's/enabled = 0/enabled = 1/' > /tmp/rpmforge.repo && \
- mv /tmp/rpmforge.repo /etc/yum.repos.d
+ sed -i 's/enabled = 0/enabled = 1/' /etc/yum.repos.d/rpmforge.repo
COPY localrepo.repo /etc/yum.repos.d/localrepo.repo
\ No newline at end of file
diff --git a/jenkins/package-testing/deb-common-test-packages.sh b/jenkins/package-testing/deb-common-test-packages.sh
index 016ab5e..0d15436 100755
--- a/jenkins/package-testing/deb-common-test-packages.sh
+++ b/jenkins/package-testing/deb-common-test-packages.sh
@@ -8,15 +8,15 @@ target=$(echo $0 | sed 's/.*test-packages-\([^.]*\)\.sh.*/\1/')
export ARV_PACKAGES_DIR=/arvados/packages/$target
-dpkg-query --show > $ARV_PACKAGES_DIR/$1.before
+dpkg-query --show > "$ARV_PACKAGES_DIR/$1.before"
apt-get -qq update
apt-get --assume-yes --force-yes install $1
-dpkg-query --show > $ARV_PACKAGES_DIR/$1.after
+dpkg-query --show > "$ARV_PACKAGES_DIR/$1.after"
set +e
-diff $ARV_PACKAGES_DIR/$1.before $ARV_PACKAGES_DIR/$1.after > $ARV_PACKAGES_DIR/$1.diff
+diff "$ARV_PACKAGES_DIR/$1.before" "$ARV_PACKAGES_DIR/$1.after" > "$ARV_PACKAGES_DIR/$1.diff"
set -e
mkdir -p /tmp/opts
@@ -26,10 +26,12 @@ export ARV_PACKAGES_DIR=/arvados/packages/$target
dpkg-deb -x $ARV_PACKAGES_DIR/"$1"_*.deb .
-for so in $(find . -name "*.so") ; do
+while read so && [ -n "$so" ]; do
echo
echo "== Packages dependencies for $so =="
- ldd $so | awk '($3 ~ /^\//){print $3}' | sort -u | xargs dpkg -S | cut -d: -f1 | sort -u
-done
+ ldd "$so" | awk '($3 ~ /^\//){print $3}' | sort -u | xargs dpkg -S | cut -d: -f1 | sort -u
+done <<EOF
+$(find -name '*.so')
+EOF
exec /jenkins/package-testing/common-test-packages.sh $1
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index e9f75a4..30ba4cd 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -264,7 +264,7 @@ handle_python_package
# arvados-src
(
- set -ex
+ set -e
cd "$WORKSPACE"
COMMIT_HASH=$(format_last_commit_here "%H")
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index dce75eb..3c62d7b 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -250,13 +250,14 @@ fpm_build () {
for i in "${fpm_depends[@]}"; do
COMMAND_ARR+=('--depends' "$i")
done
- COMMAND_ARR+=("${fpm_args[@]}")
# Append remaining function arguments directly to fpm's command line.
for i; do
COMMAND_ARR+=("$i")
done
+ COMMAND_ARR+=("${fpm_args[@]}")
+
COMMAND_ARR+=("$PACKAGE")
debug_echo -e "\n${COMMAND_ARR[@]}\n"
commit f44f65a25c1d505e7faa9e26351525c10bb2cbbd
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Jan 18 14:43:01 2016 -0500
8008: Move test files into their own directories. Fix rails package tests to
work with debian & ubuntu. Fix error code checking.
diff --git a/jenkins/common-test-packages.sh b/jenkins/common-test-packages.sh
deleted file mode 100755
index 1ff8707..0000000
--- a/jenkins/common-test-packages.sh
+++ /dev/null
@@ -1,28 +0,0 @@
-#!/bin/sh
-
-set -eu
-
-FAIL=0
-
-echo
-shared=$(find -name '*.so')
-if test -n "$shared" ; then
- for so in $shared ; do
- if ldd $so | grep "not found" ; then
- echo "^^^ Missing while scanning $so ^^^"
- FAIL=1
- fi
- done
-fi
-
-if test -x /jenkins/test-package-$1.sh ; then
- if ! /jenkins/test-package-$1.sh ; then
- FAIL=1
- fi
-fi
-
-if test $FAIL = 0 ; then
- echo "Package $1 passed"
-fi
-
-exit $FAIL
diff --git a/jenkins/dockerfiles/.gitignore b/jenkins/package-build-dockerfiles/.gitignore
similarity index 100%
rename from jenkins/dockerfiles/.gitignore
rename to jenkins/package-build-dockerfiles/.gitignore
diff --git a/jenkins/dockerfiles/Makefile b/jenkins/package-build-dockerfiles/Makefile
similarity index 100%
rename from jenkins/dockerfiles/Makefile
rename to jenkins/package-build-dockerfiles/Makefile
diff --git a/jenkins/dockerfiles/README b/jenkins/package-build-dockerfiles/README
similarity index 100%
rename from jenkins/dockerfiles/README
rename to jenkins/package-build-dockerfiles/README
diff --git a/jenkins/dockerfiles/build-all-build-containers.sh b/jenkins/package-build-dockerfiles/build-all-build-containers.sh
similarity index 100%
rename from jenkins/dockerfiles/build-all-build-containers.sh
rename to jenkins/package-build-dockerfiles/build-all-build-containers.sh
diff --git a/jenkins/dockerfiles/centos6/Dockerfile b/jenkins/package-build-dockerfiles/centos6/Dockerfile
similarity index 93%
rename from jenkins/dockerfiles/centos6/Dockerfile
rename to jenkins/package-build-dockerfiles/centos6/Dockerfile
index 68c1c74..51bad92 100644
--- a/jenkins/dockerfiles/centos6/Dockerfile
+++ b/jenkins/package-build-dockerfiles/centos6/Dockerfile
@@ -20,12 +20,13 @@ RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
RUN touch /var/lib/rpm/* && yum -q -y install python27 python33
RUN scl enable python33 "easy_install-3.3 pip" && scl enable python27 "easy_install-2.7 pip"
-RUN cd / root && \
+RUN cd /tmp && \
curl -OL 'http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm' && \
rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm && \
cat /etc/yum.repos.d/rpmforge.repo | sed 's/enabled = 0/enabled = 1/' > /tmp/rpmforge.repo && \
- mv /tmp/rpmforge.repo /etc/yum.repos.d && \
- yum install --assumeyes git
+ mv /tmp/rpmforge.repo /etc/yum.repos.d
+
+RUN touch /var/lib/rpm/* && yum install --assumeyes git
ENV WORKSPACE /arvados
CMD ["scl", "enable", "python33", "python27", "/usr/local/rvm/bin/rvm-exec default bash /jenkins/run-build-packages.sh --target centos6"]
diff --git a/jenkins/dockerfiles/debian7/Dockerfile b/jenkins/package-build-dockerfiles/debian7/Dockerfile
similarity index 100%
rename from jenkins/dockerfiles/debian7/Dockerfile
rename to jenkins/package-build-dockerfiles/debian7/Dockerfile
diff --git a/jenkins/dockerfiles/debian8/Dockerfile b/jenkins/package-build-dockerfiles/debian8/Dockerfile
similarity index 100%
rename from jenkins/dockerfiles/debian8/Dockerfile
rename to jenkins/package-build-dockerfiles/debian8/Dockerfile
diff --git a/jenkins/dockerfiles/ubuntu1204/Dockerfile b/jenkins/package-build-dockerfiles/ubuntu1204/Dockerfile
similarity index 100%
rename from jenkins/dockerfiles/ubuntu1204/Dockerfile
rename to jenkins/package-build-dockerfiles/ubuntu1204/Dockerfile
diff --git a/jenkins/dockerfiles/ubuntu1404/Dockerfile b/jenkins/package-build-dockerfiles/ubuntu1404/Dockerfile
similarity index 100%
rename from jenkins/dockerfiles/ubuntu1404/Dockerfile
rename to jenkins/package-build-dockerfiles/ubuntu1404/Dockerfile
diff --git a/jenkins/package-test-dockerfiles/centos6/Dockerfile b/jenkins/package-test-dockerfiles/centos6/Dockerfile
index 5d610c0..95d8157 100644
--- a/jenkins/package-test-dockerfiles/centos6/Dockerfile
+++ b/jenkins/package-test-dockerfiles/centos6/Dockerfile
@@ -12,7 +12,7 @@ RUN touch /var/lib/rpm/* && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
/usr/local/rvm/bin/rvm-exec default gem install bundle fpm
-RUN cd / root && \
+RUN cd /tmp && \
curl -OL 'http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm' && \
rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm && \
cat /etc/yum.repos.d/rpmforge.repo | sed 's/enabled = 0/enabled = 1/' > /tmp/rpmforge.repo && \
diff --git a/jenkins/package-testing/common-test-packages.sh b/jenkins/package-testing/common-test-packages.sh
new file mode 100755
index 0000000..2dc67ab
--- /dev/null
+++ b/jenkins/package-testing/common-test-packages.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+
+set -eu
+
+FAIL=0
+
+echo
+
+while read so && [ -n "$so" ]; do
+ if ldd "$so" | grep "not found" ; then
+ echo "^^^ Missing while scanning $so ^^^"
+ FAIL=1
+ fi
+done <<EOF
+$(find -name '*.so')
+EOF
+
+if test -x "/jenkins/package-testing/test-package-$1.sh" ; then
+ if ! "/jenkins/package-testing/test-package-$1.sh" ; then
+ FAIL=1
+ fi
+fi
+
+if test $FAIL = 0 ; then
+ echo "Package $1 passed"
+fi
+
+exit $FAIL
diff --git a/jenkins/deb-common-test-packages.sh b/jenkins/package-testing/deb-common-test-packages.sh
similarity index 65%
rename from jenkins/deb-common-test-packages.sh
rename to jenkins/package-testing/deb-common-test-packages.sh
index 1b52fc1..016ab5e 100755
--- a/jenkins/deb-common-test-packages.sh
+++ b/jenkins/package-testing/deb-common-test-packages.sh
@@ -6,9 +6,19 @@ set -eu
# from the invocation path.
target=$(echo $0 | sed 's/.*test-packages-\([^.]*\)\.sh.*/\1/')
+export ARV_PACKAGES_DIR=/arvados/packages/$target
+
+dpkg-query --show > $ARV_PACKAGES_DIR/$1.before
+
apt-get -qq update
apt-get --assume-yes --force-yes install $1
+dpkg-query --show > $ARV_PACKAGES_DIR/$1.after
+
+set +e
+diff $ARV_PACKAGES_DIR/$1.before $ARV_PACKAGES_DIR/$1.after > $ARV_PACKAGES_DIR/$1.diff
+set -e
+
mkdir -p /tmp/opts
cd /tmp/opts
@@ -22,4 +32,4 @@ for so in $(find . -name "*.so") ; do
ldd $so | awk '($3 ~ /^\//){print $3}' | sort -u | xargs dpkg -S | cut -d: -f1 | sort -u
done
-exec /jenkins/common-test-packages.sh $1
+exec /jenkins/package-testing/common-test-packages.sh $1
diff --git a/jenkins/package-testing/test-package-arvados-api-server.sh b/jenkins/package-testing/test-package-arvados-api-server.sh
new file mode 100755
index 0000000..64494aa
--- /dev/null
+++ b/jenkins/package-testing/test-package-arvados-api-server.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+set -e
+cd /var/www/arvados-api/current/
+
+case "$TARGET" in
+ debian*|ubuntu*)
+ apt-get install -y nginx
+ dpkg-reconfigure arvados-api-server
+ ;;
+ centos6)
+ yum install --assumeyes httpd
+ yum reinstall --assumeyes arvados-api-server
+ ;;
+ *)
+ echo -e "$0: Unknown target '$TARGET'.\n" >&2
+ exit 1
+ ;;
+esac
+
+/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arados-api-server.gems
diff --git a/jenkins/test-package-arvados-node-manager.sh b/jenkins/package-testing/test-package-arvados-node-manager.sh
similarity index 75%
rename from jenkins/test-package-arvados-node-manager.sh
rename to jenkins/package-testing/test-package-arvados-node-manager.sh
index c6760f8..2f416d1 100755
--- a/jenkins/test-package-arvados-node-manager.sh
+++ b/jenkins/package-testing/test-package-arvados-node-manager.sh
@@ -3,5 +3,5 @@ exec python <<EOF
import libcloud.compute.types
import libcloud.compute.providers
libcloud.compute.providers.get_driver(libcloud.compute.types.Provider.AZURE_ARM)
-print "Successly imported compatible libcloud library"
+print "Successfully imported compatible libcloud library"
EOF
diff --git a/jenkins/test-package-arvados-sso-server.sh b/jenkins/package-testing/test-package-arvados-sso-server.sh
similarity index 87%
rename from jenkins/test-package-arvados-sso-server.sh
rename to jenkins/package-testing/test-package-arvados-sso-server.sh
index 7f1221e..28c5618 100755
--- a/jenkins/test-package-arvados-sso-server.sh
+++ b/jenkins/package-testing/test-package-arvados-sso-server.sh
@@ -76,7 +76,7 @@ if [[ ! -e "/etc/arvados/sso/database.yml" ]]; then
# We need to set up our database configuration now.
if [[ "$FORMAT" == "rpm" ]]; then
# postgres packaging on CentOS6 is kind of primitive, needs an initdb
- $SUDO service postgresql initdb
+ service postgresql initdb
if [ "$TARGET" = "centos6" ]; then
sed -i -e "s/127.0.0.1\/32 ident/127.0.0.1\/32 md5/" /var/lib/pgsql/data/pg_hba.conf
sed -i -e "s/::1\/128 ident/::1\/128 md5/" /var/lib/pgsql/data/pg_hba.conf
@@ -107,12 +107,12 @@ if [[ "$FORMAT" == "deb" ]]; then
/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arvados-sso-server.gems
# Test 3: the package should remove cleanly
- $SUDO apt-get remove arvados-sso-server --yes || EXITCODE=3
+ apt-get remove arvados-sso-server --yes || EXITCODE=3
checkexit $EXITCODE "apt-get remove arvados-sso-server --yes"
# Test 4: the package configuration should remove cleanly
- $SUDO dpkg --purge arvados-sso-server || EXITCODE=4
+ dpkg --purge arvados-sso-server || EXITCODE=4
checkexit $EXITCODE "dpkg --purge arvados-sso-server"
@@ -124,7 +124,7 @@ if [[ "$FORMAT" == "deb" ]]; then
# Test 5: the package should remove cleanly with --purge
- $SUDO apt-get remove arvados-sso-server --purge --yes || EXITCODE=5
+ apt-get remove arvados-sso-server --purge --yes || EXITCODE=5
checkexit $EXITCODE "apt-get remove arvados-sso-server --purge --yes"
@@ -138,10 +138,10 @@ elif [[ "$FORMAT" == "rpm" ]]; then
# Set up Nginx first
# (courtesy of https://www.phusionpassenger.com/library/walkthroughs/deploy/ruby/ownserver/nginx/oss/el6/install_passenger.html)
- $SUDO yum install -q -y epel-release pygpgme curl
- $SUDO curl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo
- $SUDO yum install -q -y nginx passenger
- $SUDO sed -i -e 's/^# passenger/passenger/' /etc/nginx/conf.d/passenger.conf
+ yum install -q -y epel-release pygpgme curl
+ curl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo
+ yum install -q -y nginx passenger
+ sed -i -e 's/^# passenger/passenger/' /etc/nginx/conf.d/passenger.conf
# Done setting up Nginx
# Test 2: the package should reinstall cleanly
@@ -151,7 +151,7 @@ elif [[ "$FORMAT" == "rpm" ]]; then
/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arvados-sso-server.gems
# Test 3: the package should remove cleanly
- $SUDO yum -q -y remove arvados-sso-server || EXITCODE=3
+ yum -q -y remove arvados-sso-server || EXITCODE=3
checkexit $EXITCODE "yum -q -y remove arvados-sso-server"
diff --git a/jenkins/package-testing/test-package-arvados-workbench.sh b/jenkins/package-testing/test-package-arvados-workbench.sh
new file mode 100755
index 0000000..e255786
--- /dev/null
+++ b/jenkins/package-testing/test-package-arvados-workbench.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+set -e
+cd /var/www/arvados-workbench/current/
+
+case "$TARGET" in
+ debian*|ubuntu*)
+ apt-get install -y nginx
+ dpkg-reconfigure arvados-workbench
+ ;;
+ centos6)
+ yum install --assumeyes httpd
+ yum reinstall --assumeyes arvados-workbench
+ ;;
+ *)
+ echo -e "$0: Unknown target '$TARGET'.\n" >&2
+ exit 1
+ ;;
+esac
+
+/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arvados-workbench.gems
diff --git a/jenkins/test-package-python27-python-arvados-fuse.sh b/jenkins/package-testing/test-package-python27-python-arvados-fuse.sh
similarity index 55%
rename from jenkins/test-package-python27-python-arvados-fuse.sh
rename to jenkins/package-testing/test-package-python27-python-arvados-fuse.sh
index d2a1072..1654be9 100755
--- a/jenkins/test-package-python27-python-arvados-fuse.sh
+++ b/jenkins/package-testing/test-package-python27-python-arvados-fuse.sh
@@ -2,5 +2,5 @@
exec python <<EOF
import arvados_fuse
-print "Successly imported arvados_fuse"
+print "Successfully imported arvados_fuse"
EOF
diff --git a/jenkins/test-package-python27-python-arvados-python-client.sh b/jenkins/package-testing/test-package-python27-python-arvados-python-client.sh
similarity index 55%
rename from jenkins/test-package-python27-python-arvados-python-client.sh
rename to jenkins/package-testing/test-package-python27-python-arvados-python-client.sh
index c5abddd..0772fbf 100755
--- a/jenkins/test-package-python27-python-arvados-python-client.sh
+++ b/jenkins/package-testing/test-package-python27-python-arvados-python-client.sh
@@ -2,5 +2,5 @@
exec python <<EOF
import arvados
-print "Successly imported arvados"
+print "Successfully imported arvados"
EOF
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/package-testing/test-packages-centos6.sh
similarity index 86%
rename from jenkins/test-packages-centos6.sh
rename to jenkins/package-testing/test-packages-centos6.sh
index 29ad987..ad80564 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/package-testing/test-packages-centos6.sh
@@ -38,7 +38,7 @@ if test -n "$shared" ; then
fi
if test -n "$SCL" ; then
- exec $SCL "/jenkins/common-test-packages.sh $1"
+ exec $SCL "/jenkins/package-testing/common-test-packages.sh $1"
else
- exec /jenkins/common-test-packages.sh $1
+ exec /jenkins/package-testing/common-test-packages.sh $1
fi
diff --git a/jenkins/test-packages-debian7.sh b/jenkins/package-testing/test-packages-debian7.sh
similarity index 100%
rename from jenkins/test-packages-debian7.sh
rename to jenkins/package-testing/test-packages-debian7.sh
diff --git a/jenkins/test-packages-debian8.sh b/jenkins/package-testing/test-packages-debian8.sh
similarity index 100%
rename from jenkins/test-packages-debian8.sh
rename to jenkins/package-testing/test-packages-debian8.sh
diff --git a/jenkins/test-packages-ubuntu1204.sh b/jenkins/package-testing/test-packages-ubuntu1204.sh
similarity index 100%
rename from jenkins/test-packages-ubuntu1204.sh
rename to jenkins/package-testing/test-packages-ubuntu1204.sh
diff --git a/jenkins/test-packages-ubuntu1404.sh b/jenkins/package-testing/test-packages-ubuntu1404.sh
similarity index 100%
rename from jenkins/test-packages-ubuntu1404.sh
rename to jenkins/package-testing/test-packages-ubuntu1404.sh
diff --git a/jenkins/run-build-packages-all-targets.sh b/jenkins/run-build-packages-all-targets.sh
index 5cd5e74..64c1262 100755
--- a/jenkins/run-build-packages-all-targets.sh
+++ b/jenkins/run-build-packages-all-targets.sh
@@ -81,6 +81,19 @@ done
cd $(dirname $0)
+FINAL_EXITCODE=0
+
for dockerfile_path in $(find -name Dockerfile); do
./run-build-packages-one-target.sh --target "$(basename $(dirname "$dockerfile_path"))" --command "$COMMAND" $DEBUG $TEST_PACKAGES $ONLY_TEST
+
+ CODE=$?
+ if test $CODE != 0 ; then
+ FINAL_EXITCODE=$CODE
+ fi
done
+
+if test $FINAL_EXITCODE != 0 ; then
+ echo "Build packages failed with code $FINAL_EXITCODE" >&2
+fi
+
+exit $FINAL_EXITCODE
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index 0233de1..ebfc0be 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -96,7 +96,7 @@ if [[ -n "$test_packages" ]]; then
)
fi
- COMMAND="/jenkins/test-packages-$TARGET.sh"
+ COMMAND="/jenkins/package-testing/test-packages-$TARGET.sh"
IMAGE="arvados/package-test:$TARGET"
else
IMAGE="arvados/build:$TARGET"
@@ -110,7 +110,7 @@ JENKINS_DIR=$(dirname "$(readlink -e "$0")")
if [[ -n "$test_packages" ]]; then
pushd "$JENKINS_DIR/package-test-dockerfiles"
else
- pushd "$JENKINS_DIR/dockerfiles"
+ pushd "$JENKINS_DIR/package-build-dockerfiles"
fi
echo $TARGET
@@ -148,35 +148,37 @@ fi
FINAL_EXITCODE=0
package_fails=""
-set -x
+set +e
if [[ -n "$test_packages" ]]; then
for p in $packages ; do
- set +e
docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
--env ARVADOS_DEBUG=1 \
--env "TARGET=$TARGET" \
--env "WORKSPACE=/arvados" \
"$IMAGE" $COMMAND $p
- if test $? != 0 ; then
- FINAL_EXITCODE=$?
+ CODE=$?
+ if test $CODE != 0 ; then
+ FINAL_EXITCODE=$CODE
package_fails="$package_fails $p"
- echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
+ echo "ERROR: $tag test failed with exit status $FINAL_EXITCODE." >&2
fi
done
else
- if docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
- --env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND; then
- # Success - nothing more to do.
- true
+ docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
+ --env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND
+ CODE=$?
+ if test $CODE = 0 ; then
+ echo
+ echo "Build packages for $TARGET succeeded." >&2
else
- FINAL_EXITCODE=$?
+ FINAL_EXITCODE=$CODE
echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
fi
fi
if test -n "$package_fails" ; then
- echo "Failed package tests:$package_fails"
+ echo "Failed package tests:$package_fails" >&2
fi
exit $FINAL_EXITCODE
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index 6324066..cc673a6 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -149,7 +149,7 @@ umask 0022
debug_echo "umask is" `umask`
if [[ ! -d "$WORKSPACE/packages/$TARGET" ]]; then
- mkdir -p $WORKSPACE/packages/$TARGET
+ mkdir -p "$WORKSPACE/packages/$TARGET"
fi
# Build the SSO server package
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 445b3a0..e9f75a4 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -264,7 +264,7 @@ handle_python_package
# arvados-src
(
- set -e
+ set -ex
cd "$WORKSPACE"
COMMIT_HASH=$(format_last_commit_here "%H")
@@ -285,6 +285,7 @@ handle_python_package
fpm_build $SRC_BUILD_DIR/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
rm -rf "$SRC_BUILD_DIR"
+ echo $?
)
# Go binaries
diff --git a/jenkins/test-package-arvados-api-server.sh b/jenkins/test-package-arvados-api-server.sh
deleted file mode 100755
index 467872b..0000000
--- a/jenkins/test-package-arvados-api-server.sh
+++ /dev/null
@@ -1,6 +0,0 @@
-#!/bin/sh
-set -e
-cd /var/www/arvados-api/current/
-yum install --assumeyes httpd
-yum reinstall --assumeyes arvados-api-server
-/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arados-api-server.gems
diff --git a/jenkins/test-package-arvados-workbench.sh b/jenkins/test-package-arvados-workbench.sh
deleted file mode 100755
index 7111b94..0000000
--- a/jenkins/test-package-arvados-workbench.sh
+++ /dev/null
@@ -1,6 +0,0 @@
-#!/bin/sh
-set -e
-cd /var/www/arvados-workbench/current/
-yum install --assumeyes httpd
-yum reinstall --assumeyes arvados-workbench
-/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arvados-workbench.gems
commit a956a49875fc484c2c237aec228a92f8ce5b0f06
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jan 14 16:40:59 2016 -0500
8008: SSO package tests pass for all targets.
diff --git a/jenkins/package-test-dockerfiles/debian7/Dockerfile b/jenkins/package-test-dockerfiles/debian7/Dockerfile
index 9fcf3c6..3277e60 100644
--- a/jenkins/package-test-dockerfiles/debian7/Dockerfile
+++ b/jenkins/package-test-dockerfiles/debian7/Dockerfile
@@ -2,10 +2,10 @@ FROM debian:7
MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
# Install RVM
-RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+RUN apt-get update && apt-get -y install curl procps && \
+ gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
- /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
- /usr/local/rvm/bin/rvm-exec default gem install fpm
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1
RUN echo "deb file:///arvados/packages/debian7/ /" >>/etc/apt/sources.list
diff --git a/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile b/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
index 2908515..80573bf 100644
--- a/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
+++ b/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
@@ -1,12 +1,11 @@
FROM ubuntu:precise
MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
-
# Install RVM
-RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+RUN apt-get update && apt-get -y install curl && \
+ gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
- /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
- /usr/local/rvm/bin/rvm-exec default gem install fpm
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1
RUN echo "deb file:///arvados/packages/ubuntu1204/ /" >>/etc/apt/sources.list
\ No newline at end of file
diff --git a/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile b/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
index 1bd06b3..2d8ebe9 100644
--- a/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
+++ b/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
@@ -1,12 +1,11 @@
FROM ubuntu:trusty
MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
-
# Install RVM
-RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+RUN apt-get update && apt-get -y install curl && \
+ gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
- /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
- /usr/local/rvm/bin/rvm-exec default gem install fpm
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1
RUN echo "deb file:///arvados/packages/ubuntu1404/ /" >>/etc/apt/sources.list
\ No newline at end of file
diff --git a/jenkins/run-build-packages-all-targets.sh b/jenkins/run-build-packages-all-targets.sh
index 654d551..5cd5e74 100755
--- a/jenkins/run-build-packages-all-targets.sh
+++ b/jenkins/run-build-packages-all-targets.sh
@@ -38,7 +38,7 @@ fi
set -e
PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,test-packages,debug,command: \
+ help,test-packages,debug,command:,only-test: \
-- "" "$@")
if [ $? -ne 0 ]; then
exit 1
@@ -47,6 +47,7 @@ fi
COMMAND=
DEBUG=
TEST_PACKAGES=
+ONLY_TEST=
eval set -- "$PARSEDOPTS"
while [ $# -gt 0 ]; do
@@ -65,6 +66,9 @@ while [ $# -gt 0 ]; do
--test-packages)
TEST_PACKAGES="--test-packages"
;;
+ --only-test)
+ ONLY_TEST="$1 $2"; shift
+ ;;
--)
if [ $# -gt 1 ]; then
echo >&2 "$0: unrecognized argument '$2'. Try: $0 --help"
@@ -78,5 +82,5 @@ done
cd $(dirname $0)
for dockerfile_path in $(find -name Dockerfile); do
- ./run-build-packages-one-target.sh --target "$(basename $(dirname "$dockerfile_path"))" --command "$COMMAND" $DEBUG $TEST_PACKAGES
+ ./run-build-packages-one-target.sh --target "$(basename $(dirname "$dockerfile_path"))" --command "$COMMAND" $DEBUG $TEST_PACKAGES $ONLY_TEST
done
commit f84fd5c8f2292a41bbb4cd762dbedab09adbf784
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jan 14 15:45:18 2016 -0500
8008: Incorporate SSO package testing into general package testing framework.
diff --git a/jenkins/dockerfiles/centos6/Dockerfile b/jenkins/dockerfiles/centos6/Dockerfile
index 68134c4..68c1c74 100644
--- a/jenkins/dockerfiles/centos6/Dockerfile
+++ b/jenkins/dockerfiles/centos6/Dockerfile
@@ -20,5 +20,12 @@ RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
RUN touch /var/lib/rpm/* && yum -q -y install python27 python33
RUN scl enable python33 "easy_install-3.3 pip" && scl enable python27 "easy_install-2.7 pip"
+RUN cd / root && \
+ curl -OL 'http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm' && \
+ rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm && \
+ cat /etc/yum.repos.d/rpmforge.repo | sed 's/enabled = 0/enabled = 1/' > /tmp/rpmforge.repo && \
+ mv /tmp/rpmforge.repo /etc/yum.repos.d && \
+ yum install --assumeyes git
+
ENV WORKSPACE /arvados
CMD ["scl", "enable", "python33", "python27", "/usr/local/rvm/bin/rvm-exec default bash /jenkins/run-build-packages.sh --target centos6"]
diff --git a/jenkins/package-test-dockerfiles/centos6/Dockerfile b/jenkins/package-test-dockerfiles/centos6/Dockerfile
index 6870ec3..5d610c0 100644
--- a/jenkins/package-test-dockerfiles/centos6/Dockerfile
+++ b/jenkins/package-test-dockerfiles/centos6/Dockerfile
@@ -14,6 +14,8 @@ RUN touch /var/lib/rpm/* && \
RUN cd / root && \
curl -OL 'http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm' && \
- rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
+ rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm && \
+ cat /etc/yum.repos.d/rpmforge.repo | sed 's/enabled = 0/enabled = 1/' > /tmp/rpmforge.repo && \
+ mv /tmp/rpmforge.repo /etc/yum.repos.d
COPY localrepo.repo /etc/yum.repos.d/localrepo.repo
\ No newline at end of file
diff --git a/jenkins/package-test-dockerfiles/debian7/Dockerfile b/jenkins/package-test-dockerfiles/debian7/Dockerfile
index fb789c6..9fcf3c6 100644
--- a/jenkins/package-test-dockerfiles/debian7/Dockerfile
+++ b/jenkins/package-test-dockerfiles/debian7/Dockerfile
@@ -1,4 +1,11 @@
FROM debian:7
MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
+# Install RVM
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+ curl -L https://get.rvm.io | bash -s stable && \
+ /usr/local/rvm/bin/rvm install 2.1 && \
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
+ /usr/local/rvm/bin/rvm-exec default gem install fpm
+
RUN echo "deb file:///arvados/packages/debian7/ /" >>/etc/apt/sources.list
diff --git a/jenkins/package-test-dockerfiles/debian8/Dockerfile b/jenkins/package-test-dockerfiles/debian8/Dockerfile
index fbcd8f4..01abbc9 100644
--- a/jenkins/package-test-dockerfiles/debian8/Dockerfile
+++ b/jenkins/package-test-dockerfiles/debian8/Dockerfile
@@ -1,4 +1,11 @@
FROM debian:8
MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
-RUN echo "deb file:///arvados/packages/debian8/ /" >>/etc/apt/sources.list
\ No newline at end of file
+# Install RVM
+RUN apt-get update && apt-get -y install curl && \
+ gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+ curl -L https://get.rvm.io | bash -s stable && \
+ /usr/local/rvm/bin/rvm install 2.1 && \
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1
+
+RUN echo "deb file:///arvados/packages/debian8/ /" >>/etc/apt/sources.list
diff --git a/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile b/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
index 4b6e936..2908515 100644
--- a/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
+++ b/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
@@ -1,4 +1,12 @@
FROM ubuntu:precise
MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
+
+# Install RVM
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+ curl -L https://get.rvm.io | bash -s stable && \
+ /usr/local/rvm/bin/rvm install 2.1 && \
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
+ /usr/local/rvm/bin/rvm-exec default gem install fpm
+
RUN echo "deb file:///arvados/packages/ubuntu1204/ /" >>/etc/apt/sources.list
\ No newline at end of file
diff --git a/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile b/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
index 4f49d92..1bd06b3 100644
--- a/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
+++ b/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
@@ -1,4 +1,12 @@
FROM ubuntu:trusty
MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
+
+# Install RVM
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+ curl -L https://get.rvm.io | bash -s stable && \
+ /usr/local/rvm/bin/rvm install 2.1 && \
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
+ /usr/local/rvm/bin/rvm-exec default gem install fpm
+
RUN echo "deb file:///arvados/packages/ubuntu1404/ /" >>/etc/apt/sources.list
\ No newline at end of file
diff --git a/jenkins/rails-package-scripts/postinst.sh b/jenkins/rails-package-scripts/postinst.sh
index 1032d14..2feaf13 100644
--- a/jenkins/rails-package-scripts/postinst.sh
+++ b/jenkins/rails-package-scripts/postinst.sh
@@ -194,7 +194,7 @@ configure_version() {
fi
run_and_report "Running bundle install" \
- $COMMAND_PREFIX bundle install --path $SHARED_PATH/vendor_bundle --quiet
+ $COMMAND_PREFIX bundle install --path $SHARED_PATH/vendor_bundle --local --quiet
echo -n "Ensuring directory and file permissions ..."
# Ensure correct ownership of a few files
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index af6e078..0233de1 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -133,23 +133,32 @@ if test -z "$packages" ; then
keep-web
libarvados-perl"
- if test "$target" = centos6 ; then
- packages="$packages python27-python-arvados-fuse
+ case "$TARGET" in
+ centos6)
+ packages="$packages python27-python-arvados-fuse
python27-python-arvados-python-client"
- else
- packages="$packages python-arvados-fuse
+ ;;
+ *)
+ packages="$packages python-arvados-fuse
python-arvados-python-client"
- fi
+ ;;
+ esac
fi
FINAL_EXITCODE=0
package_fails=""
+set -x
if [[ -n "$test_packages" ]]; then
for p in $packages ; do
- if ! docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
- --env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND $p ; then
+ set +e
+ docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
+ --env ARVADOS_DEBUG=1 \
+ --env "TARGET=$TARGET" \
+ --env "WORKSPACE=/arvados" \
+ "$IMAGE" $COMMAND $p
+ if test $? != 0 ; then
FINAL_EXITCODE=$?
package_fails="$package_fails $p"
echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index eedb77b..6324066 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -1,6 +1,7 @@
#!/bin/bash
-. `dirname "$(readlink -f "$0")"`/run-library.sh
+JENKINS_DIR=$(dirname $(readlink -e "$0"))
+. "$JENKINS_DIR/run-library.sh"
read -rd "\000" helpmessage <<EOF
$(basename $0): Build Arvados SSO server package
@@ -10,8 +11,6 @@ Syntax:
Options:
---build-bundle-packages (default: false)
- Build package with vendor/bundle included
--debug
Output debug information (default: false)
--target
@@ -23,7 +22,6 @@ EOF
EXITCODE=0
DEBUG=${ARVADOS_DEBUG:-0}
-BUILD_BUNDLE_PACKAGES=0
TARGET=debian7
PARSEDOPTS=$(getopt --name "$0" --longoptions \
@@ -47,8 +45,8 @@ while [ $# -gt 0 ]; do
--debug)
DEBUG=1
;;
- --build-bundle-packages)
- BUILD_BUNDLE_PACKAGES=1
+ --test-packages)
+ test_packages=1
;;
--)
if [ $# -gt 1 ]; then
@@ -111,19 +109,19 @@ fi
fpm --version >/dev/null 2>&1
if [[ "$?" != 0 ]]; then
- echo >&2 "$helpmessage"
- echo >&2
- echo >&2 "Error: fpm not found"
- echo >&2
- exit 1
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: fpm not found"
+ echo >&2
+ exit 1
fi
RUN_BUILD_PACKAGES_PATH="`dirname \"$0\"`"
RUN_BUILD_PACKAGES_PATH="`( cd \"$RUN_BUILD_PACKAGES_PATH\" && pwd )`" # absolutized and normalized
if [ -z "$RUN_BUILD_PACKAGES_PATH" ] ; then
- # error; for some reason, the path is not accessible
- # to the script (e.g. permissions re-evaled after suid)
- exit 1 # fail
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
fi
debug_echo "$0 is running from $RUN_BUILD_PACKAGES_PATH"
@@ -136,20 +134,6 @@ else
GEM=gem
fi
-if [[ "$TARGET" == "centos6" ]]; then
- # CentOS6 comes with git 1.7.1, but we want at least 1.7.6
- # because we use git status --ignore in fpm-info.sh
- cd /usr/src
- install_package libcurl-devel zlib-devel wget gettext
- wget https://www.kernel.org/pub/software/scm/git/git-1.8.5.6.tar.gz
- tar xzf git-1.8.5.6.tar.gz
- cd git-1.8.5.6
- make configure
- ./configure --prefix=/usr --without-tcltk
- make all
- make install
-fi
-
# Make all files world-readable -- jenkins runs with umask 027, and has checked
# out our git tree here
chmod o+r "$WORKSPACE" -R
@@ -165,13 +149,13 @@ umask 0022
debug_echo "umask is" `umask`
if [[ ! -d "$WORKSPACE/packages/$TARGET" ]]; then
- mkdir -p $WORKSPACE/packages/$TARGET
+ mkdir -p $WORKSPACE/packages/$TARGET
fi
# Build the SSO server package
handle_rails_package arvados-sso-server "$WORKSPACE" \
- "$WORKSPACE/LICENCE" --url="https://arvados.org" \
- --description="Arvados SSO server - Arvados is a free and open source platform for big data science." \
- --license="Expat license"
+ "$WORKSPACE/LICENCE" --url="https://arvados.org" \
+ --description="Arvados SSO server - Arvados is a free and open source platform for big data science." \
+ --license="Expat license"
exit $EXITCODE
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 26486e9..445b3a0 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -27,7 +27,6 @@ EOF
EXITCODE=0
DEBUG=${ARVADOS_DEBUG:-0}
-BUILD_BUNDLE_PACKAGES=0
TARGET=debian7
COMMAND=
@@ -52,9 +51,6 @@ while [ $# -gt 0 ]; do
--debug)
DEBUG=1
;;
- --build-bundle-packages)
- BUILD_BUNDLE_PACKAGES=1
- ;;
--command)
COMMAND="$2"; shift
;;
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index 0579814..dce75eb 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -136,9 +136,7 @@ handle_rails_package() {
mkdir -p tmp
version_from_git >"$version_file"
git rev-parse HEAD >git-commit.version
- if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
- fi
+ bundle package --all
)
if [[ 0 != "$?" ]] || ! cd "$WORKSPACE/packages/$TARGET"; then
echo "ERROR: $pkgname package prep failed" >&2
@@ -160,15 +158,11 @@ handle_rails_package() {
# .git and packages are for the SSO server, which is built from its
# repository root.
for exclude in .git packages tmp log coverage \
- vendor/cache/\* Capfile\* config/deploy\*; do
+ Capfile\* config/deploy\*; do
switches+=(-x "$exclude_root/$exclude")
done
fpm_build "${pos_args[@]}" "${switches[@]}" \
-x "$exclude_root/vendor/bundle" "$@" "$license_arg"
- if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- posargs[1]="$pkgname-with-bundle"
- fpm_build "${pos_args[@]}" "${switches[@]}" "$@" "$license_arg"
- fi
rm -rf "$scripts_dir" "$version_file"
}
diff --git a/jenkins/run-test-packages-sso.sh b/jenkins/run-test-packages-sso.sh
deleted file mode 100755
index 03d4a96..0000000
--- a/jenkins/run-test-packages-sso.sh
+++ /dev/null
@@ -1,293 +0,0 @@
-#!/bin/bash
-
-. `dirname "$(readlink -f "$0")"`/run-library.sh
-
-read -rd "\000" helpmessage <<EOF
-$(basename $0): Test Arvados SSO package
-
-Syntax:
- WORKSPACE=/path/to/arvados-sso $(basename $0) [options]
-
-Options:
-
---debug
- Output debug information (default: false)
---target
- Distribution to build packages for (default: debian7)
-
-WORKSPACE=path Path to the Arvados SSO source tree
-
-EOF
-
-EXITCODE=0
-DEBUG=${ARVADOS_DEBUG:-0}
-BUILD_BUNDLE_PACKAGES=0
-TARGET=debian7
-SUDO=/usr/bin/sudo
-
-if [[ ! -e "$SUDO" ]]; then
- SUDO=
-fi
-
-PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,build-bundle-packages,debug,target: \
- -- "" "$@")
-if [ $? -ne 0 ]; then
- exit 1
-fi
-
-eval set -- "$PARSEDOPTS"
-while [ $# -gt 0 ]; do
- case "$1" in
- --help)
- echo >&2 "$helpmessage"
- echo >&2
- exit 1
- ;;
- --target)
- TARGET="$2"; shift
- ;;
- --debug)
- DEBUG=1
- ;;
- --build-bundle-packages)
- BUILD_BUNDLE_PACKAGES=1
- ;;
- --)
- if [ $# -gt 1 ]; then
- echo >&2 "$0: unrecognized argument '$2'. Try: $0 --help"
- exit 1
- fi
- ;;
- esac
- shift
-done
-
-STDOUT_IF_DEBUG=/dev/null
-STDERR_IF_DEBUG=/dev/null
-DASHQ_UNLESS_DEBUG=-q
-if [[ "$DEBUG" != 0 ]]; then
- STDOUT_IF_DEBUG=/dev/stdout
- STDERR_IF_DEBUG=/dev/stderr
- DASHQ_UNLESS_DEBUG=
-fi
-
-case "$TARGET" in
- debian7)
- FORMAT=deb
- ;;
- debian8)
- FORMAT=deb
- ;;
- ubuntu1204)
- FORMAT=deb
- ;;
- ubuntu1404)
- FORMAT=deb
- ;;
- centos6)
- FORMAT=rpm
- ;;
- *)
- echo -e "$0: Unknown target '$TARGET'.\n" >&2
- exit 1
- ;;
-esac
-
-if ! [[ -n "$WORKSPACE" ]]; then
- echo >&2 "$helpmessage"
- echo >&2
- echo >&2 "Error: WORKSPACE environment variable not set"
- echo >&2
- exit 1
-fi
-
-if ! [[ -d "$WORKSPACE" ]]; then
- echo >&2 "$helpmessage"
- echo >&2
- echo >&2 "Error: $WORKSPACE is not a directory"
- echo >&2
- exit 1
-fi
-
-title () {
- txt="********** $1 **********"
- printf "\n%*s%s\n\n" $((($COLUMNS-${#txt})/2)) "" "$txt"
-}
-
-checkexit() {
- if [[ "$1" != "0" ]]; then
- title "!!!!!! $2 FAILED !!!!!!"
- fi
-}
-
-
-# Find the SSO server package
-
-cd "$WORKSPACE"
-
-SSO_VERSION=$(version_from_git)
-PACKAGE_NAME=arvados-sso-server
-
-if [[ "$FORMAT" == "deb" ]]; then
- PACKAGE_PATH=$WORKSPACE/packages/$TARGET/${PACKAGE_NAME}_${SSO_VERSION}_amd64.deb
-elif [[ "$FORMAT" == "rpm" ]]; then
- PACKAGE_PATH=$WORKSPACE/packages/$TARGET/${PACKAGE_NAME}-${SSO_VERSION}-1.x86_64.rpm
-fi
-
-# Test 1a: the package to test must exist
-if [[ ! -f $PACKAGE_PATH ]]; then
- echo "Latest package not found at $PACKAGE_PATH. Please build the package first."
- exit 1
-fi
-
-if [[ "$FORMAT" == "deb" ]]; then
- # Test 1b: the system/container where we're running the tests must be clean
- set +e
- dpkg -l |grep $PACKAGE_NAME -q
- if [[ "$?" != "1" ]]; then
- echo "Please make sure the $PACKAGE_NAME package is not installed before running this script"
- exit 1
- fi
- set -e
-fi
-
-if [[ -e "/var/www/arvados-sso" ]]; then
- echo "Please make sure /var/www/arvados-sso does not exist before running this script"
- exit 1
-fi
-
-# Prepare the machine
-if [[ "$FORMAT" == "deb" ]]; then
- $SUDO apt-get update
-elif [[ "$FORMAT" == "rpm" ]]; then
- $SUDO yum check-update
-fi
-$SUDO mkdir -p /etc/arvados/sso
-
-if [[ ! -e "/etc/arvados/sso/application.yml" ]]; then
- RANDOM_PASSWORD=`date | md5sum |cut -f1 -d' '`
- cp config/application.yml.example /etc/arvados/sso/application.yml
- sed -i -e 's/uuid_prefix: ~/uuid_prefix: zzzzz/' /etc/arvados/sso/application.yml
- sed -i -e "s/secret_token: ~/secret_token: $RANDOM_PASSWORD/" /etc/arvados/sso/application.yml
-fi
-
-if [[ ! -e "/etc/arvados/sso/database.yml" ]]; then
- # We haven't installed our dependencies yet, but we need to set up our
- # database configuration now. Install postgresql if need be.
- if [[ "$FORMAT" == "deb" ]]; then
- install_package postgresql
- elif [[ "$FORMAT" == "rpm" ]]; then
- install_package postgresql-server
- # postgres packaging on CentOS6 is kind of primitive, needs an initdb
- $SUDO service postgresql initdb
- if [ "$TARGET" = "centos6" ]; then
- sed -i -e "s/127.0.0.1\/32 ident/127.0.0.1\/32 md5/" /var/lib/pgsql/data/pg_hba.conf
- sed -i -e "s/::1\/128 ident/::1\/128 md5/" /var/lib/pgsql/data/pg_hba.conf
- fi
- fi
- $SUDO service postgresql start
-
- RANDOM_PASSWORD=`date | md5sum |cut -f1 -d' '`
- cat >/etc/arvados/sso/database.yml <<EOF
-production:
- adapter: postgresql
- encoding: utf8
- database: sso_provider_production
- username: sso_provider_user
- password: $RANDOM_PASSWORD
- host: localhost
-EOF
- if [[ "$SUDO" != '' ]]; then
- $SUDO -u postgres psql -c "CREATE USER sso_provider_user WITH PASSWORD '$RANDOM_PASSWORD'"
- $SUDO -u postgres createdb sso_provider_production -O sso_provider_user
- else
- install_package sudo
- if [ "$TARGET" = "centos6" ]; then
- # Work around silly CentOS6 default, cf. https://bugzilla.redhat.com/show_bug.cgi?id=1020147
- sed -i -e 's/Defaults requiretty/#Defaults requiretty/' /etc/sudoers
- fi
- /usr/bin/sudo -u postgres psql -c "CREATE USER sso_provider_user WITH PASSWORD '$RANDOM_PASSWORD'"
- /usr/bin/sudo -u postgres createdb sso_provider_production -O sso_provider_user
- fi
-fi
-
-if [[ "$FORMAT" == "deb" ]]; then
- # Test 2: the package should install cleanly
- # In order to get the package dependencies, we need to first do an install
- # with dpkg that will fail, then run apt-get to install the dependencies,
- # and the subsequent dpkg installation should succeed.
- set +e
- $SUDO dpkg -i $PACKAGE_PATH > /dev/null 2>&1
- $SUDO apt-get -f install --yes
- set -e
- $SUDO dpkg -i $PACKAGE_PATH || EXITCODE=2
-
- checkexit $EXITCODE "dpkg -i $PACKAGE_PATH"
-
- # Test 3: the package should remove cleanly
- $SUDO apt-get remove $PACKAGE_NAME --yes || EXITCODE=3
-
- checkexit $EXITCODE "apt-get remove $PACKAGE_PATH --yes"
-
- # Test 4: the package configuration should remove cleanly
- $SUDO dpkg --purge $PACKAGE_NAME || EXITCODE=4
-
- checkexit $EXITCODE "dpkg --purge $PACKAGE_PATH"
-
- if [[ -e "/var/www/arvados-sso" ]]; then
- EXITCODE=4
- fi
-
- checkexit $EXITCODE "leftover items under /var/www/arvados-sso"
-
- # Test 5: the package should remove cleanly with --purge
- $SUDO dpkg -i $PACKAGE_PATH || EXITCODE=5
-
- checkexit $EXITCODE "dpkg -i $PACKAGE_PATH"
-
- $SUDO apt-get remove $PACKAGE_NAME --purge --yes || EXITCODE=5
-
- checkexit $EXITCODE "apt-get remove $PACKAGE_PATH --purge --yes"
-
- if [[ -e "/var/www/arvados-sso" ]]; then
- EXITCODE=5
- fi
-
- checkexit $EXITCODE "leftover items under /var/www/arvados-sso"
-
-elif [[ "$FORMAT" == "rpm" ]]; then
-
- # Set up Nginx first
- # (courtesy of https://www.phusionpassenger.com/library/walkthroughs/deploy/ruby/ownserver/nginx/oss/el6/install_passenger.html)
- $SUDO yum install -q -y epel-release pygpgme curl
- $SUDO curl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo
- $SUDO yum install -q -y nginx passenger
- $SUDO sed -i -e 's/^# passenger/passenger/' /etc/nginx/conf.d/passenger.conf
- # Done setting up Nginx
-
- # Test 2: the package should install cleanly
- $SUDO yum -q -y --nogpgcheck localinstall $PACKAGE_PATH || EXITCODE=3
-
- checkexit $EXITCODE "yum -q -y --nogpgcheck localinstall $PACKAGE_PATH"
-
- # Test 3: the package should remove cleanly
- $SUDO yum -q -y remove $PACKAGE_NAME || EXITCODE=3
-
- checkexit $EXITCODE "yum -q -y remove $PACKAGE_PATH"
-
- if [[ -e "/var/www/arvados-sso" ]]; then
- EXITCODE=3
- fi
-
- checkexit $EXITCODE "leftover items under /var/www/arvados-sso"
-
-fi
-
-if [[ "$EXITCODE" == "0" ]]; then
- echo "Testing complete, no errors!"
-else
- echo "Errors while testing!"
-fi
-
-exit $EXITCODE
diff --git a/jenkins/test-package-arvados-api-server.sh b/jenkins/test-package-arvados-api-server.sh
index 0f21ff5..467872b 100755
--- a/jenkins/test-package-arvados-api-server.sh
+++ b/jenkins/test-package-arvados-api-server.sh
@@ -1,5 +1,6 @@
#!/bin/sh
set -e
cd /var/www/arvados-api/current/
-/usr/local/rvm/bin/rvm-exec default bundle install
+yum install --assumeyes httpd
+yum reinstall --assumeyes arvados-api-server
/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arados-api-server.gems
diff --git a/jenkins/test-package-arvados-sso-server.sh b/jenkins/test-package-arvados-sso-server.sh
new file mode 100755
index 0000000..7f1221e
--- /dev/null
+++ b/jenkins/test-package-arvados-sso-server.sh
@@ -0,0 +1,172 @@
+#!/bin/bash
+
+set -e
+
+EXITCODE=0
+DEBUG=${ARVADOS_DEBUG:-0}
+
+STDOUT_IF_DEBUG=/dev/null
+STDERR_IF_DEBUG=/dev/null
+DASHQ_UNLESS_DEBUG=-q
+if [[ "$DEBUG" != 0 ]]; then
+ STDOUT_IF_DEBUG=/dev/stdout
+ STDERR_IF_DEBUG=/dev/stderr
+ DASHQ_UNLESS_DEBUG=
+fi
+
+case "$TARGET" in
+ debian*|ubuntu*)
+ FORMAT=deb
+ ;;
+ centos6)
+ FORMAT=rpm
+ ;;
+ *)
+ echo -e "$0: Unknown target '$TARGET'.\n" >&2
+ exit 1
+ ;;
+esac
+
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
+fi
+
+if ! [[ -d "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: $WORKSPACE is not a directory"
+ echo >&2
+ exit 1
+fi
+
+title () {
+ txt="********** $1 **********"
+ printf "\n%*s%s\n\n" $((($COLUMNS-${#txt})/2)) "" "$txt"
+}
+
+checkexit() {
+ if [[ "$1" != "0" ]]; then
+ title "!!!!!! $2 FAILED !!!!!!"
+ fi
+}
+
+
+# Find the SSO server package
+
+cd "$WORKSPACE"
+
+if [[ ! -d "/var/www/arvados-sso" ]]; then
+ echo "/var/www/arvados-sso should exist"
+ exit 1
+fi
+
+if [[ ! -e "/etc/arvados/sso/application.yml" ]]; then
+ mkdir -p /etc/arvados/sso/
+ RANDOM_PASSWORD=`date | md5sum |cut -f1 -d' '`
+ cp config/application.yml.example /etc/arvados/sso/application.yml
+ sed -i -e 's/uuid_prefix: ~/uuid_prefix: zzzzz/' /etc/arvados/sso/application.yml
+ sed -i -e "s/secret_token: ~/secret_token: $RANDOM_PASSWORD/" /etc/arvados/sso/application.yml
+fi
+
+if [[ ! -e "/etc/arvados/sso/database.yml" ]]; then
+ # We need to set up our database configuration now.
+ if [[ "$FORMAT" == "rpm" ]]; then
+ # postgres packaging on CentOS6 is kind of primitive, needs an initdb
+ $SUDO service postgresql initdb
+ if [ "$TARGET" = "centos6" ]; then
+ sed -i -e "s/127.0.0.1\/32 ident/127.0.0.1\/32 md5/" /var/lib/pgsql/data/pg_hba.conf
+ sed -i -e "s/::1\/128 ident/::1\/128 md5/" /var/lib/pgsql/data/pg_hba.conf
+ fi
+ fi
+ service postgresql start
+
+ RANDOM_PASSWORD=`date | md5sum |cut -f1 -d' '`
+ cat >/etc/arvados/sso/database.yml <<EOF
+production:
+ adapter: postgresql
+ encoding: utf8
+ database: sso_provider_production
+ username: sso_provider_user
+ password: $RANDOM_PASSWORD
+ host: localhost
+EOF
+
+ su postgres -c "psql -c \"CREATE USER sso_provider_user WITH PASSWORD '$RANDOM_PASSWORD'\""
+ su postgres -c "createdb sso_provider_production -O sso_provider_user"
+fi
+
+if [[ "$FORMAT" == "deb" ]]; then
+ # Test 2: the package should reconfigure cleanly
+ dpkg-reconfigure arvados-sso-server || EXITCODE=3
+
+ cd /var/www/arvados-sso/current/
+ /usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arvados-sso-server.gems
+
+ # Test 3: the package should remove cleanly
+ $SUDO apt-get remove arvados-sso-server --yes || EXITCODE=3
+
+ checkexit $EXITCODE "apt-get remove arvados-sso-server --yes"
+
+ # Test 4: the package configuration should remove cleanly
+ $SUDO dpkg --purge arvados-sso-server || EXITCODE=4
+
+ checkexit $EXITCODE "dpkg --purge arvados-sso-server"
+
+ if [[ -e "/var/www/arvados-sso" ]]; then
+ EXITCODE=4
+ fi
+
+ checkexit $EXITCODE "leftover items under /var/www/arvados-sso"
+
+ # Test 5: the package should remove cleanly with --purge
+
+ $SUDO apt-get remove arvados-sso-server --purge --yes || EXITCODE=5
+
+ checkexit $EXITCODE "apt-get remove arvados-sso-server --purge --yes"
+
+ if [[ -e "/var/www/arvados-sso" ]]; then
+ EXITCODE=5
+ fi
+
+ checkexit $EXITCODE "leftover items under /var/www/arvados-sso"
+
+elif [[ "$FORMAT" == "rpm" ]]; then
+
+ # Set up Nginx first
+ # (courtesy of https://www.phusionpassenger.com/library/walkthroughs/deploy/ruby/ownserver/nginx/oss/el6/install_passenger.html)
+ $SUDO yum install -q -y epel-release pygpgme curl
+ $SUDO curl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo
+ $SUDO yum install -q -y nginx passenger
+ $SUDO sed -i -e 's/^# passenger/passenger/' /etc/nginx/conf.d/passenger.conf
+ # Done setting up Nginx
+
+ # Test 2: the package should reinstall cleanly
+ yum --assumeyes reinstall arvados-sso-server || EXITCODE=3
+
+ cd /var/www/arvados-sso/current/
+ /usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arvados-sso-server.gems
+
+ # Test 3: the package should remove cleanly
+ $SUDO yum -q -y remove arvados-sso-server || EXITCODE=3
+
+ checkexit $EXITCODE "yum -q -y remove arvados-sso-server"
+
+ if [[ -e "/var/www/arvados-sso" ]]; then
+ EXITCODE=3
+ fi
+
+ checkexit $EXITCODE "leftover items under /var/www/arvados-sso"
+
+fi
+
+if [[ "$EXITCODE" == "0" ]]; then
+ echo "Testing complete, no errors!"
+else
+ echo "Errors while testing!"
+fi
+
+exit $EXITCODE
diff --git a/jenkins/test-package-arvados-workbench.sh b/jenkins/test-package-arvados-workbench.sh
index 270fc3a..7111b94 100755
--- a/jenkins/test-package-arvados-workbench.sh
+++ b/jenkins/test-package-arvados-workbench.sh
@@ -1,5 +1,6 @@
#!/bin/sh
set -e
cd /var/www/arvados-workbench/current/
-/usr/local/rvm/bin/rvm-exec default bundle install
+yum install --assumeyes httpd
+yum reinstall --assumeyes arvados-workbench
/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arvados-workbench.gems
commit 2fe4a6a068e720e72b1695de133b4363e9a69a40
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 13 16:41:39 2016 -0500
Fixing our fpm rails package building to handle fpm-info.sh.
diff --git a/jenkins/package-test-dockerfiles/centos6/Dockerfile b/jenkins/package-test-dockerfiles/centos6/Dockerfile
index 0020d5f..6870ec3 100644
--- a/jenkins/package-test-dockerfiles/centos6/Dockerfile
+++ b/jenkins/package-test-dockerfiles/centos6/Dockerfile
@@ -12,4 +12,8 @@ RUN touch /var/lib/rpm/* && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
/usr/local/rvm/bin/rvm-exec default gem install bundle fpm
+RUN cd / root && \
+ curl -OL 'http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm' && \
+ rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
+
COPY localrepo.repo /etc/yum.repos.d/localrepo.repo
\ No newline at end of file
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index ad46452..af6e078 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -21,6 +21,8 @@ WORKSPACE=path Path to the Arvados source tree to build packages from
EOF
+set -e
+
if ! [[ -n "$WORKSPACE" ]]; then
echo >&2 "$helpmessage"
echo >&2
@@ -131,7 +133,7 @@ if test -z "$packages" ; then
keep-web
libarvados-perl"
- if test $target == centos6 ; then
+ if test "$target" = centos6 ; then
packages="$packages python27-python-arvados-fuse
python27-python-arvados-python-client"
else
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index 623e844..0579814 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -235,18 +235,17 @@ fpm_build () {
# that will take precedence, as desired.
COMMAND_ARR+=(--iteration "$(default_iteration "$PACKAGE" "$VERSION")")
- # Append remaining function arguments directly to fpm's command line.
- for i; do
- COMMAND_ARR+=("$i")
- done
+ # 'dir' type packages are provided in the form /path/to/source=/path/to/dest
+ # so strip off the 2nd part to check for fpm-info below.
+ PACKAGE_DIR=$(echo $PACKAGE | sed 's/\/=.*//')
# Append --depends X and other arguments specified by fpm-info.sh in
# the package source dir. These are added last so they can override
# the arguments added by this script.
declare -a fpm_args=()
declare -a fpm_depends=()
- if [[ -d "$PACKAGE" ]]; then
- FPM_INFO="$PACKAGE/fpm-info.sh"
+ if [[ -d "$PACKAGE_DIR" ]]; then
+ FPM_INFO="$PACKAGE_DIR/fpm-info.sh"
else
FPM_INFO="${WORKSPACE}/backports/${PACKAGE_TYPE}-${PACKAGE}/fpm-info.sh"
fi
@@ -259,6 +258,11 @@ fpm_build () {
done
COMMAND_ARR+=("${fpm_args[@]}")
+ # Append remaining function arguments directly to fpm's command line.
+ for i; do
+ COMMAND_ARR+=("$i")
+ done
+
COMMAND_ARR+=("$PACKAGE")
debug_echo -e "\n${COMMAND_ARR[@]}\n"
commit ebdbb0bb23ceb1d1373472e6405fa2293f51616b
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 13 15:40:36 2016 -0500
8008: Run bundle install and bundle list for api server and workbench.
diff --git a/jenkins/deb-common-test-packages.sh b/jenkins/deb-common-test-packages.sh
index fe6a04e..1b52fc1 100755
--- a/jenkins/deb-common-test-packages.sh
+++ b/jenkins/deb-common-test-packages.sh
@@ -12,7 +12,9 @@ apt-get --assume-yes --force-yes install $1
mkdir -p /tmp/opts
cd /tmp/opts
-dpkg-deb -x /arvados/packages/$target/$1-*.deb .
+export ARV_PACKAGES_DIR=/arvados/packages/$target
+
+dpkg-deb -x $ARV_PACKAGES_DIR/"$1"_*.deb .
for so in $(find . -name "*.so") ; do
echo
diff --git a/jenkins/package-test-dockerfiles/centos6/Dockerfile b/jenkins/package-test-dockerfiles/centos6/Dockerfile
index addd259..0020d5f 100644
--- a/jenkins/package-test-dockerfiles/centos6/Dockerfile
+++ b/jenkins/package-test-dockerfiles/centos6/Dockerfile
@@ -1,6 +1,15 @@
FROM centos:6
MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
-RUN yum -q install --assumeyes scl-utils centos-release-SCL
+RUN yum -q install --assumeyes scl-utils centos-release-SCL \
+ which tar
+
+# Install RVM
+RUN touch /var/lib/rpm/* && \
+ gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+ curl -L https://get.rvm.io | bash -s stable && \
+ /usr/local/rvm/bin/rvm install 2.1 && \
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
+ /usr/local/rvm/bin/rvm-exec default gem install bundle fpm
COPY localrepo.repo /etc/yum.repos.d/localrepo.repo
\ No newline at end of file
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index 2399342..ad46452 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -117,7 +117,7 @@ time docker build --tag=$IMAGE .
popd
if test -z "$packages" ; then
-packages="arvados-api-server
+ packages="arvados-api-server
arvados-data-manager
arvados-docker-cleaner
arvados-git-httpd
@@ -129,9 +129,15 @@ packages="arvados-api-server
keep-rsync
keepstore
keep-web
- libarvados-perl
- python27-python-arvados-fuse
- python27-python-arvados-python-client"
+ libarvados-perl"
+
+ if test $target == centos6 ; then
+ packages="$packages python27-python-arvados-fuse
+ python27-python-arvados-python-client"
+ else
+ packages="$packages python-arvados-fuse
+ python-arvados-python-client"
+ fi
fi
FINAL_EXITCODE=0
diff --git a/jenkins/test-package-arvados-api-server.sh b/jenkins/test-package-arvados-api-server.sh
new file mode 100755
index 0000000..0f21ff5
--- /dev/null
+++ b/jenkins/test-package-arvados-api-server.sh
@@ -0,0 +1,5 @@
+#!/bin/sh
+set -e
+cd /var/www/arvados-api/current/
+/usr/local/rvm/bin/rvm-exec default bundle install
+/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arados-api-server.gems
diff --git a/jenkins/test-package-arvados-workbench.sh b/jenkins/test-package-arvados-workbench.sh
new file mode 100755
index 0000000..270fc3a
--- /dev/null
+++ b/jenkins/test-package-arvados-workbench.sh
@@ -0,0 +1,5 @@
+#!/bin/sh
+set -e
+cd /var/www/arvados-workbench/current/
+/usr/local/rvm/bin/rvm-exec default bundle install
+/usr/local/rvm/bin/rvm-exec default bundle list >$ARV_PACKAGES_DIR/arvados-workbench.gems
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index 9241291..29ad987 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -5,14 +5,16 @@ set -eu
yum -q clean all
touch /var/lib/rpm/*
-rpm -qa | sort > /arvados/packages/centos6/$1.before
+export ARV_PACKAGES_DIR=/arvados/packages/centos6
+
+rpm -qa | sort > $ARV_PACKAGES_DIR/$1.before
yum install --assumeyes $1
-rpm -qa | sort > /arvados/packages/centos6/$1.after
+rpm -qa | sort > $ARV_PACKAGES_DIR/$1.after
set +e
-diff /arvados/packages/centos6/$1.before /arvados/packages/centos6/$1.after > /arvados/packages/centos6/$1.diff
+diff $ARV_PACKAGES_DIR/$1.before $ARV_PACKAGES_DIR/$1.after > $ARV_PACKAGES_DIR/$1.diff
set -e
SCL=""
@@ -23,7 +25,7 @@ fi
mkdir -p /tmp/opts
cd /tmp/opts
-rpm2cpio /arvados/packages/centos6/$1-*.rpm | cpio -idm 2>/dev/null
+rpm2cpio $ARV_PACKAGES_DIR/$1-*.rpm | cpio -idm 2>/dev/null
shared=$(find -name '*.so')
if test -n "$shared" ; then
commit 60d3fdf6dedff3cb9eca478caacfeee19a752812
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 13 14:50:45 2016 -0500
8080: Successfully tests all packages. Produces before/after/diff lists for each package.
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index afb6eff..2399342 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -14,6 +14,8 @@ Syntax:
Run package install test script "test-packages-$target.sh"
--debug
Output debug information (default: false)
+--only-test
+ Test only a specific package
WORKSPACE=path Path to the Arvados source tree to build packages from
@@ -36,7 +38,7 @@ if ! [[ -d "$WORKSPACE" ]]; then
fi
PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,debug,test-packages,target:,command: \
+ help,debug,test-packages,target:,command:,only-test: \
-- "" "$@")
if [ $? -ne 0 ]; then
exit 1
@@ -57,6 +59,9 @@ while [ $# -gt 0 ]; do
--target)
TARGET="$2"; shift
;;
+ --only-test)
+ packages="$2"; shift
+ ;;
--debug)
DEBUG=" --debug"
;;
@@ -111,6 +116,7 @@ cd $TARGET
time docker build --tag=$IMAGE .
popd
+if test -z "$packages" ; then
packages="arvados-api-server
arvados-data-manager
arvados-docker-cleaner
@@ -126,14 +132,18 @@ packages="arvados-api-server
libarvados-perl
python27-python-arvados-fuse
python27-python-arvados-python-client"
+fi
FINAL_EXITCODE=0
+package_fails=""
+
if [[ -n "$test_packages" ]]; then
for p in $packages ; do
if ! docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
--env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND $p ; then
FINAL_EXITCODE=$?
+ package_fails="$package_fails $p"
echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
fi
done
@@ -148,4 +158,8 @@ else
fi
fi
+if test -n "$package_fails" ; then
+ echo "Failed package tests:$package_fails"
+fi
+
exit $FINAL_EXITCODE
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index 3d084ab..9241291 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -5,8 +5,16 @@ set -eu
yum -q clean all
touch /var/lib/rpm/*
+rpm -qa | sort > /arvados/packages/centos6/$1.before
+
yum install --assumeyes $1
+rpm -qa | sort > /arvados/packages/centos6/$1.after
+
+set +e
+diff /arvados/packages/centos6/$1.before /arvados/packages/centos6/$1.after > /arvados/packages/centos6/$1.diff
+set -e
+
SCL=""
if scl enable python27 true 2>/dev/null ; then
SCL="scl enable python27"
@@ -15,7 +23,7 @@ fi
mkdir -p /tmp/opts
cd /tmp/opts
-rpm2cpio /arvados/packages/centos6/$1-*.rpm | cpio -idm
+rpm2cpio /arvados/packages/centos6/$1-*.rpm | cpio -idm 2>/dev/null
shared=$(find -name '*.so')
if test -n "$shared" ; then
@@ -27,4 +35,8 @@ if test -n "$shared" ; then
done
fi
-exec $SCL /jenkins/common-test-packages.sh $1
+if test -n "$SCL" ; then
+ exec $SCL "/jenkins/common-test-packages.sh $1"
+else
+ exec /jenkins/common-test-packages.sh $1
+fi
commit ffd741be7af1eca4c11884a3e8aa6630be25de2d
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jan 13 14:11:30 2016 -0500
8008: Test each Arvados package individually.
diff --git a/jenkins/common-test-packages.sh b/jenkins/common-test-packages.sh
index c9d77c8..1ff8707 100755
--- a/jenkins/common-test-packages.sh
+++ b/jenkins/common-test-packages.sh
@@ -1,28 +1,28 @@
#!/bin/sh
+set -eu
+
FAIL=0
echo
-for so in $(find . -name "*.so") ; do
- if ldd $so | grep "not found" ; then
- echo "^^^ Missing while scanning $so ^^^"
- FAIL=1
- fi
-done
+shared=$(find -name '*.so')
+if test -n "$shared" ; then
+ for so in $shared ; do
+ if ldd $so | grep "not found" ; then
+ echo "^^^ Missing while scanning $so ^^^"
+ FAIL=1
+ fi
+ done
+fi
-echo
-if ! python <<EOF
-import arvados
-import arvados_fuse
-print "Successly imported arvados and arvados_fuse"
+if test -x /jenkins/test-package-$1.sh ; then
+ if ! /jenkins/test-package-$1.sh ; then
+ FAIL=1
+ fi
+fi
-import libcloud.compute.types
-import libcloud.compute.providers
-libcloud.compute.providers.get_driver(libcloud.compute.types.Provider.AZURE_ARM)
-print "Successly imported compatible libcloud library"
-EOF
-then
- FAIL=1
+if test $FAIL = 0 ; then
+ echo "Package $1 passed"
fi
exit $FAIL
diff --git a/jenkins/deb-common-test-packages.sh b/jenkins/deb-common-test-packages.sh
index 4bc4573..fe6a04e 100755
--- a/jenkins/deb-common-test-packages.sh
+++ b/jenkins/deb-common-test-packages.sh
@@ -1,22 +1,18 @@
#!/bin/bash
+set -eu
+
# Multiple .deb based distros symlink to this script, so extract the target
# from the invocation path.
target=$(echo $0 | sed 's/.*test-packages-\([^.]*\)\.sh.*/\1/')
apt-get -qq update
-if ! apt-get --assume-yes --force-yes install \
- python-arvados-python-client python-arvados-fuse arvados-node-manager
-then
- exit 1
-fi
+apt-get --assume-yes --force-yes install $1
mkdir -p /tmp/opts
cd /tmp/opts
-for r in /arvados/packages/$target/python-*amd64.deb ; do
- dpkg-deb -x $r .
-done
+dpkg-deb -x /arvados/packages/$target/$1-*.deb .
for so in $(find . -name "*.so") ; do
echo
@@ -24,4 +20,4 @@ for so in $(find . -name "*.so") ; do
ldd $so | awk '($3 ~ /^\//){print $3}' | sort -u | xargs dpkg -S | cut -d: -f1 | sort -u
done
-exec /jenkins/common-test-packages.sh
+exec /jenkins/common-test-packages.sh $1
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index 1d4249b..afb6eff 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -111,15 +111,41 @@ cd $TARGET
time docker build --tag=$IMAGE .
popd
+packages="arvados-api-server
+ arvados-data-manager
+ arvados-docker-cleaner
+ arvados-git-httpd
+ arvados-node-manager
+ arvados-src
+ arvados-workbench
+ crunchstat
+ keepproxy
+ keep-rsync
+ keepstore
+ keep-web
+ libarvados-perl
+ python27-python-arvados-fuse
+ python27-python-arvados-python-client"
+
FINAL_EXITCODE=0
-if docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
- --env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND; then
- # Success - nothing more to do.
- true
+if [[ -n "$test_packages" ]]; then
+ for p in $packages ; do
+ if ! docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
+ --env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND $p ; then
+ FINAL_EXITCODE=$?
+ echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
+ fi
+ done
else
- FINAL_EXITCODE=$?
- echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
+ if docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
+ --env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND; then
+ # Success - nothing more to do.
+ true
+ else
+ FINAL_EXITCODE=$?
+ echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
+ fi
fi
exit $FINAL_EXITCODE
diff --git a/jenkins/test-package-arvados-node-manager.sh b/jenkins/test-package-arvados-node-manager.sh
new file mode 100755
index 0000000..c6760f8
--- /dev/null
+++ b/jenkins/test-package-arvados-node-manager.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+exec python <<EOF
+import libcloud.compute.types
+import libcloud.compute.providers
+libcloud.compute.providers.get_driver(libcloud.compute.types.Provider.AZURE_ARM)
+print "Successly imported compatible libcloud library"
+EOF
diff --git a/jenkins/test-package-python27-python-arvados-fuse.sh b/jenkins/test-package-python27-python-arvados-fuse.sh
new file mode 100755
index 0000000..d2a1072
--- /dev/null
+++ b/jenkins/test-package-python27-python-arvados-fuse.sh
@@ -0,0 +1,6 @@
+#!/bin/sh
+
+exec python <<EOF
+import arvados_fuse
+print "Successly imported arvados_fuse"
+EOF
diff --git a/jenkins/test-package-python27-python-arvados-python-client.sh b/jenkins/test-package-python27-python-arvados-python-client.sh
new file mode 100755
index 0000000..c5abddd
--- /dev/null
+++ b/jenkins/test-package-python27-python-arvados-python-client.sh
@@ -0,0 +1,6 @@
+#!/bin/sh
+
+exec python <<EOF
+import arvados
+print "Successly imported arvados"
+EOF
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index 0d956af..3d084ab 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -1,25 +1,30 @@
#!/bin/bash
+set -eu
+
yum -q clean all
touch /var/lib/rpm/*
-if ! yum install --assumeyes \
- python27-python-arvados-python-client python27-python-arvados-fuse arvados-node-manager
-then
- exit 1
+
+yum install --assumeyes $1
+
+SCL=""
+if scl enable python27 true 2>/dev/null ; then
+ SCL="scl enable python27"
fi
mkdir -p /tmp/opts
cd /tmp/opts
-for r in /arvados/packages/centos6/python27-python-*x86_64.rpm ; do
- rpm2cpio $r | cpio -idm
-done
+rpm2cpio /arvados/packages/centos6/$1-*.rpm | cpio -idm
-for so in $(find -name "*.so") ; do
- echo
- echo "== Packages dependencies for $so =="
- scl enable python27 "ldd $so" \
- | awk '($3 ~ /^\//){print $3}' | sort -u | xargs rpm -qf | sort -u
-done
+shared=$(find -name '*.so')
+if test -n "$shared" ; then
+ for so in $shared ; do
+ echo
+ echo "== Packages dependencies for $so =="
+ $SCL ldd "$so" \
+ | awk '($3 ~ /^\//){print $3}' | sort -u | xargs rpm -qf | sort -u
+ done
+fi
-exec scl enable python27 /jenkins/common-test-packages.sh
+exec $SCL /jenkins/common-test-packages.sh $1
commit c9e5a801bb72c20b6c56e0b88e78ea5626fb6497
Author: radhika <radhika at curoverse.com>
Date: Wed Jan 13 11:17:11 2016 -0500
8028: add services/crunch-dispatch-local
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index a7b5fda..152319c 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -68,6 +68,7 @@ services/keepproxy
services/keepstore
services/login-sync
services/nodemanager
+services/crunch-dispatch-local
sdk/cli
sdk/pam
sdk/python
@@ -720,6 +721,7 @@ gostuff=(
services/datamanager/collection
services/datamanager/keep
services/datamanager
+ services/crunch-dispatch-local
tools/keep-rsync
)
for g in "${gostuff[@]}"
commit 1e48d4f97d643b3a36f7856284e3a95430bc8f92
Merge: 886c3c5 211b5c1
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jan 11 17:17:45 2016 -0500
Merge branch '8014-rails-postinst-scripts-wip'
Refs #8014. Closes #8036.
commit 211b5c1e8a86fe9f3a6704f76120e2fed78a9247
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jan 11 17:17:09 2016 -0500
8014: Rails upgrade scripts are removed; don't run them in deploy script.
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index f83f901..1b06c65 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -163,8 +163,6 @@ title "Updating API server"
SUM_ECODE=0
run_puppet $IDENTIFIER.arvadosapi.com ECODE
SUM_ECODE=$(($SUM_ECODE + $ECODE))
-run_command $IDENTIFIER.arvadosapi.com ECODE "/usr/local/rvm/bin/rvm-exec /usr/local/bin/arvados-api-server-upgrade.sh"
-SUM_ECODE=$(($SUM_ECODE + $ECODE))
if [ ! "$IDENTIFIER" = "c97qk" ]
then
run_command $IDENTIFIER.arvadosapi.com ECODE "dpkg -L arvados-mailchimp-plugin 2>/dev/null && apt-get install arvados-mailchimp-plugin --reinstall || echo"
@@ -239,9 +237,6 @@ if [[ `host workbench.$ARVADOS_API_HOST |cut -f4 -d' '` != `host $ARVADOS_API_HO
SUM_ECODE=$(($SUM_ECODE + $ECODE))
fi
-run_command workbench.$IDENTIFIER ECODE "/usr/local/rvm/bin/rvm-exec /usr/local/bin/arvados-workbench-upgrade.sh"
-SUM_ECODE=$(($SUM_ECODE + $ECODE))
-
if [[ "$SUM_ECODE" != "0" ]]; then
title "ERROR: Updating workbench FAILED"
EXITCODE=$(($EXITCODE + $SUM_ECODE))
commit 81afb79616d482b177d91ee4e198392e750221ab
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jan 4 18:02:08 2016 -0500
8014: Rails postinst script sets better default access for /etc/arvados.
diff --git a/jenkins/rails-package-scripts/postinst.sh b/jenkins/rails-package-scripts/postinst.sh
index 96e687c..1032d14 100644
--- a/jenkins/rails-package-scripts/postinst.sh
+++ b/jenkins/rails-package-scripts/postinst.sh
@@ -64,6 +64,14 @@ run_and_report() {
return $retcode
}
+setup_confdirs() {
+ for confdir in "$@"; do
+ if [ ! -d "$confdir" ]; then
+ install -d -g "$WWW_OWNER" -m 0750 "$confdir"
+ fi
+ done
+}
+
setup_conffile() {
# Usage: setup_conffile CONFFILE_PATH [SOURCE_PATH]
# Both paths are relative to RELEASE_CONFIG_PATH.
@@ -84,19 +92,27 @@ setup_conffile() {
if [ ! -e "$release_conffile" ]; then
ln -s "$etc_conffile" "$release_conffile"
# If there's a config file in /var/www identical to the one in /etc,
- # overwrite it with a symlink.
+ # overwrite it with a symlink after porting its permissions.
elif cmp --quiet "$release_conffile" "$etc_conffile"; then
+ local ownership="$(stat -c "%U:%G" "$release_conffile")"
+ chown "$ownership" "$etc_conffile"
+ chmod --reference="$release_conffile" "$etc_conffile"
+ chgrp "${ownership#*:}" "$CONFIG_PATH" /etc/arvados
+ chmod g+rx "$CONFIG_PATH" /etc/arvados
ln --force -s "$etc_conffile" "$release_conffile"
fi
fi
if [ -n "$conffile_source" ]; then
- cp --no-clobber "$RELEASE_CONFIG_PATH/$conffile_source" "$etc_conffile"
+ if [ ! -e "$etc_conffile" ]; then
+ install -g "$WWW_OWNER" -m 0640 \
+ "$RELEASE_CONFIG_PATH/$conffile_source" "$etc_conffile"
+ return 1
# Even if $etc_conffile already existed, it might be unmodified from
# the source. This is especially likely when a user installs, updates
# database.yml, then reconfigures before they update application.yml.
# Use cmp to be sure whether $etc_conffile is modified.
- if cmp --quiet "$RELEASE_CONFIG_PATH/$conffile_source" "$etc_conffile"; then
+ elif cmp --quiet "$RELEASE_CONFIG_PATH/$conffile_source" "$etc_conffile"; then
return 1
fi
fi
@@ -137,14 +153,14 @@ configure_version() {
if [ -e /etc/redhat-release ]; then
# Recognize any service that starts with "nginx"; e.g., nginx16.
if [ "$WEB_SERVICE" != "${WEB_SERVICE#nginx}" ]; then
- WWW_OWNER=nginx:nginx
+ WWW_OWNER=nginx
else
- WWW_OWNER=apache:apache
+ WWW_OWNER=apache
fi
else
# Assume we're on a Debian-based system for now.
# Both Apache and Nginx run as www-data by default.
- WWW_OWNER=www-data:www-data
+ WWW_OWNER=www-data
fi
echo
@@ -154,7 +170,7 @@ configure_version() {
echo
echo -n "Creating symlinks to configuration in $CONFIG_PATH ..."
- mkdir -p $CONFIG_PATH
+ setup_confdirs /etc/arvados "$CONFIG_PATH"
setup_conffile environments/production.rb environments/production.rb.example \
|| true
setup_conffile application.yml application.yml.example || APPLICATION_READY=0
@@ -182,21 +198,20 @@ configure_version() {
echo -n "Ensuring directory and file permissions ..."
# Ensure correct ownership of a few files
- chown "$WWW_OWNER" $RELEASE_PATH/config/environment.rb
- chown "$WWW_OWNER" $RELEASE_PATH/config.ru
- chown "$WWW_OWNER" $RELEASE_PATH/Gemfile.lock
- chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
- chown -R "$WWW_OWNER" $SHARED_PATH/log
+ chown "$WWW_OWNER:" $RELEASE_PATH/config/environment.rb
+ chown "$WWW_OWNER:" $RELEASE_PATH/config.ru
+ chown "$WWW_OWNER:" $RELEASE_PATH/Gemfile.lock
+ chown -R "$WWW_OWNER:" $RELEASE_PATH/tmp
+ chown -R "$WWW_OWNER:" $SHARED_PATH/log
case "$RAILSPKG_DATABASE_LOAD_TASK" in
- db:schema:load) chown "$WWW_OWNER" $RELEASE_PATH/db/schema.rb ;;
- db:structure:load) chown "$WWW_OWNER" $RELEASE_PATH/db/structure.sql ;;
+ db:schema:load) chown "$WWW_OWNER:" $RELEASE_PATH/db/schema.rb ;;
+ db:structure:load) chown "$WWW_OWNER:" $RELEASE_PATH/db/structure.sql ;;
esac
chmod 644 $SHARED_PATH/log/*
chmod -R 2775 $RELEASE_PATH/tmp
echo "... done."
if [ -n "$RAILSPKG_DATABASE_LOAD_TASK" ]; then
- chown "$WWW_OWNER" $RELEASE_PATH/config/database.yml
prepare_database
fi
@@ -213,7 +228,7 @@ configure_version() {
else
echo "Precompiling assets... skipped."
fi
- chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
+ chown -R "$WWW_OWNER:" $RELEASE_PATH/tmp
if [ ! -z "$WEB_SERVICE" ]; then
service "$WEB_SERVICE" restart
commit e6ee12052db27b61f8bcf7af89acc9dc520a1799
Author: Brett Smith <brett at curoverse.com>
Date: Sat Jan 2 12:15:53 2016 -0500
8014, 8059: Unify Rails package building.
* Introduce a handle_rails_package function, and helpers, to build a
Rails package.
* Generalize all the support scripts into unified postinst, prerm, and
postrm scripts. handle_rails_package builds these into the
packages. See jenkins/rails-package-scripts/README.md for details.
* More error checking throughout (e.g., check for failure when doing
any prep work for Rails packages, or the Workbench packages
specifically, using `set -e` in a subshell).
One behavior change from this unification: before this,
run-build-packages treated --build-bundle-packages to mean "*also*
build bundle packages", while run-build-packages-sso treated the
switch to mean "*only* build bundle packages". This commit declares
run-build-packages to be the winner, and makes run-build-packages-sso
consistent with it.
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
deleted file mode 100755
index 2fff1e7..0000000
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ /dev/null
@@ -1,118 +0,0 @@
-#!/bin/bash
-
-set -e
-
-if [ -e /etc/redhat-release ]; then
- WWW_OWNER=nginx:nginx
-else
- # Assume we're on a Debian-based system for now.
- WWW_OWNER=www-data:www-data
-fi
-
-NGINX_SERVICE=${NGINX_SERVICE:-$(service --status-all 2>/dev/null \
- | grep -Eo '\bnginx[^[:space:]]*' || true)}
-if [ -z "$NGINX_SERVICE" ]; then
- cat >&2 <<EOF
-Error: nginx service not found. Aborting.
-Set NGINX_SERVICE to the name of the service hosting the Rails server.
-EOF
- exit 1
-elif [ "$NGINX_SERVICE" != "$(echo "$NGINX_SERVICE" | head -n 1)" ]; then
- cat >&2 <<EOF
-Error: multiple nginx services found. Aborting.
-Set NGINX_SERVICE to the name of the service hosting the Rails server.
-EOF
- exit 1
-fi
-
-RELEASE_PATH=/var/www/arvados-api/current
-SHARED_PATH=/var/www/arvados-api/shared
-CONFIG_PATH=/etc/arvados/api/
-
-echo "Assumption: $NGINX_SERVICE is configured to serve your API server URL from"
-echo " /var/www/arvados-api/current"
-echo "Assumption: configuration files are in /etc/arvados/api/"
-echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
-echo
-
-echo "Copying files from $CONFIG_PATH"
-cp -f $CONFIG_PATH/database.yml $RELEASE_PATH/config/database.yml
-cp -f $RELEASE_PATH/config/environments/production.rb.example $RELEASE_PATH/config/environments/production.rb
-cp -f $CONFIG_PATH/application.yml $RELEASE_PATH/config/application.yml
-if [ -e $CONFIG_PATH/omniauth.rb ]; then
- cp -f $CONFIG_PATH/omniauth.rb $RELEASE_PATH/config/initializers/omniauth.rb
-fi
-echo "Done."
-
-# Before we do anything else, make sure some directories and files are in place
-if [[ ! -e $SHARED_PATH/log ]]; then mkdir -p $SHARED_PATH/log; fi
-if [[ ! -e $RELEASE_PATH/tmp ]]; then mkdir -p $RELEASE_PATH/tmp; fi
-if [[ ! -e $RELEASE_PATH/log ]]; then ln -s $SHARED_PATH/log $RELEASE_PATH/log; fi
-if [[ ! -e $SHARED_PATH/log/production.log ]]; then touch $SHARED_PATH/log/production.log; fi
-
-cd "$RELEASE_PATH"
-export RAILS_ENV=production
-
-echo "Making sure bundle is installed"
-set +e
-which bundle > /dev/null
-if [[ "$?" != "0" ]]; then
- gem install bundle
-fi
-set -e
-echo "Done."
-
-echo "Running bundle install"
-bundle install --path $SHARED_PATH/vendor_bundle
-echo "Done."
-
-echo "Precompiling assets"
-# precompile assets; thankfully this does not take long
-bundle exec rake assets:precompile
-echo "Done."
-
-echo "Ensuring directory and file permissions"
-# Ensure correct ownership of a few files
-chown "$WWW_OWNER" $RELEASE_PATH/config/environment.rb
-chown "$WWW_OWNER" $RELEASE_PATH/config.ru
-chown "$WWW_OWNER" $RELEASE_PATH/config/database.yml
-chown "$WWW_OWNER" $RELEASE_PATH/Gemfile.lock
-chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
-chown -R "$WWW_OWNER" $SHARED_PATH/log
-chown "$WWW_OWNER" $RELEASE_PATH/db/structure.sql
-chmod 644 $SHARED_PATH/log/*
-# Rails creates the cache directory if it doesn't exist
-if [[ -d $RELEASE_PATH/tmp/cache/ ]]; then
- chmod -R 2775 $RELEASE_PATH/tmp/cache/
-fi
-echo "Done."
-
-echo "Running sanity check"
-bundle exec rake config:check
-SANITY_CHECK_EXIT_CODE=$?
-echo "Done."
-
-if [[ "$SANITY_CHECK_EXIT_CODE" != "0" ]]; then
- echo "Sanity check failed, aborting. Please roll back to the previous version of the package."
- echo "The database has not been migrated yet, so reinstalling the previous version is safe."
- exit $SANITY_CHECK_EXIT_CODE
-fi
-
-echo "Checking database status"
-# If we use `grep -q`, rake will write a backtrace on EPIPE.
-if bundle exec rake db:migrate:status | grep '^database: ' >/dev/null; then
- echo "Starting db:migrate"
- bundle exec rake db:migrate
-elif [ 0 -eq ${PIPESTATUS[0]} ]; then
- # The database exists, but the migrations table doesn't.
- echo "Setting up database"
- bundle exec rake db:structure:load db:seed
-else
- echo "Error: Database is not ready to set up. Aborting." >&2
- exit 1
-fi
-echo "Done."
-
-echo "Restarting nginx"
-service "$NGINX_SERVICE" restart
-echo "Done."
diff --git a/jenkins/arvados-api-server-extras/postinst.sh b/jenkins/arvados-api-server-extras/postinst.sh
deleted file mode 100644
index 1aeffe9..0000000
--- a/jenkins/arvados-api-server-extras/postinst.sh
+++ /dev/null
@@ -1,12 +0,0 @@
-#!/bin/bash
-
-cd /var/www/arvados-api
-
-chown -R www-data:www-data tmp >/dev/null 2>&1
-chown -R www-data:www-data log >/dev/null 2>&1
-chown www-data:www-data db/structure.sql >/dev/null 2>&1
-chmod 644 log/* >/dev/null 2>&1
-
-# Errors above are not serious
-exit 0
-
diff --git a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
deleted file mode 100755
index e981517..0000000
--- a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
+++ /dev/null
@@ -1,90 +0,0 @@
-#!/bin/bash
-
-set -e
-
-if [ -e /etc/redhat-release ]; then
- WWW_OWNER=nginx:nginx
-else
- # Assume we're on a Debian-based system for now.
- WWW_OWNER=www-data:www-data
-fi
-
-NGINX_SERVICE=${NGINX_SERVICE:-$(service --status-all 2>/dev/null \
- | grep -Eo '\bnginx[^[:space:]]*' || true)}
-if [ -z "$NGINX_SERVICE" ]; then
- cat >&2 <<EOF
-Error: nginx service not found. Aborting.
-Set NGINX_SERVICE to the name of the service hosting the Rails server.
-EOF
- exit 1
-elif [ "$NGINX_SERVICE" != "$(echo "$NGINX_SERVICE" | head -n 1)" ]; then
- cat >&2 <<EOF
-Error: multiple nginx services found. Aborting.
-Set NGINX_SERVICE to the name of the service hosting the Rails server.
-EOF
- exit 1
-fi
-
-RELEASE_PATH=/var/www/arvados-workbench/current
-SHARED_PATH=/var/www/arvados-workbench/shared
-CONFIG_PATH=/etc/arvados/workbench/
-
-echo "Assumption: $NGINX_SERVICE is configured to serve your workbench URL from "
-echo " /var/www/arvados-workbench/current"
-echo "Assumption: configuration files are in /etc/arvados/workbench/"
-echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
-echo
-
-echo "Copying files from $CONFIG_PATH"
-cp -f $CONFIG_PATH/application.yml $RELEASE_PATH/config/application.yml
-cp -f $RELEASE_PATH/config/environments/production.rb.example $RELEASE_PATH/config/environments/production.rb
-echo "Done."
-
-# Before we do anything else, make sure some directories and files are in place
-if [[ ! -e $SHARED_PATH/log ]]; then mkdir -p $SHARED_PATH/log; fi
-if [[ ! -e $RELEASE_PATH/tmp ]]; then mkdir -p $RELEASE_PATH/tmp; fi
-if [[ ! -e $RELEASE_PATH/log ]]; then ln -s $SHARED_PATH/log $RELEASE_PATH/log; fi
-if [[ ! -e $SHARED_PATH/log/production.log ]]; then touch $SHARED_PATH/log/production.log; fi
-
-echo "Making sure bundle is installed"
-set +e
-which bundle > /dev/null
-if [[ "$?" != "0" ]]; then
- gem install bundle
-fi
-set -e
-echo "Done."
-
-echo "Running bundle install"
-(cd $RELEASE_PATH && RAILS_ENV=production bundle install --path $SHARED_PATH/vendor_bundle)
-echo "Done."
-
-# We do not need to precompile assets, they are already part of the package.
-
-echo "Ensuring directory and file permissions"
-chown "$WWW_OWNER" $RELEASE_PATH/config/environment.rb
-chown "$WWW_OWNER" $RELEASE_PATH/config.ru
-chown "$WWW_OWNER" $RELEASE_PATH/config/database.yml
-chown "$WWW_OWNER" $RELEASE_PATH/Gemfile.lock
-chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
-chown -R "$WWW_OWNER" $SHARED_PATH/log
-chown "$WWW_OWNER" $RELEASE_PATH/db/schema.rb
-chmod 644 $SHARED_PATH/log/*
-echo "Done."
-
-echo "Running sanity check"
-(cd $RELEASE_PATH && RAILS_ENV=production bundle exec rake config:check)
-SANITY_CHECK_EXIT_CODE=$?
-echo "Done."
-
-if [[ "$SANITY_CHECK_EXIT_CODE" != "0" ]]; then
- echo "Sanity check failed, aborting. Please roll back to the previous version of the package."
- exit $SANITY_CHECK_EXIT_CODE
-fi
-
-# We do not need to run db:migrate because Workbench is stateless
-
-echo "Restarting nginx"
-service "$NGINX_SERVICE" restart
-echo "Done."
-
diff --git a/jenkins/arvados-workbench-extras/postinst.sh b/jenkins/arvados-workbench-extras/postinst.sh
deleted file mode 100644
index d0aa9b0..0000000
--- a/jenkins/arvados-workbench-extras/postinst.sh
+++ /dev/null
@@ -1,12 +0,0 @@
-#!/bin/bash
-
-cd /var/www/arvados-workbench
-
-chown -R www-data:www-data tmp >/dev/null 2>&1
-chown -R www-data:www-data log >/dev/null 2>&1
-chown www-data:www-data db/schema.rb >/dev/null 2>&1
-chmod 644 log/* >/dev/null 2>&1
-
-# Errors above are not serious
-exit 0
-
diff --git a/jenkins/rails-package-scripts/README.md b/jenkins/rails-package-scripts/README.md
new file mode 100644
index 0000000..6779fb4
--- /dev/null
+++ b/jenkins/rails-package-scripts/README.md
@@ -0,0 +1,13 @@
+When run-build-packages.sh builds a Rails package, it generates the package's pre/post-inst/rm scripts by concatenating:
+
+1. package_name.sh, which defines variables about where package files live and some human-readable names about them.
+2. step2.sh, which uses those to define some utility variables and set defaults for things that aren't set.
+3. stepname.sh, like postinst.sh, prerm.sh, etc., which uses all this information to do the actual work.
+
+Since our build process is a tower of shell scripts, concatenating files seemed like the least worst option to share code between these files and packages. More advanced code generation would've been too much trouble to integrate into our build process at this time. Trying to inject portions of files into other files seemed error-prone and likely to introduce bugs to the end result.
+
+postinst.sh lets the early parts define a few hooks to control behavior:
+
+* After it installs the core configuration files (database.yml, application.yml, and production.rb) to /etc/arvados/server, it calls setup_extra_conffiles. By default this is a noop function (in step2.sh). API server defines this to set up the old omniauth.rb conffile.
+* $RAILSPKG_DATABASE_LOAD_TASK defines the Rake task to load the database. API server uses db:structure:load. SSO server uses db:schema:load. Workbench doesn't set this, which causes the postinst to skip all database work.
+* If $RAILSPKG_SUPPORTS_CONFIG_CHECK != 1, it won't run the config:check rake task. SSO clears this flag (it doesn't have that task code).
diff --git a/jenkins/rails-package-scripts/arvados-api-server.sh b/jenkins/rails-package-scripts/arvados-api-server.sh
new file mode 100644
index 0000000..4df87eb
--- /dev/null
+++ b/jenkins/rails-package-scripts/arvados-api-server.sh
@@ -0,0 +1,12 @@
+#!/bin/sh
+# This file declares variables common to all scripts for one Rails package.
+
+PACKAGE_NAME=arvados-api-server
+INSTALL_PATH=/var/www/arvados-api
+CONFIG_PATH=/etc/arvados/api
+DOC_URL="http://doc.arvados.org/install/install-api-server.html#configure"
+
+RAILSPKG_DATABASE_LOAD_TASK=db:structure:load
+setup_extra_conffiles() {
+ setup_conffile initializers/omniauth.rb
+}
diff --git a/jenkins/rails-package-scripts/arvados-sso-server.sh b/jenkins/rails-package-scripts/arvados-sso-server.sh
new file mode 100644
index 0000000..10b2ee2
--- /dev/null
+++ b/jenkins/rails-package-scripts/arvados-sso-server.sh
@@ -0,0 +1,9 @@
+#!/bin/sh
+# This file declares variables common to all scripts for one Rails package.
+
+PACKAGE_NAME=arvados-sso-server
+INSTALL_PATH=/var/www/arvados-sso
+CONFIG_PATH=/etc/arvados/sso
+DOC_URL="http://doc.arvados.org/install/install-sso.html#configure"
+RAILSPKG_DATABASE_LOAD_TASK=db:schema:load
+RAILSPKG_SUPPORTS_CONFIG_CHECK=0
diff --git a/jenkins/rails-package-scripts/arvados-workbench.sh b/jenkins/rails-package-scripts/arvados-workbench.sh
new file mode 100644
index 0000000..f2b8a56
--- /dev/null
+++ b/jenkins/rails-package-scripts/arvados-workbench.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+# This file declares variables common to all scripts for one Rails package.
+
+PACKAGE_NAME=arvados-workbench
+INSTALL_PATH=/var/www/arvados-workbench
+CONFIG_PATH=/etc/arvados/workbench
+DOC_URL="http://doc.arvados.org/install/install-workbench-app.html#configure"
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/rails-package-scripts/postinst.sh
old mode 100755
new mode 100644
similarity index 82%
rename from jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
rename to jenkins/rails-package-scripts/postinst.sh
index e72d9b8..96e687c
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/rails-package-scripts/postinst.sh
@@ -1,15 +1,8 @@
-#!/bin/bash
+#!/bin/sh
+# This code runs after package variable definitions and step2.sh.
set -e
-INSTALL_PATH=/var/www/arvados-sso
-RELEASE_PATH=$INSTALL_PATH/current
-RELEASE_CONFIG_PATH=$RELEASE_PATH/config
-SHARED_PATH=$INSTALL_PATH/shared
-CONFIG_PATH=/etc/arvados/sso
-PACKAGE_NAME=arvados-sso-server
-DOC_URL="http://doc.arvados.org/install/install-sso.html#configure"
-
DATABASE_READY=1
APPLICATION_READY=1
@@ -20,8 +13,8 @@ else
fi
report_not_ready() {
- local ready_flag=$1; shift
- local config_file=$1; shift
+ local ready_flag="$1"; shift
+ local config_file="$1"; shift
if [ "1" != "$ready_flag" ]; then cat >&2 <<EOF
PLEASE NOTE:
@@ -39,7 +32,7 @@ EOF
}
report_web_service_warning() {
- local warning=$1; shift
+ local warning="$1"; shift
cat >&2 <<EOF
WARNING: $warning.
@@ -59,7 +52,7 @@ run_and_report() {
# This is the usual wrapper that prints ACTION_MSG, runs CMD, then writes
# a message about whether CMD succeeded or failed. Returns the exit code
# of CMD.
- local action_message=$1; shift
+ local action_message="$1"; shift
local retcode=0
echo -n "$action_message..."
if "$@"; then
@@ -79,10 +72,10 @@ setup_conffile() {
# If SOURCE_PATH is given, this function will try to install that file as
# the configuration file in CONFIG_PATH, and return 1 if the file in
# CONFIG_PATH is unmodified from the source.
- local conffile_relpath=$1; shift
- local conffile_source=$1; shift
- local release_conffile=$RELEASE_CONFIG_PATH/$conffile_relpath
- local etc_conffile=$CONFIG_PATH/$(basename "$conffile_relpath")
+ local conffile_relpath="$1"; shift
+ local conffile_source="$1"
+ local release_conffile="$RELEASE_CONFIG_PATH/$conffile_relpath"
+ local etc_conffile="$CONFIG_PATH/$(basename "$conffile_relpath")"
# Note that -h can return true and -e will return false simultaneously
# when the target is a dangling symlink. We're okay with that outcome,
@@ -109,6 +102,27 @@ setup_conffile() {
fi
}
+prepare_database() {
+ DB_MIGRATE_STATUS=`$COMMAND_PREFIX bundle exec rake db:migrate:status 2>&1 || true`
+ if echo $DB_MIGRATE_STATUS | grep -qF 'Schema migrations table does not exist yet.'; then
+ # The database exists, but the migrations table doesn't.
+ run_and_report "Setting up database" $COMMAND_PREFIX bundle exec \
+ rake "$RAILSPKG_DATABASE_LOAD_TASK" db:seed
+ elif echo $DB_MIGRATE_STATUS | grep -q '^database: '; then
+ run_and_report "Running db:migrate" \
+ $COMMAND_PREFIX bundle exec rake db:migrate
+ elif echo $DB_MIGRATE_STATUS | grep -q 'database .* does not exist'; then
+ if ! run_and_report "Running db:setup" \
+ $COMMAND_PREFIX bundle exec rake db:setup 2>/dev/null; then
+ echo "Warning: unable to set up database." >&2
+ DATABASE_READY=0
+ fi
+ else
+ echo "Warning: Database is not ready to set up. Skipping database setup." >&2
+ DATABASE_READY=0
+ fi
+}
+
configure_version() {
WEB_SERVICE=${WEB_SERVICE:-$(service --status-all 2>/dev/null \
| grep -Eo '\bnginx|httpd[^[:space:]]*' || true)}
@@ -134,18 +148,20 @@ configure_version() {
fi
echo
- echo "Assumption: $WEB_SERVICE is configured to serve your SSO server URL from"
+ echo "Assumption: $WEB_SERVICE is configured to serve Rails from"
echo " $RELEASE_PATH"
- echo "Assumption: configuration files are in $CONFIG_PATH"
echo "Assumption: $WEB_SERVICE and passenger run as $WWW_OWNER"
echo
- echo -n "Symlinking files from $CONFIG_PATH ..."
+ echo -n "Creating symlinks to configuration in $CONFIG_PATH ..."
mkdir -p $CONFIG_PATH
- setup_conffile database.yml database.yml.example || DATABASE_READY=0
setup_conffile environments/production.rb environments/production.rb.example \
|| true
setup_conffile application.yml application.yml.example || APPLICATION_READY=0
+ if [ -n "$RAILSPKG_DATABASE_LOAD_TASK" ]; then
+ setup_conffile database.yml database.yml.example || DATABASE_READY=0
+ fi
+ setup_extra_conffiles
echo "... done."
# Before we do anything else, make sure some directories and files are in place
@@ -168,31 +184,25 @@ configure_version() {
# Ensure correct ownership of a few files
chown "$WWW_OWNER" $RELEASE_PATH/config/environment.rb
chown "$WWW_OWNER" $RELEASE_PATH/config.ru
- chown "$WWW_OWNER" $RELEASE_PATH/config/database.yml
chown "$WWW_OWNER" $RELEASE_PATH/Gemfile.lock
chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
chown -R "$WWW_OWNER" $SHARED_PATH/log
- chown "$WWW_OWNER" $RELEASE_PATH/db/schema.rb
+ case "$RAILSPKG_DATABASE_LOAD_TASK" in
+ db:schema:load) chown "$WWW_OWNER" $RELEASE_PATH/db/schema.rb ;;
+ db:structure:load) chown "$WWW_OWNER" $RELEASE_PATH/db/structure.sql ;;
+ esac
chmod 644 $SHARED_PATH/log/*
+ chmod -R 2775 $RELEASE_PATH/tmp
echo "... done."
- DB_MIGRATE_STATUS=`$COMMAND_PREFIX bundle exec rake db:migrate:status 2>&1 || true`
- if echo $DB_MIGRATE_STATUS | grep -qF 'Schema migrations table does not exist yet.'; then
- # The database exists, but the migrations table doesn't.
- run_and_report "Setting up database" \
- $COMMAND_PREFIX bundle exec rake db:schema:load db:seed
- elif echo $DB_MIGRATE_STATUS | grep -q '^database: '; then
- run_and_report "Running db:migrate" \
- $COMMAND_PREFIX bundle exec rake db:migrate
- elif echo $DB_MIGRATE_STATUS | grep -q 'database .* does not exist'; then
- if ! run_and_report "Running db:setup" \
- $COMMAND_PREFIX bundle exec rake db:setup 2>/dev/null; then
- echo "Warning: unable to set up database." >&2
- DATABASE_READY=0
- fi
- else
- echo "Warning: Database is not ready to set up. Skipping database setup." >&2
- DATABASE_READY=0
+ if [ -n "$RAILSPKG_DATABASE_LOAD_TASK" ]; then
+ chown "$WWW_OWNER" $RELEASE_PATH/config/database.yml
+ prepare_database
+ fi
+
+ if [ 11 = "$RAILSPKG_SUPPORTS_CONFIG_CHECK$APPLICATION_READY" ]; then
+ run_and_report "Checking application.yml for completeness" \
+ $COMMAND_PREFIX bundle exec rake config:check || APPLICATION_READY=0
fi
# precompile assets; thankfully this does not take long
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postrm b/jenkins/rails-package-scripts/postrm.sh
old mode 100755
new mode 100644
similarity index 79%
rename from jenkins/arvados-sso-server-extras/arvados-sso-server.postrm
rename to jenkins/rails-package-scripts/postrm.sh
index 070c90e..8c45d2f
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postrm
+++ b/jenkins/rails-package-scripts/postrm.sh
@@ -1,15 +1,12 @@
#!/bin/sh
+# This code runs after package variable definitions and step2.sh.
set -e
-INSTALL_PATH=/var/www/arvados-sso
-RELEASE_PATH=$INSTALL_PATH/current
-SHARED_PATH=$INSTALL_PATH/shared
-CONFIG_PATH=/etc/arvados/sso
-
purge () {
rm -rf $SHARED_PATH/vendor_bundle
rm -rf $SHARED_PATH/log
+ rm -rf $CONFIG_PATH
rmdir $SHARED_PATH || true
rmdir $INSTALL_PATH || true
}
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.prerm b/jenkins/rails-package-scripts/prerm.sh
old mode 100755
new mode 100644
similarity index 71%
rename from jenkins/arvados-sso-server-extras/arvados-sso-server.prerm
rename to jenkins/rails-package-scripts/prerm.sh
index 1a43fa2..4ef5904
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.prerm
+++ b/jenkins/rails-package-scripts/prerm.sh
@@ -1,20 +1,16 @@
#!/bin/sh
-
-set -e
-
-INSTALL_PATH=/var/www/arvados-sso
-RELEASE_PATH=$INSTALL_PATH/current
-SHARED_PATH=$INSTALL_PATH/shared
-CONFIG_PATH=/etc/arvados/sso
+# This code runs after package variable definitions and step2.sh.
remove () {
rm -f $RELEASE_PATH/config/database.yml
rm -f $RELEASE_PATH/config/environments/production.rb
rm -f $RELEASE_PATH/config/application.yml
+ # Old API server configuration file.
+ rm -f $RELEASE_PATH/config/initializers/omniauth.rb
rm -rf $RELEASE_PATH/public/assets/
rm -rf $RELEASE_PATH/tmp
rm -rf $RELEASE_PATH/.bundle
- rm $RELEASE_PATH/log || true
+ rm -rf $RELEASE_PATH/log
}
if [ "$1" = 'remove' ]; then
diff --git a/jenkins/rails-package-scripts/step2.sh b/jenkins/rails-package-scripts/step2.sh
new file mode 100644
index 0000000..6678d49
--- /dev/null
+++ b/jenkins/rails-package-scripts/step2.sh
@@ -0,0 +1,25 @@
+#!/bin/sh
+# This code runs after package variable definitions, before the actual
+# pre/post package work, to set some variable and function defaults.
+
+if [ -z "$INSTALL_PATH" ]; then
+ cat >&2 <<EOF
+
+PACKAGE BUILD ERROR: $0 is missing package metadata.
+
+This package is buggy. Please mail <support at curoverse.com> to let
+us know the name and version number of the package you tried to
+install, and we'll get it fixed.
+
+EOF
+ exit 3
+fi
+
+RELEASE_PATH=$INSTALL_PATH/current
+RELEASE_CONFIG_PATH=$RELEASE_PATH/config
+SHARED_PATH=$INSTALL_PATH/shared
+
+RAILSPKG_SUPPORTS_CONFIG_CHECK=${RAILSPKG_SUPPORTS_CONFIG_CHECK:-1}
+if ! type setup_extra_conffiles >/dev/null 2>&1; then
+ setup_extra_conffiles() { return; }
+fi
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index 2cee17f..eedb77b 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -169,73 +169,9 @@ if [[ ! -d "$WORKSPACE/packages/$TARGET" ]]; then
fi
# Build the SSO server package
-
-cd "$WORKSPACE"
-
-SSO_VERSION=$(version_from_git)
-PACKAGE_NAME=arvados-sso-server
-
-if [[ ! -d "$WORKSPACE/tmp" ]]; then
- mkdir $WORKSPACE/tmp
-fi
-
-if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
-fi
-
-/usr/bin/git rev-parse HEAD > git-commit.version
-
-cd $WORKSPACE/packages/$TARGET
-
-# Annoyingly, we require a database.yml file for rake assets:precompile to work.
-
-# TODO: add bogus database.yml file so we can precompile the assets and put them in the
-# package. Then remove that database.yml file again. It has to be a valid file though.
-#RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile
-
-# There are just 2 excludes left here, all the others are pulled in via fpm-info.sh, which
-# takes .gitignore into account via a call to git status:
-#
-# 1. The .git directory is excluded by git implicitly, so we can't pick it up from .gitignore.
-# 2. The packages directory needs to be explictly excluded here because it will only be listed
-# if it exists at the time fpm-info.sh runs. If it does not exist at that time, this script
-# will create it and when fpm runs, it will include the directory. So we add it to the exclude
-# list explicitly here, just in case.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados SSO server - Arvados is a free and open source platform for big data science.'" "--license='Expat License'" "-s" "dir" "-t" "$FORMAT" "-v" "$SSO_VERSION" "-x" "var/www/arvados-sso/current/.git" "-x" "var/www/arvados-sso/current/packages" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server.postinst" "--before-remove=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server.prerm" "--after-remove=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server.postrm" )
-
-if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- # This is the complete package with vendor/bundle included.
- # It's big, so we do not build it by default.
- COMMAND_ARR+=("-n" "${PACKAGE_NAME}-with-bundle")
-else
- # The default package excludes vendor/bundle
- COMMAND_ARR+=("-n" "${PACKAGE_NAME}" "-x" "var/www/arvados-sso/current/vendor/bundle")
-fi
-
-# Append --depends X and other arguments specified by fpm-info.sh in
-# the package source dir. These are added last so they can override
-# the arguments added by this script.
-# `--iteration 2` accommodates a postinst change on 2015-12-10.
-declare -a fpm_args=(--iteration 2)
-declare -a fpm_depends=()
-FPM_INFO="$WORKSPACE/fpm-info.sh"
-if [[ -e "$FPM_INFO" ]]; then
- debug_echo "Loading fpm overrides from $FPM_INFO"
- source "$FPM_INFO"
-fi
-
-for i in "${fpm_depends[@]}"; do
- COMMAND_ARR+=('--depends' "$i")
-done
-COMMAND_ARR+=("${fpm_args[@]}")
-COMMAND_ARR+=("$WORKSPACE/=/var/www/arvados-sso/current")
-debug_echo -e "\n${COMMAND_ARR[@]}\n"
-
-FPM_RESULTS=$("${COMMAND_ARR[@]}")
-FPM_EXIT_CODE=$?
-
-fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
-
-# SSO server package build done
+handle_rails_package arvados-sso-server "$WORKSPACE" \
+ "$WORKSPACE/LICENCE" --url="https://arvados.org" \
+ --description="Arvados SSO server - Arvados is a free and open source platform for big data science." \
+ --license="Expat license"
exit $EXITCODE
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index c6e658c..26486e9 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -412,114 +412,47 @@ for deppkg in "${PYTHON3_BACKPORTS[@]}"; do
done
# Build the API server package
-
-cd "$WORKSPACE/services/api"
-
-API_VERSION=$(version_from_git)
-PACKAGE_NAME=arvados-api-server
-
-if [[ ! -d "$WORKSPACE/services/api/tmp" ]]; then
- mkdir $WORKSPACE/services/api/tmp
-fi
-
-
-if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
-fi
-
-/usr/bin/git rev-parse HEAD > git-commit.version
-
-cd $WORKSPACE/packages/$TARGET
-
-# Annoyingly, we require a database.yml file for rake assets:precompile to work. So for now,
-# we do that in the upgrade script.
-# TODO: add bogus database.yml file so we can precompile the assets and put them in the
-# package. Then remove that database.yml file again. It has to be a valid file though.
-#RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile
-
-# This is the complete package with vendor/bundle included.
-# It's big, so we do not build it by default.
-if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$API_VERSION" "--iteration" "$(default_iteration "$PACKAGE_NAME" "$API_VERSION")" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh" "$WORKSPACE/agpl-3.0.txt=/var/www/arvados-api/current/agpl-3.0.txt")
-
- debug_echo -e "\n${COMMAND_ARR[@]}\n"
-
- FPM_RESULTS=$("${COMMAND_ARR[@]}")
- FPM_EXIT_CODE=$?
- fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
-fi
-
-# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$API_VERSION" "--iteration" "$(default_iteration "$PACKAGE_NAME" "$API_VERSION")" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh" "$WORKSPACE/agpl-3.0.txt=/var/www/arvados-api/current/agpl-3.0.txt")
-
-debug_echo -e "\n${COMMAND_ARR[@]}\n"
-
-FPM_RESULTS=$("${COMMAND_ARR[@]}")
-FPM_EXIT_CODE=$?
-fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
-
-# API server package build done
+handle_rails_package arvados-api-server "$WORKSPACE/services/api" \
+ "$WORKSPACE/agpl-3.0.txt" --url="https://arvados.org" \
+ --description="Arvados API server - Arvados is a free and open source platform for big data science." \
+ --license="GNU Affero General Public License, version 3.0"
# Build the workbench server package
+(
+ set -e
+ cd "$WORKSPACE/apps/workbench"
-cd "$WORKSPACE/apps/workbench"
-
-WORKBENCH_VERSION=$(version_from_git)
-PACKAGE_NAME=arvados-workbench
-
-if [[ ! -d "$WORKSPACE/apps/workbench/tmp" ]]; then
- mkdir $WORKSPACE/apps/workbench/tmp
-fi
-
-# We need to bundle to be ready even when we build a package without vendor directory
-# because asset compilation requires it.
-bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
+ # We need to bundle to be ready even when we build a package without vendor directory
+ # because asset compilation requires it.
+ bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
-/usr/bin/git rev-parse HEAD > git-commit.version
+ # clear the tmp directory; the asset generation step will recreate tmp/cache/assets,
+ # and we want that in the package, so it's easier to not exclude the tmp directory
+ # from the package - empty it instead.
+ rm -rf tmp
+ mkdir tmp
-# clear the tmp directory; the asset generation step will recreate tmp/cache/assets,
-# and we want that in the package, so it's easier to not exclude the tmp directory
-# from the package - empty it instead.
-rm -rf $WORKSPACE/apps/workbench/tmp/*
+ # Set up application.yml and production.rb so that asset precompilation works
+ \cp config/application.yml.example config/application.yml -f
+ \cp config/environments/production.rb.example config/environments/production.rb -f
+ sed -i 's/secret_token: ~/secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/' config/application.yml
-# Set up application.yml and production.rb so that asset precompilation works
-\cp config/application.yml.example config/application.yml -f
-\cp config/environments/production.rb.example config/environments/production.rb -f
-sed -i 's/secret_token: ~/secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/' config/application.yml
+ RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile >/dev/null
-RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile >/dev/null
+ # Remove generated configuration files so they don't go in the package.
+ rm config/application.yml config/environments/production.rb
+)
if [[ "$?" != "0" ]]; then
echo "ERROR: Asset precompilation failed"
EXITCODE=1
+else
+ handle_rails_package arvados-workbench "$WORKSPACE/apps/workbench" \
+ "$WORKSPACE/agpl-3.0.txt" --url="https://arvados.org" \
+ --description="Arvados Workbench - Arvados is a free and open source platform for big data science." \
+ --license="GNU Affero General Public License, version 3.0"
fi
-cd $WORKSPACE/packages/$TARGET
-
-# This is the complete package with vendor/bundle included.
-# It's big, so we do not build it by default.
-if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
-
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$WORKBENCH_VERSION" "--iteration" "$(default_iteration "$PACKAGE_NAME" "$WORKBENCH_VERSION")" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh" "$WORKSPACE/agpl-3.0.txt=/var/www/arvados-workbench/current/agpl-3.0.txt")
-
- debug_echo -e "\n${COMMAND_ARR[@]}\n"
-
- FPM_RESULTS=$("${COMMAND_ARR[@]}")
- FPM_EXIT_CODE=$?
- fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
-fi
-
-# Build the 'bare' package without vendor/bundle.
-
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$WORKBENCH_VERSION" "--iteration" "$(default_iteration "$PACKAGE_NAME" "$WORKBENCH_VERSION")" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh" "$WORKSPACE/agpl-3.0.txt=/var/www/arvados-workbench/current/agpl-3.0.txt")
-
-debug_echo -e "\n${COMMAND_ARR[@]}\n"
-
-FPM_RESULTS=$("${COMMAND_ARR[@]}")
-FPM_EXIT_CODE=$?
-fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
-
-# Workbench package build done
# clean up temporary GOPATH
rm -rf "$GOPATH"
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index 32a6099..623e844 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -114,6 +114,64 @@ default_iteration() {
echo $iteration
}
+_build_rails_package_scripts() {
+ local pkgname="$1"; shift
+ local destdir="$1"; shift
+ local srcdir="$RUN_BUILD_PACKAGES_PATH/rails-package-scripts"
+ for scriptname in postinst prerm postrm; do
+ cat "$srcdir/$pkgname.sh" "$srcdir/step2.sh" "$srcdir/$scriptname.sh" \
+ >"$destdir/$scriptname" || return $?
+ done
+}
+
+handle_rails_package() {
+ local pkgname="$1"; shift
+ local srcdir="$1"; shift
+ local license_path="$1"; shift
+ local scripts_dir="$(mktemp --tmpdir -d "$pkgname-XXXXXXXX.scripts")" && \
+ local version_file="$(mktemp --tmpdir "$pkgname-XXXXXXXX.version")" && (
+ set -e
+ _build_rails_package_scripts "$pkgname" "$scripts_dir"
+ cd "$srcdir"
+ mkdir -p tmp
+ version_from_git >"$version_file"
+ git rev-parse HEAD >git-commit.version
+ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
+ bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
+ fi
+ )
+ if [[ 0 != "$?" ]] || ! cd "$WORKSPACE/packages/$TARGET"; then
+ echo "ERROR: $pkgname package prep failed" >&2
+ rm -rf "$scripts_dir" "$version_file"
+ EXITCODE=1
+ return 1
+ fi
+ local railsdir="/var/www/${pkgname%-server}/current"
+ local -a pos_args=("$srcdir/=$railsdir" "$pkgname" "Curoverse, Inc." dir
+ "$(cat "$version_file")")
+ local license_arg="$license_path=$railsdir/$(basename "$license_path")"
+ # --iteration=3 accommodates the package scripts change from #8014.
+ local -a switches=(--iteration=3
+ --after-install "$scripts_dir/postinst"
+ --before-remove "$scripts_dir/prerm"
+ --after-remove "$scripts_dir/postrm")
+ # For some reason fpm excludes need to not start with /.
+ local exclude_root="${railsdir#/}"
+ # .git and packages are for the SSO server, which is built from its
+ # repository root.
+ for exclude in .git packages tmp log coverage \
+ vendor/cache/\* Capfile\* config/deploy\*; do
+ switches+=(-x "$exclude_root/$exclude")
+ done
+ fpm_build "${pos_args[@]}" "${switches[@]}" \
+ -x "$exclude_root/vendor/bundle" "$@" "$license_arg"
+ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
+ posargs[1]="$pkgname-with-bundle"
+ fpm_build "${pos_args[@]}" "${switches[@]}" "$@" "$license_arg"
+ fi
+ rm -rf "$scripts_dir" "$version_file"
+}
+
# Build packages for everything
fpm_build () {
# The package source. Depending on the source type, this can be a
commit 0814135175ab777abe515e4ba404dd0bf2929b24
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 1 19:09:08 2016 -0500
Fix quoting of local variable assignments throughout.
When you assign a variable directly (e.g., `FOO=bar`), you don't need
to quote the RHS, because the shell doesn't do expansion in that
case. But when you declare and assign a variable (e.g., `local
foo=bar`), you *do* potentially need to quote it, because `local` is a
command and all the normal expansion rules for running commands
applies.
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index 69ebb08..32a6099 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -28,7 +28,7 @@ EOF
}
format_last_commit_here() {
- local format=$1; shift
+ local format="$1"; shift
TZ=UTC git log -n1 --first-parent "--format=format:$format" .
}
@@ -59,8 +59,8 @@ handle_python_package () {
}
handle_ruby_gem() {
- local gem_name=$1; shift
- local gem_version=$(nohash_version_from_git)
+ local gem_name="$1"; shift
+ local gem_version="$(nohash_version_from_git)"
local gem_src_dir="$(pwd)"
if ! [[ -e "${gem_name}-${gem_version}.gem" ]]; then
@@ -76,7 +76,7 @@ package_go_binary() {
local src_path="$1"; shift
local prog="$1"; shift
local description="$1"; shift
- local license_file=${1:-agpl-3.0.txt}; shift
+ local license_file="${1:-agpl-3.0.txt}"; shift
debug_echo "package_go_binary $src_path as $prog"
@@ -86,8 +86,8 @@ package_go_binary() {
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
cd "$GOPATH/src/git.curoverse.com/arvados.git/$src_path"
- local version=$(version_from_git)
- local timestamp=$(timestamp_from_git)
+ local version="$(version_from_git)"
+ local timestamp="$(timestamp_from_git)"
# If the command imports anything from the Arvados SDK, bump the
# version number and build a new package whenever the SDK changes.
@@ -104,8 +104,8 @@ package_go_binary() {
}
default_iteration() {
- local package_name=$1; shift
- local package_version=$1; shift
+ local package_name="$1"; shift
+ local package_version="$1"; shift
local iteration=1
if [[ $package_version =~ ^0\.1\.([0-9]{14})(\.|$) ]] && \
[[ ${BASH_REMATCH[1]} -le $LICENSE_PACKAGE_TS ]]; then
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index a7b5fda..bdf541a 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -393,7 +393,7 @@ gem_uninstall_if_exists() {
}
setup_virtualenv() {
- local venvdest=$1; shift
+ local venvdest="$1"; shift
if ! [[ -e "$venvdest/bin/activate" ]] || ! [[ -e "$venvdest/bin/pip" ]]; then
virtualenv --setuptools "$@" "$venvdest" || fatal "virtualenv $venvdest failed"
fi
commit 9f73adf4a13fef44dd133b91c46693aea1af69df
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 1 13:38:00 2016 -0500
8014: Clean database state detection in arvados-sso postinst.
* Remove unused status code capture.
* Use `grep -q`. Previous versions of the script avoided it because
they piped directly from rake tasks, which would get upset when the
pipe broke. Now that we capture all the output, that's no longer a
concern.
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index 1ada186..e72d9b8 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -176,19 +176,15 @@ configure_version() {
chmod 644 $SHARED_PATH/log/*
echo "... done."
- set +e
- DB_MIGRATE_STATUS=`$COMMAND_PREFIX bundle exec rake db:migrate:status 2>&1`
- DB_MIGRATE_STATUS_CODE=$?
- set -e
-
- if echo $DB_MIGRATE_STATUS | grep 'Schema migrations table does not exist yet.' >/dev/null; then
+ DB_MIGRATE_STATUS=`$COMMAND_PREFIX bundle exec rake db:migrate:status 2>&1 || true`
+ if echo $DB_MIGRATE_STATUS | grep -qF 'Schema migrations table does not exist yet.'; then
# The database exists, but the migrations table doesn't.
run_and_report "Setting up database" \
$COMMAND_PREFIX bundle exec rake db:schema:load db:seed
- elif echo $DB_MIGRATE_STATUS | grep '^database: ' >/dev/null; then
+ elif echo $DB_MIGRATE_STATUS | grep -q '^database: '; then
run_and_report "Running db:migrate" \
$COMMAND_PREFIX bundle exec rake db:migrate
- elif echo $DB_MIGRATE_STATUS | grep 'database .* does not exist' >/dev/null; then
+ elif echo $DB_MIGRATE_STATUS | grep -q 'database .* does not exist'; then
if ! run_and_report "Running db:setup" \
$COMMAND_PREFIX bundle exec rake db:setup 2>/dev/null; then
echo "Warning: unable to set up database." >&2
commit c1c5b145f832bd9a647c6e07e19f148cf26ff910
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 1 12:26:20 2016 -0500
8014: Introduce run_and_report function to arvados-sso postinst.
Use this to DRY up code that says "Doing something... done."
This commit removes some `|| exit $?` that's redundant with `set -e`.
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index 530de46..1ada186 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -54,6 +54,23 @@ For RPM-based systems, then reinstall this package.
EOF
}
+run_and_report() {
+ # Usage: run_and_report ACTION_MSG CMD
+ # This is the usual wrapper that prints ACTION_MSG, runs CMD, then writes
+ # a message about whether CMD succeeded or failed. Returns the exit code
+ # of CMD.
+ local action_message=$1; shift
+ local retcode=0
+ echo -n "$action_message..."
+ if "$@"; then
+ echo " done."
+ else
+ retcode=$?
+ echo " failed."
+ fi
+ return $retcode
+}
+
setup_conffile() {
# Usage: setup_conffile CONFFILE_PATH [SOURCE_PATH]
# Both paths are relative to RELEASE_CONFIG_PATH.
@@ -140,18 +157,12 @@ configure_version() {
cd "$RELEASE_PATH"
export RAILS_ENV=production
- echo "Making sure bundle is installed ..."
- set +e
- which bundle > /dev/null
- if [[ "$?" != "0" ]]; then
- $COMMAND_PREFIX gem install bundle
+ if ! $COMMAND_PREFIX bundle --version >/dev/null; then
+ run_and_report "Installing bundle" $COMMAND_PREFIX gem install bundle
fi
- set -e
- echo "... done."
- echo -n "Running bundle install ..."
- $COMMAND_PREFIX bundle install --path $SHARED_PATH/vendor_bundle --quiet || exit $?
- echo "... done."
+ run_and_report "Running bundle install" \
+ $COMMAND_PREFIX bundle install --path $SHARED_PATH/vendor_bundle --quiet
echo -n "Ensuring directory and file permissions ..."
# Ensure correct ownership of a few files
@@ -171,52 +182,35 @@ configure_version() {
set -e
if echo $DB_MIGRATE_STATUS | grep 'Schema migrations table does not exist yet.' >/dev/null; then
- # The database exists, but the migrations table doesn't.
- echo -n "Setting up database ..."
- $COMMAND_PREFIX bundle exec rake db:schema:load db:seed || exit $?
- echo "... done."
+ # The database exists, but the migrations table doesn't.
+ run_and_report "Setting up database" \
+ $COMMAND_PREFIX bundle exec rake db:schema:load db:seed
elif echo $DB_MIGRATE_STATUS | grep '^database: ' >/dev/null; then
- echo -n "Running db:migrate ..."
- $COMMAND_PREFIX bundle exec rake db:migrate || exit $?
- echo "... done."
+ run_and_report "Running db:migrate" \
+ $COMMAND_PREFIX bundle exec rake db:migrate
elif echo $DB_MIGRATE_STATUS | grep 'database .* does not exist' >/dev/null; then
- echo -n "Running db:setup ..."
- set +e
- $COMMAND_PREFIX bundle exec rake db:setup 2>/dev/null
- if [ "$?" = "0" ]; then
- echo "... done."
- else
- echo "... failed."
- echo "Warning: unable to set up database." >&2
- DATABASE_READY=0
- fi
- set -e
+ if ! run_and_report "Running db:setup" \
+ $COMMAND_PREFIX bundle exec rake db:setup 2>/dev/null; then
+ echo "Warning: unable to set up database." >&2
+ DATABASE_READY=0
+ fi
else
echo "Warning: Database is not ready to set up. Skipping database setup." >&2
DATABASE_READY=0
fi
- echo -n "Precompiling assets ..."
# precompile assets; thankfully this does not take long
if [ "$APPLICATION_READY" = "1" ]; then
- set +e
- $COMMAND_PREFIX bundle exec rake assets:precompile -q -s 2>/dev/null
- if [ "$?" = "0" ]; then
- echo "... done."
- else
- echo "... failed."
- APPLICATION_READY=0
- fi
- set -e
+ run_and_report "Precompiling assets" \
+ $COMMAND_PREFIX bundle exec rake assets:precompile -q -s 2>/dev/null \
+ || APPLICATION_READY=0
else
- echo "... skipped."
+ echo "Precompiling assets... skipped."
fi
chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
if [ ! -z "$WEB_SERVICE" ]; then
- echo -n "Restarting $WEB_SERVICE ..."
- service "$WEB_SERVICE" restart >/dev/null || exit $?
- echo "... done."
+ service "$WEB_SERVICE" restart
fi
}
commit 9fb392b2eaa59f20f7186de726531d922408dfa1
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 1 12:17:01 2016 -0500
8014: Improve conffile handling in arvados-sso postinst.
* DRY it up into a function.
* DATABASE_READY and APPLICATION_READY care about whether the
corresponding .yml file is usable. Always detect when it is
unmodified from the .yml.example file.
* Build symlinks from /var/www to /etc in more cases where it's safe.
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index 3631bbe..530de46 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -4,6 +4,7 @@ set -e
INSTALL_PATH=/var/www/arvados-sso
RELEASE_PATH=$INSTALL_PATH/current
+RELEASE_CONFIG_PATH=$RELEASE_PATH/config
SHARED_PATH=$INSTALL_PATH/shared
CONFIG_PATH=/etc/arvados/sso
PACKAGE_NAME=arvados-sso-server
@@ -53,6 +54,44 @@ For RPM-based systems, then reinstall this package.
EOF
}
+setup_conffile() {
+ # Usage: setup_conffile CONFFILE_PATH [SOURCE_PATH]
+ # Both paths are relative to RELEASE_CONFIG_PATH.
+ # This function will try to safely ensure that a symbolic link for
+ # the configuration file points from RELEASE_CONFIG_PATH to CONFIG_PATH.
+ # If SOURCE_PATH is given, this function will try to install that file as
+ # the configuration file in CONFIG_PATH, and return 1 if the file in
+ # CONFIG_PATH is unmodified from the source.
+ local conffile_relpath=$1; shift
+ local conffile_source=$1; shift
+ local release_conffile=$RELEASE_CONFIG_PATH/$conffile_relpath
+ local etc_conffile=$CONFIG_PATH/$(basename "$conffile_relpath")
+
+ # Note that -h can return true and -e will return false simultaneously
+ # when the target is a dangling symlink. We're okay with that outcome,
+ # so check -h first.
+ if [ ! -h "$release_conffile" ]; then
+ if [ ! -e "$release_conffile" ]; then
+ ln -s "$etc_conffile" "$release_conffile"
+ # If there's a config file in /var/www identical to the one in /etc,
+ # overwrite it with a symlink.
+ elif cmp --quiet "$release_conffile" "$etc_conffile"; then
+ ln --force -s "$etc_conffile" "$release_conffile"
+ fi
+ fi
+
+ if [ -n "$conffile_source" ]; then
+ cp --no-clobber "$RELEASE_CONFIG_PATH/$conffile_source" "$etc_conffile"
+ # Even if $etc_conffile already existed, it might be unmodified from
+ # the source. This is especially likely when a user installs, updates
+ # database.yml, then reconfigures before they update application.yml.
+ # Use cmp to be sure whether $etc_conffile is modified.
+ if cmp --quiet "$RELEASE_CONFIG_PATH/$conffile_source" "$etc_conffile"; then
+ return 1
+ fi
+ fi
+}
+
configure_version() {
WEB_SERVICE=${WEB_SERVICE:-$(service --status-all 2>/dev/null \
| grep -Eo '\bnginx|httpd[^[:space:]]*' || true)}
@@ -85,36 +124,11 @@ configure_version() {
echo
echo -n "Symlinking files from $CONFIG_PATH ..."
-
- if [ ! -f $CONFIG_PATH/database.yml ]; then
- mkdir -p $CONFIG_PATH
- cp $RELEASE_PATH/config/database.yml.example $CONFIG_PATH/database.yml
- DATABASE_READY=0
- fi
-
- if [ ! -h $RELEASE_PATH/config/database.yml ]; then
- ln -s $CONFIG_PATH/database.yml $RELEASE_PATH/config/database.yml
- fi
-
- if [ ! -f $CONFIG_PATH/production.rb ]; then
- mkdir -p $CONFIG_PATH
- cp $RELEASE_PATH/config/environments/production.rb.example $CONFIG_PATH/production.rb
- fi
-
- if [ ! -h $RELEASE_PATH/config/environments/production.rb ]; then
- ln -s $CONFIG_PATH/production.rb $RELEASE_PATH/config/environments/production.rb
- fi
-
- if [ ! -f $CONFIG_PATH/application.yml ]; then
- mkdir -p $CONFIG_PATH
- cp $RELEASE_PATH/config/application.yml.example $CONFIG_PATH/application.yml
- APPLICATION_READY=0
- fi
-
- if [ ! -h $RELEASE_PATH/config/application.yml ]; then
- ln -s $CONFIG_PATH/application.yml $RELEASE_PATH/config/application.yml
- fi
-
+ mkdir -p $CONFIG_PATH
+ setup_conffile database.yml database.yml.example || DATABASE_READY=0
+ setup_conffile environments/production.rb environments/production.rb.example \
+ || true
+ setup_conffile application.yml application.yml.example || APPLICATION_READY=0
echo "... done."
# Before we do anything else, make sure some directories and files are in place
commit 1c0ce486e11b13ef5cd10959d4eca1fd948d88d1
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 1 11:10:05 2016 -0500
8014: arvados-sso postinst recognizes versioned Nginx service names.
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index 05b331e..3631bbe 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -65,7 +65,8 @@ configure_version() {
fi
if [ -e /etc/redhat-release ]; then
- if [ "$WEB_SERVICE" = "nginx" ]; then
+ # Recognize any service that starts with "nginx"; e.g., nginx16.
+ if [ "$WEB_SERVICE" != "${WEB_SERVICE#nginx}" ]; then
WWW_OWNER=nginx:nginx
else
WWW_OWNER=apache:apache
commit 26170766797e80d0c86c25a10331f43c6d0eb283
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 1 11:02:46 2016 -0500
8014: Refactor web service detection warnings in arvados-sso postinst.
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index ce15782..05b331e 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -37,35 +37,31 @@ EOF
fi
}
-configure_version() {
- WEB_SERVICE=${WEB_SERVICE:-$(service --status-all 2>/dev/null \
- | grep -Eo '\bnginx|httpd[^[:space:]]*' || true)}
- if [ -z "$WEB_SERVICE" ]; then
+report_web_service_warning() {
+ local warning=$1; shift
cat >&2 <<EOF
-Warning: web service (Nginx or Apache) not found.
+WARNING: $warning.
To override, set the WEB_SERVICE environment variable to the name of the service
-hosting the Rails server. Alternativey, install nginx.
+hosting the Rails server.
For Debian-based systems, then reconfigure this package with dpkg-reconfigure.
For RPM-based systems, then reinstall this package.
EOF
+}
+
+configure_version() {
+ WEB_SERVICE=${WEB_SERVICE:-$(service --status-all 2>/dev/null \
+ | grep -Eo '\bnginx|httpd[^[:space:]]*' || true)}
+ if [ -z "$WEB_SERVICE" ]; then
+ report_web_service_warning "Web service (Nginx or Apache) not found"
elif [ "$WEB_SERVICE" != "$(echo "$WEB_SERVICE" | head -n 1)" ]; then
WEB_SERVICE=$(echo "$WEB_SERVICE" | head -n 1)
- cat >&2 <<EOF
-Warning: multiple web services found. Choosing the first one ($WEB_SERVICE).
-
-To override, set the WEB_SERVICE environment variable to the name of the service
-hosting the Rails server.
-
-For Debian-based systems, then reconfigure this package with dpkg-reconfigure.
-
-For RPM-based systems, then reinstall this package.
-
-EOF
+ report_web_service_warning \
+ "Multiple web services found. Choosing the first one ($WEB_SERVICE)"
fi
if [ -e /etc/redhat-release ]; then
commit 46ae733bd885da0d75882831af822c9578621730
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 1 10:54:00 2016 -0500
8014: Simplify COMMAND_PREFIX-setting code.
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index f7f1ee4..ce15782 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -12,6 +12,12 @@ DOC_URL="http://doc.arvados.org/install/install-sso.html#configure"
DATABASE_READY=1
APPLICATION_READY=1
+if [ -s "$HOME/.rvm/scripts/rvm" ] || [ -s "/usr/local/rvm/scripts/rvm" ]; then
+ COMMAND_PREFIX="/usr/local/rvm/bin/rvm-exec default"
+else
+ COMMAND_PREFIX=
+fi
+
report_not_ready() {
local ready_flag=$1; shift
local config_file=$1; shift
@@ -31,22 +37,6 @@ EOF
fi
}
-setup_ruby_environment() {
- if [ -s "$HOME/.rvm/scripts/rvm" ] ; then
- using_rvm="true"
- elif [ -s "/usr/local/rvm/scripts/rvm" ] ; then
- using_rvm="true"
- else
- using_rvm="false"
- fi
-
- if [ "$using_rvm" = "true" ]; then
- COMMAND_PREFIX="/usr/local/rvm/bin/rvm-exec default"
- else
- COMMAND_PREFIX=
- fi
-}
-
configure_version() {
WEB_SERVICE=${WEB_SERVICE:-$(service --status-all 2>/dev/null \
| grep -Eo '\bnginx|httpd[^[:space:]]*' || true)}
@@ -221,11 +211,9 @@ EOF
if [ "$1" = configure ]; then
# This is a debian-based system
- setup_ruby_environment
configure_version
elif [ "$1" = "0" ] || [ "$1" = "1" ] || [ "$1" = "2" ]; then
# This is an rpm-based system
- setup_ruby_environment
configure_version
fi
commit 2bee50fa0b09c9dd785fe6d6beafe21fa961111d
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 1 10:51:01 2016 -0500
8014: Refactor "not fully configured" messages in arvados-sso postinst.
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index 070796e..f7f1ee4 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -7,10 +7,30 @@ RELEASE_PATH=$INSTALL_PATH/current
SHARED_PATH=$INSTALL_PATH/shared
CONFIG_PATH=/etc/arvados/sso
PACKAGE_NAME=arvados-sso-server
+DOC_URL="http://doc.arvados.org/install/install-sso.html#configure"
DATABASE_READY=1
APPLICATION_READY=1
+report_not_ready() {
+ local ready_flag=$1; shift
+ local config_file=$1; shift
+ if [ "1" != "$ready_flag" ]; then cat >&2 <<EOF
+
+PLEASE NOTE:
+
+The $PACKAGE_NAME package was not configured completely because
+$config_file needs some tweaking.
+Please refer to the documentation at
+<$DOC_URL> for more details.
+
+When $(basename "$config_file") has been modified,
+reconfigure or reinstall this package.
+
+EOF
+ fi
+}
+
setup_ruby_environment() {
if [ -s "$HOME/.rvm/scripts/rvm" ] ; then
using_rvm="true"
@@ -209,32 +229,5 @@ elif [ "$1" = "0" ] || [ "$1" = "1" ] || [ "$1" = "2" ]; then
configure_version
fi
-if [ "$DATABASE_READY" = "0" ]; then
- cat <<EOF
-
-PLEASE NOTE:
-
-The $PACKAGE_NAME package was not configured completely because
-$CONFIG_PATH/database.yml needs some tweaking. Please refer to the
-documentation at http://doc.arvados.org/install/install-sso.html#configure for
-more details.
-
-When database.yml has been modified, reconfigure or reinstall this package.
-EOF
-fi
-
-if [ "$APPLICATION_READY" = "0" ]; then
- cat <<EOF
-
-PLEASE NOTE:
-
-The $PACKAGE_NAME package was not configured completely because
-$CONFIG_PATH/application.yml needs some tweaking. Please refer to the
-documentation at http://doc.arvados.org/install/install-sso.html#configure for
-more details.
-
-When application.yml has been modified, reconfigure or reinstall this package.
-EOF
-fi
-
-echo
+report_not_ready "$DATABASE_READY" "$CONFIG_PATH/database.yml"
+report_not_ready "$APPLICATION_READY" "$CONFIG_PATH/application.yml"
commit 33f530c93b5bd397c6c9569385613915acf4cdbb
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 1 10:41:23 2016 -0500
8014: Remove unused $VERSION tracking from arvados-sso postinst.
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index 7423e5b..070796e 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -11,8 +11,6 @@ PACKAGE_NAME=arvados-sso-server
DATABASE_READY=1
APPLICATION_READY=1
-VERSION=`cat $RELEASE_PATH/git-commit.version`
-
setup_ruby_environment() {
if [ -s "$HOME/.rvm/scripts/rvm" ] ; then
using_rvm="true"
@@ -29,12 +27,7 @@ setup_ruby_environment() {
fi
}
-# arguments: <major version> <most recently configured package version>
configure_version() {
- VERSION="$1"
-
- [ "$VERSION" ] || { echo "Error: configure_version: need version parameter" >&2; exit 1; }
-
WEB_SERVICE=${WEB_SERVICE:-$(service --status-all 2>/dev/null \
| grep -Eo '\bnginx|httpd[^[:space:]]*' || true)}
if [ -z "$WEB_SERVICE" ]; then
@@ -209,11 +202,11 @@ EOF
if [ "$1" = configure ]; then
# This is a debian-based system
setup_ruby_environment
- configure_version $VERSION "$2"
+ configure_version
elif [ "$1" = "0" ] || [ "$1" = "1" ] || [ "$1" = "2" ]; then
# This is an rpm-based system
setup_ruby_environment
- configure_version $VERSION
+ configure_version
fi
if [ "$DATABASE_READY" = "0" ]; then
commit 4735c45083770bf8344dd80dd84695bb4e0fd169
Author: Brett Smith <brett at curoverse.com>
Date: Thu Dec 31 15:58:18 2015 -0500
8014: Remove unused sso-server-upgrade.sh script.
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server-upgrade.sh b/jenkins/arvados-sso-server-extras/arvados-sso-server-upgrade.sh
deleted file mode 100755
index 1a377dc..0000000
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server-upgrade.sh
+++ /dev/null
@@ -1,91 +0,0 @@
-#!/bin/bash
-
-set -e
-
-if [ -e /etc/redhat-release ]; then
- WWW_OWNER=nginx:nginx
-else
- # Assume we're on a Debian-based system for now.
- WWW_OWNER=www-data:www-data
-fi
-
-NGINX_SERVICE=${NGINX_SERVICE:-$(service --status-all 2>/dev/null \
- | grep -Eo '\bnginx[^[:space:]]*' || true)}
-if [ -z "$NGINX_SERVICE" ]; then
- cat >&2 <<EOF
-Error: nginx service not found. Aborting.
-Set NGINX_SERVICE to the name of the service hosting the Rails server.
-EOF
- exit 1
-elif [ "$NGINX_SERVICE" != "$(echo "$NGINX_SERVICE" | head -n 1)" ]; then
- cat >&2 <<EOF
-Error: multiple nginx services found. Aborting.
-Set NGINX_SERVICE to the name of the service hosting the Rails server.
-EOF
- exit 1
-fi
-
-RELEASE_PATH=/var/www/arvados-sso/current
-SHARED_PATH=/var/www/arvados-sso/shared
-CONFIG_PATH=/etc/arvados/sso/
-
-echo
-echo "Assumption: $NGINX_SERVICE is configured to serve your SSO server URL from"
-echo " /var/www/arvados-sso/current"
-echo "Assumption: configuration files are in /etc/arvados/sso/"
-echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
-echo
-
-echo "Copying files from $CONFIG_PATH ..."
-cp -f $CONFIG_PATH/database.yml $RELEASE_PATH/config/database.yml
-cp -f $RELEASE_PATH/config/environments/production.rb.example $RELEASE_PATH/config/environments/production.rb
-cp -f $CONFIG_PATH/application.yml $RELEASE_PATH/config/application.yml
-echo "... done."
-
-# Before we do anything else, make sure some directories and files are in place
-if [[ ! -e $SHARED_PATH/log ]]; then mkdir -p $SHARED_PATH/log; fi
-if [[ ! -e $RELEASE_PATH/tmp ]]; then mkdir -p $RELEASE_PATH/tmp; fi
-if [[ ! -e $RELEASE_PATH/log ]]; then ln -s $SHARED_PATH/log $RELEASE_PATH/log; fi
-if [[ ! -e $SHARED_PATH/log/production.log ]]; then touch $SHARED_PATH/log/production.log; fi
-
-cd "$RELEASE_PATH"
-export RAILS_ENV=production
-
-echo "Running bundle install ..."
-bundle install --path $SHARED_PATH/vendor_bundle --quiet
-echo "... done."
-
-echo "Ensuring directory and file permissions ..."
-# Ensure correct ownership of a few files
-chown "$WWW_OWNER" $RELEASE_PATH/config/environment.rb
-chown "$WWW_OWNER" $RELEASE_PATH/config.ru
-chown "$WWW_OWNER" $RELEASE_PATH/config/database.yml
-chown "$WWW_OWNER" $RELEASE_PATH/Gemfile.lock
-chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
-chown -R "$WWW_OWNER" $SHARED_PATH/log
-chown "$WWW_OWNER" $RELEASE_PATH/db/schema.rb
-chmod 644 $SHARED_PATH/log/*
-echo "... done."
-
-# If we use `grep -q`, rake will write a backtrace on EPIPE.
-if bundle exec rake db:migrate:status | grep '^database: ' >/dev/null; then
- echo "Starting db:migrate ..."
- bundle exec rake db:migrate
-elif [ 0 -eq ${PIPESTATUS[0]} ]; then
- # The database exists, but the migrations table doesn't.
- echo "Setting up database ..."
- bundle exec rake db:schema:load db:seed
-else
- echo "Error: Database is not ready to set up. Aborting." >&2
- exit 1
-fi
-echo "... done."
-
-echo "Precompiling assets ..."
-# precompile assets; thankfully this does not take long
-bundle exec rake assets:precompile -q -s
-echo "... done."
-
-echo "Restarting nginx ..."
-service "$NGINX_SERVICE" restart
-echo "... done."
commit 886c3c5c57c6419aceb390aea88c5886f43bbf42
Author: Brett Smith <brett at curoverse.com>
Date: Sat Jan 9 22:34:18 2016 -0500
7991: pip build process detects if --no-use-wheel is supported.
Refs #7991.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 72cc476..c6e658c 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -359,6 +359,15 @@ fpm_build $LIBCLOUD_DIR "$PYTHON2_PKG_PREFIX"-apache-libcloud
rm -rf $LIBCLOUD_DIR
# Python 2 dependencies
+declare -a PIP_DOWNLOAD_SWITCHES=(--no-deps)
+# Add --no-use-wheel if this pip knows it.
+pip wheel --help >/dev/null 2>&1
+case "$?" in
+ 0) PIP_DOWNLOAD_SWITCHES+=(--no-use-wheel) ;;
+ 2) ;;
+ *) echo "WARNING: `pip wheel` test returned unknown exit code $?" ;;
+esac
+
for deppkg in "${PYTHON_BACKPORTS[@]}"; do
outname=$(echo "$deppkg" | sed -e 's/^python-//' -e 's/[<=>].*//' -e 's/_/-/g' -e "s/^/${PYTHON2_PKG_PREFIX}-/")
case "$deppkg" in
@@ -368,7 +377,7 @@ for deppkg in "${PYTHON_BACKPORTS[@]}"; do
pyfpm_workdir=$(mktemp --tmpdir -d pyfpm-XXXXXX) && (
set -e
cd "$pyfpm_workdir"
- pip install --no-deps --no-use-wheel --download . "$deppkg"
+ pip install "${PIP_DOWNLOAD_SWITCHES[@]}" --download . "$deppkg"
tar -xf "$deppkg"-*.tar*
cd "$deppkg"-*/
"python$PYTHON2_VERSION" setup.py $DASHQ_UNLESS_DEBUG egg_info build
commit 5560d70bf29553b1d0719dda060d298de7bfc338
Author: Brett Smith <brett at curoverse.com>
Date: Sat Jan 9 22:33:34 2016 -0500
7991: Update error message for pip build errors.
Refs #7991.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 6f8af19..72cc476 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -382,7 +382,7 @@ for deppkg in "${PYTHON_BACKPORTS[@]}"; do
mv --no-clobber "$outname"*.$FORMAT "$WORKSPACE/packages/$TARGET"
)
if [ 0 != "$?" ]; then
- echo "ERROR: httplib2 build process failed"
+ echo "ERROR: $deppkg build process failed"
EXITCODE=1
fi
if [ -n "$pyfpm_workdir" ]; then
commit d42d4368192ec140faf161db6c98a500aa39fdcd
Merge: d42db2e e88afb9
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 8 17:20:05 2016 -0500
Merge branch '7991-google-api-python-client-perms-wip'
Closes #7991, #8136.
commit e88afb9fdcdd8b9659ad68021987fd71269d7ff4
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jan 8 15:58:28 2016 -0500
7991: Fix permissions of google-api-python-client egg info files.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index fc835b7..6f8af19 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -362,21 +362,20 @@ rm -rf $LIBCLOUD_DIR
for deppkg in "${PYTHON_BACKPORTS[@]}"; do
outname=$(echo "$deppkg" | sed -e 's/^python-//' -e 's/[<=>].*//' -e 's/_/-/g' -e "s/^/${PYTHON2_PKG_PREFIX}-/")
case "$deppkg" in
- httplib2)
- # Work around 0640 permissions on some package files on Debian 8
- # and Ubuntu 14.04. See #7591.
- httplib2_workdir=$(mktemp -d httplib2-XXXXXX) && (
+ httplib2|google-api-python-client)
+ # Work around 0640 permissions on some package files.
+ # See #7591 and #7991.
+ pyfpm_workdir=$(mktemp --tmpdir -d pyfpm-XXXXXX) && (
set -e
- cd "$httplib2_workdir"
- pip install --download . httplib2
- tar -xf httplib2-*.tar*
- cd httplib2-*/
+ cd "$pyfpm_workdir"
+ pip install --no-deps --no-use-wheel --download . "$deppkg"
+ tar -xf "$deppkg"-*.tar*
+ cd "$deppkg"-*/
"python$PYTHON2_VERSION" setup.py $DASHQ_UNLESS_DEBUG egg_info build
chmod -R go+rX .
set +e
# --iteration 2 provides an upgrade for previously built
- # buggy packages. Arguments past $outname can be removed
- # once we're building httplib2 > 0.9.2.
+ # buggy packages.
fpm_build . "$outname" "" python "" --iteration 2
# The upload step uses the package timestamp to determine
# whether it's new. --no-clobber plays nice with that.
@@ -386,8 +385,8 @@ for deppkg in "${PYTHON_BACKPORTS[@]}"; do
echo "ERROR: httplib2 build process failed"
EXITCODE=1
fi
- if [ -n "$httplib2_workdir" ]; then
- rm -rf "$httplib2_workdir"
+ if [ -n "$pyfpm_workdir" ]; then
+ rm -rf "$pyfpm_workdir"
fi
;;
*)
commit d42db2ef8638690a1a3e0f4e66ec898d529c8736
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jan 6 19:38:28 2016 -0500
Make sure not to silence package installation during the test-packages
stage, so that we can see actual failure messages if there is a problem.
No issue #
diff --git a/jenkins/deb-common-test-packages.sh b/jenkins/deb-common-test-packages.sh
index d9d0301..4bc4573 100755
--- a/jenkins/deb-common-test-packages.sh
+++ b/jenkins/deb-common-test-packages.sh
@@ -5,7 +5,7 @@
target=$(echo $0 | sed 's/.*test-packages-\([^.]*\)\.sh.*/\1/')
apt-get -qq update
-if ! apt-get -qq --assume-yes --force-yes install \
+if ! apt-get --assume-yes --force-yes install \
python-arvados-python-client python-arvados-fuse arvados-node-manager
then
exit 1
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index 7a13982..0d956af 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -2,7 +2,7 @@
yum -q clean all
touch /var/lib/rpm/*
-if ! yum -q install --assumeyes \
+if ! yum install --assumeyes \
python27-python-arvados-python-client python27-python-arvados-fuse arvados-node-manager
then
exit 1
commit fff5527a9d917345367f08f0e2fe2dbb81e00187
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jan 6 09:36:14 2016 -0500
8104: Pin the pycurl backport version.
Refs #8104.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 6369087..fc835b7 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -96,7 +96,8 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
+ ciso8601 pycrypto backports.ssl_match_hostname llfuse \
+ 'pycurl<7.21.5')
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
debian8)
@@ -108,7 +109,8 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
+ ciso8601 pycrypto backports.ssl_match_hostname llfuse \
+ 'pycurl<7.21.5')
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
ubuntu1204)
@@ -120,7 +122,8 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
+ ciso8601 pycrypto backports.ssl_match_hostname llfuse \
+ 'pycurl<7.21.5')
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
ubuntu1404)
@@ -131,7 +134,7 @@ case "$TARGET" in
PYTHON3_PKG_PREFIX=python3
PYTHON_BACKPORTS=(pyasn1==0.1.7 pyvcf pyasn1-modules==0.0.5 llfuse ciso8601 \
google-api-python-client six uritemplate oauth2client httplib2 \
- rsa pycurl backports.ssl_match_hostname)
+ rsa 'pycurl<7.21.5' backports.ssl_match_hostname)
PYTHON3_BACKPORTS=(docker-py requests websocket-client)
;;
centos6)
@@ -143,7 +146,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname pycurl
+ ciso8601 pycrypto backports.ssl_match_hostname 'pycurl<7.21.5' \
python-daemon lockfile llfuse 'pbr<1.0')
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
export PYCURL_SSL_LIBRARY=nss
commit bbed885afb9007c5d81ff55af70e8d7be92f8f37
Merge: 6db0d02 575f751
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Dec 14 16:56:28 2015 -0500
Merge branch '7901-crunchstat-summary' refs #7901
commit 575f75167b48977b3c825e30f944fca70a74f901
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Dec 10 16:00:17 2015 -0500
7901: Add crunchstat-summary
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 89e75a8..a7b5fda 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -78,6 +78,7 @@ sdk/go/manifest
sdk/go/blockdigest
sdk/go/streamer
sdk/go/crunchrunner
+tools/crunchstat-summary
tools/keep-rsync
EOF
@@ -642,6 +643,7 @@ pythonstuff=(
sdk/python
services/fuse
services/nodemanager
+ tools/crunchstat-summary
)
for p in "${pythonstuff[@]}"
do
commit 6db0d02823e4721272a9492c2a03bb16afb56469
Author: Brett Smith <brett at curoverse.com>
Date: Thu Dec 10 15:52:32 2015 -0500
7961: SSO postinst script uses bash.
Closes #7961.
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index d39cdd4..7423e5b 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
set -e
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index 2550135..2cee17f 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -215,7 +215,8 @@ fi
# Append --depends X and other arguments specified by fpm-info.sh in
# the package source dir. These are added last so they can override
# the arguments added by this script.
-declare -a fpm_args=()
+# `--iteration 2` accommodates a postinst change on 2015-12-10.
+declare -a fpm_args=(--iteration 2)
declare -a fpm_depends=()
FPM_INFO="$WORKSPACE/fpm-info.sh"
if [[ -e "$FPM_INFO" ]]; then
commit e7a2d57f559f84d9a2181526e3a4d6ca6082f3c4
Author: Brett Smith <brett at curoverse.com>
Date: Wed Dec 9 12:43:58 2015 -0500
7953: Update test procedure for CentOS 6 packages.
Don't install any Software Collections in the Dockerfile, just the
Software Collections tools. Test that our packages correctly declare
their dependency on the relevant Software Collection, and that it gets
pulled in. We want to tell users to install this way, so this makes
sure it's working.
Refs #7953, #7957.
diff --git a/jenkins/package-test-dockerfiles/centos6/Dockerfile b/jenkins/package-test-dockerfiles/centos6/Dockerfile
index 814ad47..addd259 100644
--- a/jenkins/package-test-dockerfiles/centos6/Dockerfile
+++ b/jenkins/package-test-dockerfiles/centos6/Dockerfile
@@ -1,7 +1,6 @@
FROM centos:6
MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
-RUN yum -q install --assumeyes scl-utils centos-release-SCL && \
- yum -q install --assumeyes python27
+RUN yum -q install --assumeyes scl-utils centos-release-SCL
COPY localrepo.repo /etc/yum.repos.d/localrepo.repo
\ No newline at end of file
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index 1aca953..7a13982 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -1,29 +1,25 @@
#!/bin/bash
-if ! python --version |& grep "Python 2.7" >/dev/null ; then
- # Re-execute in the software collections Python 2.7 environment.
- exec scl enable python27 $0
-else
- yum -q clean all
- touch /var/lib/rpm/*
- if ! yum -q install --assumeyes \
- python27-python-arvados-python-client python27-python-arvados-fuse arvados-node-manager
- then
- exit 1
- fi
+yum -q clean all
+touch /var/lib/rpm/*
+if ! yum -q install --assumeyes \
+ python27-python-arvados-python-client python27-python-arvados-fuse arvados-node-manager
+then
+ exit 1
+fi
- mkdir -p /tmp/opts
- cd /tmp/opts
+mkdir -p /tmp/opts
+cd /tmp/opts
- for r in /arvados/packages/centos6/python27-python-*x86_64.rpm ; do
- rpm2cpio $r | cpio -idm
- done
+for r in /arvados/packages/centos6/python27-python-*x86_64.rpm ; do
+ rpm2cpio $r | cpio -idm
+done
- for so in $(find . -name "*.so") ; do
- echo
- echo "== Packages dependencies for $so =="
- ldd $so | awk '($3 ~ /^\//){print $3}' | sort -u | xargs rpm -qf | sort -u
- done
+for so in $(find -name "*.so") ; do
+ echo
+ echo "== Packages dependencies for $so =="
+ scl enable python27 "ldd $so" \
+ | awk '($3 ~ /^\//){print $3}' | sort -u | xargs rpm -qf | sort -u
+done
- exec /jenkins/common-test-packages.sh
-fi
+exec scl enable python27 /jenkins/common-test-packages.sh
commit b4ca11159ff0d38af3750c3a109e731f295a8460
Merge: 2748e09 dbedfbc
Author: Brett Smith <brett at curoverse.com>
Date: Mon Dec 7 20:55:59 2015 -0500
Merge branch '7829-package-licenses-wip'
Refs #7829. Closes #7913.
commit dbedfbcff43c82cc5d4685a013d58c6245a5f6ab
Author: Brett Smith <brett at curoverse.com>
Date: Mon Dec 7 15:36:32 2015 -0500
7829: Add licenses to Arvados software packages.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 84ada48..6369087 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -227,8 +227,9 @@ rm -rf install
perl Makefile.PL INSTALL_BASE=install >"$STDOUT_IF_DEBUG" && \
make install INSTALLDIRS=perl >"$STDOUT_IF_DEBUG" && \
fpm_build install/lib/=/usr/share libarvados-perl \
- "Curoverse, Inc." dir "$(version_from_git)" install/man/=/usr/share/man && \
- mv libarvados-perl*.$FORMAT "$WORKSPACE/packages/$TARGET/"
+ "Curoverse, Inc." dir "$(version_from_git)" install/man/=/usr/share/man \
+ "$WORKSPACE/LICENSE-2.0.txt=/usr/share/doc/libarvados-perl/LICENSE-2.0.txt" && \
+ mv --no-clobber libarvados-perl*.$FORMAT "$WORKSPACE/packages/$TARGET/"
# Ruby gems
debug_echo -e "\nRuby gems\n"
@@ -428,7 +429,7 @@ cd $WORKSPACE/packages/$TARGET
# This is the complete package with vendor/bundle included.
# It's big, so we do not build it by default.
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$API_VERSION" "--iteration" "$(default_iteration "$PACKAGE_NAME" "$API_VERSION")" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh" "$WORKSPACE/agpl-3.0.txt=/var/www/arvados-api/current/agpl-3.0.txt")
debug_echo -e "\n${COMMAND_ARR[@]}\n"
@@ -438,7 +439,7 @@ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
fi
# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$API_VERSION" "--iteration" "$(default_iteration "$PACKAGE_NAME" "$API_VERSION")" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh" "$WORKSPACE/agpl-3.0.txt=/var/www/arvados-api/current/agpl-3.0.txt")
debug_echo -e "\n${COMMAND_ARR[@]}\n"
@@ -488,7 +489,7 @@ cd $WORKSPACE/packages/$TARGET
# It's big, so we do not build it by default.
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$WORKBENCH_VERSION" "--iteration" "$(default_iteration "$PACKAGE_NAME" "$WORKBENCH_VERSION")" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh" "$WORKSPACE/agpl-3.0.txt=/var/www/arvados-workbench/current/agpl-3.0.txt")
debug_echo -e "\n${COMMAND_ARR[@]}\n"
@@ -499,7 +500,7 @@ fi
# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$WORKBENCH_VERSION" "--iteration" "$(default_iteration "$PACKAGE_NAME" "$WORKBENCH_VERSION")" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh" "$WORKSPACE/agpl-3.0.txt=/var/www/arvados-workbench/current/agpl-3.0.txt")
debug_echo -e "\n${COMMAND_ARR[@]}\n"
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index 76d410c..69ebb08 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -2,6 +2,11 @@
# A library of functions shared by the various scripts in this directory.
+# This is the timestamp about when we merged changed to include licenses
+# with Arvados packages. We use it as a heuristic to add revisions for
+# older packages.
+LICENSE_PACKAGE_TS=20151208015500
+
debug_echo () {
echo "$@" >"$STDOUT_IF_DEBUG"
}
@@ -71,6 +76,7 @@ package_go_binary() {
local src_path="$1"; shift
local prog="$1"; shift
local description="$1"; shift
+ local license_file=${1:-agpl-3.0.txt}; shift
debug_echo "package_go_binary $src_path as $prog"
@@ -94,7 +100,18 @@ package_go_binary() {
cd $WORKSPACE/packages/$TARGET
go get "git.curoverse.com/arvados.git/$src_path"
- fpm_build "$GOPATH/bin/$basename=/usr/bin/$prog" "$prog" 'Curoverse, Inc.' dir "$version" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=$description"
+ fpm_build "$GOPATH/bin/$basename=/usr/bin/$prog" "$prog" 'Curoverse, Inc.' dir "$version" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=$description" "$WORKSPACE/$license_file=/usr/share/doc/$prog/$license_file"
+}
+
+default_iteration() {
+ local package_name=$1; shift
+ local package_version=$1; shift
+ local iteration=1
+ if [[ $package_version =~ ^0\.1\.([0-9]{14})(\.|$) ]] && \
+ [[ ${BASH_REMATCH[1]} -le $LICENSE_PACKAGE_TS ]]; then
+ iteration=2
+ fi
+ echo $iteration
}
# Build packages for everything
@@ -156,6 +173,9 @@ fpm_build () {
if [[ "$VERSION" != "" ]]; then
COMMAND_ARR+=('-v' "$VERSION")
fi
+ # We can always add an --iteration here. If another one is specified in $@,
+ # that will take precedence, as desired.
+ COMMAND_ARR+=(--iteration "$(default_iteration "$PACKAGE" "$VERSION")")
# Append remaining function arguments directly to fpm's command line.
for i; do
commit 2748e092a403d6d8ff11a01238e55935aba7055a
Author: Brett Smith <brett at curoverse.com>
Date: Mon Dec 7 15:31:09 2015 -0500
7953: Fix Docker image to test CentOS 6.6 packages.
Follows the same rationale as
bebf531a2cb9cf08843d6ac4d4ae2238d58efc2b.
Refs #7953.
diff --git a/jenkins/package-test-dockerfiles/centos6/Dockerfile b/jenkins/package-test-dockerfiles/centos6/Dockerfile
index 5e354e2..814ad47 100644
--- a/jenkins/package-test-dockerfiles/centos6/Dockerfile
+++ b/jenkins/package-test-dockerfiles/centos6/Dockerfile
@@ -1,9 +1,7 @@
FROM centos:6
MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
-RUN yum -q install --assumeyes scl-utils && \
- curl -L -O https://www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm && \
- yum -q install --assumeyes rhscl-python27-epel-6-x86_64.noarch.rpm && \
+RUN yum -q install --assumeyes scl-utils centos-release-SCL && \
yum -q install --assumeyes python27
COPY localrepo.repo /etc/yum.repos.d/localrepo.repo
\ No newline at end of file
commit d3bbd1396af6fb8e6b6c8f50a0406cbc71ee82ca
Merge: 2542d0d bebf531
Author: Brett Smith <brett at curoverse.com>
Date: Mon Dec 7 12:00:29 2015 -0500
Merge branch '7953-centos-scl-fix-wip'
Refs #7953. Closes #7955.
commit bebf531a2cb9cf08843d6ac4d4ae2238d58efc2b
Author: Brett Smith <brett at curoverse.com>
Date: Mon Dec 7 10:06:02 2015 -0500
7953: Fix Docker image to build CentOS 6.6 packages.
* Follow the Software Collections documentation at
<https://wiki.centos.org/AdditionalResources/Repositories/SCL>.
This downgrades us to Python 3.3, but that shouldn't be a
problem, since we test and build packages for Debian 7 which uses
Python 3.2. On the plus side, it removes an external dependency
on softwarecollections.org.
* The python-pip package disappeared. I don't know where it went.
Maybe there were changes to the underlying centos:6 Docker image?
Install it manually.
diff --git a/jenkins/dockerfiles/centos6/Dockerfile b/jenkins/dockerfiles/centos6/Dockerfile
index cca5e60..68134c4 100644
--- a/jenkins/dockerfiles/centos6/Dockerfile
+++ b/jenkins/dockerfiles/centos6/Dockerfile
@@ -2,7 +2,7 @@ FROM centos:6
MAINTAINER Brett Smith <brett at curoverse.com>
# Install build dependencies provided in base distribution
-RUN yum -q -y install make automake gcc gcc-c++ libyaml-devel patch readline-devel zlib-devel libffi-devel openssl-devel bzip2 libtool bison sqlite-devel rpm-build git perl-ExtUtils-MakeMaker fuse-devel libattr-devel nss-devel libcurl-devel which tar scl-utils postgresql-devel python-pip
+RUN yum -q -y install make automake gcc gcc-c++ libyaml-devel patch readline-devel zlib-devel libffi-devel openssl-devel bzip2 libtool bison sqlite-devel rpm-build git perl-ExtUtils-MakeMaker fuse-devel libattr-devel nss-devel libcurl-devel which tar scl-utils centos-release-SCL postgresql-devel
# Install golang binary
ADD generated/golang-amd64.tar.gz /usr/local/
@@ -15,14 +15,10 @@ RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
/usr/local/rvm/bin/rvm-exec default gem install bundle fpm
-# Download and install Software Collections for newer requirements
-ADD https://www.softwarecollections.org/en/scls/rhscl/rh-python34/epel-6-x86_64/download/rhscl-rh-python34-epel-6-x86_64.noarch.rpm https://www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm /opt/
-
-RUN yum -q -y install /opt/rhscl-*.rpm
-
# Need to "touch" RPM database to workaround bug in interaction between
# overlayfs and yum (https://bugzilla.redhat.com/show_bug.cgi?id=1213602)
-RUN touch /var/lib/rpm/* && yum -q -y install python27 rh-python34
+RUN touch /var/lib/rpm/* && yum -q -y install python27 python33
+RUN scl enable python33 "easy_install-3.3 pip" && scl enable python27 "easy_install-2.7 pip"
ENV WORKSPACE /arvados
-CMD ["scl", "enable", "rh-python34", "python27", "/usr/local/rvm/bin/rvm-exec default bash /jenkins/run-build-packages.sh --target centos6"]
+CMD ["scl", "enable", "python33", "python27", "/usr/local/rvm/bin/rvm-exec default bash /jenkins/run-build-packages.sh --target centos6"]
commit 2542d0dfd960962d10bc4daf603fdb9e2c22d146
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Dec 4 10:14:22 2015 -0500
Default run-deploy.sh to ssh port 22, and provide a command line option
to change the port.
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index b81218e..f83f901 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -1,6 +1,7 @@
#!/bin/bash
DEBUG=0
+SSH_PORT=22
function usage {
echo >&2
@@ -9,6 +10,7 @@ function usage {
echo >&2 " <identifier> Arvados cluster name"
echo >&2
echo >&2 "$0 options:"
+ echo >&2 " -p, --port <ssh port> SSH port to use (default 22)"
echo >&2 " -d, --debug Enable debug output"
echo >&2 " -h, --help Display this help and exit"
echo >&2
@@ -27,8 +29,8 @@ function usage {
}
# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
-TEMP=`getopt -o hd \
- --long help,debug \
+TEMP=`getopt -o hdp: \
+ --long help,debug,port: \
-n "$0" -- "$@"`
if [ $? != 0 ] ; then echo "Use -h for help"; exit 1 ; fi
@@ -38,6 +40,9 @@ eval set -- "$TEMP"
while [ $# -ge 1 ]
do
case $1 in
+ -p | --port)
+ SSH_PORT="$2"; shift 2
+ ;;
-d | --debug)
DEBUG=1
shift
@@ -91,9 +96,9 @@ function run_puppet() {
title "Running puppet on $node"
TMP_FILE=`mktemp`
if [[ "$DEBUG" != "0" ]]; then
- ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C bash -c "'$PUPPET_AGENT'" | tee $TMP_FILE
+ ssh -t -p$SSH_PORT -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C bash -c "'$PUPPET_AGENT'" | tee $TMP_FILE
else
- ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C bash -c "'$PUPPET_AGENT'" > $TMP_FILE 2>&1
+ ssh -t -p$SSH_PORT -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C bash -c "'$PUPPET_AGENT'" > $TMP_FILE 2>&1
fi
ECODE=${PIPESTATUS[0]}
@@ -129,9 +134,9 @@ function run_command() {
title "Running '$command' on $node"
TMP_FILE=`mktemp`
if [[ "$DEBUG" != "0" ]]; then
- ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C "$command" | tee $TMP_FILE
+ ssh -t -p$SSH_PORT -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C "$command" | tee $TMP_FILE
else
- ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C "$command" > $TMP_FILE 2>&1
+ ssh -t -p$SSH_PORT -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C "$command" > $TMP_FILE 2>&1
fi
ECODE=$?
commit 8c2f3c1f36572e6822d9457d13d7129efd330fd3
Merge: 6ca2758 14ef215
Author: radhika <radhika at curoverse.com>
Date: Wed Dec 2 12:51:10 2015 -0500
refs #7710
Merge branch '7710-add-go-sdk-crunchrunner'
commit 6ca2758f31fb1a4afeb4eb5aca3cd8d6af402169
Author: Brett Smith <brett at curoverse.com>
Date: Tue Dec 1 17:13:20 2015 -0500
7839: Stop backporting requests for Python 2.
This is no longer needed now that our Keep client is built on top of
pycurl. Confirmed by building all our Python packages from source
into a fresh virtualenv. requests does not appear in `pip freeze`.
requests is still needed for Python 3 because docker-py uses it.
Closes #7839.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 59aa65c..84ada48 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -95,8 +95,7 @@ case "$TARGET" in
PYTHON3_PKG_PREFIX=python3
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
- rsa uritemplate httplib2 ws4py \
- pykka requests six pyexecjs jsonschema \
+ rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -108,8 +107,7 @@ case "$TARGET" in
PYTHON3_PKG_PREFIX=python3
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
- rsa uritemplate httplib2 ws4py \
- pykka requests six pyexecjs jsonschema \
+ rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -121,8 +119,7 @@ case "$TARGET" in
PYTHON3_PKG_PREFIX=python3
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
- rsa uritemplate httplib2 ws4py \
- pykka requests six pyexecjs jsonschema \
+ rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -145,8 +142,7 @@ case "$TARGET" in
PYTHON3_PKG_PREFIX=$PYTHON3_PACKAGE
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
- rsa uritemplate httplib2 ws4py \
- pykka requests six pyexecjs jsonschema \
+ rsa uritemplate httplib2 ws4py pykka six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl
python-daemon lockfile llfuse 'pbr<1.0')
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
commit 14ef215b72d2d56986ad29f1358cf27f7be75294
Author: radhika <radhika at curoverse.com>
Date: Tue Dec 1 13:35:41 2015 -0500
7710: add crunchrunner to gostuff
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 7a3a64e..89e75a8 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -77,6 +77,7 @@ sdk/go/keepclient
sdk/go/manifest
sdk/go/blockdigest
sdk/go/streamer
+sdk/go/crunchrunner
tools/keep-rsync
EOF
@@ -706,6 +707,7 @@ gostuff=(
sdk/go/blockdigest
sdk/go/manifest
sdk/go/streamer
+ sdk/go/crunchrunner
services/arv-git-httpd
services/crunchstat
services/keep-web
commit 24129230929833c3bad1e81b6af9b975c1f605a1
Merge: abc952f fc413df
Author: radhika <radhika at curoverse.com>
Date: Thu Nov 26 10:45:53 2015 -0500
refs #7253
Merge branch '7253-add-go-sdk-manifest'
commit fc413df044d945ac7620b0220980b23698cc1f5d
Merge: c6d5c89 abc952f
Author: radhika <radhika at curoverse.com>
Date: Thu Nov 26 00:53:55 2015 -0500
Merge branch 'master' into 7253-add-go-sdk-manifest
commit c6d5c89d0581b6f93793ed9e43d3ec28bc3488e1
Author: radhika <radhika at curoverse.com>
Date: Wed Nov 25 17:19:04 2015 -0500
7253: add sdk/go/blockdigest to gostuff
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 57beafa..b959cab 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -75,6 +75,7 @@ sdk/ruby
sdk/go/arvadosclient
sdk/go/keepclient
sdk/go/manifest
+sdk/go/blockdigest
sdk/go/streamer
tools/keep-rsync
@@ -702,6 +703,7 @@ do_install services/api apiserver
declare -a gostuff
gostuff=(
sdk/go/arvadosclient
+ sdk/go/blockdigest
sdk/go/manifest
sdk/go/streamer
services/arv-git-httpd
commit abc952f3017c4bcb68578cd90d130ec08f72aeb2
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Nov 25 15:05:20 2015 -0500
Force pbr<1.7.0 when installing mock.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 99fb5bf..64335bc 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -395,7 +395,7 @@ setup_virtualenv() {
fi
"$venvdest/bin/pip" install 'setuptools>=18' 'pip>=7'
# ubuntu1404 can't seem to install mock via tests_require, but it can do this.
- "$venvdest/bin/pip" install 'mock>=1.0'
+ "$venvdest/bin/pip" install 'mock>=1.0' 'pbr<1.7.0'
}
export PERLINSTALLBASE
commit e359eafa8b63bd374153ab0bfe81680b3a476dcd
Author: radhika <radhika at curoverse.com>
Date: Wed Nov 25 10:21:50 2015 -0500
7253: add sdk/go/manifest to gostuff
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 99fb5bf..57beafa 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -74,6 +74,7 @@ sdk/python
sdk/ruby
sdk/go/arvadosclient
sdk/go/keepclient
+sdk/go/manifest
sdk/go/streamer
tools/keep-rsync
@@ -701,6 +702,7 @@ do_install services/api apiserver
declare -a gostuff
gostuff=(
sdk/go/arvadosclient
+ sdk/go/manifest
sdk/go/streamer
services/arv-git-httpd
services/crunchstat
commit 30fc3285c237081c3bbe20de8a6e3adcfbc29ca0
Merge: 723ab70 751ca54
Author: radhika <radhika at curoverse.com>
Date: Mon Nov 23 14:18:55 2015 -0500
refs #7490
Merge branch '7490-datamanager-test'
commit 751ca54d959fdf93f042800cb5c4831e11979b78
Author: radhika <radhika at curoverse.com>
Date: Thu Nov 12 13:06:27 2015 -0500
7490: add services/datamanager/keep to gostuff
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index d9a4a9c..99fb5bf 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -710,6 +710,7 @@ gostuff=(
services/keepproxy
services/datamanager/summary
services/datamanager/collection
+ services/datamanager/keep
services/datamanager
tools/keep-rsync
)
commit 723ab7027a196d66e82d373d58f767503e88f771
Merge: acc9826 5cac6f5
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Nov 11 12:14:51 2015 -0500
Merge branch '5824-keep-web-workbench' refs #5824
commit acc982607d60ae73a482a9254b6a34979832efb8
Merge: 835109e f551c4a
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Nov 11 09:20:36 2015 -0500
Merge branch '7593-arvados-cwl-runner' refs #7593
commit 835109e6384b40e3c801587849bcf76ea0b1e76f
Merge: e2a1b08 5cabf9c
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Nov 9 21:18:12 2015 -0500
Merge branch '7722-build-keep-rsync' closes #7722
commit e2a1b086e41099f52b407e4085576b00eed39b2b
Merge: d35ac7e a5bf976
Author: Nico Cesar <nico at curoverse.com>
Date: Mon Nov 9 16:48:34 2015 -0500
Merge branch '7591-httplib2-perms-fix-wip'
refs #7591
commit a5bf97676451900886f6dda7521ca7db27cdd452
Author: Brett Smith <brett at curoverse.com>
Date: Sat Nov 7 11:49:17 2015 -0500
7591: Workaround bad permissions in the httplib2 backport.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index d122336..fad3f77 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -422,10 +422,42 @@ LIBCLOUD_DIR=$(mktemp -d)
fpm_build $LIBCLOUD_DIR "$PYTHON2_PKG_PREFIX"-apache-libcloud
rm -rf $LIBCLOUD_DIR
-# A few dependencies
+# Python 2 dependencies
for deppkg in "${PYTHON_BACKPORTS[@]}"; do
outname=$(echo "$deppkg" | sed -e 's/^python-//' -e 's/[<=>].*//' -e 's/_/-/g' -e "s/^/${PYTHON2_PKG_PREFIX}-/")
- fpm_build "$deppkg" "$outname"
+ case "$deppkg" in
+ httplib2)
+ # Work around 0640 permissions on some package files on Debian 8
+ # and Ubuntu 14.04. See #7591.
+ httplib2_workdir=$(mktemp -d httplib2-XXXXXX) && (
+ set -e
+ cd "$httplib2_workdir"
+ pip install --download . httplib2
+ tar -xf httplib2-*.tar*
+ cd httplib2-*/
+ "python$PYTHON2_VERSION" setup.py $DASHQ_UNLESS_DEBUG egg_info build
+ chmod -R go+rX .
+ set +e
+ # --iteration 2 provides an upgrade for previously built
+ # buggy packages. Arguments past $outname can be removed
+ # once we're building httplib2 > 0.9.2.
+ fpm_build . "$outname" "" python "" --iteration 2
+ # The upload step uses the package timestamp to determine
+ # whether it's new. --no-clobber plays nice with that.
+ mv --no-clobber "$outname"*.$FORMAT "$WORKSPACE/packages/$TARGET"
+ )
+ if [ 0 != "$?" ]; then
+ echo "ERROR: httplib2 build process failed"
+ EXITCODE=1
+ fi
+ if [ -n "$httplib2_workdir" ]; then
+ rm -rf "$httplib2_workdir"
+ fi
+ ;;
+ *)
+ fpm_build "$deppkg" "$outname"
+ ;;
+ esac
done
# Python 3 dependencies
commit 5cac6f5fd28ab38fcc7279d5d7bd5f224d689d8a
Merge: 3adaad5 d35ac7e
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Nov 6 16:55:38 2015 -0500
5824: Merge branch 'master' into 5824-keep-web-workbench
commit d35ac7e38b0eb480a4573a77a4bdff2a84f92f28
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Nov 6 12:41:24 2015 -0500
Add libwww-perl and python mock dependencies, so tests can run on ubuntu 14.04
If we let setuptools install mock for us (because it's listed in
tests_require), "python setup.py test" fails:
running test
Searching for mock>=1.0
Best match: mock 1.3.0
Processing mock-1.3.0-py2.7.egg
Using /tmp/arvados/sdk/python/.eggs/mock-1.3.0-py2.7.egg
Traceback (most recent call last):
File "setup.py", line 49, in <module>
cmdclass={'egg_info': tagger},
[...]
File "/tmp/.cache/arvados-build/VENVDIR/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2910, in scan_list
raise RequirementParseError(msg, line, "at", line[p:])
pkg_resources.RequirementParseError: Expected version spec in funcsigs;python_version<"3.3" at ;python_version<"3.3
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 8e845cb..c528363 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -151,7 +151,7 @@ sanity_checks() {
echo Checking dependencies:
echo -n 'virtualenv: '
virtualenv --version \
- || fatal "No virtualenv. Try: apt-get install virtualenv"
+ || fatal "No virtualenv. Try: apt-get install virtualenv (on ubuntu: python-virtualenv)"
echo -n 'go: '
go version \
|| fatal "No go binary. See http://golang.org/doc/install"
@@ -173,7 +173,7 @@ sanity_checks() {
for mod in ExtUtils::MakeMaker JSON LWP Net::SSL; do
echo -n "perl $mod: "
perl -e "use $mod; print \"\$$mod::VERSION\\n\"" \
- || fatal "No $mod. Try: apt-get install perl-modules libcrypt-ssleay-perl libjson-perl"
+ || fatal "No $mod. Try: apt-get install perl-modules libcrypt-ssleay-perl libjson-perl libwww-perl"
done
echo -n 'gitolite: '
which gitolite \
@@ -392,6 +392,8 @@ setup_virtualenv() {
virtualenv --setuptools "$@" "$venvdest" || fatal "virtualenv $venvdest failed"
fi
"$venvdest/bin/pip" install 'setuptools>=18' 'pip>=7'
+ # ubuntu1404 can't seem to install mock via tests_require, but it can do this.
+ "$venvdest/bin/pip" install 'mock>=1.0'
}
export PERLINSTALLBASE
commit 5cabf9c500789ffa2a32b8c298d0978f3c08c539
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Nov 5 01:19:13 2015 -0500
7722: Build keep-rsync package. Deduplicate go->fpm recipe.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index d122336..617df64 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -285,92 +285,25 @@ handle_python_package
cd $WORKSPACE/packages/$TARGET
fpm_build $SRC_BUILD_DIR/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
- rm -r "$SRC_BUILD_DIR"
+ rm -rf "$SRC_BUILD_DIR"
)
-# Keep
+# Go binaries
export GOPATH=$(mktemp -d)
-mkdir -p "$GOPATH/src/git.curoverse.com"
-ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
-
-# keepstore
-cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keepstore"
-PKG_VERSION=$(version_from_git)
-go get "git.curoverse.com/arvados.git/services/keepstore"
-cd $WORKSPACE/packages/$TARGET
-fpm_build $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
-
-# Get GO SDK version
-cd "$GOPATH/src/git.curoverse.com/arvados.git/sdk/go"
-GO_SDK_VERSION=$(version_from_git)
-GO_SDK_TIMESTAMP=$(timestamp_from_git)
-
-# keepproxy
-cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keepproxy"
-KEEPPROXY_VERSION=$(version_from_git)
-KEEPPROXY_TIMESTAMP=$(timestamp_from_git)
-
-if [[ "$GO_SDK_TIMESTAMP" -gt "$KEEPPROXY_TIMESTAMP" ]]; then
- PKG_VERSION=$GO_SDK_VERSION
-else
- PKG_VERSION=$KEEPPROXY_VERSION
-fi
-
-go get "git.curoverse.com/arvados.git/services/keepproxy"
-cd $WORKSPACE/packages/$TARGET
-fpm_build $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
-
-# keep-web
-cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keep-web"
-KEEP_WEB_VERSION=$(version_from_git)
-KEEP_WEB_TIMESTAMP=$(timestamp_from_git)
-
-if [[ "$GO_SDK_TIMESTAMP" -gt "$KEEP_WEB_TIMESTAMP" ]]; then
- PKG_VERSION=$GO_SDK_VERSION
-else
- PKG_VERSION=$KEEP_WEB_VERSION
-fi
-
-go get "git.curoverse.com/arvados.git/services/keep-web"
-cd $WORKSPACE/packages/$TARGET
-fpm_build $GOPATH/bin/keep-web=/usr/bin/keep-web keep-web 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Static web hosting service for user data stored in Arvados Keep"
-
-# datamanager
-cd "$GOPATH/src/git.curoverse.com/arvados.git/services/datamanager"
-DATAMANAGER_VERSION=$(version_from_git)
-DATAMANAGER_TIMESTAMP=$(timestamp_from_git)
-
-if [[ "$GO_SDK_TIMESTAMP" -gt "$DATAMANAGER_TIMESTAMP" ]]; then
- PKG_VERSION=$GO_SDK_VERSION
-else
- PKG_VERSION=$DATAMANAGER_VERSION
-fi
-
-go get "git.curoverse.com/arvados.git/services/datamanager"
-cd $WORKSPACE/packages/$TARGET
-fpm_build $GOPATH/bin/datamanager=/usr/bin/arvados-data-manager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
-
-# arv-git-httpd
-cd "$GOPATH/src/git.curoverse.com/arvados.git/services/arv-git-httpd"
-ARVGITHTTPD_VERSION=$(version_from_git)
-ARVGITHTTPD_TIMESTAMP=$(timestamp_from_git)
-
-if [[ "$GO_SDK_TIMESTAMP" -gt "$ARVGITHTTPD_TIMESTAMP" ]]; then
- PKG_VERSION=$GO_SDK_VERSION
-else
- PKG_VERSION=$ARVGITHTTPD_VERSION
-fi
-
-go get "git.curoverse.com/arvados.git/services/arv-git-httpd"
-cd $WORKSPACE/packages/$TARGET
-fpm_build $GOPATH/bin/arv-git-httpd=/usr/bin/arvados-git-httpd arvados-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
-
-# crunchstat
-cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
-PKG_VERSION=$(version_from_git)
-go get "git.curoverse.com/arvados.git/services/crunchstat"
-cd $WORKSPACE/packages/$TARGET
-fpm_build $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
+package_go_binary services/keepstore keepstore \
+ "Keep storage daemon, accessible to clients on the LAN"
+package_go_binary services/keepproxy keepproxy \
+ "Make a Keep cluster accessible to clients that are not on the LAN"
+package_go_binary services/keep-web keep-web \
+ "Static web hosting service for user data stored in Arvados Keep"
+package_go_binary services/datamanager arvados-data-manager \
+ "Ensure block replication levels, report disk usage, and determine which blocks should be deleted when space is needed"
+package_go_binary services/arv-git-httpd arvados-git-httpd \
+ "Provide authenticated http access to Arvados-hosted git repositories"
+package_go_binary services/crunchstat crunchstat \
+ "Gather cpu/memory/network statistics of running Crunch jobs"
+package_go_binary tools/keep-rsync keep-rsync \
+ "Copy all data from one set of Keep servers to another"
# The Python SDK
# Please resist the temptation to add --no-python-fix-name to the fpm call here
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index f19d58b..76d410c 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -66,6 +66,37 @@ handle_ruby_gem() {
fi
}
+# Usage: package_go_binary services/foo arvados-foo "Compute foo to arbitrary precision"
+package_go_binary() {
+ local src_path="$1"; shift
+ local prog="$1"; shift
+ local description="$1"; shift
+
+ debug_echo "package_go_binary $src_path as $prog"
+
+ local basename="${src_path##*/}"
+
+ mkdir -p "$GOPATH/src/git.curoverse.com"
+ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
+
+ cd "$GOPATH/src/git.curoverse.com/arvados.git/$src_path"
+ local version=$(version_from_git)
+ local timestamp=$(timestamp_from_git)
+
+ # If the command imports anything from the Arvados SDK, bump the
+ # version number and build a new package whenever the SDK changes.
+ if grep -qr git.curoverse.com/arvados .; then
+ cd "$GOPATH/src/git.curoverse.com/arvados.git/sdk/go"
+ if [[ $(timestamp_from_git) -gt "$timestamp" ]]; then
+ version=$(version_from_git)
+ fi
+ fi
+
+ cd $WORKSPACE/packages/$TARGET
+ go get "git.curoverse.com/arvados.git/$src_path"
+ fpm_build "$GOPATH/bin/$basename=/usr/bin/$prog" "$prog" 'Curoverse, Inc.' dir "$version" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=$description"
+}
+
# Build packages for everything
fpm_build () {
# The package source. Depending on the source type, this can be a
commit 7c80a8b76a2972ef4e42dcc281f46d9e0c8839ea
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Nov 4 09:50:42 2015 -0500
arvados-api-server-upgrade.sh fix: do not try to chmod the cache
directory if it doesn't exist. This avoids an error on fresh install.
Rails will create the cache directory if it doesn't exist.
No issue #
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
index edecb21..2fff1e7 100755
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -81,7 +81,10 @@ chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
chown -R "$WWW_OWNER" $SHARED_PATH/log
chown "$WWW_OWNER" $RELEASE_PATH/db/structure.sql
chmod 644 $SHARED_PATH/log/*
-chmod -R 2775 $RELEASE_PATH/tmp/cache/
+# Rails creates the cache directory if it doesn't exist
+if [[ -d $RELEASE_PATH/tmp/cache/ ]]; then
+ chmod -R 2775 $RELEASE_PATH/tmp/cache/
+fi
echo "Done."
echo "Running sanity check"
commit 057ea788a6cfc9c0a5fc02376e0557f4d0529b38
Merge: 846f42e 57b2c57
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Nov 3 23:54:39 2015 -0500
Merge branch '5824-keep-web'
refs #5824
commit 3adaad524ee4dacb06434956bee3c96599299c98
Author: Tom Clegg <tom at curoverse.com>
Date: Sat Oct 17 03:02:03 2015 -0400
5824: Start keep-web service for Workbench integration tests.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 8e845cb..1b1e757 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -253,9 +253,10 @@ start_api() {
}
start_nginx_proxy_services() {
- echo 'Starting keepproxy, arv-git-httpd, and nginx ssl proxy...'
+ echo 'Starting keepproxy, keep-web, arv-git-httpd, and nginx ssl proxy...'
cd "$WORKSPACE" \
&& python sdk/python/tests/run_test_server.py start_keep_proxy \
+ && python sdk/python/tests/run_test_server.py start_keep-web \
&& python sdk/python/tests/run_test_server.py start_arv-git-httpd \
&& python sdk/python/tests/run_test_server.py start_nginx \
&& export ARVADOS_TEST_PROXY_SERVICES=1
@@ -267,6 +268,7 @@ stop_services() {
cd "$WORKSPACE" \
&& python sdk/python/tests/run_test_server.py stop_nginx \
&& python sdk/python/tests/run_test_server.py stop_arv-git-httpd \
+ && python sdk/python/tests/run_test_server.py stop_keep-web \
&& python sdk/python/tests/run_test_server.py stop_keep_proxy
fi
if [[ -n "$ARVADOS_TEST_API_HOST" ]]; then
commit 57b2c57db16bed3f712200f603a9e77998706cc6
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Nov 3 23:50:44 2015 -0500
5824: Add services/keep-web
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index a767b29..d122336 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -320,6 +320,21 @@ go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/packages/$TARGET
fpm_build $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
+# keep-web
+cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keep-web"
+KEEP_WEB_VERSION=$(version_from_git)
+KEEP_WEB_TIMESTAMP=$(timestamp_from_git)
+
+if [[ "$GO_SDK_TIMESTAMP" -gt "$KEEP_WEB_TIMESTAMP" ]]; then
+ PKG_VERSION=$GO_SDK_VERSION
+else
+ PKG_VERSION=$KEEP_WEB_VERSION
+fi
+
+go get "git.curoverse.com/arvados.git/services/keep-web"
+cd $WORKSPACE/packages/$TARGET
+fpm_build $GOPATH/bin/keep-web=/usr/bin/keep-web keep-web 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Static web hosting service for user data stored in Arvados Keep"
+
# datamanager
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/datamanager"
DATAMANAGER_VERSION=$(version_from_git)
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 93cf966..8e845cb 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -59,14 +59,15 @@ apps/workbench_benchmark
apps/workbench_profile
doc
services/api
+services/arv-git-httpd
services/crunchstat
services/dockercleaner
services/fuse
+services/keep-web
services/keepproxy
services/keepstore
services/login-sync
services/nodemanager
-services/arv-git-httpd
sdk/cli
sdk/pam
sdk/python
@@ -699,6 +700,7 @@ gostuff=(
sdk/go/streamer
services/arv-git-httpd
services/crunchstat
+ services/keep-web
services/keepstore
sdk/go/keepclient
services/keepproxy
commit 846f42e54e6f2e4b00802c5f1a45cb2ec1b23674
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Oct 29 12:28:15 2015 -0400
Remove unnecessary yum update. touch RPM database to work around overlayfs bug.
refs #7370
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index f8f1e1b..1aca953 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -5,7 +5,7 @@ if ! python --version |& grep "Python 2.7" >/dev/null ; then
exec scl enable python27 $0
else
yum -q clean all
- yum -q -y update
+ touch /var/lib/rpm/*
if ! yum -q install --assumeyes \
python27-python-arvados-python-client python27-python-arvados-fuse arvados-node-manager
then
commit 69e10b5d9116fed733b192ba902d2dc7ad68a076
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Oct 29 11:55:03 2015 -0400
Obsoleted by run-build-packages-all-targets.sh --test-packages refs #7370
diff --git a/jenkins/run-test-packages.sh b/jenkins/run-test-packages.sh
deleted file mode 100755
index baca7b7..0000000
--- a/jenkins/run-test-packages.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/sh
-
-if test -z "$WORKSPACE" ; then
- echo "Must set WORKSPACE"
- exit 1
-fi
-
-FAIL=0
-
-ERRORS=""
-
-for pkg in ./test-packages-*.sh ; do
- echo
- echo "== Running $pkg =="
- echo
- if ! $pkg --run-test ; then
- FAIL=1
- ERRORS="$ERRORS\n$pkg has install errors"
- fi
-done
-
-echo $ERRORS
-exit $FAIL
commit cadd265aa66c1fd41e3a385a3b23e97462094b4b
Author: Nico Cesar <nico at curoverse.com>
Date: Thu Oct 29 11:42:37 2015 -0400
reverted hkp port 80 explicitly
no issue #
diff --git a/jenkins/dockerfiles/centos6/Dockerfile b/jenkins/dockerfiles/centos6/Dockerfile
index 928350f..cca5e60 100644
--- a/jenkins/dockerfiles/centos6/Dockerfile
+++ b/jenkins/dockerfiles/centos6/Dockerfile
@@ -9,7 +9,7 @@ ADD generated/golang-amd64.tar.gz /usr/local/
RUN ln -s /usr/local/go/bin/go /usr/local/bin/
# Install RVM
-RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
diff --git a/jenkins/dockerfiles/debian7/Dockerfile b/jenkins/dockerfiles/debian7/Dockerfile
index daf02ab..21b554f 100644
--- a/jenkins/dockerfiles/debian7/Dockerfile
+++ b/jenkins/dockerfiles/debian7/Dockerfile
@@ -5,7 +5,7 @@ MAINTAINER Ward Vandewege <ward at curoverse.com>
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libpq-dev python-pip
# Install RVM
-RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
diff --git a/jenkins/dockerfiles/debian8/Dockerfile b/jenkins/dockerfiles/debian8/Dockerfile
index 87a0dee..6fb622a 100644
--- a/jenkins/dockerfiles/debian8/Dockerfile
+++ b/jenkins/dockerfiles/debian8/Dockerfile
@@ -5,7 +5,7 @@ MAINTAINER Ward Vandewege <ward at curoverse.com>
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev python-pip
# Install RVM
-RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
diff --git a/jenkins/dockerfiles/ubuntu1204/Dockerfile b/jenkins/dockerfiles/ubuntu1204/Dockerfile
index 478b05f..4984d37 100644
--- a/jenkins/dockerfiles/ubuntu1204/Dockerfile
+++ b/jenkins/dockerfiles/ubuntu1204/Dockerfile
@@ -5,7 +5,7 @@ MAINTAINER Ward Vandewege <ward at curoverse.com>
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip
# Install RVM
-RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
diff --git a/jenkins/dockerfiles/ubuntu1404/Dockerfile b/jenkins/dockerfiles/ubuntu1404/Dockerfile
index bfe6f5f..a3d6c8d 100644
--- a/jenkins/dockerfiles/ubuntu1404/Dockerfile
+++ b/jenkins/dockerfiles/ubuntu1404/Dockerfile
@@ -5,7 +5,7 @@ MAINTAINER Brett Smith <brett at curoverse.com>
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip
# Install RVM
-RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
commit 40fff998c296ff1fa3ae681a1ae4c06df7f6abe5
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Oct 28 17:21:06 2015 -0400
Make sure that the CentOS 6 test script actually does the yum update call.
refs #7370
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index c8a62b5..f8f1e1b 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -5,7 +5,7 @@ if ! python --version |& grep "Python 2.7" >/dev/null ; then
exec scl enable python27 $0
else
yum -q clean all
- yum -q update
+ yum -q -y update
if ! yum -q install --assumeyes \
python27-python-arvados-python-client python27-python-arvados-fuse arvados-node-manager
then
commit f551c4a120603e54eb6b9119b3e214df3523a8c9
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Oct 28 11:26:29 2015 -0400
7593: Add sdk/cwl to package building.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index a767b29..b4d97e0 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -257,6 +257,9 @@ handle_python_package
cd "$WORKSPACE/sdk/python"
handle_python_package
+cd "$WORKSPACE/sdk/cwl"
+handle_python_package
+
cd "$WORKSPACE/services/fuse"
handle_python_package
diff --git a/jenkins/run_upload_packages.py b/jenkins/run_upload_packages.py
index 5fc134c..04e6c80 100755
--- a/jenkins/run_upload_packages.py
+++ b/jenkins/run_upload_packages.py
@@ -189,6 +189,7 @@ PACKAGE_SUITES = {
'python': _define_suite(PythonPackageSuite,
'sdk/pam/dist/*.tar.gz',
'sdk/python/dist/*.tar.gz',
+ 'sdk/cwl/dist/*.tar.gz',
'services/nodemanager/dist/*.tar.gz',
'services/fuse/dist/*.tar.gz',
),
commit 426e8062746c72854622a7a676c8f5c51fef45ff
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Oct 27 15:16:17 2015 -0400
Add python-pip to the Arvados build images.
this is a dependency for #7591 (refs #7591)
diff --git a/jenkins/dockerfiles/centos6/Dockerfile b/jenkins/dockerfiles/centos6/Dockerfile
index 4044dd6..928350f 100644
--- a/jenkins/dockerfiles/centos6/Dockerfile
+++ b/jenkins/dockerfiles/centos6/Dockerfile
@@ -2,7 +2,7 @@ FROM centos:6
MAINTAINER Brett Smith <brett at curoverse.com>
# Install build dependencies provided in base distribution
-RUN yum -q -y install make automake gcc gcc-c++ libyaml-devel patch readline-devel zlib-devel libffi-devel openssl-devel bzip2 libtool bison sqlite-devel rpm-build git perl-ExtUtils-MakeMaker fuse-devel libattr-devel nss-devel libcurl-devel which tar scl-utils postgresql-devel
+RUN yum -q -y install make automake gcc gcc-c++ libyaml-devel patch readline-devel zlib-devel libffi-devel openssl-devel bzip2 libtool bison sqlite-devel rpm-build git perl-ExtUtils-MakeMaker fuse-devel libattr-devel nss-devel libcurl-devel which tar scl-utils postgresql-devel python-pip
# Install golang binary
ADD generated/golang-amd64.tar.gz /usr/local/
diff --git a/jenkins/dockerfiles/debian7/Dockerfile b/jenkins/dockerfiles/debian7/Dockerfile
index 5fc25c0..daf02ab 100644
--- a/jenkins/dockerfiles/debian7/Dockerfile
+++ b/jenkins/dockerfiles/debian7/Dockerfile
@@ -2,7 +2,7 @@ FROM debian:wheezy
MAINTAINER Ward Vandewege <ward at curoverse.com>
# Install dependencies and set up system.
-RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libpq-dev
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libpq-dev python-pip
# Install RVM
RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
diff --git a/jenkins/dockerfiles/debian8/Dockerfile b/jenkins/dockerfiles/debian8/Dockerfile
index 83cce32..87a0dee 100644
--- a/jenkins/dockerfiles/debian8/Dockerfile
+++ b/jenkins/dockerfiles/debian8/Dockerfile
@@ -2,7 +2,7 @@ FROM debian:jessie
MAINTAINER Ward Vandewege <ward at curoverse.com>
# Install dependencies and set up system.
-RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev python-pip
# Install RVM
RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
diff --git a/jenkins/dockerfiles/ubuntu1204/Dockerfile b/jenkins/dockerfiles/ubuntu1204/Dockerfile
index deab74e..478b05f 100644
--- a/jenkins/dockerfiles/ubuntu1204/Dockerfile
+++ b/jenkins/dockerfiles/ubuntu1204/Dockerfile
@@ -2,7 +2,7 @@ FROM ubuntu:precise
MAINTAINER Ward Vandewege <ward at curoverse.com>
# Install dependencies and set up system.
-RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip
# Install RVM
RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
diff --git a/jenkins/dockerfiles/ubuntu1404/Dockerfile b/jenkins/dockerfiles/ubuntu1404/Dockerfile
index cf735cb..bfe6f5f 100644
--- a/jenkins/dockerfiles/ubuntu1404/Dockerfile
+++ b/jenkins/dockerfiles/ubuntu1404/Dockerfile
@@ -2,7 +2,7 @@ FROM ubuntu:trusty
MAINTAINER Brett Smith <brett at curoverse.com>
# Install dependencies and set up system.
-RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip
# Install RVM
RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
commit 937bbde50c186d8d28bff1d8980fd701cb9b72ab
Author: Nico Cesar <nico at curoverse.com>
Date: Tue Oct 27 11:14:16 2015 -0400
hkp://pool.sks-keyservers.net:80 is needed to use the transaparent proxy
no issue #
diff --git a/jenkins/dockerfiles/centos6/Dockerfile b/jenkins/dockerfiles/centos6/Dockerfile
index 0a733e3..4044dd6 100644
--- a/jenkins/dockerfiles/centos6/Dockerfile
+++ b/jenkins/dockerfiles/centos6/Dockerfile
@@ -9,7 +9,7 @@ ADD generated/golang-amd64.tar.gz /usr/local/
RUN ln -s /usr/local/go/bin/go /usr/local/bin/
# Install RVM
-RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
diff --git a/jenkins/dockerfiles/debian7/Dockerfile b/jenkins/dockerfiles/debian7/Dockerfile
index 33083a0..5fc25c0 100644
--- a/jenkins/dockerfiles/debian7/Dockerfile
+++ b/jenkins/dockerfiles/debian7/Dockerfile
@@ -5,7 +5,7 @@ MAINTAINER Ward Vandewege <ward at curoverse.com>
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libpq-dev
# Install RVM
-RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
diff --git a/jenkins/dockerfiles/debian8/Dockerfile b/jenkins/dockerfiles/debian8/Dockerfile
index dc26e25..83cce32 100644
--- a/jenkins/dockerfiles/debian8/Dockerfile
+++ b/jenkins/dockerfiles/debian8/Dockerfile
@@ -5,7 +5,7 @@ MAINTAINER Ward Vandewege <ward at curoverse.com>
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev
# Install RVM
-RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
diff --git a/jenkins/dockerfiles/ubuntu1204/Dockerfile b/jenkins/dockerfiles/ubuntu1204/Dockerfile
index 642d2a0..deab74e 100644
--- a/jenkins/dockerfiles/ubuntu1204/Dockerfile
+++ b/jenkins/dockerfiles/ubuntu1204/Dockerfile
@@ -5,7 +5,7 @@ MAINTAINER Ward Vandewege <ward at curoverse.com>
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev
# Install RVM
-RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
diff --git a/jenkins/dockerfiles/ubuntu1404/Dockerfile b/jenkins/dockerfiles/ubuntu1404/Dockerfile
index 69ed2cc..cf735cb 100644
--- a/jenkins/dockerfiles/ubuntu1404/Dockerfile
+++ b/jenkins/dockerfiles/ubuntu1404/Dockerfile
@@ -5,7 +5,7 @@ MAINTAINER Brett Smith <brett at curoverse.com>
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev
# Install RVM
-RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+RUN gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
commit d3c6950d8f2945c18448a0b1edf38bf8d15ced3f
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Oct 23 22:24:51 2015 -0400
Fix path issue when running run-build-packages-all-targets.sh from
another directory.
refs #7370
diff --git a/jenkins/run-build-packages-all-targets.sh b/jenkins/run-build-packages-all-targets.sh
index 19befb3..654d551 100755
--- a/jenkins/run-build-packages-all-targets.sh
+++ b/jenkins/run-build-packages-all-targets.sh
@@ -75,6 +75,8 @@ while [ $# -gt 0 ]; do
shift
done
+cd $(dirname $0)
+
for dockerfile_path in $(find -name Dockerfile); do
./run-build-packages-one-target.sh --target "$(basename $(dirname "$dockerfile_path"))" --command "$COMMAND" $DEBUG $TEST_PACKAGES
done
commit d8b5db8865d9d2929e9a3fe708d609d10622817f
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Oct 23 16:31:18 2015 -0400
Make sure bundle is installed in arvados-api-server-upgrade.sh,
arvados-workbench-upgrade.sh and arvados-sso-server.postinst
No issue #
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
index 71105e9..edecb21 100755
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -53,6 +53,15 @@ if [[ ! -e $SHARED_PATH/log/production.log ]]; then touch $SHARED_PATH/log/produ
cd "$RELEASE_PATH"
export RAILS_ENV=production
+echo "Making sure bundle is installed"
+set +e
+which bundle > /dev/null
+if [[ "$?" != "0" ]]; then
+ gem install bundle
+fi
+set -e
+echo "Done."
+
echo "Running bundle install"
bundle install --path $SHARED_PATH/vendor_bundle
echo "Done."
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index 436f5bd..d39cdd4 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -126,6 +126,15 @@ EOF
cd "$RELEASE_PATH"
export RAILS_ENV=production
+ echo "Making sure bundle is installed ..."
+ set +e
+ which bundle > /dev/null
+ if [[ "$?" != "0" ]]; then
+ $COMMAND_PREFIX gem install bundle
+ fi
+ set -e
+ echo "... done."
+
echo -n "Running bundle install ..."
$COMMAND_PREFIX bundle install --path $SHARED_PATH/vendor_bundle --quiet || exit $?
echo "... done."
diff --git a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
index a481985..e981517 100755
--- a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
+++ b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
@@ -46,6 +46,15 @@ if [[ ! -e $RELEASE_PATH/tmp ]]; then mkdir -p $RELEASE_PATH/tmp; fi
if [[ ! -e $RELEASE_PATH/log ]]; then ln -s $SHARED_PATH/log $RELEASE_PATH/log; fi
if [[ ! -e $SHARED_PATH/log/production.log ]]; then touch $SHARED_PATH/log/production.log; fi
+echo "Making sure bundle is installed"
+set +e
+which bundle > /dev/null
+if [[ "$?" != "0" ]]; then
+ gem install bundle
+fi
+set -e
+echo "Done."
+
echo "Running bundle install"
(cd $RELEASE_PATH && RAILS_ENV=production bundle install --path $SHARED_PATH/vendor_bundle)
echo "Done."
commit d119e26a14b0bc143638ca38926a4a0d911fe8f1
Merge: 9eac45e cffee40
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Oct 23 10:29:02 2015 -0400
Merge branch '7370-package-install-testing' closes #7370
commit cffee40d749672091b0969c8ec3168ab3e5254d6
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Oct 20 21:18:23 2015 -0400
Shut up dpkg-scanpackages warnings.
refs #7370
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index 5b8f2ce..1d4249b 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -85,7 +85,7 @@ if [[ -n "$test_packages" ]]; then
if [[ -n "$(find $WORKSPACE/packages/$TARGET -name *.deb)" ]] ; then
(cd $WORKSPACE/packages/$TARGET
- dpkg-scanpackages . /dev/null | gzip -c > Packages.gz
+ dpkg-scanpackages . 2> >(grep -v 'warning' 1>&2) | gzip -c > Packages.gz
)
fi
commit 9eac45e52bf4a4b37bebfb2fc85ec5584207bfa8
Author: Brett Smith <brett at curoverse.com>
Date: Mon Oct 19 12:53:20 2015 -0400
Backport pbr < 1.0 on CentOS 6.
The lockfile backport grew a dependency on pbr < 1.0 in their version
0.11. No issue #.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index ff9acf6..a767b29 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -148,7 +148,7 @@ case "$TARGET" in
rsa uritemplate httplib2 ws4py \
pykka requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl
- python-daemon lockfile llfuse)
+ python-daemon lockfile llfuse 'pbr<1.0')
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
export PYCURL_SSL_LIBRARY=nss
;;
commit a23620386db119bf670221368016484a5a2f1f93
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Oct 16 14:29:57 2015 -0400
7370: run-build-packages-all-targets hands off to run-build-packages-one-target, suppports --test-packages option.
diff --git a/jenkins/run-build-packages-all-targets.sh b/jenkins/run-build-packages-all-targets.sh
index 8c889a0..19befb3 100755
--- a/jenkins/run-build-packages-all-targets.sh
+++ b/jenkins/run-build-packages-all-targets.sh
@@ -10,6 +10,8 @@ Options:
--command
Build command to execute (default: use built-in Docker image command)
+--test-packages
+ Run package install tests
--debug
Output debug information (default: false)
@@ -36,7 +38,7 @@ fi
set -e
PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,debug,command: \
+ help,test-packages,debug,command: \
-- "" "$@")
if [ $? -ne 0 ]; then
exit 1
@@ -44,6 +46,7 @@ fi
COMMAND=
DEBUG=
+TEST_PACKAGES=
eval set -- "$PARSEDOPTS"
while [ $# -gt 0 ]; do
@@ -54,11 +57,14 @@ while [ $# -gt 0 ]; do
exit 1
;;
--debug)
- DEBUG=" --debug"
+ DEBUG="--debug"
;;
--command)
COMMAND="$2"; shift
;;
+ --test-packages)
+ TEST_PACKAGES="--test-packages"
+ ;;
--)
if [ $# -gt 1 ]; then
echo >&2 "$0: unrecognized argument '$2'. Try: $0 --help"
@@ -69,36 +75,6 @@ while [ $# -gt 0 ]; do
shift
done
-if [[ "$COMMAND" != "" ]]; then
- COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND$DEBUG"
-fi
-
-FINAL_EXITCODE=0
-JENKINS_DIR=$(dirname "$(readlink -e "$0")")
-
-run_docker() {
- local tag=$1; shift
- if [[ "$COMMAND" != "" ]]; then
- COMMAND="$COMMAND --target $tag"
- fi
- if docker run -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
- --env ARVADOS_DEBUG=1 "arvados/build:$tag" $COMMAND; then
- # Success - nothing more to do.
- true
- else
- FINAL_EXITCODE=$?
- echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
- fi
-}
-
-# In case it's needed, build the containers. This costs just a few
-# seconds when the containers already exist, so it's not a big deal to
-# do it on each run.
-cd "$JENKINS_DIR/dockerfiles"
-time ./build-all-build-containers.sh
-
for dockerfile_path in $(find -name Dockerfile); do
- run_docker "$(basename $(dirname "$dockerfile_path"))"
+ ./run-build-packages-one-target.sh --target "$(basename $(dirname "$dockerfile_path"))" --command "$COMMAND" $DEBUG $TEST_PACKAGES
done
-
-exit $FINAL_EXITCODE
commit ad5f9554d13754e16def528a24a2669f7cbd4e5d
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Oct 15 16:40:54 2015 -0400
7370: Fix centos6 package builder container.
diff --git a/jenkins/dockerfiles/centos6/Dockerfile b/jenkins/dockerfiles/centos6/Dockerfile
index d0daef6..0a733e3 100644
--- a/jenkins/dockerfiles/centos6/Dockerfile
+++ b/jenkins/dockerfiles/centos6/Dockerfile
@@ -13,7 +13,7 @@ RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
curl -L https://get.rvm.io | bash -s stable && \
/usr/local/rvm/bin/rvm install 2.1 && \
/usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
- /usr/local/rvm/bin/rvm-exec default gem install fpm
+ /usr/local/rvm/bin/rvm-exec default gem install bundle fpm
# Download and install Software Collections for newer requirements
ADD https://www.softwarecollections.org/en/scls/rhscl/rh-python34/epel-6-x86_64/download/rhscl-rh-python34-epel-6-x86_64.noarch.rpm https://www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm /opt/
commit 72c6e7bcd898caa3a2ed718e66b71568fe66bc7d
Merge: d20e333 b67e599
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Oct 15 16:24:07 2015 -0400
Merge branch 'master' into 7370-package-install-testing
Conflicts:
jenkins/run-build-packages-one-target.sh
diff --cc jenkins/run-build-packages-one-target.sh
index 3e0e136,2d8d8df..5b8f2ce
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@@ -10,8 -10,8 +10,10 @@@ Syntax
Distribution to build packages for (default: debian7)
--command
Build command to execute (default: use built-in Docker image command)
+--test-packages
+ Run package install test script "test-packages-$target.sh"
+ --debug
+ Output debug information (default: false)
WORKSPACE=path Path to the Arvados source tree to build packages from
@@@ -34,7 -34,7 +36,7 @@@ if ! [[ -d "$WORKSPACE" ]]; the
fi
PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,test-packages,target:,command: \
- help,debug,target:,command: \
++ help,debug,test-packages,target:,command: \
-- "" "$@")
if [ $? -ne 0 ]; then
exit 1
@@@ -72,48 -73,34 +78,48 @@@ don
set -e
-if [[ "$COMMAND" != "" ]]; then
- COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND --target $TARGET$DEBUG"
+if [[ -n "$test_packages" ]]; then
+ if [[ -n "$(find $WORKSPACE/packages/$TARGET -name *.rpm)" ]] ; then
+ createrepo $WORKSPACE/packages/$TARGET
+ fi
+
+ if [[ -n "$(find $WORKSPACE/packages/$TARGET -name *.deb)" ]] ; then
+ (cd $WORKSPACE/packages/$TARGET
+ dpkg-scanpackages . /dev/null | gzip -c > Packages.gz
+ )
+ fi
+
+ COMMAND="/jenkins/test-packages-$TARGET.sh"
+ IMAGE="arvados/package-test:$TARGET"
+else
+ IMAGE="arvados/build:$TARGET"
+ if [[ "$COMMAND" != "" ]]; then
- COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND --target $TARGET"
++ COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND --target $TARGET$DEBUG"
+ fi
fi
-FINAL_EXITCODE=0
JENKINS_DIR=$(dirname "$(readlink -e "$0")")
-run_docker() {
- local tag=$1; shift
- if docker run -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
- --env ARVADOS_DEBUG=1 "arvados/build:$tag" $COMMAND; then
- # Success - nothing more to do.
- true
- else
- FINAL_EXITCODE=$?
- echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
- fi
-}
+if [[ -n "$test_packages" ]]; then
+ pushd "$JENKINS_DIR/package-test-dockerfiles"
+else
+ pushd "$JENKINS_DIR/dockerfiles"
+fi
-# In case it's needed, build the container. This costs just a few
-# seconds when the container already exist, so it's not a big deal to
-# do it on each run.
-cd "$JENKINS_DIR/dockerfiles"
echo $TARGET
cd $TARGET
-time docker build -t arvados/build:$TARGET .
-cd ..
+time docker build --tag=$IMAGE .
+popd
-run_docker $TARGET
+FINAL_EXITCODE=0
+
+if docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
+ --env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND; then
+ # Success - nothing more to do.
+ true
+else
+ FINAL_EXITCODE=$?
+ echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
+fi
exit $FINAL_EXITCODE
commit d20e33394de2a85f17399e0e4d9d7d2e52c94178
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Oct 15 16:21:56 2015 -0400
7370: Refactoring way the package test is set up and executed.
diff --git a/jenkins/deb-common-test-packages.sh b/jenkins/deb-common-test-packages.sh
index f40ba7b..d9d0301 100755
--- a/jenkins/deb-common-test-packages.sh
+++ b/jenkins/deb-common-test-packages.sh
@@ -1,29 +1,9 @@
-#!/bin/sh
+#!/bin/bash
-if test "$1" = --run-test ; then
+# Multiple .deb based distros symlink to this script, so extract the target
+# from the invocation path.
+target=$(echo $0 | sed 's/.*test-packages-\([^.]*\)\.sh.*/\1/')
- if test -z "$WORKSPACE" ; then
- echo "Must set WORKSPACE"
- exit 1
- fi
-
- self=$(readlink -f $0)
- base=$(dirname $self)
-
- cd $WORKSPACE/packages/$2
- dpkg-scanpackages . /dev/null | gzip -c9 > Packages.gz
-
- exec docker run \
- --rm \
- --volume=$WORKSPACE/packages/$2:/mnt \
- --volume=$self:/root/run-test.sh \
- --volume=$base/common-test-packages.sh:/root/common-test.sh \
- --workdir=/mnt \
- $3 \
- /root/run-test.sh
-fi
-
-echo "deb file:///mnt /" >>/etc/apt/sources.list
apt-get -qq update
if ! apt-get -qq --assume-yes --force-yes install \
python-arvados-python-client python-arvados-fuse arvados-node-manager
@@ -34,7 +14,7 @@ fi
mkdir -p /tmp/opts
cd /tmp/opts
-for r in /mnt/python-*amd64.deb ; do
+for r in /arvados/packages/$target/python-*amd64.deb ; do
dpkg-deb -x $r .
done
@@ -44,4 +24,4 @@ for so in $(find . -name "*.so") ; do
ldd $so | awk '($3 ~ /^\//){print $3}' | sort -u | xargs dpkg -S | cut -d: -f1 | sort -u
done
-exec /root/common-test.sh
+exec /jenkins/common-test-packages.sh
diff --git a/jenkins/dockerfiles/centos6/Dockerfile b/jenkins/dockerfiles/centos6/Dockerfile
index ca974df..d0daef6 100644
--- a/jenkins/dockerfiles/centos6/Dockerfile
+++ b/jenkins/dockerfiles/centos6/Dockerfile
@@ -19,7 +19,10 @@ RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
ADD https://www.softwarecollections.org/en/scls/rhscl/rh-python34/epel-6-x86_64/download/rhscl-rh-python34-epel-6-x86_64.noarch.rpm https://www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm /opt/
RUN yum -q -y install /opt/rhscl-*.rpm
-RUN yum -q -y install python27 rh-python34
+
+# Need to "touch" RPM database to workaround bug in interaction between
+# overlayfs and yum (https://bugzilla.redhat.com/show_bug.cgi?id=1213602)
+RUN touch /var/lib/rpm/* && yum -q -y install python27 rh-python34
ENV WORKSPACE /arvados
CMD ["scl", "enable", "rh-python34", "python27", "/usr/local/rvm/bin/rvm-exec default bash /jenkins/run-build-packages.sh --target centos6"]
diff --git a/jenkins/package-test-dockerfiles/centos6/Dockerfile b/jenkins/package-test-dockerfiles/centos6/Dockerfile
new file mode 100644
index 0000000..5e354e2
--- /dev/null
+++ b/jenkins/package-test-dockerfiles/centos6/Dockerfile
@@ -0,0 +1,9 @@
+FROM centos:6
+MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
+
+RUN yum -q install --assumeyes scl-utils && \
+ curl -L -O https://www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm && \
+ yum -q install --assumeyes rhscl-python27-epel-6-x86_64.noarch.rpm && \
+ yum -q install --assumeyes python27
+
+COPY localrepo.repo /etc/yum.repos.d/localrepo.repo
\ No newline at end of file
diff --git a/jenkins/package-test-dockerfiles/centos6/localrepo.repo b/jenkins/package-test-dockerfiles/centos6/localrepo.repo
new file mode 100644
index 0000000..ac6b898
--- /dev/null
+++ b/jenkins/package-test-dockerfiles/centos6/localrepo.repo
@@ -0,0 +1,5 @@
+[localrepo]
+name=Arvados Test
+baseurl=file:///arvados/packages/centos6
+gpgcheck=0
+enabled=1
diff --git a/jenkins/package-test-dockerfiles/debian7/Dockerfile b/jenkins/package-test-dockerfiles/debian7/Dockerfile
new file mode 100644
index 0000000..fb789c6
--- /dev/null
+++ b/jenkins/package-test-dockerfiles/debian7/Dockerfile
@@ -0,0 +1,4 @@
+FROM debian:7
+MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
+
+RUN echo "deb file:///arvados/packages/debian7/ /" >>/etc/apt/sources.list
diff --git a/jenkins/package-test-dockerfiles/debian8/Dockerfile b/jenkins/package-test-dockerfiles/debian8/Dockerfile
new file mode 100644
index 0000000..fbcd8f4
--- /dev/null
+++ b/jenkins/package-test-dockerfiles/debian8/Dockerfile
@@ -0,0 +1,4 @@
+FROM debian:8
+MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
+
+RUN echo "deb file:///arvados/packages/debian8/ /" >>/etc/apt/sources.list
\ No newline at end of file
diff --git a/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile b/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
new file mode 100644
index 0000000..4b6e936
--- /dev/null
+++ b/jenkins/package-test-dockerfiles/ubuntu1204/Dockerfile
@@ -0,0 +1,4 @@
+FROM ubuntu:precise
+MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
+
+RUN echo "deb file:///arvados/packages/ubuntu1204/ /" >>/etc/apt/sources.list
\ No newline at end of file
diff --git a/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile b/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
new file mode 100644
index 0000000..4f49d92
--- /dev/null
+++ b/jenkins/package-test-dockerfiles/ubuntu1404/Dockerfile
@@ -0,0 +1,4 @@
+FROM ubuntu:trusty
+MAINTAINER Peter Amstutz <peter.amstutz at curoverse.com>
+
+RUN echo "deb file:///arvados/packages/ubuntu1404/ /" >>/etc/apt/sources.list
\ No newline at end of file
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index 5451a4e..3e0e136 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -10,6 +10,8 @@ Syntax:
Distribution to build packages for (default: debian7)
--command
Build command to execute (default: use built-in Docker image command)
+--test-packages
+ Run package install test script "test-packages-$target.sh"
WORKSPACE=path Path to the Arvados source tree to build packages from
@@ -32,7 +34,7 @@ if ! [[ -d "$WORKSPACE" ]]; then
fi
PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,target:,command: \
+ help,test-packages,target:,command: \
-- "" "$@")
if [ $? -ne 0 ]; then
exit 1
@@ -55,6 +57,9 @@ while [ $# -gt 0 ]; do
--command)
COMMAND="$2"; shift
;;
+ --test-packages)
+ test_packages=1
+ ;;
--)
if [ $# -gt 1 ]; then
echo >&2 "$0: unrecognized argument '$2'. Try: $0 --help"
@@ -67,34 +72,48 @@ done
set -e
-if [[ "$COMMAND" != "" ]]; then
- COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND --target $TARGET"
+if [[ -n "$test_packages" ]]; then
+ if [[ -n "$(find $WORKSPACE/packages/$TARGET -name *.rpm)" ]] ; then
+ createrepo $WORKSPACE/packages/$TARGET
+ fi
+
+ if [[ -n "$(find $WORKSPACE/packages/$TARGET -name *.deb)" ]] ; then
+ (cd $WORKSPACE/packages/$TARGET
+ dpkg-scanpackages . /dev/null | gzip -c > Packages.gz
+ )
+ fi
+
+ COMMAND="/jenkins/test-packages-$TARGET.sh"
+ IMAGE="arvados/package-test:$TARGET"
+else
+ IMAGE="arvados/build:$TARGET"
+ if [[ "$COMMAND" != "" ]]; then
+ COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND --target $TARGET"
+ fi
fi
-FINAL_EXITCODE=0
JENKINS_DIR=$(dirname "$(readlink -e "$0")")
-run_docker() {
- local tag=$1; shift
- if docker run -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
- --env ARVADOS_DEBUG=1 "arvados/build:$tag" $COMMAND; then
- # Success - nothing more to do.
- true
- else
- FINAL_EXITCODE=$?
- echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
- fi
-}
+if [[ -n "$test_packages" ]]; then
+ pushd "$JENKINS_DIR/package-test-dockerfiles"
+else
+ pushd "$JENKINS_DIR/dockerfiles"
+fi
-# In case it's needed, build the container. This costs just a few
-# seconds when the container already exist, so it's not a big deal to
-# do it on each run.
-cd "$JENKINS_DIR/dockerfiles"
echo $TARGET
cd $TARGET
-time docker build -t arvados/build:$TARGET .
-cd ..
+time docker build --tag=$IMAGE .
+popd
-run_docker $TARGET
+FINAL_EXITCODE=0
+
+if docker run --rm -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
+ --env ARVADOS_DEBUG=1 "$IMAGE" $COMMAND; then
+ # Success - nothing more to do.
+ true
+else
+ FINAL_EXITCODE=$?
+ echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
+fi
exit $FINAL_EXITCODE
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index 555b9f1..c8a62b5 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -1,62 +1,29 @@
-#!/bin/sh
+#!/bin/bash
-if test "$1" = --run-test ; then
-
- if test -z "$WORKSPACE" ; then
- echo "Must set WORKSPACE"
+if ! python --version |& grep "Python 2.7" >/dev/null ; then
+ # Re-execute in the software collections Python 2.7 environment.
+ exec scl enable python27 $0
+else
+ yum -q clean all
+ yum -q update
+ if ! yum -q install --assumeyes \
+ python27-python-arvados-python-client python27-python-arvados-fuse arvados-node-manager
+ then
exit 1
fi
- self=$(readlink -f $0)
- base=$(dirname $self)
+ mkdir -p /tmp/opts
+ cd /tmp/opts
- createrepo $WORKSPACE/packages/centos6
+ for r in /arvados/packages/centos6/python27-python-*x86_64.rpm ; do
+ rpm2cpio $r | cpio -idm
+ done
- exec docker run \
- --rm \
- --volume=$WORKSPACE/packages/centos6:/mnt \
- --volume=$(readlink -f $0):/root/run-test.sh \
- --volume=$base/common-test-packages.sh:/root/common-test.sh \
- --workdir=/mnt \
- centos:6 \
- /root/run-test.sh --install-scl
-fi
+ for so in $(find . -name "*.so") ; do
+ echo
+ echo "== Packages dependencies for $so =="
+ ldd $so | awk '($3 ~ /^\//){print $3}' | sort -u | xargs rpm -qf | sort -u
+ done
-if test "$1" = --install-scl ; then
- yum -q install --assumeyes scl-utils
- curl -L -O https://www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm
- yum -q install --assumeyes rhscl-python27-epel-6-x86_64.noarch.rpm
- yum -q install --assumeyes python27
- exec scl enable python27 $0
+ exec /jenkins/common-test-packages.sh
fi
-
-cat >/etc/yum.repos.d/localrepo.repo <<EOF
-[localrepo]
-name=Arvados Test
-baseurl=file:///mnt
-gpgcheck=0
-enabled=1
-EOF
-
-yum -q clean all
-yum -q update
-if ! yum -q install --assumeyes \
- python27-python-arvados-python-client python27-python-arvados-fuse arvados-node-manager
-then
- exit 1
-fi
-
-mkdir -p /tmp/opts
-cd /tmp/opts
-
-for r in /mnt/python27-python-*x86_64.rpm ; do
- rpm2cpio $r | cpio -idm
-done
-
-for so in $(find . -name "*.so") ; do
- echo
- echo "== Packages dependencies for $so =="
- ldd $so | awk '($3 ~ /^\//){print $3}' | sort -u | xargs rpm -qf | sort -u
-done
-
-exec /root/common-test.sh
diff --git a/jenkins/test-packages-debian7.sh b/jenkins/test-packages-debian7.sh
deleted file mode 100755
index c5cbfe6..0000000
--- a/jenkins/test-packages-debian7.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/sh
-
-exec ./deb-common-test-packages.sh --run-test debian7 debian:wheezy
diff --git a/jenkins/test-packages-debian7.sh b/jenkins/test-packages-debian7.sh
new file mode 120000
index 0000000..54ce94c
--- /dev/null
+++ b/jenkins/test-packages-debian7.sh
@@ -0,0 +1 @@
+deb-common-test-packages.sh
\ No newline at end of file
diff --git a/jenkins/test-packages-debian8.sh b/jenkins/test-packages-debian8.sh
deleted file mode 100755
index 5502f46..0000000
--- a/jenkins/test-packages-debian8.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/sh
-
-exec ./deb-common-test-packages.sh --run-test debian8 debian:jessie
diff --git a/jenkins/test-packages-debian8.sh b/jenkins/test-packages-debian8.sh
new file mode 120000
index 0000000..54ce94c
--- /dev/null
+++ b/jenkins/test-packages-debian8.sh
@@ -0,0 +1 @@
+deb-common-test-packages.sh
\ No newline at end of file
diff --git a/jenkins/test-packages-ubuntu1204.sh b/jenkins/test-packages-ubuntu1204.sh
deleted file mode 100755
index 00727a0..0000000
--- a/jenkins/test-packages-ubuntu1204.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/sh
-
-exec ./deb-common-test-packages.sh --run-test ubuntu1204 ubuntu:precise
diff --git a/jenkins/test-packages-ubuntu1204.sh b/jenkins/test-packages-ubuntu1204.sh
new file mode 120000
index 0000000..54ce94c
--- /dev/null
+++ b/jenkins/test-packages-ubuntu1204.sh
@@ -0,0 +1 @@
+deb-common-test-packages.sh
\ No newline at end of file
diff --git a/jenkins/test-packages-ubuntu1404.sh b/jenkins/test-packages-ubuntu1404.sh
deleted file mode 100755
index bfc2c85..0000000
--- a/jenkins/test-packages-ubuntu1404.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/sh
-
-exec ./deb-common-test-packages.sh --run-test ubuntu1404 ubuntu:trusty
diff --git a/jenkins/test-packages-ubuntu1404.sh b/jenkins/test-packages-ubuntu1404.sh
new file mode 120000
index 0000000..54ce94c
--- /dev/null
+++ b/jenkins/test-packages-ubuntu1404.sh
@@ -0,0 +1 @@
+deb-common-test-packages.sh
\ No newline at end of file
commit b67e59900349a05e6ed030f1cb92020a8109e7dd
Merge: 51601a2 bc27dd2
Author: radhika <radhika at curoverse.com>
Date: Thu Oct 15 12:38:41 2015 -0400
refs #7167
Merge branch '7167-keep-rsync-test-setup'
commit bc27dd23b329c7d82deebb3065917dceba18da27
Merge: 82b4217 51601a2
Author: radhika <radhika at curoverse.com>
Date: Thu Oct 15 12:36:16 2015 -0400
Merge branch 'master' into 7167-keep-rsync-test-setup
commit 51601a277992c213e60c6bebbc876631891ce699
Merge: d3f13c1 f4d018c
Author: Brett Smith <brett at curoverse.com>
Date: Thu Oct 15 11:57:43 2015 -0400
Merge branch '7341-deploy-retries-puppet-agent-wip'
Closes #7341, #7549.
commit 39520ae3caa8227e58471b8783106a3c2dba1304
Merge: 823a16a 2d11eaa
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Oct 15 10:40:17 2015 -0400
Merge branch 'master' into 7370-package-install-testing
commit f4d018c979b333d61fed9b1884d7edab540f3c1a
Author: Brett Smith <brett at curoverse.com>
Date: Wed Oct 14 18:18:58 2015 -0400
7341: run-deploy.sh retries failed Puppet agent runs.
It simply keeps retrying any failure until ten minutes have passed,
per Nico's suggestion in the comments.
Bonus bugfix: make sure run_puppet's ECODE variable gets set to the
exit status of Puppet (or the wrapper script), and not tee.
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 4953659..b81218e 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -64,6 +64,21 @@ EXITCODE=0
COLUMNS=80
+PUPPET_AGENT='
+now() { date +%s; }
+let endtime="$(now) + 600"
+while [ "$endtime" -gt "$(now)" ]; do
+ puppet agent --test --detailed-exitcodes
+ agent_exitcode=$?
+ if [ 0 = "$agent_exitcode" ] || [ 2 = "$agent_exitcode" ]; then
+ break
+ else
+ sleep 10s
+ fi
+done
+exit ${agent_exitcode:-99}
+'
+
title () {
date=`date +'%Y-%m-%d %H:%M:%S'`
printf "$date $1\n"
@@ -76,12 +91,12 @@ function run_puppet() {
title "Running puppet on $node"
TMP_FILE=`mktemp`
if [[ "$DEBUG" != "0" ]]; then
- ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C "/usr/bin/puppet agent -t" | tee $TMP_FILE
+ ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C bash -c "'$PUPPET_AGENT'" | tee $TMP_FILE
else
- ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C "/usr/bin/puppet agent -t" > $TMP_FILE 2>&1
+ ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C bash -c "'$PUPPET_AGENT'" > $TMP_FILE 2>&1
fi
- ECODE=$?
+ ECODE=${PIPESTATUS[0]}
RESULT=$(cat $TMP_FILE)
if [[ "$ECODE" != "255" && ! ("$RESULT" =~ 'already in progress') && "$ECODE" != "2" && "$ECODE" != "0" ]]; then
commit d3f13c19bdc7b2efe160c66116461faf865fed4e
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Oct 13 10:18:55 2015 -0400
Make sure to run the arvados-api-server-upgrade.sh and
arvados-workbench-upgrade.sh scripts in the RVM environment in
run-deploy.sh.
no issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 8762aae..4953659 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -143,7 +143,7 @@ title "Updating API server"
SUM_ECODE=0
run_puppet $IDENTIFIER.arvadosapi.com ECODE
SUM_ECODE=$(($SUM_ECODE + $ECODE))
-run_command $IDENTIFIER.arvadosapi.com ECODE "/usr/local/bin/arvados-api-server-upgrade.sh"
+run_command $IDENTIFIER.arvadosapi.com ECODE "/usr/local/rvm/bin/rvm-exec /usr/local/bin/arvados-api-server-upgrade.sh"
SUM_ECODE=$(($SUM_ECODE + $ECODE))
if [ ! "$IDENTIFIER" = "c97qk" ]
then
@@ -219,7 +219,7 @@ if [[ `host workbench.$ARVADOS_API_HOST |cut -f4 -d' '` != `host $ARVADOS_API_HO
SUM_ECODE=$(($SUM_ECODE + $ECODE))
fi
-run_command workbench.$IDENTIFIER ECODE "/usr/local/bin/arvados-workbench-upgrade.sh"
+run_command workbench.$IDENTIFIER ECODE "/usr/local/rvm/bin/rvm-exec /usr/local/bin/arvados-workbench-upgrade.sh"
SUM_ECODE=$(($SUM_ECODE + $ECODE))
if [[ "$SUM_ECODE" != "0" ]]; then
commit 8f535386bf3cf4bb0e6a929e4a39796fae235f3f
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Oct 9 13:09:20 2015 -0400
Make it possible to pass --debug through to the script run inside the Docker
build containers.
No issue #
diff --git a/jenkins/run-build-packages-all-targets.sh b/jenkins/run-build-packages-all-targets.sh
index ddf87c1..8c889a0 100755
--- a/jenkins/run-build-packages-all-targets.sh
+++ b/jenkins/run-build-packages-all-targets.sh
@@ -10,6 +10,8 @@ Options:
--command
Build command to execute (default: use built-in Docker image command)
+--debug
+ Output debug information (default: false)
WORKSPACE=path Path to the Arvados source tree to build packages from
@@ -34,13 +36,14 @@ fi
set -e
PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,command: \
+ help,debug,command: \
-- "" "$@")
if [ $? -ne 0 ]; then
exit 1
fi
COMMAND=
+DEBUG=
eval set -- "$PARSEDOPTS"
while [ $# -gt 0 ]; do
@@ -50,6 +53,9 @@ while [ $# -gt 0 ]; do
echo >&2
exit 1
;;
+ --debug)
+ DEBUG=" --debug"
+ ;;
--command)
COMMAND="$2"; shift
;;
@@ -64,7 +70,7 @@ while [ $# -gt 0 ]; do
done
if [[ "$COMMAND" != "" ]]; then
- COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND"
+ COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND$DEBUG"
fi
FINAL_EXITCODE=0
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index 5451a4e..2d8d8df 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -10,6 +10,8 @@ Syntax:
Distribution to build packages for (default: debian7)
--command
Build command to execute (default: use built-in Docker image command)
+--debug
+ Output debug information (default: false)
WORKSPACE=path Path to the Arvados source tree to build packages from
@@ -32,7 +34,7 @@ if ! [[ -d "$WORKSPACE" ]]; then
fi
PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,target:,command: \
+ help,debug,target:,command: \
-- "" "$@")
if [ $? -ne 0 ]; then
exit 1
@@ -40,6 +42,7 @@ fi
TARGET=debian7
COMMAND=
+DEBUG=
eval set -- "$PARSEDOPTS"
while [ $# -gt 0 ]; do
@@ -52,6 +55,9 @@ while [ $# -gt 0 ]; do
--target)
TARGET="$2"; shift
;;
+ --debug)
+ DEBUG=" --debug"
+ ;;
--command)
COMMAND="$2"; shift
;;
@@ -68,7 +74,7 @@ done
set -e
if [[ "$COMMAND" != "" ]]; then
- COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND --target $TARGET"
+ COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND --target $TARGET$DEBUG"
fi
FINAL_EXITCODE=0
commit 3af59ee413454b94ad45bb5d8990896186b5d672
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Oct 9 13:02:39 2015 -0400
Ensure proper permissions on /usr/local/arvados/src in the arvados-src package.
refs #7397
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index f813ac1..ff9acf6 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -271,6 +271,8 @@ handle_python_package
COMMIT_HASH=$(format_last_commit_here "%H")
SRC_BUILD_DIR=$(mktemp -d)
+ # mktemp creates the directory with 0700 permissions by default
+ chmod 755 $SRC_BUILD_DIR
git clone $DASHQ_UNLESS_DEBUG "$WORKSPACE/.git" "$SRC_BUILD_DIR"
cd "$SRC_BUILD_DIR"
commit 82b42172688762c4f5fa7727a6974c94e8ccd86a
Merge: d993317 be68e90
Author: radhika <radhika at curoverse.com>
Date: Fri Oct 9 10:36:48 2015 -0400
Merge branch 'master' into 7167-keep-rsync-test-setup
commit be68e901353548dd87ccadcd62ae82551b1995d2
Author: Nico Cesar <nico at curoverse.com>
Date: Wed Oct 7 10:09:16 2015 -0400
7460: c97qk doesnt need mailchimp deployed. this caused deploy to fail in Ubuntu
refs #7460
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 06680a0..8762aae 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -145,8 +145,11 @@ run_puppet $IDENTIFIER.arvadosapi.com ECODE
SUM_ECODE=$(($SUM_ECODE + $ECODE))
run_command $IDENTIFIER.arvadosapi.com ECODE "/usr/local/bin/arvados-api-server-upgrade.sh"
SUM_ECODE=$(($SUM_ECODE + $ECODE))
-run_command $IDENTIFIER.arvadosapi.com ECODE "dpkg -L arvados-mailchimp-plugin 2>/dev/null && apt-get install arvados-mailchimp-plugin --reinstall || echo"
-SUM_ECODE=$(($SUM_ECODE + $ECODE))
+if [ ! "$IDENTIFIER" = "c97qk" ]
+then
+ run_command $IDENTIFIER.arvadosapi.com ECODE "dpkg -L arvados-mailchimp-plugin 2>/dev/null && apt-get install arvados-mailchimp-plugin --reinstall || echo"
+ SUM_ECODE=$(($SUM_ECODE + $ECODE))
+fi
if [[ "$SUM_ECODE" != "0" ]]; then
title "ERROR: Updating API server FAILED"
commit 2d11eaae9b48ebf7998b37d978466d26ddcfc769
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Oct 7 10:07:25 2015 -0400
Run CWL tests for both draft-2 and draft-3. No issue #
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
index 2aa68c2..53c0538 100755
--- a/jenkins/run-cwl-tests.sh
+++ b/jenkins/run-cwl-tests.sh
@@ -175,7 +175,8 @@ fi
./build-node-docker.sh
)
-./run_test.sh RUNNER=cwltool/cwltool/main.py
+./run_test.sh RUNNER=cwltool/cwltool/main.py DRAFT=draft-2
+./run_test.sh RUNNER=cwltool/cwltool/main.py DRAFT=draft-3
(cd cwltool
handle_python_package
commit 5f2cf1041d2579bc4c29d5691b9ebaf9250f39b8
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Oct 7 10:02:43 2015 -0400
Update cwl builder process for repository reorganization. No issue #
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
index e148171..2aa68c2 100755
--- a/jenkins/run-cwl-tests.sh
+++ b/jenkins/run-cwl-tests.sh
@@ -156,25 +156,38 @@ fi
cd "$WORKSPACE"
-pushd reference
-python setup.py install
-python setup.py test
-./build-node-docker.sh
-popd
+if test -d cwltool ; then
+ (cd cwltool
+ git fetch
+ git reset --hard origin/master
+ )
+else
+ git clone git at github.com:common-workflow-language/cwltool.git
+ (cd cwltool
+ git config user.email "sysadmin at curoverse.com"
+ git config user.name "Curoverse build bot"
+ )
+fi
-pushd conformance
-pwd
-./run_test.sh
-popd
+(cd cwltool
+ python setup.py install
+ python setup.py test
+ ./build-node-docker.sh
+)
-cd reference
-handle_python_package
+./run_test.sh RUNNER=cwltool/cwltool/main.py
-cd cwl-runner
-handle_python_package
-cd ..
+(cd cwltool
+ handle_python_package
+)
-./build-cwl-docker.sh
+(cd cwltool/cwl-runner
+ handle_python_package
+)
+
+(cd cwltool
+ ./build-cwl-docker.sh
+)
if [[ "$UPLOAD_DOCKER" != 0 ]]; then
docker push commonworkflowlanguage/cwltool_module
@@ -182,27 +195,23 @@ if [[ "$UPLOAD_DOCKER" != 0 ]]; then
docker push commonworkflowlanguage/nodejs-engine
fi
-# Setup virtualenv and build documentation.
-
-virtualenv ../venv
-. ../venv/bin/activate
-python setup.py install
-cd ..
-
if test -d common-workflow-language.github.io ; then
- cd common-workflow-language.github.io
- git fetch
- git reset --hard origin/master
- cd ..
+ (cd common-workflow-language.github.io
+ git fetch
+ git reset --hard origin/master
+ )
else
git clone git at github.com:common-workflow-language/common-workflow-language.github.io.git
- cd common-workflow-language.github.io
- git config user.email "sysadmin at curoverse.com"
- git config user.name "Curoverse build bot"
- cd ..
+ (cd common-workflow-language.github.io
+ git config user.email "sysadmin at curoverse.com"
+ git config user.name "Curoverse build bot"
+ )
fi
-python -mcwltool --outdir=$PWD/common-workflow-language.github.io specification/cwlsite.cwl specification/cwlsite-job.json
-cd common-workflow-language.github.io
-git add --all
-git diff-index --quiet HEAD || git commit -m"Build bot"
-git push
+
+python -mcwltool --outdir=$PWD/common-workflow-language.github.io site/cwlsite.cwl site/cwlsite-job.json
+
+(cd common-workflow-language.github.io
+ git add --all
+ git diff-index --quiet HEAD || git commit -m"Build bot"
+ git push
+)
commit 44b7b4e113d39b34e35d6b2323fee9fbad21611c
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Oct 5 14:21:04 2015 -0400
Update pin of libcloud fork to dev4 refs #7286
diff --git a/jenkins/libcloud-pin b/jenkins/libcloud-pin
index 4ebd37f..c399227 100644
--- a/jenkins/libcloud-pin
+++ b/jenkins/libcloud-pin
@@ -1 +1 @@
-LIBCLOUD_PIN=0.18.1.dev3
\ No newline at end of file
+LIBCLOUD_PIN=0.18.1.dev4
\ No newline at end of file
commit da075298c7ddbefa68947783a787b46f659fefdf
Merge: fec33dc 182891a
Author: Brett Smith <brett at curoverse.com>
Date: Mon Oct 5 13:14:00 2015 -0400
Merge branch '7451-dont-recreate-venv3dir-wip'
Refs #7451. Closes #7452.
commit 182891aa10947a1238195a8565c051efd1dd5480
Author: Brett Smith <brett at curoverse.com>
Date: Mon Oct 5 12:53:42 2015 -0400
7451: Improve virtualenv setup in run-tests.sh.
The code to build the Python 3 virtualenv didn't check to see whether
the virtualenv already existed first. When reusing a Python 3
virtualenv, pip would get downgraded and start failing.
Refactor the virtualenv setup code to avoid this duplication and these
sorts of bugs.
While I was at it: rather than trying to parse package versions
ourselves to figure out when to upgrade setuptools and pip, just tell
pip the versions we want. 'pip>=7' is an approximation of what we're
currently doing. The main thing we care about is that we get a
version new enough to work with setuptools 18.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 0224f33..2e5e320 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -384,6 +384,14 @@ gem_uninstall_if_exists() {
fi
}
+setup_virtualenv() {
+ local venvdest=$1; shift
+ if ! [[ -e "$venvdest/bin/activate" ]] || ! [[ -e "$venvdest/bin/pip" ]]; then
+ virtualenv --setuptools "$@" "$venvdest" || fatal "virtualenv $venvdest failed"
+ fi
+ "$venvdest/bin/pip" install 'setuptools>=18' 'pip>=7'
+}
+
export PERLINSTALLBASE
export PERLLIB="$PERLINSTALLBASE/lib/perl5:${PERLLIB:+$PERLLIB}"
@@ -392,15 +400,9 @@ mkdir -p "$GOPATH/src/git.curoverse.com"
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
|| fatal "symlink failed"
-if ! [[ -e "$VENVDIR/bin/activate" ]] || ! [[ -e "$VENVDIR/bin/pip" ]]; then
- virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
-fi
+setup_virtualenv "$VENVDIR" --python python2.7
. "$VENVDIR/bin/activate"
-if (pip install setuptools | grep setuptools-0) || [ "$($VENVDIR/bin/easy_install --version | cut -d\ -f2 | cut -d. -f1)" -lt 18 ]; then
- pip install --upgrade setuptools pip
-fi
-
# Needed for run_test_server.py which is used by certain (non-Python) tests.
pip freeze 2>/dev/null | egrep ^PyYAML= \
|| pip install PyYAML >/dev/null \
@@ -419,17 +421,7 @@ deactivate
# Otherwise, skip dependent tests.
PYTHON3=$(which python3)
if [ "0" = "$?" ]; then
- virtualenv --python "$PYTHON3" --setuptools "$VENV3DIR" \
- || fatal "python3 virtualenv $VENV3DIR failed"
-
- . "$VENV3DIR/bin/activate"
-
- if (pip install setuptools | grep setuptools-0) || [ "$($VENV3DIR/bin/easy_install --version | cut -d\ -f2 | cut -d. -f1)" -lt 18 ]; then
- pip install --upgrade setuptools pip
- fi
-
- # Deactivate Python 3 virtualenv
- deactivate
+ setup_virtualenv "$VENV3DIR" --python python3
else
PYTHON3=
skip[services/dockercleaner]=1
commit fec33dcaf67cb752c669915bb49dead9b8f11e4f
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Oct 5 12:48:56 2015 -0400
Fix incorrect line numbers on compile errors reported by "go test" in coverage mode. No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 01b471a..0224f33 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -493,15 +493,22 @@ do_test_once() {
then
covername="coverage-$(echo "$1" | sed -e 's/\//_/g')"
coverflags=("-covermode=count" "-coverprofile=$WORKSPACE/tmp/.$covername.tmp")
+ # We do "go get -t" here to catch compilation errors
+ # before trying "go test". Otherwise, coverage-reporting
+ # mode makes Go show the wrong line numbers when reporting
+ # compilation errors.
if [[ -n "${testargs[$1]}" ]]
then
# "go test -check.vv giturl" doesn't work, but this
# does:
- cd "$WORKSPACE/$1" && go test ${coverflags[@]} ${testargs[$1]}
+ cd "$WORKSPACE/$1" && \
+ go get -t "git.curoverse.com/arvados.git/$1" && \
+ go test ${coverflags[@]} ${testargs[$1]}
else
# The above form gets verbose even when testargs is
# empty, so use this form in such cases:
- go test ${coverflags[@]} "git.curoverse.com/arvados.git/$1"
+ go get -t "git.curoverse.com/arvados.git/$1" && \
+ go test ${coverflags[@]} "git.curoverse.com/arvados.git/$1"
fi
result="$?"
go tool cover -html="$WORKSPACE/tmp/.$covername.tmp" -o "$WORKSPACE/tmp/$covername.html"
commit d99331757367dbbb3607a4644a5a6572667cb262
Author: radhika <radhika at curoverse.com>
Date: Mon Oct 5 07:37:48 2015 -0400
7167: add keep-rsync to gostuff.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 01b471a..62786dc 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -74,6 +74,7 @@ sdk/ruby
sdk/go/arvadosclient
sdk/go/keepclient
sdk/go/streamer
+tools/keep-rsync
EOF
@@ -695,16 +696,17 @@ do_install services/api apiserver
declare -a gostuff
gostuff=(
+ sdk/go/arvadosclient
+ sdk/go/streamer
services/arv-git-httpd
services/crunchstat
services/keepstore
+ sdk/go/keepclient
services/keepproxy
services/datamanager/summary
services/datamanager/collection
services/datamanager
- sdk/go/arvadosclient
- sdk/go/keepclient
- sdk/go/streamer
+ tools/keep-rsync
)
for g in "${gostuff[@]}"
do
commit 5a9879c969a110494880301464c80d648fcd560b
Author: Brett Smith <brett at curoverse.com>
Date: Wed Sep 30 11:11:59 2015 -0400
7356: CentOS 6 needs a Python 3 backport for websocket-client.
Refs #7356.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index a6909df..f813ac1 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -149,7 +149,7 @@ case "$TARGET" in
pykka requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl
python-daemon lockfile llfuse)
- PYTHON3_BACKPORTS=(docker-py six requests)
+ PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
export PYCURL_SSL_LIBRARY=nss
;;
*)
commit a9f03b2a636cbc7dc16628a5de8701cc66e04871
Merge: 25ada28 1c82c1f
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Sep 29 13:49:35 2015 -0400
Merge branch '7330-postinst-for-sso-server'
refs #7330
commit 1c82c1f875ca2681ff36e804fde8f6beb8f36c77
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Sep 28 21:00:32 2015 -0400
More fixes for the SSO server packages.
refs #7330
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index ecd9b44..436f5bd 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -6,6 +6,7 @@ INSTALL_PATH=/var/www/arvados-sso
RELEASE_PATH=$INSTALL_PATH/current
SHARED_PATH=$INSTALL_PATH/shared
CONFIG_PATH=/etc/arvados/sso
+PACKAGE_NAME=arvados-sso-server
DATABASE_READY=1
APPLICATION_READY=1
@@ -78,8 +79,8 @@ EOF
echo
echo "Assumption: $WEB_SERVICE is configured to serve your SSO server URL from"
- echo " /var/www/arvados-sso/current"
- echo "Assumption: configuration files are in /etc/arvados/sso/"
+ echo " $RELEASE_PATH"
+ echo "Assumption: configuration files are in $CONFIG_PATH"
echo "Assumption: $WEB_SERVICE and passenger run as $WWW_OWNER"
echo
@@ -211,8 +212,8 @@ if [ "$DATABASE_READY" = "0" ]; then
PLEASE NOTE:
-The arvados-sso package was not configured completely because
-/etc/arvados/database.yml needs some tweaking. Please refer to the
+The $PACKAGE_NAME package was not configured completely because
+$CONFIG_PATH/database.yml needs some tweaking. Please refer to the
documentation at http://doc.arvados.org/install/install-sso.html#configure for
more details.
@@ -225,8 +226,8 @@ if [ "$APPLICATION_READY" = "0" ]; then
PLEASE NOTE:
-The arvados-sso package was not configured completely because
-/etc/arvados/application.yml needs some tweaking. Please refer to the
+The $PACKAGE_NAME package was not configured completely because
+$CONFIG_PATH/application.yml needs some tweaking. Please refer to the
documentation at http://doc.arvados.org/install/install-sso.html#configure for
more details.
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index 8d1de52..2550135 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -3,7 +3,7 @@
. `dirname "$(readlink -f "$0")"`/run-library.sh
read -rd "\000" helpmessage <<EOF
-$(basename $0): Build Arvados SSO package
+$(basename $0): Build Arvados SSO server package
Syntax:
WORKSPACE=/path/to/arvados-sso $(basename $0) [options]
@@ -173,7 +173,7 @@ fi
cd "$WORKSPACE"
SSO_VERSION=$(version_from_git)
-PACKAGE_NAME=arvados-sso
+PACKAGE_NAME=arvados-sso-server
if [[ ! -d "$WORKSPACE/tmp" ]]; then
mkdir $WORKSPACE/tmp
diff --git a/jenkins/run-test-packages-sso.sh b/jenkins/run-test-packages-sso.sh
index 02e1b04..03d4a96 100755
--- a/jenkins/run-test-packages-sso.sh
+++ b/jenkins/run-test-packages-sso.sh
@@ -110,17 +110,29 @@ if ! [[ -d "$WORKSPACE" ]]; then
exit 1
fi
+title () {
+ txt="********** $1 **********"
+ printf "\n%*s%s\n\n" $((($COLUMNS-${#txt})/2)) "" "$txt"
+}
+
+checkexit() {
+ if [[ "$1" != "0" ]]; then
+ title "!!!!!! $2 FAILED !!!!!!"
+ fi
+}
+
+
# Find the SSO server package
cd "$WORKSPACE"
SSO_VERSION=$(version_from_git)
-PACKAGE_NAME=arvados-sso
+PACKAGE_NAME=arvados-sso-server
if [[ "$FORMAT" == "deb" ]]; then
- PACKAGE_PATH=$WORKSPACE/packages/$TARGET/arvados-sso_${SSO_VERSION}_amd64.deb
+ PACKAGE_PATH=$WORKSPACE/packages/$TARGET/${PACKAGE_NAME}_${SSO_VERSION}_amd64.deb
elif [[ "$FORMAT" == "rpm" ]]; then
- PACKAGE_PATH=$WORKSPACE/packages/$TARGET/arvados-sso-${SSO_VERSION}-1.x86_64.rpm
+ PACKAGE_PATH=$WORKSPACE/packages/$TARGET/${PACKAGE_NAME}-${SSO_VERSION}-1.x86_64.rpm
fi
# Test 1a: the package to test must exist
@@ -132,9 +144,9 @@ fi
if [[ "$FORMAT" == "deb" ]]; then
# Test 1b: the system/container where we're running the tests must be clean
set +e
- dpkg -l |grep arvados-sso -q
+ dpkg -l |grep $PACKAGE_NAME -q
if [[ "$?" != "1" ]]; then
- echo "Please make sure the arvados-sso package is not installed before running this script"
+ echo "Please make sure the $PACKAGE_NAME package is not installed before running this script"
exit 1
fi
set -e
@@ -209,42 +221,73 @@ if [[ "$FORMAT" == "deb" ]]; then
$SUDO dpkg -i $PACKAGE_PATH > /dev/null 2>&1
$SUDO apt-get -f install --yes
set -e
- $SUDO dpkg -i $PACKAGE_PATH
+ $SUDO dpkg -i $PACKAGE_PATH || EXITCODE=2
+
+ checkexit $EXITCODE "dpkg -i $PACKAGE_PATH"
# Test 3: the package should remove cleanly
- $SUDO apt-get remove arvados-sso --yes
+ $SUDO apt-get remove $PACKAGE_NAME --yes || EXITCODE=3
+
+ checkexit $EXITCODE "apt-get remove $PACKAGE_PATH --yes"
# Test 4: the package configuration should remove cleanly
- $SUDO dpkg --purge arvados-sso
+ $SUDO dpkg --purge $PACKAGE_NAME || EXITCODE=4
+
+ checkexit $EXITCODE "dpkg --purge $PACKAGE_PATH"
if [[ -e "/var/www/arvados-sso" ]]; then
- echo "Error: leftover items under /var/www/arvados-sso."
- exit 4
+ EXITCODE=4
fi
+ checkexit $EXITCODE "leftover items under /var/www/arvados-sso"
+
# Test 5: the package should remove cleanly with --purge
- $SUDO dpkg -i $PACKAGE_PATH
- $SUDO apt-get remove arvados-sso --purge --yes
+ $SUDO dpkg -i $PACKAGE_PATH || EXITCODE=5
+
+ checkexit $EXITCODE "dpkg -i $PACKAGE_PATH"
+
+ $SUDO apt-get remove $PACKAGE_NAME --purge --yes || EXITCODE=5
+
+ checkexit $EXITCODE "apt-get remove $PACKAGE_PATH --purge --yes"
if [[ -e "/var/www/arvados-sso" ]]; then
- echo "Error: leftover items under /var/www/arvados-sso."
- exit 5
+ EXITCODE=5
fi
+ checkexit $EXITCODE "leftover items under /var/www/arvados-sso"
+
elif [[ "$FORMAT" == "rpm" ]]; then
+
+ # Set up Nginx first
+ # (courtesy of https://www.phusionpassenger.com/library/walkthroughs/deploy/ruby/ownserver/nginx/oss/el6/install_passenger.html)
+ $SUDO yum install -q -y epel-release pygpgme curl
+ $SUDO curl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo
+ $SUDO yum install -q -y nginx passenger
+ $SUDO sed -i -e 's/^# passenger/passenger/' /etc/nginx/conf.d/passenger.conf
+ # Done setting up Nginx
+
# Test 2: the package should install cleanly
- $SUDO yum -q -y --nogpgcheck localinstall $PACKAGE_PATH
+ $SUDO yum -q -y --nogpgcheck localinstall $PACKAGE_PATH || EXITCODE=3
+
+ checkexit $EXITCODE "yum -q -y --nogpgcheck localinstall $PACKAGE_PATH"
# Test 3: the package should remove cleanly
- $SUDO yum -q -y remove arvados-sso
+ $SUDO yum -q -y remove $PACKAGE_NAME || EXITCODE=3
+
+ checkexit $EXITCODE "yum -q -y remove $PACKAGE_PATH"
if [[ -e "/var/www/arvados-sso" ]]; then
- echo "Error: leftover items under /var/www/arvados-sso."
- exit 3
+ EXITCODE=3
fi
+ checkexit $EXITCODE "leftover items under /var/www/arvados-sso"
+
fi
-echo "Testing complete, no errors!"
+if [[ "$EXITCODE" == "0" ]]; then
+ echo "Testing complete, no errors!"
+else
+ echo "Errors while testing!"
+fi
exit $EXITCODE
commit 9ac3e54bc97741867159976495d723a06f2cf28d
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Sep 28 17:42:58 2015 -0400
A number of additional tweaks.
refs #7330
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index 736453b..ecd9b44 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -7,6 +7,9 @@ RELEASE_PATH=$INSTALL_PATH/current
SHARED_PATH=$INSTALL_PATH/shared
CONFIG_PATH=/etc/arvados/sso
+DATABASE_READY=1
+APPLICATION_READY=1
+
VERSION=`cat $RELEASE_PATH/git-commit.version`
setup_ruby_environment() {
@@ -82,18 +85,35 @@ EOF
echo -n "Symlinking files from $CONFIG_PATH ..."
+ if [ ! -f $CONFIG_PATH/database.yml ]; then
+ mkdir -p $CONFIG_PATH
+ cp $RELEASE_PATH/config/database.yml.example $CONFIG_PATH/database.yml
+ DATABASE_READY=0
+ fi
+
if [ ! -h $RELEASE_PATH/config/database.yml ]; then
ln -s $CONFIG_PATH/database.yml $RELEASE_PATH/config/database.yml
fi
+
+ if [ ! -f $CONFIG_PATH/production.rb ]; then
+ mkdir -p $CONFIG_PATH
+ cp $RELEASE_PATH/config/environments/production.rb.example $CONFIG_PATH/production.rb
+ fi
+
if [ ! -h $RELEASE_PATH/config/environments/production.rb ]; then
- if [ ! -f $CONFIG_PATH/production.rb ]; then
- cp $RELEASE_PATH/config/environments/production.rb.example $CONFIG_PATH/production.rb
- fi
ln -s $CONFIG_PATH/production.rb $RELEASE_PATH/config/environments/production.rb
fi
+
+ if [ ! -f $CONFIG_PATH/application.yml ]; then
+ mkdir -p $CONFIG_PATH
+ cp $RELEASE_PATH/config/application.yml.example $CONFIG_PATH/application.yml
+ APPLICATION_READY=0
+ fi
+
if [ ! -h $RELEASE_PATH/config/application.yml ]; then
ln -s $CONFIG_PATH/application.yml $RELEASE_PATH/config/application.yml
fi
+
echo "... done."
# Before we do anything else, make sure some directories and files are in place
@@ -122,7 +142,7 @@ EOF
echo "... done."
set +e
- DB_MIGRATE_STATUS=`$COMMAND_PREFIX bundle exec rake db:migrate:status 2>/dev/null`
+ DB_MIGRATE_STATUS=`$COMMAND_PREFIX bundle exec rake db:migrate:status 2>&1`
DB_MIGRATE_STATUS_CODE=$?
set -e
@@ -130,28 +150,50 @@ EOF
# The database exists, but the migrations table doesn't.
echo -n "Setting up database ..."
$COMMAND_PREFIX bundle exec rake db:schema:load db:seed || exit $?
+ echo "... done."
elif echo $DB_MIGRATE_STATUS | grep '^database: ' >/dev/null; then
echo -n "Running db:migrate ..."
$COMMAND_PREFIX bundle exec rake db:migrate || exit $?
+ echo "... done."
+ elif echo $DB_MIGRATE_STATUS | grep 'database .* does not exist' >/dev/null; then
+ echo -n "Running db:setup ..."
+ set +e
+ $COMMAND_PREFIX bundle exec rake db:setup 2>/dev/null
+ if [ "$?" = "0" ]; then
+ echo "... done."
+ else
+ echo "... failed."
+ echo "Warning: unable to set up database." >&2
+ DATABASE_READY=0
+ fi
+ set -e
else
- echo "Error: Database is not ready to set up. Aborting." >&2
- exit 1
+ echo "Warning: Database is not ready to set up. Skipping database setup." >&2
+ DATABASE_READY=0
fi
- echo "... done."
echo -n "Precompiling assets ..."
# precompile assets; thankfully this does not take long
- $COMMAND_PREFIX bundle exec rake assets:precompile -q -s || exit $?
+ if [ "$APPLICATION_READY" = "1" ]; then
+ set +e
+ $COMMAND_PREFIX bundle exec rake assets:precompile -q -s 2>/dev/null
+ if [ "$?" = "0" ]; then
+ echo "... done."
+ else
+ echo "... failed."
+ APPLICATION_READY=0
+ fi
+ set -e
+ else
+ echo "... skipped."
+ fi
chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
- echo "... done."
if [ ! -z "$WEB_SERVICE" ]; then
echo -n "Restarting $WEB_SERVICE ..."
- service "$WEB_SERVICE" restart || exit $?
+ service "$WEB_SERVICE" restart >/dev/null || exit $?
echo "... done."
fi
-
- echo
}
if [ "$1" = configure ]; then
@@ -163,3 +205,33 @@ elif [ "$1" = "0" ] || [ "$1" = "1" ] || [ "$1" = "2" ]; then
setup_ruby_environment
configure_version $VERSION
fi
+
+if [ "$DATABASE_READY" = "0" ]; then
+ cat <<EOF
+
+PLEASE NOTE:
+
+The arvados-sso package was not configured completely because
+/etc/arvados/database.yml needs some tweaking. Please refer to the
+documentation at http://doc.arvados.org/install/install-sso.html#configure for
+more details.
+
+When database.yml has been modified, reconfigure or reinstall this package.
+EOF
+fi
+
+if [ "$APPLICATION_READY" = "0" ]; then
+ cat <<EOF
+
+PLEASE NOTE:
+
+The arvados-sso package was not configured completely because
+/etc/arvados/application.yml needs some tweaking. Please refer to the
+documentation at http://doc.arvados.org/install/install-sso.html#configure for
+more details.
+
+When application.yml has been modified, reconfigure or reinstall this package.
+EOF
+fi
+
+echo
commit 3789a46123ea8a3c90977d8365d3720f9f6585b1
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Sep 28 00:23:16 2015 -0400
Add support for CentOS6 to the improved SSO packages.
refs #7330
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index cc23396..736453b 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -31,40 +31,53 @@ configure_version() {
[ "$VERSION" ] || { echo "Error: configure_version: need version parameter" >&2; exit 1; }
- if [ -e /etc/redhat-release ]; then
- WWW_OWNER=nginx:nginx
- else
- # Assume we're on a Debian-based system for now.
- WWW_OWNER=www-data:www-data
- fi
+ WEB_SERVICE=${WEB_SERVICE:-$(service --status-all 2>/dev/null \
+ | grep -Eo '\bnginx|httpd[^[:space:]]*' || true)}
+ if [ -z "$WEB_SERVICE" ]; then
+ cat >&2 <<EOF
+
+Warning: web service (Nginx or Apache) not found.
- NGINX_SERVICE=${NGINX_SERVICE:-$(service --status-all 2>/dev/null \
- | grep -Eo '\bnginx[^[:space:]]*' || true)}
- if [ -z "$NGINX_SERVICE" ]; then
- cat >&2 <<EOF
-Error: nginx service not found. Aborting.
+To override, set the WEB_SERVICE environment variable to the name of the service
+hosting the Rails server. Alternativey, install nginx.
-Set NGINX_SERVICE to the name of the service hosting the Rails server
-and reconfigure this package with dpkg-reconfigure.
+For Debian-based systems, then reconfigure this package with dpkg-reconfigure.
+
+For RPM-based systems, then reinstall this package.
EOF
- exit 0
- elif [ "$NGINX_SERVICE" != "$(echo "$NGINX_SERVICE" | head -n 1)" ]; then
- cat >&2 <<EOF
-Error: multiple nginx services found. Aborting.
+ elif [ "$WEB_SERVICE" != "$(echo "$WEB_SERVICE" | head -n 1)" ]; then
+ WEB_SERVICE=$(echo "$WEB_SERVICE" | head -n 1)
+ cat >&2 <<EOF
+Warning: multiple web services found. Choosing the first one ($WEB_SERVICE).
+
+To override, set the WEB_SERVICE environment variable to the name of the service
+hosting the Rails server.
+
+For Debian-based systems, then reconfigure this package with dpkg-reconfigure.
-Set NGINX_SERVICE to the name of the service hosting the Rails server
-and reconfigure this package with dpkg-reconfigure.
+For RPM-based systems, then reinstall this package.
EOF
- exit 0
+ fi
+
+ if [ -e /etc/redhat-release ]; then
+ if [ "$WEB_SERVICE" = "nginx" ]; then
+ WWW_OWNER=nginx:nginx
+ else
+ WWW_OWNER=apache:apache
+ fi
+ else
+ # Assume we're on a Debian-based system for now.
+ # Both Apache and Nginx run as www-data by default.
+ WWW_OWNER=www-data:www-data
fi
echo
- echo "Assumption: $NGINX_SERVICE is configured to serve your SSO server URL from"
+ echo "Assumption: $WEB_SERVICE is configured to serve your SSO server URL from"
echo " /var/www/arvados-sso/current"
echo "Assumption: configuration files are in /etc/arvados/sso/"
- echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
+ echo "Assumption: $WEB_SERVICE and passenger run as $WWW_OWNER"
echo
echo -n "Symlinking files from $CONFIG_PATH ..."
@@ -132,15 +145,21 @@ EOF
chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
echo "... done."
- echo -n "Restarting nginx ..."
- service "$NGINX_SERVICE" restart || exit $?
- echo "... done."
+ if [ ! -z "$WEB_SERVICE" ]; then
+ echo -n "Restarting $WEB_SERVICE ..."
+ service "$WEB_SERVICE" restart || exit $?
+ echo "... done."
+ fi
+
echo
}
if [ "$1" = configure ]; then
+ # This is a debian-based system
setup_ruby_environment
configure_version $VERSION "$2"
+elif [ "$1" = "0" ] || [ "$1" = "1" ] || [ "$1" = "2" ]; then
+ # This is an rpm-based system
+ setup_ruby_environment
+ configure_version $VERSION
fi
-
-
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postrm b/jenkins/arvados-sso-server-extras/arvados-sso-server.postrm
index 1d73a68..070c90e 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postrm
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postrm
@@ -15,8 +15,12 @@ purge () {
}
if [ "$1" = 'purge' ]; then
+ # This is a debian-based system and purge was requested
+ purge
+elif [ "$1" = "0" ] || [ "$1" = "1" ] || [ "$1" = "2" ]; then
+ # This is an rpm-based system, no guarantees are made, always purge
+ # Apparently yum doesn't actually remember what it installed.
+ # Clean those files up here, then purge.
+ rm -rf $RELEASE_PATH
purge
fi
-
-
-
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.prerm b/jenkins/arvados-sso-server-extras/arvados-sso-server.prerm
index 677a6ac..1a43fa2 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.prerm
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.prerm
@@ -14,9 +14,13 @@ remove () {
rm -rf $RELEASE_PATH/public/assets/
rm -rf $RELEASE_PATH/tmp
rm -rf $RELEASE_PATH/.bundle
- rm $RELEASE_PATH/log
+ rm $RELEASE_PATH/log || true
}
if [ "$1" = 'remove' ]; then
+ # This is a debian-based system and removal was requested
+ remove
+elif [ "$1" = "0" ] || [ "$1" = "1" ] || [ "$1" = "2" ]; then
+ # This is an rpm-based system
remove
fi
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index ada7a82..8d1de52 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -136,6 +136,20 @@ else
GEM=gem
fi
+if [[ "$TARGET" == "centos6" ]]; then
+ # CentOS6 comes with git 1.7.1, but we want at least 1.7.6
+ # because we use git status --ignore in fpm-info.sh
+ cd /usr/src
+ install_package libcurl-devel zlib-devel wget gettext
+ wget https://www.kernel.org/pub/software/scm/git/git-1.8.5.6.tar.gz
+ tar xzf git-1.8.5.6.tar.gz
+ cd git-1.8.5.6
+ make configure
+ ./configure --prefix=/usr --without-tcltk
+ make all
+ make install
+fi
+
# Make all files world-readable -- jenkins runs with umask 027, and has checked
# out our git tree here
chmod o+r "$WORKSPACE" -R
@@ -218,6 +232,7 @@ debug_echo -e "\n${COMMAND_ARR[@]}\n"
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
+
fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
# SSO server package build done
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
index f07397f..f19d58b 100755
--- a/jenkins/run-library.sh
+++ b/jenkins/run-library.sh
@@ -183,3 +183,12 @@ fpm_verify () {
echo "Error building package for $1:\n $FPM_RESULTS"
fi
}
+
+install_package() {
+ PACKAGES=$@
+ if [[ "$FORMAT" == "deb" ]]; then
+ $SUDO apt-get install $PACKAGES --yes
+ elif [[ "$FORMAT" == "rpm" ]]; then
+ $SUDO yum -q -y install $PACKAGES
+ fi
+}
diff --git a/jenkins/run-test-packages-sso.sh b/jenkins/run-test-packages-sso.sh
index b9476bb..02e1b04 100755
--- a/jenkins/run-test-packages-sso.sh
+++ b/jenkins/run-test-packages-sso.sh
@@ -110,15 +110,6 @@ if ! [[ -d "$WORKSPACE" ]]; then
exit 1
fi
-install_package() {
- PACKAGES=$@
- if [[ "$FORMAT" == "deb" ]]; then
- $SUDO apt-get install $PACKAGES --yes
- elif [[ "$FORMAT" == "rpm" ]]; then
- $SUDO yum -q -y install $PACKAGES
- fi
-}
-
# Find the SSO server package
cd "$WORKSPACE"
@@ -129,7 +120,7 @@ PACKAGE_NAME=arvados-sso
if [[ "$FORMAT" == "deb" ]]; then
PACKAGE_PATH=$WORKSPACE/packages/$TARGET/arvados-sso_${SSO_VERSION}_amd64.deb
elif [[ "$FORMAT" == "rpm" ]]; then
- PACKAGE_PATH=$WORKSPACE/packages/$TARGET/arvados-sso-${SSO_VERSION}.x86_64.rpm
+ PACKAGE_PATH=$WORKSPACE/packages/$TARGET/arvados-sso-${SSO_VERSION}-1.x86_64.rpm
fi
# Test 1a: the package to test must exist
@@ -170,7 +161,19 @@ if [[ ! -e "/etc/arvados/sso/application.yml" ]]; then
fi
if [[ ! -e "/etc/arvados/sso/database.yml" ]]; then
- install_package postgresql
+ # We haven't installed our dependencies yet, but we need to set up our
+ # database configuration now. Install postgresql if need be.
+ if [[ "$FORMAT" == "deb" ]]; then
+ install_package postgresql
+ elif [[ "$FORMAT" == "rpm" ]]; then
+ install_package postgresql-server
+ # postgres packaging on CentOS6 is kind of primitive, needs an initdb
+ $SUDO service postgresql initdb
+ if [ "$TARGET" = "centos6" ]; then
+ sed -i -e "s/127.0.0.1\/32 ident/127.0.0.1\/32 md5/" /var/lib/pgsql/data/pg_hba.conf
+ sed -i -e "s/::1\/128 ident/::1\/128 md5/" /var/lib/pgsql/data/pg_hba.conf
+ fi
+ fi
$SUDO service postgresql start
RANDOM_PASSWORD=`date | md5sum |cut -f1 -d' '`
@@ -188,12 +191,15 @@ EOF
$SUDO -u postgres createdb sso_provider_production -O sso_provider_user
else
install_package sudo
+ if [ "$TARGET" = "centos6" ]; then
+ # Work around silly CentOS6 default, cf. https://bugzilla.redhat.com/show_bug.cgi?id=1020147
+ sed -i -e 's/Defaults requiretty/#Defaults requiretty/' /etc/sudoers
+ fi
/usr/bin/sudo -u postgres psql -c "CREATE USER sso_provider_user WITH PASSWORD '$RANDOM_PASSWORD'"
/usr/bin/sudo -u postgres createdb sso_provider_production -O sso_provider_user
fi
fi
-
if [[ "$FORMAT" == "deb" ]]; then
# Test 2: the package should install cleanly
# In order to get the package dependencies, we need to first do an install
@@ -239,4 +245,6 @@ elif [[ "$FORMAT" == "rpm" ]]; then
fi
+echo "Testing complete, no errors!"
+
exit $EXITCODE
commit f23d3167e9163944b7a52ce9d822b930053bb957
Author: Ward Vandewege <ward at curoverse.com>
Date: Sun Sep 27 15:59:56 2015 -0400
Initial version of run-test-packages-sso.sh
refs #7330
diff --git a/jenkins/run-test-packages-sso.sh b/jenkins/run-test-packages-sso.sh
new file mode 100755
index 0000000..b9476bb
--- /dev/null
+++ b/jenkins/run-test-packages-sso.sh
@@ -0,0 +1,242 @@
+#!/bin/bash
+
+. `dirname "$(readlink -f "$0")"`/run-library.sh
+
+read -rd "\000" helpmessage <<EOF
+$(basename $0): Test Arvados SSO package
+
+Syntax:
+ WORKSPACE=/path/to/arvados-sso $(basename $0) [options]
+
+Options:
+
+--debug
+ Output debug information (default: false)
+--target
+ Distribution to build packages for (default: debian7)
+
+WORKSPACE=path Path to the Arvados SSO source tree
+
+EOF
+
+EXITCODE=0
+DEBUG=${ARVADOS_DEBUG:-0}
+BUILD_BUNDLE_PACKAGES=0
+TARGET=debian7
+SUDO=/usr/bin/sudo
+
+if [[ ! -e "$SUDO" ]]; then
+ SUDO=
+fi
+
+PARSEDOPTS=$(getopt --name "$0" --longoptions \
+ help,build-bundle-packages,debug,target: \
+ -- "" "$@")
+if [ $? -ne 0 ]; then
+ exit 1
+fi
+
+eval set -- "$PARSEDOPTS"
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --help)
+ echo >&2 "$helpmessage"
+ echo >&2
+ exit 1
+ ;;
+ --target)
+ TARGET="$2"; shift
+ ;;
+ --debug)
+ DEBUG=1
+ ;;
+ --build-bundle-packages)
+ BUILD_BUNDLE_PACKAGES=1
+ ;;
+ --)
+ if [ $# -gt 1 ]; then
+ echo >&2 "$0: unrecognized argument '$2'. Try: $0 --help"
+ exit 1
+ fi
+ ;;
+ esac
+ shift
+done
+
+STDOUT_IF_DEBUG=/dev/null
+STDERR_IF_DEBUG=/dev/null
+DASHQ_UNLESS_DEBUG=-q
+if [[ "$DEBUG" != 0 ]]; then
+ STDOUT_IF_DEBUG=/dev/stdout
+ STDERR_IF_DEBUG=/dev/stderr
+ DASHQ_UNLESS_DEBUG=
+fi
+
+case "$TARGET" in
+ debian7)
+ FORMAT=deb
+ ;;
+ debian8)
+ FORMAT=deb
+ ;;
+ ubuntu1204)
+ FORMAT=deb
+ ;;
+ ubuntu1404)
+ FORMAT=deb
+ ;;
+ centos6)
+ FORMAT=rpm
+ ;;
+ *)
+ echo -e "$0: Unknown target '$TARGET'.\n" >&2
+ exit 1
+ ;;
+esac
+
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
+fi
+
+if ! [[ -d "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: $WORKSPACE is not a directory"
+ echo >&2
+ exit 1
+fi
+
+install_package() {
+ PACKAGES=$@
+ if [[ "$FORMAT" == "deb" ]]; then
+ $SUDO apt-get install $PACKAGES --yes
+ elif [[ "$FORMAT" == "rpm" ]]; then
+ $SUDO yum -q -y install $PACKAGES
+ fi
+}
+
+# Find the SSO server package
+
+cd "$WORKSPACE"
+
+SSO_VERSION=$(version_from_git)
+PACKAGE_NAME=arvados-sso
+
+if [[ "$FORMAT" == "deb" ]]; then
+ PACKAGE_PATH=$WORKSPACE/packages/$TARGET/arvados-sso_${SSO_VERSION}_amd64.deb
+elif [[ "$FORMAT" == "rpm" ]]; then
+ PACKAGE_PATH=$WORKSPACE/packages/$TARGET/arvados-sso-${SSO_VERSION}.x86_64.rpm
+fi
+
+# Test 1a: the package to test must exist
+if [[ ! -f $PACKAGE_PATH ]]; then
+ echo "Latest package not found at $PACKAGE_PATH. Please build the package first."
+ exit 1
+fi
+
+if [[ "$FORMAT" == "deb" ]]; then
+ # Test 1b: the system/container where we're running the tests must be clean
+ set +e
+ dpkg -l |grep arvados-sso -q
+ if [[ "$?" != "1" ]]; then
+ echo "Please make sure the arvados-sso package is not installed before running this script"
+ exit 1
+ fi
+ set -e
+fi
+
+if [[ -e "/var/www/arvados-sso" ]]; then
+ echo "Please make sure /var/www/arvados-sso does not exist before running this script"
+ exit 1
+fi
+
+# Prepare the machine
+if [[ "$FORMAT" == "deb" ]]; then
+ $SUDO apt-get update
+elif [[ "$FORMAT" == "rpm" ]]; then
+ $SUDO yum check-update
+fi
+$SUDO mkdir -p /etc/arvados/sso
+
+if [[ ! -e "/etc/arvados/sso/application.yml" ]]; then
+ RANDOM_PASSWORD=`date | md5sum |cut -f1 -d' '`
+ cp config/application.yml.example /etc/arvados/sso/application.yml
+ sed -i -e 's/uuid_prefix: ~/uuid_prefix: zzzzz/' /etc/arvados/sso/application.yml
+ sed -i -e "s/secret_token: ~/secret_token: $RANDOM_PASSWORD/" /etc/arvados/sso/application.yml
+fi
+
+if [[ ! -e "/etc/arvados/sso/database.yml" ]]; then
+ install_package postgresql
+ $SUDO service postgresql start
+
+ RANDOM_PASSWORD=`date | md5sum |cut -f1 -d' '`
+ cat >/etc/arvados/sso/database.yml <<EOF
+production:
+ adapter: postgresql
+ encoding: utf8
+ database: sso_provider_production
+ username: sso_provider_user
+ password: $RANDOM_PASSWORD
+ host: localhost
+EOF
+ if [[ "$SUDO" != '' ]]; then
+ $SUDO -u postgres psql -c "CREATE USER sso_provider_user WITH PASSWORD '$RANDOM_PASSWORD'"
+ $SUDO -u postgres createdb sso_provider_production -O sso_provider_user
+ else
+ install_package sudo
+ /usr/bin/sudo -u postgres psql -c "CREATE USER sso_provider_user WITH PASSWORD '$RANDOM_PASSWORD'"
+ /usr/bin/sudo -u postgres createdb sso_provider_production -O sso_provider_user
+ fi
+fi
+
+
+if [[ "$FORMAT" == "deb" ]]; then
+ # Test 2: the package should install cleanly
+ # In order to get the package dependencies, we need to first do an install
+ # with dpkg that will fail, then run apt-get to install the dependencies,
+ # and the subsequent dpkg installation should succeed.
+ set +e
+ $SUDO dpkg -i $PACKAGE_PATH > /dev/null 2>&1
+ $SUDO apt-get -f install --yes
+ set -e
+ $SUDO dpkg -i $PACKAGE_PATH
+
+ # Test 3: the package should remove cleanly
+ $SUDO apt-get remove arvados-sso --yes
+
+ # Test 4: the package configuration should remove cleanly
+ $SUDO dpkg --purge arvados-sso
+
+ if [[ -e "/var/www/arvados-sso" ]]; then
+ echo "Error: leftover items under /var/www/arvados-sso."
+ exit 4
+ fi
+
+ # Test 5: the package should remove cleanly with --purge
+ $SUDO dpkg -i $PACKAGE_PATH
+ $SUDO apt-get remove arvados-sso --purge --yes
+
+ if [[ -e "/var/www/arvados-sso" ]]; then
+ echo "Error: leftover items under /var/www/arvados-sso."
+ exit 5
+ fi
+
+elif [[ "$FORMAT" == "rpm" ]]; then
+ # Test 2: the package should install cleanly
+ $SUDO yum -q -y --nogpgcheck localinstall $PACKAGE_PATH
+
+ # Test 3: the package should remove cleanly
+ $SUDO yum -q -y remove arvados-sso
+
+ if [[ -e "/var/www/arvados-sso" ]]; then
+ echo "Error: leftover items under /var/www/arvados-sso."
+ exit 3
+ fi
+
+fi
+
+exit $EXITCODE
commit 170c33818a1ac6a3e35f411d80cf77a008bd6e07
Author: Ward Vandewege <ward at curoverse.com>
Date: Sun Sep 27 15:29:15 2015 -0400
Remove bash-ism from arvados-sso-server.postinst.
refs #7330
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index e6db44c..cc23396 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -108,17 +108,21 @@ EOF
chmod 644 $SHARED_PATH/log/*
echo "... done."
- # If we use `grep -q`, rake will write a backtrace on EPIPE.
- if $COMMAND_PREFIX bundle exec rake db:migrate:status | grep '^database: ' >/dev/null; then
- echo -n "Running db:migrate ..."
- $COMMAND_PREFIX bundle exec rake db:migrate || exit $?
- elif [ 0 -eq ${PIPESTATUS[0]} ]; then
- # The database exists, but the migrations table doesn't.
- echo -n "Setting up database ..."
- $COMMAND_PREFIX bundle exec rake db:schema:load db:seed || exit $?
+ set +e
+ DB_MIGRATE_STATUS=`$COMMAND_PREFIX bundle exec rake db:migrate:status 2>/dev/null`
+ DB_MIGRATE_STATUS_CODE=$?
+ set -e
+
+ if echo $DB_MIGRATE_STATUS | grep 'Schema migrations table does not exist yet.' >/dev/null; then
+ # The database exists, but the migrations table doesn't.
+ echo -n "Setting up database ..."
+ $COMMAND_PREFIX bundle exec rake db:schema:load db:seed || exit $?
+ elif echo $DB_MIGRATE_STATUS | grep '^database: ' >/dev/null; then
+ echo -n "Running db:migrate ..."
+ $COMMAND_PREFIX bundle exec rake db:migrate || exit $?
else
- echo "Error: Database is not ready to set up. Aborting." >&2
- exit 1
+ echo "Error: Database is not ready to set up. Aborting." >&2
+ exit 1
fi
echo "... done."
commit c21753f5f91a3144395f4b259b09fb16b39b4b2d
Author: Ward Vandewege <ward at curoverse.com>
Date: Sun Sep 27 11:22:32 2015 -0400
Add prerm and postrm scripts, various cleanups.
refs #7330
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
index 383d25c..e6db44c 100755
--- a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -108,8 +108,6 @@ EOF
chmod 644 $SHARED_PATH/log/*
echo "... done."
- ls -laF $RELEASE_PATH/tmp
-
# If we use `grep -q`, rake will write a backtrace on EPIPE.
if $COMMAND_PREFIX bundle exec rake db:migrate:status | grep '^database: ' >/dev/null; then
echo -n "Running db:migrate ..."
@@ -124,16 +122,12 @@ EOF
fi
echo "... done."
- ls -laF $RELEASE_PATH/tmp
-
echo -n "Precompiling assets ..."
# precompile assets; thankfully this does not take long
$COMMAND_PREFIX bundle exec rake assets:precompile -q -s || exit $?
chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
echo "... done."
- ls -laF $RELEASE_PATH/tmp
-
echo -n "Restarting nginx ..."
service "$NGINX_SERVICE" restart || exit $?
echo "... done."
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postrm b/jenkins/arvados-sso-server-extras/arvados-sso-server.postrm
new file mode 100755
index 0000000..1d73a68
--- /dev/null
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postrm
@@ -0,0 +1,22 @@
+#!/bin/sh
+
+set -e
+
+INSTALL_PATH=/var/www/arvados-sso
+RELEASE_PATH=$INSTALL_PATH/current
+SHARED_PATH=$INSTALL_PATH/shared
+CONFIG_PATH=/etc/arvados/sso
+
+purge () {
+ rm -rf $SHARED_PATH/vendor_bundle
+ rm -rf $SHARED_PATH/log
+ rmdir $SHARED_PATH || true
+ rmdir $INSTALL_PATH || true
+}
+
+if [ "$1" = 'purge' ]; then
+ purge
+fi
+
+
+
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.prerm b/jenkins/arvados-sso-server-extras/arvados-sso-server.prerm
new file mode 100755
index 0000000..677a6ac
--- /dev/null
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.prerm
@@ -0,0 +1,22 @@
+#!/bin/sh
+
+set -e
+
+INSTALL_PATH=/var/www/arvados-sso
+RELEASE_PATH=$INSTALL_PATH/current
+SHARED_PATH=$INSTALL_PATH/shared
+CONFIG_PATH=/etc/arvados/sso
+
+remove () {
+ rm -f $RELEASE_PATH/config/database.yml
+ rm -f $RELEASE_PATH/config/environments/production.rb
+ rm -f $RELEASE_PATH/config/application.yml
+ rm -rf $RELEASE_PATH/public/assets/
+ rm -rf $RELEASE_PATH/tmp
+ rm -rf $RELEASE_PATH/.bundle
+ rm $RELEASE_PATH/log
+}
+
+if [ "$1" = 'remove' ]; then
+ remove
+fi
diff --git a/jenkins/arvados-sso-server-extras/prerm.sh b/jenkins/arvados-sso-server-extras/prerm.sh
deleted file mode 100755
index b1280b8..0000000
--- a/jenkins/arvados-sso-server-extras/prerm.sh
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/bin/sh
-
-RELEASE_PATH=/var/www/arvados-sso/current
-rm -f $RELEASE_PATH/config/database.yml
-rm -f $RELEASE_PATH/config/environments/production.rb
-rm -f $RELEASE_PATH/config/application.yml
-rm -rf $RELEASE_PATH/public/assets/
-rm -rf $RELEASE_PATH/tmp
-rm $RELEASE_PATH/log
-
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index 6c3c03b..ada7a82 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -187,7 +187,7 @@ cd $WORKSPACE/packages/$TARGET
# if it exists at the time fpm-info.sh runs. If it does not exist at that time, this script
# will create it and when fpm runs, it will include the directory. So we add it to the exclude
# list explicitly here, just in case.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados SSO server - Arvados is a free and open source platform for big data science.'" "--license='Expat License'" "-s" "dir" "-t" "$FORMAT" "-v" "$SSO_VERSION" "-x" "var/www/arvados-sso/current/.git" "-x" "var/www/arvados-sso/current/packages" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server.postinst")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados SSO server - Arvados is a free and open source platform for big data science.'" "--license='Expat License'" "-s" "dir" "-t" "$FORMAT" "-v" "$SSO_VERSION" "-x" "var/www/arvados-sso/current/.git" "-x" "var/www/arvados-sso/current/packages" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server.postinst" "--before-remove=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server.prerm" "--after-remove=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server.postrm" )
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
# This is the complete package with vendor/bundle included.
commit 656b5e388467792f437caf1afa032dd4d36b2eb1
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Sep 25 21:42:29 2015 -0400
First commit.
refs #7330
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
new file mode 100755
index 0000000..383d25c
--- /dev/null
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server.postinst
@@ -0,0 +1,148 @@
+#!/bin/sh
+
+set -e
+
+INSTALL_PATH=/var/www/arvados-sso
+RELEASE_PATH=$INSTALL_PATH/current
+SHARED_PATH=$INSTALL_PATH/shared
+CONFIG_PATH=/etc/arvados/sso
+
+VERSION=`cat $RELEASE_PATH/git-commit.version`
+
+setup_ruby_environment() {
+ if [ -s "$HOME/.rvm/scripts/rvm" ] ; then
+ using_rvm="true"
+ elif [ -s "/usr/local/rvm/scripts/rvm" ] ; then
+ using_rvm="true"
+ else
+ using_rvm="false"
+ fi
+
+ if [ "$using_rvm" = "true" ]; then
+ COMMAND_PREFIX="/usr/local/rvm/bin/rvm-exec default"
+ else
+ COMMAND_PREFIX=
+ fi
+}
+
+# arguments: <major version> <most recently configured package version>
+configure_version() {
+ VERSION="$1"
+
+ [ "$VERSION" ] || { echo "Error: configure_version: need version parameter" >&2; exit 1; }
+
+ if [ -e /etc/redhat-release ]; then
+ WWW_OWNER=nginx:nginx
+ else
+ # Assume we're on a Debian-based system for now.
+ WWW_OWNER=www-data:www-data
+ fi
+
+ NGINX_SERVICE=${NGINX_SERVICE:-$(service --status-all 2>/dev/null \
+ | grep -Eo '\bnginx[^[:space:]]*' || true)}
+ if [ -z "$NGINX_SERVICE" ]; then
+ cat >&2 <<EOF
+Error: nginx service not found. Aborting.
+
+Set NGINX_SERVICE to the name of the service hosting the Rails server
+and reconfigure this package with dpkg-reconfigure.
+
+EOF
+ exit 0
+ elif [ "$NGINX_SERVICE" != "$(echo "$NGINX_SERVICE" | head -n 1)" ]; then
+ cat >&2 <<EOF
+Error: multiple nginx services found. Aborting.
+
+Set NGINX_SERVICE to the name of the service hosting the Rails server
+and reconfigure this package with dpkg-reconfigure.
+
+EOF
+ exit 0
+ fi
+
+ echo
+ echo "Assumption: $NGINX_SERVICE is configured to serve your SSO server URL from"
+ echo " /var/www/arvados-sso/current"
+ echo "Assumption: configuration files are in /etc/arvados/sso/"
+ echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
+ echo
+
+ echo -n "Symlinking files from $CONFIG_PATH ..."
+
+ if [ ! -h $RELEASE_PATH/config/database.yml ]; then
+ ln -s $CONFIG_PATH/database.yml $RELEASE_PATH/config/database.yml
+ fi
+ if [ ! -h $RELEASE_PATH/config/environments/production.rb ]; then
+ if [ ! -f $CONFIG_PATH/production.rb ]; then
+ cp $RELEASE_PATH/config/environments/production.rb.example $CONFIG_PATH/production.rb
+ fi
+ ln -s $CONFIG_PATH/production.rb $RELEASE_PATH/config/environments/production.rb
+ fi
+ if [ ! -h $RELEASE_PATH/config/application.yml ]; then
+ ln -s $CONFIG_PATH/application.yml $RELEASE_PATH/config/application.yml
+ fi
+ echo "... done."
+
+ # Before we do anything else, make sure some directories and files are in place
+ if [ ! -e $SHARED_PATH/log ]; then mkdir -p $SHARED_PATH/log; fi
+ if [ ! -e $RELEASE_PATH/tmp ]; then mkdir -p $RELEASE_PATH/tmp; fi
+ if [ ! -e $RELEASE_PATH/log ]; then ln -s $SHARED_PATH/log $RELEASE_PATH/log; fi
+ if [ ! -e $SHARED_PATH/log/production.log ]; then touch $SHARED_PATH/log/production.log; fi
+
+ cd "$RELEASE_PATH"
+ export RAILS_ENV=production
+
+ echo -n "Running bundle install ..."
+ $COMMAND_PREFIX bundle install --path $SHARED_PATH/vendor_bundle --quiet || exit $?
+ echo "... done."
+
+ echo -n "Ensuring directory and file permissions ..."
+ # Ensure correct ownership of a few files
+ chown "$WWW_OWNER" $RELEASE_PATH/config/environment.rb
+ chown "$WWW_OWNER" $RELEASE_PATH/config.ru
+ chown "$WWW_OWNER" $RELEASE_PATH/config/database.yml
+ chown "$WWW_OWNER" $RELEASE_PATH/Gemfile.lock
+ chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
+ chown -R "$WWW_OWNER" $SHARED_PATH/log
+ chown "$WWW_OWNER" $RELEASE_PATH/db/schema.rb
+ chmod 644 $SHARED_PATH/log/*
+ echo "... done."
+
+ ls -laF $RELEASE_PATH/tmp
+
+ # If we use `grep -q`, rake will write a backtrace on EPIPE.
+ if $COMMAND_PREFIX bundle exec rake db:migrate:status | grep '^database: ' >/dev/null; then
+ echo -n "Running db:migrate ..."
+ $COMMAND_PREFIX bundle exec rake db:migrate || exit $?
+ elif [ 0 -eq ${PIPESTATUS[0]} ]; then
+ # The database exists, but the migrations table doesn't.
+ echo -n "Setting up database ..."
+ $COMMAND_PREFIX bundle exec rake db:schema:load db:seed || exit $?
+ else
+ echo "Error: Database is not ready to set up. Aborting." >&2
+ exit 1
+ fi
+ echo "... done."
+
+ ls -laF $RELEASE_PATH/tmp
+
+ echo -n "Precompiling assets ..."
+ # precompile assets; thankfully this does not take long
+ $COMMAND_PREFIX bundle exec rake assets:precompile -q -s || exit $?
+ chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
+ echo "... done."
+
+ ls -laF $RELEASE_PATH/tmp
+
+ echo -n "Restarting nginx ..."
+ service "$NGINX_SERVICE" restart || exit $?
+ echo "... done."
+ echo
+}
+
+if [ "$1" = configure ]; then
+ setup_ruby_environment
+ configure_version $VERSION "$2"
+fi
+
+
diff --git a/jenkins/arvados-sso-server-extras/postinst.sh b/jenkins/arvados-sso-server-extras/postinst.sh
deleted file mode 100755
index ffd9b14..0000000
--- a/jenkins/arvados-sso-server-extras/postinst.sh
+++ /dev/null
@@ -1,12 +0,0 @@
-#!/bin/bash
-
-cd /var/www/arvados-sso
-
-chown -R www-data:www-data tmp >/dev/null 2>&1
-chown -R www-data:www-data log >/dev/null 2>&1
-chown www-data:www-data db/schema.rb >/dev/null 2>&1
-chmod 644 log/* >/dev/null 2>&1
-
-# Errors above are not serious
-exit 0
-
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index f985882..6c3c03b 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -173,19 +173,21 @@ fi
cd $WORKSPACE/packages/$TARGET
-# Annoyingly, we require a database.yml file for rake assets:precompile to work. So for now,
-# we do that in the upgrade script.
+# Annoyingly, we require a database.yml file for rake assets:precompile to work.
+
# TODO: add bogus database.yml file so we can precompile the assets and put them in the
# package. Then remove that database.yml file again. It has to be a valid file though.
#RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile
-# There are just 2 excludes left here, all the others are pulled in via fpm-info.sh.
-# The .git directory is excluded by git implicitly, so we can't pick it up from .gitignore.
-# The packages directory needs to be explictly excluded here because it will only be listed
+# There are just 2 excludes left here, all the others are pulled in via fpm-info.sh, which
+# takes .gitignore into account via a call to git status:
+#
+# 1. The .git directory is excluded by git implicitly, so we can't pick it up from .gitignore.
+# 2. The packages directory needs to be explictly excluded here because it will only be listed
# if it exists at the time fpm-info.sh runs. If it does not exist at that time, this script
# will create it and when fpm runs, it will include the directory. So we add it to the exclude
# list explicitly here, just in case.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados SSO server - Arvados is a free and open source platform for big data science.'" "--license='Expat License'" "-s" "dir" "-t" "$FORMAT" "-v" "$SSO_VERSION" "-x" "var/www/arvados-sso/current/.git" "-x" "var/www/arvados-sso/current/packages" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/postinst.sh")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados SSO server - Arvados is a free and open source platform for big data science.'" "--license='Expat License'" "-s" "dir" "-t" "$FORMAT" "-v" "$SSO_VERSION" "-x" "var/www/arvados-sso/current/.git" "-x" "var/www/arvados-sso/current/packages" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server.postinst")
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
# This is the complete package with vendor/bundle included.
@@ -211,7 +213,7 @@ for i in "${fpm_depends[@]}"; do
COMMAND_ARR+=('--depends' "$i")
done
COMMAND_ARR+=("${fpm_args[@]}")
-COMMAND_ARR+=("$WORKSPACE/=/var/www/arvados-sso/current" "$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server-upgrade.sh=/usr/local/bin/arvados-sso-server-upgrade.sh")
+COMMAND_ARR+=("$WORKSPACE/=/var/www/arvados-sso/current")
debug_echo -e "\n${COMMAND_ARR[@]}\n"
FPM_RESULTS=$("${COMMAND_ARR[@]}")
commit 25ada289bea32adcbaee5a5181eb94809693ec9c
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Sep 25 12:05:33 2015 -0400
Fix the test that avoids running puppet multiple times on installations
where workbench runs on the API server.
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index d2c12ee..06680a0 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -210,7 +210,7 @@ KEEP_NODES=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TO
title "Updating workbench"
SUM_ECODE=0
-if [[ `host workbench.$ARVADOS_API_HOST` != `host $ARVADOS_API_HOST` ]]; then
+if [[ `host workbench.$ARVADOS_API_HOST |cut -f4 -d' '` != `host $ARVADOS_API_HOST |cut -f4 -d' '` ]]; then
# Workbench runs on a separate host. We need to run puppet there too.
run_puppet workbench.$IDENTIFIER ECODE
SUM_ECODE=$(($SUM_ECODE + $ECODE))
commit 9819eac0a0ec331264e58864e66682ae9d436e82
Author: Brett Smith <brett at curoverse.com>
Date: Thu Sep 24 12:01:20 2015 -0400
7322: Don't backport virtualenv for Arvados.
The Arvados code only needs virtualenv 1.7, which is provided with all
of our supported distros.
run-tests.sh requires 1.8.3 to use the --setuptools switch, but that
can be handled without publishing a backported package.
Refs #7322.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 396eb92..a6909df 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -96,7 +96,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
- virtualenv pykka requests six pyexecjs jsonschema \
+ pykka requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -109,7 +109,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
- virtualenv pykka requests six pyexecjs jsonschema \
+ pykka requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -122,7 +122,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
- virtualenv pykka requests six pyexecjs jsonschema \
+ pykka requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
commit 469c824c8db7dd960350997b3f4d55bcb3245456
Merge: da2a7a1 7da79d9
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Sep 23 13:02:43 2015 -0400
Merge branch '7228-group-writable-tmp-cache' refs #7228
commit 7da79d9d4ec5509eefb06751173bb9ff5793a4ce
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Sep 23 09:09:25 2015 -0400
7228: Add chmod -R 2775 $RELEASE_PATH/tmp/cache/ so that cache is group writable.
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
index b3badf7..e48701c 100755
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -72,6 +72,7 @@ chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
chown -R "$WWW_OWNER" $SHARED_PATH/log
chown "$WWW_OWNER" $RELEASE_PATH/db/structure.sql
chmod 644 $SHARED_PATH/log/*
+chmod -R 2775 $RELEASE_PATH/tmp/cache/
echo "Done."
echo "Running sanity check"
commit da2a7a1bd6ef8ca7daafbf9b88d0754be3fc9382
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Sep 21 16:56:57 2015 -0400
Make the build step for our libcloud package a lot less noisy by default.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 8fba12a..396eb92 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -396,7 +396,11 @@ LIBCLOUD_DIR=$(mktemp -d)
cd $LIBCLOUD_DIR
git clone $DASHQ_UNLESS_DEBUG https://github.com/curoverse/libcloud.git .
git checkout apache-libcloud-$LIBCLOUD_PIN
+ # libcloud is absurdly noisy without -q, so force -q here
+ OLD_DASHQ_UNLESS_DEBUG=$DASHQ_UNLESS_DEBUG
+ DASHQ_UNLESS_DEBUG=-q
handle_python_package
+ DASHQ_UNLESS_DEBUG=$OLD_DASHQ_UNLESS_DEBUG
)
fpm_build $LIBCLOUD_DIR "$PYTHON2_PKG_PREFIX"-apache-libcloud
rm -rf $LIBCLOUD_DIR
commit 5ee7a415414d023634ef3e9b9333d49068791e79
Author: Brett Smith <brett at curoverse.com>
Date: Wed Sep 16 16:33:28 2015 -0400
Close unclosed quotes in arvados-api-server-upgrade.sh.
No issue #.
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
index b3badf7..f924b8b 100755
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -29,7 +29,7 @@ RELEASE_PATH=/var/www/arvados-api/current
SHARED_PATH=/var/www/arvados-api/shared
CONFIG_PATH=/etc/arvados/api/
-echo "Assumption: $NGINX_SERVICE is configured to serve your API server URL from
+echo "Assumption: $NGINX_SERVICE is configured to serve your API server URL from"
echo " /var/www/arvados-api/current"
echo "Assumption: configuration files are in /etc/arvados/api/"
echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
commit fb189c35101622a1bcd53d4c10b589963afd8747
Merge: c43d72c 716e832
Author: Brett Smith <brett at curoverse.com>
Date: Wed Sep 16 15:25:40 2015 -0400
Merge branch 'pr/1'
Closes #7261.
commit c43d72c6d70213350abf70e81710f98afa0a0862
Merge: 9afc5f2 3022574
Author: radhika <radhika at curoverse.com>
Date: Mon Sep 14 18:56:43 2015 -0400
refs #6260
Merge branch '6260-test-datamanager'
commit 302257484801465c1e0371e36dcc0d3a28392af1
Author: radhika <radhika at curoverse.com>
Date: Mon Sep 14 18:53:16 2015 -0400
6260: since datamanager no longer needs the data-manager-token-file (only keep server does), remove the run-tests.sh update for this.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index e2e4ab6..7793b3e 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -759,10 +759,6 @@ do_test services/dockercleaner pip "$VENV3DIR/bin/"
for g in "${gostuff[@]}"
do
- if [[ "$g" == "services/datamanager" ]]
- then
- testargs["$g"]+=" -data-manager-token-file $WORKSPACE/tmp/keep.data-manager-token-file"
- fi
do_test "$g" go
done
commit 9afc5f2d9ca0af29c34ff15e58b66387c3c149a6
Merge: 33692d9 f6f099e
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Sep 11 22:25:30 2015 -0400
Merge branch '7311-temp-base' closes #7311
commit f6f099ed8d235f23a70870b2e013afd6a085db00
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Sep 11 17:04:53 2015 -0400
7311: Do not forget to make temp dirs.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index c8e1cce..639cd14 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -300,13 +300,12 @@ fi
# Set up temporary install dirs (unless existing dirs were supplied)
for tmpdir in VENVDIR VENV3DIR GOPATH GEMHOME PERLINSTALLBASE
do
- if [[ -n "${!tmpdir}" ]]; then
- # Support old user-named temp dirs (but --temp {base} would
- # be more convenient)
- mkdir -p "${!tmpdir}"
- else
+ if [[ -z "${!tmpdir}" ]]; then
eval "$tmpdir"="$temp/$tmpdir"
fi
+ if ! [[ -d "${!tmpdir}" ]]; then
+ mkdir "${!tmpdir}" || fatal "can't create ${!tmpdir} (does $temp exist?)"
+ fi
done
setup_ruby_environment() {
commit f0c491076403ee51e42604e603aacb9873944aba
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Sep 11 10:44:16 2015 -0400
7311: Make recognizable temp dirs under a single base; allow override with --temp DIR
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 43a8acb..c8e1cce 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -14,9 +14,11 @@ Options:
--skip FOO Do not test the FOO component.
--only FOO Do not test anything except the FOO component.
+--temp DIR Install components and dependencies under DIR instead of
+ making a new temporary directory. Implies --leave-temp.
--leave-temp Do not remove GOPATH, virtualenv, and other temp dirs at exit.
- Instead, show which directories were used this time so they
- can be reused in subsequent invocations.
+ Instead, show the path to give as --temp to reuse them in
+ subsequent invocations.
--skip-install Do not run any install steps. Just run tests.
You should provide GOPATH, GEMHOME, and VENVDIR options
from a previous invocation if you use this option.
@@ -91,26 +93,18 @@ PERLINSTALLBASE=
COLUMNS=80
-leave_temp=
skip_install=
+temp=
+temp_preserve=
-declare -A leave_temp
clear_temp() {
- leaving=""
- for var in VENVDIR VENV3DIR GOPATH GITDIR GEMHOME PERLINSTALLBASE
- do
- if [[ -z "${leave_temp[$var]}" ]]
- then
- if [[ -n "${!var}" ]]
- then
- rm -rf "${!var}"
- fi
- else
- leaving+=" $var=\"${!var}\""
- fi
- done
- if [[ -n "$leaving" ]]; then
- echo "Leaving behind temp dirs: $leaving"
+ if [[ -z "$temp" ]]; then
+ # we didn't even get as far as making a temp dir
+ :
+ elif [[ -z "$temp_preserve" ]]; then
+ rm -rf "$temp"
+ else
+ echo "Leaving behind temp dirs in $temp"
fi
}
@@ -222,12 +216,12 @@ do
skip_install=1
only_install="$1"; shift
;;
+ --temp)
+ temp="$1"; shift
+ temp_preserve=1
+ ;;
--leave-temp)
- leave_temp[VENVDIR]=1
- leave_temp[VENV3DIR]=1
- leave_temp[GOPATH]=1
- leave_temp[GEMHOME]=1
- leave_temp[PERLINSTALLBASE]=1
+ temp_preserve=1
;;
--retry)
retry=1
@@ -299,14 +293,19 @@ fi
cd "$WORKSPACE"
find -name '*.pyc' -delete
+if [[ -z "$temp" ]]; then
+ temp="$(mktemp -d)"
+fi
+
# Set up temporary install dirs (unless existing dirs were supplied)
for tmpdir in VENVDIR VENV3DIR GOPATH GEMHOME PERLINSTALLBASE
do
if [[ -n "${!tmpdir}" ]]; then
- leave_temp[$tmpdir]=1
+ # Support old user-named temp dirs (but --temp {base} would
+ # be more convenient)
mkdir -p "${!tmpdir}"
else
- eval "$tmpdir"='$(mktemp -d)'
+ eval "$tmpdir"="$temp/$tmpdir"
fi
done
commit 716e83202432b70111466738123fadbf38824d87
Author: Joshua C. Randall <jcrandall at alum.mit.edu>
Date: Thu Sep 10 20:20:04 2015 +0000
Adds setuptools/pip upgrade in python3 virtualenv
Moves ruby environment setup to after python3 virtualenv
setup, because otherwise activating the python3 virtualenv
breaks ruby.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 43a8acb..db450e1 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -403,17 +403,6 @@ if (pip install setuptools | grep setuptools-0) || [ "$($VENVDIR/bin/easy_instal
pip install --upgrade setuptools pip
fi
-# Note: this must be the last time we change PATH, otherwise rvm will
-# whine a lot.
-setup_ruby_environment
-
-echo "PATH is $PATH"
-
-if ! which bundler >/dev/null
-then
- gem install --user-install bundler || fatal 'Could not install bundler'
-fi
-
# Needed for run_test_server.py which is used by certain (non-Python) tests.
pip freeze 2>/dev/null | egrep ^PyYAML= \
|| pip install PyYAML >/dev/null \
@@ -425,12 +414,24 @@ pip freeze 2>/dev/null | egrep ^apache-libcloud==$LIBCLOUD_PIN \
|| pip install --pre --ignore-installed https://github.com/curoverse/libcloud/archive/apache-libcloud-$LIBCLOUD_PIN.zip >/dev/null \
|| fatal "pip install apache-libcloud failed"
+# Deactivate Python 2 virtualenv
+deactivate
+
# If Python 3 is available, set up its virtualenv in $VENV3DIR.
# Otherwise, skip dependent tests.
PYTHON3=$(which python3)
if [ "0" = "$?" ]; then
virtualenv --python "$PYTHON3" --setuptools "$VENV3DIR" \
- || fatal "python3 virtualenv $VENV3DIR failed"
+ || fatal "python3 virtualenv $VENV3DIR failed"
+
+ . "$VENV3DIR/bin/activate"
+
+ if (pip install setuptools | grep setuptools-0) || [ "$($VENV3DIR/bin/easy_install --version | cut -d\ -f2 | cut -d. -f1)" -lt 18 ]; then
+ pip install --upgrade setuptools pip
+ fi
+
+ # Deactivate Python 3 virtualenv
+ deactivate
else
PYTHON3=
skip[services/dockercleaner]=1
@@ -442,6 +443,20 @@ services/dockercleaner install and tests will be skipped
EOF
fi
+# Reactivate Python 2 virtualenv
+. "$VENVDIR/bin/activate"
+
+# Note: this must be the last time we change PATH, otherwise rvm will
+# whine a lot.
+setup_ruby_environment
+
+echo "PATH is $PATH"
+
+if ! which bundler >/dev/null
+then
+ gem install --user-install bundler || fatal 'Could not install bundler'
+fi
+
checkexit() {
if [[ "$1" != "0" ]]; then
title "!!!!!! $2 FAILED !!!!!!"
commit 33692d98e44828a7114f1b8faa54c359b769e87b
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Sep 9 11:42:39 2015 -0400
Add libcloud-pin and update to dev3 refs #7031
diff --git a/jenkins/libcloud-pin b/jenkins/libcloud-pin
new file mode 100644
index 0000000..4ebd37f
--- /dev/null
+++ b/jenkins/libcloud-pin
@@ -0,0 +1 @@
+LIBCLOUD_PIN=0.18.1.dev3
\ No newline at end of file
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 31bf708..8fba12a 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -1,6 +1,7 @@
#!/bin/bash
. `dirname "$(readlink -f "$0")"`/run-library.sh
+. `dirname "$(readlink -f "$0")"`/libcloud-pin
read -rd "\000" helpmessage <<EOF
$(basename $0): Build Arvados packages
@@ -394,7 +395,7 @@ LIBCLOUD_DIR=$(mktemp -d)
(
cd $LIBCLOUD_DIR
git clone $DASHQ_UNLESS_DEBUG https://github.com/curoverse/libcloud.git .
- git checkout apache-libcloud-0.18.1.dev2
+ git checkout apache-libcloud-$LIBCLOUD_PIN
handle_python_package
)
fpm_build $LIBCLOUD_DIR "$PYTHON2_PKG_PREFIX"-apache-libcloud
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 8264591..43a8acb 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -1,5 +1,7 @@
#!/bin/bash
+. `dirname "$(readlink -f "$0")"`/libcloud-pin
+
read -rd "\000" helpmessage <<EOF
$(basename $0): Install and test Arvados components.
@@ -419,8 +421,8 @@ pip freeze 2>/dev/null | egrep ^PyYAML= \
# Preinstall forked version of libcloud, because nodemanager "pip install"
# won't pick it up by default.
-pip freeze 2>/dev/null | egrep ^apache-libcloud==0.18.1.dev2 \
- || pip install --pre --ignore-installed https://github.com/curoverse/libcloud/archive/apache-libcloud-0.18.1.dev2.zip >/dev/null \
+pip freeze 2>/dev/null | egrep ^apache-libcloud==$LIBCLOUD_PIN \
+ || pip install --pre --ignore-installed https://github.com/curoverse/libcloud/archive/apache-libcloud-$LIBCLOUD_PIN.zip >/dev/null \
|| fatal "pip install apache-libcloud failed"
# If Python 3 is available, set up its virtualenv in $VENV3DIR.
commit ee3c9aa1a39e6864a096a6619e237534d524ef69
Author: radhika <radhika at curoverse.com>
Date: Tue Sep 8 17:26:12 2015 -0400
6260: append to testargs, instead of replacing.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 21572f6..e2e4ab6 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -761,11 +761,9 @@ for g in "${gostuff[@]}"
do
if [[ "$g" == "services/datamanager" ]]
then
- testargs["$g"]="-data-manager-token-file $WORKSPACE/tmp/keep.data-manager-token-file"
- do_test "$g" go
- else
- do_test "$g" go
+ testargs["$g"]+=" -data-manager-token-file $WORKSPACE/tmp/keep.data-manager-token-file"
fi
+ do_test "$g" go
done
test_workbench() {
commit 323701f50f452cd96a15d14c278d41bd6e6b6f25
Merge: ff7b6fb 761d43d
Author: radhika <radhika at curoverse.com>
Date: Tue Sep 8 12:34:54 2015 -0400
Merge branch 'master' into 6260-test-datamanager
commit 761d43de01eb9c197162a14f8cacd3d58d67da77
Merge: a4e96f8 d5f9f02
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Tue Sep 8 09:44:21 2015 -0400
Merge branch '7031-build-forked-libcloud' refs #7031
commit d5f9f022f9414ffb8bc1b7b2f04096468574e2e5
Merge: 96952e4 a4e96f8
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Tue Sep 8 09:43:20 2015 -0400
Merge branch 'master' into 7031-build-forked-libcloud
commit a4e96f869530a93545582f5a21ec0e6caa20bcd9
Merge: c6f9a3f fe19ce4
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Sep 7 14:13:18 2015 -0400
Merge branch '6263-go-coverage' refs #6263
commit c6f9a3fc32babe5c0d26a1b23a89a5c7917af8db
Merge: 6dab05d 5ea1be9
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Sep 7 14:10:53 2015 -0400
Merge branch '6263-gitolite-test' refs #6263
commit 6dab05d0e5f1337aaa045b785d81536766edd27c
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Sep 4 11:04:17 2015 -0400
Make sure the library can always be found.
No issue #
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index da24867..f985882 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -1,6 +1,6 @@
#!/bin/bash
-. ./run-library.sh
+. `dirname "$(readlink -f "$0")"`/run-library.sh
read -rd "\000" helpmessage <<EOF
$(basename $0): Build Arvados SSO package
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 31734a9..4daf864 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -1,6 +1,6 @@
#!/bin/bash
-. ./run-library.sh
+. `dirname "$(readlink -f "$0")"`/run-library.sh
read -rd "\000" helpmessage <<EOF
$(basename $0): Build Arvados packages
commit 96952e4e39a617dd5233772ccf31c42195150d5e
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Sep 4 10:00:23 2015 -0400
Bump version to dev2
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 00e1541..15bdc5b 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -394,7 +394,7 @@ LIBCLOUD_DIR=$(mktemp -d)
(
cd $LIBCLOUD_DIR
git clone $DASHQ_UNLESS_DEBUG https://github.com/curoverse/libcloud.git .
- git checkout apache-libcloud-0.18.1.dev1
+ git checkout apache-libcloud-0.18.1.dev2
handle_python_package
)
fpm_build $LIBCLOUD_DIR "$PYTHON2_PKG_PREFIX"-apache-libcloud
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index e2ae4c5..87c83e1 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -414,8 +414,8 @@ pip freeze 2>/dev/null | egrep ^PyYAML= \
# Preinstall forked version of libcloud, because nodemanager "pip install"
# won't pick it up by default.
-pip freeze 2>/dev/null | egrep ^apache-libcloud==0.18.1.dev1 \
- || pip install --pre --ignore-installed https://github.com/curoverse/libcloud/archive/apache-libcloud-0.18.1.dev1.zip >/dev/null \
+pip freeze 2>/dev/null | egrep ^apache-libcloud==0.18.1.dev2 \
+ || pip install --pre --ignore-installed https://github.com/curoverse/libcloud/archive/apache-libcloud-0.18.1.dev2.zip >/dev/null \
|| fatal "pip install apache-libcloud failed"
# If Python 3 is available, set up its virtualenv in $VENV3DIR.
commit 9201f7f9da3043ad5574d3cfb8f841493a76460e
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Aug 31 14:10:53 2015 -0400
7031: Fix python prefix
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index fcebf6c..00e1541 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -397,7 +397,7 @@ LIBCLOUD_DIR=$(mktemp -d)
git checkout apache-libcloud-0.18.1.dev1
handle_python_package
)
-fpm_build $LIBCLOUD_DIR python-apache-libcloud
+fpm_build $LIBCLOUD_DIR "$PYTHON2_PKG_PREFIX"-apache-libcloud
rm -rf $LIBCLOUD_DIR
# A few dependencies
commit 9dda02a389dd2989e6190e07d2d5e8727dd6842a
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Aug 31 13:38:37 2015 -0400
7031: Check out our forked libcloud from git and make packages from that.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 31734a9..fcebf6c 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -95,7 +95,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
- virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
+ virtualenv pykka requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -108,7 +108,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
- virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
+ virtualenv pykka requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -121,7 +121,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
- virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
+ virtualenv pykka requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
@@ -133,7 +133,7 @@ case "$TARGET" in
PYTHON3_PKG_PREFIX=python3
PYTHON_BACKPORTS=(pyasn1==0.1.7 pyvcf pyasn1-modules==0.0.5 llfuse ciso8601 \
google-api-python-client six uritemplate oauth2client httplib2 \
- rsa apache-libcloud pycurl backports.ssl_match_hostname)
+ rsa pycurl backports.ssl_match_hostname)
PYTHON3_BACKPORTS=(docker-py requests websocket-client)
;;
centos6)
@@ -145,7 +145,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
- pykka apache-libcloud requests six pyexecjs jsonschema \
+ pykka requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl
python-daemon lockfile llfuse)
PYTHON3_BACKPORTS=(docker-py six requests)
@@ -389,6 +389,17 @@ cd $WORKSPACE/packages/$TARGET
rm -rf "$WORKSPACE/services/dockercleaner/build"
fpm_build $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curoverse, Inc.' 'python3' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/dockercleaner/arvados_docker_cleaner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Docker image cleaner"
+# Forked libcloud
+LIBCLOUD_DIR=$(mktemp -d)
+(
+ cd $LIBCLOUD_DIR
+ git clone $DASHQ_UNLESS_DEBUG https://github.com/curoverse/libcloud.git .
+ git checkout apache-libcloud-0.18.1.dev1
+ handle_python_package
+)
+fpm_build $LIBCLOUD_DIR python-apache-libcloud
+rm -rf $LIBCLOUD_DIR
+
# A few dependencies
for deppkg in "${PYTHON_BACKPORTS[@]}"; do
outname=$(echo "$deppkg" | sed -e 's/^python-//' -e 's/[<=>].*//' -e 's/_/-/g' -e "s/^/${PYTHON2_PKG_PREFIX}-/")
commit ff7b6fbcbca05e47fa226f8bef9f70de74e39024
Merge: fd45d42 bfa2dd5
Author: radhika <radhika at curoverse.com>
Date: Thu Sep 3 15:55:31 2015 -0400
Merge branch 'master' into 6260-test-datamanager
commit bfa2dd53794c53558aecb44577ab1e4752c268a0
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Sep 3 10:06:16 2015 -0400
Another tweak to the scopes required for run-deploy.sh to do its thing.
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index fb22633..d2c12ee 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -16,9 +16,10 @@ function usage {
echo >&2 ' arv api_client_authorization create_system_auth \'
echo >&2 ' --scopes "[\"GET /arvados/v1/virtual_machines\",'
echo >&2 ' \"GET /arvados/v1/keep_services\",'
+ echo >&2 ' \"GET /arvados/v1/keep_services/\",'
echo >&2 ' \"GET /arvados/v1/groups\",'
- echo >&2 ' \"GET /arvados/v1/links\",'
echo >&2 ' \"GET /arvados/v1/groups/\",'
+ echo >&2 ' \"GET /arvados/v1/links\",'
echo >&2 ' \"GET /arvados/v1/collections\",'
echo >&2 ' \"POST /arvados/v1/collections\",'
echo >&2 ' \"POST /arvados/v1/links\"]"'
commit b4006ddaf13af4210e2019adfbc1eba3ae963c60
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Sep 2 17:05:35 2015 -0400
Fix documented scope needed for run-deploy.sh
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index c097a20..fb22633 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -15,7 +15,7 @@ function usage {
echo >&2 "Note: this script requires an arvados token created with these permissions:"
echo >&2 ' arv api_client_authorization create_system_auth \'
echo >&2 ' --scopes "[\"GET /arvados/v1/virtual_machines\",'
- echo >&2 ' \"GET /arvados/v1/keep_services/\",'
+ echo >&2 ' \"GET /arvados/v1/keep_services\",'
echo >&2 ' \"GET /arvados/v1/groups\",'
echo >&2 ' \"GET /arvados/v1/links\",'
echo >&2 ' \"GET /arvados/v1/groups/\",'
commit 5ea1be908b4b46f029111deeff5216401a9662e2
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Sep 2 02:48:18 2015 -0400
6263: Skip re-installing virtualenv if it is already set up. Upgrade pip along with setuptools.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index d1c9c9e..9ca70b8 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -392,11 +392,13 @@ mkdir -p "$GOPATH/src/git.curoverse.com"
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
|| fatal "symlink failed"
-virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
+if ! [[ -e "$VENVDIR/bin/activate" ]] || ! [[ -e "$VENVDIR/bin/pip" ]]; then
+ virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
+fi
. "$VENVDIR/bin/activate"
if (pip install setuptools | grep setuptools-0) || [ "$($VENVDIR/bin/easy_install --version | cut -d\ -f2 | cut -d. -f1)" -lt 18 ]; then
- pip install --upgrade setuptools
+ pip install --upgrade setuptools pip
fi
# Note: this must be the last time we change PATH, otherwise rvm will
commit 73fbce50c5916c0daa9939cecc2088fb572ee17a
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Sep 2 00:49:33 2015 -0400
6263: Check for gitolite in sanity checks.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index e2ae4c5..d1c9c9e 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -177,6 +177,9 @@ sanity_checks() {
perl -e "use $mod; print \"\$$mod::VERSION\\n\"" \
|| fatal "No $mod. Try: apt-get install perl-modules libcrypt-ssleay-perl libjson-perl"
done
+ echo -n 'gitolite: '
+ which gitolite \
+ || fatal "No gitolite. Try: apt-get install gitolite3"
}
rotate_logfile() {
commit 4d3faccfe98c1f00987d7bc1ab6509ef347cc2ac
Merge: b549464 e6a5235
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Sep 1 16:57:06 2015 -0400
Merge branch '6939-package-sso-server'
closes #6939
commit e6a5235b8be6eb8e884d6d648b7bc89179829d9f
Merge: 379365c b549464
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Sep 1 16:56:17 2015 -0400
Merge branch 'master' into 6939-package-sso-server
Conflicts:
jenkins/run-build-packages-all-targets.sh
refs #6939
diff --cc jenkins/run-build-packages-all-targets.sh
index 3ddb4ba,2761f9f..ddf87c1
--- a/jenkins/run-build-packages-all-targets.sh
+++ b/jenkins/run-build-packages-all-targets.sh
@@@ -4,13 -4,10 +4,15 @@@ read -rd "\000" helpmessage <<EO
$(basename $0): Orchestrate run-build-packages.sh for every target
Syntax:
- WORKSPACE=/path/to/arvados $(basename $0)
+ WORKSPACE=/path/to/arvados $(basename $0) [options]
+
- WORKSPACE=path Path to the Arvados source tree to build packages from
++Options:
+
+--command
+ Build command to execute (default: use built-in Docker image command)
+ WORKSPACE=path Path to the Arvados source tree to build packages from
+
EOF
if ! [[ -n "$WORKSPACE" ]]; then
commit 379365ca0bf7948269ed705af53874f65668d851
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Sep 1 14:52:01 2015 -0400
Split out shared functions into a new run-library.sh.
refs #6939
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index 36fb586..da24867 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -1,5 +1,6 @@
#!/bin/bash
+. ./run-library.sh
read -rd "\000" helpmessage <<EOF
$(basename $0): Build Arvados SSO package
@@ -68,10 +69,6 @@ if [[ "$DEBUG" != 0 ]]; then
DASHQ_UNLESS_DEBUG=
fi
-debug_echo () {
- echo "$@" >"$STDOUT_IF_DEBUG"
-}
-
case "$TARGET" in
debian7)
FORMAT=deb
@@ -94,7 +91,6 @@ case "$TARGET" in
;;
esac
-
if ! [[ -n "$WORKSPACE" ]]; then
echo >&2 "$helpmessage"
echo >&2
@@ -133,51 +129,6 @@ fi
debug_echo "$0 is running from $RUN_BUILD_PACKAGES_PATH"
debug_echo "Workspace is $WORKSPACE"
-format_last_commit_here() {
- local format=$1; shift
- TZ=UTC git log -n1 --first-parent "--format=format:$format" .
-}
-
-version_from_git() {
- # Generates a version number from the git log for the current working
- # directory, and writes it to stdout.
- local git_ts git_hash
- declare $(format_last_commit_here "git_ts=%ct git_hash=%h")
- echo "0.1.$(date -ud "@$git_ts" +%Y%m%d%H%M%S).$git_hash"
-}
-
-nohash_version_from_git() {
- version_from_git | cut -d. -f1-3
-}
-
-timestamp_from_git() {
- format_last_commit_here "%ct"
-}
-
-# verify build results
-fpm_verify () {
- FPM_EXIT_CODE=$1
- shift
- FPM_RESULTS=$@
-
- FPM_PACKAGE_NAME=''
- if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\.-]*\.)(deb|rpm) ]]; then
- FPM_PACKAGE_NAME=${BASH_REMATCH[1]}${BASH_REMATCH[2]}
- fi
-
- if [[ "$FPM_PACKAGE_NAME" == "" ]]; then
- EXITCODE=1
- echo "Error: $PACKAGE: Unable to figure out package name from fpm results:"
- echo
- echo $FPM_RESULTS
- echo
- elif [[ "$FPM_RESULTS" =~ "File already exists" ]]; then
- echo "Package $FPM_PACKAGE_NAME exists, not rebuilding"
- elif [[ 0 -ne "$FPM_EXIT_CODE" ]]; then
- echo "Error building package for $1:\n $FPM_RESULTS"
- fi
-}
-
if [[ -f /etc/profile.d/rvm.sh ]]; then
source /etc/profile.d/rvm.sh
GEM="rvm-exec default gem"
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 0a04788..31734a9 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -1,5 +1,6 @@
#!/bin/bash
+. ./run-library.sh
read -rd "\000" helpmessage <<EOF
$(basename $0): Build Arvados packages
@@ -79,10 +80,6 @@ if [[ "$DEBUG" != 0 ]]; then
DASHQ_UNLESS_DEBUG=
fi
-debug_echo () {
- echo "$@" >"$STDOUT_IF_DEBUG"
-}
-
declare -a PYTHON_BACKPORTS PYTHON3_BACKPORTS
PYTHON2_VERSION=2.7
@@ -180,22 +177,6 @@ if [[ "$?" != 0 ]]; then
exit 1
fi
-find_easy_install() {
- for version_suffix in "$@"; do
- if "easy_install$version_suffix" --version >/dev/null 2>&1; then
- echo "easy_install$version_suffix"
- return 0
- fi
- done
- cat >&2 <<EOF
-$helpmessage
-
-Error: easy_install$1 (from Python setuptools module) not found
-
-EOF
- exit 1
-}
-
EASY_INSTALL2=$(find_easy_install -$PYTHON2_VERSION "")
EASY_INSTALL3=$(find_easy_install -$PYTHON3_VERSION 3)
@@ -210,168 +191,6 @@ fi
debug_echo "$0 is running from $RUN_BUILD_PACKAGES_PATH"
debug_echo "Workspace is $WORKSPACE"
-format_last_commit_here() {
- local format=$1; shift
- TZ=UTC git log -n1 --first-parent "--format=format:$format" .
-}
-
-version_from_git() {
- # Generates a version number from the git log for the current working
- # directory, and writes it to stdout.
- local git_ts git_hash
- declare $(format_last_commit_here "git_ts=%ct git_hash=%h")
- echo "0.1.$(date -ud "@$git_ts" +%Y%m%d%H%M%S).$git_hash"
-}
-
-nohash_version_from_git() {
- version_from_git | cut -d. -f1-3
-}
-
-timestamp_from_git() {
- format_last_commit_here "%ct"
-}
-
-handle_python_package () {
- # This function assumes the current working directory is the python package directory
- if [ -n "$(find dist -name "*-$(nohash_version_from_git).tar.gz" -print -quit)" ]; then
- # This package doesn't need rebuilding.
- return
- fi
- # Make sure only to use sdist - that's the only format pip can deal with (sigh)
- python setup.py $DASHQ_UNLESS_DEBUG sdist
-}
-
-handle_ruby_gem() {
- local gem_name=$1; shift
- local gem_version=$(nohash_version_from_git)
- local gem_src_dir="$(pwd)"
-
- if ! [[ -e "${gem_name}-${gem_version}.gem" ]]; then
- find -maxdepth 1 -name "${gem_name}-*.gem" -delete
-
- # -q appears to be broken in gem version 2.2.2
- $GEM build "$gem_name.gemspec" $DASHQ_UNLESS_DEBUG >"$STDOUT_IF_DEBUG" 2>"$STDERR_IF_DEBUG"
- fi
-}
-
-# Build packages for everything
-fpm_build () {
- # The package source. Depending on the source type, this can be a
- # path, or the name of the package in an upstream repository (e.g.,
- # pip).
- PACKAGE=$1
- shift
- # The name of the package to build. Defaults to $PACKAGE.
- PACKAGE_NAME=${1:-$PACKAGE}
- shift
- # Optional: the vendor of the package. Should be "Curoverse, Inc." for
- # packages of our own software. Passed to fpm --vendor.
- VENDOR=$1
- shift
- # The type of source package. Passed to fpm -s. Default "python".
- PACKAGE_TYPE=${1:-python}
- shift
- # Optional: the package version number. Passed to fpm -v.
- VERSION=$1
- shift
-
- case "$PACKAGE_TYPE" in
- python)
- # All Arvados Python2 packages depend on Python 2.7.
- # Make sure we build with that for consistency.
- set -- "$@" --python-bin python2.7 \
- --python-easyinstall "$EASY_INSTALL2" \
- --python-package-name-prefix "$PYTHON2_PKG_PREFIX" \
- --depends "$PYTHON2_PACKAGE"
- ;;
- python3)
- # fpm does not actually support a python3 package type. Instead
- # we recognize it as a convenience shortcut to add several
- # necessary arguments to fpm's command line later, after we're
- # done handling positional arguments.
- PACKAGE_TYPE=python
- set -- "$@" --python-bin python3 \
- --python-easyinstall "$EASY_INSTALL3" \
- --python-package-name-prefix "$PYTHON3_PKG_PREFIX" \
- --depends "$PYTHON3_PACKAGE"
- ;;
- esac
-
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "$FORMAT")
- if [ python = "$PACKAGE_TYPE" ]; then
- COMMAND_ARR+=(--exclude=\*/{dist,site}-packages/tests/\*)
- fi
-
- if [[ "$PACKAGE_NAME" != "$PACKAGE" ]]; then
- COMMAND_ARR+=('-n' "$PACKAGE_NAME")
- fi
-
- if [[ "$VENDOR" != "" ]]; then
- COMMAND_ARR+=('--vendor' "$VENDOR")
- fi
-
- if [[ "$VERSION" != "" ]]; then
- COMMAND_ARR+=('-v' "$VERSION")
- fi
-
- # Append remaining function arguments directly to fpm's command line.
- for i; do
- COMMAND_ARR+=("$i")
- done
-
- # Append --depends X and other arguments specified by fpm-info.sh in
- # the package source dir. These are added last so they can override
- # the arguments added by this script.
- declare -a fpm_args=()
- declare -a fpm_depends=()
- if [[ -d "$PACKAGE" ]]; then
- FPM_INFO="$PACKAGE/fpm-info.sh"
- else
- FPM_INFO="${WORKSPACE}/backports/${PACKAGE_TYPE}-${PACKAGE}/fpm-info.sh"
- fi
- if [[ -e "$FPM_INFO" ]]; then
- debug_echo "Loading fpm overrides from $FPM_INFO"
- source "$FPM_INFO"
- fi
- for i in "${fpm_depends[@]}"; do
- COMMAND_ARR+=('--depends' "$i")
- done
- COMMAND_ARR+=("${fpm_args[@]}")
-
- COMMAND_ARR+=("$PACKAGE")
-
- debug_echo -e "\n${COMMAND_ARR[@]}\n"
-
- FPM_RESULTS=$("${COMMAND_ARR[@]}")
- FPM_EXIT_CODE=$?
-
- fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
-}
-
-# verify build results
-fpm_verify () {
- FPM_EXIT_CODE=$1
- shift
- FPM_RESULTS=$@
-
- FPM_PACKAGE_NAME=''
- if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\.-]*\.)(deb|rpm) ]]; then
- FPM_PACKAGE_NAME=${BASH_REMATCH[1]}${BASH_REMATCH[2]}
- fi
-
- if [[ "$FPM_PACKAGE_NAME" == "" ]]; then
- EXITCODE=1
- echo "Error: $PACKAGE: Unable to figure out package name from fpm results:"
- echo
- echo $FPM_RESULTS
- echo
- elif [[ "$FPM_RESULTS" =~ "File already exists" ]]; then
- echo "Package $FPM_PACKAGE_NAME exists, not rebuilding"
- elif [[ 0 -ne "$FPM_EXIT_CODE" ]]; then
- echo "Error building package for $1:\n $FPM_RESULTS"
- fi
-}
-
if [[ -f /etc/profile.d/rvm.sh ]]; then
source /etc/profile.d/rvm.sh
GEM="rvm-exec default gem"
diff --git a/jenkins/run-library.sh b/jenkins/run-library.sh
new file mode 100755
index 0000000..f07397f
--- /dev/null
+++ b/jenkins/run-library.sh
@@ -0,0 +1,185 @@
+#!/bin/bash
+
+# A library of functions shared by the various scripts in this directory.
+
+debug_echo () {
+ echo "$@" >"$STDOUT_IF_DEBUG"
+}
+
+find_easy_install() {
+ for version_suffix in "$@"; do
+ if "easy_install$version_suffix" --version >/dev/null 2>&1; then
+ echo "easy_install$version_suffix"
+ return 0
+ fi
+ done
+ cat >&2 <<EOF
+$helpmessage
+
+Error: easy_install$1 (from Python setuptools module) not found
+
+EOF
+ exit 1
+}
+
+format_last_commit_here() {
+ local format=$1; shift
+ TZ=UTC git log -n1 --first-parent "--format=format:$format" .
+}
+
+version_from_git() {
+ # Generates a version number from the git log for the current working
+ # directory, and writes it to stdout.
+ local git_ts git_hash
+ declare $(format_last_commit_here "git_ts=%ct git_hash=%h")
+ echo "0.1.$(date -ud "@$git_ts" +%Y%m%d%H%M%S).$git_hash"
+}
+
+nohash_version_from_git() {
+ version_from_git | cut -d. -f1-3
+}
+
+timestamp_from_git() {
+ format_last_commit_here "%ct"
+}
+
+handle_python_package () {
+ # This function assumes the current working directory is the python package directory
+ if [ -n "$(find dist -name "*-$(nohash_version_from_git).tar.gz" -print -quit)" ]; then
+ # This package doesn't need rebuilding.
+ return
+ fi
+ # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ python setup.py $DASHQ_UNLESS_DEBUG sdist
+}
+
+handle_ruby_gem() {
+ local gem_name=$1; shift
+ local gem_version=$(nohash_version_from_git)
+ local gem_src_dir="$(pwd)"
+
+ if ! [[ -e "${gem_name}-${gem_version}.gem" ]]; then
+ find -maxdepth 1 -name "${gem_name}-*.gem" -delete
+
+ # -q appears to be broken in gem version 2.2.2
+ $GEM build "$gem_name.gemspec" $DASHQ_UNLESS_DEBUG >"$STDOUT_IF_DEBUG" 2>"$STDERR_IF_DEBUG"
+ fi
+}
+
+# Build packages for everything
+fpm_build () {
+ # The package source. Depending on the source type, this can be a
+ # path, or the name of the package in an upstream repository (e.g.,
+ # pip).
+ PACKAGE=$1
+ shift
+ # The name of the package to build. Defaults to $PACKAGE.
+ PACKAGE_NAME=${1:-$PACKAGE}
+ shift
+ # Optional: the vendor of the package. Should be "Curoverse, Inc." for
+ # packages of our own software. Passed to fpm --vendor.
+ VENDOR=$1
+ shift
+ # The type of source package. Passed to fpm -s. Default "python".
+ PACKAGE_TYPE=${1:-python}
+ shift
+ # Optional: the package version number. Passed to fpm -v.
+ VERSION=$1
+ shift
+
+ case "$PACKAGE_TYPE" in
+ python)
+ # All Arvados Python2 packages depend on Python 2.7.
+ # Make sure we build with that for consistency.
+ set -- "$@" --python-bin python2.7 \
+ --python-easyinstall "$EASY_INSTALL2" \
+ --python-package-name-prefix "$PYTHON2_PKG_PREFIX" \
+ --depends "$PYTHON2_PACKAGE"
+ ;;
+ python3)
+ # fpm does not actually support a python3 package type. Instead
+ # we recognize it as a convenience shortcut to add several
+ # necessary arguments to fpm's command line later, after we're
+ # done handling positional arguments.
+ PACKAGE_TYPE=python
+ set -- "$@" --python-bin python3 \
+ --python-easyinstall "$EASY_INSTALL3" \
+ --python-package-name-prefix "$PYTHON3_PKG_PREFIX" \
+ --depends "$PYTHON3_PACKAGE"
+ ;;
+ esac
+
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "$FORMAT")
+ if [ python = "$PACKAGE_TYPE" ]; then
+ COMMAND_ARR+=(--exclude=\*/{dist,site}-packages/tests/\*)
+ fi
+
+ if [[ "$PACKAGE_NAME" != "$PACKAGE" ]]; then
+ COMMAND_ARR+=('-n' "$PACKAGE_NAME")
+ fi
+
+ if [[ "$VENDOR" != "" ]]; then
+ COMMAND_ARR+=('--vendor' "$VENDOR")
+ fi
+
+ if [[ "$VERSION" != "" ]]; then
+ COMMAND_ARR+=('-v' "$VERSION")
+ fi
+
+ # Append remaining function arguments directly to fpm's command line.
+ for i; do
+ COMMAND_ARR+=("$i")
+ done
+
+ # Append --depends X and other arguments specified by fpm-info.sh in
+ # the package source dir. These are added last so they can override
+ # the arguments added by this script.
+ declare -a fpm_args=()
+ declare -a fpm_depends=()
+ if [[ -d "$PACKAGE" ]]; then
+ FPM_INFO="$PACKAGE/fpm-info.sh"
+ else
+ FPM_INFO="${WORKSPACE}/backports/${PACKAGE_TYPE}-${PACKAGE}/fpm-info.sh"
+ fi
+ if [[ -e "$FPM_INFO" ]]; then
+ debug_echo "Loading fpm overrides from $FPM_INFO"
+ source "$FPM_INFO"
+ fi
+ for i in "${fpm_depends[@]}"; do
+ COMMAND_ARR+=('--depends' "$i")
+ done
+ COMMAND_ARR+=("${fpm_args[@]}")
+
+ COMMAND_ARR+=("$PACKAGE")
+
+ debug_echo -e "\n${COMMAND_ARR[@]}\n"
+
+ FPM_RESULTS=$("${COMMAND_ARR[@]}")
+ FPM_EXIT_CODE=$?
+
+ fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
+}
+
+# verify build results
+fpm_verify () {
+ FPM_EXIT_CODE=$1
+ shift
+ FPM_RESULTS=$@
+
+ FPM_PACKAGE_NAME=''
+ if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\.-]*\.)(deb|rpm) ]]; then
+ FPM_PACKAGE_NAME=${BASH_REMATCH[1]}${BASH_REMATCH[2]}
+ fi
+
+ if [[ "$FPM_PACKAGE_NAME" == "" ]]; then
+ EXITCODE=1
+ echo "Error: $PACKAGE: Unable to figure out package name from fpm results:"
+ echo
+ echo $FPM_RESULTS
+ echo
+ elif [[ "$FPM_RESULTS" =~ "File already exists" ]]; then
+ echo "Package $FPM_PACKAGE_NAME exists, not rebuilding"
+ elif [[ 0 -ne "$FPM_EXIT_CODE" ]]; then
+ echo "Error building package for $1:\n $FPM_RESULTS"
+ fi
+}
commit ad88edf91c59cbf2cfdf59f21f278871889636b3
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Sep 1 14:33:18 2015 -0400
First batch of improvements based on review feedback.
refs #6939
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
index ea0718e..5451a4e 100755
--- a/jenkins/run-build-packages-one-target.sh
+++ b/jenkins/run-build-packages-one-target.sh
@@ -97,8 +97,4 @@ cd ..
run_docker $TARGET
-#for dockerfile_path in $(find -name Dockerfile); do
-# run_docker "$(basename $(dirname "$dockerfile_path"))"
-#done
-
exit $FINAL_EXITCODE
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
index 141db2b..36fb586 100755
--- a/jenkins/run-build-packages-sso.sh
+++ b/jenkins/run-build-packages-sso.sh
@@ -228,6 +228,23 @@ cd $WORKSPACE/packages/$TARGET
# package. Then remove that database.yml file again. It has to be a valid file though.
#RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile
+# There are just 2 excludes left here, all the others are pulled in via fpm-info.sh.
+# The .git directory is excluded by git implicitly, so we can't pick it up from .gitignore.
+# The packages directory needs to be explictly excluded here because it will only be listed
+# if it exists at the time fpm-info.sh runs. If it does not exist at that time, this script
+# will create it and when fpm runs, it will include the directory. So we add it to the exclude
+# list explicitly here, just in case.
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados SSO server - Arvados is a free and open source platform for big data science.'" "--license='Expat License'" "-s" "dir" "-t" "$FORMAT" "-v" "$SSO_VERSION" "-x" "var/www/arvados-sso/current/.git" "-x" "var/www/arvados-sso/current/packages" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/postinst.sh")
+
+if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
+ # This is the complete package with vendor/bundle included.
+ # It's big, so we do not build it by default.
+ COMMAND_ARR+=("-n" "${PACKAGE_NAME}-with-bundle")
+else
+ # The default package excludes vendor/bundle
+ COMMAND_ARR+=("-n" "${PACKAGE_NAME}" "-x" "var/www/arvados-sso/current/vendor/bundle")
+fi
+
# Append --depends X and other arguments specified by fpm-info.sh in
# the package source dir. These are added last so they can override
# the arguments added by this script.
@@ -239,28 +256,6 @@ if [[ -e "$FPM_INFO" ]]; then
source "$FPM_INFO"
fi
-# This is the complete package with vendor/bundle included.
-# It's big, so we do not build it by default.
-if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados SSO server - Arvados is a free and open source platform for big data science.'" "--license='Expat License'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$SSO_VERSION" "-x" "var/www/arvados-sso/current/.git" "-x" "var/www/arvados-sso/current/tmp" "-x" "var/www/arvados-sso/current/log" "-x" "var/www/arvados-sso/current/vendor/cache/*" "-x" "var/www/arvados-sso/current/packages" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/postinst.sh")
-
- for i in "${fpm_depends[@]}"; do
- COMMAND_ARR+=('--depends' "$i")
- done
- COMMAND_ARR+=("${fpm_args[@]}")
- COMMAND_ARR+=("$WORKSPACE/=/var/www/arvados-sso/current" "$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server-upgrade.sh=/usr/local/bin/arvados-sso-server-upgrade.sh")
-
- debug_echo -e "\n${COMMAND_ARR[@]}\n"
-
- FPM_RESULTS=$("${COMMAND_ARR[@]}")
- FPM_EXIT_CODE=$?
- fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
-fi
-
-# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados SSO server - Arvados is a free and open source platform for big data science.'" "--license='Expat License'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$SSO_VERSION" "-x" "var/www/arvados-sso/current/.git" "-x" "var/www/arvados-sso/current/tmp" "-x" "var/www/arvados-sso/current/log" "-x" "var/www/arvados-sso/current/vendor/bundle" "-x" "var/www/arvados-sso/current/vendor/cache/*" "-x" "var/www/arvados-sso/current/packages" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/postinst.sh")
-
-
for i in "${fpm_depends[@]}"; do
COMMAND_ARR+=('--depends' "$i")
done
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 9f7c37f..0a04788 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -27,6 +27,7 @@ EXITCODE=0
DEBUG=${ARVADOS_DEBUG:-0}
BUILD_BUNDLE_PACKAGES=0
TARGET=debian7
+COMMAND=
PARSEDOPTS=$(getopt --name "$0" --longoptions \
help,build-bundle-packages,debug,target: \
commit d9b63321261cbc98a50166981213342b7c0b91f1
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Aug 31 17:04:23 2015 -0400
Add build script for the new Arvados SSO server package.
Also make the run-build-packages-all-targets.sh script a bit smarter so
that we can use it to build the SSO server package.
Add a run-build-packages-one-target.sh script to build any of our
Arvados packages for just one target.
refs #6939
diff --git a/jenkins/arvados-sso-server-extras/arvados-sso-server-upgrade.sh b/jenkins/arvados-sso-server-extras/arvados-sso-server-upgrade.sh
new file mode 100755
index 0000000..1a377dc
--- /dev/null
+++ b/jenkins/arvados-sso-server-extras/arvados-sso-server-upgrade.sh
@@ -0,0 +1,91 @@
+#!/bin/bash
+
+set -e
+
+if [ -e /etc/redhat-release ]; then
+ WWW_OWNER=nginx:nginx
+else
+ # Assume we're on a Debian-based system for now.
+ WWW_OWNER=www-data:www-data
+fi
+
+NGINX_SERVICE=${NGINX_SERVICE:-$(service --status-all 2>/dev/null \
+ | grep -Eo '\bnginx[^[:space:]]*' || true)}
+if [ -z "$NGINX_SERVICE" ]; then
+ cat >&2 <<EOF
+Error: nginx service not found. Aborting.
+Set NGINX_SERVICE to the name of the service hosting the Rails server.
+EOF
+ exit 1
+elif [ "$NGINX_SERVICE" != "$(echo "$NGINX_SERVICE" | head -n 1)" ]; then
+ cat >&2 <<EOF
+Error: multiple nginx services found. Aborting.
+Set NGINX_SERVICE to the name of the service hosting the Rails server.
+EOF
+ exit 1
+fi
+
+RELEASE_PATH=/var/www/arvados-sso/current
+SHARED_PATH=/var/www/arvados-sso/shared
+CONFIG_PATH=/etc/arvados/sso/
+
+echo
+echo "Assumption: $NGINX_SERVICE is configured to serve your SSO server URL from"
+echo " /var/www/arvados-sso/current"
+echo "Assumption: configuration files are in /etc/arvados/sso/"
+echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
+echo
+
+echo "Copying files from $CONFIG_PATH ..."
+cp -f $CONFIG_PATH/database.yml $RELEASE_PATH/config/database.yml
+cp -f $RELEASE_PATH/config/environments/production.rb.example $RELEASE_PATH/config/environments/production.rb
+cp -f $CONFIG_PATH/application.yml $RELEASE_PATH/config/application.yml
+echo "... done."
+
+# Before we do anything else, make sure some directories and files are in place
+if [[ ! -e $SHARED_PATH/log ]]; then mkdir -p $SHARED_PATH/log; fi
+if [[ ! -e $RELEASE_PATH/tmp ]]; then mkdir -p $RELEASE_PATH/tmp; fi
+if [[ ! -e $RELEASE_PATH/log ]]; then ln -s $SHARED_PATH/log $RELEASE_PATH/log; fi
+if [[ ! -e $SHARED_PATH/log/production.log ]]; then touch $SHARED_PATH/log/production.log; fi
+
+cd "$RELEASE_PATH"
+export RAILS_ENV=production
+
+echo "Running bundle install ..."
+bundle install --path $SHARED_PATH/vendor_bundle --quiet
+echo "... done."
+
+echo "Ensuring directory and file permissions ..."
+# Ensure correct ownership of a few files
+chown "$WWW_OWNER" $RELEASE_PATH/config/environment.rb
+chown "$WWW_OWNER" $RELEASE_PATH/config.ru
+chown "$WWW_OWNER" $RELEASE_PATH/config/database.yml
+chown "$WWW_OWNER" $RELEASE_PATH/Gemfile.lock
+chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
+chown -R "$WWW_OWNER" $SHARED_PATH/log
+chown "$WWW_OWNER" $RELEASE_PATH/db/schema.rb
+chmod 644 $SHARED_PATH/log/*
+echo "... done."
+
+# If we use `grep -q`, rake will write a backtrace on EPIPE.
+if bundle exec rake db:migrate:status | grep '^database: ' >/dev/null; then
+ echo "Starting db:migrate ..."
+ bundle exec rake db:migrate
+elif [ 0 -eq ${PIPESTATUS[0]} ]; then
+ # The database exists, but the migrations table doesn't.
+ echo "Setting up database ..."
+ bundle exec rake db:schema:load db:seed
+else
+ echo "Error: Database is not ready to set up. Aborting." >&2
+ exit 1
+fi
+echo "... done."
+
+echo "Precompiling assets ..."
+# precompile assets; thankfully this does not take long
+bundle exec rake assets:precompile -q -s
+echo "... done."
+
+echo "Restarting nginx ..."
+service "$NGINX_SERVICE" restart
+echo "... done."
diff --git a/jenkins/arvados-sso-server-extras/postinst.sh b/jenkins/arvados-sso-server-extras/postinst.sh
new file mode 100755
index 0000000..ffd9b14
--- /dev/null
+++ b/jenkins/arvados-sso-server-extras/postinst.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+cd /var/www/arvados-sso
+
+chown -R www-data:www-data tmp >/dev/null 2>&1
+chown -R www-data:www-data log >/dev/null 2>&1
+chown www-data:www-data db/schema.rb >/dev/null 2>&1
+chmod 644 log/* >/dev/null 2>&1
+
+# Errors above are not serious
+exit 0
+
diff --git a/jenkins/arvados-sso-server-extras/prerm.sh b/jenkins/arvados-sso-server-extras/prerm.sh
new file mode 100755
index 0000000..b1280b8
--- /dev/null
+++ b/jenkins/arvados-sso-server-extras/prerm.sh
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+RELEASE_PATH=/var/www/arvados-sso/current
+rm -f $RELEASE_PATH/config/database.yml
+rm -f $RELEASE_PATH/config/environments/production.rb
+rm -f $RELEASE_PATH/config/application.yml
+rm -rf $RELEASE_PATH/public/assets/
+rm -rf $RELEASE_PATH/tmp
+rm $RELEASE_PATH/log
+
diff --git a/jenkins/run-build-packages-all-targets.sh b/jenkins/run-build-packages-all-targets.sh
index 2761f9f..3ddb4ba 100755
--- a/jenkins/run-build-packages-all-targets.sh
+++ b/jenkins/run-build-packages-all-targets.sh
@@ -4,10 +4,13 @@ read -rd "\000" helpmessage <<EOF
$(basename $0): Orchestrate run-build-packages.sh for every target
Syntax:
- WORKSPACE=/path/to/arvados $(basename $0)
+ WORKSPACE=/path/to/arvados $(basename $0) [options]
WORKSPACE=path Path to the Arvados source tree to build packages from
+--command
+ Build command to execute (default: use built-in Docker image command)
+
EOF
if ! [[ -n "$WORKSPACE" ]]; then
@@ -18,15 +21,60 @@ if ! [[ -n "$WORKSPACE" ]]; then
exit 1
fi
+if ! [[ -d "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: $WORKSPACE is not a directory"
+ echo >&2
+ exit 1
+fi
+
set -e
+PARSEDOPTS=$(getopt --name "$0" --longoptions \
+ help,command: \
+ -- "" "$@")
+if [ $? -ne 0 ]; then
+ exit 1
+fi
+
+COMMAND=
+
+eval set -- "$PARSEDOPTS"
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --help)
+ echo >&2 "$helpmessage"
+ echo >&2
+ exit 1
+ ;;
+ --command)
+ COMMAND="$2"; shift
+ ;;
+ --)
+ if [ $# -gt 1 ]; then
+ echo >&2 "$0: unrecognized argument '$2'. Try: $0 --help"
+ exit 1
+ fi
+ ;;
+ esac
+ shift
+done
+
+if [[ "$COMMAND" != "" ]]; then
+ COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND"
+fi
+
FINAL_EXITCODE=0
JENKINS_DIR=$(dirname "$(readlink -e "$0")")
run_docker() {
local tag=$1; shift
+ if [[ "$COMMAND" != "" ]]; then
+ COMMAND="$COMMAND --target $tag"
+ fi
if docker run -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
- --env ARVADOS_DEBUG=1 "arvados/build:$tag"; then
+ --env ARVADOS_DEBUG=1 "arvados/build:$tag" $COMMAND; then
# Success - nothing more to do.
true
else
diff --git a/jenkins/run-build-packages-one-target.sh b/jenkins/run-build-packages-one-target.sh
new file mode 100755
index 0000000..ea0718e
--- /dev/null
+++ b/jenkins/run-build-packages-one-target.sh
@@ -0,0 +1,104 @@
+#!/bin/bash
+
+read -rd "\000" helpmessage <<EOF
+$(basename $0): Orchestrate run-build-packages.sh for one target
+
+Syntax:
+ WORKSPACE=/path/to/arvados $(basename $0) [options]
+
+--target <target>
+ Distribution to build packages for (default: debian7)
+--command
+ Build command to execute (default: use built-in Docker image command)
+
+WORKSPACE=path Path to the Arvados source tree to build packages from
+
+EOF
+
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
+fi
+
+if ! [[ -d "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: $WORKSPACE is not a directory"
+ echo >&2
+ exit 1
+fi
+
+PARSEDOPTS=$(getopt --name "$0" --longoptions \
+ help,target:,command: \
+ -- "" "$@")
+if [ $? -ne 0 ]; then
+ exit 1
+fi
+
+TARGET=debian7
+COMMAND=
+
+eval set -- "$PARSEDOPTS"
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --help)
+ echo >&2 "$helpmessage"
+ echo >&2
+ exit 1
+ ;;
+ --target)
+ TARGET="$2"; shift
+ ;;
+ --command)
+ COMMAND="$2"; shift
+ ;;
+ --)
+ if [ $# -gt 1 ]; then
+ echo >&2 "$0: unrecognized argument '$2'. Try: $0 --help"
+ exit 1
+ fi
+ ;;
+ esac
+ shift
+done
+
+set -e
+
+if [[ "$COMMAND" != "" ]]; then
+ COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND --target $TARGET"
+fi
+
+FINAL_EXITCODE=0
+JENKINS_DIR=$(dirname "$(readlink -e "$0")")
+
+run_docker() {
+ local tag=$1; shift
+ if docker run -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
+ --env ARVADOS_DEBUG=1 "arvados/build:$tag" $COMMAND; then
+ # Success - nothing more to do.
+ true
+ else
+ FINAL_EXITCODE=$?
+ echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
+ fi
+}
+
+# In case it's needed, build the container. This costs just a few
+# seconds when the container already exist, so it's not a big deal to
+# do it on each run.
+cd "$JENKINS_DIR/dockerfiles"
+echo $TARGET
+cd $TARGET
+time docker build -t arvados/build:$TARGET .
+cd ..
+
+run_docker $TARGET
+
+#for dockerfile_path in $(find -name Dockerfile); do
+# run_docker "$(basename $(dirname "$dockerfile_path"))"
+#done
+
+exit $FINAL_EXITCODE
diff --git a/jenkins/run-build-packages-sso.sh b/jenkins/run-build-packages-sso.sh
new file mode 100755
index 0000000..141db2b
--- /dev/null
+++ b/jenkins/run-build-packages-sso.sh
@@ -0,0 +1,277 @@
+#!/bin/bash
+
+
+read -rd "\000" helpmessage <<EOF
+$(basename $0): Build Arvados SSO package
+
+Syntax:
+ WORKSPACE=/path/to/arvados-sso $(basename $0) [options]
+
+Options:
+
+--build-bundle-packages (default: false)
+ Build package with vendor/bundle included
+--debug
+ Output debug information (default: false)
+--target
+ Distribution to build packages for (default: debian7)
+
+WORKSPACE=path Path to the Arvados SSO source tree to build packages from
+
+EOF
+
+EXITCODE=0
+DEBUG=${ARVADOS_DEBUG:-0}
+BUILD_BUNDLE_PACKAGES=0
+TARGET=debian7
+
+PARSEDOPTS=$(getopt --name "$0" --longoptions \
+ help,build-bundle-packages,debug,target: \
+ -- "" "$@")
+if [ $? -ne 0 ]; then
+ exit 1
+fi
+
+eval set -- "$PARSEDOPTS"
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --help)
+ echo >&2 "$helpmessage"
+ echo >&2
+ exit 1
+ ;;
+ --target)
+ TARGET="$2"; shift
+ ;;
+ --debug)
+ DEBUG=1
+ ;;
+ --build-bundle-packages)
+ BUILD_BUNDLE_PACKAGES=1
+ ;;
+ --)
+ if [ $# -gt 1 ]; then
+ echo >&2 "$0: unrecognized argument '$2'. Try: $0 --help"
+ exit 1
+ fi
+ ;;
+ esac
+ shift
+done
+
+STDOUT_IF_DEBUG=/dev/null
+STDERR_IF_DEBUG=/dev/null
+DASHQ_UNLESS_DEBUG=-q
+if [[ "$DEBUG" != 0 ]]; then
+ STDOUT_IF_DEBUG=/dev/stdout
+ STDERR_IF_DEBUG=/dev/stderr
+ DASHQ_UNLESS_DEBUG=
+fi
+
+debug_echo () {
+ echo "$@" >"$STDOUT_IF_DEBUG"
+}
+
+case "$TARGET" in
+ debian7)
+ FORMAT=deb
+ ;;
+ debian8)
+ FORMAT=deb
+ ;;
+ ubuntu1204)
+ FORMAT=deb
+ ;;
+ ubuntu1404)
+ FORMAT=deb
+ ;;
+ centos6)
+ FORMAT=rpm
+ ;;
+ *)
+ echo -e "$0: Unknown target '$TARGET'.\n" >&2
+ exit 1
+ ;;
+esac
+
+
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
+fi
+
+if ! [[ -d "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: $WORKSPACE is not a directory"
+ echo >&2
+ exit 1
+fi
+
+# Test for fpm
+fpm --version >/dev/null 2>&1
+
+if [[ "$?" != 0 ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: fpm not found"
+ echo >&2
+ exit 1
+fi
+
+RUN_BUILD_PACKAGES_PATH="`dirname \"$0\"`"
+RUN_BUILD_PACKAGES_PATH="`( cd \"$RUN_BUILD_PACKAGES_PATH\" && pwd )`" # absolutized and normalized
+if [ -z "$RUN_BUILD_PACKAGES_PATH" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+debug_echo "$0 is running from $RUN_BUILD_PACKAGES_PATH"
+debug_echo "Workspace is $WORKSPACE"
+
+format_last_commit_here() {
+ local format=$1; shift
+ TZ=UTC git log -n1 --first-parent "--format=format:$format" .
+}
+
+version_from_git() {
+ # Generates a version number from the git log for the current working
+ # directory, and writes it to stdout.
+ local git_ts git_hash
+ declare $(format_last_commit_here "git_ts=%ct git_hash=%h")
+ echo "0.1.$(date -ud "@$git_ts" +%Y%m%d%H%M%S).$git_hash"
+}
+
+nohash_version_from_git() {
+ version_from_git | cut -d. -f1-3
+}
+
+timestamp_from_git() {
+ format_last_commit_here "%ct"
+}
+
+# verify build results
+fpm_verify () {
+ FPM_EXIT_CODE=$1
+ shift
+ FPM_RESULTS=$@
+
+ FPM_PACKAGE_NAME=''
+ if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\.-]*\.)(deb|rpm) ]]; then
+ FPM_PACKAGE_NAME=${BASH_REMATCH[1]}${BASH_REMATCH[2]}
+ fi
+
+ if [[ "$FPM_PACKAGE_NAME" == "" ]]; then
+ EXITCODE=1
+ echo "Error: $PACKAGE: Unable to figure out package name from fpm results:"
+ echo
+ echo $FPM_RESULTS
+ echo
+ elif [[ "$FPM_RESULTS" =~ "File already exists" ]]; then
+ echo "Package $FPM_PACKAGE_NAME exists, not rebuilding"
+ elif [[ 0 -ne "$FPM_EXIT_CODE" ]]; then
+ echo "Error building package for $1:\n $FPM_RESULTS"
+ fi
+}
+
+if [[ -f /etc/profile.d/rvm.sh ]]; then
+ source /etc/profile.d/rvm.sh
+ GEM="rvm-exec default gem"
+else
+ GEM=gem
+fi
+
+# Make all files world-readable -- jenkins runs with umask 027, and has checked
+# out our git tree here
+chmod o+r "$WORKSPACE" -R
+
+# More cleanup - make sure all executables that we'll package are 755
+# No executables in the sso server package
+#find -type d -name 'bin' |xargs -I {} find {} -type f |xargs -I {} chmod 755 {}
+
+# Now fix our umask to something better suited to building and publishing
+# gems and packages
+umask 0022
+
+debug_echo "umask is" `umask`
+
+if [[ ! -d "$WORKSPACE/packages/$TARGET" ]]; then
+ mkdir -p $WORKSPACE/packages/$TARGET
+fi
+
+# Build the SSO server package
+
+cd "$WORKSPACE"
+
+SSO_VERSION=$(version_from_git)
+PACKAGE_NAME=arvados-sso
+
+if [[ ! -d "$WORKSPACE/tmp" ]]; then
+ mkdir $WORKSPACE/tmp
+fi
+
+if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
+ bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
+fi
+
+/usr/bin/git rev-parse HEAD > git-commit.version
+
+cd $WORKSPACE/packages/$TARGET
+
+# Annoyingly, we require a database.yml file for rake assets:precompile to work. So for now,
+# we do that in the upgrade script.
+# TODO: add bogus database.yml file so we can precompile the assets and put them in the
+# package. Then remove that database.yml file again. It has to be a valid file though.
+#RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile
+
+# Append --depends X and other arguments specified by fpm-info.sh in
+# the package source dir. These are added last so they can override
+# the arguments added by this script.
+declare -a fpm_args=()
+declare -a fpm_depends=()
+FPM_INFO="$WORKSPACE/fpm-info.sh"
+if [[ -e "$FPM_INFO" ]]; then
+ debug_echo "Loading fpm overrides from $FPM_INFO"
+ source "$FPM_INFO"
+fi
+
+# This is the complete package with vendor/bundle included.
+# It's big, so we do not build it by default.
+if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados SSO server - Arvados is a free and open source platform for big data science.'" "--license='Expat License'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$SSO_VERSION" "-x" "var/www/arvados-sso/current/.git" "-x" "var/www/arvados-sso/current/tmp" "-x" "var/www/arvados-sso/current/log" "-x" "var/www/arvados-sso/current/vendor/cache/*" "-x" "var/www/arvados-sso/current/packages" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/postinst.sh")
+
+ for i in "${fpm_depends[@]}"; do
+ COMMAND_ARR+=('--depends' "$i")
+ done
+ COMMAND_ARR+=("${fpm_args[@]}")
+ COMMAND_ARR+=("$WORKSPACE/=/var/www/arvados-sso/current" "$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server-upgrade.sh=/usr/local/bin/arvados-sso-server-upgrade.sh")
+
+ debug_echo -e "\n${COMMAND_ARR[@]}\n"
+
+ FPM_RESULTS=$("${COMMAND_ARR[@]}")
+ FPM_EXIT_CODE=$?
+ fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
+fi
+
+# Build the 'bare' package without vendor/bundle.
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados SSO server - Arvados is a free and open source platform for big data science.'" "--license='Expat License'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$SSO_VERSION" "-x" "var/www/arvados-sso/current/.git" "-x" "var/www/arvados-sso/current/tmp" "-x" "var/www/arvados-sso/current/log" "-x" "var/www/arvados-sso/current/vendor/bundle" "-x" "var/www/arvados-sso/current/vendor/cache/*" "-x" "var/www/arvados-sso/current/packages" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/postinst.sh")
+
+
+for i in "${fpm_depends[@]}"; do
+ COMMAND_ARR+=('--depends' "$i")
+done
+COMMAND_ARR+=("${fpm_args[@]}")
+COMMAND_ARR+=("$WORKSPACE/=/var/www/arvados-sso/current" "$RUN_BUILD_PACKAGES_PATH/arvados-sso-server-extras/arvados-sso-server-upgrade.sh=/usr/local/bin/arvados-sso-server-upgrade.sh")
+debug_echo -e "\n${COMMAND_ARR[@]}\n"
+
+FPM_RESULTS=$("${COMMAND_ARR[@]}")
+FPM_EXIT_CODE=$?
+fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
+
+# SSO server package build done
+
+exit $EXITCODE
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index fbd5fa5..9f7c37f 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -15,6 +15,9 @@ Options:
Output debug information (default: false)
--target
Distribution to build packages for (default: debian7)
+--command
+ Build command to execute (defaults to the run command defined in the
+ Docker image)
WORKSPACE=path Path to the Arvados source tree to build packages from
@@ -49,6 +52,9 @@ while [ $# -gt 0 ]; do
--build-bundle-packages)
BUILD_BUNDLE_PACKAGES=1
;;
+ --command)
+ COMMAND="$2"; shift
+ ;;
--)
if [ $# -gt 1 ]; then
echo >&2 "$0: unrecognized argument '$2'. Try: $0 --help"
@@ -59,6 +65,10 @@ while [ $# -gt 0 ]; do
shift
done
+if [[ "$COMMAND" != "" ]]; then
+ COMMAND="/usr/local/rvm/bin/rvm-exec default bash /jenkins/$COMMAND --target $TARGET"
+fi
+
STDOUT_IF_DEBUG=/dev/null
STDERR_IF_DEBUG=/dev/null
DASHQ_UNLESS_DEBUG=-q
commit 823a16ae78acdadd17d6f563dfafe4678c364906
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Aug 31 15:23:00 2015 -0400
Tests compatible libcloud package is installed.
diff --git a/jenkins/common-test-packages.sh b/jenkins/common-test-packages.sh
index 52bbba0..c9d77c8 100755
--- a/jenkins/common-test-packages.sh
+++ b/jenkins/common-test-packages.sh
@@ -15,6 +15,11 @@ if ! python <<EOF
import arvados
import arvados_fuse
print "Successly imported arvados and arvados_fuse"
+
+import libcloud.compute.types
+import libcloud.compute.providers
+libcloud.compute.providers.get_driver(libcloud.compute.types.Provider.AZURE_ARM)
+print "Successly imported compatible libcloud library"
EOF
then
FAIL=1
diff --git a/jenkins/deb-common-test-packages.sh b/jenkins/deb-common-test-packages.sh
index d781c8b..f40ba7b 100755
--- a/jenkins/deb-common-test-packages.sh
+++ b/jenkins/deb-common-test-packages.sh
@@ -25,7 +25,9 @@ fi
echo "deb file:///mnt /" >>/etc/apt/sources.list
apt-get -qq update
-if ! apt-get -qq --assume-yes --force-yes install python-arvados-python-client python-arvados-fuse ; then
+if ! apt-get -qq --assume-yes --force-yes install \
+ python-arvados-python-client python-arvados-fuse arvados-node-manager
+then
exit 1
fi
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index 580bb27..555b9f1 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -40,7 +40,9 @@ EOF
yum -q clean all
yum -q update
-if ! yum -q install --assumeyes python27-python-arvados-python-client python27-python-arvados-fuse ; then
+if ! yum -q install --assumeyes \
+ python27-python-arvados-python-client python27-python-arvados-fuse arvados-node-manager
+then
exit 1
fi
commit 21a3d01379891f2670991e4d24804e1dc87a1ab1
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Aug 31 11:18:05 2015 -0400
Log package dependencies for each shared object. Add -q so make the logs a bit
less chatty.
diff --git a/jenkins/common-test-packages.sh b/jenkins/common-test-packages.sh
index 869f054..52bbba0 100755
--- a/jenkins/common-test-packages.sh
+++ b/jenkins/common-test-packages.sh
@@ -2,6 +2,7 @@
FAIL=0
+echo
for so in $(find . -name "*.so") ; do
if ldd $so | grep "not found" ; then
echo "^^^ Missing while scanning $so ^^^"
@@ -9,6 +10,7 @@ for so in $(find . -name "*.so") ; do
fi
done
+echo
if ! python <<EOF
import arvados
import arvados_fuse
diff --git a/jenkins/deb-common-test-packages.sh b/jenkins/deb-common-test-packages.sh
index fbebe24..d781c8b 100755
--- a/jenkins/deb-common-test-packages.sh
+++ b/jenkins/deb-common-test-packages.sh
@@ -24,8 +24,8 @@ if test "$1" = --run-test ; then
fi
echo "deb file:///mnt /" >>/etc/apt/sources.list
-apt-get update
-if ! apt-get --assume-yes --force-yes install python-arvados-python-client python-arvados-fuse ; then
+apt-get -qq update
+if ! apt-get -qq --assume-yes --force-yes install python-arvados-python-client python-arvados-fuse ; then
exit 1
fi
@@ -36,4 +36,10 @@ for r in /mnt/python-*amd64.deb ; do
dpkg-deb -x $r .
done
+for so in $(find . -name "*.so") ; do
+ echo
+ echo "== Packages dependencies for $so =="
+ ldd $so | awk '($3 ~ /^\//){print $3}' | sort -u | xargs dpkg -S | cut -d: -f1 | sort -u
+done
+
exec /root/common-test.sh
diff --git a/jenkins/run-test-packages.sh b/jenkins/run-test-packages.sh
index dce7362..baca7b7 100755
--- a/jenkins/run-test-packages.sh
+++ b/jenkins/run-test-packages.sh
@@ -7,11 +7,17 @@ fi
FAIL=0
+ERRORS=""
+
for pkg in ./test-packages-*.sh ; do
+ echo
+ echo "== Running $pkg =="
+ echo
if ! $pkg --run-test ; then
FAIL=1
- echo "$pkg has install errors"
+ ERRORS="$ERRORS\n$pkg has install errors"
fi
done
+echo $ERRORS
exit $FAIL
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index ea3ed09..580bb27 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -23,10 +23,10 @@ if test "$1" = --run-test ; then
fi
if test "$1" = --install-scl ; then
- yum install --assumeyes scl-utils
+ yum -q install --assumeyes scl-utils
curl -L -O https://www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm
- yum install --assumeyes rhscl-python27-epel-6-x86_64.noarch.rpm
- yum install --assumeyes python27
+ yum -q install --assumeyes rhscl-python27-epel-6-x86_64.noarch.rpm
+ yum -q install --assumeyes python27
exec scl enable python27 $0
fi
@@ -38,9 +38,9 @@ gpgcheck=0
enabled=1
EOF
-yum clean all
-yum update
-if ! yum install --assumeyes python27-python-arvados-python-client python27-python-arvados-fuse ; then
+yum -q clean all
+yum -q update
+if ! yum -q install --assumeyes python27-python-arvados-python-client python27-python-arvados-fuse ; then
exit 1
fi
@@ -51,4 +51,10 @@ for r in /mnt/python27-python-*x86_64.rpm ; do
rpm2cpio $r | cpio -idm
done
+for so in $(find . -name "*.so") ; do
+ echo
+ echo "== Packages dependencies for $so =="
+ ldd $so | awk '($3 ~ /^\//){print $3}' | sort -u | xargs rpm -qf | sort -u
+done
+
exec /root/common-test.sh
commit 8713e250470011db40d0cc9f1a02afbf98d2853c
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Sat Aug 29 15:15:55 2015 -0400
Sets up local repositories for both centos (yum) and debian (apt), installs
from there as if an end user would.
diff --git a/jenkins/common-test-packages.sh b/jenkins/common-test-packages.sh
new file mode 100755
index 0000000..869f054
--- /dev/null
+++ b/jenkins/common-test-packages.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+FAIL=0
+
+for so in $(find . -name "*.so") ; do
+ if ldd $so | grep "not found" ; then
+ echo "^^^ Missing while scanning $so ^^^"
+ FAIL=1
+ fi
+done
+
+if ! python <<EOF
+import arvados
+import arvados_fuse
+print "Successly imported arvados and arvados_fuse"
+EOF
+then
+ FAIL=1
+fi
+
+exit $FAIL
diff --git a/jenkins/test-packages-deb-common.sh b/jenkins/deb-common-test-packages.sh
similarity index 57%
rename from jenkins/test-packages-deb-common.sh
rename to jenkins/deb-common-test-packages.sh
index d2c586c..fbebe24 100755
--- a/jenkins/test-packages-deb-common.sh
+++ b/jenkins/deb-common-test-packages.sh
@@ -8,13 +8,16 @@ if test "$1" = --run-test ; then
fi
self=$(readlink -f $0)
- cd $WORKSPACE/packages/debian7
+ base=$(dirname $self)
+
+ cd $WORKSPACE/packages/$2
dpkg-scanpackages . /dev/null | gzip -c9 > Packages.gz
exec docker run \
--rm \
--volume=$WORKSPACE/packages/$2:/mnt \
--volume=$self:/root/run-test.sh \
+ --volume=$base/common-test-packages.sh:/root/common-test.sh \
--workdir=/mnt \
$3 \
/root/run-test.sh
@@ -22,9 +25,15 @@ fi
echo "deb file:///mnt /" >>/etc/apt/sources.list
apt-get update
-apt-get --assume-yes --force-yes install python-arvados-python-client python-arvados-fuse
+if ! apt-get --assume-yes --force-yes install python-arvados-python-client python-arvados-fuse ; then
+ exit 1
+fi
+
+mkdir -p /tmp/opts
+cd /tmp/opts
+
+for r in /mnt/python-*amd64.deb ; do
+ dpkg-deb -x $r .
+done
-python <<EOF
-import arvados
-import arvados_fuse
-EOF
+exec /root/common-test.sh
diff --git a/jenkins/run-test-packages.sh b/jenkins/run-test-packages.sh
index c4f5bec..dce7362 100755
--- a/jenkins/run-test-packages.sh
+++ b/jenkins/run-test-packages.sh
@@ -5,6 +5,13 @@ if test -z "$WORKSPACE" ; then
exit 1
fi
-for pkg in test-packages-*.sh ; do
- $pkg --run-test
+FAIL=0
+
+for pkg in ./test-packages-*.sh ; do
+ if ! $pkg --run-test ; then
+ FAIL=1
+ echo "$pkg has install errors"
+ fi
done
+
+exit $FAIL
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
index 6dd87e9..ea3ed09 100755
--- a/jenkins/test-packages-centos6.sh
+++ b/jenkins/test-packages-centos6.sh
@@ -7,10 +7,16 @@ if test "$1" = --run-test ; then
exit 1
fi
+ self=$(readlink -f $0)
+ base=$(dirname $self)
+
+ createrepo $WORKSPACE/packages/centos6
+
exec docker run \
--rm \
--volume=$WORKSPACE/packages/centos6:/mnt \
--volume=$(readlink -f $0):/root/run-test.sh \
+ --volume=$base/common-test-packages.sh:/root/common-test.sh \
--workdir=/mnt \
centos:6 \
/root/run-test.sh --install-scl
@@ -24,7 +30,19 @@ if test "$1" = --install-scl ; then
exec scl enable python27 $0
fi
-yum install --assumeyes python27-python*.rpm
+cat >/etc/yum.repos.d/localrepo.repo <<EOF
+[localrepo]
+name=Arvados Test
+baseurl=file:///mnt
+gpgcheck=0
+enabled=1
+EOF
+
+yum clean all
+yum update
+if ! yum install --assumeyes python27-python-arvados-python-client python27-python-arvados-fuse ; then
+ exit 1
+fi
mkdir -p /tmp/opts
cd /tmp/opts
@@ -33,18 +51,4 @@ for r in /mnt/python27-python-*x86_64.rpm ; do
rpm2cpio $r | cpio -idm
done
-FAIL=0
-
-for so in $(find . -name "*.so") ; do
- if ldd $so | grep "not found" ; then
- echo "^^^ Missing while scanning $so ^^^"
- FAIL=1
- fi
-done
-
-python <<EOF
-import arvados
-import arvados_fuse
-EOF
-
-exit $FAIL
+exec /root/common-test.sh
diff --git a/jenkins/test-packages-debian7.sh b/jenkins/test-packages-debian7.sh
index c48761c..c5cbfe6 100755
--- a/jenkins/test-packages-debian7.sh
+++ b/jenkins/test-packages-debian7.sh
@@ -1,3 +1,3 @@
#!/bin/sh
-exec ./test-packages-deb-common.sh --run-test debian7 debian:wheezy
+exec ./deb-common-test-packages.sh --run-test debian7 debian:wheezy
diff --git a/jenkins/test-packages-debian8.sh b/jenkins/test-packages-debian8.sh
index 6e42806..5502f46 100755
--- a/jenkins/test-packages-debian8.sh
+++ b/jenkins/test-packages-debian8.sh
@@ -1,3 +1,3 @@
#!/bin/sh
-exec ./test-packages-deb-common.sh --run-test debian8 debian:jessie
+exec ./deb-common-test-packages.sh --run-test debian8 debian:jessie
diff --git a/jenkins/test-packages-ubuntu1204.sh b/jenkins/test-packages-ubuntu1204.sh
index a8af4d8..00727a0 100755
--- a/jenkins/test-packages-ubuntu1204.sh
+++ b/jenkins/test-packages-ubuntu1204.sh
@@ -1,3 +1,3 @@
#!/bin/sh
-exec ./test-packages-deb-common.sh --run-test ubuntu1204 ubuntu:precise
+exec ./deb-common-test-packages.sh --run-test ubuntu1204 ubuntu:precise
diff --git a/jenkins/test-packages-ubuntu1404.sh b/jenkins/test-packages-ubuntu1404.sh
index a70a1f6..bfc2c85 100755
--- a/jenkins/test-packages-ubuntu1404.sh
+++ b/jenkins/test-packages-ubuntu1404.sh
@@ -1,3 +1,3 @@
#!/bin/sh
-exec ./test-packages-deb-common.sh --run-test ubuntu1404 ubuntu:trusty
+exec ./deb-common-test-packages.sh --run-test ubuntu1404 ubuntu:trusty
commit 77f198433d31d492d86c38359940acd7f5012cf8
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Aug 28 17:18:20 2015 -0400
Script to automatically test python package installation on clean Docker
images.
diff --git a/jenkins/run-test-packages.sh b/jenkins/run-test-packages.sh
new file mode 100755
index 0000000..c4f5bec
--- /dev/null
+++ b/jenkins/run-test-packages.sh
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+if test -z "$WORKSPACE" ; then
+ echo "Must set WORKSPACE"
+ exit 1
+fi
+
+for pkg in test-packages-*.sh ; do
+ $pkg --run-test
+done
diff --git a/jenkins/test-packages-centos6.sh b/jenkins/test-packages-centos6.sh
new file mode 100755
index 0000000..6dd87e9
--- /dev/null
+++ b/jenkins/test-packages-centos6.sh
@@ -0,0 +1,50 @@
+#!/bin/sh
+
+if test "$1" = --run-test ; then
+
+ if test -z "$WORKSPACE" ; then
+ echo "Must set WORKSPACE"
+ exit 1
+ fi
+
+ exec docker run \
+ --rm \
+ --volume=$WORKSPACE/packages/centos6:/mnt \
+ --volume=$(readlink -f $0):/root/run-test.sh \
+ --workdir=/mnt \
+ centos:6 \
+ /root/run-test.sh --install-scl
+fi
+
+if test "$1" = --install-scl ; then
+ yum install --assumeyes scl-utils
+ curl -L -O https://www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm
+ yum install --assumeyes rhscl-python27-epel-6-x86_64.noarch.rpm
+ yum install --assumeyes python27
+ exec scl enable python27 $0
+fi
+
+yum install --assumeyes python27-python*.rpm
+
+mkdir -p /tmp/opts
+cd /tmp/opts
+
+for r in /mnt/python27-python-*x86_64.rpm ; do
+ rpm2cpio $r | cpio -idm
+done
+
+FAIL=0
+
+for so in $(find . -name "*.so") ; do
+ if ldd $so | grep "not found" ; then
+ echo "^^^ Missing while scanning $so ^^^"
+ FAIL=1
+ fi
+done
+
+python <<EOF
+import arvados
+import arvados_fuse
+EOF
+
+exit $FAIL
diff --git a/jenkins/test-packages-deb-common.sh b/jenkins/test-packages-deb-common.sh
new file mode 100755
index 0000000..d2c586c
--- /dev/null
+++ b/jenkins/test-packages-deb-common.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+
+if test "$1" = --run-test ; then
+
+ if test -z "$WORKSPACE" ; then
+ echo "Must set WORKSPACE"
+ exit 1
+ fi
+
+ self=$(readlink -f $0)
+ cd $WORKSPACE/packages/debian7
+ dpkg-scanpackages . /dev/null | gzip -c9 > Packages.gz
+
+ exec docker run \
+ --rm \
+ --volume=$WORKSPACE/packages/$2:/mnt \
+ --volume=$self:/root/run-test.sh \
+ --workdir=/mnt \
+ $3 \
+ /root/run-test.sh
+fi
+
+echo "deb file:///mnt /" >>/etc/apt/sources.list
+apt-get update
+apt-get --assume-yes --force-yes install python-arvados-python-client python-arvados-fuse
+
+python <<EOF
+import arvados
+import arvados_fuse
+EOF
diff --git a/jenkins/test-packages-debian7.sh b/jenkins/test-packages-debian7.sh
new file mode 100755
index 0000000..c48761c
--- /dev/null
+++ b/jenkins/test-packages-debian7.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+exec ./test-packages-deb-common.sh --run-test debian7 debian:wheezy
diff --git a/jenkins/test-packages-debian8.sh b/jenkins/test-packages-debian8.sh
new file mode 100755
index 0000000..6e42806
--- /dev/null
+++ b/jenkins/test-packages-debian8.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+exec ./test-packages-deb-common.sh --run-test debian8 debian:jessie
diff --git a/jenkins/test-packages-ubuntu1204.sh b/jenkins/test-packages-ubuntu1204.sh
new file mode 100755
index 0000000..a8af4d8
--- /dev/null
+++ b/jenkins/test-packages-ubuntu1204.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+exec ./test-packages-deb-common.sh --run-test ubuntu1204 ubuntu:precise
diff --git a/jenkins/test-packages-ubuntu1404.sh b/jenkins/test-packages-ubuntu1404.sh
new file mode 100755
index 0000000..a70a1f6
--- /dev/null
+++ b/jenkins/test-packages-ubuntu1404.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+exec ./test-packages-deb-common.sh --run-test ubuntu1404 ubuntu:trusty
commit b5494642ea862658418d2f9a419b10d1fe25431a
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Aug 28 11:37:35 2015 -0400
Optimization: do not run bundle install for the API server package when
we aren't including the vendor directory in the package we're building.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index cd83513..fbd5fa5 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -584,7 +584,9 @@ if [[ ! -d "$WORKSPACE/services/api/tmp" ]]; then
fi
-bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
+if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
+ bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
+fi
/usr/bin/git rev-parse HEAD > git-commit.version
@@ -630,6 +632,8 @@ if [[ ! -d "$WORKSPACE/apps/workbench/tmp" ]]; then
mkdir $WORKSPACE/apps/workbench/tmp
fi
+# We need to bundle to be ready even when we build a package without vendor directory
+# because asset compilation requires it.
bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
/usr/bin/git rev-parse HEAD > git-commit.version
commit b8568977a1c6f7ff82900291814699617f8547b1
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Aug 28 11:37:35 2015 -0400
Optimization: do not run bundle install for the API server package when
we aren't including the vendor directory in the package we're building.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index cd83513..fbd5fa5 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -584,7 +584,9 @@ if [[ ! -d "$WORKSPACE/services/api/tmp" ]]; then
fi
-bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
+if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
+ bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
+fi
/usr/bin/git rev-parse HEAD > git-commit.version
@@ -630,6 +632,8 @@ if [[ ! -d "$WORKSPACE/apps/workbench/tmp" ]]; then
mkdir $WORKSPACE/apps/workbench/tmp
fi
+# We need to bundle to be ready even when we build a package without vendor directory
+# because asset compilation requires it.
bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
/usr/bin/git rev-parse HEAD > git-commit.version
commit 8003783f717ecb167a47c11d01f459b43670fd0e
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Aug 28 10:40:06 2015 -0400
Preinstall forked version of libcloud, because nodemanager "pip install" won't
pick it up by default. refs #6507
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index cdff551..e2ae4c5 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -412,6 +412,12 @@ pip freeze 2>/dev/null | egrep ^PyYAML= \
|| pip install PyYAML >/dev/null \
|| fatal "pip install PyYAML failed"
+# Preinstall forked version of libcloud, because nodemanager "pip install"
+# won't pick it up by default.
+pip freeze 2>/dev/null | egrep ^apache-libcloud==0.18.1.dev1 \
+ || pip install --pre --ignore-installed https://github.com/curoverse/libcloud/archive/apache-libcloud-0.18.1.dev1.zip >/dev/null \
+ || fatal "pip install apache-libcloud failed"
+
# If Python 3 is available, set up its virtualenv in $VENV3DIR.
# Otherwise, skip dependent tests.
PYTHON3=$(which python3)
commit a3ec6d08fcaa76d3b16906f57eb07bbecafce545
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Aug 28 09:51:43 2015 -0400
run-build-packages-all-targets.sh should abort of WORKSPACE is not set.
No issue #
diff --git a/jenkins/run-build-packages-all-targets.sh b/jenkins/run-build-packages-all-targets.sh
index a1052a9..2761f9f 100755
--- a/jenkins/run-build-packages-all-targets.sh
+++ b/jenkins/run-build-packages-all-targets.sh
@@ -1,6 +1,22 @@
#!/bin/bash
-# Orchestrate run-build-packages.sh for every target.
+read -rd "\000" helpmessage <<EOF
+$(basename $0): Orchestrate run-build-packages.sh for every target
+
+Syntax:
+ WORKSPACE=/path/to/arvados $(basename $0)
+
+WORKSPACE=path Path to the Arvados source tree to build packages from
+
+EOF
+
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
+fi
set -e
commit 8379e4d518e9351212004e1d4155d8b869e727f1
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Aug 28 09:51:43 2015 -0400
run-build-packages-all-targets.sh should abort of WORKSPACE is not set.
No issue #
diff --git a/jenkins/run-build-packages-all-targets.sh b/jenkins/run-build-packages-all-targets.sh
index a1052a9..2761f9f 100755
--- a/jenkins/run-build-packages-all-targets.sh
+++ b/jenkins/run-build-packages-all-targets.sh
@@ -1,6 +1,22 @@
#!/bin/bash
-# Orchestrate run-build-packages.sh for every target.
+read -rd "\000" helpmessage <<EOF
+$(basename $0): Orchestrate run-build-packages.sh for every target
+
+Syntax:
+ WORKSPACE=/path/to/arvados $(basename $0)
+
+WORKSPACE=path Path to the Arvados source tree to build packages from
+
+EOF
+
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
+fi
set -e
commit fe19ce4b94f7ae3277e9016bc7886179f46ec472
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Aug 27 15:47:45 2015 -0400
6263: Generate Go coverage reports in $WORKSPACE/tmp/coverage-*
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index cdff551..281b721 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -458,22 +458,28 @@ do_test() {
}
do_test_once() {
+ unset result
if [[ -z "${skip[$1]}" ]] && ( [[ -z "$only" ]] || [[ "$only" == "$1" ]] )
then
title "Running $1 tests"
timer_reset
if [[ "$2" == "go" ]]
then
+ covername="coverage-$(echo "$1" | sed -e 's/\//_/g')"
+ coverflags=("-covermode=count" "-coverprofile=$WORKSPACE/tmp/.$covername.tmp")
if [[ -n "${testargs[$1]}" ]]
then
# "go test -check.vv giturl" doesn't work, but this
# does:
- cd "$WORKSPACE/$1" && go test ${testargs[$1]}
+ cd "$WORKSPACE/$1" && go test ${coverflags[@]} ${testargs[$1]}
else
# The above form gets verbose even when testargs is
# empty, so use this form in such cases:
- go test "git.curoverse.com/arvados.git/$1"
+ go test ${coverflags[@]} "git.curoverse.com/arvados.git/$1"
fi
+ result="$?"
+ go tool cover -html="$WORKSPACE/tmp/.$covername.tmp" -o "$WORKSPACE/tmp/$covername.html"
+ rm "$WORKSPACE/tmp/.$covername.tmp"
elif [[ "$2" == "pip" ]]
then
# $3 can name a path directory for us to use, including trailing
@@ -486,7 +492,7 @@ do_test_once() {
else
"test_$1"
fi
- result="$?"
+ result=${result:-$?}
checkexit $result "$1 tests"
title "End of $1 tests (`timer`)"
return $result
commit aafddea00106bd7e9fd6336623788dbab95b7306
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Aug 27 14:51:29 2015 -0400
Update/clarify package assumptions for api server and workbench.
No issue #
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
index cbd35d9..b3badf7 100755
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -29,8 +29,8 @@ RELEASE_PATH=/var/www/arvados-api/current
SHARED_PATH=/var/www/arvados-api/shared
CONFIG_PATH=/etc/arvados/api/
-echo "Assumption: $NGINX_SERVICE is configured to serve $HOSTNAME from /var/www/$HOSTNAME/current"
-echo "Assumption: /var/www/$HOSTNAME is symlinked to /var/www/arvados-api"
+echo "Assumption: $NGINX_SERVICE is configured to serve your API server URL from
+echo " /var/www/arvados-api/current"
echo "Assumption: configuration files are in /etc/arvados/api/"
echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
echo
diff --git a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
index 45eca77..a481985 100755
--- a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
+++ b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
@@ -29,8 +29,8 @@ RELEASE_PATH=/var/www/arvados-workbench/current
SHARED_PATH=/var/www/arvados-workbench/shared
CONFIG_PATH=/etc/arvados/workbench/
-echo "Assumption: $NGINX_SERVICE is configured to serve workbench.$HOSTNAME from /var/www/workbench.$HOSTNAME/current"
-echo "Assumption: /var/www/$HOSTNAME is symlinked to /var/www/arvados-workbench"
+echo "Assumption: $NGINX_SERVICE is configured to serve your workbench URL from "
+echo " /var/www/arvados-workbench/current"
echo "Assumption: configuration files are in /etc/arvados/workbench/"
echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
echo
commit bab8ca353f89418b0d92ec639e5964cd1d2aada5
Merge: fc1a358 feac224
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Aug 26 11:29:32 2015 -0400
Merge branch '6781-arvados-login-sync' refs #6781
commit feac22422cfdc33e14633b1a2b724c49eea69d94
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Aug 19 14:49:00 2015 -0400
Stop building broken debs for our gems. We don't use them and they have
never worked. We can reintroduce a working version later.
refs #6781
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 1c59fa8..cd83513 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -241,10 +241,6 @@ handle_ruby_gem() {
# -q appears to be broken in gem version 2.2.2
$GEM build "$gem_name.gemspec" $DASHQ_UNLESS_DEBUG >"$STDOUT_IF_DEBUG" 2>"$STDERR_IF_DEBUG"
fi
-
- cd "$WORKSPACE/packages/$TARGET"
- fpm_build "$gem_src_dir/$gem_name"-*.gem "" "Curoverse, Inc." gem "" \
- --prefix "$FPM_GEM_PREFIX"
}
# Build packages for everything
commit 0be695eef3af4788a0890060a765e6b0d2c32334
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Aug 13 11:02:51 2015 -0400
6781: Add arvados-login-sync.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 96bcea0..1c59fa8 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -418,6 +418,9 @@ handle_ruby_gem arvados
cd "$WORKSPACE/sdk/cli"
handle_ruby_gem arvados-cli
+cd "$WORKSPACE/services/login-sync"
+handle_ruby_gem arvados-login-sync
+
# Python packages
debug_echo -e "\nPython packages\n"
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index e191b44..cdff551 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -60,6 +60,7 @@ services/dockercleaner
services/fuse
services/keepproxy
services/keepstore
+services/login-sync
services/nodemanager
services/arv-git-httpd
sdk/cli
@@ -558,12 +559,18 @@ install_doc() {
}
do_install doc
-install_ruby_sdk() {
- with_test_gemset gem_uninstall_if_exists arvados \
- && cd "$WORKSPACE/sdk/ruby" \
+install_gem() {
+ gemname=$1
+ srcpath=$2
+ with_test_gemset gem_uninstall_if_exists "$gemname" \
+ && cd "$WORKSPACE/$srcpath" \
&& bundle_install_trylocal \
- && gem build arvados.gemspec \
- && with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
+ && gem build "$gemname.gemspec" \
+ && with_test_gemset gem install --no-ri --no-rdoc $(ls -t "$gemname"-*.gem|head -n1)
+}
+
+install_ruby_sdk() {
+ install_gem arvados sdk/ruby
}
do_install sdk/ruby ruby_sdk
@@ -575,14 +582,15 @@ install_perl_sdk() {
do_install sdk/perl perl_sdk
install_cli() {
- with_test_gemset gem_uninstall_if_exists arvados-cli \
- && cd "$WORKSPACE/sdk/cli" \
- && bundle_install_trylocal \
- && gem build arvados-cli.gemspec \
- && with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
+ install_gem arvados-cli sdk/cli
}
do_install sdk/cli cli
+install_login-sync() {
+ install_gem arvados-login-sync services/login-sync
+}
+do_install services/login-sync login-sync
+
# Install the Python SDK early. Various other test suites (like
# keepproxy) bring up run_test_server.py, which imports the arvados
# module. We can't actually *test* the Python SDK yet though, because
@@ -719,6 +727,12 @@ test_cli() {
}
do_test sdk/cli cli
+test_login-sync() {
+ cd "$WORKSPACE/services/login-sync" \
+ && bundle exec rake test TESTOPTS=-v ${testargs[services/login-sync]}
+}
+do_test services/login-sync login-sync
+
for p in "${pythonstuff[@]}"
do
do_test "$p" pip
diff --git a/jenkins/run_upload_packages.py b/jenkins/run_upload_packages.py
index eff30cc..5fc134c 100755
--- a/jenkins/run_upload_packages.py
+++ b/jenkins/run_upload_packages.py
@@ -192,7 +192,11 @@ PACKAGE_SUITES = {
'services/nodemanager/dist/*.tar.gz',
'services/fuse/dist/*.tar.gz',
),
- 'gems': _define_suite(GemPackageSuite, 'sdk/ruby/*.gem', 'sdk/cli/*.gem'),
+ 'gems': _define_suite(GemPackageSuite,
+ 'sdk/ruby/*.gem',
+ 'sdk/cli/*.gem',
+ 'services/login-sync/*.gem',
+ ),
}
for target in ['debian7', 'debian8', 'ubuntu1204', 'ubuntu1404']:
PACKAGE_SUITES[target] = _define_suite(
commit fd45d42570bad689429ac62df86929e6b0c86cdf
Author: radhika <radhika at curoverse.com>
Date: Tue Aug 25 12:38:35 2015 -0400
6260: add datamanager to run-tests.sh
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index e191b44..48ac24d 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -661,6 +661,7 @@ gostuff=(
services/keepproxy
services/datamanager/summary
services/datamanager/collection
+ services/datamanager
sdk/go/arvadosclient
sdk/go/keepclient
sdk/go/streamer
@@ -727,7 +728,13 @@ do_test services/dockercleaner pip "$VENV3DIR/bin/"
for g in "${gostuff[@]}"
do
- do_test "$g" go
+ if [[ "$g" == "services/datamanager" ]]
+ then
+ testargs["$g"]="-data-manager-token-file $WORKSPACE/tmp/keep.data-manager-token-file"
+ do_test "$g" go
+ else
+ do_test "$g" go
+ fi
done
test_workbench() {
commit fc1a358d5403e87bfdeca2d0c510362576a8c2e4
Author: Nico Cesar <nico at curoverse.com>
Date: Mon Aug 24 17:35:20 2015 -0400
7041: pyvcf was missing from backports in trusty
refs #7041
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index fed66e5..96bcea0 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -123,7 +123,7 @@ case "$TARGET" in
PYTHON2_PKG_PREFIX=python
PYTHON3_PACKAGE=python$PYTHON3_VERSION
PYTHON3_PKG_PREFIX=python3
- PYTHON_BACKPORTS=(pyasn1==0.1.7 pyasn1-modules==0.0.5 llfuse ciso8601 \
+ PYTHON_BACKPORTS=(pyasn1==0.1.7 pyvcf pyasn1-modules==0.0.5 llfuse ciso8601 \
google-api-python-client six uritemplate oauth2client httplib2 \
rsa apache-libcloud pycurl backports.ssl_match_hostname)
PYTHON3_BACKPORTS=(docker-py requests websocket-client)
commit 06dd898fe349cbc5a16fb5f6daf2b563fbb3c936
Author: Brett Smith <brett at curoverse.com>
Date: Thu Aug 20 11:20:54 2015 -0400
6940: Use nginx as WWW_OWNER on Red Hat distributions.
We recommend deploying with Nginx, and this script interacts with
Nginx. The default Nginx configuration that Phusion ships runs as the
nginx user on these distros. Use that as the WWW_OWNER.
Ideally we want to move to a more sophisticated place where we
introspect the Web server configuration and use the same server it
does. But for now, this is an improvement.
No issue #, but discovered when working on #6940. I cleared this with
Ward.
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
index 5160c7a..cbd35d9 100755
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -3,7 +3,7 @@
set -e
if [ -e /etc/redhat-release ]; then
- WWW_OWNER=apache:apache
+ WWW_OWNER=nginx:nginx
else
# Assume we're on a Debian-based system for now.
WWW_OWNER=www-data:www-data
diff --git a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
index 0a2bf96..45eca77 100755
--- a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
+++ b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
@@ -3,7 +3,7 @@
set -e
if [ -e /etc/redhat-release ]; then
- WWW_OWNER=apache:apache
+ WWW_OWNER=nginx:nginx
else
# Assume we're on a Debian-based system for now.
WWW_OWNER=www-data:www-data
commit 81b4b70978fe97e6bd22bcb85cdfa36131f2f012
Merge: 7701a7a b0dd008
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Aug 14 17:32:58 2015 -0400
Merge branch '6638-backport-deps' refs #6638
commit 7701a7ad84b0d4fb4642568bd75f63a5c0586658
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Aug 14 15:22:07 2015 -0400
Make sure to also deploy switchyard.
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 0084d2f..c097a20 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -224,7 +224,7 @@ if [[ "$SUM_ECODE" != "0" ]]; then
exit $EXITCODE
fi
-for n in manage $SHELL_NODES $KEEP_NODES; do
+for n in manage switchyard $SHELL_NODES $KEEP_NODES; do
ECODE=0
if [[ $n =~ $ARVADOS_API_HOST$ ]]; then
# e.g. keep.qr1hi.arvadosapi.com
commit b0dd0080e98ffccb2d48f247583f55c8937bda84
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Aug 14 14:37:12 2015 -0400
6638: Use (git clone; git checkout) -- git clone --branch doesn't work with commit hashes.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 60d7ab6..fed66e5 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -441,8 +441,11 @@ handle_python_package
COMMIT_HASH=$(format_last_commit_here "%H")
SRC_BUILD_DIR=$(mktemp -d)
- git clone $DASHQ_UNLESS_DEBUG --branch "$COMMIT_HASH" "$WORKSPACE/.git" "$SRC_BUILD_DIR"
+ git clone $DASHQ_UNLESS_DEBUG "$WORKSPACE/.git" "$SRC_BUILD_DIR"
cd "$SRC_BUILD_DIR"
+
+ # go into detached-head state
+ git checkout $DASHQ_UNLESS_DEBUG "$COMMIT_HASH"
echo "$COMMIT_HASH" >git-commit.version
cd "$SRC_BUILD_DIR"
commit fcc0a767df83518e1b42c78ab7d4805fe6b5192b
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Aug 13 20:38:53 2015 -0400
Performance optimization for the run_upload_packages.py script: only
rebuild the freight cache for the distribution that is being updated.
No issue #
diff --git a/jenkins/run_upload_packages.py b/jenkins/run_upload_packages.py
index f7ddf7f..eff30cc 100755
--- a/jenkins/run_upload_packages.py
+++ b/jenkins/run_upload_packages.py
@@ -146,7 +146,7 @@ class DebianPackageSuite(DistroPackageSuite):
cd "$1"; shift
DISTNAME=$1; shift
freight add "$@" "apt/$DISTNAME"
-freight cache
+freight cache "apt/$DISTNAME"
rm "$@"
"""
TARGET_DISTNAMES = {
commit bdffa02ff66636f6e3b28d5ab553d9fcad69037b
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Aug 13 17:06:29 2015 -0400
6934: Add sdk/pam to run_upload_packages.py refs #6934
diff --git a/jenkins/run_upload_packages.py b/jenkins/run_upload_packages.py
index b0a9706..f7ddf7f 100755
--- a/jenkins/run_upload_packages.py
+++ b/jenkins/run_upload_packages.py
@@ -187,9 +187,11 @@ def _define_suite(suite_class, *rel_globs, **kwargs):
PACKAGE_SUITES = {
'python': _define_suite(PythonPackageSuite,
+ 'sdk/pam/dist/*.tar.gz',
'sdk/python/dist/*.tar.gz',
+ 'services/nodemanager/dist/*.tar.gz',
'services/fuse/dist/*.tar.gz',
- 'services/nodemanager/dist/*.tar.gz'),
+ ),
'gems': _define_suite(GemPackageSuite, 'sdk/ruby/*.gem', 'sdk/cli/*.gem'),
}
for target in ['debian7', 'debian8', 'ubuntu1204', 'ubuntu1404']:
commit 3f03b72f0c24ee2c5abfe5654107f6e1c019bc15
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Aug 12 16:10:17 2015 -0400
6638: Fix rvm usage error. Straighten out "need new gem" and "need to build new package from gem" logic.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 3893d32..60d7ab6 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -233,20 +233,18 @@ handle_python_package () {
handle_ruby_gem() {
local gem_name=$1; shift
local gem_version=$(nohash_version_from_git)
+ local gem_src_dir="$(pwd)"
- if [ -e "${gem_name}-${gem_version}.gem" ]; then
- # This gem doesn't need rebuilding.
- return
- fi
-
- find -maxdepth 1 -name "${gem_name}-*.gem" -delete
+ if ! [[ -e "${gem_name}-${gem_version}.gem" ]]; then
+ find -maxdepth 1 -name "${gem_name}-*.gem" -delete
- # -q appears to be broken in gem version 2.2.2
- gem build "$gem_name.gemspec" $DASHQ_UNLESS_DEBUG >"$STDOUT_IF_DEBUG" 2>"$STDERR_IF_DEBUG"
+ # -q appears to be broken in gem version 2.2.2
+ $GEM build "$gem_name.gemspec" $DASHQ_UNLESS_DEBUG >"$STDOUT_IF_DEBUG" 2>"$STDERR_IF_DEBUG"
+ fi
- fpm_build "$gem_name"-*.gem "" "Curoverse, Inc." gem "" \
+ cd "$WORKSPACE/packages/$TARGET"
+ fpm_build "$gem_src_dir/$gem_name"-*.gem "" "Curoverse, Inc." gem "" \
--prefix "$FPM_GEM_PREFIX"
- mv -t "$WORKSPACE/packages/$TARGET/" "$gem_name"*."$FORMAT"
}
# Build packages for everything
@@ -368,7 +366,10 @@ fpm_verify () {
}
if [[ -f /etc/profile.d/rvm.sh ]]; then
- source /etc/profile.d/rvm.sh
+ source /etc/profile.d/rvm.sh
+ GEM="rvm-exec default gem"
+else
+ GEM=gem
fi
# Make all files world-readable -- jenkins runs with umask 027, and has checked
@@ -409,11 +410,7 @@ perl Makefile.PL INSTALL_BASE=install >"$STDOUT_IF_DEBUG" && \
# Ruby gems
debug_echo -e "\nRuby gems\n"
-if type rvm-exec >/dev/null 2>&1; then
- FPM_GEM_PREFIX=$(rvm-exec system gem environment gemdir)
-else
- FPM_GEM_PREFIX=$(gem environment gemdir)
-fi
+FPM_GEM_PREFIX=$($GEM environment gemdir)
cd "$WORKSPACE/sdk/ruby"
handle_ruby_gem arvados
commit 59c684f9e8a545ffae5036b193d09e9c8ab7e689
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Aug 12 16:09:08 2015 -0400
6638: Add postgres libraries to build containers, so bundler can build the pg gem.
diff --git a/jenkins/dockerfiles/centos6/Dockerfile b/jenkins/dockerfiles/centos6/Dockerfile
index e08665b..ca974df 100644
--- a/jenkins/dockerfiles/centos6/Dockerfile
+++ b/jenkins/dockerfiles/centos6/Dockerfile
@@ -2,7 +2,7 @@ FROM centos:6
MAINTAINER Brett Smith <brett at curoverse.com>
# Install build dependencies provided in base distribution
-RUN yum -q -y install make automake gcc gcc-c++ libyaml-devel patch readline-devel zlib-devel libffi-devel openssl-devel bzip2 libtool bison sqlite-devel rpm-build git perl-ExtUtils-MakeMaker fuse-devel libattr-devel nss-devel libcurl-devel which tar scl-utils
+RUN yum -q -y install make automake gcc gcc-c++ libyaml-devel patch readline-devel zlib-devel libffi-devel openssl-devel bzip2 libtool bison sqlite-devel rpm-build git perl-ExtUtils-MakeMaker fuse-devel libattr-devel nss-devel libcurl-devel which tar scl-utils postgresql-devel
# Install golang binary
ADD generated/golang-amd64.tar.gz /usr/local/
diff --git a/jenkins/dockerfiles/debian7/Dockerfile b/jenkins/dockerfiles/debian7/Dockerfile
index ca4cc28..33083a0 100644
--- a/jenkins/dockerfiles/debian7/Dockerfile
+++ b/jenkins/dockerfiles/debian7/Dockerfile
@@ -2,7 +2,7 @@ FROM debian:wheezy
MAINTAINER Ward Vandewege <ward at curoverse.com>
# Install dependencies and set up system.
-RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libpq-dev
# Install RVM
RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
diff --git a/jenkins/dockerfiles/debian8/Dockerfile b/jenkins/dockerfiles/debian8/Dockerfile
index 5b81434..dc26e25 100644
--- a/jenkins/dockerfiles/debian8/Dockerfile
+++ b/jenkins/dockerfiles/debian8/Dockerfile
@@ -2,7 +2,7 @@ FROM debian:jessie
MAINTAINER Ward Vandewege <ward at curoverse.com>
# Install dependencies and set up system.
-RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev
# Install RVM
RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
diff --git a/jenkins/dockerfiles/ubuntu1204/Dockerfile b/jenkins/dockerfiles/ubuntu1204/Dockerfile
index 96e331c..642d2a0 100644
--- a/jenkins/dockerfiles/ubuntu1204/Dockerfile
+++ b/jenkins/dockerfiles/ubuntu1204/Dockerfile
@@ -2,7 +2,7 @@ FROM ubuntu:precise
MAINTAINER Ward Vandewege <ward at curoverse.com>
# Install dependencies and set up system.
-RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev
# Install RVM
RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
diff --git a/jenkins/dockerfiles/ubuntu1404/Dockerfile b/jenkins/dockerfiles/ubuntu1404/Dockerfile
index 4b56bce..69ed2cc 100644
--- a/jenkins/dockerfiles/ubuntu1404/Dockerfile
+++ b/jenkins/dockerfiles/ubuntu1404/Dockerfile
@@ -2,7 +2,7 @@ FROM ubuntu:trusty
MAINTAINER Brett Smith <brett at curoverse.com>
# Install dependencies and set up system.
-RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev libpq-dev
# Install RVM
RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
commit bf4baab1c1e2a422334fc07d77a44036832583a9
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Aug 12 12:01:29 2015 -0400
6638: Build current version of arvados-src, not github master.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 0bd0a43..3893d32 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -436,37 +436,25 @@ handle_python_package
cd "$WORKSPACE/services/nodemanager"
handle_python_package
-# Arvados-src
-# We use $WORKSPACE/src-build-dir as the clean directory from which to build the src package
-if [[ ! -d "$WORKSPACE/src-build-dir" ]]; then
- mkdir "$WORKSPACE/src-build-dir"
- cd "$WORKSPACE"
- git clone $DASHQ_UNLESS_DEBUG https://github.com/curoverse/arvados.git src-build-dir
-fi
+# arvados-src
+(
+ set -e
-# Get the commit hash we're building against, from the working directory
-cd "$WORKSPACE"
-MASTER_COMMIT_HASH=$(format_last_commit_here "%H")
-
-# Make sure we check out that commit in the clean $WORKSPACE/src-build-dir directory
-cd "$WORKSPACE/src-build-dir"
-# just in case, check out master
-git checkout $DASHQ_UNLESS_DEBUG master
-git pull $DASHQ_UNLESS_DEBUG
-# go into detached-head state
-MASTER_COMMIT_HASH=$(format_last_commit_here "%H")
-git checkout $DASHQ_UNLESS_DEBUG "$MASTER_COMMIT_HASH"
-echo "$MASTER_COMMIT_HASH" >git-commit.version
-
-# Build arvados src deb package
-cd "$WORKSPACE"
-PKG_VERSION=$(version_from_git)
-cd $WORKSPACE/packages/$TARGET
-fpm_build $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
+ cd "$WORKSPACE"
+ COMMIT_HASH=$(format_last_commit_here "%H")
+
+ SRC_BUILD_DIR=$(mktemp -d)
+ git clone $DASHQ_UNLESS_DEBUG --branch "$COMMIT_HASH" "$WORKSPACE/.git" "$SRC_BUILD_DIR"
+ cd "$SRC_BUILD_DIR"
+ echo "$COMMIT_HASH" >git-commit.version
+
+ cd "$SRC_BUILD_DIR"
+ PKG_VERSION=$(version_from_git)
+ cd $WORKSPACE/packages/$TARGET
+ fpm_build $SRC_BUILD_DIR/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
-# clean up, check out master and step away from detached-head state
-cd "$WORKSPACE/src-build-dir"
-git checkout $DASHQ_UNLESS_DEBUG master
+ rm -r "$SRC_BUILD_DIR"
+)
# Keep
export GOPATH=$(mktemp -d)
commit dfc5f78444e593f24ab976a6ad472c2d1dce12b9
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Aug 12 12:05:59 2015 -0400
6638: Read package dependencies from fpm-info.sh in source tree.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 16699ba..0bd0a43 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -314,6 +314,25 @@ fpm_build () {
COMMAND_ARR+=("$i")
done
+ # Append --depends X and other arguments specified by fpm-info.sh in
+ # the package source dir. These are added last so they can override
+ # the arguments added by this script.
+ declare -a fpm_args=()
+ declare -a fpm_depends=()
+ if [[ -d "$PACKAGE" ]]; then
+ FPM_INFO="$PACKAGE/fpm-info.sh"
+ else
+ FPM_INFO="${WORKSPACE}/backports/${PACKAGE_TYPE}-${PACKAGE}/fpm-info.sh"
+ fi
+ if [[ -e "$FPM_INFO" ]]; then
+ debug_echo "Loading fpm overrides from $FPM_INFO"
+ source "$FPM_INFO"
+ fi
+ for i in "${fpm_depends[@]}"; do
+ COMMAND_ARR+=('--depends' "$i")
+ done
+ COMMAND_ARR+=("${fpm_args[@]}")
+
COMMAND_ARR+=("$PACKAGE")
debug_echo -e "\n${COMMAND_ARR[@]}\n"
commit 988b1304d57f960f1006af52dc700026061a280e
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Aug 12 09:31:32 2015 -0400
6638: Dry up debug printfs.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 58c63ac..16699ba 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -59,6 +59,19 @@ while [ $# -gt 0 ]; do
shift
done
+STDOUT_IF_DEBUG=/dev/null
+STDERR_IF_DEBUG=/dev/null
+DASHQ_UNLESS_DEBUG=-q
+if [[ "$DEBUG" != 0 ]]; then
+ STDOUT_IF_DEBUG=/dev/stdout
+ STDERR_IF_DEBUG=/dev/stderr
+ DASHQ_UNLESS_DEBUG=
+fi
+
+debug_echo () {
+ echo "$@" >"$STDOUT_IF_DEBUG"
+}
+
declare -a PYTHON_BACKPORTS PYTHON3_BACKPORTS
PYTHON2_VERSION=2.7
@@ -183,10 +196,8 @@ if [ -z "$RUN_BUILD_PACKAGES_PATH" ] ; then
exit 1 # fail
fi
-if [[ "$DEBUG" != 0 ]]; then
- echo "$0 is running from $RUN_BUILD_PACKAGES_PATH"
- echo "Workspace is $WORKSPACE"
-fi
+debug_echo "$0 is running from $RUN_BUILD_PACKAGES_PATH"
+debug_echo "Workspace is $WORKSPACE"
format_last_commit_here() {
local format=$1; shift
@@ -216,11 +227,7 @@ handle_python_package () {
return
fi
# Make sure only to use sdist - that's the only format pip can deal with (sigh)
- if [[ "$DEBUG" != 0 ]]; then
- python setup.py sdist
- else
- python setup.py -q sdist
- fi
+ python setup.py $DASHQ_UNLESS_DEBUG sdist
}
handle_ruby_gem() {
@@ -234,12 +241,8 @@ handle_ruby_gem() {
find -maxdepth 1 -name "${gem_name}-*.gem" -delete
- if [[ "$DEBUG" != 0 ]]; then
- gem build "$gem_name.gemspec"
- else
- # -q appears to be broken in gem version 2.2.2
- gem build "$gem_name.gemspec" -q >/dev/null 2>&1
- fi
+ # -q appears to be broken in gem version 2.2.2
+ gem build "$gem_name.gemspec" $DASHQ_UNLESS_DEBUG >"$STDOUT_IF_DEBUG" 2>"$STDERR_IF_DEBUG"
fpm_build "$gem_name"-*.gem "" "Curoverse, Inc." gem "" \
--prefix "$FPM_GEM_PREFIX"
@@ -313,11 +316,7 @@ fpm_build () {
COMMAND_ARR+=("$PACKAGE")
- if [[ "$DEBUG" != 0 ]]; then
- echo
- echo "${COMMAND_ARR[@]}"
- echo
- fi
+ debug_echo -e "\n${COMMAND_ARR[@]}\n"
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
@@ -364,46 +363,32 @@ find -type d -name 'bin' |xargs -I {} find {} -type f |xargs -I {} chmod 755 {}
# gems and packages
umask 0022
-if [[ "$DEBUG" != 0 ]]; then
- echo "umask is" `umask`
-fi
+debug_echo "umask is" `umask`
if [[ ! -d "$WORKSPACE/packages/$TARGET" ]]; then
mkdir -p $WORKSPACE/packages/$TARGET
fi
# Perl packages
-if [[ "$DEBUG" != 0 ]]; then
- echo -e "\nPerl packages\n"
-fi
-
-if [[ "$DEBUG" != 0 ]]; then
- PERL_OUT=/dev/stdout
-else
- PERL_OUT=/dev/null
-fi
+debug_echo -e "\nPerl packages\n"
cd "$WORKSPACE/sdk/perl"
if [[ -e Makefile ]]; then
- make realclean >"$PERL_OUT"
+ make realclean >"$STDOUT_IF_DEBUG"
fi
find -maxdepth 1 \( -name 'MANIFEST*' -or -name "libarvados-perl*.$FORMAT" \) \
-delete
rm -rf install
-perl Makefile.PL INSTALL_BASE=install >"$PERL_OUT" && \
- make install INSTALLDIRS=perl >"$PERL_OUT" && \
+perl Makefile.PL INSTALL_BASE=install >"$STDOUT_IF_DEBUG" && \
+ make install INSTALLDIRS=perl >"$STDOUT_IF_DEBUG" && \
fpm_build install/lib/=/usr/share libarvados-perl \
"Curoverse, Inc." dir "$(version_from_git)" install/man/=/usr/share/man && \
mv libarvados-perl*.$FORMAT "$WORKSPACE/packages/$TARGET/"
# Ruby gems
-if [[ "$DEBUG" != 0 ]]; then
- echo
- echo "Ruby gems"
- echo
-fi
+debug_echo -e "\nRuby gems\n"
if type rvm-exec >/dev/null 2>&1; then
FPM_GEM_PREFIX=$(rvm-exec system gem environment gemdir)
@@ -418,11 +403,7 @@ cd "$WORKSPACE/sdk/cli"
handle_ruby_gem arvados-cli
# Python packages
-if [[ "$DEBUG" != 0 ]]; then
- echo
- echo "Python packages"
- echo
-fi
+debug_echo -e "\nPython packages\n"
cd "$WORKSPACE/sdk/pam"
handle_python_package
@@ -441,11 +422,7 @@ handle_python_package
if [[ ! -d "$WORKSPACE/src-build-dir" ]]; then
mkdir "$WORKSPACE/src-build-dir"
cd "$WORKSPACE"
- if [[ "$DEBUG" != 0 ]]; then
- git clone https://github.com/curoverse/arvados.git src-build-dir
- else
- git clone -q https://github.com/curoverse/arvados.git src-build-dir
- fi
+ git clone $DASHQ_UNLESS_DEBUG https://github.com/curoverse/arvados.git src-build-dir
fi
# Get the commit hash we're building against, from the working directory
@@ -455,17 +432,11 @@ MASTER_COMMIT_HASH=$(format_last_commit_here "%H")
# Make sure we check out that commit in the clean $WORKSPACE/src-build-dir directory
cd "$WORKSPACE/src-build-dir"
# just in case, check out master
-if [[ "$DEBUG" != 0 ]]; then
- git checkout master
- git pull
- # go into detached-head state
- git checkout "$MASTER_COMMIT_HASH"
-else
- git checkout -q master
- git pull -q
- # go into detached-head state
- git checkout -q "$MASTER_COMMIT_HASH"
-fi
+git checkout $DASHQ_UNLESS_DEBUG master
+git pull $DASHQ_UNLESS_DEBUG
+# go into detached-head state
+MASTER_COMMIT_HASH=$(format_last_commit_here "%H")
+git checkout $DASHQ_UNLESS_DEBUG "$MASTER_COMMIT_HASH"
echo "$MASTER_COMMIT_HASH" >git-commit.version
# Build arvados src deb package
@@ -476,11 +447,7 @@ fpm_build $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curovers
# clean up, check out master and step away from detached-head state
cd "$WORKSPACE/src-build-dir"
-if [[ "$DEBUG" != 0 ]]; then
- git checkout master
-else
- git checkout -q master
-fi
+git checkout $DASHQ_UNLESS_DEBUG master
# Keep
export GOPATH=$(mktemp -d)
@@ -610,11 +577,8 @@ if [[ ! -d "$WORKSPACE/services/api/tmp" ]]; then
mkdir $WORKSPACE/services/api/tmp
fi
-BUNDLE_OUTPUT=`bundle install --path vendor/bundle`
-if [[ "$DEBUG" != 0 ]]; then
- echo $BUNDLE_OUTPUT
-fi
+bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
/usr/bin/git rev-parse HEAD > git-commit.version
@@ -631,11 +595,7 @@ cd $WORKSPACE/packages/$TARGET
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
- if [[ "$DEBUG" != 0 ]]; then
- echo
- echo "${COMMAND_ARR[@]}"
- echo
- fi
+ debug_echo -e "\n${COMMAND_ARR[@]}\n"
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
@@ -645,11 +605,7 @@ fi
# Build the 'bare' package without vendor/bundle.
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
-if [[ "$DEBUG" != 0 ]]; then
- echo
- echo "${COMMAND_ARR[@]}"
- echo
-fi
+debug_echo -e "\n${COMMAND_ARR[@]}\n"
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
@@ -668,11 +624,7 @@ if [[ ! -d "$WORKSPACE/apps/workbench/tmp" ]]; then
mkdir $WORKSPACE/apps/workbench/tmp
fi
-BUNDLE_OUTPUT=`bundle install --path vendor/bundle`
-
-if [[ "$DEBUG" != 0 ]]; then
- echo $BUNDLE_OUTPUT
-fi
+bundle install --path vendor/bundle >"$STDOUT_IF_DEBUG"
/usr/bin/git rev-parse HEAD > git-commit.version
@@ -701,11 +653,7 @@ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
- if [[ "$DEBUG" != 0 ]]; then
- echo
- echo "${COMMAND_ARR[@]}"
- echo
- fi
+ debug_echo -e "\n${COMMAND_ARR[@]}\n"
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
@@ -716,11 +664,7 @@ fi
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
-if [[ "$DEBUG" != 0 ]]; then
- echo
- echo "${COMMAND_ARR[@]}"
- echo
-fi
+debug_echo -e "\n${COMMAND_ARR[@]}\n"
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
commit 4e76299b9c1d4132a1fb681d332be638fc922425
Merge: 61974cf 3d595c9
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Aug 12 23:06:15 2015 -0400
Merge branch '6934-pam' refs #6934
commit 3d595c9b4cda969d75715395a29057e1740568e3
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Aug 12 22:36:13 2015 -0400
6934: Build libpam-arvados only for targets that have libpam-python.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index b1ff4be..58c63ac 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -563,9 +563,11 @@ rm -rf "$WORKSPACE/sdk/python/build"
fpm_build $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --deb-recommends=git
# The PAM module
-cd $WORKSPACE/packages/$TARGET
-rm -rf "$WORKSPACE/sdk/pam/build"
-fpm_build $WORKSPACE/sdk/pam libpam-arvados 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/pam/arvados_pam.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=PAM module for authenticating shell logins using Arvados API tokens" --depends libpam-python
+if [[ $TARGET =~ debian|ubuntu ]]; then
+ cd $WORKSPACE/packages/$TARGET
+ rm -rf "$WORKSPACE/sdk/pam/build"
+ fpm_build $WORKSPACE/sdk/pam libpam-arvados 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/pam/arvados_pam.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=PAM module for authenticating shell logins using Arvados API tokens" --depends libpam-python
+fi
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
commit 69e7600c1d70c485325a3b2c9d818ac2e1e1084d
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Aug 12 16:19:06 2015 -0400
6934: Add libpam-python dependency on all targets.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index ad1494b..b1ff4be 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -565,7 +565,7 @@ fpm_build $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'C
# The PAM module
cd $WORKSPACE/packages/$TARGET
rm -rf "$WORKSPACE/sdk/pam/build"
-fpm_build $WORKSPACE/sdk/pam libpam-arvados 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/pam/arvados_pam.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=PAM module for authenticating shell logins using Arvados API tokens"
+fpm_build $WORKSPACE/sdk/pam libpam-arvados 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/pam/arvados_pam.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=PAM module for authenticating shell logins using Arvados API tokens" --depends libpam-python
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
commit ebaa21aa60ce8f177091dc347ef47ec7e90af3ae
Merge: c18ab40 61974cf
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Aug 12 13:57:06 2015 -0400
6934: Merge branch 'master' into 6934-pam
commit c18ab4005222f73e5e50684ed1f242d8988baec4
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Aug 10 10:41:00 2015 -0400
6934: Package libpam-arvados using Python recipe. Add sdk/pam to run-tests.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 1ceecee..9dcfd1f 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -424,15 +424,16 @@ if [[ "$DEBUG" != 0 ]]; then
echo
fi
-cd "$WORKSPACE"
+cd "$WORKSPACE/sdk/pam"
+handle_python_package
-cd sdk/python
+cd "$WORKSPACE/sdk/python"
handle_python_package
-cd ../../services/fuse
+cd "$WORKSPACE/services/fuse"
handle_python_package
-cd ../../services/nodemanager
+cd "$WORKSPACE/services/nodemanager"
handle_python_package
# Arvados-src
@@ -558,6 +559,11 @@ cd $WORKSPACE/packages/$TARGET
rm -rf "$WORKSPACE/sdk/python/build"
fpm_build $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --deb-recommends=git
+# The PAM module
+cd $WORKSPACE/packages/$TARGET
+rm -rf "$WORKSPACE/sdk/pam/build"
+fpm_build $WORKSPACE/sdk/pam libpam-arvados 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/pam/arvados_pam.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=PAM module for authenticating shell logins using Arvados API tokens"
+
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
@@ -588,18 +594,6 @@ for deppkg in "${PYTHON3_BACKPORTS[@]}"; do
fpm_build "$deppkg" "$outname" "" python3
done
-# libpam-arvados
-cd $WORKSPACE/sdk/pam
-PKG_VERSION=$(version_from_git)
-cd $WORKSPACE/packages/$TARGET
-
-if [[ "$FORMAT" == "deb" ]]; then
- fpm_build $WORKSPACE/sdk/pam/debian/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" "-d libpam-python" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/debian/arvados_pam=/etc/default/arvados_pam
-#else
- # FIXME enable and test once we have the centos pam.d file
- #fpm_build $WORKSPACE/sdk/pam/centos/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" "-d libpam-python" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/centos/arvados_pam=/etc/default/arvados_pam
-fi
-
# Build the API server package
cd "$WORKSPACE/services/api"
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 7759944..e191b44 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -63,6 +63,7 @@ services/keepstore
services/nodemanager
services/arv-git-httpd
sdk/cli
+sdk/pam
sdk/python
sdk/ruby
sdk/go/arvadosclient
@@ -390,6 +391,10 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
. "$VENVDIR/bin/activate"
+if (pip install setuptools | grep setuptools-0) || [ "$($VENVDIR/bin/easy_install --version | cut -d\ -f2 | cut -d. -f1)" -lt 18 ]; then
+ pip install --upgrade setuptools
+fi
+
# Note: this must be the last time we change PATH, otherwise rvm will
# whine a lot.
setup_ruby_environment
@@ -506,10 +511,10 @@ do_install() {
# the source dist package to avoid a pip bug.
# see https://arvados.org/issues/5766 for details.
- # Also need to install twice, because if it belives the package is
+ # Also need to install twice, because if it believes the package is
# already installed, pip it won't install it. So the first "pip
# install" ensures that the dependencies are met, the second "pip
- # install" ensures that we've actually install the local package
+ # install" ensures that we've actually installed the local package
# we just built.
cd "$WORKSPACE/$1" \
&& "${3}python" setup.py sdist rotate --keep=1 --match .tar.gz \
@@ -585,6 +590,7 @@ do_install sdk/cli cli
# keepproxy).
declare -a pythonstuff
pythonstuff=(
+ sdk/pam
sdk/python
services/fuse
services/nodemanager
commit 61974cf1393c51568293d19685ea3a4aafc4a770
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Aug 7 10:56:45 2015 -0400
Make sure that we build arvados-src package with the correct version of
the git tree, not latest.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 1ceecee..7178c59 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -447,19 +447,22 @@ if [[ ! -d "$WORKSPACE/src-build-dir" ]]; then
fi
fi
+# Get the commit hash we're building against, from the working directory
+cd "$WORKSPACE"
+MASTER_COMMIT_HASH=$(format_last_commit_here "%H")
+
+# Make sure we check out that commit in the clean $WORKSPACE/src-build-dir directory
cd "$WORKSPACE/src-build-dir"
# just in case, check out master
if [[ "$DEBUG" != 0 ]]; then
git checkout master
git pull
# go into detached-head state
- MASTER_COMMIT_HASH=$(format_last_commit_here "%H")
git checkout "$MASTER_COMMIT_HASH"
else
git checkout -q master
git pull -q
# go into detached-head state
- MASTER_COMMIT_HASH=$(format_last_commit_here "%H")
git checkout -q "$MASTER_COMMIT_HASH"
fi
echo "$MASTER_COMMIT_HASH" >git-commit.version
commit b8048416d62f1c380f425d0d2dc7ed0fbbd6fe5c
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Aug 4 22:00:51 2015 -0400
Fix logic bug in run-deploy: deploy api server first before gathering
the git revision from it to make sure the correct arvados/jobs docker
image is installed.
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 054b16f..0084d2f 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -138,6 +138,21 @@ function run_command() {
eval "$return_var=$ECODE"
}
+title "Updating API server"
+SUM_ECODE=0
+run_puppet $IDENTIFIER.arvadosapi.com ECODE
+SUM_ECODE=$(($SUM_ECODE + $ECODE))
+run_command $IDENTIFIER.arvadosapi.com ECODE "/usr/local/bin/arvados-api-server-upgrade.sh"
+SUM_ECODE=$(($SUM_ECODE + $ECODE))
+run_command $IDENTIFIER.arvadosapi.com ECODE "dpkg -L arvados-mailchimp-plugin 2>/dev/null && apt-get install arvados-mailchimp-plugin --reinstall || echo"
+SUM_ECODE=$(($SUM_ECODE + $ECODE))
+
+if [[ "$SUM_ECODE" != "0" ]]; then
+ title "ERROR: Updating API server FAILED"
+ EXITCODE=$(($EXITCODE + $SUM_ECODE))
+ exit $EXITCODE
+fi
+
title "Loading ARVADOS_API_HOST and ARVADOS_API_TOKEN"
if [[ -f "$HOME/.config/arvados/$IDENTIFIER.arvadosapi.com.conf" ]]; then
. $HOME/.config/arvados/$IDENTIFIER.arvadosapi.com.conf
@@ -192,21 +207,6 @@ title "Gathering list of shell and Keep nodes"
SHELL_NODES=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv virtual_machine list |jq .items[].hostname -r`
KEEP_NODES=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv keep_service list |jq .items[].service_host -r`
-title "Updating API server"
-SUM_ECODE=0
-run_puppet $IDENTIFIER.arvadosapi.com ECODE
-SUM_ECODE=$(($SUM_ECODE + $ECODE))
-run_command $IDENTIFIER.arvadosapi.com ECODE "/usr/local/bin/arvados-api-server-upgrade.sh"
-SUM_ECODE=$(($SUM_ECODE + $ECODE))
-run_command $IDENTIFIER.arvadosapi.com ECODE "dpkg -L arvados-mailchimp-plugin 2>/dev/null && apt-get install arvados-mailchimp-plugin --reinstall || echo"
-SUM_ECODE=$(($SUM_ECODE + $ECODE))
-
-if [[ "$SUM_ECODE" != "0" ]]; then
- title "ERROR: Updating API server FAILED"
- EXITCODE=$(($EXITCODE + $SUM_ECODE))
- exit $EXITCODE
-fi
-
title "Updating workbench"
SUM_ECODE=0
if [[ `host workbench.$ARVADOS_API_HOST` != `host $ARVADOS_API_HOST` ]]; then
commit 7034b1b4089ade8f362f592ec209fcfd45b87ec3
Merge: d44585b eafd033
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Jul 30 15:38:03 2015 -0400
Merge branch '6592-test-perl' refs #6592
commit eafd0331da34dd8d61f9f3afd22f7976615b2781
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Jul 30 11:44:44 2015 -0400
6592: Build and install the Perl SDK so sdk/cli tests can use it.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 96c71b9..7759944 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -83,6 +83,7 @@ VENVDIR=
VENV3DIR=
PYTHONPATH=
GEMHOME=
+PERLINSTALLBASE=
COLUMNS=80
@@ -92,7 +93,7 @@ skip_install=
declare -A leave_temp
clear_temp() {
leaving=""
- for var in VENVDIR VENV3DIR GOPATH GITDIR GEMHOME
+ for var in VENVDIR VENV3DIR GOPATH GITDIR GEMHOME PERLINSTALLBASE
do
if [[ -z "${leave_temp[$var]}" ]]
then
@@ -166,6 +167,14 @@ sanity_checks() {
echo -n 'nginx: '
PATH="$PATH:/sbin:/usr/sbin:/usr/local/sbin" nginx -v \
|| fatal "No nginx. Try: apt-get install nginx"
+ echo -n 'perl: '
+ perl -v | grep version \
+ || fatal "No perl. Try: apt-get install perl"
+ for mod in ExtUtils::MakeMaker JSON LWP Net::SSL; do
+ echo -n "perl $mod: "
+ perl -e "use $mod; print \"\$$mod::VERSION\\n\"" \
+ || fatal "No $mod. Try: apt-get install perl-modules libcrypt-ssleay-perl libjson-perl"
+ done
}
rotate_logfile() {
@@ -211,6 +220,7 @@ do
leave_temp[VENV3DIR]=1
leave_temp[GOPATH]=1
leave_temp[GEMHOME]=1
+ leave_temp[PERLINSTALLBASE]=1
;;
--retry)
retry=1
@@ -283,7 +293,7 @@ cd "$WORKSPACE"
find -name '*.pyc' -delete
# Set up temporary install dirs (unless existing dirs were supplied)
-for tmpdir in VENVDIR VENV3DIR GOPATH GEMHOME
+for tmpdir in VENVDIR VENV3DIR GOPATH GEMHOME PERLINSTALLBASE
do
if [[ -n "${!tmpdir}" ]]; then
leave_temp[$tmpdir]=1
@@ -369,6 +379,9 @@ gem_uninstall_if_exists() {
fi
}
+export PERLINSTALLBASE
+export PERLLIB="$PERLINSTALLBASE/lib/perl5:${PERLLIB:+$PERLLIB}"
+
export GOPATH
mkdir -p "$GOPATH/src/git.curoverse.com"
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
@@ -549,6 +562,13 @@ install_ruby_sdk() {
}
do_install sdk/ruby ruby_sdk
+install_perl_sdk() {
+ cd "$WORKSPACE/sdk/perl" \
+ && perl Makefile.PL INSTALL_BASE="$PERLINSTALLBASE" \
+ && make install INSTALLDIRS=perl
+}
+do_install sdk/perl perl_sdk
+
install_cli() {
with_test_gemset gem_uninstall_if_exists arvados-cli \
&& cd "$WORKSPACE/sdk/cli" \
commit d44585be5835346f21df58aae29fd0c9a3f3d63e
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jul 27 10:50:32 2015 -0400
Rebuild Python packages before building their distro packages.
Without this, built scripts lay around between builds, with shebang
lines for the first distro they were built for. This can lead to
problems like Debian packages getting shebang lines for Red Hat
systems. See, e.g.,
<https://ci.curoverse.com/job/deploy-to-staging/1250/console>.
No issue #.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 4150a2b..1ceecee 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -555,20 +555,24 @@ fpm_build $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
cd $WORKSPACE/packages/$TARGET
+rm -rf "$WORKSPACE/sdk/python/build"
fpm_build $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --deb-recommends=git
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
cd $WORKSPACE/packages/$TARGET
+rm -rf "$WORKSPACE/services/fuse/build"
fpm_build $WORKSPACE/services/fuse "${PYTHON2_PKG_PREFIX}-arvados-fuse" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
# The node manager
cd $WORKSPACE/packages/$TARGET
+rm -rf "$WORKSPACE/services/nodemanager/build"
fpm_build $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
# The Docker image cleaner
cd $WORKSPACE/packages/$TARGET
+rm -rf "$WORKSPACE/services/dockercleaner/build"
fpm_build $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curoverse, Inc.' 'python3' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/dockercleaner/arvados_docker_cleaner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Docker image cleaner"
# A few dependencies
commit 310b52541abd4625151df391e1194ec2e7c76448
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Jul 27 09:19:46 2015 -0400
Fix parameter ordering for minor change in cwltool command line semantics. no issue #
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
index e76ad55..e148171 100755
--- a/jenkins/run-cwl-tests.sh
+++ b/jenkins/run-cwl-tests.sh
@@ -201,7 +201,7 @@ else
git config user.name "Curoverse build bot"
cd ..
fi
-python -mcwltool specification/cwlsite.cwl specification/cwlsite-job.json --outdir=$PWD/common-workflow-language.github.io
+python -mcwltool --outdir=$PWD/common-workflow-language.github.io specification/cwlsite.cwl specification/cwlsite-job.json
cd common-workflow-language.github.io
git add --all
git diff-index --quiet HEAD || git commit -m"Build bot"
commit cc2a6672fef9bb825b2fa4b8482f4306cae491a1
Merge: 310db12 ae548f2
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jul 27 08:55:43 2015 -0400
Merge branch '6691-trusty-packages-wip'
Refs #6691. Closes #6742.
commit ae548f25303fad421a20ebcc34af39a6d0ee0db8
Author: Brett Smith <brett at curoverse.com>
Date: Thu Jul 23 17:10:54 2015 -0400
6691: Build and ship packages for Ubuntu 14.04.
diff --git a/jenkins/dockerfiles/Makefile b/jenkins/dockerfiles/Makefile
index 3d6eec6..70fbf28 100644
--- a/jenkins/dockerfiles/Makefile
+++ b/jenkins/dockerfiles/Makefile
@@ -1,4 +1,4 @@
-all: centos6/generated debian7/generated debian8/generated ubuntu1204/generated
+all: centos6/generated debian7/generated debian8/generated ubuntu1204/generated ubuntu1404/generated
centos6/generated: common-generated-all
test -d centos6/generated || mkdir centos6/generated
@@ -16,6 +16,10 @@ ubuntu1204/generated: common-generated-all
test -d ubuntu1204/generated || mkdir ubuntu1204/generated
cp -rlt ubuntu1204/generated common-generated/*
+ubuntu1404/generated: common-generated-all
+ test -d ubuntu1404/generated || mkdir ubuntu1404/generated
+ cp -rlt ubuntu1404/generated common-generated/*
+
common-generated-all: common-generated/golang-amd64.tar.gz
common-generated/golang-amd64.tar.gz: common-generated
diff --git a/jenkins/dockerfiles/ubuntu1404/Dockerfile b/jenkins/dockerfiles/ubuntu1404/Dockerfile
new file mode 100644
index 0000000..4b56bce
--- /dev/null
+++ b/jenkins/dockerfiles/ubuntu1404/Dockerfile
@@ -0,0 +1,19 @@
+FROM ubuntu:trusty
+MAINTAINER Brett Smith <brett at curoverse.com>
+
+# Install dependencies and set up system.
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev
+
+# Install RVM
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+ curl -L https://get.rvm.io | bash -s stable && \
+ /usr/local/rvm/bin/rvm install 2.1 && \
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
+ /usr/local/rvm/bin/rvm-exec default gem install fpm
+
+# Install golang binary
+ADD generated/golang-amd64.tar.gz /usr/local/
+RUN ln -s /usr/local/go/bin/go /usr/local/bin/
+
+ENV WORKSPACE /arvados
+CMD ["/usr/local/rvm/bin/rvm-exec", "default", "bash", "/jenkins/run-build-packages.sh", "--target", "ubuntu1404"]
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index b8e58d7..4150a2b 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -104,6 +104,17 @@ case "$TARGET" in
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
+ ubuntu1404)
+ FORMAT=deb
+ PYTHON2_PACKAGE=python$PYTHON2_VERSION
+ PYTHON2_PKG_PREFIX=python
+ PYTHON3_PACKAGE=python$PYTHON3_VERSION
+ PYTHON3_PKG_PREFIX=python3
+ PYTHON_BACKPORTS=(pyasn1==0.1.7 pyasn1-modules==0.0.5 llfuse ciso8601 \
+ google-api-python-client six uritemplate oauth2client httplib2 \
+ rsa apache-libcloud pycurl backports.ssl_match_hostname)
+ PYTHON3_BACKPORTS=(docker-py requests websocket-client)
+ ;;
centos6)
FORMAT=rpm
PYTHON2_PACKAGE=$(rpm -qf "$(which python$PYTHON2_VERSION)" --queryformat '%{NAME}\n')
diff --git a/jenkins/run_upload_packages.py b/jenkins/run_upload_packages.py
index fb4defd..b0a9706 100755
--- a/jenkins/run_upload_packages.py
+++ b/jenkins/run_upload_packages.py
@@ -153,6 +153,7 @@ rm "$@"
'debian7': 'wheezy',
'debian8': 'jessie',
'ubuntu1204': 'precise',
+ 'ubuntu1404': 'trusty',
}
def post_uploads(self, paths):
@@ -191,7 +192,7 @@ PACKAGE_SUITES = {
'services/nodemanager/dist/*.tar.gz'),
'gems': _define_suite(GemPackageSuite, 'sdk/ruby/*.gem', 'sdk/cli/*.gem'),
}
-for target in ['debian7', 'debian8', 'ubuntu1204']:
+for target in ['debian7', 'debian8', 'ubuntu1204', 'ubuntu1404']:
PACKAGE_SUITES[target] = _define_suite(
DebianPackageSuite, os.path.join('packages', target, '*.deb'),
target=target)
commit 310db12be5e8825d11e91a0abc30b63548ec4757
Merge: c0e1dbb 4886a88
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jul 24 12:08:09 2015 -0400
Merge branch '6682-use-docker-build-cmd-wip'
Refs #6682. Closes #6741.
commit 4886a88b72e5faef32780bc2fdd60ec5c2d7f16d
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jul 22 22:21:59 2015 -0400
6682: Add script to orchestrate builds for multiple targets.
Now that we can build packages for multiple targets, it's nice to have
a one-shot script to build them all. This does that.
diff --git a/jenkins/run-build-packages-all-targets.sh b/jenkins/run-build-packages-all-targets.sh
new file mode 100755
index 0000000..a1052a9
--- /dev/null
+++ b/jenkins/run-build-packages-all-targets.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+
+# Orchestrate run-build-packages.sh for every target.
+
+set -e
+
+FINAL_EXITCODE=0
+JENKINS_DIR=$(dirname "$(readlink -e "$0")")
+
+run_docker() {
+ local tag=$1; shift
+ if docker run -v "$JENKINS_DIR:/jenkins" -v "$WORKSPACE:/arvados" \
+ --env ARVADOS_DEBUG=1 "arvados/build:$tag"; then
+ # Success - nothing more to do.
+ true
+ else
+ FINAL_EXITCODE=$?
+ echo "ERROR: $tag build failed with exit status $FINAL_EXITCODE." >&2
+ fi
+}
+
+# In case it's needed, build the containers. This costs just a few
+# seconds when the containers already exist, so it's not a big deal to
+# do it on each run.
+cd "$JENKINS_DIR/dockerfiles"
+time ./build-all-build-containers.sh
+
+for dockerfile_path in $(find -name Dockerfile); do
+ run_docker "$(basename $(dirname "$dockerfile_path"))"
+done
+
+exit $FINAL_EXITCODE
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index fc12cb4..b8e58d7 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -21,7 +21,7 @@ WORKSPACE=path Path to the Arvados source tree to build packages from
EOF
EXITCODE=0
-DEBUG=0
+DEBUG=${ARVADOS_DEBUG:-0}
BUILD_BUNDLE_PACKAGES=0
TARGET=debian7
commit c0e1dbb244f0cf8d838ebd9296e797a0a2f23fdf
Author: Brett Smith <brett at curoverse.com>
Date: Thu Jul 23 16:59:11 2015 -0400
6497: Bugfix distro names for DebianPackageSuites.
freight is expecting a codename like wheezy or jessie, but we've been
using our internal distro+version names there. This fixes that.
Nico gave a thumbs up to this patch. Refs #6497.
diff --git a/jenkins/run_upload_packages.py b/jenkins/run_upload_packages.py
index 3eac15b..fb4defd 100755
--- a/jenkins/run_upload_packages.py
+++ b/jenkins/run_upload_packages.py
@@ -144,15 +144,21 @@ class DistroPackageSuite(PackageSuite):
class DebianPackageSuite(DistroPackageSuite):
FREIGHT_SCRIPT = """
cd "$1"; shift
-TARGET=$1; shift
-freight add "$@" "apt/$TARGET"
+DISTNAME=$1; shift
+freight add "$@" "apt/$DISTNAME"
freight cache
rm "$@"
"""
+ TARGET_DISTNAMES = {
+ 'debian7': 'wheezy',
+ 'debian8': 'jessie',
+ 'ubuntu1204': 'precise',
+ }
def post_uploads(self, paths):
self._run_script(self.FREIGHT_SCRIPT, self.REMOTE_DEST_DIR,
- self.target, *self._paths_basenames(paths))
+ self.TARGET_DISTNAMES[self.target],
+ *self._paths_basenames(paths))
class RedHatPackageSuite(DistroPackageSuite):
commit 34ef97e556eff9a315160f0317a6652154d749f0
Author: Nico Cesar <nico at curoverse.com>
Date: Thu Jul 23 12:55:13 2015 -0400
one of the scopes wasnt working
closes #6700
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 43b17b8..054b16f 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -19,6 +19,7 @@ function usage {
echo >&2 ' \"GET /arvados/v1/groups\",'
echo >&2 ' \"GET /arvados/v1/links\",'
echo >&2 ' \"GET /arvados/v1/groups/\",'
+ echo >&2 ' \"GET /arvados/v1/collections\",'
echo >&2 ' \"POST /arvados/v1/collections\",'
echo >&2 ' \"POST /arvados/v1/links\"]"'
echo >&2
commit d8d561588362cf4c2e100949b39eb45661471535
Author: Nico Cesar <nico at curoverse.com>
Date: Thu Jul 23 10:53:41 2015 -0400
scopes in usage() are wrong.
refs #6700
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 1d23b4e..43b17b8 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -15,11 +15,10 @@ function usage {
echo >&2 "Note: this script requires an arvados token created with these permissions:"
echo >&2 ' arv api_client_authorization create_system_auth \'
echo >&2 ' --scopes "[\"GET /arvados/v1/virtual_machines\",'
- echo >&2 ' \"GET /arvados/v1/keep_services\",'
+ echo >&2 ' \"GET /arvados/v1/keep_services/\",'
echo >&2 ' \"GET /arvados/v1/groups\",'
echo >&2 ' \"GET /arvados/v1/links\",'
echo >&2 ' \"GET /arvados/v1/groups/\",'
- echo >&2 ' \"GET /arvados/v1/keep_services/accessible\",'
echo >&2 ' \"POST /arvados/v1/collections\",'
echo >&2 ' \"POST /arvados/v1/links\"]"'
echo >&2
commit f32deb7586783a24dd419329975f1d9429728d29
Author: Brett Smith <brett at curoverse.com>
Date: Thu Jul 23 10:07:54 2015 -0400
6750: arvados-api-server-upgrade only installs omniauth.rb if it exists.
New installs configure the SSO server via application.yml. Refs #6750.
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
index ace07c7..5160c7a 100755
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -39,7 +39,9 @@ echo "Copying files from $CONFIG_PATH"
cp -f $CONFIG_PATH/database.yml $RELEASE_PATH/config/database.yml
cp -f $RELEASE_PATH/config/environments/production.rb.example $RELEASE_PATH/config/environments/production.rb
cp -f $CONFIG_PATH/application.yml $RELEASE_PATH/config/application.yml
-cp -f $CONFIG_PATH/omniauth.rb $RELEASE_PATH/config/initializers/omniauth.rb
+if [ -e $CONFIG_PATH/omniauth.rb ]; then
+ cp -f $CONFIG_PATH/omniauth.rb $RELEASE_PATH/config/initializers/omniauth.rb
+fi
echo "Done."
# Before we do anything else, make sure some directories and files are in place
commit c31cdc09ae6aefbbf6f9e14b3c6fdf910a729b29
Merge: a5c1b7e 5cf1533
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jul 22 16:38:33 2015 -0400
Merge branch '6497-separate-package-upload-wip'
Closes #6497, #6533, #6498.
commit 5cf15331acb6250da715ee9c41a2a0db7986917c
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jul 22 16:38:23 2015 -0400
6497: Separate package build and uploader scripts.
This lets us avoid redundant uploads as we build for multiple targets,
and lets us better customize the upload process per target.
run-build-packages had code to avoid doing an upload when there was no
new package to build. run_upload_packages uses a timestamp file to
achieve a similar result.
Other changes to run-build-packages to support this:
* Better detection of when Python and Ruby packages have already been
built.
* Refactor Gem building into a function to fix various bugs (wrong
version checked for existing arvados-cli; distro packages not
installed in the proper directory).
* Refactor git log querying to reduce code duplication.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 42b42da..fc12cb4 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -2,19 +2,13 @@
read -rd "\000" helpmessage <<EOF
-$(basename $0): Build Arvados packages and (optionally) upload them.
+$(basename $0): Build Arvados packages
Syntax:
WORKSPACE=/path/to/arvados $(basename $0) [options]
Options:
---upload
- Upload packages (default: false)
---scp-user USERNAME
- scp user for repository server (only required when --upload is specified)
---scp-host HOSTNAME
- scp host for repository server (only required when --upload is specified)
--build-bundle-packages (default: false)
Build api server and workbench packages with vendor/bundle included
--debug
@@ -27,15 +21,12 @@ WORKSPACE=path Path to the Arvados source tree to build packages from
EOF
EXITCODE=0
-CALL_FREIGHT=0
-
DEBUG=0
-UPLOAD=0
BUILD_BUNDLE_PACKAGES=0
TARGET=debian7
PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,upload,scp-user:,scp-host:,build-bundle-packages,debug,target: \
+ help,build-bundle-packages,debug,target: \
-- "" "$@")
if [ $? -ne 0 ]; then
exit 1
@@ -49,21 +40,12 @@ while [ $# -gt 0 ]; do
echo >&2
exit 1
;;
- --scp-user)
- SCPUSER="$2"; shift
- ;;
- --scp-host)
- SCPHOST="$2"; shift
- ;;
--target)
TARGET="$2"; shift
;;
--debug)
DEBUG=1
;;
- --upload)
- UPLOAD=1
- ;;
--build-bundle-packages)
BUILD_BUNDLE_PACKAGES=1
;;
@@ -77,15 +59,6 @@ while [ $# -gt 0 ]; do
shift
done
-# Sanity checks
-if [[ "$UPLOAD" != '0' && ("$SCPUSER" == '' || "$SCPHOST" == '') ]]; then
- echo >&2 "$helpmessage"
- echo >&2
- echo >&2 "Error: please specify --scp-user and --scp-host if --upload is set"
- echo >&2
- exit 1
-fi
-
declare -a PYTHON_BACKPORTS PYTHON3_BACKPORTS
PYTHON2_VERSION=2.7
@@ -94,9 +67,6 @@ PYTHON3_VERSION=$(python3 -c 'import sys; print("{v.major}.{v.minor}".format(v=s
case "$TARGET" in
debian7)
FORMAT=deb
- FPM_OUTDIR=tmp
- REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rm -f *deb'
-
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON2_PKG_PREFIX=python
PYTHON3_PACKAGE=python$PYTHON3_VERSION
@@ -110,9 +80,6 @@ case "$TARGET" in
;;
debian8)
FORMAT=deb
- FPM_OUTDIR=tmp
- REPO_UPDATE_CMD='freight add *deb apt/jessie && freight cache && rm -f *deb'
-
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON2_PKG_PREFIX=python
PYTHON3_PACKAGE=python$PYTHON3_VERSION
@@ -126,9 +93,6 @@ case "$TARGET" in
;;
ubuntu1204)
FORMAT=deb
- FPM_OUTDIR=tmp
- REPO_UPDATE_CMD='freight add *deb apt/precise && freight cache && rm -f *deb'
-
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON2_PKG_PREFIX=python
PYTHON3_PACKAGE=python$PYTHON3_VERSION
@@ -142,9 +106,6 @@ case "$TARGET" in
;;
centos6)
FORMAT=rpm
- FPM_OUTDIR=rpm
- REPO_UPDATE_CMD='mv *rpm /var/www/rpm.arvados.org/CentOS/6/os/x86_64/ && createrepo /var/www/rpm.arvados.org/CentOS/6/os/x86_64/'
-
PYTHON2_PACKAGE=$(rpm -qf "$(which python$PYTHON2_VERSION)" --queryformat '%{NAME}\n')
PYTHON2_PKG_PREFIX=$PYTHON2_PACKAGE
PYTHON3_PACKAGE=$(rpm -qf "$(which python$PYTHON3_VERSION)" --queryformat '%{NAME}\n')
@@ -216,45 +177,66 @@ if [[ "$DEBUG" != 0 ]]; then
echo "Workspace is $WORKSPACE"
fi
+format_last_commit_here() {
+ local format=$1; shift
+ TZ=UTC git log -n1 --first-parent "--format=format:$format" .
+}
+
version_from_git() {
# Generates a version number from the git log for the current working
# directory, and writes it to stdout.
local git_ts git_hash
- declare $(TZ=UTC git log -n1 --first-parent \
- --format=format:"git_ts=%ct git_hash=%h" .)
+ declare $(format_last_commit_here "git_ts=%ct git_hash=%h")
echo "0.1.$(date -ud "@$git_ts" +%Y%m%d%H%M%S).$git_hash"
}
+nohash_version_from_git() {
+ version_from_git | cut -d. -f1-3
+}
+
timestamp_from_git() {
- # Generates a version number from the git log for the current working
- # directory, and writes it to stdout.
- local git_ts git_hash
- declare $(TZ=UTC git log -n1 --first-parent \
- --format=format:"git_ts=%ct git_hash=%h" .)
- echo "$git_ts"
+ format_last_commit_here "%ct"
}
handle_python_package () {
# This function assumes the current working directory is the python package directory
- if [[ "$UPLOAD" != 0 ]]; then
- # Make sure only to use sdist - that's the only format pip can deal with (sigh)
- if [[ "$DEBUG" != 0 ]]; then
- python setup.py sdist upload
- else
- python setup.py -q sdist upload
- fi
+ if [ -n "$(find dist -name "*-$(nohash_version_from_git).tar.gz" -print -quit)" ]; then
+ # This package doesn't need rebuilding.
+ return
+ fi
+ # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ if [[ "$DEBUG" != 0 ]]; then
+ python setup.py sdist
else
- # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ python setup.py -q sdist
+ fi
+}
+
+handle_ruby_gem() {
+ local gem_name=$1; shift
+ local gem_version=$(nohash_version_from_git)
+
+ if [ -e "${gem_name}-${gem_version}.gem" ]; then
+ # This gem doesn't need rebuilding.
+ return
+ fi
+
+ find -maxdepth 1 -name "${gem_name}-*.gem" -delete
+
if [[ "$DEBUG" != 0 ]]; then
- python setup.py sdist
+ gem build "$gem_name.gemspec"
else
- python setup.py -q sdist
+ # -q appears to be broken in gem version 2.2.2
+ gem build "$gem_name.gemspec" -q >/dev/null 2>&1
fi
- fi
+
+ fpm_build "$gem_name"-*.gem "" "Curoverse, Inc." gem "" \
+ --prefix "$FPM_GEM_PREFIX"
+ mv -t "$WORKSPACE/packages/$TARGET/" "$gem_name"*."$FORMAT"
}
# Build packages for everything
-fpm_build_and_scp () {
+fpm_build () {
# The package source. Depending on the source type, this can be a
# path, or the name of the package in an upstream repository (e.g.,
# pip).
@@ -329,11 +311,11 @@ fpm_build_and_scp () {
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
- fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
+ fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
}
-# verify build results and scp packages, if needed
-fpm_verify_and_scp () {
+# verify build results
+fpm_verify () {
FPM_EXIT_CODE=$1
shift
FPM_RESULTS=$@
@@ -349,19 +331,10 @@ fpm_verify_and_scp () {
echo
echo $FPM_RESULTS
echo
- else
- if [[ ! $FPM_RESULTS =~ "File already exists" ]]; then
- if [[ "$FPM_EXIT_CODE" != "0" ]]; then
- echo "Error building package for $1:\n $FPM_RESULTS"
- else
- if [[ "$UPLOAD" != 0 ]]; then
- scp -P2222 $FPM_PACKAGE_NAME $SCPUSER@$SCPHOST:$FPM_OUTDIR/
- CALL_FREIGHT=1
- fi
- fi
- else
- echo "Package $FPM_PACKAGE_NAME exists, not rebuilding"
- fi
+ elif [[ "$FPM_RESULTS" =~ "File already exists" ]]; then
+ echo "Package $FPM_PACKAGE_NAME exists, not rebuilding"
+ elif [[ 0 -ne "$FPM_EXIT_CODE" ]]; then
+ echo "Error building package for $1:\n $FPM_RESULTS"
fi
}
@@ -410,7 +383,7 @@ rm -rf install
perl Makefile.PL INSTALL_BASE=install >"$PERL_OUT" && \
make install INSTALLDIRS=perl >"$PERL_OUT" && \
- fpm_build_and_scp install/lib/=/usr/share libarvados-perl \
+ fpm_build install/lib/=/usr/share libarvados-perl \
"Curoverse, Inc." dir "$(version_from_git)" install/man/=/usr/share/man && \
mv libarvados-perl*.$FORMAT "$WORKSPACE/packages/$TARGET/"
@@ -427,64 +400,11 @@ else
FPM_GEM_PREFIX=$(gem environment gemdir)
fi
-cd "$WORKSPACE"
-cd sdk/ruby
-
-ARVADOS_GEM_EPOCH=`git log -n1 --first-parent --format=%ct`
-ARVADOS_GEM_DATE=`date --utc --date="@${ARVADOS_GEM_EPOCH}" +%Y%m%d%H%M%S`
-ARVADOS_GEM_VERSION="0.1.${ARVADOS_GEM_DATE}"
-
-# see if this gem needs building/uploading
-gem search arvados -r -a |grep -q $ARVADOS_GEM_VERSION
-
-if [[ "$?" != "0" ]]; then
- # clean up old packages
- find -maxdepth 1 \( -name 'arvados-*.gem' -or -name 'rubygem-arvados_*.deb' -or -name 'rubygem-arvados_*.rpm' \) \
- -delete
-
- if [[ "$DEBUG" != 0 ]]; then
- gem build arvados.gemspec
- else
- # -q appears to be broken in gem version 2.2.2
- gem build arvados.gemspec -q >/dev/null 2>&1
- fi
-
- if [[ "$UPLOAD" != 0 ]]; then
- # publish new gem
- gem push arvados-*gem
- fi
-
- fpm_build_and_scp arvados-*.gem "" "Curoverse, Inc." gem "" \
- --prefix "$FPM_GEM_PREFIX"
-fi
-
-# Build arvados-cli GEM
-cd "$WORKSPACE"
-cd sdk/cli
-
-ARVADOS_CLI_GEM_EPOCH=`git log -n1 --first-parent --format=%ct`
-ARVADOS_CLI_GEM_DATE=`date --utc --date="@${ARVADOS_CLI_GEM_EPOCH}" +%Y%m%d%H%M%S`
-ARVADOS_CLI_GEM_VERSION="0.1.${ARVADOS_CLI_GEM_DATE}"
-
-# see if this gem needs building/uploading
-gem search arvados-cli -r -a |grep -q $ARVADOS_GEM_VERSION
+cd "$WORKSPACE/sdk/ruby"
+handle_ruby_gem arvados
-if [[ "$?" != "0" ]]; then
- # clean up old gems
- rm -f arvados-cli*gem
-
- if [[ "$DEBUG" != 0 ]]; then
- gem build arvados-cli.gemspec
- else
- # -q appears to be broken in gem version 2.2.2
- gem build arvados-cli.gemspec -q >/dev/null
- fi
-
- if [[ "$UPLOAD" != 0 ]]; then
- # publish new gem
- gem push arvados-cli*gem
- fi
-fi
+cd "$WORKSPACE/sdk/cli"
+handle_ruby_gem arvados-cli
# Python packages
if [[ "$DEBUG" != 0 ]]; then
@@ -522,21 +442,22 @@ if [[ "$DEBUG" != 0 ]]; then
git checkout master
git pull
# go into detached-head state
- git checkout `git log --format=format:%h -n1 --first-parent .`
+ MASTER_COMMIT_HASH=$(format_last_commit_here "%H")
+ git checkout "$MASTER_COMMIT_HASH"
else
git checkout -q master
git pull -q
# go into detached-head state
- git checkout -q `git log --format=format:%h -n1 --first-parent .`
+ MASTER_COMMIT_HASH=$(format_last_commit_here "%H")
+ git checkout -q "$MASTER_COMMIT_HASH"
fi
-
-git log --format=format:%H -n1 --first-parent . > git-commit.version
+echo "$MASTER_COMMIT_HASH" >git-commit.version
# Build arvados src deb package
cd "$WORKSPACE"
PKG_VERSION=$(version_from_git)
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
+fpm_build $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
# clean up, check out master and step away from detached-head state
cd "$WORKSPACE/src-build-dir"
@@ -556,7 +477,7 @@ cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keepstore"
PKG_VERSION=$(version_from_git)
go get "git.curoverse.com/arvados.git/services/keepstore"
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
+fpm_build $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
# Get GO SDK version
cd "$GOPATH/src/git.curoverse.com/arvados.git/sdk/go"
@@ -576,7 +497,7 @@ fi
go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
+fpm_build $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
# datamanager
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/datamanager"
@@ -591,7 +512,7 @@ fi
go get "git.curoverse.com/arvados.git/services/datamanager"
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $GOPATH/bin/datamanager=/usr/bin/arvados-data-manager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
+fpm_build $GOPATH/bin/datamanager=/usr/bin/arvados-data-manager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
# arv-git-httpd
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/arv-git-httpd"
@@ -606,14 +527,14 @@ fi
go get "git.curoverse.com/arvados.git/services/arv-git-httpd"
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $GOPATH/bin/arv-git-httpd=/usr/bin/arvados-git-httpd arvados-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
+fpm_build $GOPATH/bin/arv-git-httpd=/usr/bin/arvados-git-httpd arvados-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
# crunchstat
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
PKG_VERSION=$(version_from_git)
go get "git.curoverse.com/arvados.git/services/crunchstat"
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
+fpm_build $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
# The Python SDK
# Please resist the temptation to add --no-python-fix-name to the fpm call here
@@ -623,33 +544,33 @@ fpm_build_and_scp $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curover
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --deb-recommends=git
+fpm_build $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --deb-recommends=git
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $WORKSPACE/services/fuse "${PYTHON2_PKG_PREFIX}-arvados-fuse" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
+fpm_build $WORKSPACE/services/fuse "${PYTHON2_PKG_PREFIX}-arvados-fuse" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
# The node manager
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
+fpm_build $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
# The Docker image cleaner
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curoverse, Inc.' 'python3' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/dockercleaner/arvados_docker_cleaner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Docker image cleaner"
+fpm_build $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curoverse, Inc.' 'python3' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/dockercleaner/arvados_docker_cleaner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Docker image cleaner"
# A few dependencies
for deppkg in "${PYTHON_BACKPORTS[@]}"; do
outname=$(echo "$deppkg" | sed -e 's/^python-//' -e 's/[<=>].*//' -e 's/_/-/g' -e "s/^/${PYTHON2_PKG_PREFIX}-/")
- fpm_build_and_scp "$deppkg" "$outname"
+ fpm_build "$deppkg" "$outname"
done
# Python 3 dependencies
for deppkg in "${PYTHON3_BACKPORTS[@]}"; do
outname=$(echo "$deppkg" | sed -e 's/^python-//' -e 's/[<=>].*//' -e 's/_/-/g' -e "s/^/${PYTHON3_PKG_PREFIX}-/")
# The empty string is the vendor argument: these aren't Curoverse software.
- fpm_build_and_scp "$deppkg" "$outname" "" python3
+ fpm_build "$deppkg" "$outname" "" python3
done
# libpam-arvados
@@ -658,10 +579,10 @@ PKG_VERSION=$(version_from_git)
cd $WORKSPACE/packages/$TARGET
if [[ "$FORMAT" == "deb" ]]; then
- fpm_build_and_scp $WORKSPACE/sdk/pam/debian/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" "-d libpam-python" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/debian/arvados_pam=/etc/default/arvados_pam
+ fpm_build $WORKSPACE/sdk/pam/debian/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" "-d libpam-python" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/debian/arvados_pam=/etc/default/arvados_pam
#else
# FIXME enable and test once we have the centos pam.d file
- #fpm_build_and_scp $WORKSPACE/sdk/pam/centos/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" "-d libpam-python" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/centos/arvados_pam=/etc/default/arvados_pam
+ #fpm_build $WORKSPACE/sdk/pam/centos/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" "-d libpam-python" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/centos/arvados_pam=/etc/default/arvados_pam
fi
# Build the API server package
@@ -704,7 +625,7 @@ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
- fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
+ fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
fi
# Build the 'bare' package without vendor/bundle.
@@ -718,7 +639,7 @@ fi
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
-fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
+fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
# API server package build done
@@ -774,7 +695,7 @@ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
- fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
+ fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
fi
# Build the 'bare' package without vendor/bundle.
@@ -789,23 +710,9 @@ fi
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
-fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
+fpm_verify $FPM_EXIT_CODE $FPM_RESULTS
# Workbench package build done
-
-# Finally, publish the packages, if necessary
-if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
- ssh -p2222 $SCPUSER@$SCPHOST -t bash - <<EOF
-if [ -n "\$(find $FPM_OUTDIR -name "*.$FORMAT" -print -quit)" ]; then
- cd "$FPM_OUTDIR" && $REPO_UPDATE_CMD
-fi
-EOF
-else
- if [[ "$UPLOAD" != 0 ]]; then
- echo "No new packages generated. No freight run necessary."
- fi
-fi
-
# clean up temporary GOPATH
rm -rf "$GOPATH"
diff --git a/jenkins/run_upload_packages.py b/jenkins/run_upload_packages.py
new file mode 100755
index 0000000..3eac15b
--- /dev/null
+++ b/jenkins/run_upload_packages.py
@@ -0,0 +1,259 @@
+#!/usr/bin/env python3
+
+import argparse
+import functools
+import glob
+import logging
+import os
+import pipes
+import shutil
+import subprocess
+import sys
+import time
+
+class TimestampFile:
+ def __init__(self, path):
+ self.path = path
+ self.start_time = time.time()
+
+ def last_upload(self):
+ try:
+ return os.path.getmtime(self.path)
+ except EnvironmentError:
+ return -1
+
+ def update(self):
+ os.close(os.open(self.path, os.O_CREAT | os.O_APPEND))
+ os.utime(self.path, (time.time(), self.start_time))
+
+
+class PackageSuite:
+ NEED_SSH = False
+
+ def __init__(self, glob_root, rel_globs):
+ logger_part = getattr(self, 'LOGGER_PART', os.path.basename(glob_root))
+ self.logger = logging.getLogger('arvados-dev.upload.' + logger_part)
+ self.globs = [os.path.join(glob_root, rel_glob)
+ for rel_glob in rel_globs]
+
+ def files_to_upload(self, since_timestamp):
+ for abs_glob in self.globs:
+ for path in glob.glob(abs_glob):
+ if os.path.getmtime(path) >= since_timestamp:
+ yield path
+
+ def upload_file(self, path):
+ raise NotImplementedError("PackageSuite.upload_file")
+
+ def upload_files(self, paths):
+ for path in paths:
+ self.logger.info("Uploading %s", path)
+ self.upload_file(path)
+
+ def post_uploads(self, paths):
+ pass
+
+ def update_packages(self, since_timestamp):
+ upload_paths = list(self.files_to_upload(since_timestamp))
+ if upload_paths:
+ self.upload_files(upload_paths)
+ self.post_uploads(upload_paths)
+
+
+class PythonPackageSuite(PackageSuite):
+ LOGGER_PART = 'python'
+
+ def __init__(self, glob_root, rel_globs):
+ super().__init__(glob_root, rel_globs)
+ self.seen_packages = set()
+
+ def upload_file(self, path):
+ src_dir = os.path.dirname(os.path.dirname(path))
+ if src_dir in self.seen_packages:
+ return
+ self.seen_packages.add(src_dir)
+ # NOTE: If we ever start uploading Python 3 packages, we'll need to
+ # figure out some way to adapt cmd to match. It might be easiest
+ # to give all our setup.py files the executable bit, and run that
+ # directly.
+ # We also must run `sdist` before `upload`: `upload` uploads any
+ # distributions previously generated in the command. It doesn't
+ # know how to upload distributions already on disk. We write the
+ # result to a dedicated directory to avoid interfering with our
+ # timestamp tracking.
+ cmd = ['python2.7', 'setup.py']
+ if not self.logger.isEnabledFor(logging.INFO):
+ cmd.append('--quiet')
+ cmd.extend(['sdist', '--dist-dir', '.upload_dist', 'upload'])
+ subprocess.check_call(cmd, cwd=src_dir)
+ shutil.rmtree(os.path.join(src_dir, '.upload_dist'))
+
+
+class GemPackageSuite(PackageSuite):
+ LOGGER_PART = 'gems'
+
+ def upload_file(self, path):
+ cmd = ['gem', 'push', path]
+ push_proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
+ repushed = any(line == b'Repushing of gem versions is not allowed.\n'
+ for line in push_proc.stdout)
+ # Read any remaining stdout before closing.
+ for line in push_proc.stdout:
+ pass
+ push_proc.stdout.close()
+ if (push_proc.wait() != 0) and not repushed:
+ raise subprocess.CalledProcessError(push_proc.returncode, cmd)
+
+
+class DistroPackageSuite(PackageSuite):
+ NEED_SSH = True
+ REMOTE_DEST_DIR = 'tmp'
+
+ def __init__(self, glob_root, rel_globs, target, ssh_host, ssh_opts):
+ super().__init__(glob_root, rel_globs)
+ self.target = target
+ self.ssh_host = ssh_host
+ self.ssh_opts = ['-o' + opt for opt in ssh_opts]
+ if not self.logger.isEnabledFor(logging.INFO):
+ self.ssh_opts.append('-q')
+
+ def _build_cmd(self, base_cmd, *args):
+ cmd = [base_cmd]
+ cmd.extend(self.ssh_opts)
+ cmd.extend(args)
+ return cmd
+
+ def _paths_basenames(self, paths):
+ return (os.path.basename(path) for path in paths)
+
+ def _run_script(self, script, *args):
+ # SSH will use a shell to run our bash command, so we have to
+ # quote our arguments.
+ # self.__class__.__name__ provides $0 for the script, which makes a
+ # nicer message if there's an error.
+ subprocess.check_call(self._build_cmd(
+ 'ssh', self.ssh_host, 'bash', '-ec', pipes.quote(script),
+ self.__class__.__name__, *(pipes.quote(s) for s in args)))
+
+ def upload_files(self, paths):
+ cmd = self._build_cmd('scp', *paths)
+ cmd.append('{self.ssh_host}:{self.REMOTE_DEST_DIR}'.format(self=self))
+ subprocess.check_call(cmd)
+
+
+class DebianPackageSuite(DistroPackageSuite):
+ FREIGHT_SCRIPT = """
+cd "$1"; shift
+TARGET=$1; shift
+freight add "$@" "apt/$TARGET"
+freight cache
+rm "$@"
+"""
+
+ def post_uploads(self, paths):
+ self._run_script(self.FREIGHT_SCRIPT, self.REMOTE_DEST_DIR,
+ self.target, *self._paths_basenames(paths))
+
+
+class RedHatPackageSuite(DistroPackageSuite):
+ CREATEREPO_SCRIPT = """
+cd "$1"; shift
+REPODIR=$1; shift
+rpmsign --addsign "$@" </dev/null
+mv "$@" "$REPODIR"
+createrepo "$REPODIR"
+"""
+ REPO_ROOT = '/var/www/rpm.arvados.org/'
+ TARGET_REPODIRS = {
+ 'centos6': 'CentOS/6/os/x86_64/'
+ }
+
+ def post_uploads(self, paths):
+ repo_dir = os.path.join(self.REPO_ROOT,
+ self.TARGET_REPODIRS[self.target])
+ self._run_script(self.CREATEREPO_SCRIPT, self.REMOTE_DEST_DIR,
+ repo_dir, *self._paths_basenames(paths))
+
+
+def _define_suite(suite_class, *rel_globs, **kwargs):
+ return functools.partial(suite_class, rel_globs=rel_globs, **kwargs)
+
+PACKAGE_SUITES = {
+ 'python': _define_suite(PythonPackageSuite,
+ 'sdk/python/dist/*.tar.gz',
+ 'services/fuse/dist/*.tar.gz',
+ 'services/nodemanager/dist/*.tar.gz'),
+ 'gems': _define_suite(GemPackageSuite, 'sdk/ruby/*.gem', 'sdk/cli/*.gem'),
+ }
+for target in ['debian7', 'debian8', 'ubuntu1204']:
+ PACKAGE_SUITES[target] = _define_suite(
+ DebianPackageSuite, os.path.join('packages', target, '*.deb'),
+ target=target)
+for target in ['centos6']:
+ PACKAGE_SUITES[target] = _define_suite(
+ RedHatPackageSuite, os.path.join('packages', target, '*.rpm'),
+ target=target)
+
+def parse_arguments(arguments):
+ parser = argparse.ArgumentParser(
+ prog="run_upload_packages.py",
+ description="Upload Arvados packages to various repositories")
+ parser.add_argument(
+ '--workspace', '-W', default=os.environ.get('WORKSPACE'),
+ help="Arvados source directory with built packages to upload")
+ parser.add_argument(
+ '--ssh-host', '-H',
+ help="Host specification for distribution repository server")
+ parser.add_argument('-o', action='append', default=[], dest='ssh_opts',
+ metavar='OPTION', help="Pass option to `ssh -o`")
+ parser.add_argument('--verbose', '-v', action='count', default=0,
+ help="Log more information and subcommand output")
+ parser.add_argument(
+ 'targets', nargs='*', default=['all'], metavar='target',
+ help="Upload packages to these targets (default all)\nAvailable targets: " +
+ ', '.join(sorted(PACKAGE_SUITES.keys())))
+ args = parser.parse_args(arguments)
+ if 'all' in args.targets:
+ args.targets = list(PACKAGE_SUITES.keys())
+
+ if args.workspace is None:
+ parser.error("workspace not set from command line or environment")
+ for target in args.targets:
+ try:
+ suite_class = PACKAGE_SUITES[target].func
+ except KeyError:
+ parser.error("unrecognized target {!r}".format(target))
+ if suite_class.NEED_SSH and (args.ssh_host is None):
+ parser.error(
+ "--ssh-host must be specified to upload distribution packages")
+ return args
+
+def setup_logger(stream_dest, args):
+ log_handler = logging.StreamHandler(stream_dest)
+ log_handler.setFormatter(logging.Formatter(
+ '%(asctime)s %(name)s[%(process)d] %(levelname)s: %(message)s',
+ '%Y-%m-%d %H:%M:%S'))
+ logger = logging.getLogger('arvados-dev.upload')
+ logger.addHandler(log_handler)
+ logger.setLevel(max(1, logging.WARNING - (10 * args.verbose)))
+
+def build_suite_and_upload(target, since_timestamp, args):
+ suite_def = PACKAGE_SUITES[target]
+ kwargs = {}
+ if suite_def.func.NEED_SSH:
+ kwargs.update(ssh_host=args.ssh_host, ssh_opts=args.ssh_opts)
+ suite = suite_def(args.workspace, **kwargs)
+ suite.update_packages(since_timestamp)
+
+def main(arguments, stdout=sys.stdout, stderr=sys.stderr):
+ args = parse_arguments(arguments)
+ setup_logger(stderr, args)
+ ts_file = TimestampFile(os.path.join(args.workspace, 'packages',
+ '.last_upload'))
+ last_upload_ts = ts_file.last_upload()
+ for target in args.targets:
+ build_suite_and_upload(target, last_upload_ts, args)
+ ts_file.update()
+
+if __name__ == '__main__':
+ main(sys.argv[1:])
commit a5c1b7e9798565f07e231c43b3453fe7b69c7368
Author: Nico Cesar <nico at curoverse.com>
Date: Tue Jul 21 10:10:24 2015 -0400
added exit if git pull fails
no issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 398a050..1d23b4e 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -182,6 +182,10 @@ if [[ "$?" == "0" ]]; then
else
title "Installing latest arvados/jobs Docker image"
ssh -o "StrictHostKeyChecking no" shell.$IDENTIFIER "ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN /usr/local/rvm/bin/rvm-exec default arv keep docker --pull --project-uuid=$DOCKER_IMAGES_PROJECT arvados/jobs $GIT_COMMIT"
+ if [[ "$?" -ne 0 ]]; then
+ title "'git pull' failed exiting..."
+ exit 1
+ fi
fi
title "Gathering list of shell and Keep nodes"
commit d30c7f8b3210ae1e825033e852c369d16b7e48f1
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jul 20 14:36:46 2015 -0400
6656: python-arvados-python-client Recommends: git.
Because arv-copy uses it. Refs #6656.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 8bf153c..42b42da 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -623,7 +623,7 @@ fpm_build_and_scp $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curover
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
cd $WORKSPACE/packages/$TARGET
-fpm_build_and_scp $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
+fpm_build_and_scp $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --deb-recommends=git
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
commit c9182875d51afe9407271334d0be1fdbd718c4d1
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Jul 16 18:02:30 2015 -0400
run-deploy.sh improvements: remove the need for a .ssh/config entry for
the shortname of each API server. Thanks for the suggestion Nico.
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 519c126..398a050 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -190,11 +190,11 @@ KEEP_NODES=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TO
title "Updating API server"
SUM_ECODE=0
-run_puppet $IDENTIFIER ECODE
+run_puppet $IDENTIFIER.arvadosapi.com ECODE
SUM_ECODE=$(($SUM_ECODE + $ECODE))
-run_command $IDENTIFIER ECODE "/usr/local/bin/arvados-api-server-upgrade.sh"
+run_command $IDENTIFIER.arvadosapi.com ECODE "/usr/local/bin/arvados-api-server-upgrade.sh"
SUM_ECODE=$(($SUM_ECODE + $ECODE))
-run_command $IDENTIFIER ECODE "dpkg -L arvados-mailchimp-plugin 2>/dev/null && apt-get install arvados-mailchimp-plugin --reinstall || echo"
+run_command $IDENTIFIER.arvadosapi.com ECODE "dpkg -L arvados-mailchimp-plugin 2>/dev/null && apt-get install arvados-mailchimp-plugin --reinstall || echo"
SUM_ECODE=$(($SUM_ECODE + $ECODE))
if [[ "$SUM_ECODE" != "0" ]]; then
commit ef2d37b1b95aa57a2840080995ee0555b7a64f50
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jul 15 23:14:23 2015 -0400
Make sure jessie packages end up in the jessie repository.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 3493f7c..8bf153c 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -111,7 +111,7 @@ case "$TARGET" in
debian8)
FORMAT=deb
FPM_OUTDIR=tmp
- REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rm -f *deb'
+ REPO_UPDATE_CMD='freight add *deb apt/jessie && freight cache && rm -f *deb'
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON2_PKG_PREFIX=python
commit 9cc673cf435ec02b043e2b3729b0ce411cfd5f95
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jul 15 22:59:28 2015 -0400
We need to force-tag the arvados/jobs docker image.
No issue #
diff --git a/jenkins/run-build-docker-jobs-image.sh b/jenkins/run-build-docker-jobs-image.sh
index 2cc0692..fcf849b 100755
--- a/jenkins/run-build-docker-jobs-image.sh
+++ b/jenkins/run-build-docker-jobs-image.sh
@@ -68,7 +68,7 @@ docker_push () {
then
for tag in $( echo $tags|tr "," " " )
do
- $DOCKER tag $1 $1:$tag
+ $DOCKER tag -f $1 $1:$tag
done
fi
commit 86cc2e4b923ab24ef9a6460efa23904a9f8ed742
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jul 15 22:53:50 2015 -0400
freight is already smart enough to do atomic updates.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 95baaba..3493f7c 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -95,7 +95,7 @@ case "$TARGET" in
debian7)
FORMAT=deb
FPM_OUTDIR=tmp
- REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rsync -AaHX /var/www/freight/ /var/www/freight-production/ --delete && rm -f *deb'
+ REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rm -f *deb'
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON2_PKG_PREFIX=python
@@ -111,7 +111,7 @@ case "$TARGET" in
debian8)
FORMAT=deb
FPM_OUTDIR=tmp
- REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rsync -AaHX /var/www/freight/ /var/www/freight-production/ --delete && rm -f *deb'
+ REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rm -f *deb'
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON2_PKG_PREFIX=python
@@ -127,7 +127,7 @@ case "$TARGET" in
ubuntu1204)
FORMAT=deb
FPM_OUTDIR=tmp
- REPO_UPDATE_CMD='freight add *deb apt/precise && freight cache && rsync -AaHX /var/www/freight/ /var/www/freight-production/ --delete && rm -f *deb'
+ REPO_UPDATE_CMD='freight add *deb apt/precise && freight cache && rm -f *deb'
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON2_PKG_PREFIX=python
commit 30945c85e1d6b6dd20885683851ff4f1f0575346
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jul 15 21:55:21 2015 -0400
Add dockerfiles/build-all-build-containers.sh script
refs #6504
diff --git a/jenkins/dockerfiles/build-all-build-containers.sh b/jenkins/dockerfiles/build-all-build-containers.sh
new file mode 100755
index 0000000..34ffcce
--- /dev/null
+++ b/jenkins/dockerfiles/build-all-build-containers.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+
+make
+
+for target in `find -maxdepth 1 -type d |grep -v generated`; do
+ if [[ "$target" == "." ]]; then
+ continue
+ fi
+ target=${target#./}
+ echo $target
+ cd $target
+ docker build -t arvados/build:$target .
+ cd ..
+done
+
+
commit a7d40e5a84aadcbfcbce43876d6f1ad5e7d9d824
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jul 15 13:29:00 2015 -0400
Add datamanager tests. refs #3408
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index a646e36..96c71b9 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -633,6 +633,8 @@ gostuff=(
services/crunchstat
services/keepstore
services/keepproxy
+ services/datamanager/summary
+ services/datamanager/collection
sdk/go/arvadosclient
sdk/go/keepclient
sdk/go/streamer
commit 0a0ca0b129c2061db7c6f70e46334500088b984e
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jul 15 13:14:34 2015 -0400
Build python3-websocket-client packages, which is now a dependency for
python3-docker-py, which is a dependency for arvados-docker-cleaner.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 7b1b3d7..95baaba 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -106,7 +106,7 @@ case "$TARGET" in
rsa uritemplate httplib2 ws4py \
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
- PYTHON3_BACKPORTS=(docker-py six requests)
+ PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
debian8)
FORMAT=deb
@@ -122,7 +122,7 @@ case "$TARGET" in
rsa uritemplate httplib2 ws4py \
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
- PYTHON3_BACKPORTS=(docker-py six requests)
+ PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
ubuntu1204)
FORMAT=deb
@@ -138,7 +138,7 @@ case "$TARGET" in
rsa uritemplate httplib2 ws4py \
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
- PYTHON3_BACKPORTS=(docker-py six requests)
+ PYTHON3_BACKPORTS=(docker-py six requests websocket-client)
;;
centos6)
FORMAT=rpm
commit ea0e43b2e928db9e7ef77977adc789cf44bc81c2
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Jul 13 21:02:07 2015 -0400
Make sure we have a conference/environments/production.rb file so that
we generate the workbench assets correctly before packaging.
refs #6362
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index af35285..7b1b3d7 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -746,8 +746,9 @@ fi
# from the package - empty it instead.
rm -rf $WORKSPACE/apps/workbench/tmp/*
-# Set up application.yml so that asset precompilation works
+# Set up application.yml and production.rb so that asset precompilation works
\cp config/application.yml.example config/application.yml -f
+\cp config/environments/production.rb.example config/environments/production.rb -f
sed -i 's/secret_token: ~/secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/' config/application.yml
RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile >/dev/null
commit 5b7eaf60e9ae6a239f9ef601ff615913680f054f
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Jul 13 11:39:25 2015 -0400
run_deploy.sh improvements:
* Make sure that group_class is included when creating the public docker
images project.
* Document the required permissions for the arvados token needed to run
run_deploy.sh
closes #6586
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 98e4359..519c126 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -12,6 +12,17 @@ function usage {
echo >&2 " -d, --debug Enable debug output"
echo >&2 " -h, --help Display this help and exit"
echo >&2
+ echo >&2 "Note: this script requires an arvados token created with these permissions:"
+ echo >&2 ' arv api_client_authorization create_system_auth \'
+ echo >&2 ' --scopes "[\"GET /arvados/v1/virtual_machines\",'
+ echo >&2 ' \"GET /arvados/v1/keep_services\",'
+ echo >&2 ' \"GET /arvados/v1/groups\",'
+ echo >&2 ' \"GET /arvados/v1/links\",'
+ echo >&2 ' \"GET /arvados/v1/groups/\",'
+ echo >&2 ' \"GET /arvados/v1/keep_services/accessible\",'
+ echo >&2 ' \"POST /arvados/v1/collections\",'
+ echo >&2 ' \"POST /arvados/v1/links\"]"'
+ echo >&2
}
# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
@@ -146,7 +157,7 @@ DOCKER_IMAGES_PROJECT=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARV
if [[ "$DOCKER_IMAGES_PROJECT" == "" ]]; then
title "Warning: Arvados Standard Docker Images project not found. Creating it."
- DOCKER_IMAGES_PROJECT=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv --format=uuid group create --group "{\"owner_uuid\":\"$IDENTIFIER-tpzed-000000000000000\", \"name\":\"Arvados Standard Docker Images\"}"`
+ DOCKER_IMAGES_PROJECT=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv --format=uuid group create --group "{\"owner_uuid\":\"$IDENTIFIER-tpzed-000000000000000\", \"name\":\"Arvados Standard Docker Images\", \"group_class\":\"project\"}"`
ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv link create --link "{\"tail_uuid\":\"$IDENTIFIER-j7d0g-fffffffffffffff\", \"head_uuid\":\"$DOCKER_IMAGES_PROJECT\", \"link_class\":\"permission\", \"name\":\"can_read\" }"
if [[ "$?" != "0" ]]; then
title "ERROR: could not create standard Docker images project Please create it, cf. http://doc.arvados.org/install/create-standard-objects.html"
commit 4690e2bf79cfef03c548e08fd5cf67efa709ad12
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jul 10 14:07:39 2015 -0400
Add debian8 as a build target.
No issue #
diff --git a/jenkins/dockerfiles/Makefile b/jenkins/dockerfiles/Makefile
index 9d2b4a8..3d6eec6 100644
--- a/jenkins/dockerfiles/Makefile
+++ b/jenkins/dockerfiles/Makefile
@@ -1,4 +1,4 @@
-all: centos6/generated debian7/generated ubuntu1204/generated
+all: centos6/generated debian7/generated debian8/generated ubuntu1204/generated
centos6/generated: common-generated-all
test -d centos6/generated || mkdir centos6/generated
@@ -8,6 +8,10 @@ debian7/generated: common-generated-all
test -d debian7/generated || mkdir debian7/generated
cp -rlt debian7/generated common-generated/*
+debian8/generated: common-generated-all
+ test -d debian8/generated || mkdir debian8/generated
+ cp -rlt debian8/generated common-generated/*
+
ubuntu1204/generated: common-generated-all
test -d ubuntu1204/generated || mkdir ubuntu1204/generated
cp -rlt ubuntu1204/generated common-generated/*
diff --git a/jenkins/dockerfiles/debian8/Dockerfile b/jenkins/dockerfiles/debian8/Dockerfile
new file mode 100644
index 0000000..5b81434
--- /dev/null
+++ b/jenkins/dockerfiles/debian8/Dockerfile
@@ -0,0 +1,19 @@
+FROM debian:jessie
+MAINTAINER Ward Vandewege <ward at curoverse.com>
+
+# Install dependencies and set up system.
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev
+
+# Install RVM
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+ curl -L https://get.rvm.io | bash -s stable && \
+ /usr/local/rvm/bin/rvm install 2.1 && \
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
+ /usr/local/rvm/bin/rvm-exec default gem install fpm
+
+# Install golang binary
+ADD generated/golang-amd64.tar.gz /usr/local/
+RUN ln -s /usr/local/go/bin/go /usr/local/bin/
+
+ENV WORKSPACE /arvados
+CMD ["/usr/local/rvm/bin/rvm-exec", "default", "bash", "/jenkins/run-build-packages.sh", "--target", "debian8"]
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index f2ce33a..af35285 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -108,6 +108,22 @@ case "$TARGET" in
ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests)
;;
+ debian8)
+ FORMAT=deb
+ FPM_OUTDIR=tmp
+ REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rsync -AaHX /var/www/freight/ /var/www/freight-production/ --delete && rm -f *deb'
+
+ PYTHON2_PACKAGE=python$PYTHON2_VERSION
+ PYTHON2_PKG_PREFIX=python
+ PYTHON3_PACKAGE=python$PYTHON3_VERSION
+ PYTHON3_PKG_PREFIX=python3
+ PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
+ oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
+ rsa uritemplate httplib2 ws4py \
+ virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
+ ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
+ PYTHON3_BACKPORTS=(docker-py six requests)
+ ;;
ubuntu1204)
FORMAT=deb
FPM_OUTDIR=tmp
commit bdfbb8c113d8e7f63bb8caa57010437826c59dc9
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Jul 9 22:01:36 2015 -0400
Build arvados/jobs docker image separately from the others.
refs #6528
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-build-docker-images.sh
similarity index 84%
copy from jenkins/run-docker-tests.sh
copy to jenkins/run-build-docker-images.sh
index eff75ac..71910c2 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-build-docker-images.sh
@@ -155,7 +155,6 @@ else
## even though credentials are already in .dockercfg
docker login -u arvados
-
docker_push arvados/api
docker_push arvados/compute
docker_push arvados/doc
@@ -170,33 +169,4 @@ else
fi
fi
-title "Starting docker java-bwa-samtools build"
-
-timer_reset
-
-./build.sh java-bwa-samtools-image
-
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! docker java-bwa-samtools BUILD FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
-
-title "docker build java-bwa-samtools complete (`timer`)"
-
-timer_reset
-
-if [[ "$ECODE" != "0" ]]; then
- title "upload arvados/jobs image SKIPPED because build failed"
-else
- if [[ $upload == true ]]; then
- title "upload arvados/jobs image"
- docker_push arvados/jobs
- title "upload arvados/jobs image complete (`timer`)"
- else
- title "upload arvados images SKIPPED because no --upload option set"
- fi
-fi
-
exit $EXITCODE
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-build-docker-jobs-image.sh
similarity index 77%
rename from jenkins/run-docker-tests.sh
rename to jenkins/run-build-docker-jobs-image.sh
index eff75ac..2cc0692 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-build-docker-jobs-image.sh
@@ -125,14 +125,13 @@ timer_reset
# clean up the docker build environment
cd "$WORKSPACE"
cd docker
-./build.sh realclean
-
+rm -f jobs-image
rm -f config.yml
# Get test config.yml file
cp $HOME/docker/config.yml .
-./build.sh
+./build.sh jobs-image
ECODE=$?
@@ -155,45 +154,8 @@ else
## even though credentials are already in .dockercfg
docker login -u arvados
-
- docker_push arvados/api
- docker_push arvados/compute
- docker_push arvados/doc
- docker_push arvados/workbench
- docker_push arvados/keep
- docker_push arvados/keepproxy
- docker_push arvados/shell
- docker_push arvados/sso
- title "upload arvados images complete (`timer`)"
- else
- title "upload arvados images SKIPPED because no --upload option set"
- fi
-fi
-
-title "Starting docker java-bwa-samtools build"
-
-timer_reset
-
-./build.sh java-bwa-samtools-image
-
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! docker java-bwa-samtools BUILD FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
-
-title "docker build java-bwa-samtools complete (`timer`)"
-
-timer_reset
-
-if [[ "$ECODE" != "0" ]]; then
- title "upload arvados/jobs image SKIPPED because build failed"
-else
- if [[ $upload == true ]]; then
- title "upload arvados/jobs image"
docker_push arvados/jobs
- title "upload arvados/jobs image complete (`timer`)"
+ title "upload arvados images complete (`timer`)"
else
title "upload arvados images SKIPPED because no --upload option set"
fi
commit 2edb46e696fa3ca24aeb0c820f15630ece41f0fc
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jul 8 16:31:35 2015 -0400
Add dependency on libpam-python to the libarvados-pam package.
refs #2256
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index f1f52a9..f2ce33a 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -642,10 +642,10 @@ PKG_VERSION=$(version_from_git)
cd $WORKSPACE/packages/$TARGET
if [[ "$FORMAT" == "deb" ]]; then
- fpm_build_and_scp $WORKSPACE/sdk/pam/debian/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/debian/arvados_pam=/etc/default/arvados_pam
+ fpm_build_and_scp $WORKSPACE/sdk/pam/debian/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" "-d libpam-python" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/debian/arvados_pam=/etc/default/arvados_pam
#else
# FIXME enable and test once we have the centos pam.d file
- #fpm_build_and_scp $WORKSPACE/sdk/pam/centos/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/centos/arvados_pam=/etc/default/arvados_pam
+ #fpm_build_and_scp $WORKSPACE/sdk/pam/centos/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" "-d libpam-python" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/centos/arvados_pam=/etc/default/arvados_pam
fi
# Build the API server package
commit c1aed40ede934f264217fa46f9b4bec666d3b600
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jul 8 14:56:07 2015 -0400
Add libpam-arvados package build to run-build-packages.
refs #6384
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index bb65130..f1f52a9 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -636,6 +636,18 @@ for deppkg in "${PYTHON3_BACKPORTS[@]}"; do
fpm_build_and_scp "$deppkg" "$outname" "" python3
done
+# libpam-arvados
+cd $WORKSPACE/sdk/pam
+PKG_VERSION=$(version_from_git)
+cd $WORKSPACE/packages/$TARGET
+
+if [[ "$FORMAT" == "deb" ]]; then
+ fpm_build_and_scp $WORKSPACE/sdk/pam/debian/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/debian/arvados_pam=/etc/default/arvados_pam
+#else
+ # FIXME enable and test once we have the centos pam.d file
+ #fpm_build_and_scp $WORKSPACE/sdk/pam/centos/shellinabox=/etc/pam.d/shellinabox libpam-arvados 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=Apache License, Version 2.0" "--description=PAM module for Arvados" "--config-files=/etc/default" $WORKSPACE/sdk/pam/arvados_pam.py=/usr/bin/arvados_pam.py $WORKSPACE/sdk/pam/centos/arvados_pam=/etc/default/arvados_pam
+fi
+
# Build the API server package
cd "$WORKSPACE/services/api"
commit aa569f0b4c661ec83fb5be6574250f1c3f713119
Merge: 510a6b6 08c24c2
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jul 8 10:09:25 2015 -0400
Merge branch '6389-add-dockerfiles'
Closes #6389, #6456.
commit 08c24c27a61b52063146576e9c80773b0e481f37
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jun 29 16:59:24 2015 -0400
6388: Add Dockerfile to build packages on CentOS 6.
diff --git a/jenkins/dockerfiles/Makefile b/jenkins/dockerfiles/Makefile
index 046495a..9d2b4a8 100644
--- a/jenkins/dockerfiles/Makefile
+++ b/jenkins/dockerfiles/Makefile
@@ -1,4 +1,8 @@
-all: debian7/generated ubuntu1204/generated
+all: centos6/generated debian7/generated ubuntu1204/generated
+
+centos6/generated: common-generated-all
+ test -d centos6/generated || mkdir centos6/generated
+ cp -rlt centos6/generated common-generated/*
debian7/generated: common-generated-all
test -d debian7/generated || mkdir debian7/generated
diff --git a/jenkins/dockerfiles/centos6/Dockerfile b/jenkins/dockerfiles/centos6/Dockerfile
new file mode 100644
index 0000000..e08665b
--- /dev/null
+++ b/jenkins/dockerfiles/centos6/Dockerfile
@@ -0,0 +1,25 @@
+FROM centos:6
+MAINTAINER Brett Smith <brett at curoverse.com>
+
+# Install build dependencies provided in base distribution
+RUN yum -q -y install make automake gcc gcc-c++ libyaml-devel patch readline-devel zlib-devel libffi-devel openssl-devel bzip2 libtool bison sqlite-devel rpm-build git perl-ExtUtils-MakeMaker fuse-devel libattr-devel nss-devel libcurl-devel which tar scl-utils
+
+# Install golang binary
+ADD generated/golang-amd64.tar.gz /usr/local/
+RUN ln -s /usr/local/go/bin/go /usr/local/bin/
+
+# Install RVM
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+ curl -L https://get.rvm.io | bash -s stable && \
+ /usr/local/rvm/bin/rvm install 2.1 && \
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
+ /usr/local/rvm/bin/rvm-exec default gem install fpm
+
+# Download and install Software Collections for newer requirements
+ADD https://www.softwarecollections.org/en/scls/rhscl/rh-python34/epel-6-x86_64/download/rhscl-rh-python34-epel-6-x86_64.noarch.rpm https://www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm /opt/
+
+RUN yum -q -y install /opt/rhscl-*.rpm
+RUN yum -q -y install python27 rh-python34
+
+ENV WORKSPACE /arvados
+CMD ["scl", "enable", "rh-python34", "python27", "/usr/local/rvm/bin/rvm-exec default bash /jenkins/run-build-packages.sh --target centos6"]
commit 8f04915e13d44ce5028eb07f3eed3879a592f11b
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jun 29 16:56:57 2015 -0400
6389: Add Makefile to support Docker builds.
diff --git a/jenkins/dockerfiles/.gitignore b/jenkins/dockerfiles/.gitignore
new file mode 100644
index 0000000..ceee9fa
--- /dev/null
+++ b/jenkins/dockerfiles/.gitignore
@@ -0,0 +1,2 @@
+*/generated
+common-generated/
diff --git a/jenkins/dockerfiles/Makefile b/jenkins/dockerfiles/Makefile
new file mode 100644
index 0000000..046495a
--- /dev/null
+++ b/jenkins/dockerfiles/Makefile
@@ -0,0 +1,17 @@
+all: debian7/generated ubuntu1204/generated
+
+debian7/generated: common-generated-all
+ test -d debian7/generated || mkdir debian7/generated
+ cp -rlt debian7/generated common-generated/*
+
+ubuntu1204/generated: common-generated-all
+ test -d ubuntu1204/generated || mkdir ubuntu1204/generated
+ cp -rlt ubuntu1204/generated common-generated/*
+
+common-generated-all: common-generated/golang-amd64.tar.gz
+
+common-generated/golang-amd64.tar.gz: common-generated
+ wget -cqO common-generated/golang-amd64.tar.gz https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz
+
+common-generated:
+ mkdir common-generated
diff --git a/jenkins/dockerfiles/README b/jenkins/dockerfiles/README
index 2ae68cd..0dfab94 100644
--- a/jenkins/dockerfiles/README
+++ b/jenkins/dockerfiles/README
@@ -1,29 +1,13 @@
==================
-BUILD
+DOCKER IMAGE BUILD
==================
-cd dockerfiles/ubuntu1204
-docker build -t arvados/build:ubuntu1204 .
+1. `make`
+2. `cd DISTRO`
+3. `docker build -t arvados/build:DISTRO .`
-or
+==============
+BUILD PACKAGES
+==============
-cd dockerfiles/debian7
-docker build -t arvados/build:debian7 .
-
-==================
-START
-==================
-
-docker run -v /path/to/your/arvados-dev/jenkins:/jenkins -v /path/to/your/arvados:/arvados -i -t <imagehash> /bin/bash
-
-====================
-INSIDE THE CONTAINER
-====================
-
-source /etc/profile.d/rvm.sh
-
-WORKSPACE=/arvados ./run-build-packages.sh --target debian7
-
-or
-
-WORKSPACE=/arvados ./run-build-packages.sh --target ubuntu1204
+`docker run -v /path/to/your/arvados-dev/jenkins:/jenkins -v /path/to/your/arvados:/arvados arvados/build:DISTRO`
commit 5dfd1323ad00fa86e008072e8a1b19f9869b586f
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jun 29 16:56:16 2015 -0400
6389: Dockerfiles include the CMD to build packages.
diff --git a/jenkins/dockerfiles/debian7/Dockerfile b/jenkins/dockerfiles/debian7/Dockerfile
index b4f1768..ca4cc28 100644
--- a/jenkins/dockerfiles/debian7/Dockerfile
+++ b/jenkins/dockerfiles/debian7/Dockerfile
@@ -15,3 +15,5 @@ RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
ADD generated/golang-amd64.tar.gz /usr/local/
RUN ln -s /usr/local/go/bin/go /usr/local/bin/
+ENV WORKSPACE /arvados
+CMD ["/usr/local/rvm/bin/rvm-exec", "default", "bash", "/jenkins/run-build-packages.sh", "--target", "debian7"]
diff --git a/jenkins/dockerfiles/ubuntu1204/Dockerfile b/jenkins/dockerfiles/ubuntu1204/Dockerfile
index 4cdbbf8..96e331c 100644
--- a/jenkins/dockerfiles/ubuntu1204/Dockerfile
+++ b/jenkins/dockerfiles/ubuntu1204/Dockerfile
@@ -14,3 +14,6 @@ RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
# Install golang binary
ADD generated/golang-amd64.tar.gz /usr/local/
RUN ln -s /usr/local/go/bin/go /usr/local/bin/
+
+ENV WORKSPACE /arvados
+CMD ["/usr/local/rvm/bin/rvm-exec", "default", "bash", "/jenkins/run-build-packages.sh", "--target", "ubuntu1204"]
commit 73f6f1bb2bc589b28f5779521de5db4aa0993deb
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 26 17:07:32 2015 -0400
Add dockerfiles for Debian7 and Ubuntu1204
refs #6389
diff --git a/jenkins/dockerfiles/README b/jenkins/dockerfiles/README
new file mode 100644
index 0000000..2ae68cd
--- /dev/null
+++ b/jenkins/dockerfiles/README
@@ -0,0 +1,29 @@
+==================
+BUILD
+==================
+
+cd dockerfiles/ubuntu1204
+docker build -t arvados/build:ubuntu1204 .
+
+or
+
+cd dockerfiles/debian7
+docker build -t arvados/build:debian7 .
+
+==================
+START
+==================
+
+docker run -v /path/to/your/arvados-dev/jenkins:/jenkins -v /path/to/your/arvados:/arvados -i -t <imagehash> /bin/bash
+
+====================
+INSIDE THE CONTAINER
+====================
+
+source /etc/profile.d/rvm.sh
+
+WORKSPACE=/arvados ./run-build-packages.sh --target debian7
+
+or
+
+WORKSPACE=/arvados ./run-build-packages.sh --target ubuntu1204
diff --git a/jenkins/dockerfiles/debian7/Dockerfile b/jenkins/dockerfiles/debian7/Dockerfile
new file mode 100644
index 0000000..b4f1768
--- /dev/null
+++ b/jenkins/dockerfiles/debian7/Dockerfile
@@ -0,0 +1,17 @@
+FROM debian:wheezy
+MAINTAINER Ward Vandewege <ward at curoverse.com>
+
+# Install dependencies and set up system.
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev
+
+# Install RVM
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+ curl -L https://get.rvm.io | bash -s stable && \
+ /usr/local/rvm/bin/rvm install 2.1 && \
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
+ /usr/local/rvm/bin/rvm-exec default gem install fpm
+
+# Install golang binary
+ADD generated/golang-amd64.tar.gz /usr/local/
+RUN ln -s /usr/local/go/bin/go /usr/local/bin/
+
diff --git a/jenkins/dockerfiles/ubuntu1204/Dockerfile b/jenkins/dockerfiles/ubuntu1204/Dockerfile
new file mode 100644
index 0000000..4cdbbf8
--- /dev/null
+++ b/jenkins/dockerfiles/ubuntu1204/Dockerfile
@@ -0,0 +1,16 @@
+FROM ubuntu:precise
+MAINTAINER Ward Vandewege <ward at curoverse.com>
+
+# Install dependencies and set up system.
+RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools libcurl4-gnutls-dev curl git libattr1-dev libfuse-dev
+
+# Install RVM
+RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 && \
+ curl -L https://get.rvm.io | bash -s stable && \
+ /usr/local/rvm/bin/rvm install 2.1 && \
+ /usr/local/rvm/bin/rvm alias create default ruby-2.1 && \
+ /usr/local/rvm/bin/rvm-exec default gem install fpm
+
+# Install golang binary
+ADD generated/golang-amd64.tar.gz /usr/local/
+RUN ln -s /usr/local/go/bin/go /usr/local/bin/
commit 510a6b6a1a8e263a7cfb305032ae3ef07d88a400
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jul 8 09:59:54 2015 -0400
Fix up git log statements:
* remove duplication between -n1 and --max-count=1
* make sure every git log statement has --first-parent
refs #6096
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 0eb2244..bb65130 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -204,7 +204,7 @@ version_from_git() {
# Generates a version number from the git log for the current working
# directory, and writes it to stdout.
local git_ts git_hash
- declare $(TZ=UTC git log -n1 --first-parent --max-count=1 \
+ declare $(TZ=UTC git log -n1 --first-parent \
--format=format:"git_ts=%ct git_hash=%h" .)
echo "0.1.$(date -ud "@$git_ts" +%Y%m%d%H%M%S).$git_hash"
}
@@ -213,7 +213,7 @@ timestamp_from_git() {
# Generates a version number from the git log for the current working
# directory, and writes it to stdout.
local git_ts git_hash
- declare $(TZ=UTC git log -n1 --first-parent --max-count=1 \
+ declare $(TZ=UTC git log -n1 --first-parent \
--format=format:"git_ts=%ct git_hash=%h" .)
echo "$git_ts"
}
@@ -506,15 +506,15 @@ if [[ "$DEBUG" != 0 ]]; then
git checkout master
git pull
# go into detached-head state
- git checkout `git log --format=format:%h -n1 .`
+ git checkout `git log --format=format:%h -n1 --first-parent .`
else
git checkout -q master
git pull -q
# go into detached-head state
- git checkout -q `git log --format=format:%h -n1 .`
+ git checkout -q `git log --format=format:%h -n1 --first-parent .`
fi
-git log --format=format:%H -n1 . > git-commit.version
+git log --format=format:%H -n1 --first-parent . > git-commit.version
# Build arvados src deb package
cd "$WORKSPACE"
commit 6086e2a55293b8a21883f720788d951c3545cac6
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Jul 7 22:05:01 2015 -0400
Small fixes for run-deploy.sh
refs #6096
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index d5b6de0..98e4359 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -155,7 +155,7 @@ if [[ "$DOCKER_IMAGES_PROJECT" == "" ]]; then
fi
title "Found Arvados Standard Docker Images project with uuid $DOCKER_IMAGES_PROJECT"
-GIT_COMMIT=`ssh $IDENTIFIER cat /usr/local/arvados/src/git-commit.version`
+GIT_COMMIT=`ssh -o "StrictHostKeyChecking no" $IDENTIFIER cat /usr/local/arvados/src/git-commit.version`
if [[ "$?" != "0" ]] || [[ "$GIT_COMMIT" == "" ]]; then
title "ERROR: unable to get arvados/jobs Docker image git revision"
@@ -164,13 +164,13 @@ else
title "Found git commit for arvados/jobs Docker image: $GIT_COMMIT"
fi
-run_command shell.$IDENTIFIER ECODE "/usr/local/rvm/bin/rvm-exec default arv keep docker" |grep -q $GIT_COMMIT
+run_command shell.$IDENTIFIER ECODE "ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN /usr/local/rvm/bin/rvm-exec default arv keep docker" |grep -q $GIT_COMMIT
if [[ "$?" == "0" ]]; then
title "Found latest arvados/jobs Docker image, nothing to upload"
else
title "Installing latest arvados/jobs Docker image"
- ssh shell.$IDENTIFIER "/usr/local/rvm/bin/rvm-exec default arv keep docker --pull --project-uuid=$DOCKER_IMAGES_PROJECT arvados/jobs $GIT_COMMIT"
+ ssh -o "StrictHostKeyChecking no" shell.$IDENTIFIER "ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN /usr/local/rvm/bin/rvm-exec default arv keep docker --pull --project-uuid=$DOCKER_IMAGES_PROJECT arvados/jobs $GIT_COMMIT"
fi
title "Gathering list of shell and Keep nodes"
commit 0a139ea3b0b4dd9785ce1c847e08d507a9f1bb6d
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jun 10 21:30:45 2015 -0400
Make run-deploy.rb install the correct version of the arvados/jobs
docker image on deploy.
refs #6096
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index a883b75..d5b6de0 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -138,6 +138,41 @@ if [[ "$ARVADOS_API_HOST" == "" ]] || [[ "$ARVADOS_API_TOKEN" == "" ]]; then
exit 1
fi
+title "Locating Arvados Standard Docker images project"
+
+JSON_FILTER="[[\"name\", \"=\", \"Arvados Standard Docker Images\"], [\"owner_uuid\", \"=\", \"$IDENTIFIER-tpzed-000000000000000\"]]"
+DOCKER_IMAGES_PROJECT=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv --format=uuid group list --filters="$JSON_FILTER"`
+
+if [[ "$DOCKER_IMAGES_PROJECT" == "" ]]; then
+ title "Warning: Arvados Standard Docker Images project not found. Creating it."
+
+ DOCKER_IMAGES_PROJECT=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv --format=uuid group create --group "{\"owner_uuid\":\"$IDENTIFIER-tpzed-000000000000000\", \"name\":\"Arvados Standard Docker Images\"}"`
+ ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv link create --link "{\"tail_uuid\":\"$IDENTIFIER-j7d0g-fffffffffffffff\", \"head_uuid\":\"$DOCKER_IMAGES_PROJECT\", \"link_class\":\"permission\", \"name\":\"can_read\" }"
+ if [[ "$?" != "0" ]]; then
+ title "ERROR: could not create standard Docker images project Please create it, cf. http://doc.arvados.org/install/create-standard-objects.html"
+ exit 1
+ fi
+fi
+
+title "Found Arvados Standard Docker Images project with uuid $DOCKER_IMAGES_PROJECT"
+GIT_COMMIT=`ssh $IDENTIFIER cat /usr/local/arvados/src/git-commit.version`
+
+if [[ "$?" != "0" ]] || [[ "$GIT_COMMIT" == "" ]]; then
+ title "ERROR: unable to get arvados/jobs Docker image git revision"
+ exit 1
+else
+ title "Found git commit for arvados/jobs Docker image: $GIT_COMMIT"
+fi
+
+run_command shell.$IDENTIFIER ECODE "/usr/local/rvm/bin/rvm-exec default arv keep docker" |grep -q $GIT_COMMIT
+
+if [[ "$?" == "0" ]]; then
+ title "Found latest arvados/jobs Docker image, nothing to upload"
+else
+ title "Installing latest arvados/jobs Docker image"
+ ssh shell.$IDENTIFIER "/usr/local/rvm/bin/rvm-exec default arv keep docker --pull --project-uuid=$DOCKER_IMAGES_PROJECT arvados/jobs $GIT_COMMIT"
+fi
+
title "Gathering list of shell and Keep nodes"
SHELL_NODES=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv virtual_machine list |jq .items[].hostname -r`
KEEP_NODES=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv keep_service list |jq .items[].service_host -r`
commit b3677ac8c09f7e42ee6e7cfe1ec7dfcdd0832009
Merge: 9932d70 400c39a
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jul 6 13:29:44 2015 -0400
Merge branch '6388-package-scripts-redhat-support-wip'
Refs #6388. Closes #6480.
commit 400c39aabb36fa98b8878806bd5210677495527a
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jul 6 10:26:44 2015 -0400
6388: Extend API server upgrade script to do initial database setup.
This will let administrators duplicating a lot of logic that already
exists in the script.
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
index 0d125cf..ace07c7 100755
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -48,13 +48,16 @@ if [[ ! -e $RELEASE_PATH/tmp ]]; then mkdir -p $RELEASE_PATH/tmp; fi
if [[ ! -e $RELEASE_PATH/log ]]; then ln -s $SHARED_PATH/log $RELEASE_PATH/log; fi
if [[ ! -e $SHARED_PATH/log/production.log ]]; then touch $SHARED_PATH/log/production.log; fi
+cd "$RELEASE_PATH"
+export RAILS_ENV=production
+
echo "Running bundle install"
-(cd $RELEASE_PATH && RAILS_ENV=production bundle install --path $SHARED_PATH/vendor_bundle)
+bundle install --path $SHARED_PATH/vendor_bundle
echo "Done."
echo "Precompiling assets"
# precompile assets; thankfully this does not take long
-(cd $RELEASE_PATH; RAILS_ENV=production bundle exec rake assets:precompile)
+bundle exec rake assets:precompile
echo "Done."
echo "Ensuring directory and file permissions"
@@ -70,7 +73,7 @@ chmod 644 $SHARED_PATH/log/*
echo "Done."
echo "Running sanity check"
-(cd $RELEASE_PATH && RAILS_ENV=production bundle exec rake config:check)
+bundle exec rake config:check
SANITY_CHECK_EXIT_CODE=$?
echo "Done."
@@ -80,8 +83,19 @@ if [[ "$SANITY_CHECK_EXIT_CODE" != "0" ]]; then
exit $SANITY_CHECK_EXIT_CODE
fi
-echo "Starting db:migrate"
-(cd $RELEASE_PATH && bundle exec rake RAILS_ENV=production db:migrate)
+echo "Checking database status"
+# If we use `grep -q`, rake will write a backtrace on EPIPE.
+if bundle exec rake db:migrate:status | grep '^database: ' >/dev/null; then
+ echo "Starting db:migrate"
+ bundle exec rake db:migrate
+elif [ 0 -eq ${PIPESTATUS[0]} ]; then
+ # The database exists, but the migrations table doesn't.
+ echo "Setting up database"
+ bundle exec rake db:structure:load db:seed
+else
+ echo "Error: Database is not ready to set up. Aborting." >&2
+ exit 1
+fi
echo "Done."
echo "Restarting nginx"
commit 287590ce79a8cf78573e1edf5bb8cd85c8e48210
Author: Brett Smith <brett at curoverse.com>
Date: Sun Jul 5 18:11:23 2015 -0400
6388: Rails package upgrade scripts support Red Hat systems.
* Red Hat doesn't use the www-data account name.
* When nginx is installed from a Software Collection, it may have a
name like `nginx16`. Detect and use this.
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
index 5af5487..0d125cf 100755
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -1,13 +1,38 @@
#!/bin/bash
+set -e
+
+if [ -e /etc/redhat-release ]; then
+ WWW_OWNER=apache:apache
+else
+ # Assume we're on a Debian-based system for now.
+ WWW_OWNER=www-data:www-data
+fi
+
+NGINX_SERVICE=${NGINX_SERVICE:-$(service --status-all 2>/dev/null \
+ | grep -Eo '\bnginx[^[:space:]]*' || true)}
+if [ -z "$NGINX_SERVICE" ]; then
+ cat >&2 <<EOF
+Error: nginx service not found. Aborting.
+Set NGINX_SERVICE to the name of the service hosting the Rails server.
+EOF
+ exit 1
+elif [ "$NGINX_SERVICE" != "$(echo "$NGINX_SERVICE" | head -n 1)" ]; then
+ cat >&2 <<EOF
+Error: multiple nginx services found. Aborting.
+Set NGINX_SERVICE to the name of the service hosting the Rails server.
+EOF
+ exit 1
+fi
+
RELEASE_PATH=/var/www/arvados-api/current
SHARED_PATH=/var/www/arvados-api/shared
CONFIG_PATH=/etc/arvados/api/
-echo "Assumption: nginx is configured to serve `hostname` from /var/www/`hostname`/current"
-echo "Assumption: /var/www/`hostname` is symlinked to /var/www/arvados-api"
+echo "Assumption: $NGINX_SERVICE is configured to serve $HOSTNAME from /var/www/$HOSTNAME/current"
+echo "Assumption: /var/www/$HOSTNAME is symlinked to /var/www/arvados-api"
echo "Assumption: configuration files are in /etc/arvados/api/"
-echo "Assumption: nginx and passenger run as the www-data user"
+echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
echo
echo "Copying files from $CONFIG_PATH"
@@ -34,13 +59,13 @@ echo "Done."
echo "Ensuring directory and file permissions"
# Ensure correct ownership of a few files
-chown www-data:www-data $RELEASE_PATH/config/environment.rb
-chown www-data:www-data $RELEASE_PATH/config.ru
-chown www-data:www-data $RELEASE_PATH/config/database.yml
-chown www-data:www-data $RELEASE_PATH/Gemfile.lock
-chown -R www-data:www-data $RELEASE_PATH/tmp
-chown -R www-data:www-data $SHARED_PATH/log
-chown www-data:www-data $RELEASE_PATH/db/structure.sql
+chown "$WWW_OWNER" $RELEASE_PATH/config/environment.rb
+chown "$WWW_OWNER" $RELEASE_PATH/config.ru
+chown "$WWW_OWNER" $RELEASE_PATH/config/database.yml
+chown "$WWW_OWNER" $RELEASE_PATH/Gemfile.lock
+chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
+chown -R "$WWW_OWNER" $SHARED_PATH/log
+chown "$WWW_OWNER" $RELEASE_PATH/db/structure.sql
chmod 644 $SHARED_PATH/log/*
echo "Done."
@@ -60,5 +85,5 @@ echo "Starting db:migrate"
echo "Done."
echo "Restarting nginx"
-service nginx restart
+service "$NGINX_SERVICE" restart
echo "Done."
diff --git a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
index 5b89ec7..0a2bf96 100755
--- a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
+++ b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
@@ -1,13 +1,38 @@
#!/bin/bash
+set -e
+
+if [ -e /etc/redhat-release ]; then
+ WWW_OWNER=apache:apache
+else
+ # Assume we're on a Debian-based system for now.
+ WWW_OWNER=www-data:www-data
+fi
+
+NGINX_SERVICE=${NGINX_SERVICE:-$(service --status-all 2>/dev/null \
+ | grep -Eo '\bnginx[^[:space:]]*' || true)}
+if [ -z "$NGINX_SERVICE" ]; then
+ cat >&2 <<EOF
+Error: nginx service not found. Aborting.
+Set NGINX_SERVICE to the name of the service hosting the Rails server.
+EOF
+ exit 1
+elif [ "$NGINX_SERVICE" != "$(echo "$NGINX_SERVICE" | head -n 1)" ]; then
+ cat >&2 <<EOF
+Error: multiple nginx services found. Aborting.
+Set NGINX_SERVICE to the name of the service hosting the Rails server.
+EOF
+ exit 1
+fi
+
RELEASE_PATH=/var/www/arvados-workbench/current
SHARED_PATH=/var/www/arvados-workbench/shared
CONFIG_PATH=/etc/arvados/workbench/
-echo "Assumption: nginx is configured to serve workbench.`hostname` from /var/www/workbench.`hostname`/current"
-echo "Assumption: /var/www/`hostname` is symlinked to /var/www/arvados-workbench"
+echo "Assumption: $NGINX_SERVICE is configured to serve workbench.$HOSTNAME from /var/www/workbench.$HOSTNAME/current"
+echo "Assumption: /var/www/$HOSTNAME is symlinked to /var/www/arvados-workbench"
echo "Assumption: configuration files are in /etc/arvados/workbench/"
-echo "Assumption: nginx and passenger run as the www-data user"
+echo "Assumption: $NGINX_SERVICE and passenger run as $WWW_OWNER"
echo
echo "Copying files from $CONFIG_PATH"
@@ -28,13 +53,13 @@ echo "Done."
# We do not need to precompile assets, they are already part of the package.
echo "Ensuring directory and file permissions"
-chown www-data:www-data $RELEASE_PATH/config/environment.rb
-chown www-data:www-data $RELEASE_PATH/config.ru
-chown www-data:www-data $RELEASE_PATH/config/database.yml
-chown www-data:www-data $RELEASE_PATH/Gemfile.lock
-chown -R www-data:www-data $RELEASE_PATH/tmp
-chown -R www-data:www-data $SHARED_PATH/log
-chown www-data:www-data $RELEASE_PATH/db/schema.rb
+chown "$WWW_OWNER" $RELEASE_PATH/config/environment.rb
+chown "$WWW_OWNER" $RELEASE_PATH/config.ru
+chown "$WWW_OWNER" $RELEASE_PATH/config/database.yml
+chown "$WWW_OWNER" $RELEASE_PATH/Gemfile.lock
+chown -R "$WWW_OWNER" $RELEASE_PATH/tmp
+chown -R "$WWW_OWNER" $SHARED_PATH/log
+chown "$WWW_OWNER" $RELEASE_PATH/db/schema.rb
chmod 644 $SHARED_PATH/log/*
echo "Done."
@@ -51,6 +76,6 @@ fi
# We do not need to run db:migrate because Workbench is stateless
echo "Restarting nginx"
-service nginx restart
+service "$NGINX_SERVICE" restart
echo "Done."
commit 9932d701d2ebe1d8f9d04aeaef701a1e7b616852
Author: Brett Smith <brett at curoverse.com>
Date: Sun Jul 5 11:55:25 2015 -0400
Bugfix config:check error detection in Rails upgrade scripts.
The previous code was capturing the exit status of echo, which isn't
very interesting. No issue #.
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
index cf32fdb..5af5487 100755
--- a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -45,10 +45,9 @@ chmod 644 $SHARED_PATH/log/*
echo "Done."
echo "Running sanity check"
-(cd $RELEASE_PATH; RAILS_ENV=production bundle exec rake config:check)
-echo "Done."
-
+(cd $RELEASE_PATH && RAILS_ENV=production bundle exec rake config:check)
SANITY_CHECK_EXIT_CODE=$?
+echo "Done."
if [[ "$SANITY_CHECK_EXIT_CODE" != "0" ]]; then
echo "Sanity check failed, aborting. Please roll back to the previous version of the package."
diff --git a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
index 33e05d3..5b89ec7 100755
--- a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
+++ b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
@@ -39,10 +39,9 @@ chmod 644 $SHARED_PATH/log/*
echo "Done."
echo "Running sanity check"
-(cd $RELEASE_PATH; RAILS_ENV=production bundle exec rake config:check)
-echo "Done."
-
+(cd $RELEASE_PATH && RAILS_ENV=production bundle exec rake config:check)
SANITY_CHECK_EXIT_CODE=$?
+echo "Done."
if [[ "$SANITY_CHECK_EXIT_CODE" != "0" ]]; then
echo "Sanity check failed, aborting. Please roll back to the previous version of the package."
commit e18701a42562461556c3fac9013abf76f29f44fd
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Jun 30 14:53:47 2015 -0400
6377: Don't try to uninstall arvados* gems if they are not installed.
Fixes run-tests failure when no Ruby SDK is installed on the test host
and tmpdirs are new/empty. refs #6377
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index fe33948..a646e36 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -359,7 +359,13 @@ with_test_gemset() {
if [[ "$using_rvm" == true ]]; then
"$@"
else
- GEM_HOME="$tmpdir_gem_home" "$@"
+ GEM_HOME="$tmpdir_gem_home" GEM_PATH="$tmpdir_gem_home" "$@"
+ fi
+}
+
+gem_uninstall_if_exists() {
+ if gem list "$1\$" | egrep '^\w'; then
+ gem uninstall --force --all --executables "$1"
fi
}
@@ -535,7 +541,7 @@ install_doc() {
do_install doc
install_ruby_sdk() {
- with_test_gemset gem uninstall --force --all --executables arvados \
+ with_test_gemset gem_uninstall_if_exists arvados \
&& cd "$WORKSPACE/sdk/ruby" \
&& bundle_install_trylocal \
&& gem build arvados.gemspec \
@@ -544,7 +550,7 @@ install_ruby_sdk() {
do_install sdk/ruby ruby_sdk
install_cli() {
- with_test_gemset gem uninstall --force --all --executables arvados-cli \
+ with_test_gemset gem_uninstall_if_exists arvados-cli \
&& cd "$WORKSPACE/sdk/cli" \
&& bundle_install_trylocal \
&& gem build arvados-cli.gemspec \
commit d8c6d08d57863faf78b2b582950dd8997f83a811
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Jun 30 14:50:50 2015 -0400
6377: Create user-specified tmpdirs if necessary. refs #6377
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 38290f7..fe33948 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -287,8 +287,9 @@ for tmpdir in VENVDIR VENV3DIR GOPATH GEMHOME
do
if [[ -n "${!tmpdir}" ]]; then
leave_temp[$tmpdir]=1
+ mkdir -p "${!tmpdir}"
else
- eval $tmpdir=$(mktemp -d)
+ eval "$tmpdir"='$(mktemp -d)'
fi
done
commit 264344dc26e402ac477c34f90ef1a8f52dc4ebab
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Jun 29 14:18:08 2015 -0400
Add rsync to the repo update command in run-build-packages.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index dca4c25..0eb2244 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -95,7 +95,7 @@ case "$TARGET" in
debian7)
FORMAT=deb
FPM_OUTDIR=tmp
- REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rm -f *deb'
+ REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rsync -AaHX /var/www/freight/ /var/www/freight-production/ --delete && rm -f *deb'
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON2_PKG_PREFIX=python
@@ -111,7 +111,7 @@ case "$TARGET" in
ubuntu1204)
FORMAT=deb
FPM_OUTDIR=tmp
- REPO_UPDATE_CMD='freight add *deb apt/precise && freight cache && rm -f *deb'
+ REPO_UPDATE_CMD='freight add *deb apt/precise && freight cache && rsync -AaHX /var/www/freight/ /var/www/freight-production/ --delete && rm -f *deb'
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON2_PKG_PREFIX=python
commit a0c5e16c9709e4785c92489bc5347e8fbd8f3b5e
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 26 17:09:03 2015 -0400
Make sure to also build a backported package for python-llfuse. We need
it for writeable fuse.
refs #3198
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index df0a852..dca4c25 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -105,7 +105,7 @@ case "$TARGET" in
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname pycurl)
+ ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests)
;;
ubuntu1204)
@@ -121,7 +121,7 @@ case "$TARGET" in
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname pycurl)
+ ciso8601 pycrypto backports.ssl_match_hostname pycurl llfuse)
PYTHON3_BACKPORTS=(docker-py six requests)
;;
centos6)
commit 0e301da832ce612d42488c2ad69addf1f44ee85f
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 26 15:41:47 2015 -0400
Update comment references from 'debs' to 'packages'.
refs #6389
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index d76fa07..df0a852 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -237,7 +237,7 @@ handle_python_package () {
fi
}
-# Build debs for everything
+# Build packages for everything
fpm_build_and_scp () {
# The package source. Depending on the source type, this can be a
# path, or the name of the package in an upstream repository (e.g.,
@@ -316,7 +316,7 @@ fpm_build_and_scp () {
fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
}
-# verify build results and scp debs, if needed
+# verify build results and scp packages, if needed
fpm_verify_and_scp () {
FPM_EXIT_CODE=$1
shift
commit 4044a6363ba1ccebc98369dfa70052961fcf1117
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 26 15:40:26 2015 -0400
Drop built packages into packages/$TARGET instead of the old 'debs' directory.
Also, fix the 'find' test so that built packages are actually added to
the repository after upload.
refs #6389
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index fd5a385..d76fa07 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -368,8 +368,8 @@ if [[ "$DEBUG" != 0 ]]; then
echo "umask is" `umask`
fi
-if [[ ! -d "$WORKSPACE/debs" ]]; then
- mkdir -p $WORKSPACE/debs
+if [[ ! -d "$WORKSPACE/packages/$TARGET" ]]; then
+ mkdir -p $WORKSPACE/packages/$TARGET
fi
# Perl packages
@@ -396,7 +396,7 @@ perl Makefile.PL INSTALL_BASE=install >"$PERL_OUT" && \
make install INSTALLDIRS=perl >"$PERL_OUT" && \
fpm_build_and_scp install/lib/=/usr/share libarvados-perl \
"Curoverse, Inc." dir "$(version_from_git)" install/man/=/usr/share/man && \
- mv libarvados-perl*.$FORMAT "$WORKSPACE/debs/"
+ mv libarvados-perl*.$FORMAT "$WORKSPACE/packages/$TARGET/"
# Ruby gems
if [[ "$DEBUG" != 0 ]]; then
@@ -519,7 +519,7 @@ git log --format=format:%H -n1 . > git-commit.version
# Build arvados src deb package
cd "$WORKSPACE"
PKG_VERSION=$(version_from_git)
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
fpm_build_and_scp $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
# clean up, check out master and step away from detached-head state
@@ -539,7 +539,7 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keepstore"
PKG_VERSION=$(version_from_git)
go get "git.curoverse.com/arvados.git/services/keepstore"
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
fpm_build_and_scp $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
# Get GO SDK version
@@ -559,7 +559,7 @@ else
fi
go get "git.curoverse.com/arvados.git/services/keepproxy"
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
fpm_build_and_scp $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
# datamanager
@@ -574,7 +574,7 @@ else
fi
go get "git.curoverse.com/arvados.git/services/datamanager"
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
fpm_build_and_scp $GOPATH/bin/datamanager=/usr/bin/arvados-data-manager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
# arv-git-httpd
@@ -589,14 +589,14 @@ else
fi
go get "git.curoverse.com/arvados.git/services/arv-git-httpd"
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
fpm_build_and_scp $GOPATH/bin/arv-git-httpd=/usr/bin/arvados-git-httpd arvados-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
# crunchstat
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
PKG_VERSION=$(version_from_git)
go get "git.curoverse.com/arvados.git/services/crunchstat"
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
fpm_build_and_scp $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
# The Python SDK
@@ -606,21 +606,21 @@ fpm_build_and_scp $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curover
# prefix from only one of the dependencies of a package... Maybe I could
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
fpm_build_and_scp $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
fpm_build_and_scp $WORKSPACE/services/fuse "${PYTHON2_PKG_PREFIX}-arvados-fuse" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
# The node manager
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
# The Docker image cleaner
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
fpm_build_and_scp $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curoverse, Inc.' 'python3' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/dockercleaner/arvados_docker_cleaner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Docker image cleaner"
# A few dependencies
@@ -655,7 +655,7 @@ fi
/usr/bin/git rev-parse HEAD > git-commit.version
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
# Annoyingly, we require a database.yml file for rake assets:precompile to work. So for now,
# we do that in the upgrade script.
@@ -729,7 +729,7 @@ if [[ "$?" != "0" ]]; then
EXITCODE=1
fi
-cd $WORKSPACE/debs
+cd $WORKSPACE/packages/$TARGET
# This is the complete package with vendor/bundle included.
# It's big, so we do not build it by default.
@@ -767,7 +767,7 @@ fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
# Finally, publish the packages, if necessary
if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
ssh -p2222 $SCPUSER@$SCPHOST -t bash - <<EOF
-if [ -n "\$(find -name "$FPM_OUTDIR/*.$FORMAT" -print -quit)" ]; then
+if [ -n "\$(find $FPM_OUTDIR -name "*.$FORMAT" -print -quit)" ]; then
cd "$FPM_OUTDIR" && $REPO_UPDATE_CMD
fi
EOF
commit f5d864c9681464e9984c92ccfd72189b5985d8b2
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 26 15:04:35 2015 -0400
Add ubuntu1204 target to run-build-packages.sh
refs #6389
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 37dc5a4..fd5a385 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -108,6 +108,22 @@ case "$TARGET" in
ciso8601 pycrypto backports.ssl_match_hostname pycurl)
PYTHON3_BACKPORTS=(docker-py six requests)
;;
+ ubuntu1204)
+ FORMAT=deb
+ FPM_OUTDIR=tmp
+ REPO_UPDATE_CMD='freight add *deb apt/precise && freight cache && rm -f *deb'
+
+ PYTHON2_PACKAGE=python$PYTHON2_VERSION
+ PYTHON2_PKG_PREFIX=python
+ PYTHON3_PACKAGE=python$PYTHON3_VERSION
+ PYTHON3_PKG_PREFIX=python3
+ PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
+ oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
+ rsa uritemplate httplib2 ws4py \
+ virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
+ ciso8601 pycrypto backports.ssl_match_hostname pycurl)
+ PYTHON3_BACKPORTS=(docker-py six requests)
+ ;;
centos6)
FORMAT=rpm
FPM_OUTDIR=rpm
commit 598d191de7ae6033ababaefcfcfc973e13ccdcbe
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 26 14:55:22 2015 -0400
Regex match fixes in the character class match for $FPM_RESULTS:
a) we want to match a literal dot, not any character
b) bash 4.2 does not support escaping a dash to include it in the
character class, move the dash to the last character listed in the class
refs #6389
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index b2c7ce3..37dc5a4 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -307,7 +307,7 @@ fpm_verify_and_scp () {
FPM_RESULTS=$@
FPM_PACKAGE_NAME=''
- if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\-.]*\.)(deb|rpm) ]]; then
+ if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\.-]*\.)(deb|rpm) ]]; then
FPM_PACKAGE_NAME=${BASH_REMATCH[1]}${BASH_REMATCH[2]}
fi
commit 5cb61be1c7a900d9d872c8b037f158e3c10f40bc
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jun 26 09:12:22 2015 -0400
6388: CentOS needs backports for python-daemon and its dependencies.
Refs #6388.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 6ebef74..b2c7ce3 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -122,7 +122,7 @@ case "$TARGET" in
rsa uritemplate httplib2 ws4py \
pykka apache-libcloud requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl
- daemon llfuse)
+ python-daemon lockfile llfuse)
PYTHON3_BACKPORTS=(docker-py six requests)
export PYCURL_SSL_LIBRARY=nss
;;
commit ca279f2c28d2c7130b3fba8df28fc6c86d559d1b
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jun 25 23:39:43 2015 -0400
Additionally build and upload cwl-runner package. no issue #
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
index 730261b..e76ad55 100755
--- a/jenkins/run-cwl-tests.sh
+++ b/jenkins/run-cwl-tests.sh
@@ -170,6 +170,10 @@ popd
cd reference
handle_python_package
+cd cwl-runner
+handle_python_package
+cd ..
+
./build-cwl-docker.sh
if [[ "$UPLOAD_DOCKER" != 0 ]]; then
commit 6dc673b966cf35cd390b2fa1d139fa31a59788e2
Author: Nico Cesar <nico at curoverse.com>
Date: Thu Jun 25 15:53:46 2015 -0400
login patch because DockerHub is not friendly
no issue #
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index 6bf3859..eff75ac 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -151,6 +151,11 @@ if [[ "$ECODE" != "0" ]]; then
title "upload arvados images SKIPPED because build failed"
else
if [[ $upload == true ]]; then
+ ## 20150526 nico -- *sometimes* dockerhub needs re-login
+ ## even though credentials are already in .dockercfg
+ docker login -u arvados
+
+
docker_push arvados/api
docker_push arvados/compute
docker_push arvados/doc
commit 3eecec6dc8eea0765a0821526255f682daa3f321
Merge: 76edfd7 c6de699
Author: Brett Smith <brett at curoverse.com>
Date: Thu Jun 25 15:33:01 2015 -0400
Merge branch '6388-python-backports-wip'
Refs #6388. Closes #6425.
commit c6de6999ba87c75f2856953b908ddec4af404c99
Author: Brett Smith <brett at curoverse.com>
Date: Thu Jun 25 15:22:14 2015 -0400
6388: Improve portability of Python backports in run-build-packages.
* Prefix package names with the same prefix used by the interpreter
package, to avoid conflicts with system packages when we're using
something like Software Collections.
* Always specify a dependency on that package.
* Exclude more test paths from packages to avoid conflicts.
* Do a little name munging to make sure that output package names
match dependencies generated by fpm.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 0b3fe75..6ebef74 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -98,7 +98,9 @@ case "$TARGET" in
REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rm -f *deb'
PYTHON2_PACKAGE=python$PYTHON2_VERSION
+ PYTHON2_PKG_PREFIX=python
PYTHON3_PACKAGE=python$PYTHON3_VERSION
+ PYTHON3_PKG_PREFIX=python3
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
@@ -112,7 +114,9 @@ case "$TARGET" in
REPO_UPDATE_CMD='mv *rpm /var/www/rpm.arvados.org/CentOS/6/os/x86_64/ && createrepo /var/www/rpm.arvados.org/CentOS/6/os/x86_64/'
PYTHON2_PACKAGE=$(rpm -qf "$(which python$PYTHON2_VERSION)" --queryformat '%{NAME}\n')
+ PYTHON2_PKG_PREFIX=$PYTHON2_PACKAGE
PYTHON3_PACKAGE=$(rpm -qf "$(which python$PYTHON3_VERSION)" --queryformat '%{NAME}\n')
+ PYTHON3_PKG_PREFIX=$PYTHON3_PACKAGE
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
@@ -243,7 +247,9 @@ fpm_build_and_scp () {
# All Arvados Python2 packages depend on Python 2.7.
# Make sure we build with that for consistency.
set -- "$@" --python-bin python2.7 \
- --python-easyinstall "$EASY_INSTALL2"
+ --python-easyinstall "$EASY_INSTALL2" \
+ --python-package-name-prefix "$PYTHON2_PKG_PREFIX" \
+ --depends "$PYTHON2_PACKAGE"
;;
python3)
# fpm does not actually support a python3 package type. Instead
@@ -253,11 +259,15 @@ fpm_build_and_scp () {
PACKAGE_TYPE=python
set -- "$@" --python-bin python3 \
--python-easyinstall "$EASY_INSTALL3" \
- --python-package-name-prefix python3 --depends "$PYTHON3_PACKAGE"
+ --python-package-name-prefix "$PYTHON3_PKG_PREFIX" \
+ --depends "$PYTHON3_PACKAGE"
;;
esac
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "$FORMAT" "-x" "usr/local/lib/python2.7/dist-packages/tests")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "$FORMAT")
+ if [ python = "$PACKAGE_TYPE" ]; then
+ COMMAND_ARR+=(--exclude=\*/{dist,site}-packages/tests/\*)
+ fi
if [[ "$PACKAGE_NAME" != "$PACKAGE" ]]; then
COMMAND_ARR+=('-n' "$PACKAGE_NAME")
@@ -581,17 +591,17 @@ fpm_build_and_scp $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curover
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
cd $WORKSPACE/debs
-fpm_build_and_scp $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --depends="$PYTHON2_PACKAGE"
+fpm_build_and_scp $WORKSPACE/sdk/python "${PYTHON2_PKG_PREFIX}-arvados-python-client" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
cd $WORKSPACE/debs
-fpm_build_and_scp $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver" --depends="$PYTHON2_PACKAGE"
+fpm_build_and_scp $WORKSPACE/services/fuse "${PYTHON2_PKG_PREFIX}-arvados-fuse" 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
# The node manager
cd $WORKSPACE/debs
-fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager" --depends="$PYTHON2_PACKAGE"
+fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
# The Docker image cleaner
cd $WORKSPACE/debs
@@ -599,13 +609,15 @@ fpm_build_and_scp $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curo
# A few dependencies
for deppkg in "${PYTHON_BACKPORTS[@]}"; do
- fpm_build_and_scp "$deppkg"
+ outname=$(echo "$deppkg" | sed -e 's/^python-//' -e 's/[<=>].*//' -e 's/_/-/g' -e "s/^/${PYTHON2_PKG_PREFIX}-/")
+ fpm_build_and_scp "$deppkg" "$outname"
done
# Python 3 dependencies
for deppkg in "${PYTHON3_BACKPORTS[@]}"; do
+ outname=$(echo "$deppkg" | sed -e 's/^python-//' -e 's/[<=>].*//' -e 's/_/-/g' -e "s/^/${PYTHON3_PKG_PREFIX}-/")
# The empty string is the vendor argument: these aren't Curoverse software.
- fpm_build_and_scp "$deppkg" "python3-$deppkg" "" python3
+ fpm_build_and_scp "$deppkg" "$outname" "" python3
done
# Build the API server package
commit f4fcedfae4edf9c38cf9becb228f3956f2812448
Author: Brett Smith <brett at curoverse.com>
Date: Thu Jun 25 15:19:22 2015 -0400
6388: CentOS 6 needs Python daemon and llfuse backported.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 644ebd4..0b3fe75 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -117,7 +117,8 @@ case "$TARGET" in
oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
rsa uritemplate httplib2 ws4py \
pykka apache-libcloud requests six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname pycurl)
+ ciso8601 pycrypto backports.ssl_match_hostname pycurl
+ daemon llfuse)
PYTHON3_BACKPORTS=(docker-py six requests)
export PYCURL_SSL_LIBRARY=nss
;;
commit 0b1b571382be1fbbfb4b2dbb499f0367a798e711
Author: Brett Smith <brett at curoverse.com>
Date: Thu Jun 25 15:18:58 2015 -0400
6388: Build specific backport versions for Python OAuth dependencies.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index b3b58e4..644ebd4 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -100,7 +100,8 @@ case "$TARGET" in
PYTHON2_PACKAGE=python$PYTHON2_VERSION
PYTHON3_PACKAGE=python$PYTHON3_VERSION
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
- oauth2client pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py \
+ oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
+ rsa uritemplate httplib2 ws4py \
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl)
PYTHON3_BACKPORTS=(docker-py six requests)
@@ -113,8 +114,9 @@ case "$TARGET" in
PYTHON2_PACKAGE=$(rpm -qf "$(which python$PYTHON2_VERSION)" --queryformat '%{NAME}\n')
PYTHON3_PACKAGE=$(rpm -qf "$(which python$PYTHON3_VERSION)" --queryformat '%{NAME}\n')
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
- oauth2client pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py \
- virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
+ oauth2client pyasn1==0.1.7 pyasn1-modules==0.0.5 \
+ rsa uritemplate httplib2 ws4py \
+ pykka apache-libcloud requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl)
PYTHON3_BACKPORTS=(docker-py six requests)
export PYCURL_SSL_LIBRARY=nss
commit 0181b121e163c32c932aa4585ae668d50584281f
Author: Brett Smith <brett at curoverse.com>
Date: Thu Jun 25 15:16:57 2015 -0400
6388: Build PyCURL with NSS on CentOS 6.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 7f4dd29..b3b58e4 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -117,6 +117,7 @@ case "$TARGET" in
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
ciso8601 pycrypto backports.ssl_match_hostname pycurl)
PYTHON3_BACKPORTS=(docker-py six requests)
+ export PYCURL_SSL_LIBRARY=nss
;;
*)
echo -e "$0: Unknown target '$TARGET'.\n" >&2
commit 76edfd719d2cef94494cc4f50b02cf380f88cf99
Author: Brett Smith <brett at curoverse.com>
Date: Thu Jun 25 11:54:35 2015 -0400
6388: Remove old --apt-server switch from run-build-packages.
Refs #6388.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index b54761c..7f4dd29 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -35,7 +35,7 @@ BUILD_BUNDLE_PACKAGES=0
TARGET=debian7
PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,upload,scp-user:,scp-host:,apt-server:,build-bundle-packages,debug,target: \
+ help,upload,scp-user:,scp-host:,build-bundle-packages,debug,target: \
-- "" "$@")
if [ $? -ne 0 ]; then
exit 1
@@ -52,7 +52,7 @@ while [ $# -gt 0 ]; do
--scp-user)
SCPUSER="$2"; shift
;;
- --scp-host|--apt-server)
+ --scp-host)
SCPHOST="$2"; shift
;;
--target)
commit 121e7a814e34136e77191f2eb1951ad27a623453
Merge: 0f696af 8280738
Author: Brett Smith <brett at curoverse.com>
Date: Thu Jun 25 11:41:01 2015 -0400
Merge branch '6388-centos-packages-wip'
Refs #6388. Closes #6411.
commit 82807388a1f7dd901e619419690253a5fd58663b
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jun 24 17:47:30 2015 -0400
6388: run-build-packages introspects more Python environment.
* Look for easy_install under a few different names.
* On CentOS, query the package that provides each Python, and list
that as a dependency.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 818a528..b54761c 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -88,11 +88,17 @@ fi
declare -a PYTHON_BACKPORTS PYTHON3_BACKPORTS
+PYTHON2_VERSION=2.7
+PYTHON3_VERSION=$(python3 -c 'import sys; print("{v.major}.{v.minor}".format(v=sys.version_info))')
+
case "$TARGET" in
debian7)
FORMAT=deb
FPM_OUTDIR=tmp
REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rm -f *deb'
+
+ PYTHON2_PACKAGE=python$PYTHON2_VERSION
+ PYTHON3_PACKAGE=python$PYTHON3_VERSION
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py \
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
@@ -103,6 +109,9 @@ case "$TARGET" in
FORMAT=rpm
FPM_OUTDIR=rpm
REPO_UPDATE_CMD='mv *rpm /var/www/rpm.arvados.org/CentOS/6/os/x86_64/ && createrepo /var/www/rpm.arvados.org/CentOS/6/os/x86_64/'
+
+ PYTHON2_PACKAGE=$(rpm -qf "$(which python$PYTHON2_VERSION)" --queryformat '%{NAME}\n')
+ PYTHON3_PACKAGE=$(rpm -qf "$(which python$PYTHON3_VERSION)" --queryformat '%{NAME}\n')
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py \
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
@@ -135,15 +144,24 @@ if [[ "$?" != 0 ]]; then
exit 1
fi
-if ! easy_install3 --version >/dev/null; then
- cat >&2 <<EOF
+find_easy_install() {
+ for version_suffix in "$@"; do
+ if "easy_install$version_suffix" --version >/dev/null 2>&1; then
+ echo "easy_install$version_suffix"
+ return 0
+ fi
+ done
+ cat >&2 <<EOF
$helpmessage
-Error: easy_install3 (from python3-setuptools) not found
+Error: easy_install$1 (from Python setuptools module) not found
EOF
- exit 1
-fi
+ exit 1
+}
+
+EASY_INSTALL2=$(find_easy_install -$PYTHON2_VERSION "")
+EASY_INSTALL3=$(find_easy_install -$PYTHON3_VERSION 3)
RUN_BUILD_PACKAGES_PATH="`dirname \"$0\"`"
RUN_BUILD_PACKAGES_PATH="`( cd \"$RUN_BUILD_PACKAGES_PATH\" && pwd )`" # absolutized and normalized
@@ -221,7 +239,7 @@ fpm_build_and_scp () {
# All Arvados Python2 packages depend on Python 2.7.
# Make sure we build with that for consistency.
set -- "$@" --python-bin python2.7 \
- --python-easyinstall easy_install-2.7
+ --python-easyinstall "$EASY_INSTALL2"
;;
python3)
# fpm does not actually support a python3 package type. Instead
@@ -229,8 +247,9 @@ fpm_build_and_scp () {
# necessary arguments to fpm's command line later, after we're
# done handling positional arguments.
PACKAGE_TYPE=python
- set -- "$@" --python-bin python3 --python-easyinstall easy_install3 \
- --python-package-name-prefix python3 --depends python3
+ set -- "$@" --python-bin python3 \
+ --python-easyinstall "$EASY_INSTALL3" \
+ --python-package-name-prefix python3 --depends "$PYTHON3_PACKAGE"
;;
esac
@@ -558,17 +577,17 @@ fpm_build_and_scp $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curover
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
cd $WORKSPACE/debs
-fpm_build_and_scp $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --depends=python2.7
+fpm_build_and_scp $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --depends="$PYTHON2_PACKAGE"
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
cd $WORKSPACE/debs
-fpm_build_and_scp $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver" --depends=python2.7
+fpm_build_and_scp $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver" --depends="$PYTHON2_PACKAGE"
# The node manager
cd $WORKSPACE/debs
-fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager" --depends=python2.7
+fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager" --depends="$PYTHON2_PACKAGE"
# The Docker image cleaner
cd $WORKSPACE/debs
commit 53ad1144ce5b37d4d6e7c9cdbabddf92b483f75a
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jun 24 17:06:03 2015 -0400
6388: Fix Arvados Perl SDK package build.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 3d7e55d..818a528 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -319,6 +319,10 @@ if [[ "$DEBUG" != 0 ]]; then
echo "umask is" `umask`
fi
+if [[ ! -d "$WORKSPACE/debs" ]]; then
+ mkdir -p $WORKSPACE/debs
+fi
+
# Perl packages
if [[ "$DEBUG" != 0 ]]; then
echo -e "\nPerl packages\n"
@@ -335,14 +339,15 @@ cd "$WORKSPACE/sdk/perl"
if [[ -e Makefile ]]; then
make realclean >"$PERL_OUT"
fi
-find -maxdepth 1 \( -name 'MANIFEST*' -or -name 'libarvados-perl_*.deb' \) \
+find -maxdepth 1 \( -name 'MANIFEST*' -or -name "libarvados-perl*.$FORMAT" \) \
-delete
rm -rf install
-perl Makefile.PL >"$PERL_OUT" && \
- make install PREFIX=install INSTALLDIRS=perl >"$PERL_OUT" && \
- fpm_build_and_scp install/=/usr libarvados-perl "Curoverse, Inc." dir \
- "$(version_from_git)"
+perl Makefile.PL INSTALL_BASE=install >"$PERL_OUT" && \
+ make install INSTALLDIRS=perl >"$PERL_OUT" && \
+ fpm_build_and_scp install/lib/=/usr/share libarvados-perl \
+ "Curoverse, Inc." dir "$(version_from_git)" install/man/=/usr/share/man && \
+ mv libarvados-perl*.$FORMAT "$WORKSPACE/debs/"
# Ruby gems
if [[ "$DEBUG" != 0 ]]; then
@@ -434,10 +439,6 @@ handle_python_package
cd ../../services/nodemanager
handle_python_package
-if [[ ! -d "$WORKSPACE/debs" ]]; then
- mkdir -p $WORKSPACE/debs
-fi
-
# Arvados-src
# We use $WORKSPACE/src-build-dir as the clean directory from which to build the src package
if [[ ! -d "$WORKSPACE/src-build-dir" ]]; then
commit e5561b32e943e67eb457b8d7669e005a0338395e
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jun 24 16:44:52 2015 -0400
6388: Backport pycurl package everywhere.
We need the very latest version per #6073.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 7cece07..3d7e55d 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -96,7 +96,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py \
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname)
+ ciso8601 pycrypto backports.ssl_match_hostname pycurl)
PYTHON3_BACKPORTS=(docker-py six requests)
;;
centos6)
@@ -106,7 +106,7 @@ case "$TARGET" in
PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
oauth2client pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py \
virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
- ciso8601 pycrypto backports.ssl_match_hostname)
+ ciso8601 pycrypto backports.ssl_match_hostname pycurl)
PYTHON3_BACKPORTS=(docker-py six requests)
;;
*)
commit 2f56fdaf3a039ee87d264933fe1306c2642f81fe
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jun 24 13:54:02 2015 -0400
6388: Remove version munging from Python package builds.
This code was added to accommodate version numbers that did *not*
comply with PEP 440. We've since fixed those, and they only include
alphanumerics and dots, so we don't need to munge them.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 65515c8..7cece07 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -557,23 +557,17 @@ fpm_build_and_scp $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curover
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
cd $WORKSPACE/debs
-# Python version numbering is obscure. Strip dashes and replace them with dots
-# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
-fpm_build_and_scp $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --depends=python2.7
+fpm_build_and_scp $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --depends=python2.7
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
cd $WORKSPACE/debs
-# Python version numbering is obscure. Strip dashes and replace them with dots
-# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
-fpm_build_and_scp $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver" --depends=python2.7
+fpm_build_and_scp $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver" --depends=python2.7
# The node manager
cd $WORKSPACE/debs
-# Python version numbering is obscure. Strip dashes and replace them with dots
-# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
-fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager" --depends=python2.7
+fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager" --depends=python2.7
# The Docker image cleaner
cd $WORKSPACE/debs
commit aac6af892c9bbaeb791d62ed7470a80713e66650
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jun 24 13:51:09 2015 -0400
6388: run-build-packages records python2.7 dependency where needed.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 5b2ad29..65515c8 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -216,14 +216,23 @@ fpm_build_and_scp () {
VERSION=$1
shift
- # fpm does not actually support a python3 package type. Instead we recognize
- # it as a convenience shortcut to add several necessary arguments to
- # fpm's command line later, after we're done handling positional arguments.
- if [ "python3" = "$PACKAGE_TYPE" ]; then
- PACKAGE_TYPE=python
- set -- "$@" --python-bin python3 --python-easyinstall easy_install3 \
- --python-package-name-prefix python3 --depends python3
- fi
+ case "$PACKAGE_TYPE" in
+ python)
+ # All Arvados Python2 packages depend on Python 2.7.
+ # Make sure we build with that for consistency.
+ set -- "$@" --python-bin python2.7 \
+ --python-easyinstall easy_install-2.7
+ ;;
+ python3)
+ # fpm does not actually support a python3 package type. Instead
+ # we recognize it as a convenience shortcut to add several
+ # necessary arguments to fpm's command line later, after we're
+ # done handling positional arguments.
+ PACKAGE_TYPE=python
+ set -- "$@" --python-bin python3 --python-easyinstall easy_install3 \
+ --python-package-name-prefix python3 --depends python3
+ ;;
+ esac
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "$FORMAT" "-x" "usr/local/lib/python2.7/dist-packages/tests")
@@ -550,7 +559,7 @@ fpm_build_and_scp $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curover
cd $WORKSPACE/debs
# Python version numbering is obscure. Strip dashes and replace them with dots
# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
-fpm_build_and_scp $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
+fpm_build_and_scp $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK" --depends=python2.7
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
@@ -558,13 +567,13 @@ fpm_build_and_scp $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse,
cd $WORKSPACE/debs
# Python version numbering is obscure. Strip dashes and replace them with dots
# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
-fpm_build_and_scp $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
+fpm_build_and_scp $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver" --depends=python2.7
# The node manager
cd $WORKSPACE/debs
# Python version numbering is obscure. Strip dashes and replace them with dots
# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
-fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
+fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager" --depends=python2.7
# The Docker image cleaner
cd $WORKSPACE/debs
commit 0e13d045f78e1c78c9f2be3012c6bf06bf1ea5bd
Author: Brett Smith <brett at curoverse.com>
Date: Wed Jun 24 13:24:20 2015 -0400
6388: Add distro targets to run-build-packages.
This will provide us a single mechanism to make sure that packages are
built to run on a particular distribution: not just in terms of FPM's
output format, but also repository location, backports required, etc.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index e0584c4..5b2ad29 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -19,8 +19,8 @@ Options:
Build api server and workbench packages with vendor/bundle included
--debug
Output debug information (default: false)
---format
- Package format (default: deb)
+--target
+ Distribution to build packages for (default: debian7)
WORKSPACE=path Path to the Arvados source tree to build packages from
@@ -32,10 +32,10 @@ CALL_FREIGHT=0
DEBUG=0
UPLOAD=0
BUILD_BUNDLE_PACKAGES=0
-FORMAT=deb
+TARGET=debian7
PARSEDOPTS=$(getopt --name "$0" --longoptions \
- help,upload,scp-user:,scp-host:,apt-server:,build-bundle-packages,debug,format: \
+ help,upload,scp-user:,scp-host:,apt-server:,build-bundle-packages,debug,target: \
-- "" "$@")
if [ $? -ne 0 ]; then
exit 1
@@ -55,8 +55,8 @@ while [ $# -gt 0 ]; do
--scp-host|--apt-server)
SCPHOST="$2"; shift
;;
- --format)
- FORMAT="$2"; shift
+ --target)
+ TARGET="$2"; shift
;;
--debug)
DEBUG=1
@@ -86,7 +86,36 @@ if [[ "$UPLOAD" != '0' && ("$SCPUSER" == '' || "$SCPHOST" == '') ]]; then
exit 1
fi
-# Sanity check
+declare -a PYTHON_BACKPORTS PYTHON3_BACKPORTS
+
+case "$TARGET" in
+ debian7)
+ FORMAT=deb
+ FPM_OUTDIR=tmp
+ REPO_UPDATE_CMD='freight add *deb apt/wheezy && freight cache && rm -f *deb'
+ PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
+ oauth2client pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py \
+ virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
+ ciso8601 pycrypto backports.ssl_match_hostname)
+ PYTHON3_BACKPORTS=(docker-py six requests)
+ ;;
+ centos6)
+ FORMAT=rpm
+ FPM_OUTDIR=rpm
+ REPO_UPDATE_CMD='mv *rpm /var/www/rpm.arvados.org/CentOS/6/os/x86_64/ && createrepo /var/www/rpm.arvados.org/CentOS/6/os/x86_64/'
+ PYTHON_BACKPORTS=(python-gflags pyvcf google-api-python-client \
+ oauth2client pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py \
+ virtualenv pykka apache-libcloud requests six pyexecjs jsonschema \
+ ciso8601 pycrypto backports.ssl_match_hostname)
+ PYTHON3_BACKPORTS=(docker-py six requests)
+ ;;
+ *)
+ echo -e "$0: Unknown target '$TARGET'.\n" >&2
+ exit 1
+ ;;
+esac
+
+
if ! [[ -n "$WORKSPACE" ]]; then
echo >&2 "$helpmessage"
echo >&2
@@ -252,11 +281,7 @@ fpm_verify_and_scp () {
echo "Error building package for $1:\n $FPM_RESULTS"
else
if [[ "$UPLOAD" != 0 ]]; then
- if [[ "$FORMAT" == 'deb' ]]; then
- scp -P2222 $FPM_PACKAGE_NAME $SCPUSER@$SCPHOST:tmp/
- else
- scp -P2222 $FPM_PACKAGE_NAME $SCPUSER@$SCPHOST:rpm/
- fi
+ scp -P2222 $FPM_PACKAGE_NAME $SCPUSER@$SCPHOST:$FPM_OUTDIR/
CALL_FREIGHT=1
fi
fi
@@ -546,15 +571,12 @@ cd $WORKSPACE/debs
fpm_build_and_scp $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curoverse, Inc.' 'python3' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/dockercleaner/arvados_docker_cleaner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Docker image cleaner"
# A few dependencies
-for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
- pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
- pykka apache-libcloud requests six pyexecjs jsonschema ciso8601 \
- pycrypto backports.ssl_match_hostname; do
+for deppkg in "${PYTHON_BACKPORTS[@]}"; do
fpm_build_and_scp "$deppkg"
done
# Python 3 dependencies
-for deppkg in docker-py six requests; do
+for deppkg in "${PYTHON3_BACKPORTS[@]}"; do
# The empty string is the vendor argument: these aren't Curoverse software.
fpm_build_and_scp "$deppkg" "python3-$deppkg" "" python3
done
@@ -689,11 +711,11 @@ fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
# Finally, publish the packages, if necessary
if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
- if [[ "$FORMAT" == 'deb' ]]; then
- ssh -p2222 $SCPUSER@$SCPHOST -t "cd tmp && ls -laF *deb && freight add *deb apt/wheezy && freight cache && rm -f *deb"
- else
- ssh -p2222 $SCPUSER@$SCPHOST -t "cd rpm && ls -laF *rpm && mv *rpm /var/www/rpm.arvados.org/CentOS/6/os/x86_64/ && createrepo /var/www/rpm.arvados.org/CentOS/6/os/x86_64/"
- fi
+ ssh -p2222 $SCPUSER@$SCPHOST -t bash - <<EOF
+if [ -n "\$(find -name "$FPM_OUTDIR/*.$FORMAT" -print -quit)" ]; then
+ cd "$FPM_OUTDIR" && $REPO_UPDATE_CMD
+fi
+EOF
else
if [[ "$UPLOAD" != 0 ]]; then
echo "No new packages generated. No freight run necessary."
commit f0d1f9935eadd303d6b160ae9c349398f6691bd9
Author: Brett Smith <brett at curoverse.com>
Date: Tue Jun 23 12:10:54 2015 -0400
6388: Remove Debianisms from run-build-packages.
Make sure function and variable names don't refer to Debian when they
can also be used for Red Hat.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 2e797bf..e0584c4 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -12,9 +12,9 @@ Options:
--upload
Upload packages (default: false)
--scp-user USERNAME
- Scp user for apt server (only required when --upload is specified)
---apt-server HOSTNAME
- Apt server hostname (only required when --upload is specified)
+ scp user for repository server (only required when --upload is specified)
+--scp-host HOSTNAME
+ scp host for repository server (only required when --upload is specified)
--build-bundle-packages (default: false)
Build api server and workbench packages with vendor/bundle included
--debug
@@ -34,23 +34,29 @@ UPLOAD=0
BUILD_BUNDLE_PACKAGES=0
FORMAT=deb
-while [[ -n "$1" ]]
-do
- arg="$1"; shift
- case "$arg" in
+PARSEDOPTS=$(getopt --name "$0" --longoptions \
+ help,upload,scp-user:,scp-host:,apt-server:,build-bundle-packages,debug,format: \
+ -- "" "$@")
+if [ $? -ne 0 ]; then
+ exit 1
+fi
+
+eval set -- "$PARSEDOPTS"
+while [ $# -gt 0 ]; do
+ case "$1" in
--help)
echo >&2 "$helpmessage"
echo >&2
exit 1
;;
--scp-user)
- APTUSER="$1"; shift
+ SCPUSER="$2"; shift
;;
- --apt-server)
- APTSERVER="$1"; shift
+ --scp-host|--apt-server)
+ SCPHOST="$2"; shift
;;
--format)
- FORMAT="$1"; shift
+ FORMAT="$2"; shift
;;
--debug)
DEBUG=1
@@ -61,18 +67,21 @@ do
--build-bundle-packages)
BUILD_BUNDLE_PACKAGES=1
;;
- *)
- echo >&2 "$0: Unrecognized option: '$arg'. Try: $0 --help"
- exit 1
+ --)
+ if [ $# -gt 1 ]; then
+ echo >&2 "$0: unrecognized argument '$2'. Try: $0 --help"
+ exit 1
+ fi
;;
esac
+ shift
done
# Sanity checks
-if [[ "$UPLOAD" != '0' && ("$APTUSER" == '' || "$APTSERVER" == '') ]]; then
+if [[ "$UPLOAD" != '0' && ("$SCPUSER" == '' || "$SCPHOST" == '') ]]; then
echo >&2 "$helpmessage"
echo >&2
- echo >&2 "Error: please specify --scp-user and --apt-server if --upload is set"
+ echo >&2 "Error: please specify --scp-user and --scp-host if --upload is set"
echo >&2
exit 1
fi
@@ -158,7 +167,7 @@ handle_python_package () {
}
# Build debs for everything
-build_and_scp_deb () {
+fpm_build_and_scp () {
# The package source. Depending on the source type, this can be a
# path, or the name of the package in an upstream repository (e.g.,
# pip).
@@ -217,11 +226,11 @@ build_and_scp_deb () {
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
- verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
+ fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
}
# verify build results and scp debs, if needed
-verify_and_scp_deb () {
+fpm_verify_and_scp () {
FPM_EXIT_CODE=$1
shift
FPM_RESULTS=$@
@@ -244,9 +253,9 @@ verify_and_scp_deb () {
else
if [[ "$UPLOAD" != 0 ]]; then
if [[ "$FORMAT" == 'deb' ]]; then
- scp -P2222 $FPM_PACKAGE_NAME $APTUSER@$APTSERVER:tmp/
+ scp -P2222 $FPM_PACKAGE_NAME $SCPUSER@$SCPHOST:tmp/
else
- scp -P2222 $FPM_PACKAGE_NAME $APTUSER@$APTSERVER:rpm/
+ scp -P2222 $FPM_PACKAGE_NAME $SCPUSER@$SCPHOST:rpm/
fi
CALL_FREIGHT=1
fi
@@ -298,7 +307,7 @@ rm -rf install
perl Makefile.PL >"$PERL_OUT" && \
make install PREFIX=install INSTALLDIRS=perl >"$PERL_OUT" && \
- build_and_scp_deb install/=/usr libarvados-perl "Curoverse, Inc." dir \
+ fpm_build_and_scp install/=/usr libarvados-perl "Curoverse, Inc." dir \
"$(version_from_git)"
# Ruby gems
@@ -335,16 +344,16 @@ if [[ "$?" != "0" ]]; then
# -q appears to be broken in gem version 2.2.2
gem build arvados.gemspec -q >/dev/null 2>&1
fi
-
+
if [[ "$UPLOAD" != 0 ]]; then
# publish new gem
gem push arvados-*gem
fi
-
- build_and_scp_deb arvados-*.gem "" "Curoverse, Inc." gem "" \
+
+ fpm_build_and_scp arvados-*.gem "" "Curoverse, Inc." gem "" \
--prefix "$FPM_GEM_PREFIX"
fi
-
+
# Build arvados-cli GEM
cd "$WORKSPACE"
cd sdk/cli
@@ -359,14 +368,14 @@ gem search arvados-cli -r -a |grep -q $ARVADOS_GEM_VERSION
if [[ "$?" != "0" ]]; then
# clean up old gems
rm -f arvados-cli*gem
-
+
if [[ "$DEBUG" != 0 ]]; then
gem build arvados-cli.gemspec
else
# -q appears to be broken in gem version 2.2.2
gem build arvados-cli.gemspec -q >/dev/null
fi
-
+
if [[ "$UPLOAD" != 0 ]]; then
# publish new gem
gem push arvados-cli*gem
@@ -427,7 +436,7 @@ git log --format=format:%H -n1 . > git-commit.version
cd "$WORKSPACE"
PKG_VERSION=$(version_from_git)
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
+fpm_build_and_scp $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
# clean up, check out master and step away from detached-head state
cd "$WORKSPACE/src-build-dir"
@@ -447,7 +456,7 @@ cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keepstore"
PKG_VERSION=$(version_from_git)
go get "git.curoverse.com/arvados.git/services/keepstore"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
+fpm_build_and_scp $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
# Get GO SDK version
cd "$GOPATH/src/git.curoverse.com/arvados.git/sdk/go"
@@ -467,7 +476,7 @@ fi
go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
+fpm_build_and_scp $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
# datamanager
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/datamanager"
@@ -482,7 +491,7 @@ fi
go get "git.curoverse.com/arvados.git/services/datamanager"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/datamanager=/usr/bin/arvados-data-manager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
+fpm_build_and_scp $GOPATH/bin/datamanager=/usr/bin/arvados-data-manager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
# arv-git-httpd
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/arv-git-httpd"
@@ -497,14 +506,14 @@ fi
go get "git.curoverse.com/arvados.git/services/arv-git-httpd"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/arv-git-httpd=/usr/bin/arvados-git-httpd arvados-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
+fpm_build_and_scp $GOPATH/bin/arv-git-httpd=/usr/bin/arvados-git-httpd arvados-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
# crunchstat
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
PKG_VERSION=$(version_from_git)
go get "git.curoverse.com/arvados.git/services/crunchstat"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
+fpm_build_and_scp $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
# The Python SDK
# Please resist the temptation to add --no-python-fix-name to the fpm call here
@@ -516,7 +525,7 @@ build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curover
cd $WORKSPACE/debs
# Python version numbering is obscure. Strip dashes and replace them with dots
# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
-build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
+fpm_build_and_scp $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
# The FUSE driver
# Please see comment about --no-python-fix-name above; we stay consistent and do
@@ -524,30 +533,30 @@ build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse,
cd $WORKSPACE/debs
# Python version numbering is obscure. Strip dashes and replace them with dots
# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
-build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
+fpm_build_and_scp $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
# The node manager
cd $WORKSPACE/debs
# Python version numbering is obscure. Strip dashes and replace them with dots
# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
-build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
+fpm_build_and_scp $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
# The Docker image cleaner
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curoverse, Inc.' 'python3' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/dockercleaner/arvados_docker_cleaner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Docker image cleaner"
+fpm_build_and_scp $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curoverse, Inc.' 'python3' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/dockercleaner/arvados_docker_cleaner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Docker image cleaner"
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
pykka apache-libcloud requests six pyexecjs jsonschema ciso8601 \
pycrypto backports.ssl_match_hostname; do
- build_and_scp_deb "$deppkg"
+ fpm_build_and_scp "$deppkg"
done
# Python 3 dependencies
for deppkg in docker-py six requests; do
# The empty string is the vendor argument: these aren't Curoverse software.
- build_and_scp_deb "$deppkg" "python3-$deppkg" "" python3
+ fpm_build_and_scp "$deppkg" "python3-$deppkg" "" python3
done
# Build the API server package
@@ -590,7 +599,7 @@ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
- verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
+ fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
fi
# Build the 'bare' package without vendor/bundle.
@@ -604,7 +613,7 @@ fi
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
-verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
+fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
# API server package build done
@@ -659,7 +668,7 @@ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
- verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
+ fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
fi
# Build the 'bare' package without vendor/bundle.
@@ -674,16 +683,16 @@ fi
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
-verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
+fpm_verify_and_scp $FPM_EXIT_CODE $FPM_RESULTS
# Workbench package build done
# Finally, publish the packages, if necessary
if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
if [[ "$FORMAT" == 'deb' ]]; then
- ssh -p2222 $APTUSER@$APTSERVER -t "cd tmp && ls -laF *deb && freight add *deb apt/wheezy && freight cache && rm -f *deb"
+ ssh -p2222 $SCPUSER@$SCPHOST -t "cd tmp && ls -laF *deb && freight add *deb apt/wheezy && freight cache && rm -f *deb"
else
- ssh -p2222 $APTUSER@$APTSERVER -t "cd rpm && ls -laF *rpm && mv *rpm /var/www/rpm.arvados.org/CentOS/6/os/x86_64/ && createrepo /var/www/rpm.arvados.org/CentOS/6/os/x86_64/"
+ ssh -p2222 $SCPUSER@$SCPHOST -t "cd rpm && ls -laF *rpm && mv *rpm /var/www/rpm.arvados.org/CentOS/6/os/x86_64/ && createrepo /var/www/rpm.arvados.org/CentOS/6/os/x86_64/"
fi
else
if [[ "$UPLOAD" != 0 ]]; then
commit 0f696afd56780250b3f2828b875fb7ea8e4ebe81
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 12 20:45:38 2015 -0400
Fix asset generation in the workbench package. Make sure the build step will error out if asset generation fails for whatever reason.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index e6a89b4..2e797bf 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -632,8 +632,17 @@ fi
# from the package - empty it instead.
rm -rf $WORKSPACE/apps/workbench/tmp/*
+# Set up application.yml so that asset precompilation works
+\cp config/application.yml.example config/application.yml -f
+sed -i 's/secret_token: ~/secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/' config/application.yml
+
RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile >/dev/null
+if [[ "$?" != "0" ]]; then
+ echo "ERROR: Asset precompilation failed"
+ EXITCODE=1
+fi
+
cd $WORKSPACE/debs
# This is the complete package with vendor/bundle included.
commit 591ddfb6f369bd37bf8218646219f5f3fcc561bd
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 12 14:49:30 2015 -0400
Do not build and upload gems that already exist on rubygems.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 8ad5649..e6a89b4 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -316,41 +316,61 @@ fi
cd "$WORKSPACE"
cd sdk/ruby
-# clean up old packages
-find -maxdepth 1 \( -name 'arvados-*.gem' -or -name 'rubygem-arvados_*.deb' \) \
- -delete
-if [[ "$DEBUG" != 0 ]]; then
- gem build arvados.gemspec
-else
- # -q appears to be broken in gem version 2.2.2
- gem build arvados.gemspec -q >/dev/null 2>&1
-fi
+ARVADOS_GEM_EPOCH=`git log -n1 --first-parent --format=%ct`
+ARVADOS_GEM_DATE=`date --utc --date="@${ARVADOS_GEM_EPOCH}" +%Y%m%d%H%M%S`
+ARVADOS_GEM_VERSION="0.1.${ARVADOS_GEM_DATE}"
-if [[ "$UPLOAD" != 0 ]]; then
- # publish new gem
- gem push arvados-*gem
-fi
+# see if this gem needs building/uploading
+gem search arvados -r -a |grep -q $ARVADOS_GEM_VERSION
-build_and_scp_deb arvados-*.gem "" "Curoverse, Inc." gem "" \
- --prefix "$FPM_GEM_PREFIX"
+if [[ "$?" != "0" ]]; then
+ # clean up old packages
+ find -maxdepth 1 \( -name 'arvados-*.gem' -or -name 'rubygem-arvados_*.deb' -or -name 'rubygem-arvados_*.rpm' \) \
+ -delete
+ if [[ "$DEBUG" != 0 ]]; then
+ gem build arvados.gemspec
+ else
+ # -q appears to be broken in gem version 2.2.2
+ gem build arvados.gemspec -q >/dev/null 2>&1
+ fi
+
+ if [[ "$UPLOAD" != 0 ]]; then
+ # publish new gem
+ gem push arvados-*gem
+ fi
+
+ build_and_scp_deb arvados-*.gem "" "Curoverse, Inc." gem "" \
+ --prefix "$FPM_GEM_PREFIX"
+fi
+
# Build arvados-cli GEM
cd "$WORKSPACE"
cd sdk/cli
-# clean up old gems
-rm -f arvados-cli*gem
-if [[ "$DEBUG" != 0 ]]; then
- gem build arvados-cli.gemspec
-else
- # -q appears to be broken in gem version 2.2.2
- gem build arvados-cli.gemspec -q >/dev/null
-fi
+ARVADOS_CLI_GEM_EPOCH=`git log -n1 --first-parent --format=%ct`
+ARVADOS_CLI_GEM_DATE=`date --utc --date="@${ARVADOS_CLI_GEM_EPOCH}" +%Y%m%d%H%M%S`
+ARVADOS_CLI_GEM_VERSION="0.1.${ARVADOS_CLI_GEM_DATE}"
+
+# see if this gem needs building/uploading
+gem search arvados-cli -r -a |grep -q $ARVADOS_GEM_VERSION
-if [[ "$UPLOAD" != 0 ]]; then
- # publish new gem
- gem push arvados-cli*gem
+if [[ "$?" != "0" ]]; then
+ # clean up old gems
+ rm -f arvados-cli*gem
+
+ if [[ "$DEBUG" != 0 ]]; then
+ gem build arvados-cli.gemspec
+ else
+ # -q appears to be broken in gem version 2.2.2
+ gem build arvados-cli.gemspec -q >/dev/null
+ fi
+
+ if [[ "$UPLOAD" != 0 ]]; then
+ # publish new gem
+ gem push arvados-cli*gem
+ fi
fi
# Python packages
commit c159d1d0032ce82789e0725b9a50c48ef634ecd4
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 12 11:07:44 2015 -0400
Add very rudimentary support for pushing the rpm packages into a yum
repository.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index f7f74e9..8ad5649 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -240,10 +240,14 @@ verify_and_scp_deb () {
else
if [[ ! $FPM_RESULTS =~ "File already exists" ]]; then
if [[ "$FPM_EXIT_CODE" != "0" ]]; then
- echo "Error building debian package for $1:\n $FPM_RESULTS"
+ echo "Error building package for $1:\n $FPM_RESULTS"
else
if [[ "$UPLOAD" != 0 ]]; then
- scp -P2222 $FPM_PACKAGE_NAME $APTUSER@$APTSERVER:tmp/
+ if [[ "$FORMAT" == 'deb' ]]; then
+ scp -P2222 $FPM_PACKAGE_NAME $APTUSER@$APTSERVER:tmp/
+ else
+ scp -P2222 $FPM_PACKAGE_NAME $APTUSER@$APTSERVER:rpm/
+ fi
CALL_FREIGHT=1
fi
fi
@@ -647,7 +651,11 @@ verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
# Finally, publish the packages, if necessary
if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
- ssh -p2222 $APTUSER@$APTSERVER -t "cd tmp && ls -laF *deb && freight add *deb apt/wheezy && freight cache && rm -f *deb"
+ if [[ "$FORMAT" == 'deb' ]]; then
+ ssh -p2222 $APTUSER@$APTSERVER -t "cd tmp && ls -laF *deb && freight add *deb apt/wheezy && freight cache && rm -f *deb"
+ else
+ ssh -p2222 $APTUSER@$APTSERVER -t "cd rpm && ls -laF *rpm && mv *rpm /var/www/rpm.arvados.org/CentOS/6/os/x86_64/ && createrepo /var/www/rpm.arvados.org/CentOS/6/os/x86_64/"
+ fi
else
if [[ "$UPLOAD" != 0 ]]; then
echo "No new packages generated. No freight run necessary."
commit 38f341f93f21a6ecd8c27620825c57a093677330
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 12 10:10:11 2015 -0400
Add rpm generation support to run-build-packages.sh
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 63a31cc..f7f74e9 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -19,6 +19,8 @@ Options:
Build api server and workbench packages with vendor/bundle included
--debug
Output debug information (default: false)
+--format
+ Package format (default: deb)
WORKSPACE=path Path to the Arvados source tree to build packages from
@@ -30,6 +32,7 @@ CALL_FREIGHT=0
DEBUG=0
UPLOAD=0
BUILD_BUNDLE_PACKAGES=0
+FORMAT=deb
while [[ -n "$1" ]]
do
@@ -46,6 +49,9 @@ do
--apt-server)
APTSERVER="$1"; shift
;;
+ --format)
+ FORMAT="$1"; shift
+ ;;
--debug)
DEBUG=1
;;
@@ -181,7 +187,7 @@ build_and_scp_deb () {
--python-package-name-prefix python3 --depends python3
fi
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "deb" "-x" "usr/local/lib/python2.7/dist-packages/tests")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "$FORMAT" "-x" "usr/local/lib/python2.7/dist-packages/tests")
if [[ "$PACKAGE_NAME" != "$PACKAGE" ]]; then
COMMAND_ARR+=('-n' "$PACKAGE_NAME")
@@ -221,8 +227,8 @@ verify_and_scp_deb () {
FPM_RESULTS=$@
FPM_PACKAGE_NAME=''
- if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\-.]*\.deb) ]]; then
- FPM_PACKAGE_NAME=${BASH_REMATCH[1]}
+ if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\-.]*\.)(deb|rpm) ]]; then
+ FPM_PACKAGE_NAME=${BASH_REMATCH[1]}${BASH_REMATCH[2]}
fi
if [[ "$FPM_PACKAGE_NAME" == "" ]]; then
@@ -242,7 +248,7 @@ verify_and_scp_deb () {
fi
fi
else
- echo "Debian package $FPM_PACKAGE_NAME exists, not rebuilding"
+ echo "Package $FPM_PACKAGE_NAME exists, not rebuilding"
fi
fi
}
@@ -550,7 +556,7 @@ cd $WORKSPACE/debs
# This is the complete package with vendor/bundle included.
# It's big, so we do not build it by default.
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -564,7 +570,7 @@ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
fi
# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -610,7 +616,7 @@ cd $WORKSPACE/debs
# It's big, so we do not build it by default.
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -625,7 +631,7 @@ fi
# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "$FORMAT" "-n" "${PACKAGE_NAME}" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
commit f288c6cc7eebf112c73e909cbf7b74318cbf884c
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jun 10 21:11:35 2015 -0400
Store the full git hash of the checkout in the arvados-src package, in a file called git-commit.version
refs #6096
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index eba950d..63a31cc 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -391,6 +391,8 @@ else
git checkout -q `git log --format=format:%h -n1 .`
fi
+git log --format=format:%H -n1 . > git-commit.version
+
# Build arvados src deb package
cd "$WORKSPACE"
PKG_VERSION=$(version_from_git)
commit c5dfc8a4b24c4da0a83e8bd759b361b6d1080955
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jun 10 14:43:59 2015 -0400
CWL script won't fail when there are no documentation changes to commit. no issue #
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
index e65fda1..730261b 100755
--- a/jenkins/run-cwl-tests.sh
+++ b/jenkins/run-cwl-tests.sh
@@ -200,5 +200,5 @@ fi
python -mcwltool specification/cwlsite.cwl specification/cwlsite-job.json --outdir=$PWD/common-workflow-language.github.io
cd common-workflow-language.github.io
git add --all
-git commit -m"Build bot"
+git diff-index --quiet HEAD || git commit -m"Build bot"
git push
commit 84cfa56ae6bff1649b73a63c8f23d20a5a87f9ff
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jun 10 10:49:18 2015 -0400
Set default identity for cwl repo pushes. no issue #
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
index 1a15e56..e65fda1 100755
--- a/jenkins/run-cwl-tests.sh
+++ b/jenkins/run-cwl-tests.sh
@@ -192,6 +192,10 @@ if test -d common-workflow-language.github.io ; then
cd ..
else
git clone git at github.com:common-workflow-language/common-workflow-language.github.io.git
+ cd common-workflow-language.github.io
+ git config user.email "sysadmin at curoverse.com"
+ git config user.name "Curoverse build bot"
+ cd ..
fi
python -mcwltool specification/cwlsite.cwl specification/cwlsite-job.json --outdir=$PWD/common-workflow-language.github.io
cd common-workflow-language.github.io
commit ef45c8676d34405a2c99ffa51c258b8cb7b6777a
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Jun 10 10:28:01 2015 -0400
Since jenkins reuses working directories, handle updating existing git repo of
common-workflow-language site that will be automatically pushed. no issue #
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
index ac7de4f..1a15e56 100755
--- a/jenkins/run-cwl-tests.sh
+++ b/jenkins/run-cwl-tests.sh
@@ -185,8 +185,15 @@ virtualenv ../venv
python setup.py install
cd ..
-git clone git at github.com:common-workflow-language/common-workflow-language.github.io.git
-python -mcwltool specification/cwlsite.cwl specification/cwlsite-job.json --outdir=common-workflow-language.github.io
+if test -d common-workflow-language.github.io ; then
+ cd common-workflow-language.github.io
+ git fetch
+ git reset --hard origin/master
+ cd ..
+else
+ git clone git at github.com:common-workflow-language/common-workflow-language.github.io.git
+fi
+python -mcwltool specification/cwlsite.cwl specification/cwlsite-job.json --outdir=$PWD/common-workflow-language.github.io
cd common-workflow-language.github.io
git add --all
git commit -m"Build bot"
commit fbce5ceda30fd45c1834d487ed882f394352ef8a
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Tue Jun 9 16:40:47 2015 -0400
Automatically build and upload documentation site. no issue #
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
index dcb0285..ac7de4f 100755
--- a/jenkins/run-cwl-tests.sh
+++ b/jenkins/run-cwl-tests.sh
@@ -177,3 +177,17 @@ if [[ "$UPLOAD_DOCKER" != 0 ]]; then
docker push commonworkflowlanguage/cwltool
docker push commonworkflowlanguage/nodejs-engine
fi
+
+# Setup virtualenv and build documentation.
+
+virtualenv ../venv
+. ../venv/bin/activate
+python setup.py install
+cd ..
+
+git clone git at github.com:common-workflow-language/common-workflow-language.github.io.git
+python -mcwltool specification/cwlsite.cwl specification/cwlsite-job.json --outdir=common-workflow-language.github.io
+cd common-workflow-language.github.io
+git add --all
+git commit -m"Build bot"
+git push
commit b9bcac3ebad592ae9a620f26b3841986c63614e8
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Jun 9 16:35:59 2015 -0400
Make the run-deploy.sh script much more general. It can now be used to
deploy any of our clusters after an upgrade of the versions in hiera.
refs #6096
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index a8aa6d5..a883b75 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -1,24 +1,50 @@
#!/bin/bash
+DEBUG=0
+
+function usage {
+ echo >&2
+ echo >&2 "usage: $0 [options] <identifier>"
+ echo >&2
+ echo >&2 " <identifier> Arvados cluster name"
+ echo >&2
+ echo >&2 "$0 options:"
+ echo >&2 " -d, --debug Enable debug output"
+ echo >&2 " -h, --help Display this help and exit"
+ echo >&2
+}
-read -rd "\000" helpmessage <<EOF
-$(basename $0): Deploy Arvados to a cluster
-
-Syntax:
- $(basename $0) <identifier>
-
-Options:
-
- identifier Arvados cluster name
-
-EOF
-
+# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
+TEMP=`getopt -o hd \
+ --long help,debug \
+ -n "$0" -- "$@"`
+
+if [ $? != 0 ] ; then echo "Use -h for help"; exit 1 ; fi
+# Note the quotes around `$TEMP': they are essential!
+eval set -- "$TEMP"
+
+while [ $# -ge 1 ]
+do
+ case $1 in
+ -d | --debug)
+ DEBUG=1
+ shift
+ ;;
+ --)
+ shift
+ break
+ ;;
+ *)
+ usage
+ exit 1
+ ;;
+ esac
+done
IDENTIFIER=$1
if [[ "$IDENTIFIER" == '' ]]; then
- echo >&2 "$helpmessage"
- echo >&2
+ usage
exit 1
fi
@@ -27,156 +53,145 @@ EXITCODE=0
COLUMNS=80
title () {
- printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
+ date=`date +'%Y-%m-%d %H:%M:%S'`
+ printf "$date $1\n"
}
-echo $WORKSPACE
-
function run_puppet() {
node=$1
return_var=$2
+ title "Running puppet on $node"
TMP_FILE=`mktemp`
- ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node.$IDENTIFIER -C "/usr/bin/puppet agent -t" | tee $TMP_FILE
+ if [[ "$DEBUG" != "0" ]]; then
+ ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C "/usr/bin/puppet agent -t" | tee $TMP_FILE
+ else
+ ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C "/usr/bin/puppet agent -t" > $TMP_FILE 2>&1
+ fi
ECODE=$?
RESULT=$(cat $TMP_FILE)
if [[ "$ECODE" != "255" && ! ("$RESULT" =~ 'already in progress') && "$ECODE" != "2" && "$ECODE" != "0" ]]; then
- # Puppet exists 255 if the connection timed out. Just ignore that, it's possible that this node is
- # a compute node that was being shut down.
+ # Ssh exits 255 if the connection timed out. Just ignore that.
# Puppet exits 2 if there are changes. For real!
# Puppet prints 'Notice: Run of Puppet configuration client already in progress' if another puppet process
# was already running
- echo "ERROR updating $node.$IDENTIFIER: exit code $ECODE"
+ echo "ERROR running puppet on $node: exit code $ECODE"
+ if [[ "$DEBUG" == "0" ]]; then
+ title "Command output follows:"
+ echo $RESULT
+ fi
+ fi
+ if [[ "$ECODE" == "255" ]]; then
+ title "Connection timed out"
+ ECODE=0
+ fi
+ if [[ "$ECODE" == "2" ]]; then
+ ECODE=0
fi
rm -f $TMP_FILE
- echo
eval "$return_var=$ECODE"
}
-# Deploy API server
-title "Deploying API server"
+function run_command() {
+ node=$1
+ return_var=$2
+ command=$3
-SUM_ECODE=0
+ title "Running '$command' on $node"
+ TMP_FILE=`mktemp`
+ if [[ "$DEBUG" != "0" ]]; then
+ ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C "$command" | tee $TMP_FILE
+ else
+ ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node -C "$command" > $TMP_FILE 2>&1
+ fi
-# Install updated debian packages
-title "Deploying updated arvados debian packages"
+ ECODE=$?
+ RESULT=$(cat $TMP_FILE)
-ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get -qqy install arvados-src python-arvados-fuse python-arvados-python-client arvados-api-server"
+ if [[ "$ECODE" != "255" && "$ECODE" != "0" ]]; then
+ # Ssh exists 255 if the connection timed out. Just ignore that, it's possible that this node is
+ # a shell node that is down.
+ title "ERROR running command on $node: exit code $ECODE"
+ if [[ "$DEBUG" == "0" ]]; then
+ title "Command output follows:"
+ echo $RESULT
+ fi
+ fi
+ if [[ "$ECODE" == "255" ]]; then
+ title "Connection timed out"
+ ECODE=0
+ fi
+ rm -f $TMP_FILE
+ eval "$return_var=$ECODE"
+}
-ECODE=$?
-SUM_ECODE=$(($SUM_ECODE + $ECODE))
+title "Loading ARVADOS_API_HOST and ARVADOS_API_TOKEN"
+if [[ -f "$HOME/.config/arvados/$IDENTIFIER.arvadosapi.com.conf" ]]; then
+ . $HOME/.config/arvados/$IDENTIFIER.arvadosapi.com.conf
+else
+ title "WARNING: $HOME/.config/arvados/$IDENTIFIER.arvadosapi.com.conf not found."
+fi
+if [[ "$ARVADOS_API_HOST" == "" ]] || [[ "$ARVADOS_API_TOKEN" == "" ]]; then
+ title "ERROR: ARVADOS_API_HOST and/or ARVADOS_API_TOKEN environment variables are not set."
+ exit 1
+fi
-ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "/usr/local/bin/arvados-api-server-upgrade.sh"
+title "Gathering list of shell and Keep nodes"
+SHELL_NODES=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv virtual_machine list |jq .items[].hostname -r`
+KEEP_NODES=`ARVADOS_API_HOST=$ARVADOS_API_HOST ARVADOS_API_TOKEN=$ARVADOS_API_TOKEN arv keep_service list |jq .items[].service_host -r`
-ECODE=$?
+title "Updating API server"
+SUM_ECODE=0
+run_puppet $IDENTIFIER ECODE
+SUM_ECODE=$(($SUM_ECODE + $ECODE))
+run_command $IDENTIFIER ECODE "/usr/local/bin/arvados-api-server-upgrade.sh"
+SUM_ECODE=$(($SUM_ECODE + $ECODE))
+run_command $IDENTIFIER ECODE "dpkg -L arvados-mailchimp-plugin 2>/dev/null && apt-get install arvados-mailchimp-plugin --reinstall || echo"
SUM_ECODE=$(($SUM_ECODE + $ECODE))
if [[ "$SUM_ECODE" != "0" ]]; then
- title "!!!!!! DEPLOYING DEBIAN PACKAGES FAILED !!!!!!"
+ title "ERROR: Updating API server FAILED"
EXITCODE=$(($EXITCODE + $SUM_ECODE))
exit $EXITCODE
fi
-title "Deploying updated arvados debian packages complete"
-
-# Install updated arvados gems
-title "Deploying updated arvados gems"
-
-ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "/usr/local/rvm/bin/rvm default do gem install arvados arvados-cli && /usr/local/rvm/bin/rvm default do gem clean arvados arvados-cli"
-
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! DEPLOYING ARVADOS GEMS FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
- exit $EXITCODE
+title "Updating workbench"
+SUM_ECODE=0
+if [[ `host workbench.$ARVADOS_API_HOST` != `host $ARVADOS_API_HOST` ]]; then
+ # Workbench runs on a separate host. We need to run puppet there too.
+ run_puppet workbench.$IDENTIFIER ECODE
+ SUM_ECODE=$(($SUM_ECODE + $ECODE))
fi
-title "Deploying updated arvados gems complete"
-title "Deploying API server complete"
-
-# Deploy Workbench
-title "Deploying workbench"
-
-# Install updated debian packages
-title "Deploying updated arvados debian packages"
-
-ssh -p2222 root at workbench.$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get -qqy install python-arvados-fuse python-arvados-python-client arvados-workbench"
-
-ECODE=$?
-SUM_ECODE=$(($SUM_ECODE + $ECODE))
-
-ssh -p2222 root at workbench.$IDENTIFIER.arvadosapi.com -C "/usr/local/bin/arvados-workbench-upgrade.sh"
-
-ECODE=$?
+run_command workbench.$IDENTIFIER ECODE "/usr/local/bin/arvados-workbench-upgrade.sh"
SUM_ECODE=$(($SUM_ECODE + $ECODE))
if [[ "$SUM_ECODE" != "0" ]]; then
- title "!!!!!! DEPLOYING DEBIAN PACKAGES FAILED !!!!!!"
+ title "ERROR: Updating workbench FAILED"
EXITCODE=$(($EXITCODE + $SUM_ECODE))
exit $EXITCODE
fi
-title "Deploying updated arvados debian packages complete"
-
-title "Deploying workbench complete"
-
-# Update compute node(s)
-title "Update compute node(s)"
-
-# Get list of nodes that are up
-COMPRESSED_NODE_LIST=`ssh -p2222 root@$IDENTIFIER -C "sinfo --long -p crypto -r -o "%N" -h"`
-
-if [[ "$COMPRESSED_NODE_LIST" != '' ]]; then
- COMPUTE_NODES=`ssh -p2222 root@$IDENTIFIER -C "scontrol show hostname $COMPRESSED_NODE_LIST"`
-
- SUM_ECODE=0
- for node in $COMPUTE_NODES; do
- echo "Updating $node.$IDENTIFIER"
- run_puppet $node ECODE
- SUM_ECODE=$(($SUM_ECODE + $ECODE))
- done
-
- if [[ "$SUM_ECODE" != "0" ]]; then
- title "!!!!!! Update compute node(s) FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $SUM_ECODE))
- fi
-fi
-
-title "Update compute node(s) complete"
-
-title "Update shell"
-
-run_puppet shell ECODE
-
-if [[ "$ECODE" == "2" ]]; then
- # Puppet exits '2' if there are changes. For real!
- ECODE=0
-fi
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! Update shell FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
-
-title "Update shell complete"
-
-title "Update keep0"
-
-run_puppet keep0 ECODE
-
-if [[ "$ECODE" == "2" ]]; then
- # Puppet exits '2' if there are changes. For real!
+for n in manage $SHELL_NODES $KEEP_NODES; do
ECODE=0
-fi
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! Update keep0 FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
+ if [[ $n =~ $ARVADOS_API_HOST$ ]]; then
+ # e.g. keep.qr1hi.arvadosapi.com
+ node=$n
+ else
+ # e.g. shell
+ node=$n.$ARVADOS_API_HOST
+ fi
-title "Update keep0 complete"
+ # e.g. keep.qr1hi
+ node=${node%.arvadosapi.com}
-exit $EXITCODE
+ title "Updating $node"
+ run_puppet $node ECODE
+ if [[ "$ECODE" != "0" ]]; then
+ title "ERROR: Updating $node node FAILED: exit code $ECODE"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ exit $EXITCODE
+ fi
+done
commit 6268a275acf7c9daf8561e4b969cfbe4e3f44df8
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Jun 9 10:01:12 2015 -0400
Fix links to workbench on run-tests.sh. Also shut up the PyYAML build warnings.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index cd6e1d0..38290f7 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -383,7 +383,7 @@ fi
# Needed for run_test_server.py which is used by certain (non-Python) tests.
pip freeze 2>/dev/null | egrep ^PyYAML= \
- || pip install PyYAML \
+ || pip install PyYAML >/dev/null \
|| fatal "pip install PyYAML failed"
# If Python 3 is available, set up its virtualenv in $VENV3DIR.
@@ -649,7 +649,7 @@ test_doclinkchecker() {
ARVADOS_API_HOST=qr1hi.arvadosapi.com
# Make sure python-epydoc is installed or the next line won't
# do much good!
- PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
+ PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=https://workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
)
}
do_test doc doclinkchecker
commit 848ccc7efab5dab22f7fedd920daffbd102d09a7
Author: Brett Smith <brett at curoverse.com>
Date: Mon Jun 8 13:38:06 2015 -0400
Fix directory when dropping API server database connections.
The Python one-liner is written with the expectation that it's in this
directory, and usually fails to find <config/database.yml> otherwise.
No issue #.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 1a699a8..cd6e1d0 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -609,8 +609,9 @@ install_apiserver() {
# Clear out any lingering postgresql connections to the test
# database, so that we can drop it. This assumes the current user
# is a postgresql superuser.
- test_database=$(python -c "import yaml; print yaml.load(file('config/database.yml'))['test']['database']")
- psql "$test_database" -c "SELECT pg_terminate_backend (pg_stat_activity.procpid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = '$test_database';" 2>/dev/null
+ cd "$WORKSPACE/services/api" \
+ && test_database=$(python -c "import yaml; print yaml.load(file('config/database.yml'))['test']['database']") \
+ && psql "$test_database" -c "SELECT pg_terminate_backend (pg_stat_activity.procpid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = '$test_database';" 2>/dev/null
cd "$WORKSPACE/services/api" \
&& RAILS_ENV=test bundle exec rake db:drop \
commit 855e16bace329e9a449fcfc569a0e8ba6acc2d21
Merge: 1a4ea52 530468d
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 5 16:29:33 2015 -0400
Merge branch '6096-package-rails-apps'
refs #6096
commit 530468d1224a7c2251ee54bf5ec8d1f8d875de14
Merge: 381b79b 1a4ea52
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 5 16:29:18 2015 -0400
Merge branch 'master' into 6096-package-rails-apps
commit 1a4ea520c247fd46f544ae83bb9640e9f0652577
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Fri Jun 5 09:51:30 2015 -0400
Fix "docker-node-engine.sh" renamed to "build-node-docker.sh", add additional Docker image to upload. no issue #
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
index 74e9999..dcb0285 100755
--- a/jenkins/run-cwl-tests.sh
+++ b/jenkins/run-cwl-tests.sh
@@ -159,7 +159,7 @@ cd "$WORKSPACE"
pushd reference
python setup.py install
python setup.py test
-./docker-node-engine.sh
+./build-node-docker.sh
popd
pushd conformance
@@ -173,6 +173,7 @@ handle_python_package
./build-cwl-docker.sh
if [[ "$UPLOAD_DOCKER" != 0 ]]; then
+ docker push commonworkflowlanguage/cwltool_module
docker push commonworkflowlanguage/cwltool
docker push commonworkflowlanguage/nodejs-engine
fi
commit 381b79bf5cfefe790bdcc24dd33296b8518e4c19
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Jun 4 12:40:56 2015 -0400
Be sure to include the pre-generated assets (tmp/cache/assets) in the
workbench packages.
refs #6096
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index c139c08..eba950d 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -595,6 +595,11 @@ fi
/usr/bin/git rev-parse HEAD > git-commit.version
+# clear the tmp directory; the asset generation step will recreate tmp/cache/assets,
+# and we want that in the package, so it's easier to not exclude the tmp directory
+# from the package - empty it instead.
+rm -rf $WORKSPACE/apps/workbench/tmp/*
+
RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile >/dev/null
cd $WORKSPACE/debs
@@ -603,7 +608,7 @@ cd $WORKSPACE/debs
# It's big, so we do not build it by default.
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -618,7 +623,7 @@ fi
# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
commit 301f4e3ad12f7ff3c08b352b05dc7886c629900e
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Jun 4 10:40:29 2015 -0400
Name change for the arvados-api-server and arvados-workbench packages - part 2.
refs #6096
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 168a16d..a8aa6d5 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -63,7 +63,7 @@ SUM_ECODE=0
# Install updated debian packages
title "Deploying updated arvados debian packages"
-ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get -qqy install arvados-src python-arvados-fuse python-arvados-python-client arvados-api-server-bare"
+ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get -qqy install arvados-src python-arvados-fuse python-arvados-python-client arvados-api-server"
ECODE=$?
SUM_ECODE=$(($SUM_ECODE + $ECODE))
@@ -103,7 +103,7 @@ title "Deploying workbench"
# Install updated debian packages
title "Deploying updated arvados debian packages"
-ssh -p2222 root at workbench.$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get -qqy install python-arvados-fuse python-arvados-python-client arvados-workbench-bare"
+ssh -p2222 root at workbench.$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get -qqy install python-arvados-fuse python-arvados-python-client arvados-workbench"
ECODE=$?
SUM_ECODE=$(($SUM_ECODE + $ECODE))
commit 05ee30be7534c9234ccf029076563ccc7e7a8189
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Jun 4 10:16:38 2015 -0400
Name change for the arvados-api-server and arvados-workbench packages. The slim versions are now just
arvados-api-server
arvados-workbench
and the bigger version of the package that includes vendor/bundle is called
arvados-api-server-with-bundle
arvados-workbench-with-bundle
refs #6096
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 48247f8..c139c08 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -548,7 +548,7 @@ cd $WORKSPACE/debs
# This is the complete package with vendor/bundle included.
# It's big, so we do not build it by default.
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "$PACKAGE_NAME" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -562,7 +562,7 @@ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
fi
# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-bare" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -603,7 +603,7 @@ cd $WORKSPACE/debs
# It's big, so we do not build it by default.
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "$PACKAGE_NAME" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-with-bundle" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -618,7 +618,7 @@ fi
# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-bare" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
commit 0dd517477cee1a5a4c247b765e150976f4809e58
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jun 3 17:10:20 2015 -0400
Simplified run-deploy.sh now that we have moved away from Capistrano.
refs #6096
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 51de227..168a16d 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -5,36 +5,23 @@ read -rd "\000" helpmessage <<EOF
$(basename $0): Deploy Arvados to a cluster
Syntax:
- WORKSPACE=/path/to/arvados $(basename $0) <identifier> <deploy_repo_name>
+ $(basename $0) <identifier>
Options:
-identifier Arvados cluster name
-deploy_repo_name Name for the repository with the (capistrano) deploy scripts
-
-WORKSPACE=path Path to the Arvados source tree to deploy from
+ identifier Arvados cluster name
EOF
IDENTIFIER=$1
-DEPLOY_REPO=$2
-if [[ "$IDENTIFIER" == '' || "$DEPLOY_REPO" == '' ]]; then
+if [[ "$IDENTIFIER" == '' ]]; then
echo >&2 "$helpmessage"
echo >&2
exit 1
fi
-# Sanity check
-if ! [[ -n "$WORKSPACE" ]]; then
- echo >&2 "$helpmessage"
- echo >&2
- echo >&2 "Error: WORKSPACE environment variable not set"
- echo >&2
- exit 1
-fi
-
EXITCODE=0
COLUMNS=80
@@ -43,18 +30,8 @@ title () {
printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
}
-# We only install capistrano in dev mode
-export RAILS_ENV=development
-
-source /etc/profile.d/rvm.sh
echo $WORKSPACE
-# Weirdly, jenkins/rvm ties itself in a knot.
-rvm use default
-
-# Just say what version of ruby we're running
-ruby --version
-
function run_puppet() {
node=$1
return_var=$2
@@ -78,67 +55,27 @@ function run_puppet() {
eval "$return_var=$ECODE"
}
-function ensure_symlink() {
- if [[ ! -L $WORKSPACE/$1 ]]; then
- ln -s $WORKSPACE/$DEPLOY_REPO/$1 $WORKSPACE/$1
- fi
-}
-
-# Check out/update the $DEPLOY_REPO repository
-if [[ ! -d $DEPLOY_REPO ]]; then
- mkdir $DEPLOY_REPO
- git clone git at git.curoverse.com:$DEPLOY_REPO.git
-else
- cd $DEPLOY_REPO
- git pull
-fi
-
-# Make sure the necessary symlinks are in place
-cd "$WORKSPACE"
-ensure_symlink "apps/workbench/Capfile.workbench.$IDENTIFIER"
-ensure_symlink "apps/workbench/config/deploy.common.rb"
-ensure_symlink "apps/workbench/config/deploy.curoverse.rb"
-ensure_symlink "apps/workbench/config/deploy.workbench.$IDENTIFIER.rb"
-
-ensure_symlink "services/api/Capfile.$IDENTIFIER"
-ensure_symlink "services/api/config/deploy.common.rb"
-ensure_symlink "services/api/config/deploy.curoverse.rb"
-ensure_symlink "services/api/config/deploy.$IDENTIFIER.rb"
-
# Deploy API server
title "Deploying API server"
-cd "$WORKSPACE"
-cd services/api
-bundle install --deployment
+SUM_ECODE=0
-# make sure we do not print the output of config:check
-sed -i'' -e "s/RAILS_ENV=production #{rake} config:check/RAILS_ENV=production QUIET=true #{rake} config:check/" $WORKSPACE/$DEPLOY_REPO/services/api/config/deploy.common.rb
+# Install updated debian packages
+title "Deploying updated arvados debian packages"
-bundle exec cap deploy -f Capfile.$IDENTIFIER
+ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get -qqy install arvados-src python-arvados-fuse python-arvados-python-client arvados-api-server-bare"
ECODE=$?
+SUM_ECODE=$(($SUM_ECODE + $ECODE))
-# restore unaltered deploy.common.rb
-cd $WORKSPACE/$DEPLOY_REPO
-git checkout services/api/config/deploy.common.rb
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! DEPLOYING API SERVER FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
- exit $EXITCODE
-fi
-
-title "Deploying API server complete"
+ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "/usr/local/bin/arvados-api-server-upgrade.sh"
-# Install updated debian packages
-title "Deploying updated arvados debian packages"
-
-ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get -qqy install arvados-src python-arvados-fuse python-arvados-python-client"
+ECODE=$?
+SUM_ECODE=$(($SUM_ECODE + $ECODE))
-if [[ "$ECODE" != "0" ]]; then
+if [[ "$SUM_ECODE" != "0" ]]; then
title "!!!!!! DEPLOYING DEBIAN PACKAGES FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
+ EXITCODE=$(($EXITCODE + $SUM_ECODE))
exit $EXITCODE
fi
@@ -149,6 +86,8 @@ title "Deploying updated arvados gems"
ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "/usr/local/rvm/bin/rvm default do gem install arvados arvados-cli && /usr/local/rvm/bin/rvm default do gem clean arvados arvados-cli"
+ECODE=$?
+
if [[ "$ECODE" != "0" ]]; then
title "!!!!!! DEPLOYING ARVADOS GEMS FAILED !!!!!!"
EXITCODE=$(($EXITCODE + $ECODE))
@@ -156,30 +95,32 @@ if [[ "$ECODE" != "0" ]]; then
fi
title "Deploying updated arvados gems complete"
+title "Deploying API server complete"
# Deploy Workbench
title "Deploying workbench"
-cd "$WORKSPACE"
-cd apps/workbench
-bundle install --deployment
-# make sure we do not print the output of config:check
-sed -i'' -e "s/RAILS_ENV=production #{rake} config:check/RAILS_ENV=production QUIET=true #{rake} config:check/" $WORKSPACE/$DEPLOY_REPO/apps/workbench/config/deploy.common.rb
+# Install updated debian packages
+title "Deploying updated arvados debian packages"
-bundle exec cap deploy -f Capfile.workbench.$IDENTIFIER
+ssh -p2222 root at workbench.$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get -qqy install python-arvados-fuse python-arvados-python-client arvados-workbench-bare"
ECODE=$?
+SUM_ECODE=$(($SUM_ECODE + $ECODE))
-# restore unaltered deploy.common.rb
-cd $WORKSPACE/$DEPLOY_REPO
-git checkout apps/workbench/config/deploy.common.rb
+ssh -p2222 root at workbench.$IDENTIFIER.arvadosapi.com -C "/usr/local/bin/arvados-workbench-upgrade.sh"
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! DEPLOYING WORKBENCH FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
+ECODE=$?
+SUM_ECODE=$(($SUM_ECODE + $ECODE))
+
+if [[ "$SUM_ECODE" != "0" ]]; then
+ title "!!!!!! DEPLOYING DEBIAN PACKAGES FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $SUM_ECODE))
exit $EXITCODE
fi
+title "Deploying updated arvados debian packages complete"
+
title "Deploying workbench complete"
# Update compute node(s)
commit 626ea7dba17faa4106fcd317aa7e277e9ec5227f
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jun 3 16:45:12 2015 -0400
Default BUILD_BUNDLE_PACKAGES to false.
refs #6096
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 27f1404..48247f8 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -29,6 +29,7 @@ CALL_FREIGHT=0
DEBUG=0
UPLOAD=0
+BUILD_BUNDLE_PACKAGES=0
while [[ -n "$1" ]]
do
commit 7761d1a6b70e1531c73f91043b9f69a251099701
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jun 3 16:40:56 2015 -0400
Fix license names in API and workbench packages.
refs #6096
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index c5ae462..27f1404 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -547,7 +547,7 @@ cd $WORKSPACE/debs
# This is the complete package with vendor/bundle included.
# It's big, so we do not build it by default.
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='AGPL v3'" "-s" "dir" "-t" "deb" "-n" "$PACKAGE_NAME" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "$PACKAGE_NAME" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -561,7 +561,7 @@ if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
fi
# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='AGPL v3'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-bare" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-bare" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -602,7 +602,7 @@ cd $WORKSPACE/debs
# It's big, so we do not build it by default.
if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='AGPL v3'" "-s" "dir" "-t" "deb" "-n" "$PACKAGE_NAME" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "$PACKAGE_NAME" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -617,7 +617,7 @@ fi
# Build the 'bare' package without vendor/bundle.
-declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='AGPL v3'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-bare" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='GNU Affero General Public License, version 3.0'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-bare" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
if [[ "$DEBUG" != 0 ]]; then
echo
commit 6c26aa2c194f60b2823da49166741203524a0b48
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jun 3 16:27:48 2015 -0400
Build API server and Workbench debian packages.
Refs #6096
diff --git a/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
new file mode 100755
index 0000000..cf32fdb
--- /dev/null
+++ b/jenkins/arvados-api-server-extras/arvados-api-server-upgrade.sh
@@ -0,0 +1,65 @@
+#!/bin/bash
+
+RELEASE_PATH=/var/www/arvados-api/current
+SHARED_PATH=/var/www/arvados-api/shared
+CONFIG_PATH=/etc/arvados/api/
+
+echo "Assumption: nginx is configured to serve `hostname` from /var/www/`hostname`/current"
+echo "Assumption: /var/www/`hostname` is symlinked to /var/www/arvados-api"
+echo "Assumption: configuration files are in /etc/arvados/api/"
+echo "Assumption: nginx and passenger run as the www-data user"
+echo
+
+echo "Copying files from $CONFIG_PATH"
+cp -f $CONFIG_PATH/database.yml $RELEASE_PATH/config/database.yml
+cp -f $RELEASE_PATH/config/environments/production.rb.example $RELEASE_PATH/config/environments/production.rb
+cp -f $CONFIG_PATH/application.yml $RELEASE_PATH/config/application.yml
+cp -f $CONFIG_PATH/omniauth.rb $RELEASE_PATH/config/initializers/omniauth.rb
+echo "Done."
+
+# Before we do anything else, make sure some directories and files are in place
+if [[ ! -e $SHARED_PATH/log ]]; then mkdir -p $SHARED_PATH/log; fi
+if [[ ! -e $RELEASE_PATH/tmp ]]; then mkdir -p $RELEASE_PATH/tmp; fi
+if [[ ! -e $RELEASE_PATH/log ]]; then ln -s $SHARED_PATH/log $RELEASE_PATH/log; fi
+if [[ ! -e $SHARED_PATH/log/production.log ]]; then touch $SHARED_PATH/log/production.log; fi
+
+echo "Running bundle install"
+(cd $RELEASE_PATH && RAILS_ENV=production bundle install --path $SHARED_PATH/vendor_bundle)
+echo "Done."
+
+echo "Precompiling assets"
+# precompile assets; thankfully this does not take long
+(cd $RELEASE_PATH; RAILS_ENV=production bundle exec rake assets:precompile)
+echo "Done."
+
+echo "Ensuring directory and file permissions"
+# Ensure correct ownership of a few files
+chown www-data:www-data $RELEASE_PATH/config/environment.rb
+chown www-data:www-data $RELEASE_PATH/config.ru
+chown www-data:www-data $RELEASE_PATH/config/database.yml
+chown www-data:www-data $RELEASE_PATH/Gemfile.lock
+chown -R www-data:www-data $RELEASE_PATH/tmp
+chown -R www-data:www-data $SHARED_PATH/log
+chown www-data:www-data $RELEASE_PATH/db/structure.sql
+chmod 644 $SHARED_PATH/log/*
+echo "Done."
+
+echo "Running sanity check"
+(cd $RELEASE_PATH; RAILS_ENV=production bundle exec rake config:check)
+echo "Done."
+
+SANITY_CHECK_EXIT_CODE=$?
+
+if [[ "$SANITY_CHECK_EXIT_CODE" != "0" ]]; then
+ echo "Sanity check failed, aborting. Please roll back to the previous version of the package."
+ echo "The database has not been migrated yet, so reinstalling the previous version is safe."
+ exit $SANITY_CHECK_EXIT_CODE
+fi
+
+echo "Starting db:migrate"
+(cd $RELEASE_PATH && bundle exec rake RAILS_ENV=production db:migrate)
+echo "Done."
+
+echo "Restarting nginx"
+service nginx restart
+echo "Done."
diff --git a/jenkins/arvados-api-server-extras/postinst.sh b/jenkins/arvados-api-server-extras/postinst.sh
new file mode 100644
index 0000000..1aeffe9
--- /dev/null
+++ b/jenkins/arvados-api-server-extras/postinst.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+cd /var/www/arvados-api
+
+chown -R www-data:www-data tmp >/dev/null 2>&1
+chown -R www-data:www-data log >/dev/null 2>&1
+chown www-data:www-data db/structure.sql >/dev/null 2>&1
+chmod 644 log/* >/dev/null 2>&1
+
+# Errors above are not serious
+exit 0
+
diff --git a/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
new file mode 100755
index 0000000..33e05d3
--- /dev/null
+++ b/jenkins/arvados-workbench-extras/arvados-workbench-upgrade.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+
+RELEASE_PATH=/var/www/arvados-workbench/current
+SHARED_PATH=/var/www/arvados-workbench/shared
+CONFIG_PATH=/etc/arvados/workbench/
+
+echo "Assumption: nginx is configured to serve workbench.`hostname` from /var/www/workbench.`hostname`/current"
+echo "Assumption: /var/www/`hostname` is symlinked to /var/www/arvados-workbench"
+echo "Assumption: configuration files are in /etc/arvados/workbench/"
+echo "Assumption: nginx and passenger run as the www-data user"
+echo
+
+echo "Copying files from $CONFIG_PATH"
+cp -f $CONFIG_PATH/application.yml $RELEASE_PATH/config/application.yml
+cp -f $RELEASE_PATH/config/environments/production.rb.example $RELEASE_PATH/config/environments/production.rb
+echo "Done."
+
+# Before we do anything else, make sure some directories and files are in place
+if [[ ! -e $SHARED_PATH/log ]]; then mkdir -p $SHARED_PATH/log; fi
+if [[ ! -e $RELEASE_PATH/tmp ]]; then mkdir -p $RELEASE_PATH/tmp; fi
+if [[ ! -e $RELEASE_PATH/log ]]; then ln -s $SHARED_PATH/log $RELEASE_PATH/log; fi
+if [[ ! -e $SHARED_PATH/log/production.log ]]; then touch $SHARED_PATH/log/production.log; fi
+
+echo "Running bundle install"
+(cd $RELEASE_PATH && RAILS_ENV=production bundle install --path $SHARED_PATH/vendor_bundle)
+echo "Done."
+
+# We do not need to precompile assets, they are already part of the package.
+
+echo "Ensuring directory and file permissions"
+chown www-data:www-data $RELEASE_PATH/config/environment.rb
+chown www-data:www-data $RELEASE_PATH/config.ru
+chown www-data:www-data $RELEASE_PATH/config/database.yml
+chown www-data:www-data $RELEASE_PATH/Gemfile.lock
+chown -R www-data:www-data $RELEASE_PATH/tmp
+chown -R www-data:www-data $SHARED_PATH/log
+chown www-data:www-data $RELEASE_PATH/db/schema.rb
+chmod 644 $SHARED_PATH/log/*
+echo "Done."
+
+echo "Running sanity check"
+(cd $RELEASE_PATH; RAILS_ENV=production bundle exec rake config:check)
+echo "Done."
+
+SANITY_CHECK_EXIT_CODE=$?
+
+if [[ "$SANITY_CHECK_EXIT_CODE" != "0" ]]; then
+ echo "Sanity check failed, aborting. Please roll back to the previous version of the package."
+ exit $SANITY_CHECK_EXIT_CODE
+fi
+
+# We do not need to run db:migrate because Workbench is stateless
+
+echo "Restarting nginx"
+service nginx restart
+echo "Done."
+
diff --git a/jenkins/arvados-workbench-extras/postinst.sh b/jenkins/arvados-workbench-extras/postinst.sh
new file mode 100644
index 0000000..d0aa9b0
--- /dev/null
+++ b/jenkins/arvados-workbench-extras/postinst.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+cd /var/www/arvados-workbench
+
+chown -R www-data:www-data tmp >/dev/null 2>&1
+chown -R www-data:www-data log >/dev/null 2>&1
+chown www-data:www-data db/schema.rb >/dev/null 2>&1
+chmod 644 log/* >/dev/null 2>&1
+
+# Errors above are not serious
+exit 0
+
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 691e93b..c5ae462 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -9,10 +9,16 @@ Syntax:
Options:
---upload Upload packages (default: false)
---scp-user USERNAME Scp user for apt server (only required when --upload is specified)
---apt-server HOSTNAME Apt server hostname (only required when --upload is specified)
---debug Output debug information (default: false)
+--upload
+ Upload packages (default: false)
+--scp-user USERNAME
+ Scp user for apt server (only required when --upload is specified)
+--apt-server HOSTNAME
+ Apt server hostname (only required when --upload is specified)
+--build-bundle-packages (default: false)
+ Build api server and workbench packages with vendor/bundle included
+--debug
+ Output debug information (default: false)
WORKSPACE=path Path to the Arvados source tree to build packages from
@@ -45,6 +51,9 @@ do
--upload)
UPLOAD=1
;;
+ --build-bundle-packages)
+ BUILD_BUNDLE_PACKAGES=1
+ ;;
*)
echo >&2 "$0: Unrecognized option: '$arg'. Try: $0 --help"
exit 1
@@ -91,7 +100,16 @@ EOF
exit 1
fi
+RUN_BUILD_PACKAGES_PATH="`dirname \"$0\"`"
+RUN_BUILD_PACKAGES_PATH="`( cd \"$RUN_BUILD_PACKAGES_PATH\" && pwd )`" # absolutized and normalized
+if [ -z "$RUN_BUILD_PACKAGES_PATH" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
if [[ "$DEBUG" != 0 ]]; then
+ echo "$0 is running from $RUN_BUILD_PACKAGES_PATH"
echo "Workspace is $WORKSPACE"
fi
@@ -192,6 +210,15 @@ build_and_scp_deb () {
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
+ verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
+}
+
+# verify build results and scp debs, if needed
+verify_and_scp_deb () {
+ FPM_EXIT_CODE=$1
+ shift
+ FPM_RESULTS=$@
+
FPM_PACKAGE_NAME=''
if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\-.]*\.deb) ]]; then
FPM_PACKAGE_NAME=${BASH_REMATCH[1]}
@@ -270,7 +297,7 @@ if [[ "$DEBUG" != 0 ]]; then
echo
fi
-if type rvm-exec 2>/dev/null; then
+if type rvm-exec >/dev/null 2>&1; then
FPM_GEM_PREFIX=$(rvm-exec system gem environment gemdir)
else
FPM_GEM_PREFIX=$(gem environment gemdir)
@@ -286,7 +313,7 @@ if [[ "$DEBUG" != 0 ]]; then
gem build arvados.gemspec
else
# -q appears to be broken in gem version 2.2.2
- gem build arvados.gemspec -q >/dev/null
+ gem build arvados.gemspec -q >/dev/null 2>&1
fi
if [[ "$UPLOAD" != 0 ]]; then
@@ -483,12 +510,127 @@ for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pycrypto backports.ssl_match_hostname; do
build_and_scp_deb "$deppkg"
done
+
# Python 3 dependencies
for deppkg in docker-py six requests; do
# The empty string is the vendor argument: these aren't Curoverse software.
build_and_scp_deb "$deppkg" "python3-$deppkg" "" python3
done
+# Build the API server package
+
+cd "$WORKSPACE/services/api"
+
+API_VERSION=$(version_from_git)
+PACKAGE_NAME=arvados-api-server
+
+if [[ ! -d "$WORKSPACE/services/api/tmp" ]]; then
+ mkdir $WORKSPACE/services/api/tmp
+fi
+
+BUNDLE_OUTPUT=`bundle install --path vendor/bundle`
+
+if [[ "$DEBUG" != 0 ]]; then
+ echo $BUNDLE_OUTPUT
+fi
+
+/usr/bin/git rev-parse HEAD > git-commit.version
+
+cd $WORKSPACE/debs
+
+# Annoyingly, we require a database.yml file for rake assets:precompile to work. So for now,
+# we do that in the upgrade script.
+# TODO: add bogus database.yml file so we can precompile the assets and put them in the
+# package. Then remove that database.yml file again. It has to be a valid file though.
+#RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile
+
+# This is the complete package with vendor/bundle included.
+# It's big, so we do not build it by default.
+if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='AGPL v3'" "-s" "dir" "-t" "deb" "-n" "$PACKAGE_NAME" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
+
+ if [[ "$DEBUG" != 0 ]]; then
+ echo
+ echo "${COMMAND_ARR[@]}"
+ echo
+ fi
+
+ FPM_RESULTS=$("${COMMAND_ARR[@]}")
+ FPM_EXIT_CODE=$?
+ verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
+fi
+
+# Build the 'bare' package without vendor/bundle.
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados API server - Arvados is a free and open source platform for big data science.'" "--license='AGPL v3'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-bare" "-v" "$API_VERSION" "-x" "var/www/arvados-api/current/tmp" "-x" "var/www/arvados-api/current/log" "-x" "var/www/arvados-api/current/vendor/bundle" "-x" "var/www/arvados-api/current/vendor/cache/*" "-x" "var/www/arvados-api/current/coverage" "-x" "var/www/arvados-api/current/Capfile*" "-x" "var/www/arvados-api/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/postinst.sh" "$WORKSPACE/services/api/=/var/www/arvados-api/current" "$RUN_BUILD_PACKAGES_PATH/arvados-api-server-extras/arvados-api-server-upgrade.sh=/usr/local/bin/arvados-api-server-upgrade.sh")
+
+if [[ "$DEBUG" != 0 ]]; then
+ echo
+ echo "${COMMAND_ARR[@]}"
+ echo
+fi
+
+FPM_RESULTS=$("${COMMAND_ARR[@]}")
+FPM_EXIT_CODE=$?
+verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
+
+# API server package build done
+
+# Build the workbench server package
+
+cd "$WORKSPACE/apps/workbench"
+
+WORKBENCH_VERSION=$(version_from_git)
+PACKAGE_NAME=arvados-workbench
+
+if [[ ! -d "$WORKSPACE/apps/workbench/tmp" ]]; then
+ mkdir $WORKSPACE/apps/workbench/tmp
+fi
+
+BUNDLE_OUTPUT=`bundle install --path vendor/bundle`
+
+if [[ "$DEBUG" != 0 ]]; then
+ echo $BUNDLE_OUTPUT
+fi
+
+/usr/bin/git rev-parse HEAD > git-commit.version
+
+RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile >/dev/null
+
+cd $WORKSPACE/debs
+
+# This is the complete package with vendor/bundle included.
+# It's big, so we do not build it by default.
+if [[ "$BUILD_BUNDLE_PACKAGES" != 0 ]]; then
+
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='AGPL v3'" "-s" "dir" "-t" "deb" "-n" "$PACKAGE_NAME" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+
+ if [[ "$DEBUG" != 0 ]]; then
+ echo
+ echo "${COMMAND_ARR[@]}"
+ echo
+ fi
+
+ FPM_RESULTS=$("${COMMAND_ARR[@]}")
+ FPM_EXIT_CODE=$?
+ verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
+fi
+
+# Build the 'bare' package without vendor/bundle.
+
+declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "--vendor='Curoverse, Inc.'" "--url='https://arvados.org'" "--description='Arvados Workbench - Arvados is a free and open source platform for big data science.'" "--license='AGPL v3'" "-s" "dir" "-t" "deb" "-n" "${PACKAGE_NAME}-bare" "-v" "$WORKBENCH_VERSION" "-x" "var/www/arvados-workbench/current/tmp" "-x" "var/www/arvados-workbench/current/log" "-x" "var/www/arvados-workbench/current/vendor/bundle" "-x" "var/www/arvados-workbench/current/vendor/cache/*" "-x" "var/www/arvados-workbench/current/coverage" "-x" "var/www/arvados-workbench/current/Capfile*" "-x" "var/www/arvados-workbench/current/config/deploy*" "--after-install=$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/postinst.sh" "$WORKSPACE/apps/workbench/=/var/www/arvados-workbench/current" "$RUN_BUILD_PACKAGES_PATH/arvados-workbench-extras/arvados-workbench-upgrade.sh=/usr/local/bin/arvados-workbench-upgrade.sh")
+
+if [[ "$DEBUG" != 0 ]]; then
+ echo
+ echo "${COMMAND_ARR[@]}"
+ echo
+fi
+
+FPM_RESULTS=$("${COMMAND_ARR[@]}")
+FPM_EXIT_CODE=$?
+verify_and_scp_deb $FPM_EXIT_CODE $FPM_RESULTS
+
+# Workbench package build done
+
# Finally, publish the packages, if necessary
if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
ssh -p2222 $APTUSER@$APTSERVER -t "cd tmp && ls -laF *deb && freight add *deb apt/wheezy && freight cache && rm -f *deb"
commit 44eab398d387d1e72850c6a1fedd74272ca119a9
Merge: 845e399 a572ef6
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed May 27 14:40:11 2015 -0400
Merge branch '5548-test-cwltool' refs #5548
commit a572ef644ee17b0a61e039ce691a39554031aa73
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed May 27 14:18:23 2015 -0400
5548: Remove git commands from run-cwl-tests. Remove cwltool from build-packages.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index bb0877e..691e93b 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -489,37 +489,6 @@ for deppkg in docker-py six requests; do
build_and_scp_deb "$deppkg" "python3-$deppkg" "" python3
done
-# cwltool from common-workflow-language. We use this in arv-run-pipeline-instance.
-# We use $WORKSPACE/common-workflow-language as the clean directory from which to build the cwltool package
-if [[ ! -d "$WORKSPACE/common-workflow-language" ]]; then
- mkdir "$WORKSPACE/common-workflow-language"
- cd "$WORKSPACE"
- if [[ "$DEBUG" != 0 ]]; then
- git clone https://github.com/common-workflow-language/common-workflow-language.git common-workflow-language
- else
- git clone -q https://github.com/common-workflow-language/common-workflow-language.git common-workflow-language
- fi
-fi
-
-cd "$WORKSPACE/common-workflow-language"
-if [[ "$DEBUG" != 0 ]]; then
- git checkout master
- git pull
-else
- git checkout -q master
- git pull -q
-fi
-
-cd reference
-handle_python_package
-CWLTOOL_VERSION=`git log --first-parent --max-count=1 --format='format:0.1.%ct.%h'`
-
-# Build cwltool package
-cd $WORKSPACE/debs
-# Python version numbering is obscure. Strip dashes and replace them with dots
-# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
-build_and_scp_deb $WORKSPACE/common-workflow-language/reference cwltool 'Common Workflow Language Working Group' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/common-workflow-language/reference/cwltool.egg-info/PKG-INFO)"
-
# Finally, publish the packages, if necessary
if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
ssh -p2222 $APTUSER@$APTSERVER -t "cd tmp && ls -laF *deb && freight add *deb apt/wheezy && freight cache && rm -f *deb"
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
index 917365d..74e9999 100755
--- a/jenkins/run-cwl-tests.sh
+++ b/jenkins/run-cwl-tests.sh
@@ -156,23 +156,6 @@ fi
cd "$WORKSPACE"
-# We use $WORKSPACE as the clean directory from which to build the cwltool package
-if [[ ! -d "$WORKSPACE/.git" ]]; then
- if [[ "$DEBUG" != 0 ]]; then
- git clone https://github.com/common-workflow-language/common-workflow-language.git .
- else
- git clone -q https://github.com/common-workflow-language/common-workflow-language.git .
- fi
-fi
-
-if [[ "$DEBUG" != 0 ]]; then
- git checkout master
- git pull
-else
- git checkout -q master
- git pull -q
-fi
-
pushd reference
python setup.py install
python setup.py test
commit ac6798733ce11d6a3410bf1da639cb3e5a84047f
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed May 27 12:30:51 2015 -0400
5548: Script to test, build and upload common workflow language packages and
docker images.
diff --git a/jenkins/run-cwl-tests.sh b/jenkins/run-cwl-tests.sh
new file mode 100755
index 0000000..917365d
--- /dev/null
+++ b/jenkins/run-cwl-tests.sh
@@ -0,0 +1,195 @@
+#!/bin/bash
+
+read -rd "\000" helpmessage <<EOF
+$(basename $0): Test cwl tool and (optionally) upload to PyPi and Docker Hub.
+
+Syntax:
+ WORKSPACE=/path/to/common-workflow-language $(basename $0) [options]
+
+Options:
+
+--upload-pypi Upload package to pypi (default: false)
+--upload-docker Upload packages to docker hub (default: false)
+--debug Output debug information (default: false)
+
+WORKSPACE=path Path to the common-workflow-language source tree
+
+EOF
+
+EXITCODE=0
+CALL_FREIGHT=0
+
+DEBUG=0
+UPLOAD_PYPI=0
+UPLOAD_DOCKER=0
+
+VENVDIR=
+
+leave_temp=
+
+declare -A leave_temp
+
+set -e
+
+clear_temp() {
+ leaving=""
+ for var in VENVDIR
+ do
+ if [[ -z "${leave_temp[$var]}" ]]
+ then
+ if [[ -n "${!var}" ]]
+ then
+ rm -rf "${!var}"
+ fi
+ else
+ leaving+=" $var=\"${!var}\""
+ fi
+ done
+ if [[ -n "$leaving" ]]; then
+ echo "Leaving behind temp dirs: $leaving"
+ fi
+}
+
+fatal() {
+ clear_temp
+ echo >&2 "Fatal: $* (encountered in ${FUNCNAME[1]} at ${BASH_SOURCE[1]} line ${BASH_LINENO[0]})"
+ exit 1
+}
+
+trap clear_temp INT EXIT
+
+# Set up temporary install dirs (unless existing dirs were supplied)
+for tmpdir in VENVDIR
+do
+ if [[ -n "${!tmpdir}" ]]; then
+ leave_temp[$tmpdir]=1
+ else
+ eval $tmpdir=$(mktemp -d)
+ fi
+done
+
+
+while [[ -n "$1" ]]
+do
+ arg="$1"; shift
+ case "$arg" in
+ --help)
+ echo >&2 "$helpmessage"
+ echo >&2
+ exit 1
+ ;;
+ --debug)
+ DEBUG=1
+ ;;
+ --upload-pypi)
+ UPLOAD_PYPI=1
+ ;;
+ --upload-docker)
+ UPLOAD_DOCKER=1
+ ;;
+ --leave-temp)
+ leave_temp[VENVDIR]=1
+ ;;
+ *=*)
+ eval export $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2-)\"
+ ;;
+ *)
+ echo >&2 "$0: Unrecognized option: '$arg'. Try: $0 --help"
+ exit 1
+ ;;
+ esac
+done
+
+# Sanity check
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
+fi
+
+if [[ "$DEBUG" != 0 ]]; then
+ echo "Workspace is $WORKSPACE"
+fi
+
+virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
+. "$VENVDIR/bin/activate"
+
+handle_python_package () {
+ # This function assumes the current working directory is the python package directory
+ if [[ "$UPLOAD_PYPI" != 0 ]]; then
+ # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ if [[ "$DEBUG" != 0 ]]; then
+ python setup.py sdist upload
+ else
+ python setup.py -q sdist upload
+ fi
+ else
+ # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ if [[ "$DEBUG" != 0 ]]; then
+ python setup.py sdist
+ else
+ python setup.py -q sdist
+ fi
+ fi
+}
+
+# Make all files world-readable -- jenkins runs with umask 027, and has checked
+# out our git tree here
+chmod o+r "$WORKSPACE" -R
+
+# Now fix our umask to something better suited to building and publishing
+# gems and packages
+umask 0022
+
+if [[ "$DEBUG" != 0 ]]; then
+ echo "umask is" `umask`
+fi
+
+# Python packages
+if [[ "$DEBUG" != 0 ]]; then
+ echo
+ echo "Python packages"
+ echo
+fi
+
+cd "$WORKSPACE"
+
+# We use $WORKSPACE as the clean directory from which to build the cwltool package
+if [[ ! -d "$WORKSPACE/.git" ]]; then
+ if [[ "$DEBUG" != 0 ]]; then
+ git clone https://github.com/common-workflow-language/common-workflow-language.git .
+ else
+ git clone -q https://github.com/common-workflow-language/common-workflow-language.git .
+ fi
+fi
+
+if [[ "$DEBUG" != 0 ]]; then
+ git checkout master
+ git pull
+else
+ git checkout -q master
+ git pull -q
+fi
+
+pushd reference
+python setup.py install
+python setup.py test
+./docker-node-engine.sh
+popd
+
+pushd conformance
+pwd
+./run_test.sh
+popd
+
+cd reference
+handle_python_package
+
+./build-cwl-docker.sh
+
+if [[ "$UPLOAD_DOCKER" != 0 ]]; then
+ docker push commonworkflowlanguage/cwltool
+ docker push commonworkflowlanguage/nodejs-engine
+fi
commit 845e3998bb2b4175e947a20eea4ed530629a778b
Merge: 1e7b6aa b6a5513
Author: Nico Cesar <nico at nicocesar.com>
Date: Wed May 27 10:16:55 2015 -0400
Merge branch '6135-docker-git-tag'
closes #6135
Conflicts:
jenkins/run-docker-tests.sh
diff --cc jenkins/run-docker-tests.sh
index f0afccb,4f584d8..6bf3859
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@@ -50,7 -64,16 +64,15 @@@ title ()
}
docker_push () {
- # Sometimes docker push fails; retry it a few times if necessary.
- echo $tags
+ if [[ ! -z "$tags" ]]
+ then
+ for tag in $( echo $tags|tr "," " " )
+ do
+ $DOCKER tag $1 $1:$tag
+ done
+ fi
+
+ # Sometimes docker push fails; retry it a few times if necessary.
for i in `seq 1 5`; do
$DOCKER push $*
ECODE=$?
commit b6a5513611786d2c0a513bac3ab2db00d84bf289
Author: Nico Cesar <nico at nicocesar.com>
Date: Thu May 21 14:35:54 2015 -0400
6035: add support for --upload as an option
refs #6035
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index 5be7b99..4f584d8 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -1,27 +1,91 @@
#!/bin/bash
+function usage {
+ echo >&2
+ echo >&2 "usage: $0 [options]"
+ echo >&2
+ echo >&2 "$0 options:"
+ echo >&2 " -t, --tags [csv_tags] comma separated tags"
+ echo >&2 " -u, --upload Upload the images (docker push)"
+ echo >&2 " -h, --help Display this help and exit"
+ echo >&2
+ echo >&2 " If no options are given, just builds the images."
+}
+
+upload=false
+
+# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
+TEMP=`getopt -o hut: \
+ --long help,upload,tags: \
+ -n "$0" -- "$@"`
+
+if [ $? != 0 ] ; then echo "Use -h for help"; exit 1 ; fi
+# Note the quotes around `$TEMP': they are essential!
+eval set -- "$TEMP"
+
+while [ $# -ge 1 ]
+do
+ case $1 in
+ -u | --upload)
+ upload=true
+ shift
+ ;;
+ -t | --tags)
+ case "$2" in
+ "")
+ echo "ERROR: --tags needs a parameter";
+ usage;
+ exit 1
+ ;;
+ *)
+ tags=$2;
+ shift 2
+ ;;
+ esac
+ ;;
+ --)
+ shift
+ break
+ ;;
+ *)
+ usage
+ exit 1
+ ;;
+ esac
+done
+
+
EXITCODE=0
COLUMNS=80
title () {
- printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
+ printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
}
docker_push () {
- # Sometimes docker push fails; retry it a few times if necessary.
- for i in `seq 1 5`; do
- $DOCKER push $*
- ECODE=$?
- if [[ "$ECODE" == "0" ]]; then
- break
+ echo $tags
+ if [[ ! -z "$tags" ]]
+ then
+ for tag in $( echo $tags|tr "," " " )
+ do
+ $DOCKER tag $1 $1:$tag
+ done
fi
- done
- if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! docker push $* failed !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
- fi
+ # Sometimes docker push fails; retry it a few times if necessary.
+ for i in `seq 1 5`; do
+ $DOCKER push $*
+ ECODE=$?
+ if [[ "$ECODE" == "0" ]]; then
+ break
+ fi
+ done
+
+ if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! docker push $* failed !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ fi
}
timer_reset() {
@@ -34,10 +98,10 @@ timer() {
# Sanity check
if ! [[ -n "$WORKSPACE" ]]; then
- echo >&2
- echo >&2 "Error: WORKSPACE environment variable not set"
- echo >&2
- exit 1
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
fi
echo $WORKSPACE
@@ -46,12 +110,12 @@ echo $WORKSPACE
DOCKER=`which docker.io`
if [[ "$DOCKER" == "" ]]; then
- DOCKER=`which docker`
+ DOCKER=`which docker`
fi
if [[ "$DOCKER" == "" ]]; then
- title "Error: you need to have docker installed. Could not find the docker executable."
- exit 1
+ title "Error: you need to have docker installed. Could not find the docker executable."
+ exit 1
fi
# DOCKER
@@ -74,8 +138,8 @@ cp $HOME/docker/config.yml .
ECODE=$?
if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! docker BUILD FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
+ title "!!!!!! docker BUILD FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
fi
title "docker build complete (`timer`)"
@@ -84,21 +148,24 @@ title "uploading images"
timer_reset
-if [[ "$ECODE" == "0" ]]; then
- docker_push arvados/api
- docker_push arvados/compute
- docker_push arvados/doc
- docker_push arvados/workbench
- docker_push arvados/keep
- docker_push arvados/keepproxy
- docker_push arvados/shell
- docker_push arvados/sso
+if [[ "$ECODE" != "0" ]]; then
+ title "upload arvados images SKIPPED because build failed"
else
- title "upload arvados images SKIPPED because build failed"
+ if [[ $upload == true ]]; then
+ docker_push arvados/api
+ docker_push arvados/compute
+ docker_push arvados/doc
+ docker_push arvados/workbench
+ docker_push arvados/keep
+ docker_push arvados/keepproxy
+ docker_push arvados/shell
+ docker_push arvados/sso
+ title "upload arvados images complete (`timer`)"
+ else
+ title "upload arvados images SKIPPED because no --upload option set"
+ fi
fi
-title "upload arvados images complete (`timer`)"
-
title "Starting docker java-bwa-samtools build"
timer_reset
@@ -108,21 +175,24 @@ timer_reset
ECODE=$?
if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! docker java-bwa-samtools BUILD FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
+ title "!!!!!! docker java-bwa-samtools BUILD FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
fi
title "docker build java-bwa-samtools complete (`timer`)"
-title "upload arvados/jobs image"
-
timer_reset
-if [[ "$ECODE" == "0" ]]; then
- docker_push arvados/jobs
+
+if [[ "$ECODE" != "0" ]]; then
+ title "upload arvados/jobs image SKIPPED because build failed"
else
- title "upload arvados/jobs image SKIPPED because build failed"
+ if [[ $upload == true ]]; then
+ title "upload arvados/jobs image"
+ docker_push arvados/jobs
+ title "upload arvados/jobs image complete (`timer`)"
+ else
+ title "upload arvados images SKIPPED because no --upload option set"
+ fi
fi
-title "upload arvados/jobs image complete (`timer`)"
-
exit $EXITCODE
commit 1e7b6aa1f16e06b796e3ed3f66c48f38a9287a5d
Merge: 2b9864b bb48082
Author: Nico Cesar <nico at nicocesar.com>
Date: Fri May 22 15:32:41 2015 -0400
Merge branch '6035-docker-upload-option'
closes #6035
commit bb48082657c79156ad9af20d71c1ca73eafdf0fa
Author: Nico Cesar <nico at nicocesar.com>
Date: Fri May 22 15:18:14 2015 -0400
style and fashion for classy 'if' statements
refs #6035
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index 5dcc167..f0afccb 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -125,7 +125,7 @@ title "uploading images"
timer_reset
-if [[ ! "$ECODE" == "0" ]]; then
+if [[ "$ECODE" != "0" ]]; then
title "upload arvados images SKIPPED because build failed"
else
if [[ $upload == true ]]; then
@@ -160,7 +160,7 @@ title "docker build java-bwa-samtools complete (`timer`)"
timer_reset
-if [[ ! "$ECODE" == "0" ]]; then
+if [[ "$ECODE" != "0" ]]; then
title "upload arvados/jobs image SKIPPED because build failed"
else
if [[ $upload == true ]]; then
commit 6a1a627a78a021aa164bfd1136bb5396e1defd95
Author: Nico Cesar <nico at nicocesar.com>
Date: Thu May 21 14:35:54 2015 -0400
6035: add support for --upload as an option
refs #6035
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index 5be7b99..5dcc167 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -1,27 +1,68 @@
#!/bin/bash
+function usage {
+ echo >&2
+ echo >&2 "usage: $0 [options]"
+ echo >&2
+ echo >&2 "$0 options:"
+ echo >&2 " -u, --upload Upload the images (docker push)"
+ echo >&2 " -h, --help Display this help and exit"
+ echo >&2
+ echo >&2 " If no options are given, just builds the images."
+}
+
+upload=false
+
+# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
+TEMP=`getopt -o hu \
+ --long help,upload \
+ -n "$0" -- "$@"`
+
+if [ $? != 0 ] ; then echo "Use -h for help"; exit 1 ; fi
+# Note the quotes around `$TEMP': they are essential!
+eval set -- "$TEMP"
+
+while [ $# -ge 1 ]
+do
+ case $1 in
+ -u | --upload)
+ upload=true
+ shift
+ ;;
+ --)
+ shift
+ break
+ ;;
+ *)
+ usage
+ exit 1
+ ;;
+ esac
+done
+
+
EXITCODE=0
COLUMNS=80
title () {
- printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
+ printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
}
docker_push () {
# Sometimes docker push fails; retry it a few times if necessary.
- for i in `seq 1 5`; do
- $DOCKER push $*
- ECODE=$?
- if [[ "$ECODE" == "0" ]]; then
- break
+ for i in `seq 1 5`; do
+ $DOCKER push $*
+ ECODE=$?
+ if [[ "$ECODE" == "0" ]]; then
+ break
+ fi
+ done
+
+ if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! docker push $* failed !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
fi
- done
-
- if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! docker push $* failed !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
- fi
}
timer_reset() {
@@ -34,10 +75,10 @@ timer() {
# Sanity check
if ! [[ -n "$WORKSPACE" ]]; then
- echo >&2
- echo >&2 "Error: WORKSPACE environment variable not set"
- echo >&2
- exit 1
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
fi
echo $WORKSPACE
@@ -46,12 +87,12 @@ echo $WORKSPACE
DOCKER=`which docker.io`
if [[ "$DOCKER" == "" ]]; then
- DOCKER=`which docker`
+ DOCKER=`which docker`
fi
if [[ "$DOCKER" == "" ]]; then
- title "Error: you need to have docker installed. Could not find the docker executable."
- exit 1
+ title "Error: you need to have docker installed. Could not find the docker executable."
+ exit 1
fi
# DOCKER
@@ -74,8 +115,8 @@ cp $HOME/docker/config.yml .
ECODE=$?
if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! docker BUILD FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
+ title "!!!!!! docker BUILD FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
fi
title "docker build complete (`timer`)"
@@ -84,21 +125,24 @@ title "uploading images"
timer_reset
-if [[ "$ECODE" == "0" ]]; then
- docker_push arvados/api
- docker_push arvados/compute
- docker_push arvados/doc
- docker_push arvados/workbench
- docker_push arvados/keep
- docker_push arvados/keepproxy
- docker_push arvados/shell
- docker_push arvados/sso
+if [[ ! "$ECODE" == "0" ]]; then
+ title "upload arvados images SKIPPED because build failed"
else
- title "upload arvados images SKIPPED because build failed"
+ if [[ $upload == true ]]; then
+ docker_push arvados/api
+ docker_push arvados/compute
+ docker_push arvados/doc
+ docker_push arvados/workbench
+ docker_push arvados/keep
+ docker_push arvados/keepproxy
+ docker_push arvados/shell
+ docker_push arvados/sso
+ title "upload arvados images complete (`timer`)"
+ else
+ title "upload arvados images SKIPPED because no --upload option set"
+ fi
fi
-title "upload arvados images complete (`timer`)"
-
title "Starting docker java-bwa-samtools build"
timer_reset
@@ -108,21 +152,24 @@ timer_reset
ECODE=$?
if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! docker java-bwa-samtools BUILD FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
+ title "!!!!!! docker java-bwa-samtools BUILD FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
fi
title "docker build java-bwa-samtools complete (`timer`)"
-title "upload arvados/jobs image"
-
timer_reset
-if [[ "$ECODE" == "0" ]]; then
- docker_push arvados/jobs
+
+if [[ ! "$ECODE" == "0" ]]; then
+ title "upload arvados/jobs image SKIPPED because build failed"
else
- title "upload arvados/jobs image SKIPPED because build failed"
+ if [[ $upload == true ]]; then
+ title "upload arvados/jobs image"
+ docker_push arvados/jobs
+ title "upload arvados/jobs image complete (`timer`)"
+ else
+ title "upload arvados images SKIPPED because no --upload option set"
+ fi
fi
-title "upload arvados/jobs image complete (`timer`)"
-
exit $EXITCODE
commit 2b9864be33a42c3f9f612fcb3cf51e46c040a354
Merge: 0fbece8 cc7fb93
Author: Nico Cesar <nico at nicocesar.com>
Date: Thu May 21 11:33:27 2015 -0400
Merge branch '6061-add-performance-plots' refs #6061
commit cc7fb93641cd735c8bd0ec744b1fa1b84a242626
Author: Brett Smith <brett at curoverse.com>
Date: Thu May 21 10:21:45 2015 -0400
3793: Fix how python3 fpm arguments are added.
The previous code would add arguments such that they could be
interpreted as the package version. Avoid this by adding arguments
after we're done consuming positional arguments.
Refs #3793.
no more trailing spaces
adding an empty csv in case of error
diff --git a/jenkins/create-plot-data-from-log.sh b/jenkins/create-plot-data-from-log.sh
index ed5ee53..ce3bfed 100755
--- a/jenkins/create-plot-data-from-log.sh
+++ b/jenkins/create-plot-data-from-log.sh
@@ -12,7 +12,7 @@ usage() {
if [ $# -ne 3 ]
then
- usage
+ usage
exit 1
fi
@@ -33,7 +33,7 @@ fi
## MAXLINE is the amount of lines that will read after the pattern
## is match (the logfile could be hundred thousands lines long).
## 1000 should be safe enough to capture all the output of the individual test
-MAXLINES=1000
+MAXLINES=1000
## TODO: check $build and $file make sense
@@ -46,12 +46,14 @@ do
(zgrep -i -E -A$MAXLINES "^[A-Za-z0-9]+Test: $test" $file && echo "----") | tail -n +1 | tail --lines=+3|grep -B$MAXLINES -E "^-*$" -m1 > $outputdir/$cleaned_test-$build.txt
result=$?
if [ $result -eq 0 ]
- then
+ then
echo processing $outputdir/$cleaned_test-$build.txt creating $outputdir/$cleaned_test.csv
echo $(grep ^Completed $outputdir/$cleaned_test-$build.txt | perl -n -e '/^Completed (.*) in [0-9]+ms.*$/;print "".++$line."-$1,";' | perl -p -e 's/,$//g'|tr " " "_" ) > $outputdir/$cleaned_test.csv
echo $(grep ^Completed $outputdir/$cleaned_test-$build.txt | perl -n -e '/^Completed.*in ([0-9]+)ms.*$/;print "$1,";' | perl -p -e 's/,$//g' ) >> $outputdir/$cleaned_test.csv
#echo URL=https://ci.curoverse.com/view/job/arvados-api-server/ws/apps/workbench/log/$cleaned_test-$build.txt/*view*/ >> $outputdir/$test.properties
- else
+ else
echo "$test was't found on $file"
+ cleaned_test=$(echo $test | tr -d ",.:;/")
+ > $outputdir/$cleaned_test.csv
fi
done
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 95a507a..bb0877e 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -149,18 +149,18 @@ build_and_scp_deb () {
# The type of source package. Passed to fpm -s. Default "python".
PACKAGE_TYPE=${1:-python}
shift
+ # Optional: the package version number. Passed to fpm -v.
+ VERSION=$1
+ shift
+ # fpm does not actually support a python3 package type. Instead we recognize
+ # it as a convenience shortcut to add several necessary arguments to
+ # fpm's command line later, after we're done handling positional arguments.
if [ "python3" = "$PACKAGE_TYPE" ]; then
- # fpm does not actually support this package type. Instead we recognize
- # it as a convenience shortcut to add several necessary arguments to
- # fpm's command line later.
PACKAGE_TYPE=python
set -- "$@" --python-bin python3 --python-easyinstall easy_install3 \
--python-package-name-prefix python3 --depends python3
fi
- # Optional: the package version number. Passed to fpm -v.
- VERSION=$1
- shift
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "deb" "-x" "usr/local/lib/python2.7/dist-packages/tests")
commit 32f2a67ecfd7a5b4169d4e3fbc162306771f0d34
Author: Brett Smith <brett at curoverse.com>
Date: Thu May 21 10:03:36 2015 -0400
Improve packaging of third-party Python 3 packages.
* Fix the bug where we're passing in "python3" as the vendor argument
to build_and_scp_deb.
* Build more dependencies that docker-py needs.
* Automatically add a dependency on the python3 package.
Refs #3793.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 3b9fc15..95a507a 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -156,7 +156,7 @@ build_and_scp_deb () {
# fpm's command line later.
PACKAGE_TYPE=python
set -- "$@" --python-bin python3 --python-easyinstall easy_install3 \
- --python-package-name-prefix python3
+ --python-package-name-prefix python3 --depends python3
fi
# Optional: the package version number. Passed to fpm -v.
VERSION=$1
@@ -483,7 +483,11 @@ for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pycrypto backports.ssl_match_hostname; do
build_and_scp_deb "$deppkg"
done
-build_and_scp_deb docker-py python3-docker-py python3
+# Python 3 dependencies
+for deppkg in docker-py six requests; do
+ # The empty string is the vendor argument: these aren't Curoverse software.
+ build_and_scp_deb "$deppkg" "python3-$deppkg" "" python3
+done
# cwltool from common-workflow-language. We use this in arv-run-pipeline-instance.
# We use $WORKSPACE/common-workflow-language as the clean directory from which to build the cwltool package
commit eae444a782a53e348fd5df5929cfb2713f0d6fa8
Author: Tom Clegg <tom at curoverse.com>
Date: Thu May 21 08:57:50 2015 -0400
5766: Remove "remove dependencies" step. Detect existing PyYAML faster.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index be35ae9..1a699a8 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -370,10 +370,6 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
. "$VENVDIR/bin/activate"
-# When re-using $VENVDIR, upgrade any packages (except arvados) that are
-# already installed
-pip install --quiet --upgrade `pip freeze | grep -v arvados | cut -f1 -d=`
-
# Note: this must be the last time we change PATH, otherwise rvm will
# whine a lot.
setup_ruby_environment
@@ -386,8 +382,9 @@ then
fi
# Needed for run_test_server.py which is used by certain (non-Python) tests.
-echo "pip install -q PyYAML"
-pip install --quiet PyYAML || fatal "pip install PyYAML failed"
+pip freeze 2>/dev/null | egrep ^PyYAML= \
+ || pip install PyYAML \
+ || fatal "pip install PyYAML failed"
# If Python 3 is available, set up its virtualenv in $VENV3DIR.
# Otherwise, skip dependent tests.
commit c4d584c524efe8c9c689a263bec54b99b3d89138
Author: Tom Clegg <tom at curoverse.com>
Date: Thu May 21 08:51:21 2015 -0400
Tidy up sanity checks. Add checks for fuse.h, pyconfig.h, nginx.
Suggest how to fix missing dependencies.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 70fd1a3..be35ae9 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -111,7 +111,7 @@ clear_temp() {
fatal() {
clear_temp
- echo >&2 "Fatal: $* in ${FUNCNAME[1]} at ${BASH_SOURCE[1]} line ${BASH_LINENO[0]}"
+ echo >&2 "Fatal: $* (encountered in ${FUNCNAME[1]} at ${BASH_SOURCE[1]} line ${BASH_LINENO[0]})"
exit 1
}
@@ -145,44 +145,27 @@ exit_cleanly() {
}
sanity_checks() {
- # Make sure WORKSPACE is set
- if ! [[ -n "$WORKSPACE" ]]; then
- echo >&2 "$helpmessage"
- echo >&2
- echo >&2 "Error: WORKSPACE environment variable not set"
- echo >&2
- exit 1
- fi
-
- # Make sure virtualenv is installed
- `virtualenv --help >/dev/null 2>&1`
-
- if [[ "$?" != "0" ]]; then
- echo >&2
- echo >&2 "Error: virtualenv could not be found"
- echo >&2
- exit 1
- fi
-
- # Make sure go is installed
- `go env >/dev/null 2>&1`
-
- if [[ "$?" != "0" ]]; then
- echo >&2
- echo >&2 "Error: go could not be found"
- echo >&2
- exit 1
- fi
-
- # Make sure gcc is installed
- `gcc --help >/dev/null 2>&1`
-
- if [[ "$?" != "0" ]]; then
- echo >&2
- echo >&2 "Error: gcc could not be found"
- echo >&2
- exit 1
- fi
+ ( [[ -n "$WORKSPACE" ]] && [[ -d "$WORKSPACE/services" ]] ) \
+ || fatal "WORKSPACE environment variable not set to a source directory (see: $0 --help)"
+ echo Checking dependencies:
+ echo -n 'virtualenv: '
+ virtualenv --version \
+ || fatal "No virtualenv. Try: apt-get install virtualenv"
+ echo -n 'go: '
+ go version \
+ || fatal "No go binary. See http://golang.org/doc/install"
+ echo -n 'gcc: '
+ gcc --version | egrep ^gcc \
+ || fatal "No gcc. Try: apt-get install build-essential"
+ echo -n 'fuse.h: '
+ find /usr/include -wholename '*fuse/fuse.h' \
+ || fatal "No fuse/fuse.h. Try: apt-get install libfuse-dev"
+ echo -n 'pyconfig.h: '
+ find /usr/include -name pyconfig.h | egrep --max-count=1 . \
+ || fatal "No pyconfig.h. Try: apt-get install python-dev"
+ echo -n 'nginx: '
+ PATH="$PATH:/sbin:/usr/sbin:/usr/local/sbin" nginx -v \
+ || fatal "No nginx. Try: apt-get install nginx"
}
rotate_logfile() {
commit 31e86126eba36e27c824b769ddea304e52ceeb59
Author: Nico Cesar <nico at nicocesar.com>
Date: Thu May 21 10:27:14 2015 -0400
6061: comment cleanup and final version tested. it works
closes #6061
diff --git a/jenkins/create-plot-data-from-log.sh b/jenkins/create-plot-data-from-log.sh
index 26fd655..ed5ee53 100755
--- a/jenkins/create-plot-data-from-log.sh
+++ b/jenkins/create-plot-data-from-log.sh
@@ -1,4 +1,4 @@
-#!/bin/bash -x
+#!/bin/bash
build=$1
file=$2
@@ -19,7 +19,7 @@ fi
if [ ! -e $file ]
then
usage
- echo "$file doesn't exists! exiting"
+ echo "$file doesn't exist! exiting"
exit 2
fi
if [ ! -w $outputdir ]
@@ -30,7 +30,9 @@ then
fi
#------------------------------
-## max lines that a test will output
+## MAXLINE is the amount of lines that will read after the pattern
+## is match (the logfile could be hundred thousands lines long).
+## 1000 should be safe enough to capture all the output of the individual test
MAXLINES=1000
## TODO: check $build and $file make sense
@@ -40,14 +42,15 @@ for test in \
test_Create,_show,_and_update_description_for_large_collection_with_manifest_text_of_100000 \
test_Create_one_large_collection_of_20000000_and_one_small_collection_of_10000_and_combine_them
do
-(zgrep -i -E -A$MAXLINES "^[A-Za-z0-9]+Test: $test" $file && echo "----") | tail --lines=+3|grep -B$MAXLINES -E "^-*$" -m1 > $outputdir/$test-$build.txt
+ cleaned_test=$(echo $test | tr -d ",.:;/")
+ (zgrep -i -E -A$MAXLINES "^[A-Za-z0-9]+Test: $test" $file && echo "----") | tail -n +1 | tail --lines=+3|grep -B$MAXLINES -E "^-*$" -m1 > $outputdir/$cleaned_test-$build.txt
result=$?
if [ $result -eq 0 ]
then
- echo processing $outputdir/$test-$build.txt creating $outputdir/$test.csv
- echo $(grep ^Completed $outputdir/$test-$build.txt | perl -n -e '/^Completed (.*) in [0-9]+ms.*$/;print "".++$line."-$1,";' | perl -p -e 's/,$//g'|tr " " "_" ) > $outputdir/$test.csv
- echo $(grep ^Completed $outputdir/$test-$build.txt | perl -n -e '/^Completed.*in ([0-9]+)ms.*$/;print "$1,";' | perl -p -e 's/,$//g' ) >> $outputdir/$test.csv
- #echo URL=https://ci.curoverse.com/view/job/arvados-api-server/ws/apps/workbench/log/$test-$build.txt/*view*/ >> $outputdir/$test.properties
+ echo processing $outputdir/$cleaned_test-$build.txt creating $outputdir/$cleaned_test.csv
+ echo $(grep ^Completed $outputdir/$cleaned_test-$build.txt | perl -n -e '/^Completed (.*) in [0-9]+ms.*$/;print "".++$line."-$1,";' | perl -p -e 's/,$//g'|tr " " "_" ) > $outputdir/$cleaned_test.csv
+ echo $(grep ^Completed $outputdir/$cleaned_test-$build.txt | perl -n -e '/^Completed.*in ([0-9]+)ms.*$/;print "$1,";' | perl -p -e 's/,$//g' ) >> $outputdir/$cleaned_test.csv
+ #echo URL=https://ci.curoverse.com/view/job/arvados-api-server/ws/apps/workbench/log/$cleaned_test-$build.txt/*view*/ >> $outputdir/$test.properties
else
echo "$test was't found on $file"
fi
commit 23b42a86c646eed10830b4b34ec46173728140d0
Author: Nico Cesar <nico at nicocesar.com>
Date: Wed May 20 11:43:31 2015 -0400
6061: plot data
added create plot data script to exit_cleanly()
3 initial test to use
Matching more generic test names
refs #6061
diff --git a/jenkins/create-plot-data-from-log.sh b/jenkins/create-plot-data-from-log.sh
index 60516a5..26fd655 100755
--- a/jenkins/create-plot-data-from-log.sh
+++ b/jenkins/create-plot-data-from-log.sh
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/bin/bash -x
build=$1
file=$2
@@ -36,19 +36,19 @@ MAXLINES=1000
## TODO: check $build and $file make sense
for test in \
- test_Collection_page_renders_name \
- test_combine_selected_collections_into_new_collection \
- test_combine_selected_collection_files_into_new_collection_active_foo_collection_in_aproject_true \
- test_combine_selected_collection_files_into_new_collection_active_foo_file_false \
- test_combine_selected_collection_files_into_new_collection_project_viewer_foo_collection_in_aproject_false \
- test_combine_selected_collection_files_into_new_collection_project_viewer_foo_file_false \
test_Create_and_show_large_collection_with_manifest_text_of_20000000 \
- test_Create__show__and_update_description_for_large_collection_with_manifest_text_of_100000 \
+ test_Create,_show,_and_update_description_for_large_collection_with_manifest_text_of_100000 \
test_Create_one_large_collection_of_20000000_and_one_small_collection_of_10000_and_combine_them
do
- zgrep -A$MAXLINES "^CollectionsTest: $test" $file | tail --lines=+3|grep -B$MAXLINES -E "^-*$" -m1 > $outputdir/$test-$build.txt
- echo processing $outputdir/$test-$build.txt creating $outputdir/$test.csv
- echo $(grep ^Completed $test-$build.txt | perl -n -e '/^Completed (.*) in [0-9]+ms.*$/;print "".++$line."-$1,";' | perl -p -e 's/,$//g'|tr " " "_" ) > $outputdir/$test.csv
- echo $(grep ^Completed $test-$build.txt | perl -n -e '/^Completed.*in ([0-9]+)ms.*$/;print "$1,";' | perl -p -e 's/,$//g' ) >> $outputdir/$test.csv
- #echo URL=https://ci.curoverse.com/view/job/arvados-api-server/ws/apps/workbench/log/$test-$build.txt/*view*/ >> $outputdir/$test.properties
+(zgrep -i -E -A$MAXLINES "^[A-Za-z0-9]+Test: $test" $file && echo "----") | tail --lines=+3|grep -B$MAXLINES -E "^-*$" -m1 > $outputdir/$test-$build.txt
+ result=$?
+ if [ $result -eq 0 ]
+ then
+ echo processing $outputdir/$test-$build.txt creating $outputdir/$test.csv
+ echo $(grep ^Completed $outputdir/$test-$build.txt | perl -n -e '/^Completed (.*) in [0-9]+ms.*$/;print "".++$line."-$1,";' | perl -p -e 's/,$//g'|tr " " "_" ) > $outputdir/$test.csv
+ echo $(grep ^Completed $outputdir/$test-$build.txt | perl -n -e '/^Completed.*in ([0-9]+)ms.*$/;print "$1,";' | perl -p -e 's/,$//g' ) >> $outputdir/$test.csv
+ #echo URL=https://ci.curoverse.com/view/job/arvados-api-server/ws/apps/workbench/log/$test-$build.txt/*view*/ >> $outputdir/$test.properties
+ else
+ echo "$test was't found on $file"
+ fi
done
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 375c4e0..70fd1a3 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -135,6 +135,7 @@ report_outcomes() {
exit_cleanly() {
trap - INT
+ create-plot-data-from-log.sh $BUILD_NUMBER "$WORKSPACE/apps/workbench/log/test.log" "$WORKSPACE/apps/workbench/log/"
rotate_logfile "$WORKSPACE/apps/workbench/log/" "test.log"
stop_services
rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
@@ -190,7 +191,6 @@ rotate_logfile() {
if [[ -f "$1/$2" ]]; then
THEDATE=`date +%Y%m%d%H%M%S`
mv "$1/$2" "$1/$THEDATE-$BUILD_NUMBER-$2"
- ./create-plot-data-from-log.sh $BUILD_NUMBER "$1/$THEDATE-$BUILD_NUMBER-$2" "$1"
gzip "$1/$THEDATE-$BUILD_NUMBER-$2"
fi
}
commit 0f0793754990012c75393cd517c85c261579cddd
Author: Nico Cesar <nico at nicocesar.com>
Date: Wed May 20 11:12:18 2015 -0400
adding 3 more test for the performance suite.
I'm guessing the names, lets see if they are good or not ...
refs #6061
diff --git a/jenkins/create-plot-data-from-log.sh b/jenkins/create-plot-data-from-log.sh
index daf3ef8..60516a5 100755
--- a/jenkins/create-plot-data-from-log.sh
+++ b/jenkins/create-plot-data-from-log.sh
@@ -41,7 +41,10 @@ for test in \
test_combine_selected_collection_files_into_new_collection_active_foo_collection_in_aproject_true \
test_combine_selected_collection_files_into_new_collection_active_foo_file_false \
test_combine_selected_collection_files_into_new_collection_project_viewer_foo_collection_in_aproject_false \
- test_combine_selected_collection_files_into_new_collection_project_viewer_foo_file_false
+ test_combine_selected_collection_files_into_new_collection_project_viewer_foo_file_false \
+ test_Create_and_show_large_collection_with_manifest_text_of_20000000 \
+ test_Create__show__and_update_description_for_large_collection_with_manifest_text_of_100000 \
+ test_Create_one_large_collection_of_20000000_and_one_small_collection_of_10000_and_combine_them
do
zgrep -A$MAXLINES "^CollectionsTest: $test" $file | tail --lines=+3|grep -B$MAXLINES -E "^-*$" -m1 > $outputdir/$test-$build.txt
echo processing $outputdir/$test-$build.txt creating $outputdir/$test.csv
commit a43a3fb3b393b125b7c6d4139375b4bd95890cfa
Author: Nico Cesar <nico at nicocesar.com>
Date: Tue May 19 14:44:18 2015 -0400
adding the creation of plot data that jenkins plot pluging will be able to read
refs #6061
diff --git a/jenkins/create-plot-data-from-log.sh b/jenkins/create-plot-data-from-log.sh
new file mode 100755
index 0000000..daf3ef8
--- /dev/null
+++ b/jenkins/create-plot-data-from-log.sh
@@ -0,0 +1,51 @@
+#!/bin/bash
+
+build=$1
+file=$2
+outputdir=$3
+
+usage() {
+ echo "./$0 build_number file_to_parse output_dir"
+ echo "this script will use the build output to generate *csv and *txt"
+ echo "for jenkins plugin plot https://github.com/jenkinsci/plot-plugin/"
+}
+
+if [ $# -ne 3 ]
+then
+ usage
+ exit 1
+fi
+
+if [ ! -e $file ]
+then
+ usage
+ echo "$file doesn't exists! exiting"
+ exit 2
+fi
+if [ ! -w $outputdir ]
+then
+ usage
+ echo "$outputdir isn't writeable! exiting"
+ exit 3
+fi
+
+#------------------------------
+## max lines that a test will output
+MAXLINES=1000
+
+## TODO: check $build and $file make sense
+
+for test in \
+ test_Collection_page_renders_name \
+ test_combine_selected_collections_into_new_collection \
+ test_combine_selected_collection_files_into_new_collection_active_foo_collection_in_aproject_true \
+ test_combine_selected_collection_files_into_new_collection_active_foo_file_false \
+ test_combine_selected_collection_files_into_new_collection_project_viewer_foo_collection_in_aproject_false \
+ test_combine_selected_collection_files_into_new_collection_project_viewer_foo_file_false
+do
+ zgrep -A$MAXLINES "^CollectionsTest: $test" $file | tail --lines=+3|grep -B$MAXLINES -E "^-*$" -m1 > $outputdir/$test-$build.txt
+ echo processing $outputdir/$test-$build.txt creating $outputdir/$test.csv
+ echo $(grep ^Completed $test-$build.txt | perl -n -e '/^Completed (.*) in [0-9]+ms.*$/;print "".++$line."-$1,";' | perl -p -e 's/,$//g'|tr " " "_" ) > $outputdir/$test.csv
+ echo $(grep ^Completed $test-$build.txt | perl -n -e '/^Completed.*in ([0-9]+)ms.*$/;print "$1,";' | perl -p -e 's/,$//g' ) >> $outputdir/$test.csv
+ #echo URL=https://ci.curoverse.com/view/job/arvados-api-server/ws/apps/workbench/log/$test-$build.txt/*view*/ >> $outputdir/$test.properties
+done
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 8e58d67..375c4e0 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -185,10 +185,12 @@ sanity_checks() {
}
rotate_logfile() {
+ # i.e. rotate_logfile "$WORKSPACE/apps/workbench/log/" "test.log"
# $BUILD_NUMBER is set by Jenkins if this script is being called as part of a Jenkins run
if [[ -f "$1/$2" ]]; then
THEDATE=`date +%Y%m%d%H%M%S`
mv "$1/$2" "$1/$THEDATE-$BUILD_NUMBER-$2"
+ ./create-plot-data-from-log.sh $BUILD_NUMBER "$1/$THEDATE-$BUILD_NUMBER-$2" "$1"
gzip "$1/$THEDATE-$BUILD_NUMBER-$2"
fi
}
commit 9a4afad134f563d41f2a0c84b4389698df07a8db
Author: Brett Smith <brett at curoverse.com>
Date: Tue May 12 17:01:39 2015 -0400
3793: Build Docker cleaner service in run-build-packages.sh.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 5ba1eb9..3b9fc15 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -81,6 +81,16 @@ if [[ "$?" != 0 ]]; then
exit 1
fi
+if ! easy_install3 --version >/dev/null; then
+ cat >&2 <<EOF
+$helpmessage
+
+Error: easy_install3 (from python3-setuptools) not found
+
+EOF
+ exit 1
+fi
+
if [[ "$DEBUG" != 0 ]]; then
echo "Workspace is $WORKSPACE"
fi
@@ -130,27 +140,28 @@ build_and_scp_deb () {
PACKAGE=$1
shift
# The name of the package to build. Defaults to $PACKAGE.
- PACKAGE_NAME=$1
+ PACKAGE_NAME=${1:-$PACKAGE}
shift
# Optional: the vendor of the package. Should be "Curoverse, Inc." for
# packages of our own software. Passed to fpm --vendor.
VENDOR=$1
shift
# The type of source package. Passed to fpm -s. Default "python".
- PACKAGE_TYPE=$1
+ PACKAGE_TYPE=${1:-python}
shift
+
+ if [ "python3" = "$PACKAGE_TYPE" ]; then
+ # fpm does not actually support this package type. Instead we recognize
+ # it as a convenience shortcut to add several necessary arguments to
+ # fpm's command line later.
+ PACKAGE_TYPE=python
+ set -- "$@" --python-bin python3 --python-easyinstall easy_install3 \
+ --python-package-name-prefix python3
+ fi
# Optional: the package version number. Passed to fpm -v.
VERSION=$1
shift
- if [[ "$PACKAGE_NAME" == "" ]]; then
- PACKAGE_NAME=$PACKAGE
- fi
-
- if [[ "$PACKAGE_TYPE" == "" ]]; then
- PACKAGE_TYPE='python'
- fi
-
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "deb" "-x" "usr/local/lib/python2.7/dist-packages/tests")
if [[ "$PACKAGE_NAME" != "$PACKAGE" ]]; then
@@ -461,6 +472,10 @@ cd $WORKSPACE/debs
# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
+# The Docker image cleaner
+cd $WORKSPACE/debs
+build_and_scp_deb $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curoverse, Inc.' 'python3' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/dockercleaner/arvados_docker_cleaner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Docker image cleaner"
+
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
@@ -468,6 +483,7 @@ for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pycrypto backports.ssl_match_hostname; do
build_and_scp_deb "$deppkg"
done
+build_and_scp_deb docker-py python3-docker-py python3
# cwltool from common-workflow-language. We use this in arv-run-pipeline-instance.
# We use $WORKSPACE/common-workflow-language as the clean directory from which to build the cwltool package
commit 123d3d778051c90b6be39b0f0b25da24d025d382
Author: Brett Smith <brett at curoverse.com>
Date: Tue May 12 14:57:20 2015 -0400
3793: Add services/dockercleaner to run-tests.sh.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 91c7cac..8e58d67 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -56,6 +56,7 @@ apps/workbench_profile
doc
services/api
services/crunchstat
+services/dockercleaner
services/fuse
services/keepproxy
services/keepstore
@@ -79,6 +80,7 @@ unset $(env | cut -d= -f1 | grep \^ARVADOS_)
GITDIR=
GOPATH=
VENVDIR=
+VENV3DIR=
PYTHONPATH=
GEMHOME=
@@ -90,7 +92,7 @@ skip_install=
declare -A leave_temp
clear_temp() {
leaving=""
- for var in VENVDIR GOPATH GITDIR GEMHOME
+ for var in VENVDIR VENV3DIR GOPATH GITDIR GEMHOME
do
if [[ -z "${leave_temp[$var]}" ]]
then
@@ -221,6 +223,7 @@ do
;;
--leave-temp)
leave_temp[VENVDIR]=1
+ leave_temp[VENV3DIR]=1
leave_temp[GOPATH]=1
leave_temp[GEMHOME]=1
;;
@@ -295,7 +298,7 @@ cd "$WORKSPACE"
find -name '*.pyc' -delete
# Set up temporary install dirs (unless existing dirs were supplied)
-for tmpdir in VENVDIR GOPATH GEMHOME
+for tmpdir in VENVDIR VENV3DIR GOPATH GEMHOME
do
if [[ -n "${!tmpdir}" ]]; then
leave_temp[$tmpdir]=1
@@ -401,6 +404,23 @@ fi
echo "pip install -q PyYAML"
pip install --quiet PyYAML || fatal "pip install PyYAML failed"
+# If Python 3 is available, set up its virtualenv in $VENV3DIR.
+# Otherwise, skip dependent tests.
+PYTHON3=$(which python3)
+if [ "0" = "$?" ]; then
+ virtualenv --python "$PYTHON3" --setuptools "$VENV3DIR" \
+ || fatal "python3 virtualenv $VENV3DIR failed"
+else
+ PYTHON3=
+ skip[services/dockercleaner]=1
+ cat >&2 <<EOF
+
+Warning: python3 could not be found
+services/dockercleaner install and tests will be skipped
+
+EOF
+fi
+
checkexit() {
if [[ "$1" != "0" ]]; then
title "!!!!!! $2 FAILED !!!!!!"
@@ -448,8 +468,10 @@ do_test_once() {
fi
elif [[ "$2" == "pip" ]]
then
- cd "$WORKSPACE/$1" \
- && python setup.py test ${testargs[$1]}
+ # $3 can name a path directory for us to use, including trailing
+ # slash; e.g., the bin/ subdirectory of a virtualenv.
+ cd "$WORKSPACE/$1" \
+ && "${3}python" setup.py test ${testargs[$1]}
elif [[ "$2" != "" ]]
then
"test_$2"
@@ -475,6 +497,9 @@ do_install() {
go get -t "git.curoverse.com/arvados.git/$1"
elif [[ "$2" == "pip" ]]
then
+ # $3 can name a path directory for us to use, including trailing
+ # slash; e.g., the bin/ subdirectory of a virtualenv.
+
# Need to change to a different directory after creating
# the source dist package to avoid a pip bug.
# see https://arvados.org/issues/5766 for details.
@@ -485,10 +510,10 @@ do_install() {
# install" ensures that we've actually install the local package
# we just built.
cd "$WORKSPACE/$1" \
- && python setup.py sdist rotate --keep=1 --match .tar.gz \
+ && "${3}python" setup.py sdist rotate --keep=1 --match .tar.gz \
&& cd "$WORKSPACE" \
- && pip install --quiet "$WORKSPACE/$1/dist"/*.tar.gz \
- && pip install --quiet --no-deps --ignore-installed "$WORKSPACE/$1/dist"/*.tar.gz
+ && "${3}pip" install --quiet "$WORKSPACE/$1/dist"/*.tar.gz \
+ && "${3}pip" install --quiet --no-deps --ignore-installed "$WORKSPACE/$1/dist"/*.tar.gz
elif [[ "$2" != "" ]]
then
"install_$2"
@@ -559,6 +584,9 @@ for p in "${pythonstuff[@]}"
do
do_install "$p" pip
done
+if [ -n "$PYTHON3" ]; then
+ do_install services/dockercleaner pip "$VENV3DIR/bin/"
+fi
install_apiserver() {
cd "$WORKSPACE/services/api" \
@@ -677,6 +705,7 @@ for p in "${pythonstuff[@]}"
do
do_test "$p" pip
done
+do_test services/dockercleaner pip "$VENV3DIR/bin/"
for g in "${gostuff[@]}"
do
commit 1553f4842dba02aaca9fd7e5a9ac426d7532c462
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Apr 22 11:07:16 2015 -0400
5766: Make sure it doesn't try to fetch arvados packages from PyPi, make sure
we only install the local ones.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index ef2e40d..91c7cac 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -18,6 +18,7 @@ Options:
--skip-install Do not run any install steps. Just run tests.
You should provide GOPATH, GEMHOME, and VENVDIR options
from a previous invocation if you use this option.
+--only-install Run specific install step
WORKSPACE=path Arvados source tree to test.
CONFIGSRC=path Dir with api server config files to copy into source tree.
(If none given, leave config files alone in source tree.)
@@ -214,6 +215,10 @@ do
--skip-install)
skip_install=1
;;
+ --only-install)
+ skip_install=1
+ only_install="$1"; shift
+ ;;
--leave-temp)
leave_temp[VENVDIR]=1
leave_temp[GOPATH]=1
@@ -377,6 +382,10 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
. "$VENVDIR/bin/activate"
+# When re-using $VENVDIR, upgrade any packages (except arvados) that are
+# already installed
+pip install --quiet --upgrade `pip freeze | grep -v arvados | cut -f1 -d=`
+
# Note: this must be the last time we change PATH, otherwise rvm will
# whine a lot.
setup_ruby_environment
@@ -390,7 +399,7 @@ fi
# Needed for run_test_server.py which is used by certain (non-Python) tests.
echo "pip install -q PyYAML"
-pip install -q PyYAML || fatal "pip install PyYAML failed"
+pip install --quiet PyYAML || fatal "pip install PyYAML failed"
checkexit() {
if [[ "$1" != "0" ]]; then
@@ -457,7 +466,7 @@ do_test_once() {
}
do_install() {
- if [[ -z "$skip_install" ]]
+ if [[ -z "$skip_install" || (-n "$only_install" && "$only_install" == "$1") ]]
then
title "Running $1 install"
timer_reset
@@ -466,9 +475,20 @@ do_install() {
go get -t "git.curoverse.com/arvados.git/$1"
elif [[ "$2" == "pip" ]]
then
+ # Need to change to a different directory after creating
+ # the source dist package to avoid a pip bug.
+ # see https://arvados.org/issues/5766 for details.
+
+ # Also need to install twice, because if it belives the package is
+ # already installed, pip it won't install it. So the first "pip
+ # install" ensures that the dependencies are met, the second "pip
+ # install" ensures that we've actually install the local package
+ # we just built.
cd "$WORKSPACE/$1" \
&& python setup.py sdist rotate --keep=1 --match .tar.gz \
- && pip install -q --upgrade dist/*.tar.gz
+ && cd "$WORKSPACE" \
+ && pip install --quiet "$WORKSPACE/$1/dist"/*.tar.gz \
+ && pip install --quiet --no-deps --ignore-installed "$WORKSPACE/$1/dist"/*.tar.gz
elif [[ "$2" != "" ]]
then
"install_$2"
commit ffa8b415658e6d3ba3836c6e208fe35979de0bf4
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Apr 8 22:09:22 2015 -0400
5416: Support testargs for go bits, like "services/arv-git-httpd_test=-check.vv".
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 6f5dd38..ef2e40d 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -27,6 +27,9 @@ sdk/python_test="--test-suite test.test_keep_locator"
Restrict Python SDK tests to the given class
apps/workbench_test="TEST=test/integration/pipeline_instances_test.rb"
Restrict Workbench tests to the given file
+services/arv-git-httpd_test="-check.vv"
+ Show all log messages, even when tests pass (also works
+ with services/keepstore_test etc.)
ARVADOS_DEBUG=1
Print more debug messages
envvar=value Set \$envvar to value. Primarily useful for WORKSPACE,
@@ -424,7 +427,16 @@ do_test_once() {
timer_reset
if [[ "$2" == "go" ]]
then
- go test ${testargs[$1]} "git.curoverse.com/arvados.git/$1"
+ if [[ -n "${testargs[$1]}" ]]
+ then
+ # "go test -check.vv giturl" doesn't work, but this
+ # does:
+ cd "$WORKSPACE/$1" && go test ${testargs[$1]}
+ else
+ # The above form gets verbose even when testargs is
+ # empty, so use this form in such cases:
+ go test "git.curoverse.com/arvados.git/$1"
+ fi
elif [[ "$2" == "pip" ]]
then
cd "$WORKSPACE/$1" \
commit a3465f97627494bde0f148e27559046308cbedda
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Apr 8 10:17:16 2015 -0400
5416: Run keepproxy, arv-git-httpd, and SSL proxy services during Workbench test suites.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 6700712..6f5dd38 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -130,7 +130,7 @@ report_outcomes() {
exit_cleanly() {
trap - INT
rotate_logfile "$WORKSPACE/apps/workbench/log/" "test.log"
- stop_api
+ stop_services
rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
report_outcomes
clear_temp
@@ -243,7 +243,23 @@ start_api() {
&& (env | egrep ^ARVADOS)
}
-stop_api() {
+start_nginx_proxy_services() {
+ echo 'Starting keepproxy, arv-git-httpd, and nginx ssl proxy...'
+ cd "$WORKSPACE" \
+ && python sdk/python/tests/run_test_server.py start_keep_proxy \
+ && python sdk/python/tests/run_test_server.py start_arv-git-httpd \
+ && python sdk/python/tests/run_test_server.py start_nginx \
+ && export ARVADOS_TEST_PROXY_SERVICES=1
+}
+
+stop_services() {
+ if [[ -n "$ARVADOS_TEST_PROXY_SERVICES" ]]; then
+ unset ARVADOS_TEST_PROXY_SERVICES
+ cd "$WORKSPACE" \
+ && python sdk/python/tests/run_test_server.py stop_nginx \
+ && python sdk/python/tests/run_test_server.py stop_arv-git-httpd \
+ && python sdk/python/tests/run_test_server.py stop_keep_proxy
+ fi
if [[ -n "$ARVADOS_TEST_API_HOST" ]]; then
unset ARVADOS_TEST_API_HOST
cd "$WORKSPACE" \
@@ -595,7 +611,7 @@ test_doclinkchecker() {
}
do_test doc doclinkchecker
-stop_api
+stop_services
test_apiserver() {
cd "$WORKSPACE/services/api" \
@@ -636,19 +652,22 @@ do
done
test_workbench() {
- cd "$WORKSPACE/apps/workbench" \
+ start_nginx_proxy_services \
+ && cd "$WORKSPACE/apps/workbench" \
&& RAILS_ENV=test bundle exec rake test TESTOPTS=-v ${testargs[apps/workbench]}
}
do_test apps/workbench workbench
test_workbench_benchmark() {
- cd "$WORKSPACE/apps/workbench" \
+ start_nginx_proxy_services \
+ && cd "$WORKSPACE/apps/workbench" \
&& RAILS_ENV=test bundle exec rake test:benchmark ${testargs[apps/workbench_benchmark]}
}
do_test apps/workbench_benchmark workbench_benchmark
test_workbench_profile() {
- cd "$WORKSPACE/apps/workbench" \
+ start_nginx_proxy_services \
+ && cd "$WORKSPACE/apps/workbench" \
&& RAILS_ENV=test bundle exec rake test:profile ${testargs[apps/workbench_profile]}
}
do_test apps/workbench_profile workbench_profile
commit 5b7e2250d5802b20b73242ae4d0641b32f72cce7
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Apr 2 20:15:58 2015 -0400
5416: Terminate connections on the configured test database (not necessarily arvados_test).
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index f835f95..6700712 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -548,9 +548,11 @@ install_apiserver() {
&& git add tmp \
&& git commit -m 'initial commit'
- # Clear out any lingering postgresql connections to arvados_test, so that we can drop it
- # This assumes the current user is a postgresql superuser
- psql arvados_test -c "SELECT pg_terminate_backend (pg_stat_activity.procpid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'arvados_test';" 2>/dev/null
+ # Clear out any lingering postgresql connections to the test
+ # database, so that we can drop it. This assumes the current user
+ # is a postgresql superuser.
+ test_database=$(python -c "import yaml; print yaml.load(file('config/database.yml'))['test']['database']")
+ psql "$test_database" -c "SELECT pg_terminate_backend (pg_stat_activity.procpid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = '$test_database';" 2>/dev/null
cd "$WORKSPACE/services/api" \
&& RAILS_ENV=test bundle exec rake db:drop \
commit 4bfcd9f168a3371b19bbd12429a8da37883256c6
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Apr 6 09:45:40 2015 -0400
run-tests.sh can now run the arv-git-httpd tests too.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index c81f3bd..f835f95 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -56,6 +56,7 @@ services/fuse
services/keepproxy
services/keepstore
services/nodemanager
+services/arv-git-httpd
sdk/cli
sdk/python
sdk/ruby
commit 422349074a47c6cf6394234a3dba975c84b94b09
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Apr 3 11:11:26 2015 -0400
Be consistent in our package and binary naming:
arv-git-httpd becomes arvados-git-httpd
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 62bd6f9..5ba1eb9 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -426,7 +426,7 @@ fi
go get "git.curoverse.com/arvados.git/services/arv-git-httpd"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/arv-git-httpd=/usr/bin/arv-git-httpd arv-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
+build_and_scp_deb $GOPATH/bin/arv-git-httpd=/usr/bin/arvados-git-httpd arvados-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
# crunchstat
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
commit 72c3457917a25dede1695c8abda316aa4e5afd5e
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Mar 17 11:01:55 2015 -0400
5416: Add --retry flag: when a test suite fails, prompt to try that suite again.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 38d8318..c81f3bd 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -215,6 +215,9 @@ do
leave_temp[GOPATH]=1
leave_temp[GEMHOME]=1
;;
+ --retry)
+ retry=1
+ ;;
*_test=*)
suite="${arg%%_test=*}"
args="${arg#*=}"
@@ -370,11 +373,11 @@ echo "pip install -q PyYAML"
pip install -q PyYAML || fatal "pip install PyYAML failed"
checkexit() {
- if [[ "$?" != "0" ]]; then
- title "!!!!!! $1 FAILED !!!!!!"
- failures+=("$1 (`timer`)")
+ if [[ "$1" != "0" ]]; then
+ title "!!!!!! $2 FAILED !!!!!!"
+ failures+=("$2 (`timer`)")
else
- successes+=("$1 (`timer`)")
+ successes+=("$2 (`timer`)")
fi
}
@@ -387,6 +390,17 @@ timer() {
}
do_test() {
+ while ! do_test_once ${@} && [[ "$retry" == 1 ]]
+ do
+ read -p 'Try again? [Y/n] ' x
+ if [[ "$x" != "y" ]] && [[ "$x" != "" ]]
+ then
+ break
+ fi
+ done
+}
+
+do_test_once() {
if [[ -z "${skip[$1]}" ]] && ( [[ -z "$only" ]] || [[ "$only" == "$1" ]] )
then
title "Running $1 tests"
@@ -404,8 +418,10 @@ do_test() {
else
"test_$1"
fi
- checkexit "$1 tests"
+ result="$?"
+ checkexit $result "$1 tests"
title "End of $1 tests (`timer`)"
+ return $result
else
title "Skipping $1 tests"
fi
@@ -430,7 +446,7 @@ do_install() {
else
"install_$1"
fi
- checkexit "$1 install"
+ checkexit $? "$1 install"
title "End of $1 install (`timer`)"
else
title "Skipping $1 install"
commit bf95f654a1587dceb6d2d27c03d6da84a5a2a92a
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Mar 17 11:00:45 2015 -0400
5416: Add arv-git-httpd.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 006e3a4..62bd6f9 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -413,6 +413,21 @@ go get "git.curoverse.com/arvados.git/services/datamanager"
cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/datamanager=/usr/bin/arvados-data-manager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
+# arv-git-httpd
+cd "$GOPATH/src/git.curoverse.com/arvados.git/services/arv-git-httpd"
+ARVGITHTTPD_VERSION=$(version_from_git)
+ARVGITHTTPD_TIMESTAMP=$(timestamp_from_git)
+
+if [[ "$GO_SDK_TIMESTAMP" -gt "$ARVGITHTTPD_TIMESTAMP" ]]; then
+ PKG_VERSION=$GO_SDK_VERSION
+else
+ PKG_VERSION=$ARVGITHTTPD_VERSION
+fi
+
+go get "git.curoverse.com/arvados.git/services/arv-git-httpd"
+cd $WORKSPACE/debs
+build_and_scp_deb $GOPATH/bin/arv-git-httpd=/usr/bin/arv-git-httpd arv-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
+
# crunchstat
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
PKG_VERSION=$(version_from_git)
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 713ffda..38d8318 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -544,6 +544,7 @@ do_install services/api apiserver
declare -a gostuff
gostuff=(
+ services/arv-git-httpd
services/crunchstat
services/keepstore
services/keepproxy
commit 0fbece8ea4acf263da8bc312ba3451c1adb7ea43
Author: Brett Smith <brett at curoverse.com>
Date: Thu May 21 10:21:45 2015 -0400
3793: Fix how python3 fpm arguments are added.
The previous code would add arguments such that they could be
interpreted as the package version. Avoid this by adding arguments
after we're done consuming positional arguments.
Refs #3793.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 95a507a..bb0877e 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -149,18 +149,18 @@ build_and_scp_deb () {
# The type of source package. Passed to fpm -s. Default "python".
PACKAGE_TYPE=${1:-python}
shift
+ # Optional: the package version number. Passed to fpm -v.
+ VERSION=$1
+ shift
+ # fpm does not actually support a python3 package type. Instead we recognize
+ # it as a convenience shortcut to add several necessary arguments to
+ # fpm's command line later, after we're done handling positional arguments.
if [ "python3" = "$PACKAGE_TYPE" ]; then
- # fpm does not actually support this package type. Instead we recognize
- # it as a convenience shortcut to add several necessary arguments to
- # fpm's command line later.
PACKAGE_TYPE=python
set -- "$@" --python-bin python3 --python-easyinstall easy_install3 \
--python-package-name-prefix python3 --depends python3
fi
- # Optional: the package version number. Passed to fpm -v.
- VERSION=$1
- shift
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "deb" "-x" "usr/local/lib/python2.7/dist-packages/tests")
commit 9fa534460eaa10ac16d1de355e45898517306ec5
Author: Brett Smith <brett at curoverse.com>
Date: Thu May 21 10:03:36 2015 -0400
Improve packaging of third-party Python 3 packages.
* Fix the bug where we're passing in "python3" as the vendor argument
to build_and_scp_deb.
* Build more dependencies that docker-py needs.
* Automatically add a dependency on the python3 package.
Refs #3793.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 3b9fc15..95a507a 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -156,7 +156,7 @@ build_and_scp_deb () {
# fpm's command line later.
PACKAGE_TYPE=python
set -- "$@" --python-bin python3 --python-easyinstall easy_install3 \
- --python-package-name-prefix python3
+ --python-package-name-prefix python3 --depends python3
fi
# Optional: the package version number. Passed to fpm -v.
VERSION=$1
@@ -483,7 +483,11 @@ for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pycrypto backports.ssl_match_hostname; do
build_and_scp_deb "$deppkg"
done
-build_and_scp_deb docker-py python3-docker-py python3
+# Python 3 dependencies
+for deppkg in docker-py six requests; do
+ # The empty string is the vendor argument: these aren't Curoverse software.
+ build_and_scp_deb "$deppkg" "python3-$deppkg" "" python3
+done
# cwltool from common-workflow-language. We use this in arv-run-pipeline-instance.
# We use $WORKSPACE/common-workflow-language as the clean directory from which to build the cwltool package
commit 08d02e15e140740902dfab5895295c08d4ca0d16
Merge: 858cd93 42e438d
Author: Tom Clegg <tom at curoverse.com>
Date: Thu May 21 09:12:22 2015 -0400
Merge branch 'more-sanity-check-less-download' refs #5766
commit 42e438d97a95d8e46f045acfaf82298d8bb3dcab
Author: Tom Clegg <tom at curoverse.com>
Date: Thu May 21 08:57:50 2015 -0400
5766: Remove "remove dependencies" step. Detect existing PyYAML faster.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 150247c..fd0023c 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -368,10 +368,6 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
. "$VENVDIR/bin/activate"
-# When re-using $VENVDIR, upgrade any packages (except arvados) that are
-# already installed
-pip install --quiet --upgrade `pip freeze | grep -v arvados | cut -f1 -d=`
-
# Note: this must be the last time we change PATH, otherwise rvm will
# whine a lot.
setup_ruby_environment
@@ -384,8 +380,9 @@ then
fi
# Needed for run_test_server.py which is used by certain (non-Python) tests.
-echo "pip install -q PyYAML"
-pip install --quiet PyYAML || fatal "pip install PyYAML failed"
+pip freeze 2>/dev/null | egrep ^PyYAML= \
+ || pip install PyYAML \
+ || fatal "pip install PyYAML failed"
# If Python 3 is available, set up its virtualenv in $VENV3DIR.
# Otherwise, skip dependent tests.
commit 303a6e2ad510ea2a4c36e7bc2b7ce789ce0b0a7a
Author: Tom Clegg <tom at curoverse.com>
Date: Thu May 21 08:51:21 2015 -0400
Tidy up sanity checks. Add checks for fuse.h, pyconfig.h, nginx.
Suggest how to fix missing dependencies.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 8e58d67..150247c 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -111,7 +111,7 @@ clear_temp() {
fatal() {
clear_temp
- echo >&2 "Fatal: $* in ${FUNCNAME[1]} at ${BASH_SOURCE[1]} line ${BASH_LINENO[0]}"
+ echo >&2 "Fatal: $* (encountered in ${FUNCNAME[1]} at ${BASH_SOURCE[1]} line ${BASH_LINENO[0]})"
exit 1
}
@@ -144,44 +144,27 @@ exit_cleanly() {
}
sanity_checks() {
- # Make sure WORKSPACE is set
- if ! [[ -n "$WORKSPACE" ]]; then
- echo >&2 "$helpmessage"
- echo >&2
- echo >&2 "Error: WORKSPACE environment variable not set"
- echo >&2
- exit 1
- fi
-
- # Make sure virtualenv is installed
- `virtualenv --help >/dev/null 2>&1`
-
- if [[ "$?" != "0" ]]; then
- echo >&2
- echo >&2 "Error: virtualenv could not be found"
- echo >&2
- exit 1
- fi
-
- # Make sure go is installed
- `go env >/dev/null 2>&1`
-
- if [[ "$?" != "0" ]]; then
- echo >&2
- echo >&2 "Error: go could not be found"
- echo >&2
- exit 1
- fi
-
- # Make sure gcc is installed
- `gcc --help >/dev/null 2>&1`
-
- if [[ "$?" != "0" ]]; then
- echo >&2
- echo >&2 "Error: gcc could not be found"
- echo >&2
- exit 1
- fi
+ ( [[ -n "$WORKSPACE" ]] && [[ -d "$WORKSPACE/services" ]] ) \
+ || fatal "WORKSPACE environment variable not set to a source directory (see: $0 --help)"
+ echo Checking dependencies:
+ echo -n 'virtualenv: '
+ virtualenv --version \
+ || fatal "No virtualenv. Try: apt-get install virtualenv"
+ echo -n 'go: '
+ go version \
+ || fatal "No go binary. See http://golang.org/doc/install"
+ echo -n 'gcc: '
+ gcc --version | egrep ^gcc \
+ || fatal "No gcc. Try: apt-get install build-essential"
+ echo -n 'fuse.h: '
+ find /usr/include -wholename '*fuse/fuse.h' \
+ || fatal "No fuse/fuse.h. Try: apt-get install libfuse-dev"
+ echo -n 'pyconfig.h: '
+ find /usr/include -name pyconfig.h | egrep --max-count=1 . \
+ || fatal "No pyconfig.h. Try: apt-get install python-dev"
+ echo -n 'nginx: '
+ PATH="$PATH:/sbin:/usr/sbin:/usr/local/sbin" nginx -v \
+ || fatal "No nginx. Try: apt-get install nginx"
}
rotate_logfile() {
commit 858cd93950d0fd7b621e9e4656261856e60856ad
Merge: 274c7ab 034677a
Author: Brett Smith <brettcsmith at brettcsmith.org>
Date: Sat May 16 17:28:59 2015 -0400
Merge branch '3793-docker-cleaner-wip'
Refs #3793. Closes #6003.
commit 034677aa057f89be2d5bf923d6956164396d11cb
Author: Brett Smith <brett at curoverse.com>
Date: Tue May 12 17:01:39 2015 -0400
3793: Build Docker cleaner service in run-build-packages.sh.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 5ba1eb9..3b9fc15 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -81,6 +81,16 @@ if [[ "$?" != 0 ]]; then
exit 1
fi
+if ! easy_install3 --version >/dev/null; then
+ cat >&2 <<EOF
+$helpmessage
+
+Error: easy_install3 (from python3-setuptools) not found
+
+EOF
+ exit 1
+fi
+
if [[ "$DEBUG" != 0 ]]; then
echo "Workspace is $WORKSPACE"
fi
@@ -130,27 +140,28 @@ build_and_scp_deb () {
PACKAGE=$1
shift
# The name of the package to build. Defaults to $PACKAGE.
- PACKAGE_NAME=$1
+ PACKAGE_NAME=${1:-$PACKAGE}
shift
# Optional: the vendor of the package. Should be "Curoverse, Inc." for
# packages of our own software. Passed to fpm --vendor.
VENDOR=$1
shift
# The type of source package. Passed to fpm -s. Default "python".
- PACKAGE_TYPE=$1
+ PACKAGE_TYPE=${1:-python}
shift
+
+ if [ "python3" = "$PACKAGE_TYPE" ]; then
+ # fpm does not actually support this package type. Instead we recognize
+ # it as a convenience shortcut to add several necessary arguments to
+ # fpm's command line later.
+ PACKAGE_TYPE=python
+ set -- "$@" --python-bin python3 --python-easyinstall easy_install3 \
+ --python-package-name-prefix python3
+ fi
# Optional: the package version number. Passed to fpm -v.
VERSION=$1
shift
- if [[ "$PACKAGE_NAME" == "" ]]; then
- PACKAGE_NAME=$PACKAGE
- fi
-
- if [[ "$PACKAGE_TYPE" == "" ]]; then
- PACKAGE_TYPE='python'
- fi
-
declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "deb" "-x" "usr/local/lib/python2.7/dist-packages/tests")
if [[ "$PACKAGE_NAME" != "$PACKAGE" ]]; then
@@ -461,6 +472,10 @@ cd $WORKSPACE/debs
# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
+# The Docker image cleaner
+cd $WORKSPACE/debs
+build_and_scp_deb $WORKSPACE/services/dockercleaner arvados-docker-cleaner 'Curoverse, Inc.' 'python3' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/dockercleaner/arvados_docker_cleaner.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Docker image cleaner"
+
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
@@ -468,6 +483,7 @@ for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pycrypto backports.ssl_match_hostname; do
build_and_scp_deb "$deppkg"
done
+build_and_scp_deb docker-py python3-docker-py python3
# cwltool from common-workflow-language. We use this in arv-run-pipeline-instance.
# We use $WORKSPACE/common-workflow-language as the clean directory from which to build the cwltool package
commit acdfc466ec52fbbf394f0e0b00544e809e74cec8
Author: Brett Smith <brett at curoverse.com>
Date: Tue May 12 14:57:20 2015 -0400
3793: Add services/dockercleaner to run-tests.sh.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 91c7cac..8e58d67 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -56,6 +56,7 @@ apps/workbench_profile
doc
services/api
services/crunchstat
+services/dockercleaner
services/fuse
services/keepproxy
services/keepstore
@@ -79,6 +80,7 @@ unset $(env | cut -d= -f1 | grep \^ARVADOS_)
GITDIR=
GOPATH=
VENVDIR=
+VENV3DIR=
PYTHONPATH=
GEMHOME=
@@ -90,7 +92,7 @@ skip_install=
declare -A leave_temp
clear_temp() {
leaving=""
- for var in VENVDIR GOPATH GITDIR GEMHOME
+ for var in VENVDIR VENV3DIR GOPATH GITDIR GEMHOME
do
if [[ -z "${leave_temp[$var]}" ]]
then
@@ -221,6 +223,7 @@ do
;;
--leave-temp)
leave_temp[VENVDIR]=1
+ leave_temp[VENV3DIR]=1
leave_temp[GOPATH]=1
leave_temp[GEMHOME]=1
;;
@@ -295,7 +298,7 @@ cd "$WORKSPACE"
find -name '*.pyc' -delete
# Set up temporary install dirs (unless existing dirs were supplied)
-for tmpdir in VENVDIR GOPATH GEMHOME
+for tmpdir in VENVDIR VENV3DIR GOPATH GEMHOME
do
if [[ -n "${!tmpdir}" ]]; then
leave_temp[$tmpdir]=1
@@ -401,6 +404,23 @@ fi
echo "pip install -q PyYAML"
pip install --quiet PyYAML || fatal "pip install PyYAML failed"
+# If Python 3 is available, set up its virtualenv in $VENV3DIR.
+# Otherwise, skip dependent tests.
+PYTHON3=$(which python3)
+if [ "0" = "$?" ]; then
+ virtualenv --python "$PYTHON3" --setuptools "$VENV3DIR" \
+ || fatal "python3 virtualenv $VENV3DIR failed"
+else
+ PYTHON3=
+ skip[services/dockercleaner]=1
+ cat >&2 <<EOF
+
+Warning: python3 could not be found
+services/dockercleaner install and tests will be skipped
+
+EOF
+fi
+
checkexit() {
if [[ "$1" != "0" ]]; then
title "!!!!!! $2 FAILED !!!!!!"
@@ -448,8 +468,10 @@ do_test_once() {
fi
elif [[ "$2" == "pip" ]]
then
- cd "$WORKSPACE/$1" \
- && python setup.py test ${testargs[$1]}
+ # $3 can name a path directory for us to use, including trailing
+ # slash; e.g., the bin/ subdirectory of a virtualenv.
+ cd "$WORKSPACE/$1" \
+ && "${3}python" setup.py test ${testargs[$1]}
elif [[ "$2" != "" ]]
then
"test_$2"
@@ -475,6 +497,9 @@ do_install() {
go get -t "git.curoverse.com/arvados.git/$1"
elif [[ "$2" == "pip" ]]
then
+ # $3 can name a path directory for us to use, including trailing
+ # slash; e.g., the bin/ subdirectory of a virtualenv.
+
# Need to change to a different directory after creating
# the source dist package to avoid a pip bug.
# see https://arvados.org/issues/5766 for details.
@@ -485,10 +510,10 @@ do_install() {
# install" ensures that we've actually install the local package
# we just built.
cd "$WORKSPACE/$1" \
- && python setup.py sdist rotate --keep=1 --match .tar.gz \
+ && "${3}python" setup.py sdist rotate --keep=1 --match .tar.gz \
&& cd "$WORKSPACE" \
- && pip install --quiet "$WORKSPACE/$1/dist"/*.tar.gz \
- && pip install --quiet --no-deps --ignore-installed "$WORKSPACE/$1/dist"/*.tar.gz
+ && "${3}pip" install --quiet "$WORKSPACE/$1/dist"/*.tar.gz \
+ && "${3}pip" install --quiet --no-deps --ignore-installed "$WORKSPACE/$1/dist"/*.tar.gz
elif [[ "$2" != "" ]]
then
"install_$2"
@@ -559,6 +584,9 @@ for p in "${pythonstuff[@]}"
do
do_install "$p" pip
done
+if [ -n "$PYTHON3" ]; then
+ do_install services/dockercleaner pip "$VENV3DIR/bin/"
+fi
install_apiserver() {
cd "$WORKSPACE/services/api" \
@@ -677,6 +705,7 @@ for p in "${pythonstuff[@]}"
do
do_test "$p" pip
done
+do_test services/dockercleaner pip "$VENV3DIR/bin/"
for g in "${gostuff[@]}"
do
commit 274c7abf3aee7dee8bfe648f485087a5a0ab59ed
Merge: 3002e7a 8b51595
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Apr 29 16:46:01 2015 -0400
Merge branch '5766-pip-sdk-install' closes #5766
commit 3002e7a6bc67564e900a5da490ef4e6b3b43224f
Merge: ccb48a8 d648569
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Apr 22 14:27:23 2015 -0400
Merge branch '5416-ssl-proxy' refs #5416
commit 8b51595b8ee2a1a554d365e857dda5cfde440e53
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Apr 22 11:07:16 2015 -0400
5766: Make sure it doesn't try to fetch arvados packages from PyPi, make sure
we only install the local ones.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index f835f95..5ef64db 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -18,6 +18,7 @@ Options:
--skip-install Do not run any install steps. Just run tests.
You should provide GOPATH, GEMHOME, and VENVDIR options
from a previous invocation if you use this option.
+--only-install Run specific install step
WORKSPACE=path Arvados source tree to test.
CONFIGSRC=path Dir with api server config files to copy into source tree.
(If none given, leave config files alone in source tree.)
@@ -211,6 +212,10 @@ do
--skip-install)
skip_install=1
;;
+ --only-install)
+ skip_install=1
+ only_install="$1"; shift
+ ;;
--leave-temp)
leave_temp[VENVDIR]=1
leave_temp[GOPATH]=1
@@ -358,6 +363,10 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
. "$VENVDIR/bin/activate"
+# When re-using $VENVDIR, upgrade any packages (except arvados) that are
+# already installed
+pip install --quiet --upgrade `pip freeze | grep -v arvados | cut -f1 -d=`
+
# Note: this must be the last time we change PATH, otherwise rvm will
# whine a lot.
setup_ruby_environment
@@ -371,7 +380,7 @@ fi
# Needed for run_test_server.py which is used by certain (non-Python) tests.
echo "pip install -q PyYAML"
-pip install -q PyYAML || fatal "pip install PyYAML failed"
+pip install --quiet PyYAML || fatal "pip install PyYAML failed"
checkexit() {
if [[ "$1" != "0" ]]; then
@@ -429,7 +438,7 @@ do_test_once() {
}
do_install() {
- if [[ -z "$skip_install" ]]
+ if [[ -z "$skip_install" || (-n "$only_install" && "$only_install" == "$1") ]]
then
title "Running $1 install"
timer_reset
@@ -438,9 +447,20 @@ do_install() {
go get -t "git.curoverse.com/arvados.git/$1"
elif [[ "$2" == "pip" ]]
then
+ # Need to change to a different directory after creating
+ # the source dist package to avoid a pip bug.
+ # see https://arvados.org/issues/5766 for details.
+
+ # Also need to install twice, because if it belives the package is
+ # already installed, pip it won't install it. So the first "pip
+ # install" ensures that the dependencies are met, the second "pip
+ # install" ensures that we've actually install the local package
+ # we just built.
cd "$WORKSPACE/$1" \
&& python setup.py sdist rotate --keep=1 --match .tar.gz \
- && pip install -q --upgrade dist/*.tar.gz
+ && cd "$WORKSPACE" \
+ && pip install --quiet "$WORKSPACE/$1/dist"/*.tar.gz \
+ && pip install --quiet --no-deps --ignore-installed "$WORKSPACE/$1/dist"/*.tar.gz
elif [[ "$2" != "" ]]
then
"install_$2"
commit d64856940e5568d3b8bd83d60f36d13e7bc11fa2
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Apr 8 22:09:22 2015 -0400
5416: Support testargs for go bits, like "services/arv-git-httpd_test=-check.vv".
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 6f5dd38..ef2e40d 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -27,6 +27,9 @@ sdk/python_test="--test-suite test.test_keep_locator"
Restrict Python SDK tests to the given class
apps/workbench_test="TEST=test/integration/pipeline_instances_test.rb"
Restrict Workbench tests to the given file
+services/arv-git-httpd_test="-check.vv"
+ Show all log messages, even when tests pass (also works
+ with services/keepstore_test etc.)
ARVADOS_DEBUG=1
Print more debug messages
envvar=value Set \$envvar to value. Primarily useful for WORKSPACE,
@@ -424,7 +427,16 @@ do_test_once() {
timer_reset
if [[ "$2" == "go" ]]
then
- go test ${testargs[$1]} "git.curoverse.com/arvados.git/$1"
+ if [[ -n "${testargs[$1]}" ]]
+ then
+ # "go test -check.vv giturl" doesn't work, but this
+ # does:
+ cd "$WORKSPACE/$1" && go test ${testargs[$1]}
+ else
+ # The above form gets verbose even when testargs is
+ # empty, so use this form in such cases:
+ go test "git.curoverse.com/arvados.git/$1"
+ fi
elif [[ "$2" == "pip" ]]
then
cd "$WORKSPACE/$1" \
commit f9bc13bd47c5404da529d5a71153500d2ee755c7
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Apr 8 10:17:16 2015 -0400
5416: Run keepproxy, arv-git-httpd, and SSL proxy services during Workbench test suites.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 6700712..6f5dd38 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -130,7 +130,7 @@ report_outcomes() {
exit_cleanly() {
trap - INT
rotate_logfile "$WORKSPACE/apps/workbench/log/" "test.log"
- stop_api
+ stop_services
rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
report_outcomes
clear_temp
@@ -243,7 +243,23 @@ start_api() {
&& (env | egrep ^ARVADOS)
}
-stop_api() {
+start_nginx_proxy_services() {
+ echo 'Starting keepproxy, arv-git-httpd, and nginx ssl proxy...'
+ cd "$WORKSPACE" \
+ && python sdk/python/tests/run_test_server.py start_keep_proxy \
+ && python sdk/python/tests/run_test_server.py start_arv-git-httpd \
+ && python sdk/python/tests/run_test_server.py start_nginx \
+ && export ARVADOS_TEST_PROXY_SERVICES=1
+}
+
+stop_services() {
+ if [[ -n "$ARVADOS_TEST_PROXY_SERVICES" ]]; then
+ unset ARVADOS_TEST_PROXY_SERVICES
+ cd "$WORKSPACE" \
+ && python sdk/python/tests/run_test_server.py stop_nginx \
+ && python sdk/python/tests/run_test_server.py stop_arv-git-httpd \
+ && python sdk/python/tests/run_test_server.py stop_keep_proxy
+ fi
if [[ -n "$ARVADOS_TEST_API_HOST" ]]; then
unset ARVADOS_TEST_API_HOST
cd "$WORKSPACE" \
@@ -595,7 +611,7 @@ test_doclinkchecker() {
}
do_test doc doclinkchecker
-stop_api
+stop_services
test_apiserver() {
cd "$WORKSPACE/services/api" \
@@ -636,19 +652,22 @@ do
done
test_workbench() {
- cd "$WORKSPACE/apps/workbench" \
+ start_nginx_proxy_services \
+ && cd "$WORKSPACE/apps/workbench" \
&& RAILS_ENV=test bundle exec rake test TESTOPTS=-v ${testargs[apps/workbench]}
}
do_test apps/workbench workbench
test_workbench_benchmark() {
- cd "$WORKSPACE/apps/workbench" \
+ start_nginx_proxy_services \
+ && cd "$WORKSPACE/apps/workbench" \
&& RAILS_ENV=test bundle exec rake test:benchmark ${testargs[apps/workbench_benchmark]}
}
do_test apps/workbench_benchmark workbench_benchmark
test_workbench_profile() {
- cd "$WORKSPACE/apps/workbench" \
+ start_nginx_proxy_services \
+ && cd "$WORKSPACE/apps/workbench" \
&& RAILS_ENV=test bundle exec rake test:profile ${testargs[apps/workbench_profile]}
}
do_test apps/workbench_profile workbench_profile
commit decff518125e54dc044170fb4e31a5b86d566bb5
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Apr 2 20:15:58 2015 -0400
5416: Terminate connections on the configured test database (not necessarily arvados_test).
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index f835f95..6700712 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -548,9 +548,11 @@ install_apiserver() {
&& git add tmp \
&& git commit -m 'initial commit'
- # Clear out any lingering postgresql connections to arvados_test, so that we can drop it
- # This assumes the current user is a postgresql superuser
- psql arvados_test -c "SELECT pg_terminate_backend (pg_stat_activity.procpid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'arvados_test';" 2>/dev/null
+ # Clear out any lingering postgresql connections to the test
+ # database, so that we can drop it. This assumes the current user
+ # is a postgresql superuser.
+ test_database=$(python -c "import yaml; print yaml.load(file('config/database.yml'))['test']['database']")
+ psql "$test_database" -c "SELECT pg_terminate_backend (pg_stat_activity.procpid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = '$test_database';" 2>/dev/null
cd "$WORKSPACE/services/api" \
&& RAILS_ENV=test bundle exec rake db:drop \
commit ccb48a84de0d9adff1809574aac4df77b9b4575c
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Apr 6 09:45:40 2015 -0400
run-tests.sh can now run the arv-git-httpd tests too.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index c81f3bd..f835f95 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -56,6 +56,7 @@ services/fuse
services/keepproxy
services/keepstore
services/nodemanager
+services/arv-git-httpd
sdk/cli
sdk/python
sdk/ruby
commit 672107d387a86334aa70711d071bc3c2bf9c5502
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Apr 3 11:11:26 2015 -0400
Be consistent in our package and binary naming:
arv-git-httpd becomes arvados-git-httpd
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 62bd6f9..5ba1eb9 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -426,7 +426,7 @@ fi
go get "git.curoverse.com/arvados.git/services/arv-git-httpd"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/arv-git-httpd=/usr/bin/arv-git-httpd arv-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
+build_and_scp_deb $GOPATH/bin/arv-git-httpd=/usr/bin/arvados-git-httpd arvados-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
# crunchstat
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
commit 40ee991675ff42fe531bb8f2d1bd44311e9fe370
Merge: aa9dc78 a00ef86
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Mar 30 15:57:45 2015 -0400
Merge branch '5416-arv-git-httpd' refs #5416
commit aa9dc78611ff1339d02edcce329540d32fafd9b9
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Mar 24 09:54:38 2015 -0400
The easy-docker-install method should also get the keepproxy and sso
docker images.
closes #5529
diff --git a/install/easy-docker-install.sh b/install/easy-docker-install.sh
index d4d5a4e..fe6e186 100755
--- a/install/easy-docker-install.sh
+++ b/install/easy-docker-install.sh
@@ -58,8 +58,10 @@ main () {
docker_pull arvados/doc
docker_pull arvados/keep
docker_pull arvados/shell
+ docker_pull arvados/sso
docker_pull arvados/compute
docker_pull arvados/keep
+ docker_pull arvados/keepproxy
docker_pull arvados/api
docker_pull crosbymichael/skydns
docker_pull crosbymichael/skydock
commit 5cdd9c170ff4c5c3ac4c5101ee4a9ada47a4adce
Merge: 5032541 65ecef6
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Mar 23 16:49:46 2015 -0400
Merge branch '5529-build-and-test-keepproxy-docker-image'
refs #5529
commit 65ecef658fb337d97db7025bcdd4107276988543
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Mar 23 16:49:08 2015 -0400
Make sure to also push the keepproxy and sso images to the Docker registry.
refs #5529
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index e0c2288..5be7b99 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -90,7 +90,9 @@ if [[ "$ECODE" == "0" ]]; then
docker_push arvados/doc
docker_push arvados/workbench
docker_push arvados/keep
+ docker_push arvados/keepproxy
docker_push arvados/shell
+ docker_push arvados/sso
else
title "upload arvados images SKIPPED because build failed"
fi
commit a00ef8633a3fbed5baecc5c47e9a330943dc8f8e
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Mar 17 11:01:55 2015 -0400
5416: Add --retry flag: when a test suite fails, prompt to try that suite again.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 38d8318..c81f3bd 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -215,6 +215,9 @@ do
leave_temp[GOPATH]=1
leave_temp[GEMHOME]=1
;;
+ --retry)
+ retry=1
+ ;;
*_test=*)
suite="${arg%%_test=*}"
args="${arg#*=}"
@@ -370,11 +373,11 @@ echo "pip install -q PyYAML"
pip install -q PyYAML || fatal "pip install PyYAML failed"
checkexit() {
- if [[ "$?" != "0" ]]; then
- title "!!!!!! $1 FAILED !!!!!!"
- failures+=("$1 (`timer`)")
+ if [[ "$1" != "0" ]]; then
+ title "!!!!!! $2 FAILED !!!!!!"
+ failures+=("$2 (`timer`)")
else
- successes+=("$1 (`timer`)")
+ successes+=("$2 (`timer`)")
fi
}
@@ -387,6 +390,17 @@ timer() {
}
do_test() {
+ while ! do_test_once ${@} && [[ "$retry" == 1 ]]
+ do
+ read -p 'Try again? [Y/n] ' x
+ if [[ "$x" != "y" ]] && [[ "$x" != "" ]]
+ then
+ break
+ fi
+ done
+}
+
+do_test_once() {
if [[ -z "${skip[$1]}" ]] && ( [[ -z "$only" ]] || [[ "$only" == "$1" ]] )
then
title "Running $1 tests"
@@ -404,8 +418,10 @@ do_test() {
else
"test_$1"
fi
- checkexit "$1 tests"
+ result="$?"
+ checkexit $result "$1 tests"
title "End of $1 tests (`timer`)"
+ return $result
else
title "Skipping $1 tests"
fi
@@ -430,7 +446,7 @@ do_install() {
else
"install_$1"
fi
- checkexit "$1 install"
+ checkexit $? "$1 install"
title "End of $1 install (`timer`)"
else
title "Skipping $1 install"
commit 3c0d0df8f6d287e83ea959881afedddc5d4b784a
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Mar 17 11:00:45 2015 -0400
5416: Add arv-git-httpd.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 006e3a4..62bd6f9 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -413,6 +413,21 @@ go get "git.curoverse.com/arvados.git/services/datamanager"
cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/datamanager=/usr/bin/arvados-data-manager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
+# arv-git-httpd
+cd "$GOPATH/src/git.curoverse.com/arvados.git/services/arv-git-httpd"
+ARVGITHTTPD_VERSION=$(version_from_git)
+ARVGITHTTPD_TIMESTAMP=$(timestamp_from_git)
+
+if [[ "$GO_SDK_TIMESTAMP" -gt "$ARVGITHTTPD_TIMESTAMP" ]]; then
+ PKG_VERSION=$GO_SDK_VERSION
+else
+ PKG_VERSION=$ARVGITHTTPD_VERSION
+fi
+
+go get "git.curoverse.com/arvados.git/services/arv-git-httpd"
+cd $WORKSPACE/debs
+build_and_scp_deb $GOPATH/bin/arv-git-httpd=/usr/bin/arv-git-httpd arv-git-httpd 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Provides authenticated http access to Arvados-hosted git repositories."
+
# crunchstat
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
PKG_VERSION=$(version_from_git)
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 713ffda..38d8318 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -544,6 +544,7 @@ do_install services/api apiserver
declare -a gostuff
gostuff=(
+ services/arv-git-httpd
services/crunchstat
services/keepstore
services/keepproxy
commit 503254168510f46f55da0fcf6c49436c900920b5
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Mar 16 09:58:47 2015 -0400
Add python-backports.ssl-match-hostname to the list of python packages
we debianize. It's a dependency for libcloud 0.17.
Fix a typo.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 9bb4914..006e3a4 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -433,7 +433,7 @@ cd $WORKSPACE/debs
build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
# The FUSE driver
-# Please seem comment about --no-python-fix-name above; we stay consistent and do
+# Please see comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
cd $WORKSPACE/debs
# Python version numbering is obscure. Strip dashes and replace them with dots
@@ -450,7 +450,7 @@ build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curovers
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
pykka apache-libcloud requests six pyexecjs jsonschema ciso8601 \
- pycrypto; do
+ pycrypto backports.ssl_match_hostname; do
build_and_scp_deb "$deppkg"
done
commit 6c6ca86bfa400d880d051fd87c7c1fb9b8905114
Merge: 341344e 7b26427
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Mar 10 16:16:15 2015 -0400
Merge branch 'sigint' closes #5178
commit 7b264271f10d442953f216d66f439d770e0fadad
Merge: 03d58fa 341344e
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Mar 10 16:02:21 2015 -0400
Merge branch 'master' into sigint
Conflicts:
jenkins/run-tests.sh
diff --cc jenkins/run-tests.sh
index b87a7f5,a706068..713ffda
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@@ -582,6 -573,14 +583,13 @@@ test_apiserver()
}
do_test services/api apiserver
+ # Shortcut for when we're only running apiserver tests. This saves a bit of time,
+ # because we don't need to start up the api server for subsequent tests.
+ if [ ! -z "$only" ] && [ "$only" == "services/api" ]; then
+ rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
-
- clean_up
++ exit_cleanly
+ fi
+
start_api
test_ruby_sdk() {
commit 341344e17cc68abe57ed2a2ef97807f5d01d9232
Author: Brett Smith <brett at curoverse.com>
Date: Tue Mar 10 14:04:26 2015 -0400
5313: Build a .deb for the PyCrypto Python package.
libcloud's GCE driver requires this, so building it supports deploying
Node Manager on GCE. Refs #5313.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index fff87f4..9bb4914 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -449,7 +449,8 @@ build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curovers
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
- pykka apache-libcloud requests six pyexecjs jsonschema ciso8601; do
+ pykka apache-libcloud requests six pyexecjs jsonschema ciso8601 \
+ pycrypto; do
build_and_scp_deb "$deppkg"
done
commit 8c01f6309392198702e89946f4dd6eb76c3f2772
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Mar 2 09:30:32 2015 -0500
Be consistent in the naming of the arvados-data-manager package and the binary in the package.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 2355f8e..fff87f4 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -411,7 +411,7 @@ fi
go get "git.curoverse.com/arvados.git/services/datamanager"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/datamanager=/usr/bin/datamanager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
+build_and_scp_deb $GOPATH/bin/datamanager=/usr/bin/arvados-data-manager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
# crunchstat
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
commit c50da8ea8cdc343407b53e41fcaf971294cf2a65
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Mar 2 08:57:10 2015 -0500
Build arvados-data-manager deb.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 86d3157..2355f8e 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -378,11 +378,12 @@ go get "git.curoverse.com/arvados.git/services/keepstore"
cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
-# keepproxy
+# Get GO SDK version
cd "$GOPATH/src/git.curoverse.com/arvados.git/sdk/go"
GO_SDK_VERSION=$(version_from_git)
GO_SDK_TIMESTAMP=$(timestamp_from_git)
+# keepproxy
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keepproxy"
KEEPPROXY_VERSION=$(version_from_git)
KEEPPROXY_TIMESTAMP=$(timestamp_from_git)
@@ -397,6 +398,21 @@ go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
+# datamanager
+cd "$GOPATH/src/git.curoverse.com/arvados.git/services/datamanager"
+DATAMANAGER_VERSION=$(version_from_git)
+DATAMANAGER_TIMESTAMP=$(timestamp_from_git)
+
+if [[ "$GO_SDK_TIMESTAMP" -gt "$DATAMANAGER_TIMESTAMP" ]]; then
+ PKG_VERSION=$GO_SDK_VERSION
+else
+ PKG_VERSION=$DATAMANAGER_VERSION
+fi
+
+go get "git.curoverse.com/arvados.git/services/datamanager"
+cd $WORKSPACE/debs
+build_and_scp_deb $GOPATH/bin/datamanager=/usr/bin/datamanager arvados-data-manager 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Datamanager ensures block replication levels, reports on disk usage and determines which blocks should be deleted when space is needed."
+
# crunchstat
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
PKG_VERSION=$(version_from_git)
commit 91a75ac34ec1b4f78ba2b9b63fe650941ab72dce
Author: Brett Smith <brettcsmith at brettcsmith.org>
Date: Sun Feb 22 22:41:22 2015 -0500
5277: Build .deb for new ciso8601 dependency.
Refs #5277.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 1dae4bc..86d3157 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -433,7 +433,7 @@ build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curovers
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
- pykka apache-libcloud requests six pyexecjs jsonschema; do
+ pykka apache-libcloud requests six pyexecjs jsonschema ciso8601; do
build_and_scp_deb "$deppkg"
done
commit 7383891fa3dfa915967e3d15c5e678f9545ffb26
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Feb 18 11:20:26 2015 -0500
Reflect deploy changes in run-deploy.sh
Refs #5226
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index d4e31f1..51de227 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -102,6 +102,7 @@ ensure_symlink "apps/workbench/config/deploy.workbench.$IDENTIFIER.rb"
ensure_symlink "services/api/Capfile.$IDENTIFIER"
ensure_symlink "services/api/config/deploy.common.rb"
+ensure_symlink "services/api/config/deploy.curoverse.rb"
ensure_symlink "services/api/config/deploy.$IDENTIFIER.rb"
# Deploy API server
commit 02d27b88fe5afeb52e69b387929ef8d59663c09c
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Feb 14 14:21:14 2015 -0500
When run-tests.sh is invoked to only run the api server tests, do not
spin up an api server for all the tests that will not be run.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 03928ce..a706068 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -546,6 +546,13 @@ stop_api() {
&& python sdk/python/tests/run_test_server.py stop
}
+clean_up() {
+ report_outcomes
+ clear_temp
+
+ exit ${#failures}
+}
+
test_doclinkchecker() {
(
set -e
@@ -566,6 +573,14 @@ test_apiserver() {
}
do_test services/api apiserver
+# Shortcut for when we're only running apiserver tests. This saves a bit of time,
+# because we don't need to start up the api server for subsequent tests.
+if [ ! -z "$only" ] && [ "$only" == "services/api" ]; then
+ rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
+
+ clean_up
+fi
+
start_api
test_ruby_sdk() {
@@ -612,9 +627,6 @@ do_test apps/workbench_profile workbench_profile
rotate_logfile "$WORKSPACE/apps/workbench/log/" "test.log"
stop_api
-rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
-report_outcomes
-clear_temp
-
-exit ${#failures}
+rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
+clean_up
commit be33b03fe8aef6e9e23f82ee7653bc46fabef161
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Feb 14 14:06:36 2015 -0500
Make pip install quiet, no point in printing loads of useless warnings
(looking at you, PyYAML!).
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 689405b..03928ce 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -334,7 +334,8 @@ then
fi
# Needed for run_test_server.py which is used by certain (non-Python) tests.
-pip install PyYAML || fatal "pip install PyYAML failed"
+echo "pip install -q PyYAML"
+pip install -q PyYAML || fatal "pip install PyYAML failed"
checkexit() {
if [[ "$?" != "0" ]]; then
@@ -390,7 +391,7 @@ do_install() {
then
cd "$WORKSPACE/$1" \
&& python setup.py sdist rotate --keep=1 --match .tar.gz \
- && pip install --upgrade dist/*.tar.gz
+ && pip install -q --upgrade dist/*.tar.gz
elif [[ "$2" != "" ]]
then
"install_$2"
commit 03d58fa94878d11cac6d25cdb67180060d1e2fe5
Author: Tom Clegg <tom at curoverse.com>
Date: Sun Feb 8 18:59:26 2015 -0500
Clean up and exit on SIGINT.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 689405b..b87a7f5 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -126,6 +126,16 @@ report_outcomes() {
fi
}
+exit_cleanly() {
+ trap - INT
+ rotate_logfile "$WORKSPACE/apps/workbench/log/" "test.log"
+ stop_api
+ rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
+ report_outcomes
+ clear_temp
+ exit ${#failures}
+}
+
sanity_checks() {
# Make sure WORKSPACE is set
if ! [[ -n "$WORKSPACE" ]]; then
@@ -165,7 +175,6 @@ sanity_checks() {
echo >&2
exit 1
fi
-
}
rotate_logfile() {
@@ -221,6 +230,29 @@ do
esac
done
+start_api() {
+ echo 'Starting API server...'
+ cd "$WORKSPACE" \
+ && eval $(python sdk/python/tests/run_test_server.py start --auth admin) \
+ && export ARVADOS_TEST_API_HOST="$ARVADOS_API_HOST" \
+ && export ARVADOS_TEST_API_INSTALLED="$$" \
+ && (env | egrep ^ARVADOS)
+}
+
+stop_api() {
+ if [[ -n "$ARVADOS_TEST_API_HOST" ]]; then
+ unset ARVADOS_TEST_API_HOST
+ cd "$WORKSPACE" \
+ && python sdk/python/tests/run_test_server.py stop
+ fi
+}
+
+interrupt() {
+ failures+=("($(basename $0) interrupted)")
+ exit_cleanly
+}
+trap interrupt INT
+
sanity_checks
echo "WORKSPACE=$WORKSPACE"
@@ -530,21 +562,6 @@ install_workbench() {
}
do_install apps/workbench workbench
-start_api() {
- echo 'Starting API server...'
- cd "$WORKSPACE" \
- && eval $(python sdk/python/tests/run_test_server.py start --auth admin) \
- && export ARVADOS_TEST_API_HOST="$ARVADOS_API_HOST" \
- && export ARVADOS_TEST_API_INSTALLED="$$" \
- && (env | egrep ^ARVADOS)
-}
-
-stop_api() {
- unset ARVADOS_TEST_API_HOST
- cd "$WORKSPACE" \
- && python sdk/python/tests/run_test_server.py stop
-}
-
test_doclinkchecker() {
(
set -e
@@ -608,12 +625,4 @@ test_workbench_profile() {
}
do_test apps/workbench_profile workbench_profile
-rotate_logfile "$WORKSPACE/apps/workbench/log/" "test.log"
-
-stop_api
-rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
-
-report_outcomes
-clear_temp
-
-exit ${#failures}
+exit_cleanly
commit 2d03ecf4a147db64708991bd3bf2f1d9fc9b358e
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Feb 12 11:21:36 2015 -0500
Make tmp/cache during workbench install. No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index a0fefaa..689405b 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -525,6 +525,7 @@ done
install_workbench() {
cd "$WORKSPACE/apps/workbench" \
+ && mkdir -p tmp/cache \
&& RAILS_ENV=test bundle_install_trylocal
}
do_install apps/workbench workbench
commit e8492641dbf4051c456d355e536126740198892a
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Feb 12 11:21:06 2015 -0500
Default to TESTOPTS=-v for rake test.
(except workbench performance tests, because Minitest bug)
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 445509d..a0fefaa 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -560,7 +560,7 @@ stop_api
test_apiserver() {
cd "$WORKSPACE/services/api" \
- && RAILS_ENV=test bundle exec rake test ${testargs[services/api]}
+ && RAILS_ENV=test bundle exec rake test TESTOPTS=-v ${testargs[services/api]}
}
do_test services/api apiserver
@@ -568,14 +568,14 @@ start_api
test_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
- && bundle exec rake test ${testargs[sdk/ruby]}
+ && bundle exec rake test TESTOPTS=-v ${testargs[sdk/ruby]}
}
do_test sdk/ruby ruby_sdk
test_cli() {
cd "$WORKSPACE/sdk/cli" \
&& mkdir -p /tmp/keep \
- && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test ${testargs[sdk/cli]}
+ && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test TESTOPTS=-v ${testargs[sdk/cli]}
}
do_test sdk/cli cli
@@ -591,7 +591,7 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
- && RAILS_ENV=test bundle exec rake test ${testargs[apps/workbench]}
+ && RAILS_ENV=test bundle exec rake test TESTOPTS=-v ${testargs[apps/workbench]}
}
do_test apps/workbench workbench
commit 6c63e6c0a80c1f77a9f96be9956aa2a31147a89e
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Feb 9 23:39:13 2015 -0500
Make sure tmp/cache exists before we run our performance test suite.
refs #4754
diff --git a/jenkins/run-performance-suite.sh b/jenkins/run-performance-suite.sh
index c594620..2944bda 100755
--- a/jenkins/run-performance-suite.sh
+++ b/jenkins/run-performance-suite.sh
@@ -67,6 +67,8 @@ if [[ ! -d tmp ]]; then
mkdir tmp
fi
+mkdir -p tmp/cache
+
RAILS_ENV=performance bundle exec rake test:benchmark
ECODE=$?
commit 41d0494a83f08584f7427f6a050517cb1a6dd70d
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Feb 9 23:36:56 2015 -0500
Run the performance tests the documented way. This is possible after
commit d65b683af52e072b3d179b6f32edfbf37e108011 has been merged in the
arvados repository.
refs #4754
diff --git a/jenkins/run-performance-suite.sh b/jenkins/run-performance-suite.sh
index 6ffeff8..c594620 100755
--- a/jenkins/run-performance-suite.sh
+++ b/jenkins/run-performance-suite.sh
@@ -67,7 +67,7 @@ if [[ ! -d tmp ]]; then
mkdir tmp
fi
-RAILS_ENV=performance bundle exec rake TEST=test/performance/browsing_test.rb
+RAILS_ENV=performance bundle exec rake test:benchmark
ECODE=$?
commit abcdf588c84675cb4f5ce528ee30e4f40cac99b0
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Feb 9 22:46:59 2015 -0500
Add a script to run our performance tests.
refs #4754
diff --git a/jenkins/run-performance-suite.sh b/jenkins/run-performance-suite.sh
new file mode 100755
index 0000000..6ffeff8
--- /dev/null
+++ b/jenkins/run-performance-suite.sh
@@ -0,0 +1,86 @@
+#!/bin/bash
+
+EXITCODE=0
+
+INSTANCE=$1
+REVISION=$2
+
+if [[ "$INSTANCE" == '' ]]; then
+ echo "Syntax: $0 <instance> [revision]"
+ exit 1
+fi
+
+if [[ "$REVISION" == '' ]]; then
+ # See if there's a configuration file with the revision?
+ CONFIG_PATH=/home/jenkins/configuration/$INSTANCE.arvadosapi.com-versions.conf
+ if [[ -f $CONFIG_PATH ]]; then
+ echo "Loading git revision from $CONFIG_PATH"
+ . $CONFIG_PATH
+ REVISION=$ARVADOS_GIT_REVISION
+ fi
+fi
+
+if [[ "$REVISION" != '' ]]; then
+ echo "Git revision is $REVISION"
+else
+ echo "No valid git revision found, proceeding with what is in place."
+fi
+
+# Sanity check
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo "WORKSPACE environment variable not set"
+ exit 1
+fi
+
+title () {
+ txt="********** $1 **********"
+ printf "\n%*s%s\n\n" $((($COLUMNS-${#txt})/2)) "" "$txt"
+}
+
+timer_reset() {
+ t0=$SECONDS
+}
+
+timer() {
+ echo -n "$(($SECONDS - $t0))s"
+}
+
+source /etc/profile.d/rvm.sh
+echo $WORKSPACE
+
+title "Starting performance test"
+timer_reset
+
+cd $WORKSPACE
+
+if [[ "$REVISION" != '' ]]; then
+ git checkout $REVISION
+fi
+
+cp -f /home/jenkins/diagnostics/arvados-workbench/$INSTANCE-application.yml $WORKSPACE/apps/workbench/config/application.yml
+
+cd $WORKSPACE/apps/workbench
+
+HOME="$GEMHOME" bundle install --no-deployment
+
+if [[ ! -d tmp ]]; then
+ mkdir tmp
+fi
+
+RAILS_ENV=performance bundle exec rake TEST=test/performance/browsing_test.rb
+
+ECODE=$?
+
+if [[ "$REVISION" != '' ]]; then
+ git checkout master
+fi
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! PERFORMANCE TESTS FAILED (`timer`) !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ exit $EXITCODE
+fi
+
+title "Performance tests complete (`timer`)"
+
+exit $EXITCODE
commit f3e822f9fe498c6c9ef5fc0602b98695349e6e5e
Author: Brett Smith <brettcsmith at brettcsmith.org>
Date: Mon Feb 9 17:38:02 2015 -0500
Revert 11339c91. Don't use virtualenv to build packages.
fpm and virtualenv don't play nicely together. Under virtualenv, fpm
ends up building Python packages that include the files under their
virtualenv paths. This is not what we want. Refs #5175.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index cdf1d17..1dae4bc 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -85,14 +85,6 @@ if [[ "$DEBUG" != 0 ]]; then
echo "Workspace is $WORKSPACE"
fi
-# We need a recent setuptools. Older ones replace + with - in our
-# version numbers, making them non-PEP-440-compliant and
-# non-uploadable.
-VENV=$(mktemp -d)
-echo "Using virtualenv $VENV"
-virtualenv "$VENV"
-. "$VENV/bin/activate"
-
version_from_git() {
# Generates a version number from the git log for the current working
# directory, and writes it to stdout.
@@ -485,8 +477,7 @@ else
fi
fi
-# clean up temporary GOPATH and VENV
+# clean up temporary GOPATH
rm -rf "$GOPATH"
-rm -rf "$VENV"
exit $EXITCODE
commit 11339c91b5e7efdc214897856b103d0c532df45e
Author: Tom Clegg <tom at curoverse.com>
Date: Sat Feb 7 04:56:15 2015 -0500
Use virtualenv to get a newer setuptools that doesn't munge Python versions. No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 1dae4bc..cdf1d17 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -85,6 +85,14 @@ if [[ "$DEBUG" != 0 ]]; then
echo "Workspace is $WORKSPACE"
fi
+# We need a recent setuptools. Older ones replace + with - in our
+# version numbers, making them non-PEP-440-compliant and
+# non-uploadable.
+VENV=$(mktemp -d)
+echo "Using virtualenv $VENV"
+virtualenv "$VENV"
+. "$VENV/bin/activate"
+
version_from_git() {
# Generates a version number from the git log for the current working
# directory, and writes it to stdout.
@@ -477,7 +485,8 @@ else
fi
fi
-# clean up temporary GOPATH
+# clean up temporary GOPATH and VENV
rm -rf "$GOPATH"
+rm -rf "$VENV"
exit $EXITCODE
commit d7f1a63c8396715e501be04addefb0a158303e69
Merge: 6772d21 f6c53e0
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Feb 6 17:58:35 2015 -0500
Merge branch '3021-leave-api-running' refs #3021
commit 6772d21bda502b479d29f21030816560f110c04d
Author: Brett Smith <brett at curoverse.com>
Date: Fri Feb 6 11:54:12 2015 -0500
3021: Remove obsolete accommodation for buggy python-daemon.
This bug has been fixed in newer versions. Refs #3021.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index c5e5656..5425412 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -336,10 +336,6 @@ fi
# Needed for run_test_server.py which is used by certain (non-Python) tests.
pip install PyYAML || fatal "pip install PyYAML failed"
-# Needed for python-daemon 2.0.2, which breaks otherwise with
-# "ImportError: No module named docutils.core"
-pip install docutils || fatal "pip install docutils failed"
-
checkexit() {
if [[ "$?" != "0" ]]; then
title "!!!!!! $1 FAILED !!!!!!"
commit f6c53e0e14c29950c6b29f2b3c6a46c13071fbd2
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Feb 5 16:58:33 2015 -0500
3021: Print reassuring messages about expected warning messages during "bundle install --local".
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 07f5cdd..e9ed493 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -413,11 +413,21 @@ title () {
printf "\n%*s%s\n\n" $((($COLUMNS-${#txt})/2)) "" "$txt"
}
+bundle_install_trylocal() {
+ (
+ set -e
+ echo "(Running bundle install --local. 'could not find package' messages are OK.)"
+ if ! bundle install --local --no-deployment; then
+ echo "(Running bundle install again, without --local.)"
+ bundle install --no-deployment
+ fi
+ bundle package --all
+ )
+}
+
install_doc() {
cd "$WORKSPACE/doc" \
- && (bundle install --local --no-deployment \
- || bundle install --no-deployment) \
- && bundle package --all \
+ && bundle_install_trylocal \
&& rm -rf .site
}
do_install doc
@@ -425,9 +435,7 @@ do_install doc
install_ruby_sdk() {
with_test_gemset gem uninstall --force --all --executables arvados \
&& cd "$WORKSPACE/sdk/ruby" \
- && (bundle install --local --no-deployment \
- || bundle install --no-deployment) \
- && bundle package --all \
+ && bundle_install_trylocal \
&& gem build arvados.gemspec \
&& with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
@@ -436,9 +444,7 @@ do_install sdk/ruby ruby_sdk
install_cli() {
with_test_gemset gem uninstall --force --all --executables arvados-cli \
&& cd "$WORKSPACE/sdk/cli" \
- && (bundle install --local --no-deployment \
- || bundle install --no-deployment) \
- && bundle package --all \
+ && bundle_install_trylocal \
&& gem build arvados-cli.gemspec \
&& with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
@@ -462,9 +468,7 @@ done
install_apiserver() {
cd "$WORKSPACE/services/api" \
- && (RAILS_ENV=test bundle install --local --no-deployment \
- || RAILS_ENV=test bundle install --no-deployment) \
- && bundle package --all
+ && RAILS_ENV=test bundle_install_trylocal
rm -f config/environments/test.rb
cp config/environments/test.rb.example config/environments/test.rb
@@ -525,9 +529,7 @@ done
install_workbench() {
cd "$WORKSPACE/apps/workbench" \
- && (RAILS_ENV=test bundle install --local --no-deployment \
- || RAILS_ENV=test bundle install --no-deployment) \
- && bundle package --all
+ && RAILS_ENV=test bundle_install_trylocal
}
do_install apps/workbench workbench
commit 501a707f36ac0b2ac13dc074020641b58128e175
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Feb 3 15:49:49 2015 -0500
* Only source /etc/profile.d/rvm.sh if it exists and is a regular file
* Fix typo
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 7ec33e6..1dae4bc 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -208,7 +208,9 @@ build_and_scp_deb () {
fi
}
-source /etc/profile.d/rvm.sh
+if [[ -f /etc/profile.d/rvm.sh ]]; then
+ source /etc/profile.d/rvm.sh
+fi
# Make all files world-readable -- jenkins runs with umask 027, and has checked
# out our git tree here
@@ -409,7 +411,7 @@ build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curover
# prefix from only one of the dependencies of a package... Maybe I could
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
-cd $WORKSPACE/debs2
+cd $WORKSPACE/debs
# Python version numbering is obscure. Strip dashes and replace them with dots
# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
commit a526319c2a822c1a2b06b09d734fa3da6f84ec31
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Feb 2 10:27:01 2015 -0500
3021: Fix up some error checking.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 98734ca..07f5cdd 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -533,17 +533,17 @@ do_install apps/workbench workbench
start_api() {
echo 'Starting API server...'
- cd "$WORKSPACE" && \
- eval $(python sdk/python/tests/run_test_server.py start --auth admin) && \
- export ARVADOS_TEST_API_HOST="$ARVADOS_API_HOST" && \
- export ARVADOS_TEST_API_INSTALLED="$$" && \
- (env | egrep ^ARVADOS)
+ cd "$WORKSPACE" \
+ && eval $(python sdk/python/tests/run_test_server.py start --auth admin) \
+ && export ARVADOS_TEST_API_HOST="$ARVADOS_API_HOST" \
+ && export ARVADOS_TEST_API_INSTALLED="$$" \
+ && (env | egrep ^ARVADOS)
}
stop_api() {
unset ARVADOS_TEST_API_HOST
- cd "$WORKSPACE" && \
- python sdk/python/tests/run_test_server.py stop
+ cd "$WORKSPACE" \
+ && python sdk/python/tests/run_test_server.py stop
}
test_doclinkchecker() {
@@ -561,8 +561,8 @@ do_test doc doclinkchecker
stop_api
test_apiserver() {
- cd "$WORKSPACE/services/api"
- RAILS_ENV=test bundle exec rake test ${testargs[services/api]}
+ cd "$WORKSPACE/services/api" \
+ && RAILS_ENV=test bundle exec rake test ${testargs[services/api]}
}
do_test services/api apiserver
commit e925baa047b64d387c539473746b495972310813
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Feb 2 05:29:38 2015 -0500
3021: Cache packages with bundle package --all
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index febe276..98734ca 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -417,6 +417,7 @@ install_doc() {
cd "$WORKSPACE/doc" \
&& (bundle install --local --no-deployment \
|| bundle install --no-deployment) \
+ && bundle package --all \
&& rm -rf .site
}
do_install doc
@@ -426,6 +427,7 @@ install_ruby_sdk() {
&& cd "$WORKSPACE/sdk/ruby" \
&& (bundle install --local --no-deployment \
|| bundle install --no-deployment) \
+ && bundle package --all \
&& gem build arvados.gemspec \
&& with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
@@ -436,6 +438,7 @@ install_cli() {
&& cd "$WORKSPACE/sdk/cli" \
&& (bundle install --local --no-deployment \
|| bundle install --no-deployment) \
+ && bundle package --all \
&& gem build arvados-cli.gemspec \
&& with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
@@ -460,7 +463,8 @@ done
install_apiserver() {
cd "$WORKSPACE/services/api" \
&& (RAILS_ENV=test bundle install --local --no-deployment \
- || RAILS_ENV=test bundle install --no-deployment)
+ || RAILS_ENV=test bundle install --no-deployment) \
+ && bundle package --all
rm -f config/environments/test.rb
cp config/environments/test.rb.example config/environments/test.rb
@@ -522,7 +526,8 @@ done
install_workbench() {
cd "$WORKSPACE/apps/workbench" \
&& (RAILS_ENV=test bundle install --local --no-deployment \
- || RAILS_ENV=test bundle install --no-deployment)
+ || RAILS_ENV=test bundle install --no-deployment) \
+ && bundle package --all
}
do_install apps/workbench workbench
commit a6969ab968422f146df9cf9ef301fb5b0b422ded
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Feb 2 05:27:49 2015 -0500
3021: Let integration tests write to the api server log before rotating it.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 63c1d7c..febe276 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -561,8 +561,6 @@ test_apiserver() {
}
do_test services/api apiserver
-rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
-
start_api
test_ruby_sdk() {
@@ -609,6 +607,7 @@ do_test apps/workbench_profile workbench_profile
rotate_logfile "$WORKSPACE/apps/workbench/log/" "test.log"
stop_api
+rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
report_outcomes
clear_temp
commit 3e70317d25333921718191ac3ba9be91e0408449
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Feb 2 03:24:48 2015 -0500
3021: Try bundle install --local before resorting to downloading the internet.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 87441f0..63c1d7c 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -414,16 +414,18 @@ title () {
}
install_doc() {
- cd "$WORKSPACE/doc"
- bundle install --no-deployment
- rm -rf .site
+ cd "$WORKSPACE/doc" \
+ && (bundle install --local --no-deployment \
+ || bundle install --no-deployment) \
+ && rm -rf .site
}
do_install doc
install_ruby_sdk() {
with_test_gemset gem uninstall --force --all --executables arvados \
&& cd "$WORKSPACE/sdk/ruby" \
- && bundle install --no-deployment \
+ && (bundle install --local --no-deployment \
+ || bundle install --no-deployment) \
&& gem build arvados.gemspec \
&& with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
@@ -432,7 +434,8 @@ do_install sdk/ruby ruby_sdk
install_cli() {
with_test_gemset gem uninstall --force --all --executables arvados-cli \
&& cd "$WORKSPACE/sdk/cli" \
- && bundle install --no-deployment \
+ && (bundle install --local --no-deployment \
+ || bundle install --no-deployment) \
&& gem build arvados-cli.gemspec \
&& with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
@@ -455,8 +458,9 @@ do
done
install_apiserver() {
- cd "$WORKSPACE/services/api"
- RAILS_ENV=test bundle install --no-deployment
+ cd "$WORKSPACE/services/api" \
+ && (RAILS_ENV=test bundle install --local --no-deployment \
+ || RAILS_ENV=test bundle install --no-deployment)
rm -f config/environments/test.rb
cp config/environments/test.rb.example config/environments/test.rb
@@ -517,7 +521,8 @@ done
install_workbench() {
cd "$WORKSPACE/apps/workbench" \
- && RAILS_ENV=test bundle install --no-deployment
+ && (RAILS_ENV=test bundle install --local --no-deployment \
+ || RAILS_ENV=test bundle install --no-deployment)
}
do_install apps/workbench workbench
@@ -562,14 +567,12 @@ start_api
test_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
- && bundle install --no-deployment \
&& bundle exec rake test ${testargs[sdk/ruby]}
}
do_test sdk/ruby ruby_sdk
test_cli() {
cd "$WORKSPACE/sdk/cli" \
- && bundle install --no-deployment \
&& mkdir -p /tmp/keep \
&& KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test ${testargs[sdk/cli]}
}
commit 440a06656f8c81cf3d7757dd00991872683fc513
Author: Tom Clegg <tom at curoverse.com>
Date: Sun Feb 1 21:15:20 2015 -0500
3021: Leave an API server running while test suites run.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index c5e5656..87441f0 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -521,15 +521,45 @@ install_workbench() {
}
do_install apps/workbench workbench
+start_api() {
+ echo 'Starting API server...'
+ cd "$WORKSPACE" && \
+ eval $(python sdk/python/tests/run_test_server.py start --auth admin) && \
+ export ARVADOS_TEST_API_HOST="$ARVADOS_API_HOST" && \
+ export ARVADOS_TEST_API_INSTALLED="$$" && \
+ (env | egrep ^ARVADOS)
+}
+
+stop_api() {
+ unset ARVADOS_TEST_API_HOST
+ cd "$WORKSPACE" && \
+ python sdk/python/tests/run_test_server.py stop
+}
+
test_doclinkchecker() {
- cd "$WORKSPACE/doc"
- # Make sure python-epydoc is installed or the next line won't do much good!
- ARVADOS_API_HOST=qr1hi.arvadosapi.com
- PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
- unset ARVADOS_API_HOST
+ (
+ set -e
+ cd "$WORKSPACE/doc"
+ ARVADOS_API_HOST=qr1hi.arvadosapi.com
+ # Make sure python-epydoc is installed or the next line won't
+ # do much good!
+ PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
+ )
}
do_test doc doclinkchecker
+stop_api
+
+test_apiserver() {
+ cd "$WORKSPACE/services/api"
+ RAILS_ENV=test bundle exec rake test ${testargs[services/api]}
+}
+do_test services/api apiserver
+
+rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
+
+start_api
+
test_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
&& bundle install --no-deployment \
@@ -545,14 +575,6 @@ test_cli() {
}
do_test sdk/cli cli
-test_apiserver() {
- cd "$WORKSPACE/services/api"
- RAILS_ENV=test bundle exec rake test ${testargs[services/api]}
-}
-do_test services/api apiserver
-
-rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
-
for p in "${pythonstuff[@]}"
do
do_test "$p" pip
@@ -583,6 +605,8 @@ do_test apps/workbench_profile workbench_profile
rotate_logfile "$WORKSPACE/apps/workbench/log/" "test.log"
+stop_api
+
report_outcomes
clear_temp
commit 4af2bd565b75a9dcbb553975599d5e0b8856e940
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Jan 17 13:43:58 2015 -0500
Fix filename when compressing old test logs.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index a88d1bc..c5e5656 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -173,7 +173,7 @@ rotate_logfile() {
if [[ -f "$1/$2" ]]; then
THEDATE=`date +%Y%m%d%H%M%S`
mv "$1/$2" "$1/$THEDATE-$BUILD_NUMBER-$2"
- gzip "$1/$THEDATE-$2"
+ gzip "$1/$THEDATE-$BUILD_NUMBER-$2"
fi
}
commit 253673e89ef17f5f19b5c3b7cbe32d8348e1ba4f
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Jan 17 10:25:50 2015 -0500
Automatically rotate and compress the test.log files for the api server
and workbench.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index e271536..a88d1bc 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -168,6 +168,15 @@ sanity_checks() {
}
+rotate_logfile() {
+ # $BUILD_NUMBER is set by Jenkins if this script is being called as part of a Jenkins run
+ if [[ -f "$1/$2" ]]; then
+ THEDATE=`date +%Y%m%d%H%M%S`
+ mv "$1/$2" "$1/$THEDATE-$BUILD_NUMBER-$2"
+ gzip "$1/$THEDATE-$2"
+ fi
+}
+
declare -a failures
declare -A skip
declare -A testargs
@@ -542,6 +551,8 @@ test_apiserver() {
}
do_test services/api apiserver
+rotate_logfile "$WORKSPACE/services/api/log/" "test.log"
+
for p in "${pythonstuff[@]}"
do
do_test "$p" pip
@@ -570,6 +581,8 @@ test_workbench_profile() {
}
do_test apps/workbench_profile workbench_profile
+rotate_logfile "$WORKSPACE/apps/workbench/log/" "test.log"
+
report_outcomes
clear_temp
commit 20354a24f9582cc0abb268d45adacf5f386a10ed
Merge: 0f143ed 21bcc1b
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Jan 13 17:10:18 2015 -0500
Merge branch '3021-test-perf' refs #3021
commit 21bcc1b7eb165fb094db43f60b8073250074c3c1
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Jan 13 17:08:40 2015 -0500
3021: Install docutils (unlisted dependency of python-daemon).
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 6f425f8..e271536 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -327,6 +327,10 @@ fi
# Needed for run_test_server.py which is used by certain (non-Python) tests.
pip install PyYAML || fatal "pip install PyYAML failed"
+# Needed for python-daemon 2.0.2, which breaks otherwise with
+# "ImportError: No module named docutils.core"
+pip install docutils || fatal "pip install docutils failed"
+
checkexit() {
if [[ "$?" != "0" ]]; then
title "!!!!!! $1 FAILED !!!!!!"
commit 960bdb270fb93118e9241ffc28648962f136a37f
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Jan 13 16:43:58 2015 -0500
3021: Do not install *.egg after running tests: "pip install PyYAML" does what we need.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 146e7f1..6f425f8 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -354,16 +354,8 @@ do_test() {
go test ${testargs[$1]} "git.curoverse.com/arvados.git/$1"
elif [[ "$2" == "pip" ]]
then
- # Other test suites can depend on tests_require
- # dependencies of this package. For example, keepproxy runs
- # run_test_server.py, which depends on the yaml package,
- # which is in sdk/python's tests_require but not
- # install_requires, and therefore does not get installed by
- # setuptools until we run "setup.py test" *and* install the
- # .egg files that setup.py downloads.
cd "$WORKSPACE/$1" \
- && python setup.py test ${testargs[$1]} \
- && (easy_install *.egg || true)
+ && python setup.py test ${testargs[$1]}
elif [[ "$2" != "" ]]
then
"test_$2"
@@ -546,8 +538,6 @@ test_apiserver() {
}
do_test services/api apiserver
-# We must test sdk/python before testing services/keepproxy, because
-# keepproxy depends on sdk/python's test dependencies.
for p in "${pythonstuff[@]}"
do
do_test "$p" pip
commit e1fc53f3f8db8a6770dce3f11231191734217211
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Jan 13 13:08:48 2015 -0500
3021: Check for errors when installing PyYAML.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 9c86615..146e7f1 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -325,7 +325,7 @@ then
fi
# Needed for run_test_server.py which is used by certain (non-Python) tests.
-pip install PyYAML
+pip install PyYAML || fatal "pip install PyYAML failed"
checkexit() {
if [[ "$?" != "0" ]]; then
commit 82295301ecb7223887e4860ded71ee6373d3bb94
Author: Tom Clegg <tom at curoverse.com>
Date: Sun Jan 11 20:09:36 2015 -0500
3021: Allow '--only X' to override '--skip X'.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index e561816..9c86615 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -187,7 +187,7 @@ do
skip[$skipwhat]=1
;;
--only)
- only="$1"; shift
+ only="$1"; skip[$1]=""; shift
;;
--skip-install)
skip_install=1
commit 04a389ca22184133efe626dc83d8e8951c802551
Author: Tom Clegg <tom at curoverse.com>
Date: Sun Jan 11 20:08:46 2015 -0500
3021: Add workbench_profile target. Rename _performance to _benchmark.
These names match the rake tasks, test:benchmark and test:profile.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 14a0a69..e561816 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -47,7 +47,8 @@ https://arvados.org/projects/arvados/wiki/Running_tests
Available tests:
apps/workbench
-apps/workbench_performance
+apps/workbench_benchmark
+apps/workbench_profile
doc
services/api
services/crunchstat
@@ -170,6 +171,7 @@ sanity_checks() {
declare -a failures
declare -A skip
declare -A testargs
+skip[apps/workbench_profile]=1
while [[ -n "$1" ]]
do
@@ -562,11 +564,17 @@ test_workbench() {
}
do_test apps/workbench workbench
-test_workbench_performance() {
+test_workbench_benchmark() {
cd "$WORKSPACE/apps/workbench" \
- && RAILS_ENV=test bundle exec rake test:benchmark
+ && RAILS_ENV=test bundle exec rake test:benchmark ${testargs[apps/workbench_benchmark]}
}
-do_test apps/workbench_performance workbench_performance
+do_test apps/workbench_benchmark workbench_benchmark
+
+test_workbench_profile() {
+ cd "$WORKSPACE/apps/workbench" \
+ && RAILS_ENV=test bundle exec rake test:profile ${testargs[apps/workbench_profile]}
+}
+do_test apps/workbench_profile workbench_profile
report_outcomes
clear_temp
commit 5fb964860b2cacdf87a18b5295cd0c492aace3f6
Author: Tom Clegg <tom at curoverse.com>
Date: Sun Jan 11 20:04:02 2015 -0500
3021: Load test fixtures during api install.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index b044ab9..14a0a69 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -489,8 +489,8 @@ install_apiserver() {
cd "$WORKSPACE/services/api" \
&& RAILS_ENV=test bundle exec rake db:drop \
- && RAILS_ENV=test bundle exec rake db:create \
- && RAILS_ENV=test bundle exec rake db:setup
+ && RAILS_ENV=test bundle exec rake db:setup \
+ && RAILS_ENV=test bundle exec rake db:fixtures:load
}
do_install services/api apiserver
commit 0f143ed9e47ec9d07c0b17758c3bc466e631366a
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jan 9 16:43:52 2015 -0500
Python version numbering is obscure. For our debianized Python packages,
strip dashes and replace them with dots to match our other version
numbers. Cf. commit 4afcb8c in the arvados repository, compliance with
PEP-440.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index a359abb..7ec33e6 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -409,18 +409,24 @@ build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curover
# prefix from only one of the dependencies of a package... Maybe I could
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
-cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
+cd $WORKSPACE/debs2
+# Python version numbering is obscure. Strip dashes and replace them with dots
+# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
+build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
# The FUSE driver
# Please seem comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
+# Python version numbering is obscure. Strip dashes and replace them with dots
+# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
+build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
# The node manager
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
+# Python version numbering is obscure. Strip dashes and replace them with dots
+# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
+build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
@@ -456,8 +462,9 @@ CWLTOOL_VERSION=`git log --first-parent --max-count=1 --format='format:0.1.%ct.%
# Build cwltool package
cd $WORKSPACE/debs
-
-build_and_scp_deb $WORKSPACE/common-workflow-language/reference cwltool 'Common Workflow Language Working Group' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/common-workflow-language/reference/cwltool.egg-info/PKG-INFO)"
+# Python version numbering is obscure. Strip dashes and replace them with dots
+# to match our other version numbers. Cf. commit 4afcb8c, compliance with PEP-440.
+build_and_scp_deb $WORKSPACE/common-workflow-language/reference cwltool 'Common Workflow Language Working Group' 'python' "$(awk '($1 == "Version:"){ gsub(/-/,".",$2); print $2 }' $WORKSPACE/common-workflow-language/reference/cwltool.egg-info/PKG-INFO)"
# Finally, publish the packages, if necessary
if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
commit 0c5278fc1b23cae26dfea01be30a759b4413f35d
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Jan 6 13:45:35 2015 -0500
Rename {workbench,apiserver}_test to {apps/workbench,services/api}_test,
to match help message and become more consistent with overall naming
scheme.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 93d1c2b..b044ab9 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -540,7 +540,7 @@ do_test sdk/cli cli
test_apiserver() {
cd "$WORKSPACE/services/api"
- RAILS_ENV=test bundle exec rake test ${testargs[apiserver]}
+ RAILS_ENV=test bundle exec rake test ${testargs[services/api]}
}
do_test services/api apiserver
@@ -558,7 +558,7 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
- && RAILS_ENV=test bundle exec rake test ${testargs[workbench]}
+ && RAILS_ENV=test bundle exec rake test ${testargs[apps/workbench]}
}
do_test apps/workbench workbench
commit d5774ee9d1d5ff2cc1b8fc7edcc9aeac704e7d39
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Jan 6 12:46:16 2015 -0500
Make sure to build a new package for keepproxy if the go SDK has been updated.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 105f74d..a359abb 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -94,6 +94,15 @@ version_from_git() {
echo "0.1.$(date -ud "@$git_ts" +%Y%m%d%H%M%S).$git_hash"
}
+timestamp_from_git() {
+ # Generates a version number from the git log for the current working
+ # directory, and writes it to stdout.
+ local git_ts git_hash
+ declare $(TZ=UTC git log -n1 --first-parent --max-count=1 \
+ --format=format:"git_ts=%ct git_hash=%h" .)
+ echo "$git_ts"
+}
+
handle_python_package () {
# This function assumes the current working directory is the python package directory
if [[ "$UPLOAD" != 0 ]]; then
@@ -368,8 +377,20 @@ cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
# keepproxy
+cd "$GOPATH/src/git.curoverse.com/arvados.git/sdk/go"
+GO_SDK_VERSION=$(version_from_git)
+GO_SDK_TIMESTAMP=$(timestamp_from_git)
+
cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keepproxy"
-PKG_VERSION=$(version_from_git)
+KEEPPROXY_VERSION=$(version_from_git)
+KEEPPROXY_TIMESTAMP=$(timestamp_from_git)
+
+if [[ "$GO_SDK_TIMESTAMP" -gt "$KEEPPROXY_TIMESTAMP" ]]; then
+ PKG_VERSION=$GO_SDK_VERSION
+else
+ PKG_VERSION=$KEEPPROXY_VERSION
+fi
+
go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
commit 80f4c0641ca89efa0813feaa3429de9b471dd816
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Dec 30 10:04:46 2014 -0500
Add some more sanity checks in run-tests.sh, and refactor that part of the script
a little.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 23e89c0..93d1c2b 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -124,6 +124,49 @@ report_outcomes() {
done
fi
}
+
+sanity_checks() {
+ # Make sure WORKSPACE is set
+ if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
+ fi
+
+ # Make sure virtualenv is installed
+ `virtualenv --help >/dev/null 2>&1`
+
+ if [[ "$?" != "0" ]]; then
+ echo >&2
+ echo >&2 "Error: virtualenv could not be found"
+ echo >&2
+ exit 1
+ fi
+
+ # Make sure go is installed
+ `go env >/dev/null 2>&1`
+
+ if [[ "$?" != "0" ]]; then
+ echo >&2
+ echo >&2 "Error: go could not be found"
+ echo >&2
+ exit 1
+ fi
+
+ # Make sure gcc is installed
+ `gcc --help >/dev/null 2>&1`
+
+ if [[ "$?" != "0" ]]; then
+ echo >&2
+ echo >&2 "Error: gcc could not be found"
+ echo >&2
+ exit 1
+ fi
+
+}
+
declare -a failures
declare -A skip
declare -A testargs
@@ -167,14 +210,7 @@ do
esac
done
-# Sanity check
-if ! [[ -n "$WORKSPACE" ]]; then
- echo >&2 "$helpmessage"
- echo >&2
- echo >&2 "Error: WORKSPACE environment variable not set"
- echo >&2
- exit 1
-fi
+sanity_checks
echo "WORKSPACE=$WORKSPACE"
commit c05dc4cdc80d754a284e93478e643fd38dec123f
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Dec 29 16:37:55 2014 -0500
Fix go test args. No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index e437a21..23e89c0 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -313,7 +313,7 @@ do_test() {
timer_reset
if [[ "$2" == "go" ]]
then
- go test "${testargs[$1]}" "git.curoverse.com/arvados.git/$1"
+ go test ${testargs[$1]} "git.curoverse.com/arvados.git/$1"
elif [[ "$2" == "pip" ]]
then
# Other test suites can depend on tests_require
commit 7efe1eb66d31585bd6cd4a52ef0d12e0bdd91a88
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Dec 29 16:01:11 2014 -0500
Set RAILS_ENV consistently. Support "sevices/keepstore_test=-v" etc. No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 05cc65d..e437a21 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -313,7 +313,7 @@ do_test() {
timer_reset
if [[ "$2" == "go" ]]
then
- go test "git.curoverse.com/arvados.git/$1"
+ go test "${testargs[$1]}" "git.curoverse.com/arvados.git/$1"
elif [[ "$2" == "pip" ]]
then
# Other test suites can depend on tests_require
@@ -413,8 +413,7 @@ done
install_apiserver() {
cd "$WORKSPACE/services/api"
- export RAILS_ENV=test
- bundle install --no-deployment
+ RAILS_ENV=test bundle install --no-deployment
rm -f config/environments/test.rb
cp config/environments/test.rb.example config/environments/test.rb
@@ -453,9 +452,9 @@ install_apiserver() {
psql arvados_test -c "SELECT pg_terminate_backend (pg_stat_activity.procpid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'arvados_test';" 2>/dev/null
cd "$WORKSPACE/services/api" \
- && bundle exec rake db:drop \
- && bundle exec rake db:create \
- && bundle exec rake db:setup
+ && RAILS_ENV=test bundle exec rake db:drop \
+ && RAILS_ENV=test bundle exec rake db:create \
+ && RAILS_ENV=test bundle exec rake db:setup
}
do_install services/api apiserver
@@ -505,7 +504,7 @@ do_test sdk/cli cli
test_apiserver() {
cd "$WORKSPACE/services/api"
- bundle exec rake test ${testargs[apiserver]}
+ RAILS_ENV=test bundle exec rake test ${testargs[apiserver]}
}
do_test services/api apiserver
@@ -523,13 +522,13 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
- && bundle exec rake test ${testargs[workbench]}
+ && RAILS_ENV=test bundle exec rake test ${testargs[workbench]}
}
do_test apps/workbench workbench
test_workbench_performance() {
cd "$WORKSPACE/apps/workbench" \
- && bundle exec rake test:benchmark
+ && RAILS_ENV=test bundle exec rake test:benchmark
}
do_test apps/workbench_performance workbench_performance
commit 152e3b9509574666dc8e27d750b8bbe2d79b88a6
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Dec 23 15:22:48 2014 -0500
Remove superfluous output while running sdk/cli tests through run-tests.sh
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 5d751d0..05cc65d 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -496,7 +496,6 @@ test_ruby_sdk() {
do_test sdk/ruby ruby_sdk
test_cli() {
- title "Starting SDK CLI tests"
cd "$WORKSPACE/sdk/cli" \
&& bundle install --no-deployment \
&& mkdir -p /tmp/keep \
commit 48c6d3e3a7f10c6351a1cbc8fab1ea06cb616348
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Dec 23 14:55:16 2014 -0500
Minor run-tests.sh refactoring:
* sanitize naming of tests
* list available tests when --help is given
* no need to run rake generate during the doc install task, because the
linkchecker will redo that anyway
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 4b265a4..5d751d0 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -21,11 +21,11 @@ Options:
WORKSPACE=path Arvados source tree to test.
CONFIGSRC=path Dir with api server config files to copy into source tree.
(If none given, leave config files alone in source tree.)
-apiserver_test="TEST=test/functional/arvados/v1/collections_controller_test.rb"
+services/api_test="TEST=test/functional/arvados/v1/collections_controller_test.rb"
Restrict apiserver tests to the given file
sdk/python_test="--test-suite test.test_keep_locator"
Restrict Python SDK tests to the given class
-workbench_test="TEST=test/integration/pipeline_instances_test.rb"
+apps/workbench_test="TEST=test/integration/pipeline_instances_test.rb"
Restrict Workbench tests to the given file
ARVADOS_DEBUG=1
Print more debug messages
@@ -43,6 +43,25 @@ defaults to $HOME/arvados-api-server if that directory exists.
More information and background:
https://arvados.org/projects/arvados/wiki/Running_tests
+
+Available tests:
+
+apps/workbench
+apps/workbench_performance
+doc
+services/api
+services/crunchstat
+services/fuse
+services/keepproxy
+services/keepstore
+services/nodemanager
+sdk/cli
+sdk/python
+sdk/ruby
+sdk/go/arvadosclient
+sdk/go/keepclient
+sdk/go/streamer
+
EOF
# First make sure to remove any ARVADOS_ variables from the calling
@@ -307,6 +326,9 @@ do_test() {
cd "$WORKSPACE/$1" \
&& python setup.py test ${testargs[$1]} \
&& (easy_install *.egg || true)
+ elif [[ "$2" != "" ]]
+ then
+ "test_$2"
else
"test_$1"
fi
@@ -330,6 +352,9 @@ do_install() {
cd "$WORKSPACE/$1" \
&& python setup.py sdist rotate --keep=1 --match .tar.gz \
&& pip install --upgrade dist/*.tar.gz
+ elif [[ "$2" != "" ]]
+ then
+ "install_$2"
else
"install_$1"
fi
@@ -345,16 +370,12 @@ title () {
printf "\n%*s%s\n\n" $((($COLUMNS-${#txt})/2)) "" "$txt"
}
-install_docs() {
+install_doc() {
cd "$WORKSPACE/doc"
bundle install --no-deployment
rm -rf .site
- # Make sure python-epydoc is installed or the next line won't do much good!
- ARVADOS_API_HOST=qr1hi.arvadosapi.com
- PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
- unset ARVADOS_API_HOST
}
-do_install docs
+do_install doc
install_ruby_sdk() {
with_test_gemset gem uninstall --force --all --executables arvados \
@@ -363,7 +384,7 @@ install_ruby_sdk() {
&& gem build arvados.gemspec \
&& with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
-do_install ruby_sdk
+do_install sdk/ruby ruby_sdk
install_cli() {
with_test_gemset gem uninstall --force --all --executables arvados-cli \
@@ -372,7 +393,7 @@ install_cli() {
&& gem build arvados-cli.gemspec \
&& with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
-do_install cli
+do_install sdk/cli cli
# Install the Python SDK early. Various other test suites (like
# keepproxy) bring up run_test_server.py, which imports the arvados
@@ -436,7 +457,7 @@ install_apiserver() {
&& bundle exec rake db:create \
&& bundle exec rake db:setup
}
-do_install apiserver
+do_install services/api apiserver
declare -a gostuff
gostuff=(
@@ -456,20 +477,23 @@ install_workbench() {
cd "$WORKSPACE/apps/workbench" \
&& RAILS_ENV=test bundle install --no-deployment
}
-do_install workbench
+do_install apps/workbench workbench
test_doclinkchecker() {
cd "$WORKSPACE/doc"
- bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
+ # Make sure python-epydoc is installed or the next line won't do much good!
+ ARVADOS_API_HOST=qr1hi.arvadosapi.com
+ PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
+ unset ARVADOS_API_HOST
}
-do_test doclinkchecker
+do_test doc doclinkchecker
test_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
&& bundle install --no-deployment \
&& bundle exec rake test ${testargs[sdk/ruby]}
}
-do_test ruby_sdk
+do_test sdk/ruby ruby_sdk
test_cli() {
title "Starting SDK CLI tests"
@@ -478,13 +502,13 @@ test_cli() {
&& mkdir -p /tmp/keep \
&& KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test ${testargs[sdk/cli]}
}
-do_test cli
+do_test sdk/cli cli
test_apiserver() {
cd "$WORKSPACE/services/api"
bundle exec rake test ${testargs[apiserver]}
}
-do_test apiserver
+do_test services/api apiserver
# We must test sdk/python before testing services/keepproxy, because
# keepproxy depends on sdk/python's test dependencies.
@@ -502,13 +526,13 @@ test_workbench() {
cd "$WORKSPACE/apps/workbench" \
&& bundle exec rake test ${testargs[workbench]}
}
-do_test workbench
+do_test apps/workbench workbench
test_workbench_performance() {
cd "$WORKSPACE/apps/workbench" \
&& bundle exec rake test:benchmark
}
-do_test workbench_performance
+do_test apps/workbench_performance workbench_performance
report_outcomes
clear_temp
commit 49b655c14efe7bc5d73c099813db22fc09050d83
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Dec 16 11:39:40 2014 -0500
Use gem uninstall --force to skip dependency prompt. refs #4785
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 6682c0d..4b265a4 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -357,7 +357,7 @@ install_docs() {
do_install docs
install_ruby_sdk() {
- with_test_gemset gem uninstall --all --executables arvados \
+ with_test_gemset gem uninstall --force --all --executables arvados \
&& cd "$WORKSPACE/sdk/ruby" \
&& bundle install --no-deployment \
&& gem build arvados.gemspec \
@@ -366,7 +366,7 @@ install_ruby_sdk() {
do_install ruby_sdk
install_cli() {
- with_test_gemset gem uninstall --all --executables arvados-cli \
+ with_test_gemset gem uninstall --force --all --executables arvados-cli \
&& cd "$WORKSPACE/sdk/cli" \
&& bundle install --no-deployment \
&& gem build arvados-cli.gemspec \
commit fce78b35e3fa0534d4173762b979c916cc463cdd
Merge: 511d087 c479436
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Dec 15 15:54:20 2014 -0500
Merge branch '4156-bundle-install-user' refs #4156
commit c479436a110f977879ff278e9a550efe47f26e1d
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Dec 15 15:38:41 2014 -0500
4156: Uninstall old gems just before building new ones. Avoid an extra source of truth for the "gems we install" list.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index bf7448a..6682c0d 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -212,16 +212,7 @@ setup_ruby_environment() {
|| fatal 'rvm gemset setup'
rvm env
-
- # Remove previously installed versions of our own gems. This
- # ensures the test suites only have access to [a] published
- # gems and [b] the gems we build and install right now --
- # never unpublished gems left over from previous builds.
- gem uninstall --all --executables arvados arvados-cli \
- || fatal 'clean arvados gems'
else
- echo "RVM not found. Will install gems-under-test into \"$GEM_HOME\"."
-
# When our "bundle install"s need to install new gems to
# satisfy dependencies, we want them to go where "gem install
# --user-install" would put them. (However, if the caller has
@@ -366,7 +357,8 @@ install_docs() {
do_install docs
install_ruby_sdk() {
- cd "$WORKSPACE/sdk/ruby" \
+ with_test_gemset gem uninstall --all --executables arvados \
+ && cd "$WORKSPACE/sdk/ruby" \
&& bundle install --no-deployment \
&& gem build arvados.gemspec \
&& with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
@@ -374,7 +366,8 @@ install_ruby_sdk() {
do_install ruby_sdk
install_cli() {
- cd "$WORKSPACE/sdk/cli" \
+ with_test_gemset gem uninstall --all --executables arvados-cli \
+ && cd "$WORKSPACE/sdk/cli" \
&& bundle install --no-deployment \
&& gem build arvados-cli.gemspec \
&& with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
commit 7490e64759f56a8da52c8259da8b96baa769a6fe
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Dec 15 14:53:56 2014 -0500
4156: Update comments, remove excess cut, tweak GEM_HOME-reporting.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index ede7302..bf7448a 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -222,11 +222,11 @@ setup_ruby_environment() {
else
echo "RVM not found. Will install gems-under-test into \"$GEM_HOME\"."
- # When our "bundle install"s need to install new gems to satisfy
- # dependencies, we want them to go where "gem install --user-install"
- # would put them. If rvm is in use (or something else has set
- # GEM_HOME) we assume "bundle install" already does something
- # reasonable.
+ # When our "bundle install"s need to install new gems to
+ # satisfy dependencies, we want them to go where "gem install
+ # --user-install" would put them. (However, if the caller has
+ # already set GEM_HOME, we assume that's where dependencies
+ # should be installed, and we should leave it alone.)
if [ -z "$GEM_HOME" ]; then
user_gempath="$(gem env gempath)"
@@ -240,14 +240,15 @@ setup_ruby_environment() {
# happen to be installed in $user_gempath, system dirs, etc.
tmpdir_gem_home="$(env - PATH="$PATH" HOME="$GEMHOME" gem env gempath | cut -f1 -d:)"
- PATH="${tmpdir_gem_home%%:*}/bin:$PATH"
+ PATH="$tmpdir_gem_home/bin:$PATH"
export GEM_PATH="$tmpdir_gem_home:$(gem env gempath)"
- echo "Will install dependencies to $GEM_HOME"
+ echo "Will install dependencies to $(gem env gemdir)"
echo "Will install arvados gems to $tmpdir_gem_home"
echo "Gem search path is GEM_PATH=$GEM_PATH"
fi
}
+
with_test_gemset() {
if [[ "$using_rvm" == true ]]; then
"$@"
commit 2fb06b518ef7c6f402d7d8832577400a0814dab0
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Dec 15 14:43:20 2014 -0500
4156: Move "bundle install" into separate install_ task like apiserver. Remove one more GEMHOME after merge.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index f5fc1e1..ede7302 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -458,6 +458,12 @@ do
do_install "$g" go
done
+install_workbench() {
+ cd "$WORKSPACE/apps/workbench" \
+ && RAILS_ENV=test bundle install --no-deployment
+}
+do_install workbench
+
test_doclinkchecker() {
cd "$WORKSPACE/doc"
bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
@@ -500,15 +506,13 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
- && bundle install --no-deployment \
&& bundle exec rake test ${testargs[workbench]}
}
do_test workbench
test_workbench_performance() {
cd "$WORKSPACE/apps/workbench" \
- && HOME="$GEMHOME" bundle install --no-deployment \
- && HOME="$GEMHOME" bundle exec rake test:benchmark
+ && bundle exec rake test:benchmark
}
do_test workbench_performance
commit 061a0fa3af58c1c1db584e21cb069547288f7e82
Merge: 8bfa3e7 511d087
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Dec 15 14:28:06 2014 -0500
4156: Merge branch 'master' into 4156-bundle-install-user
commit 8bfa3e75783933f7a9e484b18074336456fbc834
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Dec 15 10:06:31 2014 -0500
4156: Become rvm-aware.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 9e5f47b..f1f5f8e 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -62,11 +62,6 @@ COLUMNS=80
leave_temp=
skip_install=
-if [[ -f /etc/profile.d/rvm.sh ]]
-then
- source /etc/profile.d/rvm.sh
-fi
-
declare -A leave_temp
clear_temp() {
leaving=""
@@ -183,36 +178,83 @@ do
fi
done
-# When our "bundle install"s need to install new gems to satisfy
-# dependencies, we want them to go where "gem install --user-install"
-# would put them. If rvm is in use (or something else has set
-# GEM_HOME) we assume "bundle install" already does something
-# reasonable.
+setup_ruby_environment() {
+ if [[ -s "$HOME/.rvm/scripts/rvm" ]] ; then
+ source "$HOME/.rvm/scripts/rvm"
+ using_rvm=true
+ elif [[ -s "/usr/local/rvm/scripts/rvm" ]] ; then
+ source "/usr/local/rvm/scripts/rvm"
+ using_rvm=true
+ else
+ using_rvm=false
+ fi
-if [ -z "$GEM_HOME" ]; then
- user_gempath="$(gem env gempath)"
- export GEM_HOME="${user_gempath%%:*}"
-fi
-PATH="$(gem env gemdir)/bin:$PATH"
+ if [[ "$using_rvm" == true ]]; then
+ # If rvm is in use, we can't just put separate "dependencies"
+ # and "gems-under-test" paths to GEM_PATH: passenger resets
+ # the environment to the "current gemset", which would lose
+ # our GEM_PATH and prevent our test suites from running ruby
+ # programs (for example, the Workbench test suite could not
+ # boot an API server or run arv). Instead, we have to make an
+ # rvm gemset and use it for everything.
+
+ [[ `type rvm | head -n1` == "rvm is a function" ]] \
+ || fatal 'rvm check'
+
+ # Put rvm's favorite path back in first place (overriding
+ # virtualenv, which just put itself there). Ignore rvm's
+ # complaint about not being in first place already.
+ rvm use @default 2>/dev/null
+
+ # Create (if needed) and switch to an @arvados-tests
+ # gemset. (Leave the choice of ruby to the caller.)
+ rvm use @arvados-tests --create \
+ || fatal 'rvm gemset setup'
+
+ rvm env
+
+ # Remove previously installed versions of our own gems. This
+ # ensures the test suites only have access to [a] published
+ # gems and [b] the gems we build and install right now --
+ # never unpublished gems left over from previous builds.
+ gem uninstall --all --executables arvados arvados-cli \
+ || fatal 'clean arvados gems'
+ else
+ echo "RVM not found. Will install gems-under-test into \"$GEM_HOME\"."
-# When we build and install our own gems, we install them in our
-# $GEMHOME tmpdir, and we want them to be at the front of GEM_PATH and
-# PATH so integration tests prefer them over other versions that
-# happen to be installed in $user_gempath, system dirs, etc.
+ # When our "bundle install"s need to install new gems to satisfy
+ # dependencies, we want them to go where "gem install --user-install"
+ # would put them. If rvm is in use (or something else has set
+ # GEM_HOME) we assume "bundle install" already does something
+ # reasonable.
-tmpdir_gem_home="$(env - PATH="$PATH" HOME="$GEMHOME" gem env gempath | cut -f1 -d:)"
-PATH="${tmpdir_gem_home%%:*}/bin:$PATH"
-export GEM_PATH="$tmpdir_gem_home:$(gem env gempath)"
+ if [ -z "$GEM_HOME" ]; then
+ user_gempath="$(gem env gempath)"
+ export GEM_HOME="${user_gempath%%:*}"
+ fi
+ PATH="$(gem env gemdir)/bin:$PATH"
-echo "PATH=$PATH"
-echo "Will install dependencies to $GEM_HOME"
-echo "Will install arvados gems to $tmpdir_gem_home"
-echo "Gem search path is GEM_PATH=$GEM_PATH"
+ # When we build and install our own gems, we install them in our
+ # $GEMHOME tmpdir, and we want them to be at the front of GEM_PATH and
+ # PATH so integration tests prefer them over other versions that
+ # happen to be installed in $user_gempath, system dirs, etc.
-if ! which bundler >/dev/null
-then
- gem install --user-install bundler || fatal 'Could not install bundler'
-fi
+ tmpdir_gem_home="$(env - PATH="$PATH" HOME="$GEMHOME" gem env gempath | cut -f1 -d:)"
+ PATH="${tmpdir_gem_home%%:*}/bin:$PATH"
+ export GEM_PATH="$tmpdir_gem_home:$(gem env gempath)"
+
+ echo "Will install dependencies to $GEM_HOME"
+ echo "Will install arvados gems to $tmpdir_gem_home"
+ echo "Gem search path is GEM_PATH=$GEM_PATH"
+ fi
+}
+with_test_gemset() {
+ if [[ "$using_rvm" == true ]]; then
+ "$@"
+ else
+ GEM_HOME="$tmpdir_gem_home" "$@"
+ fi
+}
export GOPATH
mkdir -p "$GOPATH/src/git.curoverse.com"
@@ -222,6 +264,17 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
. "$VENVDIR/bin/activate"
+# Note: this must be the last time we change PATH, otherwise rvm will
+# whine a lot.
+setup_ruby_environment
+
+echo "PATH is $PATH"
+
+if ! which bundler >/dev/null
+then
+ gem install --user-install bundler || fatal 'Could not install bundler'
+fi
+
# Needed for run_test_server.py which is used by certain (non-Python) tests.
pip install PyYAML
@@ -315,7 +368,7 @@ install_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
&& bundle install --no-deployment \
&& gem build arvados.gemspec \
- && GEM_HOME="$tmpdir_gem_home" gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
+ && with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
do_install ruby_sdk
@@ -323,7 +376,7 @@ install_cli() {
cd "$WORKSPACE/sdk/cli" \
&& bundle install --no-deployment \
&& gem build arvados-cli.gemspec \
- && GEM_HOME="$tmpdir_gem_home" gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
+ && with_test_gemset gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
do_install cli
commit 511d0877b038c3ce33f3fcfa72def70f7da84692
Merge: ebf8ab1 8c7f4b7
Author: Radhika Chippada <radhika at curoverse.com>
Date: Fri Dec 12 09:16:00 2014 -0500
refs #4754
Merge branch '4754-add-workbench-performance-test'
commit 72216082a3e042745707e69692e4b8664fc07a6e
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Dec 10 17:45:30 2014 -0500
4156: Do not use --user-install -- just install to the GEM_HOME we selected.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 6d8c1c3..9e5f47b 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -315,7 +315,7 @@ install_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
&& bundle install --no-deployment \
&& gem build arvados.gemspec \
- && GEM_HOME="$tmpdir_gem_home" gem install --user-install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
+ && GEM_HOME="$tmpdir_gem_home" gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
do_install ruby_sdk
@@ -323,7 +323,7 @@ install_cli() {
cd "$WORKSPACE/sdk/cli" \
&& bundle install --no-deployment \
&& gem build arvados-cli.gemspec \
- && GEM_HOME="$tmpdir_gem_home" gem install --user-install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
+ && GEM_HOME="$tmpdir_gem_home" gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
do_install cli
commit 7b41632216dcdc83aba5bc48b61189f705bd3e97
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Dec 10 16:59:54 2014 -0500
4156: Fix install destination for gem install --user-install. Make tmp-gem-home discovery work under rvm.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index b141187..6d8c1c3 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -185,25 +185,29 @@ done
# When our "bundle install"s need to install new gems to satisfy
# dependencies, we want them to go where "gem install --user-install"
-# would put them. This could be ~/.gem/..., or something rvm has set
-# up. We don't want to install them to our GEMHOME tmpdir, though:
-# that would mean re-downloading all dependencies each time we run
-# tests with clean tmpdirs. The first dir in `gem env gempath` seems
-# to give us this dir. (Note: this is a no-op if rvm is in use.)
-user_gempath="$(gem env gempath)"
-export GEM_HOME="${user_gempath%%:*}"
+# would put them. If rvm is in use (or something else has set
+# GEM_HOME) we assume "bundle install" already does something
+# reasonable.
+
+if [ -z "$GEM_HOME" ]; then
+ user_gempath="$(gem env gempath)"
+ export GEM_HOME="${user_gempath%%:*}"
+fi
PATH="$(gem env gemdir)/bin:$PATH"
-# Wherever "HOME=$GEMHOME gem install --user-install" installs stuff
-# to, we want its bin to be in our PATH. (Normally, this gempath is
-# inside our nice clean $GEMHOME, which means we can install the
-# current version there and expect integration tests to find it before
-# finding any other versions that happen to be installed
-# somewhere. But if rvm is enforcing its own idea where gempath should
-# go, we'll just assume rvm has been set up correctly rather than
-# fight it.)
-tmpdir_gempath="$(HOME="$GEMHOME" gem env gempath)"
-PATH="${tmpdir_gempath%%:*}/bin:$PATH"
+# When we build and install our own gems, we install them in our
+# $GEMHOME tmpdir, and we want them to be at the front of GEM_PATH and
+# PATH so integration tests prefer them over other versions that
+# happen to be installed in $user_gempath, system dirs, etc.
+
+tmpdir_gem_home="$(env - PATH="$PATH" HOME="$GEMHOME" gem env gempath | cut -f1 -d:)"
+PATH="${tmpdir_gem_home%%:*}/bin:$PATH"
+export GEM_PATH="$tmpdir_gem_home:$(gem env gempath)"
+
+echo "PATH=$PATH"
+echo "Will install dependencies to $GEM_HOME"
+echo "Will install arvados gems to $tmpdir_gem_home"
+echo "Gem search path is GEM_PATH=$GEM_PATH"
if ! which bundler >/dev/null
then
@@ -256,7 +260,7 @@ do_test() {
# setuptools until we run "setup.py test" *and* install the
# .egg files that setup.py downloads.
cd "$WORKSPACE/$1" \
- && HOME="$GEMHOME" python setup.py test ${testargs[$1]} \
+ && python setup.py test ${testargs[$1]} \
&& (easy_install *.egg || true)
else
"test_$1"
@@ -298,28 +302,28 @@ title () {
install_docs() {
cd "$WORKSPACE/doc"
- HOME="$GEMHOME" bundle install --no-deployment
+ bundle install --no-deployment
rm -rf .site
# Make sure python-epydoc is installed or the next line won't do much good!
ARVADOS_API_HOST=qr1hi.arvadosapi.com
- PYTHONPATH=$WORKSPACE/sdk/python/ HOME="$GEMHOME" bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
+ PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
unset ARVADOS_API_HOST
}
do_install docs
install_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
- && HOME="$GEMHOME" bundle install --no-deployment \
- && HOME="$GEMHOME" gem build arvados.gemspec \
- && HOME="$GEMHOME" gem install --user-install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
+ && bundle install --no-deployment \
+ && gem build arvados.gemspec \
+ && GEM_HOME="$tmpdir_gem_home" gem install --user-install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
do_install ruby_sdk
install_cli() {
cd "$WORKSPACE/sdk/cli" \
- && HOME="$GEMHOME" bundle install --no-deployment \
- && HOME="$GEMHOME" gem build arvados-cli.gemspec \
- && HOME="$GEMHOME" gem install --user-install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
+ && bundle install --no-deployment \
+ && gem build arvados-cli.gemspec \
+ && GEM_HOME="$tmpdir_gem_home" gem install --user-install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
do_install cli
@@ -342,7 +346,7 @@ done
install_apiserver() {
cd "$WORKSPACE/services/api"
export RAILS_ENV=test
- HOME="$GEMHOME" bundle install --no-deployment
+ bundle install --no-deployment
rm -f config/environments/test.rb
cp config/environments/test.rb.example config/environments/test.rb
@@ -381,9 +385,9 @@ install_apiserver() {
psql arvados_test -c "SELECT pg_terminate_backend (pg_stat_activity.procpid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'arvados_test';" 2>/dev/null
cd "$WORKSPACE/services/api" \
- && HOME="$GEMHOME" bundle exec rake db:drop \
- && HOME="$GEMHOME" bundle exec rake db:create \
- && HOME="$GEMHOME" bundle exec rake db:setup
+ && bundle exec rake db:drop \
+ && bundle exec rake db:create \
+ && bundle exec rake db:setup
}
do_install apiserver
@@ -403,29 +407,29 @@ done
test_doclinkchecker() {
cd "$WORKSPACE/doc"
- HOME="$GEMHOME" bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
+ bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
}
do_test doclinkchecker
test_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
- && HOME="$GEMHOME" bundle install --no-deployment \
- && HOME="$GEMHOME" bundle exec rake test ${testargs[sdk/ruby]}
+ && bundle install --no-deployment \
+ && bundle exec rake test ${testargs[sdk/ruby]}
}
do_test ruby_sdk
test_cli() {
title "Starting SDK CLI tests"
cd "$WORKSPACE/sdk/cli" \
- && HOME="$GEMHOME" bundle install --no-deployment \
+ && bundle install --no-deployment \
&& mkdir -p /tmp/keep \
- && KEEP_LOCAL_STORE=/tmp/keep HOME="$GEMHOME" bundle exec rake test ${testargs[sdk/cli]}
+ && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test ${testargs[sdk/cli]}
}
do_test cli
test_apiserver() {
cd "$WORKSPACE/services/api"
- HOME="$GEMHOME" bundle exec rake test ${testargs[apiserver]}
+ bundle exec rake test ${testargs[apiserver]}
}
do_test apiserver
@@ -443,8 +447,8 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
- && HOME="$GEMHOME" bundle install --no-deployment \
- && HOME="$GEMHOME" bundle exec rake test ${testargs[workbench]}
+ && bundle install --no-deployment \
+ && bundle exec rake test ${testargs[workbench]}
}
do_test workbench
commit 8c7f4b71ffe3976dca29a8337e3d487ad7547096
Author: Radhika Chippada <radhika at curoverse.com>
Date: Tue Dec 9 20:33:05 2014 -0500
4754: add test_workbench_performance
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 97cb27b..115db83 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -420,6 +420,13 @@ test_workbench() {
}
do_test workbench
+test_workbench_performance() {
+ cd "$WORKSPACE/apps/workbench" \
+ && HOME="$GEMHOME" bundle install --no-deployment \
+ && HOME="$GEMHOME" bundle exec rake test:benchmark
+}
+do_test workbench_performance
+
report_outcomes
clear_temp
commit f85eb7e10ef091aa9989d811705f04236bbd897a
Merge: e28c8a9 ebf8ab1
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Dec 5 12:59:44 2014 -0500
4156: Merge branch 'master' into 4156-bundle-install-user
commit ebf8ab12ebbbdd4b4b317b1b14f1e65d67e381c3
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Dec 3 10:40:10 2014 -0500
Add timing information to the docker tests.
No issue #
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index 0b5148e..e0c2288 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -24,6 +24,14 @@ docker_push () {
fi
}
+timer_reset() {
+ t0=$SECONDS
+}
+
+timer() {
+ echo -n "$(($SECONDS - $t0))s"
+}
+
# Sanity check
if ! [[ -n "$WORKSPACE" ]]; then
echo >&2
@@ -49,6 +57,8 @@ fi
# DOCKER
title "Starting docker build"
+timer_reset
+
# clean up the docker build environment
cd "$WORKSPACE"
cd docker
@@ -68,10 +78,12 @@ if [[ "$ECODE" != "0" ]]; then
EXITCODE=$(($EXITCODE + $ECODE))
fi
-title "docker build complete"
+title "docker build complete (`timer`)"
title "uploading images"
+timer_reset
+
if [[ "$ECODE" == "0" ]]; then
docker_push arvados/api
docker_push arvados/compute
@@ -83,10 +95,12 @@ else
title "upload arvados images SKIPPED because build failed"
fi
-title "upload arvados images complete"
+title "upload arvados images complete (`timer`)"
title "Starting docker java-bwa-samtools build"
+timer_reset
+
./build.sh java-bwa-samtools-image
ECODE=$?
@@ -96,16 +110,17 @@ if [[ "$ECODE" != "0" ]]; then
EXITCODE=$(($EXITCODE + $ECODE))
fi
-title "docker build java-bwa-samtools complete"
+title "docker build java-bwa-samtools complete (`timer`)"
title "upload arvados/jobs image"
+timer_reset
if [[ "$ECODE" == "0" ]]; then
docker_push arvados/jobs
else
title "upload arvados/jobs image SKIPPED because build failed"
fi
-title "upload arvados/jobs image complete"
+title "upload arvados/jobs image complete (`timer`)"
exit $EXITCODE
commit 4d4d004399af1dc0a99d67893f818756a8d19a90
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Mon Dec 1 15:46:35 2014 -0500
Install PyYAML into VENVDIR for non-Python tests that use run_test_server.py. no issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index ad8e9f6..97cb27b 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -191,6 +191,9 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
. "$VENVDIR/bin/activate"
+# Needed for run_test_server.py which is used by certain (non-Python) tests.
+pip install PyYAML
+
checkexit() {
if [[ "$?" != "0" ]]; then
title "!!!!!! $1 FAILED !!!!!!"
commit e28c8a92c6ca9b1c6a32d727cf5913a44229ec4c
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Nov 27 16:32:14 2014 -0500
4156: Install dependencies to user gempath, not system gempath, even
if not using rvm. Without this, run-tests asks for a sudo password
whenever "bundle install" has any work to do, which is especially
disruptive when two tested components have differing versions of the
arvados gem listed in their Gemfile.lock.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index ad8e9f6..a42440d 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -182,7 +182,34 @@ do
eval $tmpdir=$(mktemp -d)
fi
done
-PATH="$GEMHOME/.gem/ruby/2.1.0/bin:$PATH"
+
+# When our "bundle install"s need to install new gems to satisfy
+# dependencies, we want them to go where "gem install --user-install"
+# would put them. This could be ~/.gem/..., or something rvm has set
+# up. We don't want to install them to our GEMHOME tmpdir, though:
+# that would mean re-downloading all dependencies each time we run
+# tests with clean tmpdirs. The first dir in `gem env gempath` seems
+# to give us this dir. (Note: this is a no-op if rvm is in use.)
+user_gempath="$(gem env gempath)"
+export GEM_HOME="${user_gempath%%:*}"
+PATH="$(gem env gemdir)/bin:$PATH"
+
+# Wherever "HOME=$GEMHOME gem install --user-install" installs stuff
+# to, we want its bin to be in our PATH. (Normally, this gempath is
+# inside our nice clean $GEMHOME, which means we can install the
+# current version there and expect integration tests to find it before
+# finding any other versions that happen to be installed
+# somewhere. But if rvm is enforcing its own idea where gempath should
+# go, we'll just assume rvm has been set up correctly rather than
+# fight it.)
+tmpdir_gempath="$(HOME="$GEMHOME" gem env gempath)"
+PATH="${tmpdir_gempath%%:*}/bin:$PATH"
+
+if ! which bundler >/dev/null
+then
+ gem install --user-install bundler || fatal 'Could not install bundler'
+fi
+
export GOPATH
mkdir -p "$GOPATH/src/git.curoverse.com"
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
@@ -280,14 +307,15 @@ do_install docs
install_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
&& HOME="$GEMHOME" bundle install --no-deployment \
- && gem build arvados.gemspec \
+ && HOME="$GEMHOME" gem build arvados.gemspec \
&& HOME="$GEMHOME" gem install --user-install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
do_install ruby_sdk
install_cli() {
cd "$WORKSPACE/sdk/cli" \
- && gem build arvados-cli.gemspec \
+ && HOME="$GEMHOME" bundle install --no-deployment \
+ && HOME="$GEMHOME" gem build arvados-cli.gemspec \
&& HOME="$GEMHOME" gem install --user-install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
do_install cli
commit d0e6362a7388d6e17d43f92ec64d13d5ee206894
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Nov 26 14:23:03 2014 -0500
Pull the common-workflow-language code from the new
common-workflow-language github organization.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 9b567d4..105f74d 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -414,9 +414,9 @@ if [[ ! -d "$WORKSPACE/common-workflow-language" ]]; then
mkdir "$WORKSPACE/common-workflow-language"
cd "$WORKSPACE"
if [[ "$DEBUG" != 0 ]]; then
- git clone https://github.com/rabix/common-workflow-language.git common-workflow-language
+ git clone https://github.com/common-workflow-language/common-workflow-language.git common-workflow-language
else
- git clone -q https://github.com/rabix/common-workflow-language.git common-workflow-language
+ git clone -q https://github.com/common-workflow-language/common-workflow-language.git common-workflow-language
fi
fi
commit e52b761b16170faf94bad9dd407cbd3238bf371b
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Nov 24 21:14:00 2014 -0500
Ignore failure at "easy_install *.egg". refs #4400
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 82c45db..ad8e9f6 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -227,7 +227,7 @@ do_test() {
# .egg files that setup.py downloads.
cd "$WORKSPACE/$1" \
&& HOME="$GEMHOME" python setup.py test ${testargs[$1]} \
- && easy_install *.egg
+ && (easy_install *.egg || true)
else
"test_$1"
fi
commit bcbbd1e9d64d4527644072364a267eca13af434d
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Nov 24 13:09:01 2014 -0500
Make sure that our go packages get generated with a version number that
maps the last change to their code (i.e. the git hash for the directory
their source lives in), just like all our other packages.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 99d0681..9b567d4 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -301,7 +301,6 @@ if [[ "$DEBUG" != 0 ]]; then
fi
cd "$WORKSPACE"
-PKG_VERSION=$(version_from_git)
cd sdk/python
handle_python_package
@@ -343,6 +342,8 @@ else
fi
# Build arvados src deb package
+cd "$WORKSPACE"
+PKG_VERSION=$(version_from_git)
cd $WORKSPACE/debs
build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
@@ -360,16 +361,22 @@ mkdir -p "$GOPATH/src/git.curoverse.com"
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
# keepstore
+cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keepstore"
+PKG_VERSION=$(version_from_git)
go get "git.curoverse.com/arvados.git/services/keepstore"
cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
# keepproxy
+cd "$GOPATH/src/git.curoverse.com/arvados.git/services/keepproxy"
+PKG_VERSION=$(version_from_git)
go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
# crunchstat
+cd "$GOPATH/src/git.curoverse.com/arvados.git/services/crunchstat"
+PKG_VERSION=$(version_from_git)
go get "git.curoverse.com/arvados.git/services/crunchstat"
cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
commit 8a2221bea7f28533a62a1c39e822f8c9d30b0b95
Merge: 358ad8e 5ab76d5
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Nov 24 09:55:22 2014 -0500
Merge branch '4400-dry-python-tests' refs #4400
diff --cc jenkins/run-tests.sh
index 20c92bf,f5bef8c..82c45db
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@@ -118,7 -112,11 +112,8 @@@ report_outcomes()
}
declare -a failures
declare -A skip
+ declare -A testargs
-# Always skip CLI tests. They don't know how to use run_test_server.py.
-skip[cli]=1
-
while [[ -n "$1" ]]
do
arg="$1"; shift
commit 358ad8eabdc9eec2e8b690ca0310b213dc4f6098
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Nov 21 17:37:43 2014 -0500
run-tests: make the error message displayed when WORKSPACE isn't set a
bit prettier.
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 1f65c95..20c92bf 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -154,8 +154,15 @@ do
done
# Sanity check
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
+fi
+
echo "WORKSPACE=$WORKSPACE"
-[[ -n "$WORKSPACE" ]] || fatal "WORKSPACE not set"
if [[ -z "$CONFIGSRC" ]] && [[ -d "$HOME/arvados-api-server" ]]; then
# Jenkins expects us to use this by default.
commit 4a96daeef2dd5a209ec685ee3e1d6864987d1f4f
Merge: b5c641c ce997a9
Author: radhika <radhika at radhika>
Date: Fri Nov 21 15:04:32 2014 -0500
closes #4156
Merge branch '4156-enable-cli-tests'
commit 5ab76d533d0189c66d68220229b7453e33fbf4b7
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Nov 21 11:56:14 2014 -0500
4400: Fix syntax error.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 816fe83..f5bef8c 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -390,7 +390,7 @@ do_test cli
test_apiserver() {
cd "$WORKSPACE/services/api"
- HOME="$GEMHOME" bundle exec rake test $testargs[apiserver]
+ HOME="$GEMHOME" bundle exec rake test ${testargs[apiserver]}
}
do_test apiserver
@@ -409,7 +409,7 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
&& HOME="$GEMHOME" bundle install --no-deployment \
- && HOME="$GEMHOME" bundle exec rake test $testargs[workbench]
+ && HOME="$GEMHOME" bundle exec rake test ${testargs[workbench]}
}
do_test workbench
commit ce997a9f7f79d7ee4c5ed9ca9ed072c3626f82a6
Author: radhika <radhika at radhika>
Date: Fri Nov 21 08:53:58 2014 -0500
4156: Since all the cli tests that require apiserver are skipped, all cli tests pass now. Hence, do not skip running cli tests anymore.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 40a6daa..1f65c95 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -119,9 +119,6 @@ report_outcomes() {
declare -a failures
declare -A skip
-# Always skip CLI tests. They don't know how to use run_test_server.py.
-skip[cli]=1
-
while [[ -n "$1" ]]
do
arg="$1"; shift
commit ed9e8d8616e919f81703134a7d5cbac62f151877
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Nov 20 16:55:41 2014 -0500
4400: Dry up test recipes for python bits. Tweak "extra test args for suite X" feature.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 40a6daa..816fe83 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -23,7 +23,7 @@ CONFIGSRC=path Dir with api server config files to copy into source tree.
(If none given, leave config files alone in source tree.)
apiserver_test="TEST=test/functional/arvados/v1/collections_controller_test.rb"
Restrict apiserver tests to the given file
-python_sdk_test="--test-suite test.test_keep_locator"
+sdk/python_test="--test-suite test.test_keep_locator"
Restrict Python SDK tests to the given class
workbench_test="TEST=test/integration/pipeline_instances_test.rb"
Restrict Workbench tests to the given file
@@ -59,12 +59,6 @@ GEMHOME=
COLUMNS=80
-cli_test=
-workbench_test=
-apiserver_test=
-python_sdk_test=
-ruby_sdk_test=
-fuse_test=
leave_temp=
skip_install=
@@ -118,6 +112,7 @@ report_outcomes() {
}
declare -a failures
declare -A skip
+declare -A testargs
# Always skip CLI tests. They don't know how to use run_test_server.py.
skip[cli]=1
@@ -146,6 +141,11 @@ do
leave_temp[GOPATH]=1
leave_temp[GEMHOME]=1
;;
+ *_test=*)
+ suite="${arg%%_test=*}"
+ args="${arg#*=}"
+ testargs["$suite"]="$args"
+ ;;
*=*)
eval export $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2-)\"
;;
@@ -212,6 +212,18 @@ do_test() {
if [[ "$2" == "go" ]]
then
go test "git.curoverse.com/arvados.git/$1"
+ elif [[ "$2" == "pip" ]]
+ then
+ # Other test suites can depend on tests_require
+ # dependencies of this package. For example, keepproxy runs
+ # run_test_server.py, which depends on the yaml package,
+ # which is in sdk/python's tests_require but not
+ # install_requires, and therefore does not get installed by
+ # setuptools until we run "setup.py test" *and* install the
+ # .egg files that setup.py downloads.
+ cd "$WORKSPACE/$1" \
+ && HOME="$GEMHOME" python setup.py test ${testargs[$1]} \
+ && easy_install *.egg
else
"test_$1"
fi
@@ -230,6 +242,11 @@ do_install() {
if [[ "$2" == "go" ]]
then
go get -t "git.curoverse.com/arvados.git/$1"
+ elif [[ "$2" == "pip" ]]
+ then
+ cd "$WORKSPACE/$1" \
+ && python setup.py sdist rotate --keep=1 --match .tar.gz \
+ && pip install --upgrade dist/*.tar.gz
else
"install_$1"
fi
@@ -271,25 +288,21 @@ install_cli() {
}
do_install cli
-install_python_sdk() {
- # Install the Python SDK early. Various other test suites (like
- # keepproxy) bring up run_test_server.py, which imports the arvados
- # module. We can't actually *test* the Python SDK yet though, because
- # its own test suite brings up some of those other programs (like
- # keepproxy).
-
- cd "$WORKSPACE/sdk/python" \
- && python setup.py sdist rotate --keep=1 --match .tar.gz \
- && pip install dist/arvados-python-client-0.1.*.tar.gz
-}
-do_install python_sdk
-
-install_fuse() {
- cd "$WORKSPACE/services/fuse" \
- && python setup.py sdist rotate --keep=1 --match .tar.gz \
- && pip install dist/arvados_fuse-0.1.*.tar.gz
-}
-do_install fuse
+# Install the Python SDK early. Various other test suites (like
+# keepproxy) bring up run_test_server.py, which imports the arvados
+# module. We can't actually *test* the Python SDK yet though, because
+# its own test suite brings up some of those other programs (like
+# keepproxy).
+declare -a pythonstuff
+pythonstuff=(
+ sdk/python
+ services/fuse
+ services/nodemanager
+ )
+for p in "${pythonstuff[@]}"
+do
+ do_install "$p" pip
+done
install_apiserver() {
cd "$WORKSPACE/services/api"
@@ -362,7 +375,7 @@ do_test doclinkchecker
test_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
&& HOME="$GEMHOME" bundle install --no-deployment \
- && HOME="$GEMHOME" bundle exec rake test
+ && HOME="$GEMHOME" bundle exec rake test ${testargs[sdk/ruby]}
}
do_test ruby_sdk
@@ -371,45 +384,22 @@ test_cli() {
cd "$WORKSPACE/sdk/cli" \
&& HOME="$GEMHOME" bundle install --no-deployment \
&& mkdir -p /tmp/keep \
- && KEEP_LOCAL_STORE=/tmp/keep HOME="$GEMHOME" bundle exec rake test $cli_test
+ && KEEP_LOCAL_STORE=/tmp/keep HOME="$GEMHOME" bundle exec rake test ${testargs[sdk/cli]}
}
do_test cli
test_apiserver() {
cd "$WORKSPACE/services/api"
- HOME="$GEMHOME" bundle exec rake test $apiserver_test
+ HOME="$GEMHOME" bundle exec rake test $testargs[apiserver]
}
do_test apiserver
-test_python_sdk() {
- # Python SDK. We test this before testing keepproxy: keepproxy runs
- # run_test_server.py, which depends on the yaml package, which is in
- # tests_require but not install_requires, and therefore does not get
- # installed by setuptools until we run "setup.py test" *and* install
- # the .egg files that setup.py downloads.
-
- cd "$WORKSPACE/sdk/python" \
- && python setup.py test $python_sdk_test
- r=$?
- easy_install *.egg
- return $r
-}
-do_test python_sdk
-
-test_fuse() {
- # Install test dependencies here too, in case run_test_server needs them.
- cd "$WORKSPACE/services/fuse" \
- && python setup.py test $fuse_test
- r=$?
- easy_install *.egg
- return $r
-}
-do_test fuse
-
-test_nodemanager() {
- cd "$WORKSPACE/services/nodemanager" && python setup.py test
-}
-do_test nodemanager
+# We must test sdk/python before testing services/keepproxy, because
+# keepproxy depends on sdk/python's test dependencies.
+for p in "${pythonstuff[@]}"
+do
+ do_test "$p" pip
+done
for g in "${gostuff[@]}"
do
@@ -419,7 +409,7 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
&& HOME="$GEMHOME" bundle install --no-deployment \
- && HOME="$GEMHOME" bundle exec rake test $workbench_test
+ && HOME="$GEMHOME" bundle exec rake test $testargs[workbench]
}
do_test workbench
commit b5c641cae15249d888c7bb41c937338af8f7102d
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Nov 19 08:57:18 2014 -0500
Refactor deploy script a bit; make it less sensitive to transient errors
we don't care about.
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 599dff3..d4e31f1 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -1,15 +1,37 @@
#!/bin/bash
+
+read -rd "\000" helpmessage <<EOF
+$(basename $0): Deploy Arvados to a cluster
+
+Syntax:
+ WORKSPACE=/path/to/arvados $(basename $0) <identifier> <deploy_repo_name>
+
+Options:
+
+identifier Arvados cluster name
+deploy_repo_name Name for the repository with the (capistrano) deploy scripts
+
+WORKSPACE=path Path to the Arvados source tree to deploy from
+
+EOF
+
+
IDENTIFIER=$1
DEPLOY_REPO=$2
-if [[ "$IDENTIFIER" == '' ]]; then
- echo "Syntax: $0 <identifier> <deploy_repo_name>"
+if [[ "$IDENTIFIER" == '' || "$DEPLOY_REPO" == '' ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
exit 1
fi
-if [[ "$DEPLOY_REPO" == '' ]]; then
- echo "Syntax: $0 <identifier> <deploy_repo_name>"
+# Sanity check
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
exit 1
fi
@@ -33,6 +55,29 @@ rvm use default
# Just say what version of ruby we're running
ruby --version
+function run_puppet() {
+ node=$1
+ return_var=$2
+
+ TMP_FILE=`mktemp`
+ ssh -t -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node.$IDENTIFIER -C "/usr/bin/puppet agent -t" | tee $TMP_FILE
+
+ ECODE=$?
+ RESULT=$(cat $TMP_FILE)
+
+ if [[ "$ECODE" != "255" && ! ("$RESULT" =~ 'already in progress') && "$ECODE" != "2" && "$ECODE" != "0" ]]; then
+ # Puppet exists 255 if the connection timed out. Just ignore that, it's possible that this node is
+ # a compute node that was being shut down.
+ # Puppet exits 2 if there are changes. For real!
+ # Puppet prints 'Notice: Run of Puppet configuration client already in progress' if another puppet process
+ # was already running
+ echo "ERROR updating $node.$IDENTIFIER: exit code $ECODE"
+ fi
+ rm -f $TMP_FILE
+ echo
+ eval "$return_var=$ECODE"
+}
+
function ensure_symlink() {
if [[ ! -L $WORKSPACE/$1 ]]; then
ln -s $WORKSPACE/$DEPLOY_REPO/$1 $WORKSPACE/$1
@@ -148,15 +193,8 @@ if [[ "$COMPRESSED_NODE_LIST" != '' ]]; then
SUM_ECODE=0
for node in $COMPUTE_NODES; do
echo "Updating $node.$IDENTIFIER"
- RESULT=`ssh -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node.$IDENTIFIER -C "/usr/bin/puppet agent -t"`
- ECODE=$?
- if [[ "$ECODE" != "255" && ! ("$RESULT" =~ 'already in progress') && "$ECODE" != "2" && "$ECODE" != "0" ]]; then
- # 255 -> connection timed out. Just ignore that, it's possible the compute node was being shut down.
- # Puppet exits '2' if there are changes. For real!
- # Puppet prints 'Notice: Run of Puppet configuration client already in progress' if another puppet process was already running
- SUM_ECODE=$(($SUM_ECODE + $ECODE))
- echo "ERROR updating $node.$IDENTIFIER: exit code $ECODE"
- fi
+ run_puppet $node ECODE
+ SUM_ECODE=$(($SUM_ECODE + $ECODE))
done
if [[ "$SUM_ECODE" != "0" ]]; then
@@ -169,9 +207,7 @@ title "Update compute node(s) complete"
title "Update shell"
-ssh -p2222 root at shell.$IDENTIFIER -C "/usr/bin/puppet agent -t"
-
-ECODE=$?
+run_puppet shell ECODE
if [[ "$ECODE" == "2" ]]; then
# Puppet exits '2' if there are changes. For real!
@@ -187,9 +223,7 @@ title "Update shell complete"
title "Update keep0"
-ssh -p2222 root at keep0.$IDENTIFIER -C "/usr/bin/puppet agent -t"
-
-ECODE=$?
+run_puppet keep0 ECODE
if [[ "$ECODE" == "2" ]]; then
# Puppet exits '2' if there are changes. For real!
commit 27d5f6ed785b6e127cf9a2f1145fac21627a9a89
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Nov 18 16:31:54 2014 -0500
Do not blow up when another puppet process is already running during deploy.
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index a55678b..599dff3 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -148,11 +148,12 @@ if [[ "$COMPRESSED_NODE_LIST" != '' ]]; then
SUM_ECODE=0
for node in $COMPUTE_NODES; do
echo "Updating $node.$IDENTIFIER"
- ssh -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node.$IDENTIFIER -C "/usr/bin/puppet agent -t"
+ RESULT=`ssh -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node.$IDENTIFIER -C "/usr/bin/puppet agent -t"`
ECODE=$?
- if [[ "$ECODE" != "255" && "$ECODE" != "2" && "$ECODE" != "0" ]]; then
+ if [[ "$ECODE" != "255" && ! ("$RESULT" =~ 'already in progress') && "$ECODE" != "2" && "$ECODE" != "0" ]]; then
# 255 -> connection timed out. Just ignore that, it's possible the compute node was being shut down.
# Puppet exits '2' if there are changes. For real!
+ # Puppet prints 'Notice: Run of Puppet configuration client already in progress' if another puppet process was already running
SUM_ECODE=$(($SUM_ECODE + $ECODE))
echo "ERROR updating $node.$IDENTIFIER: exit code $ECODE"
fi
commit cd13ebefd14bc536d069f395bc866e902550ffff
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Nov 18 16:22:21 2014 -0500
Build a python-jsonschema package, because cwltool requires version 2.4
which is newer than what's available in wheezy(-backports).
refs #4572
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 8e0290d..99d0681 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -397,7 +397,7 @@ build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curovers
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
- pykka apache-libcloud requests six pyexecjs; do
+ pykka apache-libcloud requests six pyexecjs jsonschema; do
build_and_scp_deb "$deppkg"
done
commit 44c80db735811f803760159dd3f1065ab0822179
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Nov 18 13:26:23 2014 -0500
Also build a deb package for python-pyexecjs, which is a dependency for
cwltool
refs #4572
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 71ac1fe..8e0290d 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -397,7 +397,7 @@ build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curovers
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
- pykka apache-libcloud requests six; do
+ pykka apache-libcloud requests six pyexecjs; do
build_and_scp_deb "$deppkg"
done
commit 85ea947a0961e4217be9d53763fb5f40aed6e3db
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Nov 18 12:01:10 2014 -0500
Package cwltool.
Refs #4572
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index b78cf7f..71ac1fe 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -401,6 +401,36 @@ for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
build_and_scp_deb "$deppkg"
done
+# cwltool from common-workflow-language. We use this in arv-run-pipeline-instance.
+# We use $WORKSPACE/common-workflow-language as the clean directory from which to build the cwltool package
+if [[ ! -d "$WORKSPACE/common-workflow-language" ]]; then
+ mkdir "$WORKSPACE/common-workflow-language"
+ cd "$WORKSPACE"
+ if [[ "$DEBUG" != 0 ]]; then
+ git clone https://github.com/rabix/common-workflow-language.git common-workflow-language
+ else
+ git clone -q https://github.com/rabix/common-workflow-language.git common-workflow-language
+ fi
+fi
+
+cd "$WORKSPACE/common-workflow-language"
+if [[ "$DEBUG" != 0 ]]; then
+ git checkout master
+ git pull
+else
+ git checkout -q master
+ git pull -q
+fi
+
+cd reference
+handle_python_package
+CWLTOOL_VERSION=`git log --first-parent --max-count=1 --format='format:0.1.%ct.%h'`
+
+# Build cwltool package
+cd $WORKSPACE/debs
+
+build_and_scp_deb $WORKSPACE/common-workflow-language/reference cwltool 'Common Workflow Language Working Group' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/common-workflow-language/reference/cwltool.egg-info/PKG-INFO)"
+
# Finally, publish the packages, if necessary
if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
ssh -p2222 $APTUSER@$APTSERVER -t "cd tmp && ls -laF *deb && freight add *deb apt/wheezy && freight cache && rm -f *deb"
commit adb6ea84650d517054fc67c097b456714057d07f
Merge: c3db5a6 6268ada
Author: Brett Smith <brett at curoverse.com>
Date: Mon Nov 17 16:46:41 2014 -0500
Merge branch '4027-sdk-debian-packages-wip'
Refs #4027. Closes #4557.
commit 6268adae9d3812084c7854d62e1be8b5412605f8
Author: Brett Smith <brett at curoverse.com>
Date: Mon Nov 17 14:02:06 2014 -0500
4027: Build packages for Ruby and Perl SDKs.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 621df96..b78cf7f 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -85,6 +85,15 @@ if [[ "$DEBUG" != 0 ]]; then
echo "Workspace is $WORKSPACE"
fi
+version_from_git() {
+ # Generates a version number from the git log for the current working
+ # directory, and writes it to stdout.
+ local git_ts git_hash
+ declare $(TZ=UTC git log -n1 --first-parent --max-count=1 \
+ --format=format:"git_ts=%ct git_hash=%h" .)
+ echo "0.1.$(date -ud "@$git_ts" +%Y%m%d%H%M%S).$git_hash"
+}
+
handle_python_package () {
# This function assumes the current working directory is the python package directory
if [[ "$UPLOAD" != 0 ]]; then
@@ -106,14 +115,22 @@ handle_python_package () {
# Build debs for everything
build_and_scp_deb () {
+ # The package source. Depending on the source type, this can be a
+ # path, or the name of the package in an upstream repository (e.g.,
+ # pip).
PACKAGE=$1
shift
+ # The name of the package to build. Defaults to $PACKAGE.
PACKAGE_NAME=$1
shift
+ # Optional: the vendor of the package. Should be "Curoverse, Inc." for
+ # packages of our own software. Passed to fpm --vendor.
VENDOR=$1
shift
+ # The type of source package. Passed to fpm -s. Default "python".
PACKAGE_TYPE=$1
shift
+ # Optional: the package version number. Passed to fpm -v.
VERSION=$1
shift
@@ -139,6 +156,7 @@ build_and_scp_deb () {
COMMAND_ARR+=('-v' "$VERSION")
fi
+ # Append remaining function arguments directly to fpm's command line.
for i; do
COMMAND_ARR+=("$i")
done
@@ -198,6 +216,31 @@ if [[ "$DEBUG" != 0 ]]; then
echo "umask is" `umask`
fi
+# Perl packages
+if [[ "$DEBUG" != 0 ]]; then
+ echo -e "\nPerl packages\n"
+fi
+
+if [[ "$DEBUG" != 0 ]]; then
+ PERL_OUT=/dev/stdout
+else
+ PERL_OUT=/dev/null
+fi
+
+cd "$WORKSPACE/sdk/perl"
+
+if [[ -e Makefile ]]; then
+ make realclean >"$PERL_OUT"
+fi
+find -maxdepth 1 \( -name 'MANIFEST*' -or -name 'libarvados-perl_*.deb' \) \
+ -delete
+rm -rf install
+
+perl Makefile.PL >"$PERL_OUT" && \
+ make install PREFIX=install INSTALLDIRS=perl >"$PERL_OUT" && \
+ build_and_scp_deb install/=/usr libarvados-perl "Curoverse, Inc." dir \
+ "$(version_from_git)"
+
# Ruby gems
if [[ "$DEBUG" != 0 ]]; then
echo
@@ -205,10 +248,17 @@ if [[ "$DEBUG" != 0 ]]; then
echo
fi
+if type rvm-exec 2>/dev/null; then
+ FPM_GEM_PREFIX=$(rvm-exec system gem environment gemdir)
+else
+ FPM_GEM_PREFIX=$(gem environment gemdir)
+fi
+
cd "$WORKSPACE"
cd sdk/ruby
-# clean up old gems
-rm -f arvados-*gem
+# clean up old packages
+find -maxdepth 1 \( -name 'arvados-*.gem' -or -name 'rubygem-arvados_*.deb' \) \
+ -delete
if [[ "$DEBUG" != 0 ]]; then
gem build arvados.gemspec
@@ -222,6 +272,9 @@ if [[ "$UPLOAD" != 0 ]]; then
gem push arvados-*gem
fi
+build_and_scp_deb arvados-*.gem "" "Curoverse, Inc." gem "" \
+ --prefix "$FPM_GEM_PREFIX"
+
# Build arvados-cli GEM
cd "$WORKSPACE"
cd sdk/cli
@@ -248,10 +301,7 @@ if [[ "$DEBUG" != 0 ]]; then
fi
cd "$WORKSPACE"
-
-GIT_TIMESTAMP=`git log --first-parent --max-count=1 --format=format:%ct -n1 .`
-HUMAN_READABLE_TIMESTAMP=`TZ=UTC date -d @$GIT_TIMESTAMP +%Y%m%d%H%M%S`
-GIT_HASH=`git log --first-parent --max-count=1 --format=format:$HUMAN_READABLE_TIMESTAMP.%h -n1 .`
+PKG_VERSION=$(version_from_git)
cd sdk/python
handle_python_package
@@ -294,7 +344,7 @@ fi
# Build arvados src deb package
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
+build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
# clean up, check out master and step away from detached-head state
cd "$WORKSPACE/src-build-dir"
@@ -312,17 +362,17 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
# keepstore
go get "git.curoverse.com/arvados.git/services/keepstore"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
+build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
# keepproxy
go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
+build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
# crunchstat
go get "git.curoverse.com/arvados.git/services/crunchstat"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
+build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "$PKG_VERSION" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
# The Python SDK
# Please resist the temptation to add --no-python-fix-name to the fpm call here
commit c3db5a62be3c07e4ddaff6a76a8adcaaee1a0319
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Nov 17 15:55:01 2014 -0500
Retry docker push a few times if it fails.
No issue #
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index 8587996..0b5148e 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -9,9 +9,14 @@ title () {
}
docker_push () {
- $DOCKER push $*
-
- ECODE=$?
+ # Sometimes docker push fails; retry it a few times if necessary.
+ for i in `seq 1 5`; do
+ $DOCKER push $*
+ ECODE=$?
+ if [[ "$ECODE" == "0" ]]; then
+ break
+ fi
+ done
if [[ "$ECODE" != "0" ]]; then
title "!!!!!! docker push $* failed !!!!!!"
commit 91e919d5ef0dc210dc1e4c9c150acb7b082ce92a
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Nov 17 10:12:11 2014 -0500
Also build a package for python-six.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 976338f..621df96 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -347,7 +347,7 @@ build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curovers
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
- pykka apache-libcloud requests; do
+ pykka apache-libcloud requests six; do
build_and_scp_deb "$deppkg"
done
commit d3f844f115bdcb6b349b9aad3d46e32dd4d18d33
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Nov 12 14:58:05 2014 -0500
Export vars provided on command line, e.g., ARVADOS_DEBUG=1. No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index ec6f4b9..40a6daa 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -147,7 +147,7 @@ do
leave_temp[GEMHOME]=1
;;
*=*)
- eval $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2-)\"
+ eval export $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2-)\"
;;
*)
echo >&2 "$0: Unrecognized option: '$arg'. Try: $0 --help"
commit 0536d6d65bc15cc4286bf26e21b893d94eb9f4a2
Author: Brett Smith <brett at curoverse.com>
Date: Wed Nov 12 12:00:33 2014 -0500
Refs #3857: Build modern requests package for PySDK.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index eca16fc..976338f 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -347,7 +347,7 @@ build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curovers
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
- pykka apache-libcloud; do
+ pykka apache-libcloud requests; do
build_and_scp_deb "$deppkg"
done
commit 0528a6d4df9d81622f9813c2a8925dd7322a81bb
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Nov 4 11:14:18 2014 -0500
Small tweaks: fix example for running one API test in run-tests.sh;
capitalize Arvados in easy-docker-install.sh
no issue #
diff --git a/install/easy-docker-install.sh b/install/easy-docker-install.sh
index cc6f11e..d4d5a4e 100755
--- a/install/easy-docker-install.sh
+++ b/install/easy-docker-install.sh
@@ -50,7 +50,7 @@ main () {
fi
echo
- echo "If necessary, this command will download the latest arvados docker images."
+ echo "If necessary, this command will download the latest Arvados docker images."
echo "The download can take a long time, depending on the speed of your internet connection."
echo "When the images are downloaded, it will then start an Arvados environment on this computer."
echo
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 93245a6..ec6f4b9 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -21,7 +21,7 @@ Options:
WORKSPACE=path Arvados source tree to test.
CONFIGSRC=path Dir with api server config files to copy into source tree.
(If none given, leave config files alone in source tree.)
-apiserver_test="TEST=test/functional/arvados/v1/collections_test.rb"
+apiserver_test="TEST=test/functional/arvados/v1/collections_controller_test.rb"
Restrict apiserver tests to the given file
python_sdk_test="--test-suite test.test_keep_locator"
Restrict Python SDK tests to the given class
commit 076ba4ed7de93d81211c905551e38768f79a5518
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Nov 1 20:48:00 2014 -0400
Rename easy installation script. Update comments at the top of the script.
refs #4383
diff --git a/install/easy-evaluation.sh b/install/easy-docker-install.sh
similarity index 83%
rename from install/easy-evaluation.sh
rename to install/easy-docker-install.sh
index 16139d9..cc6f11e 100755
--- a/install/easy-evaluation.sh
+++ b/install/easy-docker-install.sh
@@ -1,14 +1,16 @@
#!/usr/bin/env bash
-# This script is intended for easy Arvados evaluation. It will download the
+# This script is intended to make Arvados installation easy. It will download the
# latest copy of the Arvados docker images as well as the arvdock command. It
-# then uses arvdoc to spin up Arvados on this computer.
+# then uses arvdock to spin up Arvados on this computer.
#
-# A version of this script is available at http://get.arvados.org, so that this
+# The latest version of this script is available at http://get.arvados.org, so that this
# command does the right thing:
#
# $ \curl -sSL http://get.arvados.org | bash
#
+# Prerequisites: working docker installation. Run this script as a user who is a member
+# of the docker group.
COLUMNS=80
commit 7ae95cd88f8de821b56d81a49a6999669d823c76
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Nov 1 20:41:48 2014 -0400
Add first version of the easy-evaluation script.
refs #4383
diff --git a/install/easy-evaluation.sh b/install/easy-evaluation.sh
new file mode 100755
index 0000000..16139d9
--- /dev/null
+++ b/install/easy-evaluation.sh
@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+
+# This script is intended for easy Arvados evaluation. It will download the
+# latest copy of the Arvados docker images as well as the arvdock command. It
+# then uses arvdoc to spin up Arvados on this computer.
+#
+# A version of this script is available at http://get.arvados.org, so that this
+# command does the right thing:
+#
+# $ \curl -sSL http://get.arvados.org | bash
+#
+
+COLUMNS=80
+
+fail () {
+ title "$*"
+ exit 1
+}
+
+title () {
+ printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
+}
+
+docker_pull () {
+ $DOCKER pull $*
+
+ ECODE=$?
+
+ if [[ "$ECODE" != "0" ]]; then
+ title "$DOCKER pull $* failed"
+ exit $ECODE
+ fi
+}
+
+main () {
+
+ \which which >/dev/null 2>&1 || fail "Error: could not find 'which' command."
+
+ # find the docker binary
+ DOCKER=`which docker.io`
+
+ if [[ "$DOCKER" == "" ]]; then
+ DOCKER=`which docker`
+ fi
+
+ if [[ "$DOCKER" == "" ]]; then
+ fail "Error: you need to have docker installed. Could not find the docker executable."
+ fi
+
+ echo
+ echo "If necessary, this command will download the latest arvados docker images."
+ echo "The download can take a long time, depending on the speed of your internet connection."
+ echo "When the images are downloaded, it will then start an Arvados environment on this computer."
+ echo
+ docker_pull arvados/workbench
+ docker_pull arvados/doc
+ docker_pull arvados/keep
+ docker_pull arvados/shell
+ docker_pull arvados/compute
+ docker_pull arvados/keep
+ docker_pull arvados/api
+ docker_pull crosbymichael/skydns
+ docker_pull crosbymichael/skydock
+
+ # Now download arvdock and start the containers
+ echo
+ echo Downloading arvdock
+ echo
+ \curl -sSL https://raw.githubusercontent.com/curoverse/arvados/master/docker/arvdock -o arvdock
+ chmod 755 arvdock
+
+ echo
+ echo Starting the docker containers
+ echo
+ ./arvdock start
+
+ echo To stop the containers, run
+ echo
+ echo ./arvdock stop
+ echo
+}
+
+main
commit 1b9ca54af0f7f70c65382ebe748a22e6763130e6
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Nov 1 15:08:28 2014 -0400
Make run-docker-tests.sh a bit smarter about detecting common failures.
No issue #
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index 8fbf562..8587996 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -9,7 +9,7 @@ title () {
}
docker_push () {
- docker push $*
+ $DOCKER push $*
ECODE=$?
@@ -19,8 +19,28 @@ docker_push () {
fi
}
+# Sanity check
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
+ exit 1
+fi
+
echo $WORKSPACE
+# find the docker binary
+DOCKER=`which docker.io`
+
+if [[ "$DOCKER" == "" ]]; then
+ DOCKER=`which docker`
+fi
+
+if [[ "$DOCKER" == "" ]]; then
+ title "Error: you need to have docker installed. Could not find the docker executable."
+ exit 1
+fi
+
# DOCKER
title "Starting docker build"
commit e9db1de8aaa0a519f31684561e3c660461ccf22e
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Oct 31 20:40:58 2014 -0400
Make sure all executables we package are executable.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 3a8f9c9..eca16fc 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -187,6 +187,9 @@ source /etc/profile.d/rvm.sh
# out our git tree here
chmod o+r "$WORKSPACE" -R
+# More cleanup - make sure all executables that we'll package are 755
+find -type d -name 'bin' |xargs -I {} find {} -type f |xargs -I {} chmod 755 {}
+
# Now fix our umask to something better suited to building and publishing
# gems and packages
umask 0022
commit 7b3296a9decf3443b47b386a399aecb5b996c3fa
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Oct 31 20:35:30 2014 -0400
Upload our docker images to the docker.io registry.
refs #4383
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index 9a3e566..8fbf562 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -8,6 +8,17 @@ title () {
printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
}
+docker_push () {
+ docker push $*
+
+ ECODE=$?
+
+ if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! docker push $* failed !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ fi
+}
+
echo $WORKSPACE
# DOCKER
@@ -34,6 +45,21 @@ fi
title "docker build complete"
+title "uploading images"
+
+if [[ "$ECODE" == "0" ]]; then
+ docker_push arvados/api
+ docker_push arvados/compute
+ docker_push arvados/doc
+ docker_push arvados/workbench
+ docker_push arvados/keep
+ docker_push arvados/shell
+else
+ title "upload arvados images SKIPPED because build failed"
+fi
+
+title "upload arvados images complete"
+
title "Starting docker java-bwa-samtools build"
./build.sh java-bwa-samtools-image
@@ -49,13 +75,10 @@ title "docker build java-bwa-samtools complete"
title "upload arvados/jobs image"
-docker push arvados/jobs
-
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! upload arvados/jobs FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
+if [[ "$ECODE" == "0" ]]; then
+ docker_push arvados/jobs
+else
+ title "upload arvados/jobs image SKIPPED because build failed"
fi
title "upload arvados/jobs image complete"
commit e71be8357f42713487c158be2493cd245963deb2
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Oct 24 16:06:47 2014 -0400
Clean up *.pyc files between tests.
closes #4313
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index c4c585f..93245a6 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -165,6 +165,10 @@ if [[ -z "$CONFIGSRC" ]] && [[ -d "$HOME/arvados-api-server" ]]; then
CONFIGSRC="$HOME/arvados-api-server"
fi
+# Clean up .pyc files that may exist in the workspace
+cd "$WORKSPACE"
+find -name '*.pyc' -delete
+
# Set up temporary install dirs (unless existing dirs were supplied)
for tmpdir in VENVDIR GOPATH GEMHOME
do
commit a522b5843859343b885850afff05130fa0b7345b
Author: Tom Clegg <tom at curoverse.com>
Date: Wed Oct 22 11:03:40 2014 -0400
Add services/crunchstat to test suite. refs #3826
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 412aac0..c4c585f 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -337,6 +337,7 @@ do_install apiserver
declare -a gostuff
gostuff=(
+ services/crunchstat
services/keepstore
services/keepproxy
sdk/go/arvadosclient
commit 1c44466e5983383785eaf25f3b5da01936cd2cad
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Oct 22 09:35:49 2014 -0400
Make run-deploy.sh a bit smarter about which compute nodes are actually
up.
Note: there are still race conditions to be addressed here. Or maybe
better, we probably just shouldn't care about compute node puppet runs
not completing succesfully. Either the node will get shut down by node
manager, or a regular hourly puppet run will DTRT.
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index a7ffac7..a55678b 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -136,24 +136,35 @@ fi
title "Deploying workbench complete"
-# Update compute0
-title "Update compute0"
-
-ssh -p2222 root at compute0.$IDENTIFIER -C "/usr/bin/puppet agent -t"
-
-ECODE=$?
-
-if [[ "$ECODE" == "2" ]]; then
- # Puppet exits '2' if there are changes. For real!
- ECODE=0
-fi
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! Update compute0 FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
+# Update compute node(s)
+title "Update compute node(s)"
+
+# Get list of nodes that are up
+COMPRESSED_NODE_LIST=`ssh -p2222 root@$IDENTIFIER -C "sinfo --long -p crypto -r -o "%N" -h"`
+
+if [[ "$COMPRESSED_NODE_LIST" != '' ]]; then
+ COMPUTE_NODES=`ssh -p2222 root@$IDENTIFIER -C "scontrol show hostname $COMPRESSED_NODE_LIST"`
+
+ SUM_ECODE=0
+ for node in $COMPUTE_NODES; do
+ echo "Updating $node.$IDENTIFIER"
+ ssh -p2222 -o "StrictHostKeyChecking no" -o "ConnectTimeout 5" root@$node.$IDENTIFIER -C "/usr/bin/puppet agent -t"
+ ECODE=$?
+ if [[ "$ECODE" != "255" && "$ECODE" != "2" && "$ECODE" != "0" ]]; then
+ # 255 -> connection timed out. Just ignore that, it's possible the compute node was being shut down.
+ # Puppet exits '2' if there are changes. For real!
+ SUM_ECODE=$(($SUM_ECODE + $ECODE))
+ echo "ERROR updating $node.$IDENTIFIER: exit code $ECODE"
+ fi
+ done
+
+ if [[ "$SUM_ECODE" != "0" ]]; then
+ title "!!!!!! Update compute node(s) FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $SUM_ECODE))
+ fi
fi
-title "Update compute0 complete"
+title "Update compute node(s) complete"
title "Update shell"
commit a013dc7e4f0d91499d7349eee553a771551cd13e
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Oct 21 09:00:59 2014 -0400
* exclude /usr/local/lib/python2.7/dist-packages/tests from our fpm packages
* exclude /usr/local/arvados/src/.git from the arvados-src package (bugfix)
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index edefe88..3a8f9c9 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -125,7 +125,7 @@ build_and_scp_deb () {
PACKAGE_TYPE='python'
fi
- declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "deb")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "deb" "-x" "usr/local/lib/python2.7/dist-packages/tests")
if [[ "$PACKAGE_NAME" != "$PACKAGE" ]]; then
COMMAND_ARR+=('-n' "$PACKAGE_NAME")
@@ -291,7 +291,7 @@ fi
# Build arvados src deb package
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "-x 'usr/local/arvados/src/.git*'" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
+build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
# clean up, check out master and step away from detached-head state
cd "$WORKSPACE/src-build-dir"
commit 1a655bf26c6bbec82c13d79e15f9fdd43fb18c1d
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Oct 21 08:39:06 2014 -0400
run-build-packages: Test that fpm is callable before trying to do
anything.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index aed423d..edefe88 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -70,6 +70,17 @@ if ! [[ -n "$WORKSPACE" ]]; then
exit 1
fi
+# Test for fpm
+fpm --version >/dev/null 2>&1
+
+if [[ "$?" != 0 ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: fpm not found"
+ echo >&2
+ exit 1
+fi
+
if [[ "$DEBUG" != 0 ]]; then
echo "Workspace is $WORKSPACE"
fi
commit 8f4aa687e70abdd7329a2ebf928a175eb529744b
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Oct 21 08:17:38 2014 -0400
Build a few more dependency packages for node manager.
refs #3244
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index c2bb150..aed423d 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -332,7 +332,8 @@ build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curovers
# A few dependencies
for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
- pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv; do
+ pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv \
+ pykka apache-libcloud; do
build_and_scp_deb "$deppkg"
done
commit 8d7ac59313253c4858bbe2cbd4db746b1d16e0c8
Author: Brett Smith <brett at curoverse.com>
Date: Mon Oct 20 16:50:44 2014 -0400
4239: Build more dependencies of new google-api-python-client.
Refs #4239.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index bea2c63..c2bb150 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -331,14 +331,10 @@ cd $WORKSPACE/debs
build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
# A few dependencies
-build_and_scp_deb python-gflags
-build_and_scp_deb pyvcf
-build_and_scp_deb google-api-python-client
-build_and_scp_deb oauth2client
-build_and_scp_deb uritemplate
-build_and_scp_deb httplib2
-build_and_scp_deb ws4py
-build_and_scp_deb virtualenv
+for deppkg in python-gflags pyvcf google-api-python-client oauth2client \
+ pyasn1 pyasn1-modules rsa uritemplate httplib2 ws4py virtualenv; do
+ build_and_scp_deb "$deppkg"
+done
# Finally, publish the packages, if necessary
if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
commit 14b624d40b6267456bffbbd516410dd802f143df
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Oct 17 11:18:06 2014 -0400
A few improvements for run-build-packages.sh:
* fix suggestion of env variable setting
* add a few new python packages that were split off from
google-api-python-client in version 1.3
* print package name when fpm build goes wrong
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 7f8e711..bea2c63 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -5,7 +5,7 @@ read -rd "\000" helpmessage <<EOF
$(basename $0): Build Arvados packages and (optionally) upload them.
Syntax:
- $(basename $0) WORKSPACE=/path/to/arvados [options]
+ WORKSPACE=/path/to/arvados $(basename $0) [options]
Options:
@@ -150,7 +150,7 @@ build_and_scp_deb () {
if [[ "$FPM_PACKAGE_NAME" == "" ]]; then
EXITCODE=1
- echo "Error: Unable to figure out package name from fpm results:"
+ echo "Error: $PACKAGE: Unable to figure out package name from fpm results:"
echo
echo $FPM_RESULTS
echo
@@ -334,6 +334,8 @@ build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curovers
build_and_scp_deb python-gflags
build_and_scp_deb pyvcf
build_and_scp_deb google-api-python-client
+build_and_scp_deb oauth2client
+build_and_scp_deb uritemplate
build_and_scp_deb httplib2
build_and_scp_deb ws4py
build_and_scp_deb virtualenv
commit 8dce81a50c4aa76f2d2bcb3742c9708e26b5dee1
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Oct 15 14:25:34 2014 -0400
Add blacklist to commit hook. no issue #
diff --git a/git/hooks/coding-standards.sh b/git/hooks/coding-standards.sh
index 79b0cdd..d4e4c71 100755
--- a/git/hooks/coding-standards.sh
+++ b/git/hooks/coding-standards.sh
@@ -13,6 +13,20 @@ $oldrev = ARGV[1]
$newrev = ARGV[2]
$user = ENV['USER']
+def blacklist bl
+ all_revs = `git rev-list #{$oldrev}..#{$newrev}`.split("\n")
+ all_revs.each do |rev|
+ bl.each do |b|
+ if rev == b
+ puts "Revision #{b} is blacklisted, you must remove it from your branch (possibly using git rebase) before you can push."
+ exit 1
+ end
+ end
+ end
+end
+
+blacklist ['26d74dc0524c87c5dcc0c76040ce413a4848b57a']
+
# Only enforce policy on the master branch
exit 0 if $refname != 'refs/heads/master'
commit 23fa740736501e4cf023a3a94a3f9e567e95c2fa
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Oct 15 10:55:32 2014 -0400
Make sure to catch the merge commit when doing git log in the directory
that is to be packaged.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 5f0d13e..7f8e711 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -235,9 +235,9 @@ fi
cd "$WORKSPACE"
-GIT_TIMESTAMP=`git log --format=format:%ct -n1 .`
+GIT_TIMESTAMP=`git log --first-parent --max-count=1 --format=format:%ct -n1 .`
HUMAN_READABLE_TIMESTAMP=`TZ=UTC date -d @$GIT_TIMESTAMP +%Y%m%d%H%M%S`
-GIT_HASH=`git log --format=format:$HUMAN_READABLE_TIMESTAMP.%h -n1 .`
+GIT_HASH=`git log --first-parent --max-count=1 --format=format:$HUMAN_READABLE_TIMESTAMP.%h -n1 .`
cd sdk/python
handle_python_package
commit ea45d699313db83eb9fc941bbd65872e1790b1e2
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Oct 14 13:17:18 2014 -0400
Upload our arvados/jobs image to the Docker Hub.
No issue #
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index fccdb23..9a3e566 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -47,4 +47,17 @@ fi
title "docker build java-bwa-samtools complete"
+title "upload arvados/jobs image"
+
+docker push arvados/jobs
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! upload arvados/jobs FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "upload arvados/jobs image complete"
+
exit $EXITCODE
commit aa68e1bfb8b5fe86127c4a3157183f423c9179b4
Author: Brett Smith <brett at curoverse.com>
Date: Tue Oct 14 10:15:53 2014 -0400
2881: Re-enable Node Manager tests.
The work done in refs #4139 should make these more reliable.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 05f009c..412aac0 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -122,12 +122,6 @@ declare -A skip
# Always skip CLI tests. They don't know how to use run_test_server.py.
skip[cli]=1
-# Skip Node Manager tests. Because these tests are multithreaded, their
-# performance is a little unpredictable, and Jenkins regularly has trouble
-# with them. Brett has a plan for making them more robust, but it's going
-# to take a little time to implement. -2014-10-10
-skip[nodemanager]=1
-
while [[ -n "$1" ]]
do
arg="$1"; shift
commit 284acd7cf652e9ca0fcd57a230fedcc2a155db67
Author: Brett Smith <brett at curoverse.com>
Date: Fri Oct 10 11:38:34 2014 -0400
2881: Skip Node Manager tests for now.
See comment for detailed rationale. Refs #2881, #4139.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 412aac0..05f009c 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -122,6 +122,12 @@ declare -A skip
# Always skip CLI tests. They don't know how to use run_test_server.py.
skip[cli]=1
+# Skip Node Manager tests. Because these tests are multithreaded, their
+# performance is a little unpredictable, and Jenkins regularly has trouble
+# with them. Brett has a plan for making them more robust, but it's going
+# to take a little time to implement. -2014-10-10
+skip[nodemanager]=1
+
while [[ -n "$1" ]]
do
arg="$1"; shift
commit 5060bf0df9927f1c4c20640db5d956692818f3cc
Author: Brett Smith <brett at curoverse.com>
Date: Thu Oct 9 10:33:45 2014 -0400
2881: run-tests tests the Node Manager.
Refs #2881.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 96fc4b7..412aac0 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -401,6 +401,11 @@ test_fuse() {
}
do_test fuse
+test_nodemanager() {
+ cd "$WORKSPACE/services/nodemanager" && python setup.py test
+}
+do_test nodemanager
+
for g in "${gostuff[@]}"
do
do_test "$g" go
commit 27880e36c02ea90649f0badfb8e88894ede6f6ce
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Oct 9 10:16:46 2014 -0400
Switch to human readable timestamps (YYYYMMDDhhmmss in UTC) for all our
debian packages.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 878a9c2..5f0d13e 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -235,7 +235,9 @@ fi
cd "$WORKSPACE"
-GIT_HASH=`git log --format=format:%ct.%h -n1 .`
+GIT_TIMESTAMP=`git log --format=format:%ct -n1 .`
+HUMAN_READABLE_TIMESTAMP=`TZ=UTC date -d @$GIT_TIMESTAMP +%Y%m%d%H%M%S`
+GIT_HASH=`git log --format=format:$HUMAN_READABLE_TIMESTAMP.%h -n1 .`
cd sdk/python
handle_python_package
commit 1530c4fdb9c691ac378ebc67ff3c5775f2f9ee9b
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Oct 9 10:12:39 2014 -0400
Improve output when fpm call fails.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index bbd4a2a..878a9c2 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -150,7 +150,10 @@ build_and_scp_deb () {
if [[ "$FPM_PACKAGE_NAME" == "" ]]; then
EXITCODE=1
- echo "Error: Unable to figure out package name from fpm results:\n $FPM_RESULTS"
+ echo "Error: Unable to figure out package name from fpm results:"
+ echo
+ echo $FPM_RESULTS
+ echo
else
if [[ ! $FPM_RESULTS =~ "File already exists" ]]; then
if [[ "$FPM_EXIT_CODE" != "0" ]]; then
commit eefe18d01ac964d062e32b9bbc7eed306fee9a8c
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Oct 8 17:41:08 2014 -0400
More updates for run-build-packages.sh:
* more code cleanups
* create node manager package
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index e02cc3d..bbd4a2a 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -70,110 +70,28 @@ if ! [[ -n "$WORKSPACE" ]]; then
exit 1
fi
-source /etc/profile.d/rvm.sh
-
if [[ "$DEBUG" != 0 ]]; then
echo "Workspace is $WORKSPACE"
fi
-# Make all files world-readable -- jenkins runs with umask 027, and has checked
-# out our git tree here
-chmod o+r "$WORKSPACE" -R
-
-# Now fix our umask to something better suited to building and publishing
-# gems and packages
-umask 0022
-
-if [[ "$DEBUG" != 0 ]]; then
- echo "umask is" `umask`
-fi
-
-# Build arvados GEM
-if [[ "$DEBUG" != 0 ]]; then
- echo "Build and publish ruby gems"
-fi
-
-cd "$WORKSPACE"
-cd sdk/ruby
-# clean up old gems
-rm -f arvados-*gem
-
-if [[ "$DEBUG" != 0 ]]; then
- gem build arvados.gemspec
-else
- # -q appears to be broken in gem version 2.2.2
- gem build arvados.gemspec -q >/dev/null
-fi
-
-if [[ "$UPLOAD" != 0 ]]; then
- # publish new gem
- gem push arvados-*gem
-fi
-
-# Build arvados-cli GEM
-cd "$WORKSPACE"
-cd sdk/cli
-# clean up old gems
-rm -f arvados-cli*gem
-
-if [[ "$DEBUG" != 0 ]]; then
- gem build arvados-cli.gemspec
-else
- # -q appears to be broken in gem version 2.2.2
- gem build arvados-cli.gemspec -q >/dev/null
-fi
-
-if [[ "$UPLOAD" != 0 ]]; then
- # publish new gem
- gem push arvados-cli*gem
-fi
-
-# Build arvados-python-client Python package
-if [[ "$DEBUG" != 0 ]]; then
- echo "Build and publish arvados-python-client package"
-fi
-
-cd "$WORKSPACE"
-
-GIT_HASH=`git log --format=format:%ct.%h -n1 .`
-
-cd sdk/python
-
-# Make sure only to use sdist - that's the only format pip can deal with (sigh)
-
-if [[ "$UPLOAD" != 0 ]]; then
- # Make sure only to use sdist - that's the only format pip can deal with (sigh)
- if [[ "$DEBUG" != 0 ]]; then
- python setup.py sdist upload
- else
- python setup.py -q sdist upload
- fi
-else
- # Make sure only to use sdist - that's the only format pip can deal with (sigh)
- if [[ "$DEBUG" != 0 ]]; then
- python setup.py sdist
- else
- python setup.py -q sdist
- fi
-fi
-
-cd ../../services/fuse
-
-if [[ "$UPLOAD" != 0 ]]; then
- # Make sure only to use sdist - that's the only format pip can deal with (sigh)
- if [[ "$DEBUG" != 0 ]]; then
- python setup.py sdist upload
- else
- python setup.py -q sdist upload
- fi
-else
- # Make sure only to use sdist - that's the only format pip can deal with (sigh)
- if [[ "$DEBUG" != 0 ]]; then
- python setup.py sdist
+handle_python_package () {
+ # This function assumes the current working directory is the python package directory
+ if [[ "$UPLOAD" != 0 ]]; then
+ # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ if [[ "$DEBUG" != 0 ]]; then
+ python setup.py sdist upload
+ else
+ python setup.py -q sdist upload
+ fi
else
- python setup.py -q sdist
+ # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ if [[ "$DEBUG" != 0 ]]; then
+ python setup.py sdist
+ else
+ python setup.py -q sdist
+ fi
fi
-fi
+}
# Build debs for everything
build_and_scp_deb () {
@@ -218,7 +136,6 @@ build_and_scp_deb () {
if [[ "$DEBUG" != 0 ]]; then
echo
- echo "Fpm command:"
echo "${COMMAND_ARR[@]}"
echo
fi
@@ -250,6 +167,82 @@ build_and_scp_deb () {
fi
}
+source /etc/profile.d/rvm.sh
+
+# Make all files world-readable -- jenkins runs with umask 027, and has checked
+# out our git tree here
+chmod o+r "$WORKSPACE" -R
+
+# Now fix our umask to something better suited to building and publishing
+# gems and packages
+umask 0022
+
+if [[ "$DEBUG" != 0 ]]; then
+ echo "umask is" `umask`
+fi
+
+# Ruby gems
+if [[ "$DEBUG" != 0 ]]; then
+ echo
+ echo "Ruby gems"
+ echo
+fi
+
+cd "$WORKSPACE"
+cd sdk/ruby
+# clean up old gems
+rm -f arvados-*gem
+
+if [[ "$DEBUG" != 0 ]]; then
+ gem build arvados.gemspec
+else
+ # -q appears to be broken in gem version 2.2.2
+ gem build arvados.gemspec -q >/dev/null
+fi
+
+if [[ "$UPLOAD" != 0 ]]; then
+ # publish new gem
+ gem push arvados-*gem
+fi
+
+# Build arvados-cli GEM
+cd "$WORKSPACE"
+cd sdk/cli
+# clean up old gems
+rm -f arvados-cli*gem
+
+if [[ "$DEBUG" != 0 ]]; then
+ gem build arvados-cli.gemspec
+else
+ # -q appears to be broken in gem version 2.2.2
+ gem build arvados-cli.gemspec -q >/dev/null
+fi
+
+if [[ "$UPLOAD" != 0 ]]; then
+ # publish new gem
+ gem push arvados-cli*gem
+fi
+
+# Python packages
+if [[ "$DEBUG" != 0 ]]; then
+ echo
+ echo "Python packages"
+ echo
+fi
+
+cd "$WORKSPACE"
+
+GIT_HASH=`git log --format=format:%ct.%h -n1 .`
+
+cd sdk/python
+handle_python_package
+
+cd ../../services/fuse
+handle_python_package
+
+cd ../../services/nodemanager
+handle_python_package
+
if [[ ! -d "$WORKSPACE/debs" ]]; then
mkdir -p $WORKSPACE/debs
fi
@@ -328,6 +321,10 @@ build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse,
cd $WORKSPACE/debs
build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
+# The node manager
+cd $WORKSPACE/debs
+build_and_scp_deb $WORKSPACE/services/nodemanager arvados-node-manager 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/nodemanager/arvados_node_manager.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados node manager"
+
# A few dependencies
build_and_scp_deb python-gflags
build_and_scp_deb pyvcf
commit 7fa64db2c56414cf9db215a1771266a4f9b6161b
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Oct 8 17:22:06 2014 -0400
Make run-build-packages.sh much more quiet by default. Also make
uploading optional. Change command line argument syntax to bring it in
line with run-tests.sh.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 173943e..e02cc3d 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -1,29 +1,80 @@
#!/bin/bash
-EXITCODE=0
-CALL_FREIGHT=0
-APTUSER=$1
-APTSERVER=$2
+read -rd "\000" helpmessage <<EOF
+$(basename $0): Build Arvados packages and (optionally) upload them.
-if [[ "$APTUSER" == '' ]]; then
- echo "Syntax: $0 <aptuser> <aptserver>"
- exit 1
-fi
+Syntax:
+ $(basename $0) WORKSPACE=/path/to/arvados [options]
+
+Options:
+
+--upload Upload packages (default: false)
+--scp-user USERNAME Scp user for apt server (only required when --upload is specified)
+--apt-server HOSTNAME Apt server hostname (only required when --upload is specified)
+--debug Output debug information (default: false)
+
+WORKSPACE=path Path to the Arvados source tree to build packages from
-if [[ "$APTSERVER" == '' ]]; then
- echo "Syntax: $0 <aptuser> <aptserver>"
+EOF
+
+EXITCODE=0
+CALL_FREIGHT=0
+
+DEBUG=0
+UPLOAD=0
+
+while [[ -n "$1" ]]
+do
+ arg="$1"; shift
+ case "$arg" in
+ --help)
+ echo >&2 "$helpmessage"
+ echo >&2
+ exit 1
+ ;;
+ --scp-user)
+ APTUSER="$1"; shift
+ ;;
+ --apt-server)
+ APTSERVER="$1"; shift
+ ;;
+ --debug)
+ DEBUG=1
+ ;;
+ --upload)
+ UPLOAD=1
+ ;;
+ *)
+ echo >&2 "$0: Unrecognized option: '$arg'. Try: $0 --help"
+ exit 1
+ ;;
+ esac
+done
+
+# Sanity checks
+if [[ "$UPLOAD" != '0' && ("$APTUSER" == '' || "$APTSERVER" == '') ]]; then
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: please specify --scp-user and --apt-server if --upload is set"
+ echo >&2
exit 1
fi
# Sanity check
if ! [[ -n "$WORKSPACE" ]]; then
- echo "WORKSPACE environment variable not set"
+ echo >&2 "$helpmessage"
+ echo >&2
+ echo >&2 "Error: WORKSPACE environment variable not set"
+ echo >&2
exit 1
fi
source /etc/profile.d/rvm.sh
-echo $WORKSPACE
+
+if [[ "$DEBUG" != 0 ]]; then
+ echo "Workspace is $WORKSPACE"
+fi
# Make all files world-readable -- jenkins runs with umask 027, and has checked
# out our git tree here
@@ -33,31 +84,55 @@ chmod o+r "$WORKSPACE" -R
# gems and packages
umask 0022
-echo "umask is"
-umask
+if [[ "$DEBUG" != 0 ]]; then
+ echo "umask is" `umask`
+fi
# Build arvados GEM
-echo "Build and publish ruby gem"
+if [[ "$DEBUG" != 0 ]]; then
+ echo "Build and publish ruby gems"
+fi
+
cd "$WORKSPACE"
cd sdk/ruby
# clean up old gems
rm -f arvados-*gem
-gem build arvados.gemspec
-# publish new gem
-gem push arvados-*gem
+
+if [[ "$DEBUG" != 0 ]]; then
+ gem build arvados.gemspec
+else
+ # -q appears to be broken in gem version 2.2.2
+ gem build arvados.gemspec -q >/dev/null
+fi
+
+if [[ "$UPLOAD" != 0 ]]; then
+ # publish new gem
+ gem push arvados-*gem
+fi
# Build arvados-cli GEM
-echo "Build and publish ruby gem"
cd "$WORKSPACE"
cd sdk/cli
# clean up old gems
rm -f arvados-cli*gem
-gem build arvados-cli.gemspec
-# publish new gem
-gem push arvados-cli*gem
+
+if [[ "$DEBUG" != 0 ]]; then
+ gem build arvados-cli.gemspec
+else
+ # -q appears to be broken in gem version 2.2.2
+ gem build arvados-cli.gemspec -q >/dev/null
+fi
+
+if [[ "$UPLOAD" != 0 ]]; then
+ # publish new gem
+ gem push arvados-cli*gem
+fi
# Build arvados-python-client Python package
-echo "Build and publish arvados-python-client package"
+if [[ "$DEBUG" != 0 ]]; then
+ echo "Build and publish arvados-python-client package"
+fi
+
cd "$WORKSPACE"
GIT_HASH=`git log --format=format:%ct.%h -n1 .`
@@ -65,12 +140,40 @@ GIT_HASH=`git log --format=format:%ct.%h -n1 .`
cd sdk/python
# Make sure only to use sdist - that's the only format pip can deal with (sigh)
-python setup.py sdist upload
+
+if [[ "$UPLOAD" != 0 ]]; then
+ # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ if [[ "$DEBUG" != 0 ]]; then
+ python setup.py sdist upload
+ else
+ python setup.py -q sdist upload
+ fi
+else
+ # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ if [[ "$DEBUG" != 0 ]]; then
+ python setup.py sdist
+ else
+ python setup.py -q sdist
+ fi
+fi
cd ../../services/fuse
-# Make sure only to use sdist - that's the only format pip can deal with (sigh)
-python setup.py sdist upload
+if [[ "$UPLOAD" != 0 ]]; then
+ # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ if [[ "$DEBUG" != 0 ]]; then
+ python setup.py sdist upload
+ else
+ python setup.py -q sdist upload
+ fi
+else
+ # Make sure only to use sdist - that's the only format pip can deal with (sigh)
+ if [[ "$DEBUG" != 0 ]]; then
+ python setup.py sdist
+ else
+ python setup.py -q sdist
+ fi
+fi
# Build debs for everything
build_and_scp_deb () {
@@ -113,6 +216,13 @@ build_and_scp_deb () {
COMMAND_ARR+=("$PACKAGE")
+ if [[ "$DEBUG" != 0 ]]; then
+ echo
+ echo "Fpm command:"
+ echo "${COMMAND_ARR[@]}"
+ echo
+ fi
+
FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
@@ -123,14 +233,16 @@ build_and_scp_deb () {
if [[ "$FPM_PACKAGE_NAME" == "" ]]; then
EXITCODE=1
- echo "Error: Unabled figure out package name from fpm results:\n $FPM_RESULTS"
+ echo "Error: Unable to figure out package name from fpm results:\n $FPM_RESULTS"
else
if [[ ! $FPM_RESULTS =~ "File already exists" ]]; then
if [[ "$FPM_EXIT_CODE" != "0" ]]; then
echo "Error building debian package for $1:\n $FPM_RESULTS"
else
- scp -P2222 $FPM_PACKAGE_NAME $APTUSER@$APTSERVER:tmp/
- CALL_FREIGHT=1
+ if [[ "$UPLOAD" != 0 ]]; then
+ scp -P2222 $FPM_PACKAGE_NAME $APTUSER@$APTSERVER:tmp/
+ CALL_FREIGHT=1
+ fi
fi
else
echo "Debian package $FPM_PACKAGE_NAME exists, not rebuilding"
@@ -147,16 +259,26 @@ fi
if [[ ! -d "$WORKSPACE/src-build-dir" ]]; then
mkdir "$WORKSPACE/src-build-dir"
cd "$WORKSPACE"
- git clone https://github.com/curoverse/arvados.git src-build-dir
+ if [[ "$DEBUG" != 0 ]]; then
+ git clone https://github.com/curoverse/arvados.git src-build-dir
+ else
+ git clone -q https://github.com/curoverse/arvados.git src-build-dir
+ fi
fi
cd "$WORKSPACE/src-build-dir"
# just in case, check out master
-git checkout master
-git pull
-
-# go into detached-head state
-git checkout `git log --format=format:%h -n1 .`
+if [[ "$DEBUG" != 0 ]]; then
+ git checkout master
+ git pull
+ # go into detached-head state
+ git checkout `git log --format=format:%h -n1 .`
+else
+ git checkout -q master
+ git pull -q
+ # go into detached-head state
+ git checkout -q `git log --format=format:%h -n1 .`
+fi
# Build arvados src deb package
cd $WORKSPACE/debs
@@ -164,7 +286,11 @@ build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src '
# clean up, check out master and step away from detached-head state
cd "$WORKSPACE/src-build-dir"
-git checkout master
+if [[ "$DEBUG" != 0 ]]; then
+ git checkout master
+else
+ git checkout -q master
+fi
# Keep
export GOPATH=$(mktemp -d)
@@ -211,10 +337,12 @@ build_and_scp_deb ws4py
build_and_scp_deb virtualenv
# Finally, publish the packages, if necessary
-if [[ "$CALL_FREIGHT" != "0" ]]; then
+if [[ "$UPLOAD" != 0 && "$CALL_FREIGHT" != 0 ]]; then
ssh -p2222 $APTUSER@$APTSERVER -t "cd tmp && ls -laF *deb && freight add *deb apt/wheezy && freight cache && rm -f *deb"
else
- echo "No new packages generated. No freight run necessary."
+ if [[ "$UPLOAD" != 0 ]]; then
+ echo "No new packages generated. No freight run necessary."
+ fi
fi
# clean up temporary GOPATH
commit c1ab6401ba557f8ca015eb462f8bbba692c7b88a
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Oct 8 10:56:24 2014 -0400
Allow for run-diagnostics-suite.sh to pull the git revision to test from
a configuration file.
No issue #
diff --git a/jenkins/run-diagnostics-suite.sh b/jenkins/run-diagnostics-suite.sh
index e1229ff..015a053 100755
--- a/jenkins/run-diagnostics-suite.sh
+++ b/jenkins/run-diagnostics-suite.sh
@@ -10,6 +10,22 @@ if [[ "$INSTANCE" == '' ]]; then
exit 1
fi
+if [[ "$REVISION" == '' ]]; then
+ # See if there's a configuration file with the revision?
+ CONFIG_PATH=/home/jenkins/configuration/$INSTANCE.arvadosapi.com-versions.conf
+ if [[ -f $CONFIG_PATH ]]; then
+ echo "Loading git revision from $CONFIG_PATH"
+ . $CONFIG_PATH
+ REVISION=$ARVADOS_GIT_REVISION
+ fi
+fi
+
+if [[ "$REVISION" != '' ]]; then
+ echo "Git revision is $REVISION"
+else
+ echo "No valid git revision found, proceeding with what is in place."
+fi
+
# Sanity check
if ! [[ -n "$WORKSPACE" ]]; then
echo "WORKSPACE environment variable not set"
commit 0ec1187c3f9b73dec2078ddfd98d3a248c3a1648
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Oct 7 23:04:52 2014 -0400
Allow specification of revision to test with diagnostics suite.
No issue #
diff --git a/jenkins/run-diagnostics-suite.sh b/jenkins/run-diagnostics-suite.sh
index dfd03a8..e1229ff 100755
--- a/jenkins/run-diagnostics-suite.sh
+++ b/jenkins/run-diagnostics-suite.sh
@@ -3,9 +3,10 @@
EXITCODE=0
INSTANCE=$1
+REVISION=$2
if [[ "$INSTANCE" == '' ]]; then
- echo "Syntax: $0 <instance>"
+ echo "Syntax: $0 <instance> [revision]"
exit 1
fi
@@ -36,6 +37,10 @@ timer_reset
cd $WORKSPACE
+if [[ "$REVISION" != '' ]]; then
+ git checkout $REVISION
+fi
+
cp -f /home/jenkins/diagnostics/arvados-workbench/$INSTANCE-application.yml $WORKSPACE/apps/workbench/config/application.yml
cd $WORKSPACE/apps/workbench
@@ -50,6 +55,10 @@ RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb
ECODE=$?
+if [[ "$REVISION" != '' ]]; then
+ git checkout master
+fi
+
if [[ "$ECODE" != "0" ]]; then
title "!!!!!! DIAGNOSTICS FAILED (`timer`) !!!!!!"
EXITCODE=$(($EXITCODE + $ECODE))
commit 0d50b5b6183156ba3dbe1d6fe16588dccfcdc962
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Oct 7 21:27:04 2014 -0400
Fix stupid bug in run-diagnostics-suite.sh - destination config file
needs to be application.yml
No issue #
diff --git a/jenkins/run-diagnostics-suite.sh b/jenkins/run-diagnostics-suite.sh
index 83e20be..dfd03a8 100755
--- a/jenkins/run-diagnostics-suite.sh
+++ b/jenkins/run-diagnostics-suite.sh
@@ -36,7 +36,7 @@ timer_reset
cd $WORKSPACE
-cp -f /home/jenkins/diagnostics/arvados-workbench/$INSTANCE-application.yml $WORKSPACE/apps/workbench/config/
+cp -f /home/jenkins/diagnostics/arvados-workbench/$INSTANCE-application.yml $WORKSPACE/apps/workbench/config/application.yml
cd $WORKSPACE/apps/workbench
commit 2923ce9f48f072119e8c5d0771ae3480a2679864
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Oct 7 20:34:07 2014 -0400
Parameterize the instance to run the diagnostics suite against.
No issue #
diff --git a/jenkins/run-diagnostics-suite.sh b/jenkins/run-diagnostics-suite.sh
index be4e559..83e20be 100755
--- a/jenkins/run-diagnostics-suite.sh
+++ b/jenkins/run-diagnostics-suite.sh
@@ -2,6 +2,13 @@
EXITCODE=0
+INSTANCE=$1
+
+if [[ "$INSTANCE" == '' ]]; then
+ echo "Syntax: $0 <instance>"
+ exit 1
+fi
+
# Sanity check
if ! [[ -n "$WORKSPACE" ]]; then
echo "WORKSPACE environment variable not set"
@@ -29,7 +36,7 @@ timer_reset
cd $WORKSPACE
-cp -f /home/jenkins/diagnostics/arvados-workbench/application.yml $WORKSPACE/apps/workbench/config/
+cp -f /home/jenkins/diagnostics/arvados-workbench/$INSTANCE-application.yml $WORKSPACE/apps/workbench/config/
cd $WORKSPACE/apps/workbench
commit c65e0ee82ed25c9161dfa63e3d1ff298bd0c4c17
Merge: cc48478 da1182c
Author: Brett Smith <brettcsmith at brettcsmith.org>
Date: Sun Oct 5 16:21:39 2014 -0400
Merge branch '3848-build-packages-pyversion'
Refs #3848. Closes #4110.
commit da1182c77e96a58bb6a829bf825d732cd299f807
Author: Brett Smith <brettcsmith at brettcsmith.org>
Date: Sun Oct 5 14:34:10 2014 -0400
3848: build-packages respects egg-info versions.
Now that our setup.py files know how to introspect git and generate
build tags, build-packages can use that version number rather than
overriding it.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 9881ded..173943e 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -64,24 +64,15 @@ GIT_HASH=`git log --format=format:%ct.%h -n1 .`
cd sdk/python
-# We mess with this file below, reset it here
-git checkout setup.py
-
# Make sure only to use sdist - that's the only format pip can deal with (sigh)
-python setup.py egg_info -b ".$GIT_HASH" sdist upload
+python setup.py sdist upload
cd ../../services/fuse
-# We mess with this file below, reset it here
-git checkout setup.py
-
# Make sure only to use sdist - that's the only format pip can deal with (sigh)
-python setup.py egg_info -b ".$GIT_HASH" sdist upload
+python setup.py sdist upload
# Build debs for everything
-
-# Build arvados src deb package
-
build_and_scp_deb () {
PACKAGE=$1
shift
@@ -167,6 +158,7 @@ git pull
# go into detached-head state
git checkout `git log --format=format:%h -n1 .`
+# Build arvados src deb package
cd $WORKSPACE/debs
build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "-x 'usr/local/arvados/src/.git*'" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
@@ -195,28 +187,20 @@ cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
# The Python SDK
-cd $WORKSPACE/sdk/python
-sed -i'' -e "s:version='0.1':version='0.1.$GIT_HASH':" setup.py
-
-cd $WORKSPACE/debs
-
# Please resist the temptation to add --no-python-fix-name to the fpm call here
# (which would remove the python- prefix from the package name), because this
# package is a dependency of arvados-fuse, and fpm can not omit the python-
# prefix from only one of the dependencies of a package... Maybe I could
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
-build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "0.1.${GIT_HASH}" "--url=https://arvados.org" "--description=The Arvados Python SDK"
-
-# The FUSE driver
-cd $WORKSPACE/services/fuse
-sed -i'' -e "s:version='0.1':version='0.1.$GIT_HASH':" setup.py
-
cd $WORKSPACE/debs
+build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/sdk/python/arvados_python_client.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Arvados Python SDK"
+# The FUSE driver
# Please seem comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
-build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "0.1.${GIT_HASH}" "--url=https://arvados.org" "--description=The Keep FUSE driver"
+cd $WORKSPACE/debs
+build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "$(awk '($1 == "Version:"){print $2}' $WORKSPACE/services/fuse/arvados_fuse.egg-info/PKG-INFO)" "--url=https://arvados.org" "--description=The Keep FUSE driver"
# A few dependencies
build_and_scp_deb python-gflags
commit cc48478bd40eed336631339af3c272dd1a1d86bf
Author: Ward Vandewege <ward at curoverse.com>
Date: Sun Oct 5 10:18:26 2014 -0400
Remove spurious newline
No issue #
diff --git a/README b/README
index 10aa02f..b076f0b 100644
--- a/README
+++ b/README
@@ -7,7 +7,6 @@ If you are interested in using Arvados or setting up your own Arvados
installation, you most likely do not need this source tree.
For the Arvados source code, check out the git repository at
-
https://github.com/curoverse/arvados
The main Arvados web site is
commit 7114c08c46f867f9e668d34b3fddeaf662a9b214
Author: Ward Vandewege <ward at curoverse.com>
Date: Sun Oct 5 10:17:01 2014 -0400
Add README, COPYING and agpl-3.0.txt.
No issue #
diff --git a/COPYING b/COPYING
new file mode 100644
index 0000000..af63e41
--- /dev/null
+++ b/COPYING
@@ -0,0 +1,2 @@
+This code is licenced under the GNU Affero General Public License version 3
+(see agpl-3.0.txt)
diff --git a/README b/README
new file mode 100644
index 0000000..10aa02f
--- /dev/null
+++ b/README
@@ -0,0 +1,31 @@
+Welcome to Arvados!
+
+This is the arvados-dev source tree. It contains scripts that can be useful
+if you want to hack on Arvados itself.
+
+If you are interested in using Arvados or setting up your own Arvados
+installation, you most likely do not need this source tree.
+
+For the Arvados source code, check out the git repository at
+
+ https://github.com/curoverse/arvados
+
+The main Arvados web site is
+ https://arvados.org
+
+The Arvados public wiki is located at
+ https://arvados.org/projects/arvados/wiki
+
+The Arvados public bug tracker is located at
+ https://arvados.org/projects/arvados/issues
+
+For support see
+ http://doc.arvados.org/user/getting_started/community.html
+
+Installation documentation is located at
+ http://doc.arvados.org/install
+
+If you wish to build the documentation yourself, follow the instructions in
+doc/README to build the documentation, then consult the "Install Guide".
+
+See COPYING for information about Arvados Free Software licenses.
diff --git a/agpl-3.0.txt b/agpl-3.0.txt
new file mode 100644
index 0000000..dba13ed
--- /dev/null
+++ b/agpl-3.0.txt
@@ -0,0 +1,661 @@
+ GNU AFFERO GENERAL PUBLIC LICENSE
+ Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+ Preamble
+
+ The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+ The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works. By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+ When we speak of free software, we are referring to freedom, not
+price. Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+ Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+ A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate. Many developers of free software are heartened and
+encouraged by the resulting cooperation. However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+ The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community. It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server. Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+ An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals. This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+ The precise terms and conditions for copying, distribution and
+modification follow.
+
+ TERMS AND CONDITIONS
+
+ 0. Definitions.
+
+ "This License" refers to version 3 of the GNU Affero General Public License.
+
+ "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+ "The Program" refers to any copyrightable work licensed under this
+License. Each licensee is addressed as "you". "Licensees" and
+"recipients" may be individuals or organizations.
+
+ To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy. The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+ A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+ To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy. Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+ To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies. Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+ An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License. If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+ 1. Source Code.
+
+ The "source code" for a work means the preferred form of the work
+for making modifications to it. "Object code" means any non-source
+form of a work.
+
+ A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+ The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form. A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+ The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities. However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work. For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+ The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+ The Corresponding Source for a work in source code form is that
+same work.
+
+ 2. Basic Permissions.
+
+ All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met. This License explicitly affirms your unlimited
+permission to run the unmodified Program. The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work. This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+ You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force. You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright. Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+ Conveying under any other circumstances is permitted solely under
+the conditions stated below. Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+ No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+ When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+ 4. Conveying Verbatim Copies.
+
+ You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+ You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+ 5. Conveying Modified Source Versions.
+
+ You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+ a) The work must carry prominent notices stating that you modified
+ it, and giving a relevant date.
+
+ b) The work must carry prominent notices stating that it is
+ released under this License and any conditions added under section
+ 7. This requirement modifies the requirement in section 4 to
+ "keep intact all notices".
+
+ c) You must license the entire work, as a whole, under this
+ License to anyone who comes into possession of a copy. This
+ License will therefore apply, along with any applicable section 7
+ additional terms, to the whole of the work, and all its parts,
+ regardless of how they are packaged. This License gives no
+ permission to license the work in any other way, but it does not
+ invalidate such permission if you have separately received it.
+
+ d) If the work has interactive user interfaces, each must display
+ Appropriate Legal Notices; however, if the Program has interactive
+ interfaces that do not display Appropriate Legal Notices, your
+ work need not make them do so.
+
+ A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit. Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+ 6. Conveying Non-Source Forms.
+
+ You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+ a) Convey the object code in, or embodied in, a physical product
+ (including a physical distribution medium), accompanied by the
+ Corresponding Source fixed on a durable physical medium
+ customarily used for software interchange.
+
+ b) Convey the object code in, or embodied in, a physical product
+ (including a physical distribution medium), accompanied by a
+ written offer, valid for at least three years and valid for as
+ long as you offer spare parts or customer support for that product
+ model, to give anyone who possesses the object code either (1) a
+ copy of the Corresponding Source for all the software in the
+ product that is covered by this License, on a durable physical
+ medium customarily used for software interchange, for a price no
+ more than your reasonable cost of physically performing this
+ conveying of source, or (2) access to copy the
+ Corresponding Source from a network server at no charge.
+
+ c) Convey individual copies of the object code with a copy of the
+ written offer to provide the Corresponding Source. This
+ alternative is allowed only occasionally and noncommercially, and
+ only if you received the object code with such an offer, in accord
+ with subsection 6b.
+
+ d) Convey the object code by offering access from a designated
+ place (gratis or for a charge), and offer equivalent access to the
+ Corresponding Source in the same way through the same place at no
+ further charge. You need not require recipients to copy the
+ Corresponding Source along with the object code. If the place to
+ copy the object code is a network server, the Corresponding Source
+ may be on a different server (operated by you or a third party)
+ that supports equivalent copying facilities, provided you maintain
+ clear directions next to the object code saying where to find the
+ Corresponding Source. Regardless of what server hosts the
+ Corresponding Source, you remain obligated to ensure that it is
+ available for as long as needed to satisfy these requirements.
+
+ e) Convey the object code using peer-to-peer transmission, provided
+ you inform other peers where the object code and Corresponding
+ Source of the work are being offered to the general public at no
+ charge under subsection 6d.
+
+ A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+ A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling. In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage. For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product. A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+ "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source. The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+ If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information. But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+ The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed. Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+ Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+ 7. Additional Terms.
+
+ "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law. If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+ When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it. (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.) You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+ Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+ a) Disclaiming warranty or limiting liability differently from the
+ terms of sections 15 and 16 of this License; or
+
+ b) Requiring preservation of specified reasonable legal notices or
+ author attributions in that material or in the Appropriate Legal
+ Notices displayed by works containing it; or
+
+ c) Prohibiting misrepresentation of the origin of that material, or
+ requiring that modified versions of such material be marked in
+ reasonable ways as different from the original version; or
+
+ d) Limiting the use for publicity purposes of names of licensors or
+ authors of the material; or
+
+ e) Declining to grant rights under trademark law for use of some
+ trade names, trademarks, or service marks; or
+
+ f) Requiring indemnification of licensors and authors of that
+ material by anyone who conveys the material (or modified versions of
+ it) with contractual assumptions of liability to the recipient, for
+ any liability that these contractual assumptions directly impose on
+ those licensors and authors.
+
+ All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10. If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term. If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+ If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+ Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+ 8. Termination.
+
+ You may not propagate or modify a covered work except as expressly
+provided under this License. Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+ However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+ Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+ Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License. If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+ 9. Acceptance Not Required for Having Copies.
+
+ You are not required to accept this License in order to receive or
+run a copy of the Program. Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance. However,
+nothing other than this License grants you permission to propagate or
+modify any covered work. These actions infringe copyright if you do
+not accept this License. Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+ 10. Automatic Licensing of Downstream Recipients.
+
+ Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License. You are not responsible
+for enforcing compliance by third parties with this License.
+
+ An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations. If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+ You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License. For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+ 11. Patents.
+
+ A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based. The
+work thus licensed is called the contributor's "contributor version".
+
+ A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version. For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+ Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+ In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement). To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+ If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients. "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+ If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+ A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License. You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+ Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+ 12. No Surrender of Others' Freedom.
+
+ If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all. For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+ 13. Remote Network Interaction; Use with the GNU General Public License.
+
+ Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software. This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+ Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work. The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+ 14. Revised Versions of this License.
+
+ The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time. Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation. If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+ If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+ Later license versions may give you additional or different
+permissions. However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+ 15. Disclaimer of Warranty.
+
+ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+ 16. Limitation of Liability.
+
+ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+ 17. Interpretation of Sections 15 and 16.
+
+ If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+ END OF TERMS AND CONDITIONS
+
+ How to Apply These Terms to Your New Programs
+
+ If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+ To do so, attach the following notices to the program. It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+ <one line to give the program's name and a brief idea of what it does.>
+ Copyright (C) <year> <name of author>
+
+ This program is free software: you can redistribute it and/or modify
+ it under the terms of the GNU Affero General Public License as published by
+ the Free Software Foundation, either version 3 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU Affero General Public License for more details.
+
+ You should have received a copy of the GNU Affero General Public License
+ along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+ If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source. For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code. There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+ You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<http://www.gnu.org/licenses/>.
commit ad3df9216ff20898eb96353e5d6c52b271a87375
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Oct 3 21:29:43 2014 -0400
Make run-deploy.sh do a bit more work to update the rest of the staging
cluster.
No issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index b379fff..a7ffac7 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -88,7 +88,7 @@ title "Deploying API server complete"
# Install updated debian packages
title "Deploying updated arvados debian packages"
-ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get install arvados-src python-arvados-fuse python-arvados-python-client"
+ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get -qqy install arvados-src python-arvados-fuse python-arvados-python-client"
if [[ "$ECODE" != "0" ]]; then
title "!!!!!! DEPLOYING DEBIAN PACKAGES FAILED !!!!!!"
@@ -136,4 +136,59 @@ fi
title "Deploying workbench complete"
+# Update compute0
+title "Update compute0"
+
+ssh -p2222 root at compute0.$IDENTIFIER -C "/usr/bin/puppet agent -t"
+
+ECODE=$?
+
+if [[ "$ECODE" == "2" ]]; then
+ # Puppet exits '2' if there are changes. For real!
+ ECODE=0
+fi
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! Update compute0 FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "Update compute0 complete"
+
+title "Update shell"
+
+ssh -p2222 root at shell.$IDENTIFIER -C "/usr/bin/puppet agent -t"
+
+ECODE=$?
+
+if [[ "$ECODE" == "2" ]]; then
+ # Puppet exits '2' if there are changes. For real!
+ ECODE=0
+fi
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! Update shell FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "Update shell complete"
+
+title "Update keep0"
+
+ssh -p2222 root at keep0.$IDENTIFIER -C "/usr/bin/puppet agent -t"
+
+ECODE=$?
+
+if [[ "$ECODE" == "2" ]]; then
+ # Puppet exits '2' if there are changes. For real!
+ ECODE=0
+fi
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! Update keep0 FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "Update keep0 complete"
+
exit $EXITCODE
commit 77dffc7b9308abb3d331c165c3042ec6ff41992a
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Oct 2 20:15:35 2014 -0400
Deal with Jenkins' new, tight umask in a more thorough way.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index fe64f89..9881ded 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -25,6 +25,17 @@ fi
source /etc/profile.d/rvm.sh
echo $WORKSPACE
+# Make all files world-readable -- jenkins runs with umask 027, and has checked
+# out our git tree here
+chmod o+r "$WORKSPACE" -R
+
+# Now fix our umask to something better suited to building and publishing
+# gems and packages
+umask 0022
+
+echo "umask is"
+umask
+
# Build arvados GEM
echo "Build and publish ruby gem"
cd "$WORKSPACE"
@@ -127,7 +138,6 @@ build_and_scp_deb () {
if [[ "$FPM_EXIT_CODE" != "0" ]]; then
echo "Error building debian package for $1:\n $FPM_RESULTS"
else
- chmod 644 $FPM_PACKAGE_NAME
scp -P2222 $FPM_PACKAGE_NAME $APTUSER@$APTSERVER:tmp/
CALL_FREIGHT=1
fi
@@ -147,15 +157,15 @@ if [[ ! -d "$WORKSPACE/src-build-dir" ]]; then
mkdir "$WORKSPACE/src-build-dir"
cd "$WORKSPACE"
git clone https://github.com/curoverse/arvados.git src-build-dir
-fi
+fi
cd "$WORKSPACE/src-build-dir"
# just in case, check out master
git checkout master
git pull
+
# go into detached-head state
git checkout `git log --format=format:%h -n1 .`
-cd $WORKSPACE
cd $WORKSPACE/debs
build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "-x 'usr/local/arvados/src/.git*'" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
commit 30522a0e950b024366b07c8a25f24a66935600a0
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Oct 1 17:22:51 2014 -0400
Make sure our generated packages are world-readable, otherwise they
won't be useful in the package repo.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 56a43b1..fe64f89 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -127,6 +127,7 @@ build_and_scp_deb () {
if [[ "$FPM_EXIT_CODE" != "0" ]]; then
echo "Error building debian package for $1:\n $FPM_RESULTS"
else
+ chmod 644 $FPM_PACKAGE_NAME
scp -P2222 $FPM_PACKAGE_NAME $APTUSER@$APTSERVER:tmp/
CALL_FREIGHT=1
fi
commit d99b860348c603932ca80c1c5b80c3e6880c1f35
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Oct 1 16:28:42 2014 -0400
Add new run-diagnostics-suite.sh jenkins script.
No issue #
diff --git a/jenkins/run-diagnostics-suite.sh b/jenkins/run-diagnostics-suite.sh
new file mode 100755
index 0000000..be4e559
--- /dev/null
+++ b/jenkins/run-diagnostics-suite.sh
@@ -0,0 +1,54 @@
+#!/bin/bash
+
+EXITCODE=0
+
+# Sanity check
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo "WORKSPACE environment variable not set"
+ exit 1
+fi
+
+title () {
+ txt="********** $1 **********"
+ printf "\n%*s%s\n\n" $((($COLUMNS-${#txt})/2)) "" "$txt"
+}
+
+timer_reset() {
+ t0=$SECONDS
+}
+
+timer() {
+ echo -n "$(($SECONDS - $t0))s"
+}
+
+source /etc/profile.d/rvm.sh
+echo $WORKSPACE
+
+title "Starting diagnostics"
+timer_reset
+
+cd $WORKSPACE
+
+cp -f /home/jenkins/diagnostics/arvados-workbench/application.yml $WORKSPACE/apps/workbench/config/
+
+cd $WORKSPACE/apps/workbench
+
+HOME="$GEMHOME" bundle install --no-deployment
+
+if [[ ! -d tmp ]]; then
+ mkdir tmp
+fi
+
+RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! DIAGNOSTICS FAILED (`timer`) !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ exit $EXITCODE
+fi
+
+title "Diagnostics complete (`timer`)"
+
+exit $EXITCODE
commit 0869894ffaad8b028c4775bafb853535d1183689
Merge: 60dbf9f 88474d0
Author: Tim Pierce <twp at unchi.org>
Date: Tue Sep 30 13:27:41 2014 -0400
Merge branch '4038-commit-hooks-accept-fixes'
Closes #4038.
commit 88474d0ad55e159de59e35a74cf9686d3e641f92
Author: Tim Pierce <twp at unchi.org>
Date: Tue Sep 30 13:10:32 2014 -0400
4038: add "fixes #" to acceptable merge commit messages
diff --git a/git/hooks/coding-standards.sh b/git/hooks/coding-standards.sh
index bc75a58..79b0cdd 100755
--- a/git/hooks/coding-standards.sh
+++ b/git/hooks/coding-standards.sh
@@ -24,7 +24,7 @@ $broken_commit_message = /Please enter a commit message to explain why this merg
$wrong_way_merge_master = /Merge( remote-tracking)? branch '([^\/]+\/)?master' into/
$merge_master = /Merge branch '[^']+'((?! into)| into master)/
$pull_merge = /Merge branch 'master' of /
-$refs_or_closes_or_no_issue = /(refs #|closes #|no issue #)/i
+$refs_or_closes_or_no_issue = /(refs #|closes #|fixes #|no issue #)/i
# enforced custom commit message format
def check_message_format
commit 60dbf9f65de31341aae26c1804e77be6bf268dd0
Author: Brett Smith <brettcsmith at brettcsmith.org>
Date: Mon Sep 29 12:25:53 2014 -0400
3848: Stop manually tagging Python builds.
This is now done in setup.py. Refs #3848.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 2c25bb7..96fc4b7 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -275,14 +275,14 @@ install_python_sdk() {
# keepproxy).
cd "$WORKSPACE/sdk/python" \
- && python setup.py egg_info -b ".$(git log --first-parent --format=format:%ci.%h -n1 . | tr -dc 0-9a-f. | cut -c 1-14,19-)" sdist rotate --keep=1 --match .tar.gz \
+ && python setup.py sdist rotate --keep=1 --match .tar.gz \
&& pip install dist/arvados-python-client-0.1.*.tar.gz
}
do_install python_sdk
install_fuse() {
cd "$WORKSPACE/services/fuse" \
- && python setup.py egg_info -b ".$(git log --first-parent --format=format:%ci.%h -n1 . | tr -dc 0-9a-f. | cut -c 1-14,19-)" sdist rotate --keep=1 --match .tar.gz \
+ && python setup.py sdist rotate --keep=1 --match .tar.gz \
&& pip install dist/arvados_fuse-0.1.*.tar.gz
}
do_install fuse
commit ca3061e04ae8556d3b66fb3391dc309df3c80010
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Sep 26 21:40:08 2014 -0400
Improvements to our package building script: fill in more of the
metadata fields on the deb packages.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index ce9b8f7..56a43b1 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -16,6 +16,12 @@ if [[ "$APTSERVER" == '' ]]; then
exit 1
fi
+# Sanity check
+if ! [[ -n "$WORKSPACE" ]]; then
+ echo "WORKSPACE environment variable not set"
+ exit 1
+fi
+
source /etc/profile.d/rvm.sh
echo $WORKSPACE
@@ -67,13 +73,15 @@ python setup.py egg_info -b ".$GIT_HASH" sdist upload
build_and_scp_deb () {
PACKAGE=$1
- PACKAGE_NAME=$2
- # Put spaces in $3 and you will regret it. Despite the use of arrays below.
- # Because, bash sucks.
- VENDOR=${3// /_}
- PACKAGE_TYPE=$4
- VERSION=$5
- EXTRA_ARGUMENTS=$6
+ shift
+ PACKAGE_NAME=$1
+ shift
+ VENDOR=$1
+ shift
+ PACKAGE_TYPE=$1
+ shift
+ VERSION=$1
+ shift
if [[ "$PACKAGE_NAME" == "" ]]; then
PACKAGE_NAME=$PACKAGE
@@ -83,7 +91,7 @@ build_and_scp_deb () {
PACKAGE_TYPE='python'
fi
- COMMAND_ARR=("fpm" "-s" "$PACKAGE_TYPE" "-t" "deb")
+ declare -a COMMAND_ARR=("fpm" "--maintainer=Ward Vandewege <ward at curoverse.com>" "-s" "$PACKAGE_TYPE" "-t" "deb")
if [[ "$PACKAGE_NAME" != "$PACKAGE" ]]; then
COMMAND_ARR+=('-n' "$PACKAGE_NAME")
@@ -97,15 +105,14 @@ build_and_scp_deb () {
COMMAND_ARR+=('-v' "$VERSION")
fi
- for a in $EXTRA_ARGUMENTS; do
- COMMAND_ARR+=("$a")
+ for i; do
+ COMMAND_ARR+=("$i")
done
COMMAND_ARR+=("$PACKAGE")
- FPM_RESULTS=$(${COMMAND_ARR[@]})
+ FPM_RESULTS=$("${COMMAND_ARR[@]}")
FPM_EXIT_CODE=$?
- echo ${COMMAND_ARR[@]}
FPM_PACKAGE_NAME=''
if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\-.]*\.deb) ]]; then
@@ -150,7 +157,7 @@ git checkout `git log --format=format:%h -n1 .`
cd $WORKSPACE
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "-x 'usr/local/arvados/src/.git*'"
+build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "-x 'usr/local/arvados/src/.git*'" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
# clean up, check out master and step away from detached-head state
cd "$WORKSPACE/src-build-dir"
@@ -164,17 +171,17 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
# keepstore
go get "git.curoverse.com/arvados.git/services/keepstore"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH"
+build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepstore is the Keep storage daemon, accessible to clients on the LAN"
# keepproxy
go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH"
+build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Keepproxy makes a Keep cluster accessible to clients that are not on the LAN"
# crunchstat
go get "git.curoverse.com/arvados.git/services/crunchstat"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH"
+build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=Crunchstat gathers cpu/memory/network statistics of running Crunch jobs"
# The Python SDK
cd $WORKSPACE/sdk/python
@@ -188,7 +195,7 @@ cd $WORKSPACE/debs
# prefix from only one of the dependencies of a package... Maybe I could
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
-build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "0.1.${GIT_HASH}"
+build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "0.1.${GIT_HASH}" "--url=https://arvados.org" "--description=The Arvados Python SDK"
# The FUSE driver
cd $WORKSPACE/services/fuse
@@ -198,7 +205,7 @@ cd $WORKSPACE/debs
# Please seem comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
-build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "0.1.${GIT_HASH}"
+build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "0.1.${GIT_HASH}" "--url=https://arvados.org" "--description=The Keep FUSE driver"
# A few dependencies
build_and_scp_deb python-gflags
commit 56508d6b4085d51c640df635ebfa64ca824cdb94
Author: Brett Smith <brett at curoverse.com>
Date: Tue Sep 23 10:56:57 2014 -0400
run-tests.sh uses virtualenv's activate script.
The activate script takes care of additional isolation steps, like
clearing $PYTHONHOME and rehashing $PATH. Prefer that over simply
setting $PATH ourselves. No issue #.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 45cdadc..2c25bb7 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -181,7 +181,7 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
|| fatal "symlink failed"
virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
-PATH="$VENVDIR/bin:$PATH"
+. "$VENVDIR/bin/activate"
checkexit() {
if [[ "$?" != "0" ]]; then
commit fef361d38a31b29fdda31e8a5eda13c908c04966
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Sep 23 10:21:30 2014 -0400
Fix another -z that should have been -n. refs #3894
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index ee4a9c4..45cdadc 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -88,7 +88,7 @@ clear_temp() {
leaving+=" $var=\"${!var}\""
fi
done
- if [[ -z "$leaving" ]]; then
+ if [[ -n "$leaving" ]]; then
echo "Leaving behind temp dirs: $leaving"
fi
}
commit b25bb303ff9c56f3b4349ba1e1cab399acf404f5
Merge: d9ef4f2 8e51498
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Sep 22 16:34:22 2014 -0400
Merge branch '3894-improve-run-tests' closes #3894
commit 8e51498f431f233a38da76c169ea2e897e91f051
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Sep 22 15:40:00 2014 -0400
3894: Add comments about CONFIGSRC in --help text.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 2b52830..ee4a9c4 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -20,6 +20,7 @@ Options:
from a previous invocation if you use this option.
WORKSPACE=path Arvados source tree to test.
CONFIGSRC=path Dir with api server config files to copy into source tree.
+ (If none given, leave config files alone in source tree.)
apiserver_test="TEST=test/functional/arvados/v1/collections_test.rb"
Restrict apiserver tests to the given file
python_sdk_test="--test-suite test.test_keep_locator"
@@ -36,6 +37,9 @@ into \$GOPATH, \$VENDIR, and \$GEMHOME before running any tests. Many
test suites depend on other components being installed, and installing
everything tends to be quicker than debugging dependencies.
+As a special concession to the current CI server config, CONFIGSRC
+defaults to $HOME/arvados-api-server if that directory exists.
+
More information and background:
https://arvados.org/projects/arvados/wiki/Running_tests
commit 63517baef2f2e2d2e1ea10d4952189ef377da635
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Sep 22 15:37:03 2014 -0400
3894: Fix -n that should have been -z. Flatten condition.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 742baeb..2b52830 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -156,11 +156,9 @@ done
echo "WORKSPACE=$WORKSPACE"
[[ -n "$WORKSPACE" ]] || fatal "WORKSPACE not set"
-if [[ -n "$CONFIGSRC" ]]; then
- if [[ -d "$HOME/arvados-api-server" ]]; then
- # Jenkins expects us to use this by default.
- CONFIGSRC="$HOME/arvados-api-server"
- fi
+if [[ -z "$CONFIGSRC" ]] && [[ -d "$HOME/arvados-api-server" ]]; then
+ # Jenkins expects us to use this by default.
+ CONFIGSRC="$HOME/arvados-api-server"
fi
# Set up temporary install dirs (unless existing dirs were supplied)
commit ee10e6bbf12c994dd6335560acac2327392ba3b0
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Sep 22 15:32:50 2014 -0400
3894: Use --first-parent and readable timestamp in fuse package as
well. Use "tr -dc | cut" to eliminate the horrible list of byte
positions. Fix git command to use the desired human-readable timestamp
format.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 612f208..742baeb 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -273,14 +273,14 @@ install_python_sdk() {
# keepproxy).
cd "$WORKSPACE/sdk/python" \
- && python setup.py egg_info -b ".$(git log --first-parent --format=format:%ct.%h -n1 . | cut -c 1-4,6-7,9-10,12-13,15-16,18-19,26-33)" sdist rotate --keep=1 --match .tar.gz \
+ && python setup.py egg_info -b ".$(git log --first-parent --format=format:%ci.%h -n1 . | tr -dc 0-9a-f. | cut -c 1-14,19-)" sdist rotate --keep=1 --match .tar.gz \
&& pip install dist/arvados-python-client-0.1.*.tar.gz
}
do_install python_sdk
install_fuse() {
cd "$WORKSPACE/services/fuse" \
- && python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz \
+ && python setup.py egg_info -b ".$(git log --first-parent --format=format:%ci.%h -n1 . | tr -dc 0-9a-f. | cut -c 1-14,19-)" sdist rotate --keep=1 --match .tar.gz \
&& pip install dist/arvados_fuse-0.1.*.tar.gz
}
do_install fuse
commit 07b0b88d99a07be66b564d7d8cd76fbaf27e5909
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Sep 22 11:07:13 2014 -0400
3894: Use readable timestamp (yyyymmddhhmmss) for python package
version, and use first-parent to ensure each merge to master results
in a higher version number.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 5830565..612f208 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -273,7 +273,7 @@ install_python_sdk() {
# keepproxy).
cd "$WORKSPACE/sdk/python" \
- && python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz \
+ && python setup.py egg_info -b ".$(git log --first-parent --format=format:%ct.%h -n1 . | cut -c 1-4,6-7,9-10,12-13,15-16,18-19,26-33)" sdist rotate --keep=1 --match .tar.gz \
&& pip install dist/arvados-python-client-0.1.*.tar.gz
}
do_install python_sdk
commit e4f90e4d14a89ff08ba654333d316cd3e5e97225
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Sep 22 11:05:39 2014 -0400
3894: Remove bundle install stage from sdk/cli.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 1db3857..5830565 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -260,7 +260,6 @@ do_install ruby_sdk
install_cli() {
cd "$WORKSPACE/sdk/cli" \
- && HOME="$GEMHOME" bundle install --no-deployment \
&& gem build arvados-cli.gemspec \
&& HOME="$GEMHOME" gem install --user-install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
commit 0b16f3e4e4beccf000f04f63129276e0716c141c
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Sep 22 11:04:08 2014 -0400
3894: Move report_outcomes to a function.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 680ae0a..1db3857 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -95,6 +95,23 @@ fatal() {
exit 1
}
+report_outcomes() {
+ for x in "${successes[@]}"
+ do
+ echo "Pass: $x"
+ done
+
+ if [[ ${#failures[@]} == 0 ]]
+ then
+ echo "All test suites passed."
+ else
+ echo "Failures (${#failures[@]}):"
+ for x in "${failures[@]}"
+ do
+ echo "Fail: $x"
+ done
+ fi
+}
declare -a failures
declare -A skip
@@ -395,21 +412,7 @@ test_workbench() {
}
do_test workbench
+report_outcomes
clear_temp
-for x in "${successes[@]}"
-do
- echo "Pass: $x"
-done
-
-if [[ ${#failures[@]} == 0 ]]
-then
- echo "All test suites passed."
-else
- echo "Failures (${#failures[@]}):"
- for x in "${failures[@]}"
- do
- echo "Fail: $x"
- done
-fi
exit ${#failures}
commit 73b46d8b02451bb782f7262253778593181fd02d
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Sep 19 23:39:11 2014 -0400
3894: Use $GEMHOME tmpdir instead of user's $HOME for userspace gem installs.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 852bd43..680ae0a 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -45,12 +45,16 @@ EOF
# environment that could interfere with the tests.
unset $(env | cut -d= -f1 | grep \^ARVADOS_)
-COLUMNS=80
-
+# Reset other variables that could affect our [tests'] behavior by
+# accident.
GITDIR=
GOPATH=
VENVDIR=
PYTHONPATH=
+GEMHOME=
+
+COLUMNS=80
+
cli_test=
workbench_test=
apiserver_test=
@@ -68,7 +72,7 @@ fi
declare -A leave_temp
clear_temp() {
leaving=""
- for var in VENVDIR GOPATH GITDIR
+ for var in VENVDIR GOPATH GITDIR GEMHOME
do
if [[ -z "${leave_temp[$var]}" ]]
then
@@ -101,15 +105,15 @@ while [[ -n "$1" ]]
do
arg="$1"; shift
case "$arg" in
- --skip)
- skipwhat="$1"; shift
- skip[$skipwhat]=1
- ;;
--help)
echo >&2 "$helpmessage"
echo >&2
exit 1
;;
+ --skip)
+ skipwhat="$1"; shift
+ skip[$skipwhat]=1
+ ;;
--only)
only="$1"; shift
;;
@@ -119,6 +123,7 @@ do
--leave-temp)
leave_temp[VENVDIR]=1
leave_temp[GOPATH]=1
+ leave_temp[GEMHOME]=1
;;
*=*)
eval $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2-)\"
@@ -142,16 +147,15 @@ if [[ -n "$CONFIGSRC" ]]; then
fi
# Set up temporary install dirs (unless existing dirs were supplied)
-if [[ -n "$VENVDIR" ]]; then
- leave_temp[VENVDIR]=1
-else
- VENVDIR=$(mktemp -d)
-fi
-if [[ -n "$GOPATH" ]]; then
- leave_temp[GOPATH]=1
-else
- GOPATH=$(mktemp -d)
-fi
+for tmpdir in VENVDIR GOPATH GEMHOME
+do
+ if [[ -n "${!tmpdir}" ]]; then
+ leave_temp[$tmpdir]=1
+ else
+ eval $tmpdir=$(mktemp -d)
+ fi
+done
+PATH="$GEMHOME/.gem/ruby/2.1.0/bin:$PATH"
export GOPATH
mkdir -p "$GOPATH/src/git.curoverse.com"
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
@@ -220,26 +224,28 @@ title () {
install_docs() {
cd "$WORKSPACE/doc"
- bundle install --no-deployment
+ HOME="$GEMHOME" bundle install --no-deployment
rm -rf .site
# Make sure python-epydoc is installed or the next line won't do much good!
ARVADOS_API_HOST=qr1hi.arvadosapi.com
- PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
+ PYTHONPATH=$WORKSPACE/sdk/python/ HOME="$GEMHOME" bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
unset ARVADOS_API_HOST
}
do_install docs
install_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
+ && HOME="$GEMHOME" bundle install --no-deployment \
&& gem build arvados.gemspec \
- && gem install --user-install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
+ && HOME="$GEMHOME" gem install --user-install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
do_install ruby_sdk
install_cli() {
cd "$WORKSPACE/sdk/cli" \
+ && HOME="$GEMHOME" bundle install --no-deployment \
&& gem build arvados-cli.gemspec \
- && gem install --user-install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
+ && HOME="$GEMHOME" gem install --user-install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
do_install cli
@@ -265,7 +271,8 @@ do_install fuse
install_apiserver() {
cd "$WORKSPACE/services/api"
- bundle install --no-deployment
+ export RAILS_ENV=test
+ HOME="$GEMHOME" bundle install --no-deployment
rm -f config/environments/test.rb
cp config/environments/test.rb.example config/environments/test.rb
@@ -285,8 +292,6 @@ install_apiserver() {
sed -i'' -e "s:SECRET_TOKEN:$SECRET_TOKEN:" config/application.yml
sed -i'' -e "s:BLOB_SIGNING_KEY:$BLOB_SIGNING_KEY:" config/application.yml
- export RAILS_ENV=test
-
# Set up empty git repo (for git tests)
GITDIR=$(mktemp -d)
sed -i'' -e "s:/var/cache/git:$GITDIR:" config/application.default.yml
@@ -306,9 +311,9 @@ install_apiserver() {
psql arvados_test -c "SELECT pg_terminate_backend (pg_stat_activity.procpid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'arvados_test';" 2>/dev/null
cd "$WORKSPACE/services/api" \
- && bundle exec rake db:drop \
- && bundle exec rake db:create \
- && bundle exec rake db:setup
+ && HOME="$GEMHOME" bundle exec rake db:drop \
+ && HOME="$GEMHOME" bundle exec rake db:create \
+ && HOME="$GEMHOME" bundle exec rake db:setup
}
do_install apiserver
@@ -327,29 +332,29 @@ done
test_doclinkchecker() {
cd "$WORKSPACE/doc"
- bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
+ HOME="$GEMHOME" bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
}
do_test doclinkchecker
test_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
- && bundle install --no-deployment \
- && bundle exec rake test
+ && HOME="$GEMHOME" bundle install --no-deployment \
+ && HOME="$GEMHOME" bundle exec rake test
}
do_test ruby_sdk
test_cli() {
title "Starting SDK CLI tests"
cd "$WORKSPACE/sdk/cli" \
- && bundle install --no-deployment \
+ && HOME="$GEMHOME" bundle install --no-deployment \
&& mkdir -p /tmp/keep \
- && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test $cli_test
+ && KEEP_LOCAL_STORE=/tmp/keep HOME="$GEMHOME" bundle exec rake test $cli_test
}
do_test cli
test_apiserver() {
cd "$WORKSPACE/services/api"
- bundle exec rake test $apiserver_test
+ HOME="$GEMHOME" bundle exec rake test $apiserver_test
}
do_test apiserver
@@ -385,8 +390,8 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
- && bundle install --no-deployment \
- && bundle exec rake test $workbench_test
+ && HOME="$GEMHOME" bundle install --no-deployment \
+ && HOME="$GEMHOME" bundle exec rake test $workbench_test
}
do_test workbench
commit 1b30b40e3b83876ef1d96bcbffb9356dce94c5b7
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Sep 19 23:35:30 2014 -0400
3894: Add --help message.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 9fe3218..852bd43 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -1,49 +1,48 @@
#!/bin/bash
-# Install and test Arvados components.
-#
-# Exit non-zero if any tests fail.
-#
-# Arguments:
-# --skip FOO Do not test the FOO component.
-# --only FOO Do not test anything except the FOO component.
-# WORKSPACE=path Arvados source tree to test.
-# CONFIGSRC=path Dir with api server config files to copy into source tree.
-# envvar=value Set $envvar to value
-#
-# Regardless of which components are tested, install all components in
-# the usual sequence. (Many test suites depend on other components
-# being installed.)
-#
-# To run a specific Ruby test, set $workbench_test, $apiserver_test or
-# $cli_test on the command line:
-#
-# $ run-tests.sh --only workbench workbench_test=TEST=test/integration/pipeline_instances_test.rb
-#
-#
-# To run a specific Python test set $python_sdk_test or $fuse_test.
-#
-# $ run-tests.sh --only python_sdk python_sdk_test="--test-suite tests.test_keep_locator"
-#
-#
-# You can also pass "export ARVADOS_DEBUG=1" to enable additional debugging output:
-#
-# $ run-tests.sh "export ARVADOS_DEBUG=1"
-#
-#
-# Finally, you can skip the installation steps on subsequent runs this way:
-#
-## First run
-# $ run-tests.sh --leave-temp
-#
-## Subsequent runs: record the values of VENVDIR and GOPATH from the first run, and
-# provide them on the command line in subsequent runs:
-#
-# $ run-tests.sh --skip-install VENVDIR="/tmp/tmp.y3tsTmigio" GOPATH="/tmp/tmp.3r4sSA9F3l"
-
-
-# First make sure to remove any ARVADOS_ variables from the calling environment
-# that could interfer with the tests.
+read -rd "\000" helpmessage <<EOF
+$(basename $0): Install and test Arvados components.
+
+Exit non-zero if any tests fail.
+
+Syntax:
+ $(basename $0) WORKSPACE=/path/to/arvados [options]
+
+Options:
+
+--skip FOO Do not test the FOO component.
+--only FOO Do not test anything except the FOO component.
+--leave-temp Do not remove GOPATH, virtualenv, and other temp dirs at exit.
+ Instead, show which directories were used this time so they
+ can be reused in subsequent invocations.
+--skip-install Do not run any install steps. Just run tests.
+ You should provide GOPATH, GEMHOME, and VENVDIR options
+ from a previous invocation if you use this option.
+WORKSPACE=path Arvados source tree to test.
+CONFIGSRC=path Dir with api server config files to copy into source tree.
+apiserver_test="TEST=test/functional/arvados/v1/collections_test.rb"
+ Restrict apiserver tests to the given file
+python_sdk_test="--test-suite test.test_keep_locator"
+ Restrict Python SDK tests to the given class
+workbench_test="TEST=test/integration/pipeline_instances_test.rb"
+ Restrict Workbench tests to the given file
+ARVADOS_DEBUG=1
+ Print more debug messages
+envvar=value Set \$envvar to value. Primarily useful for WORKSPACE,
+ *_test, and other examples shown above.
+
+Assuming --skip-install is not given, all components are installed
+into \$GOPATH, \$VENDIR, and \$GEMHOME before running any tests. Many
+test suites depend on other components being installed, and installing
+everything tends to be quicker than debugging dependencies.
+
+More information and background:
+
+https://arvados.org/projects/arvados/wiki/Running_tests
+EOF
+
+# First make sure to remove any ARVADOS_ variables from the calling
+# environment that could interfere with the tests.
unset $(env | cut -d= -f1 | grep \^ARVADOS_)
COLUMNS=80
@@ -106,6 +105,11 @@ do
skipwhat="$1"; shift
skip[$skipwhat]=1
;;
+ --help)
+ echo >&2 "$helpmessage"
+ echo >&2
+ exit 1
+ ;;
--only)
only="$1"; shift
;;
@@ -120,7 +124,7 @@ do
eval $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2-)\"
;;
*)
- echo >&2 "$0: Unrecognized option: '$arg'"
+ echo >&2 "$0: Unrecognized option: '$arg'. Try: $0 --help"
exit 1
;;
esac
commit d9ef4f277d847a62e17f944cd42a892eb00aab67
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Sep 19 16:46:10 2014 -0400
Make regexp to match fpm's output slightly more robust.
no issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 9844974..ce9b8f7 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -108,7 +108,7 @@ build_and_scp_deb () {
echo ${COMMAND_ARR[@]}
FPM_PACKAGE_NAME=''
- if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\-.]*.deb) ]]; then
+ if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\-.]*\.deb) ]]; then
FPM_PACKAGE_NAME=${BASH_REMATCH[1]}
fi
commit 68d7d4f2b6fce5f755f4ccecc978a8b16d73c802
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Sep 19 16:23:16 2014 -0400
Switch from PRM to freight for publishing our packages. It's way, way
faster.
no issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 2124adc..9844974 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -1,7 +1,7 @@
#!/bin/bash
EXITCODE=0
-CALL_PRM=0
+CALL_FREIGHT=0
APTUSER=$1
APTSERVER=$2
@@ -106,15 +106,26 @@ build_and_scp_deb () {
FPM_RESULTS=$(${COMMAND_ARR[@]})
FPM_EXIT_CODE=$?
echo ${COMMAND_ARR[@]}
- if [[ ! $FPM_RESULTS =~ "File already exists" ]]; then
- if [[ "$FPM_EXIT_CODE" != "0" ]]; then
- echo "Error building debian package for $1:\n $FPM_RESULTS"
+
+ FPM_PACKAGE_NAME=''
+ if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\-.]*.deb) ]]; then
+ FPM_PACKAGE_NAME=${BASH_REMATCH[1]}
+ fi
+
+ if [[ "$FPM_PACKAGE_NAME" == "" ]]; then
+ EXITCODE=1
+ echo "Error: Unabled figure out package name from fpm results:\n $FPM_RESULTS"
+ else
+ if [[ ! $FPM_RESULTS =~ "File already exists" ]]; then
+ if [[ "$FPM_EXIT_CODE" != "0" ]]; then
+ echo "Error building debian package for $1:\n $FPM_RESULTS"
+ else
+ scp -P2222 $FPM_PACKAGE_NAME $APTUSER@$APTSERVER:tmp/
+ CALL_FREIGHT=1
+ fi
else
- scp -P2222 "$PACKAGE_NAME"_"$VERSION"*.deb $APTUSER@$APTSERVER:tmp/
- CALL_PRM=1
+ echo "Debian package $FPM_PACKAGE_NAME exists, not rebuilding"
fi
- else
- echo "Debian package for $1 exists, not rebuilding"
fi
}
@@ -122,9 +133,6 @@ if [[ ! -d "$WORKSPACE/debs" ]]; then
mkdir -p $WORKSPACE/debs
fi
-# Make sure our destination directory on $APTSERVER exists - prm can delete it when invoked improperly
-ssh -p2222 $APTUSER@$APTSERVER mkdir tmp
-
# Arvados-src
# We use $WORKSPACE/src-build-dir as the clean directory from which to build the src package
if [[ ! -d "$WORKSPACE/src-build-dir" ]]; then
@@ -153,14 +161,12 @@ export GOPATH=$(mktemp -d)
mkdir -p "$GOPATH/src/git.curoverse.com"
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
-# Keep -> keepstore
+# keepstore
go get "git.curoverse.com/arvados.git/services/keepstore"
cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH"
-# Keep proxy
-
-# Keep -> keepproxy
+# keepproxy
go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/debs
build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH"
@@ -203,11 +209,13 @@ build_and_scp_deb ws4py
build_and_scp_deb virtualenv
# Finally, publish the packages, if necessary
-if [[ "$CALL_PRM" != "0" ]]; then
- ssh -p2222 $APTUSER@$APTSERVER -t "cd /var/www/$APTSERVER; /usr/local/rvm/bin/rvm default do prm --type deb -p . --component main --release wheezy --arch amd64 -d /home/$APTUSER/tmp/ --gpg 1078ECD7"
+if [[ "$CALL_FREIGHT" != "0" ]]; then
+ ssh -p2222 $APTUSER@$APTSERVER -t "cd tmp && ls -laF *deb && freight add *deb apt/wheezy && freight cache && rm -f *deb"
else
- echo "No new packages generated. No PRM run necessary."
+ echo "No new packages generated. No freight run necessary."
fi
# clean up temporary GOPATH
rm -rf "$GOPATH"
+
+exit $EXITCODE
commit eb8d11d83e21d6554dd69784254898c7454a2727
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Sep 19 12:05:41 2014 -0400
Copy only the debs that were generated to the remote server.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 4a39391..2124adc 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -72,7 +72,8 @@ build_and_scp_deb () {
# Because, bash sucks.
VENDOR=${3// /_}
PACKAGE_TYPE=$4
- EXTRA_ARGUMENTS=$5
+ VERSION=$5
+ EXTRA_ARGUMENTS=$6
if [[ "$PACKAGE_NAME" == "" ]]; then
PACKAGE_NAME=$PACKAGE
@@ -91,6 +92,11 @@ build_and_scp_deb () {
if [[ "$VENDOR" != "" ]]; then
COMMAND_ARR+=('--vendor' "$VENDOR")
fi
+
+ if [[ "$VERSION" != "" ]]; then
+ COMMAND_ARR+=('-v' "$VERSION")
+ fi
+
for a in $EXTRA_ARGUMENTS; do
COMMAND_ARR+=("$a")
done
@@ -104,7 +110,7 @@ build_and_scp_deb () {
if [[ "$FPM_EXIT_CODE" != "0" ]]; then
echo "Error building debian package for $1:\n $FPM_RESULTS"
else
- scp -P2222 $PACKAGE_NAME*.deb $APTUSER@$APTSERVER:tmp/
+ scp -P2222 "$PACKAGE_NAME"_"$VERSION"*.deb $APTUSER@$APTSERVER:tmp/
CALL_PRM=1
fi
else
@@ -136,7 +142,7 @@ git checkout `git log --format=format:%h -n1 .`
cd $WORKSPACE
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH -x 'usr/local/arvados/src/.git*'"
+build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH" "-x 'usr/local/arvados/src/.git*'"
# clean up, check out master and step away from detached-head state
cd "$WORKSPACE/src-build-dir"
@@ -150,19 +156,19 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
# Keep -> keepstore
go get "git.curoverse.com/arvados.git/services/keepstore"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH"
# Keep proxy
# Keep -> keepproxy
go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH"
# crunchstat
go get "git.curoverse.com/arvados.git/services/crunchstat"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "0.1.$GIT_HASH"
# The Python SDK
cd $WORKSPACE/sdk/python
@@ -176,7 +182,7 @@ cd $WORKSPACE/debs
# prefix from only one of the dependencies of a package... Maybe I could
# whip up a patch and send it upstream, but that will be for another day. Ward,
# 2014-05-15
-build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "-v 0.1.${GIT_HASH}"
+build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "0.1.${GIT_HASH}"
# The FUSE driver
cd $WORKSPACE/services/fuse
@@ -186,7 +192,7 @@ cd $WORKSPACE/debs
# Please seem comment about --no-python-fix-name above; we stay consistent and do
# not omit the python- prefix first.
-build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "-v 0.1.${GIT_HASH}"
+build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "0.1.${GIT_HASH}"
# A few dependencies
build_and_scp_deb python-gflags
commit 0f9882d559fc996a260fb6e00254fd9d5f7de218
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Sep 19 10:57:20 2014 -0400
3894: Fix sequence: do all installs, then all tests.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 44bb603..9fe3218 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -214,7 +214,7 @@ title () {
printf "\n%*s%s\n\n" $((($COLUMNS-${#txt})/2)) "" "$txt"
}
-test_docs() {
+install_docs() {
cd "$WORKSPACE/doc"
bundle install --no-deployment
rm -rf .site
@@ -223,20 +223,7 @@ test_docs() {
PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
unset ARVADOS_API_HOST
}
-do_test docs
-
-test_doclinkchecker() {
- cd "$WORKSPACE/doc"
- bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
-}
-do_test doclinkchecker
-
-test_ruby_sdk() {
- cd "$WORKSPACE/sdk/ruby" \
- && bundle install --no-deployment \
- && bundle exec rake test
-}
-do_test ruby_sdk
+do_install docs
install_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
@@ -252,14 +239,25 @@ install_cli() {
}
do_install cli
-test_cli() {
- title "Starting SDK CLI tests"
- cd "$WORKSPACE/sdk/cli" \
- && bundle install --no-deployment \
- && mkdir -p /tmp/keep \
- && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test $cli_test
+install_python_sdk() {
+ # Install the Python SDK early. Various other test suites (like
+ # keepproxy) bring up run_test_server.py, which imports the arvados
+ # module. We can't actually *test* the Python SDK yet though, because
+ # its own test suite brings up some of those other programs (like
+ # keepproxy).
+
+ cd "$WORKSPACE/sdk/python" \
+ && python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz \
+ && pip install dist/arvados-python-client-0.1.*.tar.gz
}
-do_test cli
+do_install python_sdk
+
+install_fuse() {
+ cd "$WORKSPACE/services/fuse" \
+ && python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz \
+ && pip install dist/arvados_fuse-0.1.*.tar.gz
+}
+do_install fuse
install_apiserver() {
cd "$WORKSPACE/services/api"
@@ -310,12 +308,6 @@ install_apiserver() {
}
do_install apiserver
-test_apiserver() {
- cd "$WORKSPACE/services/api"
- bundle exec rake test $apiserver_test
-}
-do_test apiserver
-
declare -a gostuff
gostuff=(
services/keepstore
@@ -329,25 +321,33 @@ do
do_install "$g" go
done
-install_python_sdk() {
- # Install the Python SDK early. Various other test suites (like
- # keepproxy) bring up run_test_server.py, which imports the arvados
- # module. We can't actually *test* the Python SDK yet though, because
- # its own test suite brings up some of those other programs (like
- # keepproxy).
+test_doclinkchecker() {
+ cd "$WORKSPACE/doc"
+ bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
+}
+do_test doclinkchecker
- cd "$WORKSPACE/sdk/python" \
- && python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz \
- && pip install dist/arvados-python-client-0.1.*.tar.gz
+test_ruby_sdk() {
+ cd "$WORKSPACE/sdk/ruby" \
+ && bundle install --no-deployment \
+ && bundle exec rake test
}
-do_install python_sdk
+do_test ruby_sdk
-install_fuse() {
- cd "$WORKSPACE/services/fuse" \
- && python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz \
- && pip install dist/arvados_fuse-0.1.*.tar.gz
+test_cli() {
+ title "Starting SDK CLI tests"
+ cd "$WORKSPACE/sdk/cli" \
+ && bundle install --no-deployment \
+ && mkdir -p /tmp/keep \
+ && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test $cli_test
}
-do_install fuse
+do_test cli
+
+test_apiserver() {
+ cd "$WORKSPACE/services/api"
+ bundle exec rake test $apiserver_test
+}
+do_test apiserver
test_python_sdk() {
# Python SDK. We test this before testing keepproxy: keepproxy runs
commit 254c4352cf3f0fba7861bd51b42498e996036227
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Sep 19 10:56:27 2014 -0400
3894: Use gem install --user-install to avoid sudo/rvm requirement
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index d6fc40a..44bb603 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -241,14 +241,14 @@ do_test ruby_sdk
install_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
&& gem build arvados.gemspec \
- && gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
+ && gem install --user-install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
do_install ruby_sdk
install_cli() {
cd "$WORKSPACE/sdk/cli" \
&& gem build arvados-cli.gemspec \
- && gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
+ && gem install --user-install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
do_install cli
commit cf97dfcc0a04149057688f302252d696c2d575fa
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Sep 19 10:54:11 2014 -0400
3894: Scrub PYTHONPATH
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index f2cd322..d6fc40a 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -51,6 +51,7 @@ COLUMNS=80
GITDIR=
GOPATH=
VENVDIR=
+PYTHONPATH=
cli_test=
workbench_test=
apiserver_test=
commit 12a671a65b2cc1ec7431f8cd8b024cf70f269f7a
Author: Tom Clegg <tom at curoverse.com>
Date: Fri Sep 19 10:26:53 2014 -0400
3894: Use mktemp instead of $WORKSPACE/tmpgit for test repo
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 54c378b..f2cd322 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -48,6 +48,7 @@ unset $(env | cut -d= -f1 | grep \^ARVADOS_)
COLUMNS=80
+GITDIR=
GOPATH=
VENVDIR=
cli_test=
@@ -67,7 +68,7 @@ fi
declare -A leave_temp
clear_temp() {
leaving=""
- for var in VENVDIR GOPATH
+ for var in VENVDIR GOPATH GITDIR
do
if [[ -z "${leave_temp[$var]}" ]]
then
@@ -284,7 +285,7 @@ install_apiserver() {
export RAILS_ENV=test
# Set up empty git repo (for git tests)
- GITDIR="$WORKSPACE/tmpgit"
+ GITDIR=$(mktemp -d)
sed -i'' -e "s:/var/cache/git:$GITDIR:" config/application.default.yml
rm -rf $GITDIR
commit 8c67902ccd67de79c0c86c85116c46b5dcfd82bb
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Sep 18 17:00:58 2014 -0400
3894: Fix backward empty/nonempty logic.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index f63267e..54c378b 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -64,6 +64,26 @@ then
source /etc/profile.d/rvm.sh
fi
+declare -A leave_temp
+clear_temp() {
+ leaving=""
+ for var in VENVDIR GOPATH
+ do
+ if [[ -z "${leave_temp[$var]}" ]]
+ then
+ if [[ -n "${!var}" ]]
+ then
+ rm -rf "${!var}"
+ fi
+ else
+ leaving+=" $var=\"${!var}\""
+ fi
+ done
+ if [[ -z "$leaving" ]]; then
+ echo "Leaving behind temp dirs: $leaving"
+ fi
+}
+
fatal() {
clear_temp
echo >&2 "Fatal: $* in ${FUNCNAME[1]} at ${BASH_SOURCE[1]} line ${BASH_LINENO[0]}"
@@ -72,7 +92,6 @@ fatal() {
declare -a failures
declare -A skip
-declare -A leave_temp
# Always skip CLI tests. They don't know how to use run_test_server.py.
skip[cli]=1
@@ -118,14 +137,14 @@ fi
# Set up temporary install dirs (unless existing dirs were supplied)
if [[ -n "$VENVDIR" ]]; then
- VENVDIR=$(mktemp -d)
-else
leave_temp[VENVDIR]=1
+else
+ VENVDIR=$(mktemp -d)
fi
if [[ -n "$GOPATH" ]]; then
- GOPATH=$(mktemp -d)
-else
leave_temp[GOPATH]=1
+else
+ GOPATH=$(mktemp -d)
fi
export GOPATH
mkdir -p "$GOPATH/src/git.curoverse.com"
@@ -193,21 +212,6 @@ title () {
printf "\n%*s%s\n\n" $((($COLUMNS-${#txt})/2)) "" "$txt"
}
-clear_temp() {
- for var in VENVDIR GOPATH
- do
- if [[ -z "${leave_temp[$var]}" ]]
- then
- if [[ -n "${!var}" ]]
- then
- rm -rf "${!var}"
- fi
- else
- echo "Leaving $var=\"${!var}\""
- fi
- done
-}
-
test_docs() {
cd "$WORKSPACE/doc"
bundle install --no-deployment
commit 3052322166d7f44270479f7421275732582542e0
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Sep 18 13:35:06 2014 -0400
3894: Do not create and abandon new temp dirs if caller has already
provided existing ones. Never delete temp dirs supplied by caller even
if the caller (like the help text) forgets to specify --leave-temp.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 30712eb..f63267e 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -48,8 +48,8 @@ unset $(env | cut -d= -f1 | grep \^ARVADOS_)
COLUMNS=80
-export GOPATH=$(mktemp -d)
-VENVDIR=$(mktemp -d)
+GOPATH=
+VENVDIR=
cli_test=
workbench_test=
apiserver_test=
@@ -72,6 +72,7 @@ fatal() {
declare -a failures
declare -A skip
+declare -A leave_temp
# Always skip CLI tests. They don't know how to use run_test_server.py.
skip[cli]=1
@@ -91,7 +92,8 @@ do
skip_install=1
;;
--leave-temp)
- leave_temp=1
+ leave_temp[VENVDIR]=1
+ leave_temp[GOPATH]=1
;;
*=*)
eval $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2-)\"
@@ -114,7 +116,18 @@ if [[ -n "$CONFIGSRC" ]]; then
fi
fi
-# Set up temporary install dirs
+# Set up temporary install dirs (unless existing dirs were supplied)
+if [[ -n "$VENVDIR" ]]; then
+ VENVDIR=$(mktemp -d)
+else
+ leave_temp[VENVDIR]=1
+fi
+if [[ -n "$GOPATH" ]]; then
+ GOPATH=$(mktemp -d)
+else
+ leave_temp[GOPATH]=1
+fi
+export GOPATH
mkdir -p "$GOPATH/src/git.curoverse.com"
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
|| fatal "symlink failed"
@@ -181,19 +194,18 @@ title () {
}
clear_temp() {
- if [[ -z "$leave_temp" ]]
- then
- for t in "$VENVDIR" "$GOPATH"
- do
- if [[ -n "$t" ]]
+ for var in VENVDIR GOPATH
+ do
+ if [[ -z "${leave_temp[$var]}" ]]
+ then
+ if [[ -n "${!var}" ]]
then
- rm -rf "$t"
+ rm -rf "${!var}"
fi
- done
- else
- echo "Leaving VENVDIR=\"$VENVDIR\""
- echo "Leaving GOPATH=\"$GOPATH\""
- fi
+ else
+ echo "Leaving $var=\"${!var}\""
+ fi
+ done
}
test_docs() {
commit 9022eaebb83e9ab8c53e0ca61c1b44841f0cc809
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Sep 18 13:14:50 2014 -0400
3894: Accept WORKSPACE=foo on command line. Copy config from CONFIGSRC
instead of hard-coded location. Skip this if they don't exist in the
previously hard-coded location and CONFIGSRC is not given.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 8e04550..30712eb 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -7,6 +7,8 @@
# Arguments:
# --skip FOO Do not test the FOO component.
# --only FOO Do not test anything except the FOO component.
+# WORKSPACE=path Arvados source tree to test.
+# CONFIGSRC=path Dir with api server config files to copy into source tree.
# envvar=value Set $envvar to value
#
# Regardless of which components are tested, install all components in
@@ -68,18 +70,6 @@ fatal() {
exit 1
}
-# Sanity check
-echo "WORKSPACE=$WORKSPACE"
-[[ -n "$WORKSPACE" ]] || fatal "WORKSPACE not set"
-
-# Set up temporary install dirs
-mkdir -p "$GOPATH/src/git.curoverse.com"
-ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
- || fatal "symlink failed"
-
-virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
-PATH="$VENVDIR/bin:$PATH"
-
declare -a failures
declare -A skip
@@ -113,6 +103,25 @@ do
esac
done
+# Sanity check
+echo "WORKSPACE=$WORKSPACE"
+[[ -n "$WORKSPACE" ]] || fatal "WORKSPACE not set"
+
+if [[ -n "$CONFIGSRC" ]]; then
+ if [[ -d "$HOME/arvados-api-server" ]]; then
+ # Jenkins expects us to use this by default.
+ CONFIGSRC="$HOME/arvados-api-server"
+ fi
+fi
+
+# Set up temporary install dirs
+mkdir -p "$GOPATH/src/git.curoverse.com"
+ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
+ || fatal "symlink failed"
+
+virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
+PATH="$VENVDIR/bin:$PATH"
+
checkexit() {
if [[ "$?" != "0" ]]; then
title "!!!!!! $1 FAILED !!!!!!"
@@ -241,8 +250,13 @@ install_apiserver() {
rm -f config/environments/test.rb
cp config/environments/test.rb.example config/environments/test.rb
- cp $HOME/arvados-api-server/database.yml config/ || fatal "database.yml"
- cp $HOME/arvados-api-server/application.yml config/ || fatal "application.yml"
+ if [ -n "$CONFIGSRC" ]
+ then
+ for f in database.yml application.yml
+ do
+ cp "$CONFIGSRC/$f" config/ || fatal "$f"
+ done
+ fi
# Fill in a random secret_token and blob_signing_key for testing
SECRET_TOKEN=`echo 'puts rand(2**512).to_s(36)' |ruby`
commit 3c37f8e662a46e78b00e498ab7be26165018f87c
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Sep 16 23:14:56 2014 -0400
Undo two hunks that I accidentally committed in 565c31196f3ccd4346c8a1ac1e19b1deda22eea6.
no issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index af2b3e2..8e04550 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -137,7 +137,7 @@ do_test() {
timer_reset
if [[ "$2" == "go" ]]
then
- go test -timeout 20s "git.curoverse.com/arvados.git/$1"
+ go test "git.curoverse.com/arvados.git/$1"
else
"test_$1"
fi
@@ -155,7 +155,6 @@ do_install() {
timer_reset
if [[ "$2" == "go" ]]
then
- pip install pyyaml
go get -t "git.curoverse.com/arvados.git/$1"
else
"install_$1"
commit 3de7432a187e32c3ca0dd3fb94b3d52c7563dc04
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Sep 16 21:42:47 2014 -0400
Add some robustness in the postgresql department.
no issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 79a7c62..af2b3e2 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -268,6 +268,10 @@ install_apiserver() {
&& git add tmp \
&& git commit -m 'initial commit'
+ # Clear out any lingering postgresql connections to arvados_test, so that we can drop it
+ # This assumes the current user is a postgresql superuser
+ psql arvados_test -c "SELECT pg_terminate_backend (pg_stat_activity.procpid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'arvados_test';" 2>/dev/null
+
cd "$WORKSPACE/services/api" \
&& bundle exec rake db:drop \
&& bundle exec rake db:create \
commit 565c31196f3ccd4346c8a1ac1e19b1deda22eea6
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Sep 16 17:01:29 2014 -0400
Only install the latest gem (i.e. the one we just built).
No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 4724cfa..79a7c62 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -137,7 +137,7 @@ do_test() {
timer_reset
if [[ "$2" == "go" ]]
then
- go test "git.curoverse.com/arvados.git/$1"
+ go test -timeout 20s "git.curoverse.com/arvados.git/$1"
else
"test_$1"
fi
@@ -155,6 +155,7 @@ do_install() {
timer_reset
if [[ "$2" == "go" ]]
then
+ pip install pyyaml
go get -t "git.curoverse.com/arvados.git/$1"
else
"install_$1"
@@ -214,14 +215,14 @@ do_test ruby_sdk
install_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
&& gem build arvados.gemspec \
- && gem install --no-ri --no-rdoc arvados-*.gem
+ && gem install --no-ri --no-rdoc `ls -t arvados-*.gem|head -n1`
}
do_install ruby_sdk
install_cli() {
cd "$WORKSPACE/sdk/cli" \
&& gem build arvados-cli.gemspec \
- && gem install --no-ri --no-rdoc arvados-cli-*.gem
+ && gem install --no-ri --no-rdoc `ls -t arvados-cli-*.gem|head -n1`
}
do_install cli
commit 7cf1194ba3e69a861a7f88740cd6490193339e67
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Sep 16 16:52:01 2014 -0400
Also build our java-bwa-samtools docker image.
no issue #
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index ee57dd3..fccdb23 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -34,4 +34,17 @@ fi
title "docker build complete"
+title "Starting docker java-bwa-samtools build"
+
+./build.sh java-bwa-samtools-image
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! docker java-bwa-samtools BUILD FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "docker build java-bwa-samtools complete"
+
exit $EXITCODE
commit 61a24458626de4c717cd8999ae57b1c573e3e160
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Sep 12 16:33:14 2014 -0400
Fixes for the deploy stage of the build pipeline: actually upgrade our
debian packages and gems.
no issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index 6509e91..b379fff 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -88,7 +88,7 @@ title "Deploying API server complete"
# Install updated debian packages
title "Deploying updated arvados debian packages"
-ssh -p2222 $IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get install arvados-src python-arvados-fuse python-arvados-python-client"
+ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get install arvados-src python-arvados-fuse python-arvados-python-client"
if [[ "$ECODE" != "0" ]]; then
title "!!!!!! DEPLOYING DEBIAN PACKAGES FAILED !!!!!!"
@@ -98,6 +98,19 @@ fi
title "Deploying updated arvados debian packages complete"
+# Install updated arvados gems
+title "Deploying updated arvados gems"
+
+ssh -p2222 root@$IDENTIFIER.arvadosapi.com -C "/usr/local/rvm/bin/rvm default do gem install arvados arvados-cli && /usr/local/rvm/bin/rvm default do gem clean arvados arvados-cli"
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! DEPLOYING ARVADOS GEMS FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ exit $EXITCODE
+fi
+
+title "Deploying updated arvados gems complete"
+
# Deploy Workbench
title "Deploying workbench"
cd "$WORKSPACE"
commit f7878b7f8780cb2e17078486b2239661be6aeedc
Author: Ward Vandewege <ward at curoverse.com>
Date: Sat Sep 6 00:06:15 2014 -0400
Save a bit of time: do not install RI and RDoc for the gems we install
during testing.
no issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 4f4839a..4724cfa 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -214,14 +214,14 @@ do_test ruby_sdk
install_ruby_sdk() {
cd "$WORKSPACE/sdk/ruby" \
&& gem build arvados.gemspec \
- && gem install arvados-*.gem
+ && gem install --no-ri --no-rdoc arvados-*.gem
}
do_install ruby_sdk
install_cli() {
cd "$WORKSPACE/sdk/cli" \
&& gem build arvados-cli.gemspec \
- && gem install arvados-cli-*.gem
+ && gem install --no-ri --no-rdoc arvados-cli-*.gem
}
do_install cli
commit 01f16229133a6f5fda6d92b7f86edda5e08dac8f
Author: Brett Smith <brett at curoverse.com>
Date: Thu Sep 4 16:13:31 2014 -0400
Install & test Ruby SDK, then use that version for other tests.
This commit gives our Ruby SDK the same treatment as other parts of
our build process: we build and install it early, then test it. We
use that built version to run other tests, to make sure all the
components are synced up throughout the run.
This required turning off `bundle install --deployment`, because
Bundler will not use locally installed Gems in that mode. This does
make our build process a little less like what we use in production,
but the benefits of consistent testing and tightening the build
loop (you can update a Gem and dependent Gemfiles in one push) seem
worth that cost.
Refs #3720.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index b79adfb..4f4839a 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -52,6 +52,7 @@ cli_test=
workbench_test=
apiserver_test=
python_sdk_test=
+ruby_sdk_test=
fuse_test=
leave_temp=
skip_install=
@@ -188,7 +189,7 @@ clear_temp() {
test_docs() {
cd "$WORKSPACE/doc"
- bundle install --deployment
+ bundle install --no-deployment
rm -rf .site
# Make sure python-epydoc is installed or the next line won't do much good!
ARVADOS_API_HOST=qr1hi.arvadosapi.com
@@ -203,9 +204,39 @@ test_doclinkchecker() {
}
do_test doclinkchecker
+test_ruby_sdk() {
+ cd "$WORKSPACE/sdk/ruby" \
+ && bundle install --no-deployment \
+ && bundle exec rake test
+}
+do_test ruby_sdk
+
+install_ruby_sdk() {
+ cd "$WORKSPACE/sdk/ruby" \
+ && gem build arvados.gemspec \
+ && gem install arvados-*.gem
+}
+do_install ruby_sdk
+
+install_cli() {
+ cd "$WORKSPACE/sdk/cli" \
+ && gem build arvados-cli.gemspec \
+ && gem install arvados-cli-*.gem
+}
+do_install cli
+
+test_cli() {
+ title "Starting SDK CLI tests"
+ cd "$WORKSPACE/sdk/cli" \
+ && bundle install --no-deployment \
+ && mkdir -p /tmp/keep \
+ && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test $cli_test
+}
+do_test cli
+
install_apiserver() {
cd "$WORKSPACE/services/api"
- bundle install --deployment
+ bundle install --no-deployment
rm -f config/environments/test.rb
cp config/environments/test.rb.example config/environments/test.rb
@@ -249,12 +280,6 @@ test_apiserver() {
}
do_test apiserver
-install_cli() {
- cd "$WORKSPACE/sdk/cli"
- bundle install --deployment
-}
-do_install cli
-
declare -a gostuff
gostuff=(
services/keepstore
@@ -320,20 +345,11 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
- && bundle install --deployment \
+ && bundle install --no-deployment \
&& bundle exec rake test $workbench_test
}
do_test workbench
-test_cli() {
- title "Starting SDK CLI tests"
- cd "$WORKSPACE/sdk/cli" \
- && bundle install --deployment \
- && mkdir -p /tmp/keep \
- && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test $cli_test
-}
-do_test cli
-
clear_temp
for x in "${successes[@]}"
commit ac72156ab86f90457a2907620cb2df5733c57184
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Sep 4 17:11:09 2014 -0400
Fix environment variable setting on command line. no issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index b93aa92..b79adfb 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -103,7 +103,7 @@ do
leave_temp=1
;;
*=*)
- eval $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2)\"
+ eval $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2-)\"
;;
*)
echo >&2 "$0: Unrecognized option: '$arg'"
commit 7c797619a3bf856a5f3cee92fb40722c6e771336
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Aug 28 15:21:31 2014 -0400
Fix $leave_temp in clear_temp() no issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 37c7ba9..b93aa92 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -171,11 +171,14 @@ title () {
}
clear_temp() {
- if [[ -n "$t" && -z "$leave_temp" ]]
+ if [[ -z "$leave_temp" ]]
then
for t in "$VENVDIR" "$GOPATH"
do
- rm -rf "$t"
+ if [[ -n "$t" ]]
+ then
+ rm -rf "$t"
+ fi
done
else
echo "Leaving VENVDIR=\"$VENVDIR\""
commit 38770e0e9fd272061c2bf80ee8bcce25c2611e6a
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Tue Aug 26 10:43:06 2014 -0400
Enhanced run-tests.sh command line supporting choosing specific Python and Ruby
tests, and re-using temp directories. no issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index d9e6db1..37c7ba9 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -13,10 +13,31 @@
# the usual sequence. (Many test suites depend on other components
# being installed.)
#
-# Useful environment variables include $workbench_test, $apiserver_test and
-# $cli_test. To run a specific test, use a command line such as:
+# To run a specific Ruby test, set $workbench_test, $apiserver_test or
+# $cli_test on the command line:
#
-# $ ./run-tests.sh --only workbench workbench_test=TEST=test/integration/pipeline_instances_test.rb
+# $ run-tests.sh --only workbench workbench_test=TEST=test/integration/pipeline_instances_test.rb
+#
+#
+# To run a specific Python test set $python_sdk_test or $fuse_test.
+#
+# $ run-tests.sh --only python_sdk python_sdk_test="--test-suite tests.test_keep_locator"
+#
+#
+# You can also pass "export ARVADOS_DEBUG=1" to enable additional debugging output:
+#
+# $ run-tests.sh "export ARVADOS_DEBUG=1"
+#
+#
+# Finally, you can skip the installation steps on subsequent runs this way:
+#
+## First run
+# $ run-tests.sh --leave-temp
+#
+## Subsequent runs: record the values of VENVDIR and GOPATH from the first run, and
+# provide them on the command line in subsequent runs:
+#
+# $ run-tests.sh --skip-install VENVDIR="/tmp/tmp.y3tsTmigio" GOPATH="/tmp/tmp.3r4sSA9F3l"
# First make sure to remove any ARVADOS_ variables from the calling environment
@@ -30,8 +51,15 @@ VENVDIR=$(mktemp -d)
cli_test=
workbench_test=
apiserver_test=
+python_sdk_test=
+fuse_test=
+leave_temp=
+skip_install=
-source /etc/profile.d/rvm.sh
+if [[ -f /etc/profile.d/rvm.sh ]]
+then
+ source /etc/profile.d/rvm.sh
+fi
fatal() {
clear_temp
@@ -75,7 +103,7 @@ do
leave_temp=1
;;
*=*)
- eval $arg
+ eval $(echo $arg | cut -d= -f1)=\"$(echo $arg | cut -d= -f2)\"
;;
*)
echo >&2 "$0: Unrecognized option: '$arg'"
@@ -143,15 +171,16 @@ title () {
}
clear_temp() {
- for t in "$VENVDIR" "$GOPATH"
- do
- if [[ -n "$t" && -z "$leave-temp" ]]
- then
+ if [[ -n "$t" && -z "$leave_temp" ]]
+ then
+ for t in "$VENVDIR" "$GOPATH"
+ do
rm -rf "$t"
- else
- echo "Leaving $t"
- fi
- done
+ done
+ else
+ echo "Leaving VENVDIR=\"$VENVDIR\""
+ echo "Leaving GOPATH=\"$GOPATH\""
+ fi
}
test_docs() {
@@ -264,7 +293,7 @@ test_python_sdk() {
# the .egg files that setup.py downloads.
cd "$WORKSPACE/sdk/python" \
- && python setup.py test
+ && python setup.py test $python_sdk_test
r=$?
easy_install *.egg
return $r
@@ -274,7 +303,7 @@ do_test python_sdk
test_fuse() {
# Install test dependencies here too, in case run_test_server needs them.
cd "$WORKSPACE/services/fuse" \
- && python setup.py test
+ && python setup.py test $fuse_test
r=$?
easy_install *.egg
return $r
commit 043ff05cfca336d99cf40480e93ec4cae06a7fdd
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Aug 21 12:00:09 2014 -0400
Declare empty variables up top to avoid bash parsing errors. no issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index c7531e2..d9e6db1 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -27,6 +27,9 @@ COLUMNS=80
export GOPATH=$(mktemp -d)
VENVDIR=$(mktemp -d)
+cli_test=
+workbench_test=
+apiserver_test=
source /etc/profile.d/rvm.sh
@@ -65,6 +68,12 @@ do
--only)
only="$1"; shift
;;
+ --skip-install)
+ skip_install=1
+ ;;
+ --leave-temp)
+ leave_temp=1
+ ;;
*=*)
eval $arg
;;
@@ -111,16 +120,21 @@ do_test() {
}
do_install() {
- title "Running $1 install"
- timer_reset
- if [[ "$2" == "go" ]]
+ if [[ -z "$skip_install" ]]
then
- go get -t "git.curoverse.com/arvados.git/$1"
+ title "Running $1 install"
+ timer_reset
+ if [[ "$2" == "go" ]]
+ then
+ go get -t "git.curoverse.com/arvados.git/$1"
+ else
+ "install_$1"
+ fi
+ checkexit "$1 install"
+ title "End of $1 install (`timer`)"
else
- "install_$1"
+ title "Skipping $1 install"
fi
- checkexit "$1 install"
- title "End of $1 install (`timer`)"
}
title () {
@@ -131,9 +145,11 @@ title () {
clear_temp() {
for t in "$VENVDIR" "$GOPATH"
do
- if [[ -n "$t" ]]
+ if [[ -n "$t" && -z "$leave-temp" ]]
then
rm -rf "$t"
+ else
+ echo "Leaving $t"
fi
done
}
@@ -197,7 +213,7 @@ do_install apiserver
test_apiserver() {
cd "$WORKSPACE/services/api"
- bundle exec rake test "$apiserver_test"
+ bundle exec rake test $apiserver_test
}
do_test apiserver
@@ -273,7 +289,7 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
&& bundle install --deployment \
- && bundle exec rake test "$workbench_test"
+ && bundle exec rake test $workbench_test
}
do_test workbench
@@ -282,7 +298,7 @@ test_cli() {
cd "$WORKSPACE/sdk/cli" \
&& bundle install --deployment \
&& mkdir -p /tmp/keep \
- && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test "$cli_test"
+ && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test $cli_test
}
do_test cli
commit d7e56bd7e8744b54af33675cb875859ca9d053ea
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Aug 21 10:21:39 2014 -0400
Permit setting environment variables on command line. Added workbench_test,
apiserver_test and cli_test variables to provide parameters to the respective
rake tests. no issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 2e93e3d..c7531e2 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -5,14 +5,20 @@
# Exit non-zero if any tests fail.
#
# Arguments:
-# --skip FOO Do not test the FOO component.
-# --only FOO Do not test anything except the FOO component.
+# --skip FOO Do not test the FOO component.
+# --only FOO Do not test anything except the FOO component.
+# envvar=value Set $envvar to value
#
# Regardless of which components are tested, install all components in
# the usual sequence. (Many test suites depend on other components
# being installed.)
#
+# Useful environment variables include $workbench_test, $apiserver_test and
+# $cli_test. To run a specific test, use a command line such as:
#
+# $ ./run-tests.sh --only workbench workbench_test=TEST=test/integration/pipeline_instances_test.rb
+
+
# First make sure to remove any ARVADOS_ variables from the calling environment
# that could interfer with the tests.
unset $(env | cut -d= -f1 | grep \^ARVADOS_)
@@ -59,6 +65,9 @@ do
--only)
only="$1"; shift
;;
+ *=*)
+ eval $arg
+ ;;
*)
echo >&2 "$0: Unrecognized option: '$arg'"
exit 1
@@ -188,7 +197,7 @@ do_install apiserver
test_apiserver() {
cd "$WORKSPACE/services/api"
- bundle exec rake test
+ bundle exec rake test "$apiserver_test"
}
do_test apiserver
@@ -264,7 +273,7 @@ done
test_workbench() {
cd "$WORKSPACE/apps/workbench" \
&& bundle install --deployment \
- && bundle exec rake test
+ && bundle exec rake test "$workbench_test"
}
do_test workbench
@@ -273,7 +282,7 @@ test_cli() {
cd "$WORKSPACE/sdk/cli" \
&& bundle install --deployment \
&& mkdir -p /tmp/keep \
- && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test
+ && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test "$cli_test"
}
do_test cli
commit 327681d1eee19c941e33beaef75359ea72b0c2f9
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Wed Aug 20 15:21:06 2014 -0400
Unset ARVADOS_ variable from the environment before running tests. no issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index d472965..2e93e3d 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -11,11 +11,14 @@
# Regardless of which components are tested, install all components in
# the usual sequence. (Many test suites depend on other components
# being installed.)
+#
+#
+# First make sure to remove any ARVADOS_ variables from the calling environment
+# that could interfer with the tests.
+unset $(env | cut -d= -f1 | grep \^ARVADOS_)
COLUMNS=80
-ARVADOS_API_HOST=qr1hi.arvadosapi.com
-
export GOPATH=$(mktemp -d)
VENVDIR=$(mktemp -d)
@@ -131,7 +134,9 @@ test_docs() {
bundle install --deployment
rm -rf .site
# Make sure python-epydoc is installed or the next line won't do much good!
+ ARVADOS_API_HOST=qr1hi.arvadosapi.com
PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
+ unset ARVADOS_API_HOST
}
do_test docs
commit 9336a8a8a11983f28e61db93b9e2262eff793f39
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Aug 19 16:26:32 2014 -0400
Fix failure detection for python tests. No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 4c5f070..d472965 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -235,7 +235,9 @@ test_python_sdk() {
cd "$WORKSPACE/sdk/python" \
&& python setup.py test
+ r=$?
easy_install *.egg
+ return $r
}
do_test python_sdk
@@ -243,7 +245,9 @@ test_fuse() {
# Install test dependencies here too, in case run_test_server needs them.
cd "$WORKSPACE/services/fuse" \
&& python setup.py test
+ r=$?
easy_install *.egg
+ return $r
}
do_test fuse
commit 79df4374ee643ae843e93abc14b80145c9154730
Author: Tom Clegg <tom at curoverse.com>
Date: Tue Aug 19 15:04:56 2014 -0400
Install py test deps even if py tests fail. Make docs skippable. No issue #
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 9f0b5e6..4c5f070 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -126,14 +126,14 @@ clear_temp() {
done
}
-install_docs() {
+test_docs() {
cd "$WORKSPACE/doc"
bundle install --deployment
rm -rf .site
# Make sure python-epydoc is installed or the next line won't do much good!
PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
}
-do_install docs
+do_test docs
test_doclinkchecker() {
cd "$WORKSPACE/doc"
@@ -234,16 +234,16 @@ test_python_sdk() {
# the .egg files that setup.py downloads.
cd "$WORKSPACE/sdk/python" \
- && python setup.py test \
- && easy_install *.egg
+ && python setup.py test
+ easy_install *.egg
}
do_test python_sdk
test_fuse() {
# Install test dependencies here too, in case run_test_server needs them.
cd "$WORKSPACE/services/fuse" \
- && python setup.py test \
- && easy_install *.egg
+ && python setup.py test
+ easy_install *.egg
}
do_test fuse
commit 346ebae28e3a179b51fe79948212006a09d6940e
Merge: b70e4ce 6882ae2
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Aug 19 10:22:58 2014 -0400
Merge branch 'skip-args'
no issue #
commit 6882ae221f4dcecda48162696b8dd756828b37c3
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Aug 18 14:37:45 2014 -0400
Fix swapped install/test sections for fuse.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 0d89dcf..9f0b5e6 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -219,12 +219,12 @@ install_python_sdk() {
}
do_install python_sdk
-test_fuse() {
+install_fuse() {
cd "$WORKSPACE/services/fuse" \
&& python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz \
&& pip install dist/arvados_fuse-0.1.*.tar.gz
}
-do_test fuse
+do_install fuse
test_python_sdk() {
# Python SDK. We test this before testing keepproxy: keepproxy runs
@@ -239,13 +239,13 @@ test_python_sdk() {
}
do_test python_sdk
-install_fuse() {
+test_fuse() {
# Install test dependencies here too, in case run_test_server needs them.
cd "$WORKSPACE/services/fuse" \
&& python setup.py test \
&& easy_install *.egg
}
-do_install fuse
+do_test fuse
for g in "${gostuff[@]}"
do
commit 05e6f52d0cf418ee7d727b5e034d45a60a246ba3
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Aug 18 10:48:19 2014 -0400
Display elapsed time for each test/install.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 028f0a3..0d89dcf 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -66,16 +66,25 @@ done
checkexit() {
if [[ "$?" != "0" ]]; then
title "!!!!!! $1 FAILED !!!!!!"
- failures+=("$1")
+ failures+=("$1 (`timer`)")
else
- successes+=("$1")
+ successes+=("$1 (`timer`)")
fi
}
+timer_reset() {
+ t0=$SECONDS
+}
+
+timer() {
+ echo -n "$(($SECONDS - $t0))s"
+}
+
do_test() {
if [[ -z "${skip[$1]}" ]] && ( [[ -z "$only" ]] || [[ "$only" == "$1" ]] )
then
title "Running $1 tests"
+ timer_reset
if [[ "$2" == "go" ]]
then
go test "git.curoverse.com/arvados.git/$1"
@@ -83,7 +92,7 @@ do_test() {
"test_$1"
fi
checkexit "$1 tests"
- title "End of $1 tests"
+ title "End of $1 tests (`timer`)"
else
title "Skipping $1 tests"
fi
@@ -91,6 +100,7 @@ do_test() {
do_install() {
title "Running $1 install"
+ timer_reset
if [[ "$2" == "go" ]]
then
go get -t "git.curoverse.com/arvados.git/$1"
@@ -98,7 +108,7 @@ do_install() {
"install_$1"
fi
checkexit "$1 install"
- title "End of $1 install"
+ title "End of $1 install (`timer`)"
}
title () {
commit 9b75f439dcdfeb850487c59863c680c9edd772a1
Author: Tom Clegg <tom at curoverse.com>
Date: Mon Aug 18 10:48:00 2014 -0400
Add missing "install fuse" and "test workbench".
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index f4e729a..028f0a3 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -235,17 +235,19 @@ install_fuse() {
&& python setup.py test \
&& easy_install *.egg
}
+do_install fuse
for g in "${gostuff[@]}"
do
do_test "$g" go
done
-install_workbench() {
+test_workbench() {
cd "$WORKSPACE/apps/workbench" \
&& bundle install --deployment \
&& bundle exec rake test
}
+do_test workbench
test_cli() {
title "Starting SDK CLI tests"
commit 06f6b31925a51c5100017c4a05bba430d29252a4
Author: Tom Clegg <tom at curoverse.com>
Date: Sat Aug 16 02:00:23 2014 -0400
Cleanup, add some error checks, add --skip and --only args.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index e924150..f4e729a 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -1,227 +1,276 @@
#!/bin/bash
-EXITCODE=0
+# Install and test Arvados components.
+#
+# Exit non-zero if any tests fail.
+#
+# Arguments:
+# --skip FOO Do not test the FOO component.
+# --only FOO Do not test anything except the FOO component.
+#
+# Regardless of which components are tested, install all components in
+# the usual sequence. (Many test suites depend on other components
+# being installed.)
COLUMNS=80
ARVADOS_API_HOST=qr1hi.arvadosapi.com
-title () {
- printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
-}
+export GOPATH=$(mktemp -d)
+VENVDIR=$(mktemp -d)
source /etc/profile.d/rvm.sh
-echo $WORKSPACE
-export GOPATH=$(mktemp -d)
+fatal() {
+ clear_temp
+ echo >&2 "Fatal: $* in ${FUNCNAME[1]} at ${BASH_SOURCE[1]} line ${BASH_LINENO[0]}"
+ exit 1
+}
+
+# Sanity check
+echo "WORKSPACE=$WORKSPACE"
+[[ -n "$WORKSPACE" ]] || fatal "WORKSPACE not set"
+
+# Set up temporary install dirs
mkdir -p "$GOPATH/src/git.curoverse.com"
-ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
+ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" \
+ || fatal "symlink failed"
-VENVDIR=$(mktemp -d)
-virtualenv --setuptools "$VENVDIR"
+virtualenv --setuptools "$VENVDIR" || fatal "virtualenv $VENVDIR failed"
PATH="$VENVDIR/bin:$PATH"
-# DOCS
-title "Starting DOC build"
-cd "$WORKSPACE"
-cd doc
-bundle install --deployment
-rm -rf .site
-# Make sure python-epydoc is installed or the next line won't do much good!
-PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
+declare -a failures
+declare -A skip
-checkexit() {
- ECODE=$?
+# Always skip CLI tests. They don't know how to use run_test_server.py.
+skip[cli]=1
- if [[ "$ECODE" != "0" ]]; then
+while [[ -n "$1" ]]
+do
+ arg="$1"; shift
+ case "$arg" in
+ --skip)
+ skipwhat="$1"; shift
+ skip[$skipwhat]=1
+ ;;
+ --only)
+ only="$1"; shift
+ ;;
+ *)
+ echo >&2 "$0: Unrecognized option: '$arg'"
+ exit 1
+ ;;
+ esac
+done
+
+checkexit() {
+ if [[ "$?" != "0" ]]; then
title "!!!!!! $1 FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
+ failures+=("$1")
+ else
+ successes+=("$1")
fi
}
-goget() {
- go get -t "git.curoverse.com/arvados.git/$1"
- checkexit "$1 install"
+do_test() {
+ if [[ -z "${skip[$1]}" ]] && ( [[ -z "$only" ]] || [[ "$only" == "$1" ]] )
+ then
+ title "Running $1 tests"
+ if [[ "$2" == "go" ]]
+ then
+ go test "git.curoverse.com/arvados.git/$1"
+ else
+ "test_$1"
+ fi
+ checkexit "$1 tests"
+ title "End of $1 tests"
+ else
+ title "Skipping $1 tests"
+ fi
}
-gotest() {
- title "Starting $1 tests"
- cd "$WORKSPACE"
-
- go test "git.curoverse.com/arvados.git/$1"
-
- checkexit "$1 tests"
- title "$1 tests complete"
+do_install() {
+ title "Running $1 install"
+ if [[ "$2" == "go" ]]
+ then
+ go get -t "git.curoverse.com/arvados.git/$1"
+ else
+ "install_$1"
+ fi
+ checkexit "$1 install"
+ title "End of $1 install"
}
-checkexit "Doc build"
-title "DOC build complete"
-
-# DOC linkchecker
-title "Starting DOC linkchecker"
-cd "$WORKSPACE"
-cd doc
-bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
-
-checkexit "Doc linkchecker"
-title "DOC linkchecker complete"
+title () {
+ txt="********** $1 **********"
+ printf "\n%*s%s\n\n" $((($COLUMNS-${#txt})/2)) "" "$txt"
+}
-# API SERVER
-title "Starting API server tests"
-cd "$WORKSPACE"
-cd services/api
-bundle install --deployment
+clear_temp() {
+ for t in "$VENVDIR" "$GOPATH"
+ do
+ if [[ -n "$t" ]]
+ then
+ rm -rf "$t"
+ fi
+ done
+}
-rm -f config/database.yml
-rm -f config/environments/test.rb
-cp config/environments/test.rb.example config/environments/test.rb
+install_docs() {
+ cd "$WORKSPACE/doc"
+ bundle install --deployment
+ rm -rf .site
+ # Make sure python-epydoc is installed or the next line won't do much good!
+ PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
+}
+do_install docs
-# Get test database config
-cp $HOME/arvados-api-server/database.yml config/
-# Get test application.yml file
-cp $HOME/arvados-api-server/application.yml config/
+test_doclinkchecker() {
+ cd "$WORKSPACE/doc"
+ bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
+}
+do_test doclinkchecker
-# Fill in a random secret_token and blob_signing_key for testing
-SECRET_TOKEN=`echo 'puts rand(2**512).to_s(36)' |ruby`
-BLOB_SIGNING_KEY=`echo 'puts rand(2**512).to_s(36)' |ruby`
+install_apiserver() {
+ cd "$WORKSPACE/services/api"
+ bundle install --deployment
-sed -i'' -e "s:SECRET_TOKEN:$SECRET_TOKEN:" config/application.yml
-sed -i'' -e "s:BLOB_SIGNING_KEY:$BLOB_SIGNING_KEY:" config/application.yml
+ rm -f config/environments/test.rb
+ cp config/environments/test.rb.example config/environments/test.rb
-export RAILS_ENV=test
+ cp $HOME/arvados-api-server/database.yml config/ || fatal "database.yml"
+ cp $HOME/arvados-api-server/application.yml config/ || fatal "application.yml"
-# Set up empty git repo (for git tests)
-GITDIR=$WORKSPACE/tmpgit
-rm -rf $GITDIR
-mkdir $GITDIR
-sed -i'' -e "s:/var/cache/git:$GITDIR:" config/application.default.yml
+ # Fill in a random secret_token and blob_signing_key for testing
+ SECRET_TOKEN=`echo 'puts rand(2**512).to_s(36)' |ruby`
+ BLOB_SIGNING_KEY=`echo 'puts rand(2**512).to_s(36)' |ruby`
-rm -rf $GITDIR
-mkdir -p $GITDIR/test
-cd $GITDIR/test
-/usr/bin/git init
-/usr/bin/git config user.email "jenkins at ci.curoverse.com"
-/usr/bin/git config user.name "Jenkins, CI"
-touch tmp
-/usr/bin/git add tmp
-/usr/bin/git commit -m 'initial commit'
+ sed -i'' -e "s:SECRET_TOKEN:$SECRET_TOKEN:" config/application.yml
+ sed -i'' -e "s:BLOB_SIGNING_KEY:$BLOB_SIGNING_KEY:" config/application.yml
-cd "$WORKSPACE"
-cd services/api
+ export RAILS_ENV=test
-bundle exec rake db:drop
-bundle exec rake db:create
-bundle exec rake db:setup
-bundle exec rake test
+ # Set up empty git repo (for git tests)
+ GITDIR="$WORKSPACE/tmpgit"
+ sed -i'' -e "s:/var/cache/git:$GITDIR:" config/application.default.yml
-checkexit "API server tests"
-title "API server tests complete"
+ rm -rf $GITDIR
+ mkdir -p $GITDIR/test
+ cd $GITDIR/test \
+ && git init \
+ && git config user.email "jenkins at ci.curoverse.com" \
+ && git config user.name "Jenkins, CI" \
+ && touch tmp \
+ && git add tmp \
+ && git commit -m 'initial commit'
-# Install CLI gem's dependencies.
+ cd "$WORKSPACE/services/api" \
+ && bundle exec rake db:drop \
+ && bundle exec rake db:create \
+ && bundle exec rake db:setup
+}
+do_install apiserver
-cd "$WORKSPACE/sdk/cli"
-bundle install --deployment
+test_apiserver() {
+ cd "$WORKSPACE/services/api"
+ bundle exec rake test
+}
+do_test apiserver
-cd "$WORKSPACE"
-gostuff="services/keepstore services/keepproxy sdk/go/arvadosclient sdk/go/keepclient sdk/go/streamer"
-for dir in $gostuff
+install_cli() {
+ cd "$WORKSPACE/sdk/cli"
+ bundle install --deployment
+}
+do_install cli
+
+declare -a gostuff
+gostuff=(
+ services/keepstore
+ services/keepproxy
+ sdk/go/arvadosclient
+ sdk/go/keepclient
+ sdk/go/streamer
+ )
+for g in "${gostuff[@]}"
do
- goget "$dir"
+ do_install "$g" go
done
-# Install the Python SDK early. Various other test suites (like
-# keepproxy) bring up run_test_server.py, which imports the arvados
-# module. We can't actually *test* the Python SDK yet though, because
-# its own test suite brings up some of those other programs (like
-# keepproxy).
-
-cd "$WORKSPACE/sdk/python"
-python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
-pip install dist/arvados-python-client-0.1.*.tar.gz
-
-checkexit "Python SDK install"
-
-cd "$WORKSPACE/services/fuse"
-python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
-pip install dist/arvados_fuse-0.1.*.tar.gz
-
-checkexit "FUSE install"
-
-# Python SDK. We test this before testing keepproxy: keepproxy runs
-# run_test_server.py, which depends on the yaml package, which is in
-# tests_require but not install_requires, and therefore does not get
-# installed by setuptools until we run "setup.py test" *and* install
-# the .egg files that setup.py downloads.
+install_python_sdk() {
+ # Install the Python SDK early. Various other test suites (like
+ # keepproxy) bring up run_test_server.py, which imports the arvados
+ # module. We can't actually *test* the Python SDK yet though, because
+ # its own test suite brings up some of those other programs (like
+ # keepproxy).
-title "Starting Python SDK tests"
-cd "$WORKSPACE/sdk/python"
-
-python setup.py test
-checkexit "Python SDK tests"
-easy_install *.egg
-
-title "Python tests complete"
-
-# FUSE driver
-
-title "Starting FUSE tests"
-cd "$WORKSPACE/services/fuse"
-
-python setup.py test
-checkexit "FUSE tests"
-easy_install *.egg
-
-title "FUSE tests complete"
+ cd "$WORKSPACE/sdk/python" \
+ && python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz \
+ && pip install dist/arvados-python-client-0.1.*.tar.gz
+}
+do_install python_sdk
+test_fuse() {
+ cd "$WORKSPACE/services/fuse" \
+ && python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz \
+ && pip install dist/arvados_fuse-0.1.*.tar.gz
+}
+do_test fuse
+
+test_python_sdk() {
+ # Python SDK. We test this before testing keepproxy: keepproxy runs
+ # run_test_server.py, which depends on the yaml package, which is in
+ # tests_require but not install_requires, and therefore does not get
+ # installed by setuptools until we run "setup.py test" *and* install
+ # the .egg files that setup.py downloads.
+
+ cd "$WORKSPACE/sdk/python" \
+ && python setup.py test \
+ && easy_install *.egg
+}
+do_test python_sdk
-# Go SDK packages
+install_fuse() {
+ # Install test dependencies here too, in case run_test_server needs them.
+ cd "$WORKSPACE/services/fuse" \
+ && python setup.py test \
+ && easy_install *.egg
+}
-for dir in $gostuff
+for g in "${gostuff[@]}"
do
- gotest "$dir"
+ do_test "$g" go
done
-# Workbench
-title "Starting workbench tests"
-cd "$WORKSPACE"
-cd apps/workbench
-bundle install --deployment
-
-echo $PATH
-
-bundle exec rake test
-
-checkexit "Workbench tests"
-title "Workbench tests complete"
-
-# Clean up temporary virtualenv and GOPATH
-rm -rf "$VENVDIR"
-rm -rf "$GOPATH"
-
-# The CLI SDK tests require a working API server, so let's skip those for now.
-exit $EXITCODE
-
-########################################################################
-
-# CLI SDK
-title "Starting SDK CLI tests"
-cd "$WORKSPACE"
-cd sdk/cli
-bundle install --deployment
+install_workbench() {
+ cd "$WORKSPACE/apps/workbench" \
+ && bundle install --deployment \
+ && bundle exec rake test
+}
-# Set up Python SDK and dependencies
-cd ../python
-rm -rf $HOME/lib/python
-mkdir $HOME/lib/python
-PYTHONPATH="$HOME/lib/python" easy_install --install-dir=$HOME/lib/python --upgrade google-api-python-client
-PYTHONPATH="$HOME/lib/python" python setup.py install --home=$HOME
+test_cli() {
+ title "Starting SDK CLI tests"
+ cd "$WORKSPACE/sdk/cli" \
+ && bundle install --deployment \
+ && mkdir -p /tmp/keep \
+ && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test
+}
+do_test cli
-cd ../cli
-mkdir -p /tmp/keep
-export KEEP_LOCAL_STORE=/tmp/keep
-PYTHONPATH="$HOME/lib/python" bundle exec rake test
+clear_temp
-checkexit "SDK CLI tests"
-title "SDK CLI tests complete"
+for x in "${successes[@]}"
+do
+ echo "Pass: $x"
+done
-exit $EXITCODE
+if [[ ${#failures[@]} == 0 ]]
+then
+ echo "All test suites passed."
+else
+ echo "Failures (${#failures[@]}):"
+ for x in "${failures[@]}"
+ do
+ echo "Fail: $x"
+ done
+fi
+exit ${#failures}
commit b70e4ce5d5ea65e788a172e757af5c07b016fe4f
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Aug 15 19:34:44 2014 -0400
Remove duplicated cleanup commands; the realclean target takes care of
that now.
no issue #
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index 244c0d7..ee57dd3 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -13,11 +13,6 @@ echo $WORKSPACE
# DOCKER
title "Starting docker build"
-# clean up existing docker containers and images
-docker.io stop $(docker.io ps -a -q)
-docker.io rm $(docker.io ps -a -q)
-docker.io rmi $(docker.io images -q)
-
# clean up the docker build environment
cd "$WORKSPACE"
cd docker
commit 2d816cf50f92cb6d984a8821f6c6ddd621f5140b
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Aug 15 09:23:54 2014 -0400
keep is now called keepstore
refs #3551
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index fb522f8..4a39391 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -150,7 +150,7 @@ ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
# Keep -> keepstore
go get "git.curoverse.com/arvados.git/services/keepstore"
cd $WORKSPACE/debs
-build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keep keep 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keepstore keepstore 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
# Keep proxy
commit 17341ef86b68cab0352535d12c5925814609e4d2
Merge: d9b0e60 bae7945
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Aug 15 08:51:33 2014 -0400
Merge branch '3551-go-layout'
refs #3551
commit bae79452d0fb211338eb2a327f3679ebc5210ca2
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Aug 14 18:46:37 2014 -0400
3551: Install Python and Go bits, force installation of Python test dependencies, *then* run tests.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 0557ab1..e924150 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -39,12 +39,16 @@ checkexit() {
fi
}
+goget() {
+ go get -t "git.curoverse.com/arvados.git/$1"
+ checkexit "$1 install"
+}
+
gotest() {
title "Starting $1 tests"
cd "$WORKSPACE"
- go get -t "git.curoverse.com/arvados.git/$1" \
- && go test "git.curoverse.com/arvados.git/$1"
+ go test "git.curoverse.com/arvados.git/$1"
checkexit "$1 tests"
title "$1 tests complete"
@@ -115,61 +119,69 @@ title "API server tests complete"
# Install CLI gem's dependencies.
-cd "$WORKSPACE"
-cd sdk/cli
+cd "$WORKSPACE/sdk/cli"
bundle install --deployment
+cd "$WORKSPACE"
+gostuff="services/keepstore services/keepproxy sdk/go/arvadosclient sdk/go/keepclient sdk/go/streamer"
+for dir in $gostuff
+do
+ goget "$dir"
+done
+
# Install the Python SDK early. Various other test suites (like
# keepproxy) bring up run_test_server.py, which imports the arvados
# module. We can't actually *test* the Python SDK yet though, because
# its own test suite brings up some of those other programs (like
# keepproxy).
+cd "$WORKSPACE/sdk/python"
python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
pip install dist/arvados-python-client-0.1.*.tar.gz
checkexit "Python SDK install"
-# Keep daemons
-
-gotest services/keepstore
+cd "$WORKSPACE/services/fuse"
+python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
+pip install dist/arvados_fuse-0.1.*.tar.gz
-gotest services/keepproxy
+checkexit "FUSE install"
-# Python SDK (already installed, but we didn't run its tests yet)
+# Python SDK. We test this before testing keepproxy: keepproxy runs
+# run_test_server.py, which depends on the yaml package, which is in
+# tests_require but not install_requires, and therefore does not get
+# installed by setuptools until we run "setup.py test" *and* install
+# the .egg files that setup.py downloads.
title "Starting Python SDK tests"
-cd "$WORKSPACE"
-cd sdk/python
+cd "$WORKSPACE/sdk/python"
python setup.py test
-
checkexit "Python SDK tests"
+easy_install *.egg
+
+title "Python tests complete"
# FUSE driver
-cd "$WORKSPACE"
-cd services/fuse
+title "Starting FUSE tests"
+cd "$WORKSPACE/services/fuse"
python setup.py test
-
checkexit "FUSE tests"
+easy_install *.egg
-python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
-pip install dist/arvados_fuse-0.1.*.tar.gz
-
-checkexit "FUSE install"
+title "FUSE tests complete"
-title "Python SDK tests complete"
# Go SDK packages
-for dir in sdk/go/arvadosclient sdk/go/keepclient sdk/go/streamer
+for dir in $gostuff
do
gotest "$dir"
done
-# WORKBENCH
+# Workbench
title "Starting workbench tests"
cd "$WORKSPACE"
cd apps/workbench
commit 32468a5b95280d671f4648dea6d2ff10d171378d
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Aug 14 17:18:33 2014 -0400
3551: Fix dependency cycle: Install Python SDK, then install+test Go stuff, then test Python SDK.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index f52b585..0557ab1 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -40,14 +40,14 @@ checkexit() {
}
gotest() {
- title "Starting $1 tests"
- cd "$WORKSPACE"
+ title "Starting $1 tests"
+ cd "$WORKSPACE"
- go get -t "git.curoverse.com/arvados.git/$1" \
- && go test "git.curoverse.com/arvados.git/$1"
+ go get -t "git.curoverse.com/arvados.git/$1" \
+ && go test "git.curoverse.com/arvados.git/$1"
- checkexit "$1 tests"
- title "$1 tests complete"
+ checkexit "$1 tests"
+ title "$1 tests complete"
}
checkexit "Doc build"
@@ -113,18 +113,30 @@ bundle exec rake test
checkexit "API server tests"
title "API server tests complete"
-# Keepstore. The keepstore binary is used by many other test suites,
-# so we test and install it early.
-
-gotest services/keepstore
-
-# Python SDK
+# Install CLI gem's dependencies.
cd "$WORKSPACE"
cd sdk/cli
bundle install --deployment
-# Set up Python SDK and dependencies
+# Install the Python SDK early. Various other test suites (like
+# keepproxy) bring up run_test_server.py, which imports the arvados
+# module. We can't actually *test* the Python SDK yet though, because
+# its own test suite brings up some of those other programs (like
+# keepproxy).
+
+python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
+pip install dist/arvados-python-client-0.1.*.tar.gz
+
+checkexit "Python SDK install"
+
+# Keep daemons
+
+gotest services/keepstore
+
+gotest services/keepproxy
+
+# Python SDK (already installed, but we didn't run its tests yet)
title "Starting Python SDK tests"
cd "$WORKSPACE"
@@ -134,15 +146,11 @@ python setup.py test
checkexit "Python SDK tests"
-python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
-pip install dist/arvados-python-client-0.1.*.tar.gz
-
-checkexit "Python SDK install"
+# FUSE driver
cd "$WORKSPACE"
cd services/fuse
-# We reuse $VENVDIR from the Python SDK tests above
python setup.py test
checkexit "FUSE tests"
@@ -154,10 +162,12 @@ checkexit "FUSE install"
title "Python SDK tests complete"
-for dir in services/keepproxy sdk/go/arvadosclient sdk/go/keepclient sdk/go/streamer
+# Go SDK packages
+
+for dir in sdk/go/arvadosclient sdk/go/keepclient sdk/go/streamer
do
gotest "$dir"
-end
+done
# WORKBENCH
title "Starting workbench tests"
@@ -179,6 +189,8 @@ rm -rf "$GOPATH"
# The CLI SDK tests require a working API server, so let's skip those for now.
exit $EXITCODE
+########################################################################
+
# CLI SDK
title "Starting SDK CLI tests"
cd "$WORKSPACE"
commit 35532d9eec8564a63e91af98a08117203e859d97
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Aug 14 16:30:56 2014 -0400
3551: Test and install keepstore before Python SDK.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index bee8002..f52b585 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -17,6 +17,10 @@ export GOPATH=$(mktemp -d)
mkdir -p "$GOPATH/src/git.curoverse.com"
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
+VENVDIR=$(mktemp -d)
+virtualenv --setuptools "$VENVDIR"
+PATH="$VENVDIR/bin:$PATH"
+
# DOCS
title "Starting DOC build"
cd "$WORKSPACE"
@@ -35,6 +39,17 @@ checkexit() {
fi
}
+gotest() {
+ title "Starting $1 tests"
+ cd "$WORKSPACE"
+
+ go get -t "git.curoverse.com/arvados.git/$1" \
+ && go test "git.curoverse.com/arvados.git/$1"
+
+ checkexit "$1 tests"
+ title "$1 tests complete"
+}
+
checkexit "Doc build"
title "DOC build complete"
@@ -98,53 +113,29 @@ bundle exec rake test
checkexit "API server tests"
title "API server tests complete"
-# Install and test Go bits. keepstore must come before keepproxy and keepclient.
-for dir in services/keepstore services/keepproxy sdk/go/arvadosclient sdk/go/keepclient sdk/go/streamer
-do
- title "Starting $dir tests"
- cd "$WORKSPACE"
-
- go get -t "git.curoverse.com/arvados.git/$dir" \
- && go test "git.curoverse.com/arvados.git/$dir"
-
- checkexit "$dir tests"
- title "$dir tests complete"
-done
-
-
-# WORKBENCH
-title "Starting workbench tests"
-cd "$WORKSPACE"
-cd apps/workbench
-bundle install --deployment
-
-echo $PATH
-
-
-bundle exec rake test
+# Keepstore. The keepstore binary is used by many other test suites,
+# so we test and install it early.
-checkexit "Workbench tests"
-title "Workbench tests complete"
+gotest services/keepstore
# Python SDK
-title "Starting Python SDK tests"
+
cd "$WORKSPACE"
cd sdk/cli
bundle install --deployment
# Set up Python SDK and dependencies
+title "Starting Python SDK tests"
cd "$WORKSPACE"
cd sdk/python
-VENVDIR=$(mktemp -d)
-virtualenv --setuptools "$VENVDIR"
-"$VENVDIR/bin/python" setup.py test
+python setup.py test
checkexit "Python SDK tests"
-"$VENVDIR/bin/python" setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
-"$VENVDIR/bin/pip" install dist/arvados-python-client-0.1.*.tar.gz
+python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
+pip install dist/arvados-python-client-0.1.*.tar.gz
checkexit "Python SDK install"
@@ -152,19 +143,38 @@ cd "$WORKSPACE"
cd services/fuse
# We reuse $VENVDIR from the Python SDK tests above
-"$VENVDIR/bin/python" setup.py test
+python setup.py test
checkexit "FUSE tests"
-"$VENVDIR/bin/python" setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
-"$VENVDIR/bin/pip" install dist/arvados_fuse-0.1.*.tar.gz
+python setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
+pip install dist/arvados_fuse-0.1.*.tar.gz
checkexit "FUSE install"
title "Python SDK tests complete"
-# Clean up $VENVDIR
+for dir in services/keepproxy sdk/go/arvadosclient sdk/go/keepclient sdk/go/streamer
+do
+ gotest "$dir"
+end
+
+# WORKBENCH
+title "Starting workbench tests"
+cd "$WORKSPACE"
+cd apps/workbench
+bundle install --deployment
+
+echo $PATH
+
+bundle exec rake test
+
+checkexit "Workbench tests"
+title "Workbench tests complete"
+
+# Clean up temporary virtualenv and GOPATH
rm -rf "$VENVDIR"
+rm -rf "$GOPATH"
# The CLI SDK tests require a working API server, so let's skip those for now.
exit $EXITCODE
commit 1d1cdff657785ca3c9d930d1cf3ef462d98a1b24
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Aug 14 16:30:13 2014 -0400
3551: Clean up temp dir.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 65d0a9f..fb522f8 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -182,7 +182,6 @@ build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse,
cd $WORKSPACE/services/fuse
sed -i'' -e "s:version='0.1':version='0.1.$GIT_HASH':" setup.py
-
cd $WORKSPACE/debs
# Please seem comment about --no-python-fix-name above; we stay consistent and do
@@ -204,3 +203,5 @@ else
echo "No new packages generated. No PRM run necessary."
fi
+# clean up temporary GOPATH
+rm -rf "$GOPATH"
commit d9b0e6003e265412de46174ea0e2705a8e872f2b
Merge: 1ae46a0 45ab2f4
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Aug 14 16:00:28 2014 -0400
Merge branch '3551-go-layout'
refs #3551
commit 45ab2f46c34ab5336768d095c0232f24efb6e04d
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Aug 14 15:45:27 2014 -0400
3551: Use mktemp for GOPATH. Update run-build-packages.sh with new Go recipes.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 5a331c1..65d0a9f 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -143,28 +143,26 @@ cd "$WORKSPACE/src-build-dir"
git checkout master
# Keep
-cd $WORKSPACE/services/keep
-./go.sh install keep
+export GOPATH=$(mktemp -d)
+mkdir -p "$GOPATH/src/git.curoverse.com"
+ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
+
+# Keep -> keepstore
+go get "git.curoverse.com/arvados.git/services/keepstore"
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/services/keep/bin/keep=/usr/bin/keep keep 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+build_and_scp_deb $GOPATH/bin/keepstore=/usr/bin/keep keep 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
# Keep proxy
-# First build the keepclient library
-cd $WORKSPACE/sdk/go
-./go.sh install arvados.org/keepclient
-
-# Then keepproxy
-cd $WORKSPACE/services/keep
-./go.sh install arvados.org/keepproxy
+# Keep -> keepproxy
+go get "git.curoverse.com/arvados.git/services/keepproxy"
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/services/keep/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+build_and_scp_deb $GOPATH/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
-# The crunchstat wrapper
-cd $WORKSPACE/services/crunch/crunchstat
-./go.sh install arvados.org/crunchstat
+# crunchstat
+go get "git.curoverse.com/arvados.git/services/crunchstat"
cd $WORKSPACE/debs
-build_and_scp_deb $WORKSPACE/services/crunch/crunchstat/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+build_and_scp_deb $GOPATH/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
# The Python SDK
cd $WORKSPACE/sdk/python
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 33a25be..bee8002 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -13,7 +13,7 @@ title () {
source /etc/profile.d/rvm.sh
echo $WORKSPACE
-export GOPATH="$HOME/gocode"
+export GOPATH=$(mktemp -d)
mkdir -p "$GOPATH/src/git.curoverse.com"
ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
commit 1ae46a07c7f2d1453a82a6dad0c546b224485e64
Merge: 95620c2 61ed4ec
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Aug 14 15:29:36 2014 -0400
Merge branch '3551-go-layout'
refs #3551
commit 61ed4ec992c01d2bb7c052983de9f917d3407edc
Merge: 2da54dc 95620c2
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Aug 14 15:25:04 2014 -0400
Merge branch 'master' into 3551-go-layout
commit 95620c2efb18c5a89cc88995f5c1a09a054d56a9
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Aug 14 15:24:09 2014 -0400
Make use of the realclean build.sh feature to build the docker images.
no issue #
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index 8ff8736..244c0d7 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -10,23 +10,24 @@ title () {
echo $WORKSPACE
+# DOCKER
+title "Starting docker build"
+
# clean up existing docker containers and images
docker.io stop $(docker.io ps -a -q)
docker.io rm $(docker.io ps -a -q)
docker.io rmi $(docker.io images -q)
-# clean up build files so we can re-build
-rm -f $WORKSPACE/docker/*-image
+# clean up the docker build environment
+cd "$WORKSPACE"
+cd docker
+./build.sh realclean
-rm -f docker/config.yml
+rm -f config.yml
# Get test config.yml file
-cp $HOME/docker/config.yml docker/
+cp $HOME/docker/config.yml .
-# DOCS
-title "Starting docker build"
-cd "$WORKSPACE"
-cd docker
./build.sh
ECODE=$?
commit 2da54dc5851ad6bf0837e344b880bda01fc466a0
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Aug 14 15:21:17 2014 -0400
3551: Dry up pass/fail checks with a checkexit() function.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index aeb5eb1..33a25be 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -26,13 +26,16 @@ rm -rf .site
# Make sure python-epydoc is installed or the next line won't do much good!
PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
-ECODE=$?
+checkexit() {
+ ECODE=$?
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! DOC BUILD FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
+ if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! $1 FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ fi
+}
+checkexit "Doc build"
title "DOC build complete"
# DOC linkchecker
@@ -41,13 +44,7 @@ cd "$WORKSPACE"
cd doc
bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! DOC LINKCHECKER FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
-
+checkexit "Doc linkchecker"
title "DOC linkchecker complete"
# API SERVER
@@ -98,13 +95,7 @@ bundle exec rake db:create
bundle exec rake db:setup
bundle exec rake test
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! API SERVER TESTS FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
-
+checkexit "API server tests"
title "API server tests complete"
# Install and test Go bits. keepstore must come before keepproxy and keepclient.
@@ -116,13 +107,7 @@ do
go get -t "git.curoverse.com/arvados.git/$dir" \
&& go test "git.curoverse.com/arvados.git/$dir"
- ECODE=$?
-
- if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! Keep TESTS FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
- fi
-
+ checkexit "$dir tests"
title "$dir tests complete"
done
@@ -138,13 +123,7 @@ echo $PATH
bundle exec rake test
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! WORKBENCH TESTS FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
-
+checkexit "Workbench tests"
title "Workbench tests complete"
# Python SDK
@@ -162,22 +141,12 @@ VENVDIR=$(mktemp -d)
virtualenv --setuptools "$VENVDIR"
"$VENVDIR/bin/python" setup.py test
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! Python SDK TESTS FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
+checkexit "Python SDK tests"
"$VENVDIR/bin/python" setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
"$VENVDIR/bin/pip" install dist/arvados-python-client-0.1.*.tar.gz
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! Python SDK install FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
+checkexit "Python SDK install"
cd "$WORKSPACE"
cd services/fuse
@@ -185,22 +154,12 @@ cd services/fuse
# We reuse $VENVDIR from the Python SDK tests above
"$VENVDIR/bin/python" setup.py test
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! Fuse TESTS FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
+checkexit "FUSE tests"
"$VENVDIR/bin/python" setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
"$VENVDIR/bin/pip" install dist/arvados_fuse-0.1.*.tar.gz
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! Fuse install FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
+checkexit "FUSE install"
title "Python SDK tests complete"
@@ -228,13 +187,7 @@ mkdir -p /tmp/keep
export KEEP_LOCAL_STORE=/tmp/keep
PYTHONPATH="$HOME/lib/python" bundle exec rake test
-ECODE=$?
-
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! SDK CLI TESTS FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
-
+checkexit "SDK CLI tests"
title "SDK CLI tests complete"
exit $EXITCODE
commit 3d678bcc827a44ff27152db3dc5f1b0084f1e101
Author: Tom Clegg <tom at curoverse.com>
Date: Thu Aug 14 13:13:18 2014 -0400
Update Go tests to new layout, rename keepstore, add missing tests for keepproxy and Go SDK libraries. refs #3551
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index ad2851d..aeb5eb1 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -13,6 +13,10 @@ title () {
source /etc/profile.d/rvm.sh
echo $WORKSPACE
+export GOPATH="$HOME/gocode"
+mkdir -p "$GOPATH/src/git.curoverse.com"
+ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git"
+
# DOCS
title "Starting DOC build"
cd "$WORKSPACE"
@@ -103,20 +107,25 @@ fi
title "API server tests complete"
-# Keep
-title "Starting Keep tests"
-cd "$WORKSPACE"
-cd services/keep/src/keep
-GOPATH=$HOME/gocode go test
+# Install and test Go bits. keepstore must come before keepproxy and keepclient.
+for dir in services/keepstore services/keepproxy sdk/go/arvadosclient sdk/go/keepclient sdk/go/streamer
+do
+ title "Starting $dir tests"
+ cd "$WORKSPACE"
-ECODE=$?
+ go get -t "git.curoverse.com/arvados.git/$dir" \
+ && go test "git.curoverse.com/arvados.git/$dir"
-if [[ "$ECODE" != "0" ]]; then
- title "!!!!!! Keep TESTS FAILED !!!!!!"
- EXITCODE=$(($EXITCODE + $ECODE))
-fi
+ ECODE=$?
+
+ if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! Keep TESTS FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ fi
+
+ title "$dir tests complete"
+done
-title "Keep tests complete"
# WORKBENCH
title "Starting workbench tests"
@@ -151,7 +160,7 @@ cd sdk/python
VENVDIR=$(mktemp -d)
virtualenv --setuptools "$VENVDIR"
-GOPATH="$HOME/gocode" "$VENVDIR/bin/python" setup.py test
+"$VENVDIR/bin/python" setup.py test
ECODE=$?
@@ -174,7 +183,7 @@ cd "$WORKSPACE"
cd services/fuse
# We reuse $VENVDIR from the Python SDK tests above
-GOPATH="$HOME/gocode" "$VENVDIR/bin/python" setup.py test
+"$VENVDIR/bin/python" setup.py test
ECODE=$?
commit aa4d046ce529b28b783c4b8044983d8676a2e7ca
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jul 18 10:05:35 2014 -0400
Python egg build fix for the sdist rotate command
Refs #3252
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 80a551a..ad2851d 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -160,7 +160,7 @@ if [[ "$ECODE" != "0" ]]; then
EXITCODE=$(($EXITCODE + $ECODE))
fi
-"$VENVDIR/bin/python" setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1
+"$VENVDIR/bin/python" setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
"$VENVDIR/bin/pip" install dist/arvados-python-client-0.1.*.tar.gz
ECODE=$?
@@ -183,7 +183,7 @@ if [[ "$ECODE" != "0" ]]; then
EXITCODE=$(($EXITCODE + $ECODE))
fi
-"$VENVDIR/bin/python" setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1
+"$VENVDIR/bin/python" setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1 --match .tar.gz
"$VENVDIR/bin/pip" install dist/arvados_fuse-0.1.*.tar.gz
ECODE=$?
commit 35f3cb05ac5c3319072b0f7f86b49f244e3cc3e7
Author: Brett Smith <brett at curoverse.com>
Date: Fri Jul 18 09:27:54 2014 -0400
Update Python package test procedure.
Refs #3252 simplifies the way we express Python package dependencies.
This lets us invoke tests directly from setup.py, so do that.
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
index 52fee76..80a551a 100755
--- a/jenkins/run-tests.sh
+++ b/jenkins/run-tests.sh
@@ -103,7 +103,7 @@ fi
title "API server tests complete"
-# Keep
+# Keep
title "Starting Keep tests"
cd "$WORKSPACE"
cd services/keep/src/keep
@@ -150,9 +150,8 @@ cd "$WORKSPACE"
cd sdk/python
VENVDIR=$(mktemp -d)
-virtualenv --setuptools --system-site-packages "$VENVDIR"
-"$VENVDIR/bin/pip" install -e .
-GOPATH="$HOME/gocode" "$VENVDIR/bin/python" -m unittest discover tests
+virtualenv --setuptools "$VENVDIR"
+GOPATH="$HOME/gocode" "$VENVDIR/bin/python" setup.py test
ECODE=$?
@@ -175,8 +174,7 @@ cd "$WORKSPACE"
cd services/fuse
# We reuse $VENVDIR from the Python SDK tests above
-"$VENVDIR/bin/pip" install -e .
-GOPATH="$HOME/gocode" "$VENVDIR/bin/python" -m unittest discover tests
+GOPATH="$HOME/gocode" "$VENVDIR/bin/python" setup.py test
ECODE=$?
commit 84dd5e80955f6fd3c5a26c0eb752c765e5011c5c
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Jul 7 15:11:37 2014 -0400
Make sure to bring the arvados debian packages up to the latest version
when deploying via jenkins.
no issue #
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
index c23aefc..6509e91 100755
--- a/jenkins/run-deploy.sh
+++ b/jenkins/run-deploy.sh
@@ -85,6 +85,19 @@ fi
title "Deploying API server complete"
+# Install updated debian packages
+title "Deploying updated arvados debian packages"
+
+ssh -p2222 $IDENTIFIER.arvadosapi.com -C "apt-get update && apt-get install arvados-src python-arvados-fuse python-arvados-python-client"
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! DEPLOYING DEBIAN PACKAGES FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ exit $EXITCODE
+fi
+
+title "Deploying updated arvados debian packages complete"
+
# Deploy Workbench
title "Deploying workbench"
cd "$WORKSPACE"
commit b67325fa722e4cf3d2e68cffa401ee1304866396
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Jul 1 15:38:00 2014 -0400
Make sure to exit with $EXITCODE
no issue #
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
index fc7f789..8ff8736 100755
--- a/jenkins/run-docker-tests.sh
+++ b/jenkins/run-docker-tests.sh
@@ -38,3 +38,4 @@ fi
title "docker build complete"
+exit $EXITCODE
commit 2e7d59676bf6098d0c3f277e450ffd38a0bf938d
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Jul 1 11:41:17 2014 -0400
Add docker test script.
no issue #
diff --git a/jenkins/run-docker-tests.sh b/jenkins/run-docker-tests.sh
new file mode 100755
index 0000000..fc7f789
--- /dev/null
+++ b/jenkins/run-docker-tests.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+
+EXITCODE=0
+
+COLUMNS=80
+
+title () {
+ printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
+}
+
+echo $WORKSPACE
+
+# clean up existing docker containers and images
+docker.io stop $(docker.io ps -a -q)
+docker.io rm $(docker.io ps -a -q)
+docker.io rmi $(docker.io images -q)
+
+# clean up build files so we can re-build
+rm -f $WORKSPACE/docker/*-image
+
+rm -f docker/config.yml
+
+# Get test config.yml file
+cp $HOME/docker/config.yml docker/
+
+# DOCS
+title "Starting docker build"
+cd "$WORKSPACE"
+cd docker
+./build.sh
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! docker BUILD FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "docker build complete"
+
commit 9e4360d3c323f6d9b25055b142b87295236bc648
Author: Ward Vandewege <ward at curoverse.com>
Date: Mon Jun 30 13:35:51 2014 -0400
Make sure the arvados-src package actually contains the latest source. Duh.
no issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 3891d97..5a331c1 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -128,13 +128,20 @@ if [[ ! -d "$WORKSPACE/src-build-dir" ]]; then
fi
cd "$WORKSPACE/src-build-dir"
-git fetch -a
-git checkout $GIT_REV
+# just in case, check out master
+git checkout master
+git pull
+# go into detached-head state
+git checkout `git log --format=format:%h -n1 .`
cd $WORKSPACE
cd $WORKSPACE/debs
build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH -x 'usr/local/arvados/src/.git*'"
+# clean up, check out master and step away from detached-head state
+cd "$WORKSPACE/src-build-dir"
+git checkout master
+
# Keep
cd $WORKSPACE/services/keep
./go.sh install keep
commit 8ada64214ea1a4cf63748b32a7edaa2317709f7d
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 13 11:16:38 2014 -0400
Remove superfluous directory from arvados-src path.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 9ff93a2..3891d97 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -123,8 +123,8 @@ ssh -p2222 $APTUSER@$APTSERVER mkdir tmp
# We use $WORKSPACE/src-build-dir as the clean directory from which to build the src package
if [[ ! -d "$WORKSPACE/src-build-dir" ]]; then
mkdir "$WORKSPACE/src-build-dir"
- cd "$WORKSPACE/src-build-dir"
- git clone https://github.com/curoverse/arvados.git
+ cd "$WORKSPACE"
+ git clone https://github.com/curoverse/arvados.git src-build-dir
fi
cd "$WORKSPACE/src-build-dir"
commit 17a7412040734fa4b906b1e6c5743ddc41bacf2f
Author: Ward Vandewege <ward at curoverse.com>
Date: Fri Jun 13 10:49:46 2014 -0400
Build a new arvados-src package.
No issue #
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index d9d3d18..9ff93a2 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -63,6 +63,8 @@ python setup.py egg_info -b ".$GIT_HASH" sdist upload
# Build debs for everything
+# Build arvados src deb package
+
build_and_scp_deb () {
PACKAGE=$1
PACKAGE_NAME=$2
@@ -117,6 +119,22 @@ fi
# Make sure our destination directory on $APTSERVER exists - prm can delete it when invoked improperly
ssh -p2222 $APTUSER@$APTSERVER mkdir tmp
+# Arvados-src
+# We use $WORKSPACE/src-build-dir as the clean directory from which to build the src package
+if [[ ! -d "$WORKSPACE/src-build-dir" ]]; then
+ mkdir "$WORKSPACE/src-build-dir"
+ cd "$WORKSPACE/src-build-dir"
+ git clone https://github.com/curoverse/arvados.git
+fi
+
+cd "$WORKSPACE/src-build-dir"
+git fetch -a
+git checkout $GIT_REV
+cd $WORKSPACE
+
+cd $WORKSPACE/debs
+build_and_scp_deb $WORKSPACE/src-build-dir/=/usr/local/arvados/src arvados-src 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH -x 'usr/local/arvados/src/.git*'"
+
# Keep
cd $WORKSPACE/services/keep
./go.sh install keep
commit 722ec7495f5ad00a404c31d8b72052be41a780da
Merge: be1ff42 874f78e
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Jun 5 14:27:34 2014 -0400
Merge branch 'new-git-hooks'
no issue #
commit 874f78e7f17a15da00e4aad0c9bc14620330dded
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Jun 5 14:25:47 2014 -0400
A few fixes:
a) clarify error message when doing a fast-forward merge rejection
b) reduce number of gratuitous newlines
c) do not test for issue numbers when a fast-forward merge is being
attempted, because that test will complain about every commit, obscuring
the real problem (the fast-forward merge)
diff --git a/git/hooks/coding-standards.sh b/git/hooks/coding-standards.sh
index d1b03dc..bc75a58 100755
--- a/git/hooks/coding-standards.sh
+++ b/git/hooks/coding-standards.sh
@@ -32,37 +32,42 @@ def check_message_format
merge_revs = `git rev-list --first-parent --min-parents=2 #{$oldrev}..#{$newrev}`.split("\n")
# single_revs = `git rev-list --first-parent --max-parents=1 #{$oldrev}..#{$newrev}`.split("\n")
broken = false
+ no_ff = false
merge_revs.each do |rev|
message = `git cat-file commit #{rev} | sed '1,/^$/d'`
if $wrong_way_merge_master.match(message)
- puts "\n[POLICY] This appears to be a merge from master into a feature\n"
- puts "\nbranch. Commits to master must merge from the feature\n"
- puts "\nbranch into master.\n\n"
+ puts "\n[POLICY] Only non-fast-forward merges into master are allowed. Please"
+ puts "reset your master branch:"
+ puts " git reset --hard origin/master"
+ puts "and then merge your branch with the --no-ff option:"
+ puts " git merge your-branch --no-ff\n"
+ puts "Remember to add a reference to an issue number in the merge commit!\n"
puts "\n******************************************************************\n"
- puts "\nOffending commit: #{rev}\n\n"
- puts "\nOffending commit message:\n\n"
+ puts "\nOffending commit: #{rev}\n"
+ puts "\nOffending commit message:\n"
puts message
puts "\n******************************************************************\n"
puts "\n\n"
broken = true
+ no_ff = true
elsif $pull_merge.match(message)
- puts "\n[POLICY] This appears to be a git pull merge of remote master into local\n"
- puts "\nmaster. In order to maintain a linear first-parent history of master,\n"
- puts "\nplease reset your branch and remerge or rebase using the latest master.\n"
+ puts "\n[POLICY] This appears to be a git pull merge of remote master into local"
+ puts "master. In order to maintain a linear first-parent history of master,"
+ puts "please reset your branch and remerge or rebase using the latest master.\n"
puts "\n******************************************************************\n"
- puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit: #{rev}\n"
puts "\nOffending commit message:\n\n"
puts message
puts "\n******************************************************************\n"
puts "\n\n"
broken = true
elsif not $merge_master.match(message) and not
- puts "\n[POLICY] This does not appear to be a merge of a feature\n"
- puts "\nbranch into master. Merges must follow the format\n"
- puts "\n\"Merge branch 'feature-branch'\".\n"
+ puts "\n[POLICY] This does not appear to be a merge of a feature"
+ puts "branch into master. Merges must follow the format"
+ puts "\"Merge branch 'feature-branch'\".\n"
puts "\n******************************************************************\n"
- puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit: #{rev}\n"
puts "\nOffending commit message:\n\n"
puts message
puts "\n******************************************************************\n"
@@ -74,10 +79,10 @@ def check_message_format
all_revs.each do |rev|
message = `git cat-file commit #{rev} | sed '1,/^$/d'`
if $broken_commit_message.match(message)
- puts "\n[POLICY] Rejected broken commit message for including boilerplate\n"
- puts "\ninstruction text.\n\n"
+ puts "\n[POLICY] Rejected broken commit message for including boilerplate"
+ puts "instruction text.\n"
puts "\n******************************************************************\n"
- puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit: #{rev}\n"
puts "\nOffending commit message:\n\n"
puts message
puts "\n******************************************************************\n"
@@ -85,11 +90,14 @@ def check_message_format
broken = true
end
- if not $refs_or_closes_or_no_issue.match(message)
- puts "\n[POLICY] All commits to master must include an issue using \"refs #\" or\n"
- puts "\n\"closes #\", or specify \"no issue #\"\n\n"
+ # Do not test when the commit is a no_ff merge (which will be rejected), because
+ # this test will complain about *every* commit in the merge otherwise, obscuring
+ # the real reason for the rejection (the no_ff merge)
+ if not no_ff and not $refs_or_closes_or_no_issue.match(message)
+ puts "\n[POLICY] All commits to master must include an issue using \"refs #\" or"
+ puts "\"closes #\", or specify \"no issue #\"\n"
puts "\n******************************************************************\n"
- puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit: #{rev}\n"
puts "\nOffending commit message:\n\n"
puts message
puts "\n******************************************************************\n"
commit e688effcdc8513fe5acc24116857c71edb306267
Merge: be1ff42 2147a53
Author: Ward Vandewege <ward at curoverse.com>
Date: Thu Jun 5 13:52:39 2014 -0400
Merge branch 'new-git-hooks' of git.curoverse.com:arvados-dev into new-git-hooks
commit 2147a538d2b64392a2c86e90a123c47e7efaac24
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jun 5 11:43:26 2014 -0400
Added mention of git reset to instructions for remerging/rebasing.
diff --git a/git/hooks/coding-standards.sh b/git/hooks/coding-standards.sh
index af98921..d1b03dc 100755
--- a/git/hooks/coding-standards.sh
+++ b/git/hooks/coding-standards.sh
@@ -49,7 +49,7 @@ def check_message_format
elsif $pull_merge.match(message)
puts "\n[POLICY] This appears to be a git pull merge of remote master into local\n"
puts "\nmaster. In order to maintain a linear first-parent history of master,\n"
- puts "\nplease remerge or rebase using the latest master.\n"
+ puts "\nplease reset your branch and remerge or rebase using the latest master.\n"
puts "\n******************************************************************\n"
puts "\nOffending commit: #{rev}\n\n"
puts "\nOffending commit message:\n\n"
commit 1815b1113c4983d8e2666c76ac23401086440c01
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jun 5 11:41:22 2014 -0400
Disallow commits that merge local and remote master, in order to maintain
linear history of master.
diff --git a/git/hooks/coding-standards.sh b/git/hooks/coding-standards.sh
index 18c050d..af98921 100755
--- a/git/hooks/coding-standards.sh
+++ b/git/hooks/coding-standards.sh
@@ -46,11 +46,21 @@ def check_message_format
puts "\n******************************************************************\n"
puts "\n\n"
broken = true
- elsif not $merge_master.match(message) and not $pull_merge.match(message)
+ elsif $pull_merge.match(message)
+ puts "\n[POLICY] This appears to be a git pull merge of remote master into local\n"
+ puts "\nmaster. In order to maintain a linear first-parent history of master,\n"
+ puts "\nplease remerge or rebase using the latest master.\n"
+ puts "\n******************************************************************\n"
+ puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit message:\n\n"
+ puts message
+ puts "\n******************************************************************\n"
+ puts "\n\n"
+ broken = true
+ elsif not $merge_master.match(message) and not
puts "\n[POLICY] This does not appear to be a merge of a feature\n"
- puts "\nbranch into master or a merge of master heads. Merges must follow\n"
- puts "\nthe format \"Merge branch 'feature-branch'\" or\n"
- puts "\n\"Merge branch 'master' of git.example.com:repository\".\n\n"
+ puts "\nbranch into master. Merges must follow the format\n"
+ puts "\n\"Merge branch 'feature-branch'\".\n"
puts "\n******************************************************************\n"
puts "\nOffending commit: #{rev}\n\n"
puts "\nOffending commit message:\n\n"
@@ -76,8 +86,8 @@ def check_message_format
end
if not $refs_or_closes_or_no_issue.match(message)
- puts "\n[POLICY] All commits to master must include an issue (\"refs #\" or\n"
- puts "\n\"closes #\") or specify \"no issue #\"\n\n"
+ puts "\n[POLICY] All commits to master must include an issue using \"refs #\" or\n"
+ puts "\n\"closes #\", or specify \"no issue #\"\n\n"
puts "\n******************************************************************\n"
puts "\nOffending commit: #{rev}\n\n"
puts "\nOffending commit message:\n\n"
commit 760afe4c383fc4136c949c26b0f3b14169e1e2ca
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jun 5 11:26:01 2014 -0400
Added pattern to permit commits that merge master heads.
diff --git a/git/hooks/coding-standards.sh b/git/hooks/coding-standards.sh
index f8bb91b..18c050d 100755
--- a/git/hooks/coding-standards.sh
+++ b/git/hooks/coding-standards.sh
@@ -23,6 +23,7 @@ $regex = /\[ref: (\d+)\]/
$broken_commit_message = /Please enter a commit message to explain why this merge is necessary/
$wrong_way_merge_master = /Merge( remote-tracking)? branch '([^\/]+\/)?master' into/
$merge_master = /Merge branch '[^']+'((?! into)| into master)/
+$pull_merge = /Merge branch 'master' of /
$refs_or_closes_or_no_issue = /(refs #|closes #|no issue #)/i
# enforced custom commit message format
@@ -45,10 +46,11 @@ def check_message_format
puts "\n******************************************************************\n"
puts "\n\n"
broken = true
- elsif not $merge_master.match(message)
+ elsif not $merge_master.match(message) and not $pull_merge.match(message)
puts "\n[POLICY] This does not appear to be a merge of a feature\n"
- puts "\nbranch into master, or it does not follow the required\n"
- puts "\nformat \"Merge branch 'feature-branch'\".\n\n"
+ puts "\nbranch into master or a merge of master heads. Merges must follow\n"
+ puts "\nthe format \"Merge branch 'feature-branch'\" or\n"
+ puts "\n\"Merge branch 'master' of git.example.com:repository\".\n\n"
puts "\n******************************************************************\n"
puts "\nOffending commit: #{rev}\n\n"
puts "\nOffending commit message:\n\n"
commit 6ed1a66a9ea26c74a60643c1c99191f2bb43a1c5
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date: Thu Jun 5 11:09:54 2014 -0400
Proposed new git hook policy: only applies to first-parent, enforces master is
first parent on all merges, enforces issue number (or explicitly stating no
issue number) on all master commits.
diff --git a/git/hooks/coding-standards.sh b/git/hooks/coding-standards.sh
index de8c49f..f8bb91b 100755
--- a/git/hooks/coding-standards.sh
+++ b/git/hooks/coding-standards.sh
@@ -21,43 +21,74 @@ puts "Enforcing Policies... \n(#{$refname}) (#{$oldrev[0,6]}) (#{$newrev[0,6]})"
$regex = /\[ref: (\d+)\]/
$broken_commit_message = /Please enter a commit message to explain why this merge is necessary/
-
-$merge_commit = /Merge branch/
-$merge_master_commit = /Merge branch 'master' (of|into)/
-$refs_or_closes_found = /(refs #|closes #)/i
-
-# If the next command has output, this is a non-merge commit.
-#`git rev-list --max-parents=1 --first-parent #{$oldrev}..#{$newrev}`
+$wrong_way_merge_master = /Merge( remote-tracking)? branch '([^\/]+\/)?master' into/
+$merge_master = /Merge branch '[^']+'((?! into)| into master)/
+$refs_or_closes_or_no_issue = /(refs #|closes #|no issue #)/i
# enforced custom commit message format
def check_message_format
- missed_revs = `git rev-list #{$oldrev}..#{$newrev}`.split("\n")
- missed_revs.each do |rev|
+ all_revs = `git rev-list --first-parent #{$oldrev}..#{$newrev}`.split("\n")
+ merge_revs = `git rev-list --first-parent --min-parents=2 #{$oldrev}..#{$newrev}`.split("\n")
+ # single_revs = `git rev-list --first-parent --max-parents=1 #{$oldrev}..#{$newrev}`.split("\n")
+ broken = false
+
+ merge_revs.each do |rev|
+ message = `git cat-file commit #{rev} | sed '1,/^$/d'`
+ if $wrong_way_merge_master.match(message)
+ puts "\n[POLICY] This appears to be a merge from master into a feature\n"
+ puts "\nbranch. Commits to master must merge from the feature\n"
+ puts "\nbranch into master.\n\n"
+ puts "\n******************************************************************\n"
+ puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit message:\n\n"
+ puts message
+ puts "\n******************************************************************\n"
+ puts "\n\n"
+ broken = true
+ elsif not $merge_master.match(message)
+ puts "\n[POLICY] This does not appear to be a merge of a feature\n"
+ puts "\nbranch into master, or it does not follow the required\n"
+ puts "\nformat \"Merge branch 'feature-branch'\".\n\n"
+ puts "\n******************************************************************\n"
+ puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit message:\n\n"
+ puts message
+ puts "\n******************************************************************\n"
+ puts "\n\n"
+ broken = true
+ end
+ end
+
+ all_revs.each do |rev|
message = `git cat-file commit #{rev} | sed '1,/^$/d'`
if $broken_commit_message.match(message)
- puts "\n[POLICY] Please avoid broken commit messages, rejected\n\n"
- puts "\n******************************************************************\n"
- puts "\nOffending commit: #{rev}\n\n"
- puts "\nOffending commit message:\n\n"
- puts message
- puts "\n******************************************************************\n"
- puts "\n\n"
- exit 1
+ puts "\n[POLICY] Rejected broken commit message for including boilerplate\n"
+ puts "\ninstruction text.\n\n"
+ puts "\n******************************************************************\n"
+ puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit message:\n\n"
+ puts message
+ puts "\n******************************************************************\n"
+ puts "\n\n"
+ broken = true
+ end
+
+ if not $refs_or_closes_or_no_issue.match(message)
+ puts "\n[POLICY] All commits to master must include an issue (\"refs #\" or\n"
+ puts "\n\"closes #\") or specify \"no issue #\"\n\n"
+ puts "\n******************************************************************\n"
+ puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit message:\n\n"
+ puts message
+ puts "\n******************************************************************\n"
+ puts "\n\n"
+ broken = true
end
- if $merge_commit.match(message) and
- not $merge_master_commit.match(message) and
- not $refs_or_closes_found.match(message)
- puts "\n[POLICY] Please make sure to refer to an issue in all branch merge commits, rejected\n\n"
- puts "\n******************************************************************\n"
- puts "\nOffending commit: #{rev}\n\n"
- puts "\nOffending commit message:\n\n"
- puts message
- puts "\n******************************************************************\n"
- puts "\n\n"
- exit 1
- end
+ end
+
+ if broken
+ exit 1
end
end
check_message_format
-
commit be1ff42eb4380029fa7118b9fb822fa88dfb1594
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed Jun 4 16:33:10 2014 -0400
Build crunchstat package.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
index 78ee2ee..d9d3d18 100755
--- a/jenkins/run-build-packages.sh
+++ b/jenkins/run-build-packages.sh
@@ -135,6 +135,12 @@ cd $WORKSPACE/services/keep
cd $WORKSPACE/debs
build_and_scp_deb $WORKSPACE/services/keep/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+# The crunchstat wrapper
+cd $WORKSPACE/services/crunch/crunchstat
+./go.sh install arvados.org/crunchstat
+cd $WORKSPACE/debs
+build_and_scp_deb $WORKSPACE/services/crunch/crunchstat/bin/crunchstat=/usr/bin/crunchstat crunchstat 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+
# The Python SDK
cd $WORKSPACE/sdk/python
sed -i'' -e "s:version='0.1':version='0.1.$GIT_HASH':" setup.py
commit 984848632f6d553a26661058c9b3561303e8d747
Author: Ward Vandewege <ward at curoverse.com>
Date: Tue Jun 3 11:25:10 2014 -0400
Initial commit of the scripts used in our Jenkins pipelines.
diff --git a/jenkins/run-build-packages.sh b/jenkins/run-build-packages.sh
new file mode 100755
index 0000000..78ee2ee
--- /dev/null
+++ b/jenkins/run-build-packages.sh
@@ -0,0 +1,177 @@
+#!/bin/bash
+
+EXITCODE=0
+CALL_PRM=0
+
+APTUSER=$1
+APTSERVER=$2
+
+if [[ "$APTUSER" == '' ]]; then
+ echo "Syntax: $0 <aptuser> <aptserver>"
+ exit 1
+fi
+
+if [[ "$APTSERVER" == '' ]]; then
+ echo "Syntax: $0 <aptuser> <aptserver>"
+ exit 1
+fi
+
+source /etc/profile.d/rvm.sh
+echo $WORKSPACE
+
+# Build arvados GEM
+echo "Build and publish ruby gem"
+cd "$WORKSPACE"
+cd sdk/ruby
+# clean up old gems
+rm -f arvados-*gem
+gem build arvados.gemspec
+# publish new gem
+gem push arvados-*gem
+
+# Build arvados-cli GEM
+echo "Build and publish ruby gem"
+cd "$WORKSPACE"
+cd sdk/cli
+# clean up old gems
+rm -f arvados-cli*gem
+gem build arvados-cli.gemspec
+# publish new gem
+gem push arvados-cli*gem
+
+# Build arvados-python-client Python package
+echo "Build and publish arvados-python-client package"
+cd "$WORKSPACE"
+
+GIT_HASH=`git log --format=format:%ct.%h -n1 .`
+
+cd sdk/python
+
+# We mess with this file below, reset it here
+git checkout setup.py
+
+# Make sure only to use sdist - that's the only format pip can deal with (sigh)
+python setup.py egg_info -b ".$GIT_HASH" sdist upload
+
+cd ../../services/fuse
+
+# We mess with this file below, reset it here
+git checkout setup.py
+
+# Make sure only to use sdist - that's the only format pip can deal with (sigh)
+python setup.py egg_info -b ".$GIT_HASH" sdist upload
+
+# Build debs for everything
+
+build_and_scp_deb () {
+ PACKAGE=$1
+ PACKAGE_NAME=$2
+ # Put spaces in $3 and you will regret it. Despite the use of arrays below.
+ # Because, bash sucks.
+ VENDOR=${3// /_}
+ PACKAGE_TYPE=$4
+ EXTRA_ARGUMENTS=$5
+
+ if [[ "$PACKAGE_NAME" == "" ]]; then
+ PACKAGE_NAME=$PACKAGE
+ fi
+
+ if [[ "$PACKAGE_TYPE" == "" ]]; then
+ PACKAGE_TYPE='python'
+ fi
+
+ COMMAND_ARR=("fpm" "-s" "$PACKAGE_TYPE" "-t" "deb")
+
+ if [[ "$PACKAGE_NAME" != "$PACKAGE" ]]; then
+ COMMAND_ARR+=('-n' "$PACKAGE_NAME")
+ fi
+
+ if [[ "$VENDOR" != "" ]]; then
+ COMMAND_ARR+=('--vendor' "$VENDOR")
+ fi
+ for a in $EXTRA_ARGUMENTS; do
+ COMMAND_ARR+=("$a")
+ done
+
+ COMMAND_ARR+=("$PACKAGE")
+
+ FPM_RESULTS=$(${COMMAND_ARR[@]})
+ FPM_EXIT_CODE=$?
+ echo ${COMMAND_ARR[@]}
+ if [[ ! $FPM_RESULTS =~ "File already exists" ]]; then
+ if [[ "$FPM_EXIT_CODE" != "0" ]]; then
+ echo "Error building debian package for $1:\n $FPM_RESULTS"
+ else
+ scp -P2222 $PACKAGE_NAME*.deb $APTUSER@$APTSERVER:tmp/
+ CALL_PRM=1
+ fi
+ else
+ echo "Debian package for $1 exists, not rebuilding"
+ fi
+}
+
+if [[ ! -d "$WORKSPACE/debs" ]]; then
+ mkdir -p $WORKSPACE/debs
+fi
+
+# Make sure our destination directory on $APTSERVER exists - prm can delete it when invoked improperly
+ssh -p2222 $APTUSER@$APTSERVER mkdir tmp
+
+# Keep
+cd $WORKSPACE/services/keep
+./go.sh install keep
+cd $WORKSPACE/debs
+build_and_scp_deb $WORKSPACE/services/keep/bin/keep=/usr/bin/keep keep 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+
+# Keep proxy
+
+# First build the keepclient library
+cd $WORKSPACE/sdk/go
+./go.sh install arvados.org/keepclient
+
+# Then keepproxy
+cd $WORKSPACE/services/keep
+./go.sh install arvados.org/keepproxy
+cd $WORKSPACE/debs
+build_and_scp_deb $WORKSPACE/services/keep/bin/keepproxy=/usr/bin/keepproxy keepproxy 'Curoverse, Inc.' 'dir' "-v 0.1.$GIT_HASH"
+
+# The Python SDK
+cd $WORKSPACE/sdk/python
+sed -i'' -e "s:version='0.1':version='0.1.$GIT_HASH':" setup.py
+
+cd $WORKSPACE/debs
+
+# Please resist the temptation to add --no-python-fix-name to the fpm call here
+# (which would remove the python- prefix from the package name), because this
+# package is a dependency of arvados-fuse, and fpm can not omit the python-
+# prefix from only one of the dependencies of a package... Maybe I could
+# whip up a patch and send it upstream, but that will be for another day. Ward,
+# 2014-05-15
+build_and_scp_deb $WORKSPACE/sdk/python python-arvados-python-client 'Curoverse, Inc.' 'python' "-v 0.1.${GIT_HASH}"
+
+# The FUSE driver
+cd $WORKSPACE/services/fuse
+sed -i'' -e "s:version='0.1':version='0.1.$GIT_HASH':" setup.py
+
+
+cd $WORKSPACE/debs
+
+# Please seem comment about --no-python-fix-name above; we stay consistent and do
+# not omit the python- prefix first.
+build_and_scp_deb $WORKSPACE/services/fuse python-arvados-fuse 'Curoverse, Inc.' 'python' "-v 0.1.${GIT_HASH}"
+
+# A few dependencies
+build_and_scp_deb python-gflags
+build_and_scp_deb pyvcf
+build_and_scp_deb google-api-python-client
+build_and_scp_deb httplib2
+build_and_scp_deb ws4py
+build_and_scp_deb virtualenv
+
+# Finally, publish the packages, if necessary
+if [[ "$CALL_PRM" != "0" ]]; then
+ ssh -p2222 $APTUSER@$APTSERVER -t "cd /var/www/$APTSERVER; /usr/local/rvm/bin/rvm default do prm --type deb -p . --component main --release wheezy --arch amd64 -d /home/$APTUSER/tmp/ --gpg 1078ECD7"
+else
+ echo "No new packages generated. No PRM run necessary."
+fi
+
diff --git a/jenkins/run-deploy.sh b/jenkins/run-deploy.sh
new file mode 100755
index 0000000..c23aefc
--- /dev/null
+++ b/jenkins/run-deploy.sh
@@ -0,0 +1,113 @@
+#!/bin/bash
+
+IDENTIFIER=$1
+DEPLOY_REPO=$2
+
+if [[ "$IDENTIFIER" == '' ]]; then
+ echo "Syntax: $0 <identifier> <deploy_repo_name>"
+ exit 1
+fi
+
+if [[ "$DEPLOY_REPO" == '' ]]; then
+ echo "Syntax: $0 <identifier> <deploy_repo_name>"
+ exit 1
+fi
+
+EXITCODE=0
+
+COLUMNS=80
+
+title () {
+ printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
+}
+
+# We only install capistrano in dev mode
+export RAILS_ENV=development
+
+source /etc/profile.d/rvm.sh
+echo $WORKSPACE
+
+# Weirdly, jenkins/rvm ties itself in a knot.
+rvm use default
+
+# Just say what version of ruby we're running
+ruby --version
+
+function ensure_symlink() {
+ if [[ ! -L $WORKSPACE/$1 ]]; then
+ ln -s $WORKSPACE/$DEPLOY_REPO/$1 $WORKSPACE/$1
+ fi
+}
+
+# Check out/update the $DEPLOY_REPO repository
+if [[ ! -d $DEPLOY_REPO ]]; then
+ mkdir $DEPLOY_REPO
+ git clone git at git.curoverse.com:$DEPLOY_REPO.git
+else
+ cd $DEPLOY_REPO
+ git pull
+fi
+
+# Make sure the necessary symlinks are in place
+cd "$WORKSPACE"
+ensure_symlink "apps/workbench/Capfile.workbench.$IDENTIFIER"
+ensure_symlink "apps/workbench/config/deploy.common.rb"
+ensure_symlink "apps/workbench/config/deploy.curoverse.rb"
+ensure_symlink "apps/workbench/config/deploy.workbench.$IDENTIFIER.rb"
+
+ensure_symlink "services/api/Capfile.$IDENTIFIER"
+ensure_symlink "services/api/config/deploy.common.rb"
+ensure_symlink "services/api/config/deploy.$IDENTIFIER.rb"
+
+# Deploy API server
+title "Deploying API server"
+cd "$WORKSPACE"
+cd services/api
+
+bundle install --deployment
+
+# make sure we do not print the output of config:check
+sed -i'' -e "s/RAILS_ENV=production #{rake} config:check/RAILS_ENV=production QUIET=true #{rake} config:check/" $WORKSPACE/$DEPLOY_REPO/services/api/config/deploy.common.rb
+
+bundle exec cap deploy -f Capfile.$IDENTIFIER
+
+ECODE=$?
+
+# restore unaltered deploy.common.rb
+cd $WORKSPACE/$DEPLOY_REPO
+git checkout services/api/config/deploy.common.rb
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! DEPLOYING API SERVER FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ exit $EXITCODE
+fi
+
+title "Deploying API server complete"
+
+# Deploy Workbench
+title "Deploying workbench"
+cd "$WORKSPACE"
+cd apps/workbench
+bundle install --deployment
+
+# make sure we do not print the output of config:check
+sed -i'' -e "s/RAILS_ENV=production #{rake} config:check/RAILS_ENV=production QUIET=true #{rake} config:check/" $WORKSPACE/$DEPLOY_REPO/apps/workbench/config/deploy.common.rb
+
+bundle exec cap deploy -f Capfile.workbench.$IDENTIFIER
+
+ECODE=$?
+
+# restore unaltered deploy.common.rb
+cd $WORKSPACE/$DEPLOY_REPO
+git checkout apps/workbench/config/deploy.common.rb
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! DEPLOYING WORKBENCH FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+ exit $EXITCODE
+fi
+
+title "Deploying workbench complete"
+
+exit $EXITCODE
diff --git a/jenkins/run-tapestry-tests.sh b/jenkins/run-tapestry-tests.sh
new file mode 100755
index 0000000..851a81d
--- /dev/null
+++ b/jenkins/run-tapestry-tests.sh
@@ -0,0 +1,54 @@
+#!/bin/bash
+
+EXITCODE=0
+
+COLUMNS=80
+
+title () {
+ printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
+}
+
+source /etc/profile.d/rvm.sh
+
+# This shouldn't really be necessary... but the jenkins/rvm integration seems a
+# bit wonky occasionally.
+rvm use ree
+
+echo $WORKSPACE
+
+# Tapestry
+title "Starting tapestry tests"
+cd "$WORKSPACE"
+
+# There are a few submodules
+git submodule init && git submodule update
+
+# Use sqlite for testing
+sed -i'' -e "s:mysql:sqlite3:" Gemfile
+
+# Tapestry is not set up yet to use --deployment
+#bundle install --deployment
+bundle install
+
+rm -f config/database.yml
+rm -f config/environments/test.rb
+cp $HOME/tapestry/test.rb config/environments/
+cp $HOME/tapestry/database.yml config/
+
+export RAILS_ENV=test
+
+bundle exec rake db:drop
+bundle exec rake db:create
+bundle exec rake db:setup
+bundle exec rake test
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! TAPESTRY TESTS FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "Tapestry tests complete"
+
+exit $EXITCODE
diff --git a/jenkins/run-tests.sh b/jenkins/run-tests.sh
new file mode 100755
index 0000000..52fee76
--- /dev/null
+++ b/jenkins/run-tests.sh
@@ -0,0 +1,233 @@
+#!/bin/bash
+
+EXITCODE=0
+
+COLUMNS=80
+
+ARVADOS_API_HOST=qr1hi.arvadosapi.com
+
+title () {
+ printf "\n%*s\n\n" $(((${#title}+$COLUMNS)/2)) "********** $1 **********"
+}
+
+source /etc/profile.d/rvm.sh
+echo $WORKSPACE
+
+# DOCS
+title "Starting DOC build"
+cd "$WORKSPACE"
+cd doc
+bundle install --deployment
+rm -rf .site
+# Make sure python-epydoc is installed or the next line won't do much good!
+PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake generate baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! DOC BUILD FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "DOC build complete"
+
+# DOC linkchecker
+title "Starting DOC linkchecker"
+cd "$WORKSPACE"
+cd doc
+bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! DOC LINKCHECKER FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "DOC linkchecker complete"
+
+# API SERVER
+title "Starting API server tests"
+cd "$WORKSPACE"
+cd services/api
+bundle install --deployment
+
+rm -f config/database.yml
+rm -f config/environments/test.rb
+cp config/environments/test.rb.example config/environments/test.rb
+
+# Get test database config
+cp $HOME/arvados-api-server/database.yml config/
+# Get test application.yml file
+cp $HOME/arvados-api-server/application.yml config/
+
+# Fill in a random secret_token and blob_signing_key for testing
+SECRET_TOKEN=`echo 'puts rand(2**512).to_s(36)' |ruby`
+BLOB_SIGNING_KEY=`echo 'puts rand(2**512).to_s(36)' |ruby`
+
+sed -i'' -e "s:SECRET_TOKEN:$SECRET_TOKEN:" config/application.yml
+sed -i'' -e "s:BLOB_SIGNING_KEY:$BLOB_SIGNING_KEY:" config/application.yml
+
+export RAILS_ENV=test
+
+# Set up empty git repo (for git tests)
+GITDIR=$WORKSPACE/tmpgit
+rm -rf $GITDIR
+mkdir $GITDIR
+sed -i'' -e "s:/var/cache/git:$GITDIR:" config/application.default.yml
+
+rm -rf $GITDIR
+mkdir -p $GITDIR/test
+cd $GITDIR/test
+/usr/bin/git init
+/usr/bin/git config user.email "jenkins at ci.curoverse.com"
+/usr/bin/git config user.name "Jenkins, CI"
+touch tmp
+/usr/bin/git add tmp
+/usr/bin/git commit -m 'initial commit'
+
+cd "$WORKSPACE"
+cd services/api
+
+bundle exec rake db:drop
+bundle exec rake db:create
+bundle exec rake db:setup
+bundle exec rake test
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! API SERVER TESTS FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "API server tests complete"
+
+# Keep
+title "Starting Keep tests"
+cd "$WORKSPACE"
+cd services/keep/src/keep
+GOPATH=$HOME/gocode go test
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! Keep TESTS FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "Keep tests complete"
+
+# WORKBENCH
+title "Starting workbench tests"
+cd "$WORKSPACE"
+cd apps/workbench
+bundle install --deployment
+
+echo $PATH
+
+
+bundle exec rake test
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! WORKBENCH TESTS FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "Workbench tests complete"
+
+# Python SDK
+title "Starting Python SDK tests"
+cd "$WORKSPACE"
+cd sdk/cli
+bundle install --deployment
+
+# Set up Python SDK and dependencies
+
+cd "$WORKSPACE"
+cd sdk/python
+
+VENVDIR=$(mktemp -d)
+virtualenv --setuptools --system-site-packages "$VENVDIR"
+"$VENVDIR/bin/pip" install -e .
+GOPATH="$HOME/gocode" "$VENVDIR/bin/python" -m unittest discover tests
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! Python SDK TESTS FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+"$VENVDIR/bin/python" setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1
+"$VENVDIR/bin/pip" install dist/arvados-python-client-0.1.*.tar.gz
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! Python SDK install FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+cd "$WORKSPACE"
+cd services/fuse
+
+# We reuse $VENVDIR from the Python SDK tests above
+"$VENVDIR/bin/pip" install -e .
+GOPATH="$HOME/gocode" "$VENVDIR/bin/python" -m unittest discover tests
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! Fuse TESTS FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+"$VENVDIR/bin/python" setup.py egg_info -b ".$(git log --format=format:%ct.%h -n1 .)" sdist rotate --keep=1
+"$VENVDIR/bin/pip" install dist/arvados_fuse-0.1.*.tar.gz
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! Fuse install FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "Python SDK tests complete"
+
+# Clean up $VENVDIR
+rm -rf "$VENVDIR"
+
+# The CLI SDK tests require a working API server, so let's skip those for now.
+exit $EXITCODE
+
+# CLI SDK
+title "Starting SDK CLI tests"
+cd "$WORKSPACE"
+cd sdk/cli
+bundle install --deployment
+
+# Set up Python SDK and dependencies
+cd ../python
+rm -rf $HOME/lib/python
+mkdir $HOME/lib/python
+PYTHONPATH="$HOME/lib/python" easy_install --install-dir=$HOME/lib/python --upgrade google-api-python-client
+PYTHONPATH="$HOME/lib/python" python setup.py install --home=$HOME
+
+cd ../cli
+mkdir -p /tmp/keep
+export KEEP_LOCAL_STORE=/tmp/keep
+PYTHONPATH="$HOME/lib/python" bundle exec rake test
+
+ECODE=$?
+
+if [[ "$ECODE" != "0" ]]; then
+ title "!!!!!! SDK CLI TESTS FAILED !!!!!!"
+ EXITCODE=$(($EXITCODE + $ECODE))
+fi
+
+title "SDK CLI tests complete"
+
+exit $EXITCODE
commit e1f0d735e521b0487b2cd8bb179054932ca58201
Author: Ward Vandewege <ward at jhvc.com>
Date: Wed May 28 12:01:16 2014 -0400
Only check master, because bad things happen otherwise.
diff --git a/git/hooks/coding-standards.sh b/git/hooks/coding-standards.sh
index 610abe8..de8c49f 100755
--- a/git/hooks/coding-standards.sh
+++ b/git/hooks/coding-standards.sh
@@ -13,9 +13,8 @@ $oldrev = ARGV[1]
$newrev = ARGV[2]
$user = ENV['USER']
-# Only enforce policy on the master branch -- on second thought, do it on all
-# branches so that people don't get surprised at merge-to-master time.
-#exit 0 if $refname != 'refs/heads/master'
+# Only enforce policy on the master branch
+exit 0 if $refname != 'refs/heads/master'
puts "Enforcing Policies... \n(#{$refname}) (#{$oldrev[0,6]}) (#{$newrev[0,6]})"
commit fcfdab3ca5bc40939aed45fcabf436ff3c3a67bb
Author: Ward Vandewege <ward at curoverse.com>
Date: Wed May 28 11:45:43 2014 -0400
Coding standards hook, initial commit.
diff --git a/git/hooks/coding-standards.sh b/git/hooks/coding-standards.sh
new file mode 100755
index 0000000..610abe8
--- /dev/null
+++ b/git/hooks/coding-standards.sh
@@ -0,0 +1,64 @@
+#!/usr/bin/env ruby
+
+# This script can be installed as a git update hook.
+
+# It can also be installed as a gitolite 'hooklet' in the
+# hooks/common/update.secondary.d/ directory.
+
+# NOTE: this script runs under the same assumptions as the 'update' hook, so
+# the starting directory must be maintained and arguments must be passed on.
+
+$refname = ARGV[0]
+$oldrev = ARGV[1]
+$newrev = ARGV[2]
+$user = ENV['USER']
+
+# Only enforce policy on the master branch -- on second thought, do it on all
+# branches so that people don't get surprised at merge-to-master time.
+#exit 0 if $refname != 'refs/heads/master'
+
+puts "Enforcing Policies... \n(#{$refname}) (#{$oldrev[0,6]}) (#{$newrev[0,6]})"
+
+$regex = /\[ref: (\d+)\]/
+
+$broken_commit_message = /Please enter a commit message to explain why this merge is necessary/
+
+$merge_commit = /Merge branch/
+$merge_master_commit = /Merge branch 'master' (of|into)/
+$refs_or_closes_found = /(refs #|closes #)/i
+
+# If the next command has output, this is a non-merge commit.
+#`git rev-list --max-parents=1 --first-parent #{$oldrev}..#{$newrev}`
+
+# enforced custom commit message format
+def check_message_format
+ missed_revs = `git rev-list #{$oldrev}..#{$newrev}`.split("\n")
+ missed_revs.each do |rev|
+ message = `git cat-file commit #{rev} | sed '1,/^$/d'`
+ if $broken_commit_message.match(message)
+ puts "\n[POLICY] Please avoid broken commit messages, rejected\n\n"
+ puts "\n******************************************************************\n"
+ puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit message:\n\n"
+ puts message
+ puts "\n******************************************************************\n"
+ puts "\n\n"
+ exit 1
+ end
+ if $merge_commit.match(message) and
+ not $merge_master_commit.match(message) and
+ not $refs_or_closes_found.match(message)
+ puts "\n[POLICY] Please make sure to refer to an issue in all branch merge commits, rejected\n\n"
+ puts "\n******************************************************************\n"
+ puts "\nOffending commit: #{rev}\n\n"
+ puts "\nOffending commit message:\n\n"
+ puts message
+ puts "\n******************************************************************\n"
+ puts "\n\n"
+ exit 1
+ end
+ end
+end
+
+check_message_format
+
-----------------------------------------------------------------------
hooks/post-receive
--
More information about the arvados-commits
mailing list