<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div markdown-here-wrapper-content-modified="true" style=""
data-md-original="%3Cdiv%20class%3D%22moz-cite-prefix%22%3EOn%2004%2F07%2F2014%2001%3A42%20PM%2C%20Tom%20Clegg%20wrote%3A%3Cbr%3E%3C%2Fdiv%3E%3Cblockquote%20cite%3D%22mid%3ACAOtdHJGFzp5pEJ7fmoc8Z9kUbhAMZ_kLHfRBGBazL_LK0kuoOA%40mail.gmail.com%22%20type%3D%22cite%22%3E%3Cpre%20wrap%3D%22%22%3EOn%20Mon%2C%20Apr%207%2C%202014%20at%2010%3A29%20AM%2C%20Brett%20Smith%20%26lt%3Bbrett%40curoverse.com%26gt%3B%20wrote%3A%0A%3C%2Fpre%3E%3Cblockquote%20type%3D%22cite%22%3E%3Cpre%20wrap%3D%22%22%3EOne%20quirk%20about%20the%20current%20recipe%20is%20that%20the%20Docker%20image%20has%20to%20be%20run%20in%0Aprivileged%20mode.%20arv-crunch-job%20automatically%20mounts%20Keep%2C%20and%20Docker's%0Adefault%20security%20policy%20will%20prohibit%20this%20mount.%20Longer-term%2C%20I%20wonder%20if%0Ait%20would%20make%20sense%20for%20the%20parent%20compute%20node%20to%20mount%20Keep%20and%20make%20it%0Aavailable%20to%20the%20job%20container%20as%20a%20volume.%0A%3C%2Fpre%3E%3C%2Fblockquote%3E%3Cpre%20wrap%3D%
22%22%3E%0AOne%20complication%20here%20is%20that%20the%20arv-mount%20process%20has%20to%20use%20the%0Aper-job%20auth%20token%3A%20keeping%20arv-mount%20alive%20longer%20should%20give%20us%0Abetter%20performance%20in%20general%2C%20but%20we%20don't%20want%20to%20share%20an%20arv-mount%0Aprocess%20between%20consecutive%20jobs%20running%20on%20the%20same%20compute%20node.%0A%0AIt's%20possible%20for%20arv-mount%20to%20inspect%20the%20credentials%20in%20the%20calling%0Aprocess's%20environment%2C%20which%20could%20give%20us%20a%20safety%20net%20in%20case%0Aarv-mount%20doesn't%20get%20set%20up%20%2F%20taken%20down%20correctly%20between%20jobs.%0A%22Calling%20process's%20credentials%20don't%20match%20what%20I'm%20using%20-%26gt%3B%20exit.%22%3C%2Fpre%3E%3C%2Fblockquote%3EHmm.%C2%A0%20Noodling%20over%20this%2C%20I've%20started%20to%20think%20that%20we%20maybe%20can%20and%20should%20move%20Docker%20to%20a%20lower%20level.%C2%A0%20Rather%20than%20running%20crunch-job%20in%20a%20container%2C%20cr
unch-job%20should%20invoke%20Docker%20to%20run%20the%20script%20in%20a%20container%20(i.e.%2C%20call%20%60arv-mount%20%E2%80%A6%20--exec%20docker%20run%20arvados%2Fjobs%20%E2%80%A6%20script%60.%C2%A0%20That%20would%20be%20more%20consistent%20with%20our%20goal%20of%20making%20the%20smallest%20possible%20container%3A%3Cbr%3E%3Cblockquote%20type%3D%22cite%22%3E%0A%3Cblockquote%20type%3D%22cite%22%3E%3Cpre%20wrap%3D%22%22%3EAfter%20thinking%20it%20over%20some%2C%20I%20do%20think%20that%20this%20is%20the%20right%20layer%20to%0Aintroduce%20Docker%20at%3B%20or%20at%20least%2C%20this%20is%20the%20layer%20where%20we%20need%20to%20be%20able%0Ato%20prepare%20and%20run%20customized%20Docker%20images.%20If%20we%20try%20to%20make%20this%20image's%0Aresponsibilities%20part%20of%20a%20larger%20Docker%20container%2C%20we're%20introducing%20space%0Awhere%20users%20have%20to%20worry%20that%20their%20customizations%20might%20interfere%20with%0AArvados'%20general%20operations%2C%20which%20is%20something%2
0we%20should%20avoid.%20We%20can%0Astill%20containerize%20the%20compute%20nodes%3B%20we%20should%20just%20keep%20it%20separate%20from%0Athis%20work.%0A%3C%2Fpre%3E%3C%2Fblockquote%3E%0A%3Cpre%20wrap%3D%22%22%3EAbsolutely.%20We%20want%20minimal%20restrictions%2Frequirements%20for%20these%0Aper-job%20containers.%20What's%20the%20cleanest%20way%20to%20provide%20the%20Arvados%0ASDKs%20without%20imposing%20on%20user-provided%20images%3F%20Apart%20from%20the%20SDKs%0Aused%20in%20the%20job%2C%20there%20should%20be%20no%20need%20for%20any%20Arvados%20stuff%20at%20all%0Ain%20the%20containers%20where%20job%20tasks%20run%2C%20right%3F%3Cbr%3E%3C%2Fpre%3E%3C%2Fblockquote%3ERight%20now%20the%20arvados%2Fjobs%20Docker%20image%20is%20meant%20to%20be%20the%20smallest%20possible%20base%20so%20that%20people%20who%20want%20to%20containerize%20their%20Crunch%20scripts%20can%20write%20a%20Dockerfile%20that%20starts%20%60FROM%20arvados%2Fjobs%60%20and%20then%20mixes%20in%20what%20they%20need.%C2%A0%20It%20
just%20has%20the%20SDKs.%C2%A0%20If%20we%20try%20to%20run%20crunch-job%20inside%20the%20container%2C%20we'll%20have%20to%20add%20SLURM%20and%20maybe%20some%20other%20support%20tools%20that%20crunch-job%20itself%20needs.%0A%3Cblockquote%20cite%3D%22mid%3ACAOtdHJGFzp5pEJ7fmoc8Z9kUbhAMZ_kLHfRBGBazL_LK0kuoOA%40mail.gmail.com%22%20type%3D%22cite%22%3E%3Cblockquote%20type%3D%22cite%22%3E%3Cpre%20wrap%3D%22%22%3EIf%20we'd%20like%20to%20make%20this%20usable%20as%20quickly%20as%20possible%2C%20probably%20the%20best%0Away%20to%20do%20that%20would%20be%20to%20add%20Dockerfiles%20as%20a%20new%20installation%20path%20in%0Acrunch-job.%20Much%20like%20it%20looks%20for%20install%20scripts%20in%20the%20repository%2C%20if%20it%0Asees%20a%20Dockerfile%2C%20it%20should%20build%20that%20image%20and%20run%20the%20job%20inside%20it.%20The%0Acode%20for%20that%20is%20relatively%20straightforward%3B%20the%20hard%20part%20of%20this%20would%20be%0Asetting%20up%20Docker%20on%20the%20compute%20nodes%2C%20and%20thi
nking%20through%20the%0Aconfiguration%20to%20make%20it%20robust.%0A%3C%2Fpre%3E%3C%2Fblockquote%3E%3Cpre%20wrap%3D%22%22%3E%0AWouldn't%20it%20be%20easier%20(and%20more%20reproducibility-friendly)%20to%20do%20the%0Acontainer-building%20before%20the%20job%20is%20submitted%2C%20and%20provide%20a%0Acontainer%20ID%20or%20collection%20hash%20in%20runtime_constraints%20when%20submitting%0Athe%20job%3F%3C%2Fpre%3E%3C%2Fblockquote%3EAs%20we%20discussed%2C%20this%20is%20easier%20in%20part%20because%20it%20does%20less%3A%20it%20doesn't%20specify%20how%20users%20get%20images%20into%20the%20system.%C2%A0%20But%20still%2C%20your%20point%20stands%2C%20and%20I'm%20happy%20to%20adopt%20this%20approach%20and%20leave%20that%20question%20for%20later.%3Cbr%3E%3Cbr%3EThanks%2C%3Cbr%3E%3Cbr%3E"
class="markdown-here-wrapper" data-md-url="null"
id="markdown-here-wrapper-624518">
<p style="margin: 1.2em 0px ! important;">On 04/07/2014 01:42 PM,
Tom Clegg wrote:</p>
<p style="margin: 1.2em 0px ! important;"></p>
<div class="markdown-here-exclude">
<p></p>
<blockquote
cite="mid:CAOtdHJGFzp5pEJ7fmoc8Z9kUbhAMZ_kLHfRBGBazL_LK0kuoOA@mail.gmail.com"
type="cite">
<pre wrap="">On Mon, Apr 7, 2014 at 10:29 AM, Brett Smith <a class="moz-txt-link-rfc2396E" href="mailto:brett@curoverse.com"><brett@curoverse.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">One quirk about the current recipe is that the Docker image has to be run in
privileged mode. arv-crunch-job automatically mounts Keep, and Docker's
default security policy will prohibit this mount. Longer-term, I wonder if
it would make sense for the parent compute node to mount Keep and make it
available to the job container as a volume.
</pre>
</blockquote>
<pre wrap="">One complication here is that the arv-mount process has to use the
per-job auth token: keeping arv-mount alive longer should give us
better performance in general, but we don't want to share an arv-mount
process between consecutive jobs running on the same compute node.
It's possible for arv-mount to inspect the credentials in the calling
process's environment, which could give us a safety net in case
arv-mount doesn't get set up / taken down correctly between jobs.
"Calling process's credentials don't match what I'm using -> exit."</pre>
</blockquote>
<p></p>
</div>
<p style="margin: 1.2em 0px ! important;"></p>
<p style="margin: 1.2em 0px ! important;">Hmm. Noodling over this,
I’ve started to think that we maybe can and should move Docker
to a lower level. Rather than running crunch-job in a container,
crunch-job should invoke Docker to run the script in a container
(i.e., call <code style="font-size: 0.85em; font-family:
Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em;
padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid
rgb(234, 234, 234); background-color: rgb(248, 248, 248);
border-radius: 3px 3px 3px 3px; display: inline;">arv-mount …
--exec docker run arvados/jobs … script</code>. That would
address the privilege issue immediately, and be more consistent
with our goal of making the smallest possible container:</p>
<p style="margin: 1.2em 0px ! important;"></p>
<div class="markdown-here-exclude">
<p></p>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">After thinking it over some, I do think that this is the right layer to
introduce Docker at; or at least, this is the layer where we need to be able
to prepare and run customized Docker images. If we try to make this image's
responsibilities part of a larger Docker container, we're introducing space
where users have to worry that their customizations might interfere with
Arvados' general operations, which is something we should avoid. We can
still containerize the compute nodes; we should just keep it separate from
this work.
</pre>
</blockquote>
<pre wrap="">Absolutely. We want minimal restrictions/requirements for these
per-job containers. What's the cleanest way to provide the Arvados
SDKs without imposing on user-provided images? Apart from the SDKs
used in the job, there should be no need for any Arvados stuff at all
in the containers where job tasks run, right?
</pre>
</blockquote>
<p></p>
</div>
<p style="margin: 1.2em 0px ! important;"></p>
<p style="margin: 1.2em 0px ! important;">Right now the
arvados/jobs Docker image is meant to be the smallest possible
base so that people who want to containerize their Crunch
scripts can write a Dockerfile that starts <code
style="font-size: 0.85em; font-family:
Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em;
padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid
rgb(234, 234, 234); background-color: rgb(248, 248, 248);
border-radius: 3px 3px 3px 3px; display: inline;">FROM
arvados/jobs</code> and then mixes in what they need. It just
has the SDKs. If we try to run crunch-job inside the container,
we’ll have to add SLURM and maybe some other support tools that
crunch-job itself needs. </p>
<p style="margin: 1.2em 0px ! important;"></p>
<div class="markdown-here-exclude">
<p></p>
<blockquote
cite="mid:CAOtdHJGFzp5pEJ7fmoc8Z9kUbhAMZ_kLHfRBGBazL_LK0kuoOA@mail.gmail.com"
type="cite">
<blockquote type="cite">
<pre wrap="">If we'd like to make this usable as quickly as possible, probably the best
way to do that would be to add Dockerfiles as a new installation path in
crunch-job. Much like it looks for install scripts in the repository, if it
sees a Dockerfile, it should build that image and run the job inside it. The
code for that is relatively straightforward; the hard part of this would be
setting up Docker on the compute nodes, and thinking through the
configuration to make it robust.
</pre>
</blockquote>
<pre wrap="">Wouldn't it be easier (and more reproducibility-friendly) to do the
container-building before the job is submitted, and provide a
container ID or collection hash in runtime_constraints when submitting
the job?</pre>
</blockquote>
<p></p>
</div>
<p style="margin: 1.2em 0px ! important;"></p>
<p style="margin: 1.2em 0px ! important;">As we discussed, this is
easier in part because it does less: it doesn’t specify how
users get images into the system. But still, your point stands,
and I’m happy to adopt this approach and leave that question for
later.</p>
<p style="margin: 1.2em 0px ! important;">Thanks,</p>
</div>
<div class="moz-signature markdown-here-signature">-- <br>
Brett Smith</div>
</body>
</html>