<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div style=""
data-md-original="I%20just%20pushed%20the%202492-docker-crunch-jobs%20branch%20to%20git.%C2%A0%20It%20has%20the%20recipe%20to%20build%20a%20Docker%20image%20that%20has%20all%20the%20base%20software%20necessary%20to%20install%20and%20run%20a%20Crunch%20job.%C2%A0%20It's%20intentionally%20minimalistic%20because%20in%20the%20future%20I'd%20like%20for%20Crunch%20jobs%20to%20be%20able%20to%20specify%20their%20own%20Dockerfiles%20that%20use%20this%20image%20as%20a%20base%2C%20adding%20only%20the%20software%20and%20configuration%20necessary%20for%20their%20specific%20job.%C2%A0%20But%20even%20this%20is%20enough%20to%20install%20a%20script%20and%20run%20it%20with%20arv-crunch-job.%3Cbr%3E%3Cbr%3EOne%20quirk%20about%20the%20current%20recipe%20is%20that%20the%20Docker%20image%20has%20to%20be%20run%20in%20privileged%20mode.%C2%A0%20arv-crunch-job%20automatically%20mounts%20Keep%2C%20and%20Docker's%20default%20security%20policy%20will%20prohibit%20this%20mount.%C2%A0%20Longer-term%2C%20I%20wonder
%20if%20it%20would%20make%20sense%20for%20the%20parent%20compute%20node%20to%20mount%20Keep%20and%20make%20it%20available%20to%20the%20job%20container%20as%20a%20volume.%3Cbr%3E%3Cbr%3EIf%20we'd%20like%20to%20make%20this%20usable%20as%20quickly%20as%20possible%2C%20probably%20the%20best%20way%20to%20do%20that%20would%20be%20to%20add%20Dockerfiles%20as%20a%20new%20installation%20path%20in%20crunch-job.%C2%A0%20Much%20like%20it%20looks%20for%20install%20scripts%20in%20the%20repository%2C%20if%20it%20sees%20a%20Dockerfile%2C%20it%20should%20build%20that%20image%20and%20run%20the%20job%20inside%20it.%C2%A0%20The%20code%20for%20that%20is%20relatively%20straightforward%3B%20the%20hard%20part%20of%20this%20would%20be%20setting%20up%20Docker%20on%20the%20compute%20nodes%2C%20and%20thinking%20through%20the%20configuration%20to%20make%20it%20robust.%3Cbr%3E%3Cbr%3EWe%20could%20also%20decide%20to%20roll%20that%20work%20into%20the%20larger%20Crunch%20v2%20effort.%3Cbr%3E%3Cbr%3EAfter%20thinking%2
0it%20over%20some%2C%20I%20do%20think%20that%20this%20is%20the%20right%20layer%20to%20introduce%20Docker%20at%3B%20or%20at%20least%2C%20this%20is%20the%20layer%20where%20we%20need%20to%20be%20able%20to%20prepare%20and%20run%20customized%20Docker%20images.%C2%A0%20If%20we%20try%20to%20make%20this%20image's%20responsibilities%20part%20of%20a%20larger%20Docker%20container%2C%20we're%20introducing%20space%20where%20users%20have%20to%20worry%20that%20their%20customizations%20might%20interfere%20with%20Arvados'%20general%20operations%2C%20which%20is%20something%20we%20should%20avoid.%C2%A0%20We%20can%20still%20containerize%20the%20compute%20nodes%3B%20we%20should%20just%20keep%20it%20separate%20from%20this%20work.%3Cbr%3E%3Cbr%3EI'm%20interesting%20in%20hearing%20other%20folks'%20thoughts%20about%20where%20work%20should%20be%20focused%20next.%3Cbr%3E%3Cbr%3E"
class="markdown-here-wrapper" data-md-url="null"
id="markdown-here-wrapper-718288">
<p style="margin: 1.2em 0px ! important;">I just pushed the
2492-docker-crunch-jobs branch to git. It has the recipe to
build a Docker image that has all the base software necessary to
install and run a Crunch job. It’s intentionally minimalistic
because in the future I’d like for Crunch jobs to be able to
specify their own Dockerfiles that use this image as a base,
adding only the software and configuration necessary for their
specific job. But even this is enough to install a script and
run it with arv-crunch-job.</p>
<p style="margin: 1.2em 0px ! important;">One quirk about the
current recipe is that the Docker image has to be run in
privileged mode. arv-crunch-job automatically mounts Keep, and
Docker’s default security policy will prohibit this mount.
Longer-term, I wonder if it would make sense for the parent
compute node to mount Keep and make it available to the job
container as a volume.</p>
<p style="margin: 1.2em 0px ! important;">If we’d like to make
this usable as quickly as possible, probably the best way to do
that would be to add Dockerfiles as a new installation path in
crunch-job. Much like it looks for install scripts in the
repository, if it sees a Dockerfile, it should build that image
and run the job inside it. The code for that is relatively
straightforward; the hard part of this would be setting up
Docker on the compute nodes, and thinking through the
configuration to make it robust.</p>
<p style="margin: 1.2em 0px ! important;">We could also decide to
roll that work into the larger Crunch v2 effort.</p>
<p style="margin: 1.2em 0px ! important;">After thinking it over
some, I do think that this is the right layer to introduce
Docker at; or at least, this is the layer where we need to be
able to prepare and run customized Docker images. If we try to
make this image’s responsibilities part of a larger Docker
container, we’re introducing space where users have to worry
that their customizations might interfere with Arvados’ general
operations, which is something we should avoid. We can still
containerize the compute nodes; we should just keep it separate
from this work.</p>
<p style="margin: 1.2em 0px ! important;">I’m interesting in
hearing other folks’ thoughts about where work should be focused
next.</p>
</div>
<div class="moz-signature markdown-here-signature">-- <br>
Brett Smith</div>
</body>
</html>