<div dir="ltr">To whom it may concern -<div><br></div><div>We are experiencing a large number of "Reader failed checksum" errors when using the keepproxy and I cannot seem to figure out why.</div><div><br></div><div>Here is an snippet from our log for the keepproxy.</div><div><br></div><div><div>Sep 13 11:21:12 arvados keepproxy: 2017/09/13 11:21:12 128.248.171.19,<a href="http://172.17.0.231:52236">172.17.0.231:52236</a> GET /fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9 200 67108864 34647446 <a href="http://arvados-keep01.cri.local:25107/fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9">http://arvados-keep01.cri.local:25107/fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9</a> Reader failed checksum</div><div>Sep 13 11:21:29 arvados keepproxy: 2017/09/13 11:21:29 128.248.171.19,<a href="http://172.17.0.231:51688">172.17.0.231:51688</a> GET /fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9 200 67108864 30612102 <a href="http://arvados-keep01.cri.local:25107/fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9">http://arvados-keep01.cri.local:25107/fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9</a> Reader failed checksum</div><div>Sep 13 11:21:44 arvados keepproxy: 2017/09/13 11:21:44 128.248.171.19,<a href="http://172.17.0.231:50472">172.17.0.231:50472</a> GET /fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9 200 67108864 34956478 <a href="http://arvados-keep01.cri.local:25107/fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9">http://arvados-keep01.cri.local:25107/fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9</a> Reader failed checksum</div><div>Sep 13 11:22:00 arvados keepproxy: 2017/09/13 11:22:00 128.248.171.19,<a href="http://172.17.0.231:52286">172.17.0.231:52286</a> GET /fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9 200 67108864 63292254 <a href="http://arvados-keep01.cri.local:25107/fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9">http://arvados-keep01.cri.local:25107/fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9</a> Reader failed checksum</div><div><br></div><div>If I look at the keepstore, I see the following messages associated with those requests from the keepproxy... In all cases the status code is 200.<br></div><div><br></div><div><div>Sep 13 11:21:12 prometheus keepstore: time="2017-09-13T11:21:12.588429359-05:00" level=info msg=response RequestID=bfpza7ye6fmw remoteAddr="<a href="http://172.17.0.227:59396">172.17.0.227:59396</a>" reqBytes=0 reqForwardedFor= reqMethod=GET reqPath= fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9" respBytes=67108864 respStatus=OK respStatusCode=200 timeToStatus=1.628499 timeTotal=15.000262 timeWriteBody=13.371763</div><div>Sep 13 11:21:14 prometheus keepstore: time="2017-09-13T11:21:14.127563080-05:00" level=debug msg=request RequestID=bfpzafjxba0b remoteAddr="<a href="http://172.17.0.227:59406">172.17.0.227:59406</a>" reqBytes=0 reqForwardedFor= reqMethod=GET reqPath= fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9"</div><div>Sep 13 11:21:29 prometheus keepstore: time="2017-09-13T11:21:29.125244128-05:00" level=info msg=response RequestID=bfpzafjxba0b remoteAddr="<a href="http://172.17.0.227:59406">172.17.0.227:59406</a>" reqBytes=0 reqForwardedFor= reqMethod=GET reqPath= fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9" respBytes=67108864 respStatus=OK respStatusCode=200 timeToStatus=0.179427 timeTotal=14.997671 timeWriteBody=14.818244</div><div>Sep 13 11:21:29 prometheus keepstore: time="2017-09-13T11:21:29.838397727-05:00" level=debug msg=request RequestID=bfpzamrr4y9i remoteAddr="<a href="http://172.17.0.227:59408">172.17.0.227:59408</a>" reqBytes=0 reqForwardedFor= reqMethod=GET reqPath= fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9"</div><div>Sep 13 11:21:44 prometheus keepstore: time="2017-09-13T11:21:44.838600482-05:00" level=info msg=response RequestID=bfpzamrr4y9i remoteAddr="<a href="http://172.17.0.227:59408">172.17.0.227:59408</a>" reqBytes=0 reqForwardedFor= reqMethod=GET reqPath= fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9" respBytes=67108864 respStatus=OK respStatusCode=200 timeToStatus=0.179266 timeTotal=15.000209 timeWriteBody=14.820943</div><div>Sep 13 11:21:45 prometheus keepstore: time="2017-09-13T11:21:45.863245412-05:00" level=debug msg=request RequestID=bfpzau4rwy8i remoteAddr="<a href="http://172.17.0.227:59412">172.17.0.227:59412</a>" reqBytes=0 reqForwardedFor= reqMethod=GET reqPath= fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9"</div><div>Sep 13 11:22:00 prometheus keepstore: time="2017-09-13T11:22:00.828749773-05:00" level=info msg=response RequestID=bfpzau4rwy8i remoteAddr="<a href="http://172.17.0.227:59412">172.17.0.227:59412</a>" reqBytes=0 reqForwardedFor= reqMethod=GET reqPath= fe194b0a4576d5c8d9d0a2055b132b60+67108864+Ab7cbd46f674287a7e906d009c18227475a83edbb@59cbcfe9" respBytes=67108864 respStatus=OK respStatusCode=200 timeToStatus=0.508386 timeTotal=14.965493 timeWriteBody=14.457107</div></div><div><br></div><div>When I check the blocks on the keepstore it seems that the blocks are not corrupted.</div><div><br></div><div><div>ROOT [prometheus:/mnt/keep/fe1] # openssl dgst -md5 -hex *</div><div>MD5(fe125c2660a88a250a8d95542a80a665)= fe125c2660a88a250a8d95542a80a665</div><div>MD5(fe1341f67e7c0a53a0a67c664b1385a0)= fe1341f67e7c0a53a0a67c664b1385a0</div><div>MD5(fe1763cc1a8713b6c768852bca9207b1)= fe1763cc1a8713b6c768852bca9207b1</div><div>MD5(fe194b0a4576d5c8d9d0a2055b132b60)= fe194b0a4576d5c8d9d0a2055b132b60</div><div>MD5(fe1b04d3a017e84bef791ecaca2fb764)= fe1b04d3a017e84bef791ecaca2fb764</div></div><div><br></div><div>We are running the following versions of keepproxy and keepstore.</div><div><br></div><div>keepproxy-0.1.20170906144951.22418ed-1.x86_64<br></div><div><div>keepstore-0.1.20170906144951.22418ed-1.x86_64</div></div><div><br></div><div>Any help would be greatly appreciated.</div><div><br></div><div>Thanks!</div><div><br></div><div><div class="gmail_signature">George Chlipala, Ph.D.<br>Senior Research Specialist<br>Research Resources Center<br>University of Illinois at Chicago<br><br>phone: 312-413-1700<br>email: <a href="mailto:gchlip2@uic.edu" target="_blank">gchlip2@uic.edu</a></div></div>
</div></div>