[ARVADOS] created: 1.3.0-434-g07da7af1b

Git user git at public.curoverse.com
Tue Mar 5 14:01:53 EST 2019

        at  07da7af1bac6dbf346ad6b97924c41564ce42997 (commit)

commit 07da7af1bac6dbf346ad6b97924c41564ce42997
Author: Lucas Di Pentima <ldipentima at veritasgenetics.com>
Date:   Tue Mar 5 16:00:58 2019 -0300

    14916: Adds upgrading note with script example for collection checking.
    Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima at veritasgenetics.com>

diff --git a/doc/admin/upgrading.html.textile.liquid b/doc/admin/upgrading.html.textile.liquid
index eda25d5cc..e00ef1536 100644
--- a/doc/admin/upgrading.html.textile.liquid
+++ b/doc/admin/upgrading.html.textile.liquid
@@ -32,6 +32,59 @@ TODO: extract this information based on git commit messages and generate changel
 h3. current master branch
+h4. Stricter collection manifest validation on the API server
+As a consequence of "#14482":https://dev.arvados.org/issues/14482, the Ruby SDK does a more rigorous collection manifest validation. Additionally, the API server's dependency on the Ruby SDK was updated on "#14870":https://dev.arvados.org/issues/14870 so it may be convenient to check your existing collections against this updated validation code before upgrading.
+One way you could easily do this is by setting up a new rvm gemset and install the specific arvados gem for testing, like so:
+<pre><code>~$ <span class="userinput">rvm gemset create rubysdk-test</span>
+~$ <span class="userinput">rvm gemset use rubysdk-test</span>
+~$ <span class="userinput">gem install arvados -v</span>
+Next, you can run the following script using admin credentials, it will scan the whole collection database and report any collection that didn't pass the check:
+{% codeblock as python %}
+require 'arvados'
+require 'arvados/keep'
+api = Arvados.new
+offset = 0
+batch_size = 100
+invalid = []
+while true
+    begin
+        req = api.collection.index(
+            :select => [:uuid, :created_at, :manifest_text],
+            :include_trash => true, :include_old_versions => true,
+            :limit => batch_size, :offset => offset)
+    rescue
+        invalid.each {|c| puts "#{c[:uuid]} (Created at #{c[:created_at]}): #{c[:error]}" }
+        raise
+    end
+    req[:items].each do |col|
+        begin
+            Keep::Manifest.validate! col[:manifest_text]
+        rescue Exception => e
+            puts "Collection #{col[:uuid]} manifest not valid"
+            invalid << {uuid: col[:uuid], error: e, created_at: col[:created_at]}
+        end
+    end
+    puts "Checked #{offset} / #{req[:items_available]} - Invalid: #{invalid.size}"
+    offset += req[:limit]
+    break if offset > req[:items_available]
+invalid.each {|c| puts "#{c[:uuid]} (Created at #{c[:created_at]}): #{c[:error]}" }
+{% endcodeblock %}
+The script will return a final report enumerating any invalid collection by UUID, with its creation date and error message so you can take the proper correction measures, if needed.
 h4. Python packaging change
 As part of story "#9945":https://dev.arvados.org/issues/9945, the distribution packaging (deb/rpm) of our Python packages has changed. These packages now include a built-in virtualenv to reduce dependencies on system packages. We have also stopped packaging and publishing backports for all the Python dependencies of our packages, as they are no longer needed.



More information about the arvados-commits mailing list