[ARVADOS] updated: 3895b3adf05dc6c8d7b66e6bdd42e576a03b1f09

Git user git at public.curoverse.com
Mon Sep 25 22:12:00 EDT 2017


Summary of changes:
 apps/workbench/app/helpers/provenance_helper.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

  discards  5098820fed0920a31dd87a12c4f027318d7a1bd6 (commit)
  discards  4acab1500c713175d90962d78f78dfd8e5966528 (commit)
  discards  3858382147f85129e473606e2d8ae978bb33d9da (commit)
  discards  1b3578c44b8d005f70863cf332b487c915af9f28 (commit)
  discards  6bfb4fc09e5b6f1f54c784dd1f7e2c6700020bb3 (commit)
  discards  8503943dc642ee5a716667a3972d758e5080cd48 (commit)
  discards  61627102af1a1b7a83d547efad5180a584dd9a0c (commit)
       via  3895b3adf05dc6c8d7b66e6bdd42e576a03b1f09 (commit)
       via  78b938ccd86cee5f9d01ccf5b7f42865e1db6651 (commit)
       via  52634bc31df608c30a032d0d6df632dd17207699 (commit)
       via  5c4cbf1ae8496930deb7e35c9880af05941cda75 (commit)
       via  28d09b8cf3b349d7802e04ed2e9ad91f7f0ea210 (commit)

This update added new revisions after undoing existing revisions.  That is
to say, the old revision is not a strict subset of the new revision.  This
situation occurs when you --force push a change and generate a repository
containing something like this:

 * -- * -- B -- O -- O -- O (5098820fed0920a31dd87a12c4f027318d7a1bd6)
            \
             N -- N -- N (3895b3adf05dc6c8d7b66e6bdd42e576a03b1f09)

When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.


commit 3895b3adf05dc6c8d7b66e6bdd42e576a03b1f09
Author: Ward Vandewege <wvandewege at veritasgenetics.com>
Date:   Mon Sep 25 16:53:21 2017 -0400

    Do not blow up the provenance graph if a PDH used in the workflow no
    longer exists.
    
    refs #12316
    
    Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege at veritasgenetics.com>

diff --git a/apps/workbench/app/helpers/provenance_helper.rb b/apps/workbench/app/helpers/provenance_helper.rb
index 94092a1..9b4d265 100644
--- a/apps/workbench/app/helpers/provenance_helper.rb
+++ b/apps/workbench/app/helpers/provenance_helper.rb
@@ -232,7 +232,11 @@ module ProvenanceHelper
                   # Search for any collection with this PDH
                   cols = @opts[:input_collections][pdh]
                 end
-                names = cols.collect{|x| x[:name]}.uniq
+                if cols
+                  names = cols.collect{|x| x[:name]}.uniq
+                else
+                  names = ['(collection not found)']
+                end
                 input_name = names.first
                 if names.length > 1
                   input_name += " + #{names.length - 1} more"

commit 78b938ccd86cee5f9d01ccf5b7f42865e1db6651
Author: Tom Clegg <tclegg at veritasgenetics.com>
Date:   Fri Sep 22 17:00:07 2017 -0400

    12276: Ensure dirent cache is disabled.
    
    Setting this to zero fixed flakiness in #7751, so we might still rely
    on it being zero for reliability.
    
    Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg at veritasgenetics.com>

diff --git a/services/fuse/arvados_fuse/__init__.py b/services/fuse/arvados_fuse/__init__.py
index de0338e..418f748 100644
--- a/services/fuse/arvados_fuse/__init__.py
+++ b/services/fuse/arvados_fuse/__init__.py
@@ -437,6 +437,7 @@ class Operations(llfuse.Operations):
         entry = llfuse.EntryAttributes()
         entry.st_ino = inode
         entry.generation = 0
+        entry.entry_timeout = 0
         entry.attr_timeout = e.time_to_next_poll() if e.allow_attr_cache else 0
 
         entry.st_mode = stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH

commit 52634bc31df608c30a032d0d6df632dd17207699
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Fri Sep 22 14:43:26 2017 -0400

    12276: Remove debug logging message.
    
    Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>

diff --git a/services/fuse/arvados_fuse/fusedir.py b/services/fuse/arvados_fuse/fusedir.py
index 12c02e2..7bd00d5 100644
--- a/services/fuse/arvados_fuse/fusedir.py
+++ b/services/fuse/arvados_fuse/fusedir.py
@@ -194,7 +194,6 @@ class Directory(FreshBase):
         # Calling the public items() method might trigger a refresh,
         # which we definitely don't want, so read the internal dict directly.
         for k,v in parent._entries.items():
-            _logger.debug("looking at %s %s, self is %s", k, v, self)
             if v is self:
                 self.inodes.invalidate_entry(parent, k)
                 break

commit 5c4cbf1ae8496930deb7e35c9880af05941cda75
Author: Peter Amstutz <peter.amstutz at curoverse.com>
Date:   Fri Sep 22 11:21:07 2017 -0400

    12276: Reduce number of spurious invalidations sent to kernel.
    
    * New policy to only send invalidations on objects that have a nonzero
      kernel reference count.
    * When clearing the contents of a CollectionDirectory, only invalidate the
      topmost directory entry, this should take care of invalidating all paths
      beneath it as well.
    * Don't send invalidate_inode() when deleting an inode (by definition, the
      inode is unreferenced by the kernel).
    * Remove allow_dirent_cache for now (this only relates to caching negative
      lookups, which we don't support).
    * Set attribute attr_timeout based on the polling period of the underlying
      object.
    * FuncToJSONFile sets allow_attr_cache = False instead of allow_dirent_cache =
      False (which is useless in this case).
    
    Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz at veritasgenetics.com>

diff --git a/services/fuse/arvados_fuse/__init__.py b/services/fuse/arvados_fuse/__init__.py
index 30770fc..de0338e 100644
--- a/services/fuse/arvados_fuse/__init__.py
+++ b/services/fuse/arvados_fuse/__init__.py
@@ -159,8 +159,8 @@ class InodeCache(object):
             if obj.in_use():
                 _logger.debug("InodeCache cannot clear inode %i, in use", obj.inode)
                 return
+            obj.kernel_invalidate()
             if obj.has_ref(True):
-                obj.kernel_invalidate()
                 _logger.debug("InodeCache sent kernel invalidate inode %i", obj.inode)
                 return
             obj.clear()
@@ -266,17 +266,22 @@ class Inodes(object):
             del self._entries[entry.inode]
             with llfuse.lock_released:
                 entry.finalize()
-            self.invalidate_inode(entry.inode)
             entry.inode = None
         else:
             entry.dead = True
             _logger.debug("del_entry on inode %i with refcount %i", entry.inode, entry.ref_count)
 
-    def invalidate_inode(self, inode):
-        llfuse.invalidate_inode(inode)
+    def invalidate_inode(self, entry):
+        if entry.has_ref(False):
+            # Only necessary if the kernel has previously done a lookup on this
+            # inode and hasn't yet forgotten about it.
+            llfuse.invalidate_inode(entry.inode)
 
-    def invalidate_entry(self, inode, name):
-        llfuse.invalidate_entry(inode, name.encode(self.encoding))
+    def invalidate_entry(self, entry, name):
+        if entry.has_ref(False):
+            # Only necessary if the kernel has previously done a lookup on this
+            # inode and hasn't yet forgotten about it.
+            llfuse.invalidate_entry(entry.inode, name.encode(self.encoding))
 
     def clear(self):
         self.inode_cache.clear()
@@ -432,8 +437,7 @@ class Operations(llfuse.Operations):
         entry = llfuse.EntryAttributes()
         entry.st_ino = inode
         entry.generation = 0
-        entry.entry_timeout = 60 if e.allow_dirent_cache else 0
-        entry.attr_timeout = 60 if e.allow_attr_cache else 0
+        entry.attr_timeout = e.time_to_next_poll() if e.allow_attr_cache else 0
 
         entry.st_mode = stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH
         if isinstance(e, Directory):
diff --git a/services/fuse/arvados_fuse/fresh.py b/services/fuse/arvados_fuse/fresh.py
index a51dd90..8b680f0 100644
--- a/services/fuse/arvados_fuse/fresh.py
+++ b/services/fuse/arvados_fuse/fresh.py
@@ -70,8 +70,9 @@ class FreshBase(object):
         self.dead = False
         self.cache_size = 0
         self.cache_uuid = None
+
+        # Can the kernel cache attributes?
         self.allow_attr_cache = True
-        self.allow_dirent_cache = True
 
     def invalidate(self):
         """Indicate that object contents should be refreshed from source."""
@@ -142,3 +143,13 @@ class FreshBase(object):
 
     def child_event(self, ev):
         pass
+
+    def time_to_next_poll(self):
+        if self._poll:
+            t = (self._last_update + self._poll_time) - self._atime
+            if t < 0:
+                return 0
+            else:
+                return t
+        else:
+            return self._poll_time
diff --git a/services/fuse/arvados_fuse/fusedir.py b/services/fuse/arvados_fuse/fusedir.py
index 0178fe5..12c02e2 100644
--- a/services/fuse/arvados_fuse/fusedir.py
+++ b/services/fuse/arvados_fuse/fusedir.py
@@ -150,12 +150,12 @@ class Directory(FreshBase):
         # delete any other directory entries that were not in found in 'items'
         for i in oldentries:
             _logger.debug("Forgetting about entry '%s' on inode %i", i, self.inode)
-            self.inodes.invalidate_entry(self.inode, i.encode(self.inodes.encoding))
+            self.inodes.invalidate_entry(self, i)
             self.inodes.del_entry(oldentries[i])
             changed = True
 
         if changed:
-            self.inodes.invalidate_inode(self.inode)
+            self.inodes.invalidate_inode(self)
             self._mtime = time.time()
 
         self.fresh()
@@ -182,16 +182,22 @@ class Directory(FreshBase):
         self._entries = {}
         for n in oldentries:
             oldentries[n].clear()
-            self.inodes.invalidate_entry(self.inode, n.encode(self.inodes.encoding))
             self.inodes.del_entry(oldentries[n])
-        self.inodes.invalidate_inode(self.inode)
         self.invalidate()
 
     def kernel_invalidate(self):
-        for n, e in self._entries.iteritems():
-            self.inodes.invalidate_entry(self.inode, n.encode(self.inodes.encoding))
-            e.kernel_invalidate()
-        self.inodes.invalidate_inode(self.inode)
+        # Invalidating the dentry on the parent implies invalidating all paths
+        # below it as well.
+        parent = self.inodes[self.parent_inode]
+
+        # Find self on the parent in order to invalidate this path.
+        # Calling the public items() method might trigger a refresh,
+        # which we definitely don't want, so read the internal dict directly.
+        for k,v in parent._entries.items():
+            _logger.debug("looking at %s %s, self is %s", k, v, self)
+            if v is self:
+                self.inodes.invalidate_entry(parent, k)
+                break
 
     def mtime(self):
         return self._mtime
@@ -266,13 +272,13 @@ class CollectionDirectoryBase(Directory):
                 elif event == arvados.collection.DEL:
                     ent = self._entries[name]
                     del self._entries[name]
-                    self.inodes.invalidate_entry(self.inode, name.encode(self.inodes.encoding))
+                    self.inodes.invalidate_entry(self, name)
                     self.inodes.del_entry(ent)
                 elif event == arvados.collection.MOD:
                     if hasattr(item, "fuse_entry") and item.fuse_entry is not None:
-                        self.inodes.invalidate_inode(item.fuse_entry.inode)
+                        self.inodes.invalidate_inode(item.fuse_entry)
                     elif name in self._entries:
-                        self.inodes.invalidate_inode(self._entries[name].inode)
+                        self.inodes.invalidate_inode(self._entries[name])
 
     def populate(self, mtime):
         self._mtime = mtime
@@ -547,7 +553,7 @@ class TmpCollectionDirectory(CollectionDirectoryBase):
         if self.collection_record_file:
             with llfuse.lock:
                 self.collection_record_file.invalidate()
-            self.inodes.invalidate_inode(self.collection_record_file.inode)
+            self.inodes.invalidate_inode(self.collection_record_file)
             _logger.debug("%s invalidated collection record", self)
 
     def collection_record(self):
@@ -639,6 +645,7 @@ will appear if it exists.
             return False
 
         try:
+            e = None
             e = self.inodes.add_entry(CollectionDirectory(
                     self.inode, self.inodes, self.api, self.num_retries, k))
 
@@ -649,12 +656,13 @@ will appear if it exists.
                     self.inodes.del_entry(e)
                 return True
             else:
-                self.inodes.invalidate_entry(self.inode, k)
+                self.inodes.invalidate_entry(self, k)
                 self.inodes.del_entry(e)
                 return False
         except Exception as ex:
-            _logger.debug('arv-mount exception keep %s', ex)
-            self.inodes.del_entry(e)
+            _logger.exception("arv-mount lookup '%s':", k)
+            if e is not None:
+                self.inodes.del_entry(e)
             return False
 
     def __getitem__(self, item):
@@ -963,7 +971,7 @@ class ProjectDirectory(Directory):
         # Acually move the entry from source directory to this directory.
         del src._entries[name_old]
         self._entries[name_new] = ent
-        self.inodes.invalidate_entry(src.inode, name_old.encode(self.inodes.encoding))
+        self.inodes.invalidate_entry(src, name_old)
 
     @use_counter
     def child_event(self, ev):
@@ -1000,7 +1008,7 @@ class ProjectDirectory(Directory):
             if old_name in self._entries:
                 ent = self._entries[old_name]
                 del self._entries[old_name]
-                self.inodes.invalidate_entry(self.inode, old_name.encode(self.inodes.encoding))
+                self.inodes.invalidate_entry(self, old_name)
 
             if new_name:
                 if ent is not None:
diff --git a/services/fuse/arvados_fuse/fusefile.py b/services/fuse/arvados_fuse/fusefile.py
index 8189a19..5855361 100644
--- a/services/fuse/arvados_fuse/fusefile.py
+++ b/services/fuse/arvados_fuse/fusefile.py
@@ -122,12 +122,11 @@ class FuncToJSONFile(StringFile):
         super(FuncToJSONFile, self).__init__(parent_inode, "", 0)
         self.func = func
 
-        # invalidate_inode() and invalidate_entry() are asynchronous
-        # with no callback to wait for. In order to guarantee
-        # userspace programs don't get stale data that was generated
-        # before the last invalidate(), we must disallow dirent
+        # invalidate_inode() is asynchronous with no callback to wait for. In
+        # order to guarantee userspace programs don't get stale data that was
+        # generated before the last invalidate(), we must disallow inode
         # caching entirely.
-        self.allow_dirent_cache = False
+        self.allow_attr_cache = False
 
     def size(self):
         self._update()

commit 28d09b8cf3b349d7802e04ed2e9ad91f7f0ea210
Author: Tom Clegg <tclegg at veritasgenetics.com>
Date:   Wed Sep 20 17:44:22 2017 -0400

    12298: Allow non-null log when cancelling an unrunnable container.
    
    Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg at veritasgenetics.com>

diff --git a/services/api/app/models/container.rb b/services/api/app/models/container.rb
index 36c87af..e95c1e8 100644
--- a/services/api/app/models/container.rb
+++ b/services/api/app/models/container.rb
@@ -389,7 +389,7 @@ class Container < ArvadosModel
       when Running
         permitted.push :finished_at, :output, :log
       when Queued, Locked
-        permitted.push :finished_at
+        permitted.push :finished_at, :log
       end
 
     else
diff --git a/services/api/test/unit/container_test.rb b/services/api/test/unit/container_test.rb
index 0f3dab5..0b75974 100644
--- a/services/api/test/unit/container_test.rb
+++ b/services/api/test/unit/container_test.rb
@@ -451,6 +451,17 @@ class ContainerTest < ActiveSupport::TestCase
     check_no_change_from_cancelled c
   end
 
+  test "Container locked cancel with log" do
+    c, _ = minimal_new
+    set_user_from_auth :dispatch1
+    assert c.lock, show_errors(c)
+    assert c.update_attributes(
+             state: Container::Cancelled,
+             log: collections(:real_log_collection).portable_data_hash,
+           ), show_errors(c)
+    check_no_change_from_cancelled c
+  end
+
   test "Container running cancel" do
     c, _ = minimal_new
     set_user_from_auth :dispatch1

-----------------------------------------------------------------------


hooks/post-receive
-- 




More information about the arvados-commits mailing list