• Dan Mace's avatar
    Fix quota controller worker deadlock · 010a1273
    Dan Mace authored
    The resource quota controller worker pool can deadlock when:
    
    * Worker goroutines are idle waiting for work from queues
    * The Sync() method detects discovery updates to apply
    
    The problem is workers acquire a read lock while idle, making write lock
    acquisition dependent upon the presence of work in the queues.
    
    The Sync() method blocks on a pending write lock acquisition and won't unblock
    until every existing worker processes one item from their queue and releases
    their read lock. While the Sync() method's lock is pending, all new read lock
    acquisitions will block; if a worker does process work and release its lock, it
    will then become blocked on a read lock acquisition; they become blocked on
    Sync(). This can easily deadlock all the workers processing from one queue while
    any workers on the other queue remain blocked waiting for work.
    
    Fix the deadlock by refactoring workers to acquire a read lock *after* work is
    popped from the queue. This allows writers to get locks while workers are idle,
    while preserving the worker pause semantics necessary to allow safe sync.
    010a1273
Name
Last commit
Last update
..
BUILD Loading commit data...
OWNERS Loading commit data...
doc.go Loading commit data...
resource_quota_controller.go Loading commit data...
resource_quota_controller_test.go Loading commit data...
resource_quota_monitor.go Loading commit data...