• Kubernetes Submit Queue's avatar
    Merge pull request #40417 from jsravn/fix-reconciler-external-updates-race · ddf4a0ca
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue (batch tested with PRs 41531, 40417, 41434)
    
    Always detach volumes in operator executor
    
    **What this PR does / why we need it**:
    
    Instead of marking a volume as detached immediately in Kubelet's
    reconciler, delegate the marking asynchronously to the operator
    executor. This is necessary to prevent race conditions with other
    operations mutating the same volume state.
    
    An example of one such problem:
    
    1. pod is created, volume is added to desired state of the world
    2. reconciler process starts
    3. reconciler starts MountVolume, which is kicked off asynchronously via
       operation_executor.go
    4. MountVolume mounts the volume, but hasn't yet marked it as mounted
    5. pod is deleted, volume is removed from desired state of the world
    6. reconciler reaches detach volume section, detects volume is no longer in desired state of world,
       removes it from volumes in use
    7. MountVolume tries to mark mount, throws an error because
       volume is no longer in actual state of world list. After this, kubelet isn't aware of the mount
       so doesn't try to unmount again.
    8. controller-manager tries to detach the volume, this fails because it
       is still mounted to the OS.
    9. EBS gets stuck indefinitely in busy state trying to detach.
    
    
    
    **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #32881, fixes ##37854 (maybe)
    
    **Special notes for your reviewer**:
    
    **Release note**:
    
    ```release-note
    ```
    ddf4a0ca
Name
Last commit
Last update
..
aws_ebs Loading commit data...
azure_dd Loading commit data...
azure_file Loading commit data...
cephfs Loading commit data...
cinder Loading commit data...
configmap Loading commit data...
downwardapi Loading commit data...
empty_dir Loading commit data...
fc Loading commit data...
flexvolume Loading commit data...
flocker Loading commit data...
gce_pd Loading commit data...
git_repo Loading commit data...
glusterfs Loading commit data...
host_path Loading commit data...
iscsi Loading commit data...
nfs Loading commit data...
photon_pd Loading commit data...
quobyte Loading commit data...
rbd Loading commit data...
secret Loading commit data...
testing Loading commit data...
util Loading commit data...
vsphere_volume Loading commit data...
BUILD Loading commit data...
OWNERS Loading commit data...
doc.go Loading commit data...
metrics_cached.go Loading commit data...
metrics_du.go Loading commit data...
metrics_du_test.go Loading commit data...
metrics_errors.go Loading commit data...
metrics_nil.go Loading commit data...
metrics_nil_test.go Loading commit data...
metrics_statfs.go Loading commit data...
metrics_statfs_test.go Loading commit data...
plugins.go Loading commit data...
plugins_test.go Loading commit data...
util.go Loading commit data...
util_test.go Loading commit data...
volume.go Loading commit data...
volume_linux.go Loading commit data...
volume_unsupported.go Loading commit data...