• Kubernetes Submit Queue's avatar
    Merge pull request #38339 from gnufied/backoff-on-volume-delete · 8f11cc78
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue (batch tested with PRs 38377, 36365, 36648, 37691, 38339)
    
    Exponential back off when volume delete fails
    
    **What this PR does / why we need it**:
    
    This PR implements ability in pv_controller to back off when deleting a volume fails from plugin API. 
    
    **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: 
    
    Partly fixes #38295 , but I think volume delete is most problematic thing happening in pv_controller without any sort of backoff. 
    
    After this change the attempts of volume deletion look like:
    
    ```
    controller : I1208 00:18:35.532061   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
    controller : I1208 00:20:50.578325   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
    controller : I1208 00:23:05.563488   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
    controller : I1208 00:25:20.599158   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
    controller : I1208 00:27:35.560009   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
    controller : I1208 00:29:50.594967   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
    controller : I1208 00:32:05.539168   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
    controller : I1208 00:34:20.581665   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
    ```
    8f11cc78
Name
Last commit
Last update
..
certificates Loading commit data...
cronjob Loading commit data...
daemon Loading commit data...
deployment Loading commit data...
disruption Loading commit data...
endpoint Loading commit data...
garbagecollector Loading commit data...
informers Loading commit data...
job Loading commit data...
namespace Loading commit data...
node Loading commit data...
petset Loading commit data...
podautoscaler Loading commit data...
podgc Loading commit data...
replicaset Loading commit data...
replication Loading commit data...
resourcequota Loading commit data...
route Loading commit data...
service Loading commit data...
serviceaccount Loading commit data...
volume Loading commit data...
.import-restrictions Loading commit data...
BUILD Loading commit data...
OWNERS Loading commit data...
client_builder.go Loading commit data...
controller_ref_manager.go Loading commit data...
controller_utils.go Loading commit data...
controller_utils_test.go Loading commit data...
doc.go Loading commit data...
lookup_cache.go Loading commit data...