Merge pull request #68131 from wojtek-t/fix_ipam_retrying

Automatic merge from submit-queue (batch tested with PRs 67756, 64149, 68076, 68131, 68120). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md. Fix ipam controller to not drop nodes forever in case of failures https://github.com/kubernetes/kubernetes/pull/68084 introduced a problem that when 10 attempts to assign a cidr would fail, we stop retrying but the node is still marked as "in-processing" so it would never be reconciled again. This PR is fixing this problem.

Merge pull request #68131 from wojtek-t/fix_ipam_retrying
240a086c · Kubernetes Submit Queue · GitHub · 36187c6a · fcd28827 · 240a086c
Unverified Commit 240a086c authored Aug 31, 2018 by Kubernetes Submit Queue Committed by GitHub Aug 31, 2018
Hide whitespace changes
Inline Side-by-side

Showing with 1 addition and 1 deletion

cloud_cidr_allocator.go pkg/controller/nodeipam/ipam/cloud_cidr_allocator.go +1 -1

No files found.
--- a/pkg/controller/nodeipam/ipam/cloud_cidr_allocator.go
+++ b/pkg/controller/nodeipam/ipam/cloud_cidr_allocator.go
@@ -157,7 +157,6 @@ func (ca *cloudCIDRAllocator) worker(stopChan <-chan struct{}) {
 			}
 			if err := ca.updateCIDRAllocation(workItem); err == nil {
 				glog.V(3).Infof("Updated CIDR for %q", workItem)
-				ca.removeNodeFromProcessing(workItem)
 			} else {
 				glog.Errorf("Error updating CIDR for %q: %v", workItem, err)
 				if canRetry, timeout := ca.retryParams(workItem); canRetry {
@@ -170,6 +169,7 @@ func (ca *cloudCIDRAllocator) worker(stopChan <-chan struct{}) {
 				}
 				glog.Errorf("Exceeded retry count for %q, dropping from queue", workItem)
 			}
+			ca.removeNodeFromProcessing(workItem)
 		case <-stopChan:
 			return
 		}