Merge pull request #62602 from shyamjvs/retryable-api-error-change

Automatic merge from submit-queue (batch tested with PRs 62407, 62602, 62539, 62639, 62647). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Add ConnectionReset, InternalError, etc also as retryable API errors Ref https://github.com/kubernetes/kubernetes/issues/55860 We're seeing transient InternalError causing failures for e.g in our GKE large-scale tests: - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance/34 - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance/35 I'm making the change from what we're doing for webhooks - https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/util/webhook/webhook.go#L107-L114 ```release-note NONE ``` /cc @wojtek-t

Merge pull request #62602 from shyamjvs/retryable-api-error-change
c68f0552 · Kubernetes Submit Queue · GitHub · 5899ee63 · 7ab5bea9 · c68f0552
Unverified Commit c68f0552 authored Apr 16, 2018 by Kubernetes Submit Queue Committed by GitHub Apr 16, 2018
Hide whitespace changes
Inline Side-by-side

Showing with 10 additions and 1 deletion

create_resources.go test/utils/create_resources.go +10 -1

No files found.
--- a/test/utils/create_resources.go
+++ b/test/utils/create_resources.go
@@ -51,7 +51,16 @@ func RetryWithExponentialBackOff(fn wait.ConditionFunc) error {
 }
 func IsRetryableAPIError(err error) bool {
-	return apierrs.IsTimeout(err) || apierrs.IsServerTimeout(err) || apierrs.IsTooManyRequests(err) || utilnet.IsProbableEOF(err)
+	// These errors may indicate a transient error that we can retry in tests.
+	if apierrs.IsInternalError(err) || apierrs.IsTimeout(err) || apierrs.IsServerTimeout(err) ||
+		apierrs.IsTooManyRequests(err) || utilnet.IsProbableEOF(err) || utilnet.IsConnectionReset(err) {
+		return true
+	}
+	// If the error sends the Retry-After header, we respect it as an explicit confirmation we should retry.
+	if _, shouldRetry := apierrs.SuggestsClientDelay(err); shouldRetry {
+		return true
+	}
+	return false
 }
 func CreatePodWithRetries(c clientset.Interface, namespace string, obj *v1.Pod) error {