Merge pull request #59767 from sjenning/check-phase-transition

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. kubelet: check for illegal phase transition I have been unable to root cause the transition from `Failed` phase to `Succeeded`. I have been unable to recreate as well. However, our CI in Origin, where we have controllers that look for these transitions and rely on the phase transition rules being respected, obviously indicate that the illegal transition does occur. This PR will prevent the illegal phase transition from propagating into the kubelet caches or the API server. Fixes https://github.com/kubernetes/kubernetes/issues/58711 xref https://github.com/openshift/origin/issues/17595 @dashpole @yujuhong @derekwaynecarr @smarterclayton @tnozicka

Merge pull request #59767 from sjenning/check-phase-transition
4efbb71b · Kubernetes Submit Queue · GitHub · f072871b · 9ab9ddeb · 4efbb71b
Unverified Commit 4efbb71b authored Feb 12, 2018 by Kubernetes Submit Queue Committed by GitHub Feb 12, 2018
Hide whitespace changes
Inline Side-by-side

Showing with 9 additions and 0 deletions

kubelet_pods.go pkg/kubelet/kubelet_pods.go +9 -0

No files found.
--- a/pkg/kubelet/kubelet_pods.go
+++ b/pkg/kubelet/kubelet_pods.go
@@ -1359,6 +1359,15 @@ func (kl *Kubelet) generateAPIPodStatus(pod *v1.Pod, podStatus *kubecontainer.Po
 	spec := &pod.Spec
 	allStatus := append(append([]v1.ContainerStatus{}, s.ContainerStatuses...), s.InitContainerStatuses...)
 	s.Phase = getPhase(spec, allStatus)
+	// Check for illegal phase transition
+	if pod.Status.Phase == v1.PodFailed || pod.Status.Phase == v1.PodSucceeded {
+		// API server shows terminal phase; transitions are not allowed
+		if s.Phase != pod.Status.Phase {
+			glog.Errorf("Pod attempted illegal phase transition from %s to %s: %v", pod.Status.Phase, s.Phase, s)
+			// Force back to phase from the API server
+			s.Phase = pod.Status.Phase
+		}
+	}
 	kl.probeManager.UpdatePodStatus(pod.UID, s)
 	s.Conditions = append(s.Conditions, status.GeneratePodInitializedCondition(spec, s.InitContainerStatuses, s.Phase))
 	s.Conditions = append(s.Conditions, status.GeneratePodReadyCondition(spec, s.ContainerStatuses, s.Phase))