Merge pull request #62113 from juju-solutions/bug/worker-upgrade

Automatic merge from submit-queue (batch tested with PRs 61705, 61609, 62103, 62113, 62115). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Remove GPU label during upgrade if needed **What this PR does / why we need it**: During upgrade of juju charms we might fail to remove the GPU labels. With this PR we try to remove this label only when GPUs are available. In case the label removal fails we print a warning in hope the GPUs will continue to be available after the upgrade. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/537 **Special notes for your reviewer**: **Release note**: ```release-note NONE ```

Merge pull request #62113 from juju-solutions/bug/worker-upgrade
e57a9891 · Kubernetes Submit Queue · GitHub · 485d2155 · b0eaf0b0 · e57a9891
Unverified Commit e57a9891 authored Apr 04, 2018 by Kubernetes Submit Queue Committed by GitHub Apr 04, 2018
Hide whitespace changes
Inline Side-by-side

Showing with 8 additions and 2 deletions

kubernetes_worker.py ...ju/layers/kubernetes-worker/reactive/kubernetes_worker.py +8 -2

No files found.
--- a/cluster/juju/layers/kubernetes-worker/reactive/kubernetes_worker.py
+++ b/cluster/juju/layers/kubernetes-worker/reactive/kubernetes_worker.py
@@ -69,8 +69,14 @@ def upgrade_charm():

    # Remove gpu.enabled state so we can reconfigure gpu-related kubelet flags,
    # since they can differ between k8s versions
-    remove_state('kubernetes-worker.gpu.enabled')
-    disable_gpu()
+    if is_state('kubernetes-worker.gpu.enabled'):
+        remove_state('kubernetes-worker.gpu.enabled')
+        try:
+            disable_gpu()
+        except ApplyNodeLabelFailed:
+            # Removing node label failed. Probably the master is unavailable.
+            # Proceed with the upgrade in hope GPUs will still be there.
+            hookenv.log('Failed to remove GPU labels. Proceed with upgrade.')

    remove_state('kubernetes-worker.cni-plugins.installed')
    remove_state('kubernetes-worker.config.created')