• Kubernetes Submit Queue's avatar
    Merge pull request #42942 from vishh/gpu-cont-fix · 6de28fab
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue (batch tested with PRs 42942, 42935)
    
    [Bug] Handle container restarts and avoid using runtime pod cache while allocating GPUs
    
    Fixes #42412
    
    **Background**
    Support for multiple GPUs is an experimental feature in v1.6. 
    Container restarts were handled incorrectly which resulted in stranding of GPUs
    Kubelet is incorrectly using runtime cache to track running pods which can result in race conditions (as it did in other parts of kubelet). This can result in same GPU being assigned to multiple pods.
    
    **What does this PR do**
    This PR tracks assignment of GPUs to containers and returns pre-allocated GPUs instead of (incorrectly) allocating new GPUs.
    GPU manager is updated to consume a list of active pods derived from apiserver cache instead of runtime cache.
    Node e2e has been extended to validate this failure scenario.
    
    **Risk**
    Minimal/None since support for GPUs is an experimental feature that is turned off by default. The code is also isolated to GPU manager in kubelet.
    
    **Workarounds**
    In the absence of this PR, users can mitigate the original issue by setting `RestartPolicyNever`  in their pods.
    There is no workaround for the race condition caused by using the runtime cache though.
    Hence it is worth including this fix in v1.6.0.
    
    cc @jianzhangbjz @seelam @kubernetes/sig-node-pr-reviews 
    
    Replaces #42560
    6de28fab
Name
Last commit
Last update
..
api Loading commit data...
cadvisor Loading commit data...
certificate Loading commit data...
client Loading commit data...
cm Loading commit data...
config Loading commit data...
container Loading commit data...
custommetrics Loading commit data...
dockershim Loading commit data...
dockertools Loading commit data...
envvars Loading commit data...
events Loading commit data...
eviction Loading commit data...
gpu Loading commit data...
images Loading commit data...
kuberuntime Loading commit data...
leaky Loading commit data...
lifecycle Loading commit data...
metrics Loading commit data...
network Loading commit data...
pleg Loading commit data...
pod Loading commit data...
preemption Loading commit data...
prober Loading commit data...
qos Loading commit data...
remote Loading commit data...
rkt Loading commit data...
rktshim Loading commit data...
secret Loading commit data...
server Loading commit data...
status Loading commit data...
sysctl Loading commit data...
types Loading commit data...
util Loading commit data...
volumemanager Loading commit data...
BUILD Loading commit data...
OWNERS Loading commit data...
active_deadline.go Loading commit data...
active_deadline_test.go Loading commit data...
disk_manager.go Loading commit data...
disk_manager_test.go Loading commit data...
doc.go Loading commit data...
kubelet.go Loading commit data...
kubelet_cadvisor.go Loading commit data...
kubelet_cadvisor_test.go Loading commit data...
kubelet_getters.go Loading commit data...
kubelet_getters_test.go Loading commit data...
kubelet_network.go Loading commit data...
kubelet_network_test.go Loading commit data...
kubelet_node_status.go Loading commit data...
kubelet_node_status_test.go Loading commit data...
kubelet_pods.go Loading commit data...
kubelet_pods_test.go Loading commit data...
kubelet_pods_windows_test.go Loading commit data...
kubelet_resources.go Loading commit data...
kubelet_resources_test.go Loading commit data...
kubelet_test.go Loading commit data...
kubelet_volumes.go Loading commit data...
kubelet_volumes_test.go Loading commit data...
networks.go Loading commit data...
oom_watcher.go Loading commit data...
oom_watcher_test.go Loading commit data...
pod_container_deletor.go Loading commit data...
pod_container_deletor_test.go Loading commit data...
pod_workers.go Loading commit data...
pod_workers_test.go Loading commit data...
reason_cache.go Loading commit data...
reason_cache_test.go Loading commit data...
runonce.go Loading commit data...
runonce_test.go Loading commit data...
runtime.go Loading commit data...
util.go Loading commit data...
volume_host.go Loading commit data...