• Kubernetes Submit Queue's avatar
    Merge pull request #42942 from vishh/gpu-cont-fix · 6de28fab
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue (batch tested with PRs 42942, 42935)
    
    [Bug] Handle container restarts and avoid using runtime pod cache while allocating GPUs
    
    Fixes #42412
    
    **Background**
    Support for multiple GPUs is an experimental feature in v1.6. 
    Container restarts were handled incorrectly which resulted in stranding of GPUs
    Kubelet is incorrectly using runtime cache to track running pods which can result in race conditions (as it did in other parts of kubelet). This can result in same GPU being assigned to multiple pods.
    
    **What does this PR do**
    This PR tracks assignment of GPUs to containers and returns pre-allocated GPUs instead of (incorrectly) allocating new GPUs.
    GPU manager is updated to consume a list of active pods derived from apiserver cache instead of runtime cache.
    Node e2e has been extended to validate this failure scenario.
    
    **Risk**
    Minimal/None since support for GPUs is an experimental feature that is turned off by default. The code is also isolated to GPU manager in kubelet.
    
    **Workarounds**
    In the absence of this PR, users can mitigate the original issue by setting `RestartPolicyNever`  in their pods.
    There is no workaround for the race condition caused by using the runtime cache though.
    Hence it is worth including this fix in v1.6.0.
    
    cc @jianzhangbjz @seelam @kubernetes/sig-node-pr-reviews 
    
    Replaces #42560
    6de28fab
Name
Last commit
Last update
..
boilerplate Loading commit data...
cmd/teststale Loading commit data...
e2e-internal Loading commit data...
gen-swagger-doc Loading commit data...
jenkins Loading commit data...
lib Loading commit data...
make-rules Loading commit data...
testdata Loading commit data...
verify-flags Loading commit data...
.linted_packages Loading commit data...
BUILD Loading commit data...
OWNERS Loading commit data...
autogenerated_placeholder.txt Loading commit data...
benchmark-go.sh Loading commit data...
benchmark-integration.sh Loading commit data...
build-cross.sh Loading commit data...
build-go.sh Loading commit data...
build-ui.sh Loading commit data...
cherry_pick_pull.sh Loading commit data...
dev-build-and-push.sh Loading commit data...
dev-build-and-up.sh Loading commit data...
dev-push-hyperkube.sh Loading commit data...
e2e-node-test.sh Loading commit data...
e2e.go Loading commit data...
e2e_test.go Loading commit data...
federated-ginkgo-e2e.sh Loading commit data...
generate-bindata.sh Loading commit data...
generate-docs.sh Loading commit data...
get-build.sh Loading commit data...
ginkgo-e2e.sh Loading commit data...
godep-restore.sh Loading commit data...
godep-save.sh Loading commit data...
grab-profiles.sh Loading commit data...
install-etcd.sh Loading commit data...
list-feature-tests.sh Loading commit data...
local-up-cluster.sh Loading commit data...
lookup_pull.py Loading commit data...
print-workspace-status.sh Loading commit data...
run-in-gopath.sh Loading commit data...
test-cmd.sh Loading commit data...
test-go.sh Loading commit data...
test-integration.sh Loading commit data...
test-update-storage-objects.sh Loading commit data...
update-all.sh Loading commit data...
update-api-reference-docs.sh Loading commit data...
update-bazel.sh Loading commit data...
update-codecgen.sh Loading commit data...
update-codegen.sh Loading commit data...
update-federation-api-reference-docs.sh Loading commit data...
update-federation-generated-swagger-docs.sh Loading commit data...
update-federation-openapi-spec.sh Loading commit data...
update-federation-swagger-spec.sh Loading commit data...
update-generated-docs.sh Loading commit data...
update-generated-protobuf-dockerized.sh Loading commit data...
update-generated-protobuf.sh Loading commit data...
update-generated-runtime-dockerized.sh Loading commit data...
update-generated-runtime.sh Loading commit data...
update-generated-swagger-docs.sh Loading commit data...
update-godep-licenses.sh Loading commit data...
update-gofmt.sh Loading commit data...
update-openapi-spec.sh Loading commit data...
update-staging-client-go.sh Loading commit data...
update-staging-godeps.sh Loading commit data...
update-swagger-spec.sh Loading commit data...
update-translations.sh Loading commit data...
update_owners.py Loading commit data...
verify-all.sh Loading commit data...
verify-api-groups.sh Loading commit data...
verify-api-reference-docs.sh Loading commit data...
verify-bazel.sh Loading commit data...
verify-boilerplate.sh Loading commit data...
verify-cli-conventions.sh Loading commit data...
verify-codecgen.sh Loading commit data...
verify-codegen.sh Loading commit data...
verify-description.sh Loading commit data...
verify-federation-openapi-spec.sh Loading commit data...
verify-flags-underscore.py Loading commit data...
verify-generated-docs.sh Loading commit data...
verify-generated-protobuf.sh Loading commit data...
verify-generated-runtime.sh Loading commit data...
verify-generated-swagger-docs.sh Loading commit data...
verify-godep-licenses.sh Loading commit data...
verify-godeps.sh Loading commit data...
verify-gofmt.sh Loading commit data...
verify-golint.sh Loading commit data...
verify-govet.sh Loading commit data...
verify-import-boss.sh Loading commit data...
verify-linkcheck.sh Loading commit data...
verify-openapi-spec.sh Loading commit data...
verify-pkg-names.sh Loading commit data...
verify-readonly-packages.sh Loading commit data...
verify-staging-client-go.sh Loading commit data...
verify-staging-godeps.sh Loading commit data...
verify-staging-imports.sh Loading commit data...
verify-swagger-spec.sh Loading commit data...
verify-symbols.sh Loading commit data...
verify-test-images.sh Loading commit data...
verify-test-owners.sh Loading commit data...