• k8s-merge-robot's avatar
    Merge pull request #29641 from ivan4th/fix-configmap-race · e86b3f26
    k8s-merge-robot authored
    Automatic merge from submit-queue
    
    Fix wrapped volume race
    
    **EDIT:** now covers configmap, secret, downwardapi & git_repo volume plugins.
    
    Fixes #29297.
    
    wrappedVolumeSpec used by configMapVolumeMounter and
    configMapVolumeUnmounter contained a pointer to api.Volume which was
    being patched by NewWrapperMounter/NewWrapperUnmounter, causing race
    condition during configmap volume mounts.
    
    See https://github.com/kubernetes/kubernetes/issues/29297#issuecomment-235403806 for complete explanation.
    The subtle bug was introduced by #18445, it also can affect other volume plugins utilizing `wrappedVolumeSpec` technique, if this PR is correct/accepted will make more PRs for secrets etc. Although tmpfs variety of inner `emptyDir` volume appears to be less susceptible to this race, there's chance it can fail too.
    
    The errors produced by this race look like this:
    ```Jul 19 17:05:21 ubuntu1604 kubelet[17097]: I0719 17:05:21.854303   17097 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/configmap/foo-files"
     (spec.Name: "files") to pod "11786582-4dbf-11e6-9fc9-64cca009c636" (UID: "11786582-4dbf-11e6-9fc9-64cca009c636").
    Jul 19 17:05:21 ubuntu1604 kubelet[17097]: I0719 17:05:21.854842   17097 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/configmap/bar-file
    s" (spec.Name: "files") to pod "117d2c22-4dbf-11e6-9fc9-64cca009c636" (UID: "117d2c22-4dbf-11e6-9fc9-64cca009c636").
    Jul 19 17:05:21 ubuntu1604 kubelet[17097]: E0719 17:05:21.860796   17097 configmap.go:171] Error creating atomic writer: stat /var/lib/kubelet/pods/117d2c22-4dbf-11e6-9fc9-64cca009c636/volumes/kubernetes.io~configmap/files: no such file or directory
    Jul 19 17:05:21 ubuntu1604 kubelet[17097]: E0719 17:05:21.861070   17097 goroutinemap.go:155] Operation for "kubernetes.io/configmap/bar-files" failed. No retries permitted until 2016-07-19 17:07:21.861036886 +0200 CEST (durationBeforeRetry 2m0s). error: MountVolume.SetUp failed for volume "kubernetes.io/configmap/bar-files" (spec.Name: "files") pod "117d2c22-4dbf-11e6-9fc9-64cca009c636" (UID: "117d2c22-4dbf-11e6-9fc9-64cca009c636") with: stat /var/lib/kubelet/pods/117d2c22-4dbf-11e6-9fc9-64cca009c636/volumes/kubernetes.io~configmap/files: no such file or directory
    Jul 19 17:05:21 ubuntu1604 kubelet[17097]: E0719 17:05:21.861271   17097 configmap.go:171] Error creating atomic writer: stat /var/lib/kubelet/pods/11786582-4dbf-11e6-9fc9-64cca009c636/volumes/kubernetes.io~configmap/files: no such file or directory
    Jul 19 17:05:21 ubuntu1604 kubelet[17097]: E0719 17:05:21.862284   17097 goroutinemap.go:155] Operation for "kubernetes.io/configmap/foo-files" failed. No retries permitted until 2016-07-19 17:07:21.862275753 +0200 CEST (durationBeforeRetry 2m0s). error: MountVolume.SetUp failed for volume "kubernetes.io/configmap/foo-files" (spec.Name: "files") pod "11786582-4dbf-11e6-9fc9-64cca009c636" (UID: "11786582-4dbf-11e6-9fc9-64cca009c636") with: stat /var/lib/kubelet/pods/11786582-4dbf-11e6-9fc9-64cca009c636/volumes/kubernetes.io~configmap/files: no such file or directory```
    
    Note "Error creating atomic writer" errors.
    This problem can be reproduced by making kubelet mount multiple config map volumes in parallel.
    e86b3f26
Name
Last commit
Last update
..
admission Loading commit data...
api Loading commit data...
apimachinery Loading commit data...
apis Loading commit data...
apiserver Loading commit data...
auth Loading commit data...
capabilities Loading commit data...
client Loading commit data...
cloudprovider Loading commit data...
controller Loading commit data...
conversion Loading commit data...
credentialprovider Loading commit data...
dns Loading commit data...
fieldpath Loading commit data...
fields Loading commit data...
genericapiserver Loading commit data...
healthz Loading commit data...
httplog Loading commit data...
hyperkube Loading commit data...
kubectl Loading commit data...
kubelet Loading commit data...
kubemark Loading commit data...
labels Loading commit data...
master Loading commit data...
metrics Loading commit data...
probe Loading commit data...
proxy Loading commit data...
quota Loading commit data...
registry Loading commit data...
runtime Loading commit data...
security Loading commit data...
securitycontext Loading commit data...
serviceaccount Loading commit data...
ssh Loading commit data...
storage Loading commit data...
types Loading commit data...
ui Loading commit data...
util Loading commit data...
version Loading commit data...
volume Loading commit data...
watch Loading commit data...
OWNERS Loading commit data...