• Kubernetes Submit Queue's avatar
    Merge pull request #35038 from sjenning/nfs-nonblock-reader2 · 84aa5f69
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue
    
    kubelet: storage: don't hang kubelet on unresponsive nfs
    
    Fixes #31272 
    
    Currently, due to the nature of nfs, an unresponsive nfs volume in a pod can wedge the kubelet such that additional pods can not be run.
    
    The discussion thus far surrounding this issue was to wrap the `lstat`, the syscall that ends up hanging in uninterruptible sleep, in a goroutine and limiting the number of goroutines that hang to one per-pod per-volume.
    
    However, in my investigation, I found that the callsites that request a listing of the volumes from a particular volume plugin directory don't care anything about the properties provided by the `lstat` call.  They only care about whether or not a directory exists.
    
    Given that constraint, this PR just avoids the `lstat` call by using `Readdirnames()` instead of `ReadDir()` or `ReadDirNoExit()`
    
    ### More detail for reviewers
    Consider the pod mounted nfs volume at `/var/lib/kubelet/pods/881341b5-9551-11e6-af4c-fa163e815edd/volumes/kubernetes.io~nfs/myvol`.  The kubelet wedges because when we do a `ReadDir()` or `ReadDirNoExit()` it calls `syscall.Lstat` on `myvol` which requires communication with the nfs server.  If the nfs server is unreachable, this call hangs forever.
    
    However, for our code, we only care what about the names of files/directory contained in `kubernetes.io~nfs` directory, not any of the more detailed information the `Lstat` call provides.  Getting the names can be done with `Readdirnames()`, which doesn't need to involve the nfs server.
    
    @pmorie @eparis @ncdc @derekwaynecarr @saad-ali @thockin @vishh @kubernetes/rh-cluster-infra
    84aa5f69
Name
Last commit
Last update
..
async Loading commit data...
bandwidth Loading commit data...
cache Loading commit data...
cert Loading commit data...
chmod Loading commit data...
chown Loading commit data...
clock Loading commit data...
codeinspector Loading commit data...
config Loading commit data...
configz Loading commit data...
crlf Loading commit data...
dbus Loading commit data...
diff Loading commit data...
ebtables Loading commit data...
env Loading commit data...
errors Loading commit data...
exec Loading commit data...
flag Loading commit data...
flock Loading commit data...
flowcontrol Loading commit data...
flushwriter Loading commit data...
framer Loading commit data...
goroutinemap Loading commit data...
hash Loading commit data...
homedir Loading commit data...
httpstream Loading commit data...
initsystem Loading commit data...
integer Loading commit data...
interrupt Loading commit data...
intstr Loading commit data...
io Loading commit data...
iptables Loading commit data...
json Loading commit data...
jsonpath Loading commit data...
keymutex Loading commit data...
labels Loading commit data...
limitwriter Loading commit data...
logs Loading commit data...
maps Loading commit data...
metrics Loading commit data...
mount Loading commit data...
net Loading commit data...
node Loading commit data...
oom Loading commit data...
parsers Loading commit data...
pod Loading commit data...
procfs Loading commit data...
proxy Loading commit data...
rand Loading commit data...
replicaset Loading commit data...
resourcecontainer Loading commit data...
rlimit Loading commit data...
runtime Loading commit data...
selinux Loading commit data...
sets Loading commit data...
slice Loading commit data...
strategicpatch Loading commit data...
strings Loading commit data...
sysctl Loading commit data...
system Loading commit data...
term Loading commit data...
testing Loading commit data...
threading Loading commit data...
uuid Loading commit data...
validation Loading commit data...
wait Loading commit data...
workqueue Loading commit data...
wsstream Loading commit data...
yaml Loading commit data...
doc.go Loading commit data...
template.go Loading commit data...
template_test.go Loading commit data...
trace.go Loading commit data...
trie.go Loading commit data...
umask.go Loading commit data...
umask_windows.go Loading commit data...
util.go Loading commit data...
util_test.go Loading commit data...