Merge pull request #63755 from tomoe/dumpstack-docker

Automatic merge from submit-queue (batch tested with PRs 63434, 64172, 63975, 64180, 63755). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Dump Stack when docker fails on healthcheck Save stack dump of docker daemon in order to be able to investigate why docker daemon was unresposive to `docker ps` See https://github.com/moby/moby/blob/master/daemon/daemon.go on how docker sets up a trap for SIGUSR1 with `setupDumpStackTrap()` **What this PR does / why we need it**: This allows us to investigate why docker daemon was unresponsive to "docker ps" command. **Special notes for your reviewer**: Manually tested on Ubuntu and COS. **Release note**: ```release-note NONE ```

Merge pull request #63755 from tomoe/dumpstack-docker
972a74e2 · Kubernetes Submit Queue · GitHub · 2feb0937 · da23396e · 972a74e2
Unverified Commit 972a74e2 authored May 24, 2018 by Kubernetes Submit Queue Committed by GitHub May 24, 2018
Hide whitespace changes
Inline Side-by-side

Showing with 6 additions and 0 deletions

health-monitor.sh cluster/gce/gci/health-monitor.sh +6 -0

No files found.
--- a/cluster/gce/gci/health-monitor.sh
+++ b/cluster/gce/gci/health-monitor.sh
@@ -50,6 +50,12 @@ function container_runtime_monitoring {
  while true; do
    if ! timeout 60 ${healthcheck_command} > /dev/null; then
      echo "Container runtime ${container_runtime_name} failed!"
+      if [[ "$container_runtime_name" == "docker" ]]; then
+          # Dump stack of docker daemon for investigation.
+          # Log fle name looks like goroutine-stacks-TIMESTAMP and will be saved to
+          # the exec root directory, which is /var/run/docker/ on Ubuntu and COS.
+          pkill -SIGUSR1 dockerd
+      fi
      systemctl kill --kill-who=main "${container_runtime_name}"
      # Wait for a while, as we don't want to kill it again before it is really up.
      sleep 120