Dump Stack when docker fails on healthcheck

Send SIGUSR1 to dockerd to save stack dump of docker daemon in order to be able to investigate why docker daemon was unresposive to health check done by `docker ps`. See https://github.com/moby/moby/blob/master/daemon/daemon.go on how docker sets up a trap for SIGUSR1 with `setupDumpStackTrap()`

Dump Stack when docker fails on healthcheck
da23396e · Tomoe Sugihara · 8ea1d92d · da23396e
Commit da23396e authored May 21, 2018 by Tomoe Sugihara
Hide whitespace changes
Inline Side-by-side

Showing with 6 additions and 0 deletions

health-monitor.sh cluster/gce/gci/health-monitor.sh +6 -0

No files found.
--- a/cluster/gce/gci/health-monitor.sh
+++ b/cluster/gce/gci/health-monitor.sh
@@ -50,6 +50,12 @@ function container_runtime_monitoring {
  while true; do
    if ! timeout 60 ${healthcheck_command} > /dev/null; then
      echo "Container runtime ${container_runtime_name} failed!"
+      if [[ "$container_runtime_name" == "docker" ]]; then
+          # Dump stack of docker daemon for investigation.
+          # Log fle name looks like goroutine-stacks-TIMESTAMP and will be saved to
+          # the exec root directory, which is /var/run/docker/ on Ubuntu and COS.
+          pkill -SIGUSR1 dockerd
+      fi
      systemctl kill --kill-who=main "${container_runtime_name}"
      # Wait for a while, as we don't want to kill it again before it is really up.
      sleep 120