Skip to content

K
k3s
Commit da23396e authored by Tomoe Sugihara's avatar Tomoe Sugihara
Browse files
Options

Dump Stack when docker fails on healthcheck

Send SIGUSR1 to dockerd to save stack dump of docker daemon in order to be able to investigate why docker daemon was unresposive to health check done by `docker ps`. See https://github.com/moby/moby/blob/master/daemon/daemon.go on how docker sets up a trap for SIGUSR1 with `setupDumpStackTrap()`
parent 8ea1d92d
Hide whitespace changes
Inline Side-by-side
Showing with 6 additions and 0 deletions
cluster/gce/gci/health-monitor.sh
View file @ da23396e
...@@ -50,6 +50,12 @@ function container_runtime_monitoring { ...@@ -50,6 +50,12 @@ function container_runtime_monitoring {
while true; do while true; do
if ! timeout 60 ${healthcheck_command} > /dev/null; then if ! timeout 60 ${healthcheck_command} > /dev/null; then
echo "Container runtime ${container_runtime_name} failed!" echo "Container runtime ${container_runtime_name} failed!"
if [[ "$container_runtime_name" == "docker" ]]; then
# Dump stack of docker daemon for investigation.
# Log fle name looks like goroutine-stacks-TIMESTAMP and will be saved to
# the exec root directory, which is /var/run/docker/ on Ubuntu and COS.
pkill -SIGUSR1 dockerd
fi
systemctl kill --kill-who=main "${container_runtime_name}" systemctl kill --kill-who=main "${container_runtime_name}"
# Wait for a while, as we don't want to kill it again before it is really up. # Wait for a while, as we don't want to kill it again before it is really up.
sleep 120 sleep 120
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment