fix(docker): taosd 被 OOM 杀掉时让容器退出，让 --restart 拉起来 (#34179) by uk0 · Pull Request #35368 · taosdata/TDengine

uk0 · 2026-05-27T20:24:40Z

issue #34179：tsdb:3.3.8.4 容器里 taosd 被系统 OOM 杀掉之后，taosadapter 还在跑，容器对 Docker 仍然显示 healthy，--restart unless-stopped 永远不会触发，服务实际已挂死。

原因

packaging/docker/bin/entrypoint.sh 末尾是 wait，这条命令只在所有后台子进程都退出后才返回。taosd 被杀以后，taosadapter / taoskeeper / taos-explorer 任何一个还活着，wait 就继续阻塞，PID 1 不退出，Docker 不重启。

改动

每个 & 启动的关键进程（taosd、taosadapter、taoskeeper、taos-explorer）把 $! 追加到 PIDS
末尾的 wait 换成 watchdog：每 2s 检查一次 PIDS 里每个 PID，任意一个死了就 pkill -P $$ 把兄弟进程清掉、exit 1
容器以非 0 退出后 Docker 的 --restart 策略就会把它拉起来
which xxx 检查从命令链改成 if 守卫，跳过未安装时不会留空 PID 进 PIDS
边角情况：cluster non-first 节点且所有 sidecar 都 disable 时 PIDS 为空，进 idle 循环让 SIGINT/SIGTERM trap 仍然能停容器

用 polling 而不是 wait -n 是为了不依赖 bash 4.3+；2s 心跳的 CPU 开销可以忽略。

端到端验证

在 lvm1 上用 tdengine/tdengine:latest（server 3.3.6.13）跑三个 case：

step	scenario	期望	实测
1	stock entrypoint + `pkill -9 taosd`	bug：容器仍 Running、taosadapter 还活着	✓ `Running=true ExitCode=0`，`pgrep taosadapter` 返回 PID 416
2	fix entrypoint mount 进去 + `pkill -9 taosd`	容器 Exited(1)	✓ `Running=false ExitCode=1`；log 末尾 `child process 44 exited; shutting container down so Docker can restart it`
3	step 2 基础上加 `--restart unless-stopped`	Docker 自动拉起，`RestartCount` +1	✓ `RestartCount=1`，`State.StartedAt` 已经被刷新

复现命令（lvm1 / Ubuntu 22.04 / Docker 28.2.2）：

# bug 复现
docker run -d --name t1 -e TAOS_DISABLE_KEEPER=1 -e TAOS_DISABLE_EXPLORER=1 \
    tdengine/tdengine:latest
sleep 30 && docker exec t1 pkill -9 taosd && sleep 5
docker inspect t1 --format '{{.State.Running}} {{.State.ExitCode}}'   # → true 0

# fix 验证（mount 修复后的 entrypoint）
docker run -d --name t2 --restart unless-stopped \
    -e TAOS_DISABLE_KEEPER=1 -e TAOS_DISABLE_EXPLORER=1 \
    -v $(pwd)/entrypoint.sh:/usr/bin/entrypoint.sh:ro \
    tdengine/tdengine:latest
sleep 30 && docker exec t2 pkill -9 taosd && sleep 15
docker inspect t2 --format '{{.RestartCount}} {{.State.Running}}'      # → 1 true

行为变化

服务都健康时：行为不变（watchdog 只是空跑）。
任一关键进程死了：容器以 ExitCode=1 退出，--restart 重新拉起，原 wait 实现里是"容器假活"——这才是 issue 报告的真问题。
docker stop / Ctrl-C：现有的 SIGINT/SIGTERM trap 不变，pkill -P $$ 清理子进程后 exit 0。
edge case：PIDS 为空时进 idle 等信号，trap 仍能停。
进程依赖：watchdog 一视同仁地看四个关键进程；taosadapter 单独崩同样会让容器重启——这跟 taosd 单独崩对称，是 issue reporter 期望的语义（"挂掉了假死会丢数据"）。

Refs #34179

…ta#34179) The container entrypoint started taosd / taosadapter / taoskeeper / taos-explorer as background jobs and then called `wait`, which only returns when *all* children have exited. When taosd is OOM-killed the remaining processes (most importantly taosadapter) stay alive, `wait` keeps blocking, and the container looks healthy to Docker even though the database is gone -- so `--restart unless-stopped` never fires and the service is effectively dead. Capture the PID of each background TDengine process as it's started and replace the final `wait` with a 2s-interval watchdog: if any one PID disappears, pkill the rest and `exit 1`, which lets Docker's restart policy bring the container back. Empty-PIDS edge case (cluster non-first node with every sidecar disabled) idles waiting for the existing SIGINT/SIGTERM trap. Mock-verified end-to-end: simulated OOM of one mock service triggers watchdog detection within ~1s, kills siblings, and exits 1. No behaviour change when all services stay healthy.

gemini-code-assist

Code Review

This pull request introduces a watchdog mechanism in the Docker entrypoint script (entrypoint.sh) to track the PIDs of background processes (taosd, taosadapter, taoskeeper, and taos-explorer) and shut down the container if any of them exits unexpectedly. The review comments identify critical improvement opportunities to prevent infinite hangs or unnecessary delays during startup. Specifically, they suggest checking process survival (using kill -0) within the initialization wait loops for taosd, taosadapter, and taoskeeper so that the script fails fast if a process crashes immediately.

gemini-code-assist · 2026-05-27T20:26:36Z

    taosd &
+    PIDS="$PIDS $!"
    while true; do
        es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")


在 taosd 启动后的等待循环中，如果 taosd 因为配置错误或端口冲突等原因在启动时立即崩溃，taos --check 将永远无法成功，导致此 while true 循环变成死循环，容器将无限期挂起而无法触发 Docker 的 --restart 策略。

建议在循环中保存 taosd 的 PID，并在每次循环时检查其是否仍在运行。如果已退出，则应立即报错并退出容器，实现快速失败（Fail-Fast）。

Suggested change

taosd &

PIDS="$PIDS $!"

while true; do

es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")

taosd &

TAOSD_PID=$!

PIDS="$PIDS $TAOSD_PID"

while true; do

if ! kill -0 "$TAOSD_PID" 2>/dev/null; then

echo "taosd died during startup"

exit 1

fi

es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")

gemini-code-assist · 2026-05-27T20:26:36Z

    if [ "$TAOS_FIRST_EP" = "" ]; then
        echo "run TDengine with single node."
        taosd &
+        PIDS="$PIDS $!"
    fi
    while true; do
        es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")


同样地，在单节点或非首节点初始化的 else 分支中，如果 taosd 启动失败，此处的 while true 循环也会陷入死循环。

建议在此处也保存 taosd 的 PID，并在循环中进行健康检查，以避免容器在启动失败时无限挂起。

Suggested change

if [ "$TAOS_FIRST_EP" = "" ]; then

echo "run TDengine with single node."

taosd &

PIDS="$PIDS $!"

fi

while true; do

es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")

TAOSD_PID=""

if [ "$TAOS_FIRST_EP" = "" ]; then

echo "run TDengine with single node."

taosd &

TAOSD_PID=$!

PIDS="$PIDS $TAOSD_PID"

fi

while true; do

if [ -n "$TAOSD_PID" ] && ! kill -0 "$TAOSD_PID" 2>/dev/null; then

echo "taosd died during startup"

exit 1

fi

es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")

gemini-code-assist · 2026-05-27T20:26:36Z

+    taosadapter &
+    PIDS="$PIDS $!"
    # wait for 6041 port ready
    for _ in $(seq 1 20); do
        curl -sf http://localhost:6041/metrics && break
        sleep 0.5
    done


如果 taosadapter 在启动时立即崩溃（例如由于端口冲突或配置错误），curl 探测会持续失败，导致脚本在 seq 1 20 循环中等待整整 10 秒（20 * 0.5s）才会继续。

建议在等待循环中加入进程存活检查（kill -0）。如果进程已经退出，应立即中断等待并报错退出，实现快速失败（Fail-Fast）。

Suggested change

taosadapter &

PIDS="$PIDS $!"

# wait for 6041 port ready

for _ in $(seq 1 20); do

curl -sf http://localhost:6041/metrics && break

sleep 0.5

done

taosadapter &

ADAPTER_PID=$!

PIDS="$PIDS $ADAPTER_PID"

# wait for 6041 port ready

for _ in $(seq 1 20); do

if ! kill -0 "$ADAPTER_PID" 2>/dev/null; then

echo "taosadapter died during startup"

exit 1

fi

curl -sf http://localhost:6041/metrics && break

sleep 0.5

done

gemini-code-assist · 2026-05-27T20:26:36Z

+    taoskeeper &
+    PIDS="$PIDS $!"
    # wait for 6043 port ready
    for _ in $(seq 1 20); do
        curl -sf http://localhost:6043/metrics && break
        sleep 0.5
    done


同样地，对于 taoskeeper，如果启动即崩溃，等待循环会白白消耗 10 秒。建议加入进程存活检查以实现快速失败。

Suggested change

taoskeeper &

PIDS="$PIDS $!"

# wait for 6043 port ready

for _ in $(seq 1 20); do

curl -sf http://localhost:6043/metrics && break

sleep 0.5

done

taoskeeper &

KEEPER_PID=$!

PIDS="$PIDS $KEEPER_PID"

# wait for 6043 port ready

for _ in $(seq 1 20); do

if ! kill -0 "$KEEPER_PID" 2>/dev/null; then

echo "taoskeeper died during startup"

exit 1

fi

curl -sf http://localhost:6043/metrics && break

sleep 0.5

done

…data#34179) Tighten the watchdog so any path out of entrypoint.sh -- watchdog detecting a dead child, SIGTERM from docker stop, set -e failure during initdb/create-snode, etc -- runs the same cleanup_children helper. Without this, a set -e failure during early setup would leave the background services orphaned for tini to reap, instead of going through the normal pkill -P $$ path. Verified end-to-end on lvm1 against tdengine/tdengine:latest with the new entrypoint mounted into the container: A happy path (10s idle) Running=true ExitCode=0 B SIGKILL taosd Running=false ExitCode=1 C SIGKILL taosadapter (symmetric) Running=false ExitCode=1 D docker stop (SIGTERM) Running=false ExitCode=0 E --restart + SIGKILL taosd RestartCount=1, restarted

…#34179) Addresses the gemini-code-assist review on PR taosdata#35368. Before this commit, the startup-time loops blindly waited for taos --check (or a /metrics curl) to succeed; if the just-spawned taosd / taosadapter / taoskeeper / taos-explorer crashed immediately (bad config, port in use, etc.) the script would either loop forever ("taos --check" never succeeding against a dead server) or burn the full 10s curl timeout in silence. Now: - Both taosd startup --check loops check kill -0 $TAOSD_PID on each iteration and exit 1 the moment the process disappears, so a dead taosd no longer turns into a forever-hanging container. - The three sidecar metric-wait loops are hoisted into a wait_for_metric_or_die helper that checks kill -0 inside the wait loop and exits 1 on early death (covers taosadapter, taoskeeper, taos-explorer; the gemini review missed taos-explorer — added too). - The taos --check pipeline is hardened with || true and an explicit empty-result guard so a transient connection error during polling doesn't trip set -e before the kill -0 check gets a chance to run. - The EXIT trap now re-exits with the captured rc instead of letting cleanup_children's last command (a best-effort pkill) overwrite the status code. E2E verified on lvm1 against tdengine/tdengine:latest: A happy path (10s idle) Running=true ExitCode=0 B SIGKILL taosd Running=false ExitCode=1 C SIGKILL taosadapter (symmetric) Running=false ExitCode=1 D docker stop (SIGTERM) Running=false ExitCode=0 E --restart + SIGKILL taosd RestartCount=1, restarted F SIGKILL taosd during startup loop Running=false ExitCode!=0 F2 same as F with --restart unless-stopped RestartCount=1, restarted

uk0 · 2026-05-27T20:38:26Z

Thanks for the review — all four fail-fast suggestions are valid and adopted in commit 2cebb43, plus the missing taos-explorer startup wait gets the same treatment.

What landed:

Both taosd startup --check loops now check kill -0 $TAOSD_PID on every iteration and exit 1 the moment the process disappears, so an immediately-crashing taosd (bad config, port-in-use, etc.) can no longer turn into a forever-hanging container. The taos --check pipeline is also hardened with || true + an explicit empty-result guard so a transient connection error during polling doesn't trip set -e ahead of our kill -0 check.
taosadapter / taoskeeper / taos-explorer metric-wait loops are factored into a wait_for_metric_or_die helper that checks kill -0 inside the wait loop and exit 1s on early death — burns less time and produces a proper failure message instead of timing out silently.
The EXIT trap is also fixed to re-exit with the captured rc instead of letting cleanup_children's last pkill overwrite the status code.

End-to-end matrix on tdengine/tdengine:latest, all green:

case	scenario	result
A	happy path, 10s idle	Running=true, ExitCode=0
B	SIGKILL `taosd`	Running=false, ExitCode=1
C	SIGKILL `taosadapter` (symmetric)	Running=false, ExitCode=1
D	`docker stop` (SIGTERM)	Running=false, ExitCode=0
E	`--restart unless-stopped` + SIGKILL `taosd`	RestartCount=1, restarted
F	SIGKILL `taosd` during startup loop	Running=false, ExitCode!=0
F2	F with `--restart unless-stopped`	RestartCount=1, restarted

CLAassistant · 2026-07-16T02:37:17Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

uk0 requested a review from feici02 as a code owner May 27, 2026 20:24

taosdata-bot Bot added the from community label May 27, 2026

gemini-code-assist Bot reviewed May 27, 2026

View reviewed changes

uk0 added 2 commits May 28, 2026 04:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(docker): taosd 被 OOM 杀掉时让容器退出，让 --restart 拉起来 (#34179)#35368

fix(docker): taosd 被 OOM 杀掉时让容器退出，让 --restart 拉起来 (#34179)#35368
uk0 wants to merge 3 commits into
taosdata:mainfrom
uk0:fix/issue-34179-container-exit-on-taosd-death

uk0 commented May 27, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot May 27, 2026

Uh oh!

uk0 May 27, 2026

Uh oh!

gemini-code-assist Bot May 27, 2026

Uh oh!

uk0 May 27, 2026

Uh oh!

gemini-code-assist Bot May 27, 2026

Uh oh!

uk0 May 27, 2026

Uh oh!

gemini-code-assist Bot May 27, 2026

Uh oh!

uk0 May 27, 2026

Uh oh!

uk0 commented May 27, 2026

Uh oh!

CLAassistant commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

uk0 commented May 27, 2026

原因

改动

端到端验证

行为变化

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 27, 2026

Choose a reason for hiding this comment

Uh oh!

uk0 May 27, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 27, 2026

Choose a reason for hiding this comment

Uh oh!

uk0 May 27, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 27, 2026

Choose a reason for hiding this comment

Uh oh!

uk0 May 27, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 27, 2026

Choose a reason for hiding this comment

Uh oh!

uk0 May 27, 2026

Choose a reason for hiding this comment

Uh oh!

uk0 commented May 27, 2026

Uh oh!

CLAassistant commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants