tests: fix race condition in start_conmon_with_default_args by jnovy · Pull Request #648 · containers/conmon

jnovy · 2026-03-16T12:28:22Z

Fix the race condition in start_conmon_with_default_args() in test/test_helper.bash.

Problem

When start_conmon_with_default_args is called a second time with --exec, the original container may have already transitioned to stopped state. The function only checked for running state before returning early, so a stopped container would fall through to wait_for_runtime_status "$CTR_ID" created — which would never succeed, resulting in a timeout.

PR #642 proposed masking this by increasing the timeout from 5s to 30s, but that does not address the root cause.

Fix

Add a check for stopped state alongside the existing running check, so both states cause an early return. This properly handles the case where the container has already finished by the time the exec conmon instance starts.

All 119 tests pass with this fix, including the 7 exec tests, 3 attach tests, and 19 ctrl tests that exercise the affected code path.

Related: #642

Signed-off-by: Jindrich Novy jnovy@redhat.com

packit-as-a-service · 2026-03-16T12:31:48Z

Ephemeral COPR build failed. @containers/packit-build please check.

packit-as-a-service · 2026-03-16T12:31:48Z

Ephemeral COPR build failed. @containers/packit-build please check.

jnovy · 2026-03-16T12:35:13Z

@ricardobranco777 PTAL

ricardobranco777 · 2026-03-16T14:02:36Z

@ricardobranco777 PTAL

Thanks for looking into this.

It passed on x86_64: https://openqa.opensuse.org/tests/5751790

On aarch64 we have this flake:
https://openqa-assets.opensuse.org/tests/5751789/file/conmon-conmon-runc-root.tap.txt

ricardobranco777 · 2026-03-16T14:20:35Z

On aarch64 we have this flake: https://openqa-assets.opensuse.org/tests/5751789/file/conmon-conmon-runc-root.tap.txt

When backporting #645 and this patch we get this on aarch64:

http://openqa-assets.opensuse.org/tests/5751813/file/conmon-conmon-runc-user.tap.txt
http://openqa-assets.opensuse.org/tests/5751813/file/conmon-conmon-runc-root.tap.txt

#| FAIL: /usr/bin/runc start failed with 1: time="2026-03-16T10:24:46-04:00" level=error msg="cannot start a container that has stopped"

Also:
http://openqa-assets.opensuse.org/tests/5751894/file/conmon-conmon-runc-root.tap.txt
#| FAIL: timed out waiting for 'running' from conmon-test-1773675527-34682-29452

ricardobranco777 · 2026-03-16T16:31:43Z

I gave up on #645

This PR doesn't seem to solve the issue on aarch64 as I'm getting these on ctr logs: journald partial message:

http://openqa-assets.opensuse.org/tests/5751931/file/conmon-conmon-runc-root.tap.txt

#| FAIL: timed out waiting for 'stopped' from conmon-test-1773678124-26964-30349

Fix two race conditions in start_conmon_with_default_args(): 1. The previous fix checked for "running" or "stopped" state immediately after conmon returned, to handle --exec on already-running containers. However, on slow systems (e.g. aarch64), runc state can return a bogus "stopped" state (pid=0, created=epoch) during container creation, causing the function to return early without ever calling runc start. Fix: only skip the create/start flow when --exec is in the arguments, since that is the only case where the container is already managed. 2. On slow systems, there is a race condition where the container init process (created by runc create) may exit between the "created" state check and the runc start call. runc start then fails with "cannot start a container that has stopped". Fix: detect this specific error and treat it as success, since the container did run to completion. Fixes the flaky test failures seen on aarch64 in openQA: - https://openqa-assets.opensuse.org/tests/5751789/file/conmon-conmon-runc-root.tap.txt - http://openqa-assets.opensuse.org/tests/5751813/file/conmon-conmon-runc-root.tap.txt - http://openqa-assets.opensuse.org/tests/5751894/file/conmon-conmon-runc-root.tap.txt Signed-off-by: Jindrich Novy <jnovy@redhat.com>

jnovy · 2026-03-16T16:38:05Z

@ricardobranco777 Thanks for the detailed test results! I investigated the failures thoroughly and found two distinct race conditions:

Race Condition 1 (caused by our original fix):
On aarch64, runc state can return a bogus "stopped" state (with pid: 0 and created: "0001-01-01T00:00:00Z") during container creation - before runc create has fully completed. Our original blanket check for stopped state matched this transient state and returned early, causing runc start to never be called. The container then stayed stuck in created state forever.

Fix: Only skip the create/start flow when --exec is in the arguments, since that is the only legitimate case where the container is already managed by a previous call.

Race Condition 2 (pre-existing, exposed by PR #645):
On slow systems, the container init process (created by runc create) can exit between the wait_for_runtime_status created check and the runc start call. runc start then fails with "cannot start a container that has stopped". This was always possible but PR #645 made it visible by adding die on start failure.

Fix: Detect the specific "cannot start a container that has stopped" error from runc start and treat it as success, since the container did run to completion. Also fall back to checking current state via runc state.

The updated commit addresses both issues.

jnovy · 2026-03-16T17:32:02Z

@ricardobranco777 Seems we are chasing ghosts here. In fact this one is a runc regression fixed in runc 1.4.1 (released 2026-03-13) and was backported to 1.3.x branch of runc too - opencontainers/runc#5153

Can you retry with updated runc?

ricardobranco777

@ricardobranco777 Seems we are chasing ghosts here. In fact this one is a runc regression fixed in runc 1.4.1 (released 2026-03-13) and was backported to 1.3.x branch of runc too - opencontainers/runc#5153

Can you retry with updated runc?

LGTM.

aarch64: https://openqa.opensuse.org/tests/5752031
x86_64: https://openqa.opensuse.org/tests/5752032

ricardobranco777 · 2026-03-16T18:21:24Z

test/test_helper.bash

+    # stopped). In that case, we should not try to start it again.
+    local is_exec=false
+    for arg in "${extra_args[@]}"; do
+        if [[ "$arg" == "--exec" ]]; then


Not to be pedantic, but no double-quotes needed for variables with [[. Also, == assumes a pattern. But this is harmless.

Suggested change

if [[ "$arg" == "--exec" ]]; then

if [[ $arg = "--exec" ]]; then

ricardobranco777 · 2026-03-16T18:25:13Z

test/test_helper.bash

+        # and the start call. runc start then fails with "cannot start a
+        # container that has stopped". Check if this is the case and treat
+        # it as success since the container did run to completion.
+        if expr "$start_output" : ".*cannot start a container that has stopped" > /dev/null; then


I believe we can replace all uses of expr with Bash's [[ just to avoid spawning new processes.

Suggested change

if expr "$start_output" : ".*cannot start a container that has stopped" > /dev/null; then

if [[ $start_output =~ ".*cannot start a container that has stopped" ]]; then

jnovy · 2026-03-16T18:29:30Z

If it runs for you with the new runc - even the old code should @ricardobranco777 - we don't need to merge this - it was a red herring.

ricardobranco777 · 2026-03-16T18:31:06Z

If it runs for you with the new runc - even the old code should @ricardobranco777 - we don't need to merge this - it was a red herring.

The above runs were done with this patch and runc v1.4.1.

ricardobranco777 · 2026-03-16T19:30:02Z

If it runs for you with the new runc - even the old code should @ricardobranco777 - we don't need to merge this - it was a red herring.

The above runs were done with this patch and runc v1.4.1.

FWIW, the tests pass without this patch and runc v1.4.1:

aarch64: https://openqa.opensuse.org/tests/5753417
x86_64: https://openqa.opensuse.org/tests/5753418

Thanks for helping with #642

jnovy mentioned this pull request Mar 16, 2026

tests: Increase timeout for wait_for_runtime_status #642

Closed

ricardobranco777 mentioned this pull request Mar 16, 2026

bats/conmon: Drop PR#642 in favor of 648 os-autoinst/os-autoinst-distri-opensuse#24980

Closed

jnovy force-pushed the fix-race-condition-start-conmon branch from 20e757c to 681a041 Compare March 16, 2026 16:37

ricardobranco777 approved these changes Mar 16, 2026

View reviewed changes

jnovy closed this Mar 16, 2026

	if [[ "$arg" == "--exec" ]]; then
	if [[ $arg = "--exec" ]]; then

	if expr "$start_output" : ".*cannot start a container that has stopped" > /dev/null; then
	if [[ $start_output =~ ".*cannot start a container that has stopped" ]]; then

Conversation

jnovy commented Mar 16, 2026

Problem

Fix

Uh oh!

packit-as-a-service bot commented Mar 16, 2026

Uh oh!

packit-as-a-service bot commented Mar 16, 2026

Uh oh!

jnovy commented Mar 16, 2026

Uh oh!

ricardobranco777 commented Mar 16, 2026

Uh oh!

ricardobranco777 commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ricardobranco777 commented Mar 16, 2026

Uh oh!

jnovy commented Mar 16, 2026

Uh oh!

jnovy commented Mar 16, 2026

Uh oh!

ricardobranco777 left a comment

Choose a reason for hiding this comment

Uh oh!

ricardobranco777 Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

ricardobranco777 Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnovy commented Mar 16, 2026

Uh oh!

ricardobranco777 commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ricardobranco777 commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ricardobranco777 commented Mar 16, 2026 •

edited

Loading

ricardobranco777 Mar 16, 2026 •

edited

Loading

ricardobranco777 commented Mar 16, 2026 •

edited

Loading

ricardobranco777 commented Mar 16, 2026 •

edited

Loading