Problem Statement
Each Struct-typed channel's State creates its own DuckDB connection with no thread limit, so DuckDB sizes its TaskScheduler to the host core count. Total worker threads ≈ (number of Struct channels) × (cores − 1). On large hosts this is thousands of mostly-idle threads and can exhaust the cgroup v2 pids.max (containers fail with RuntimeError: can't start new thread).
csp_gateway/server/gateway/csp/state.py:
_USE_DUCKDB_STATE = True (line 29) — DuckDB state is the default.
DuckDBState.init (line 200): self._con = duckdb.connect() — one connection per channel-state, no config={"threads": ...}.
State.init (line 524) creates a DuckDBState for every channel whose record type is a Struct.
DuckDB defaults threads to the core count, so each connection starts cores − 1 workers.
Live container (gdb backtrace of all threads): 1764 / 1985 threads in duckdb::TaskScheduler::ExecuteForever.
Measured per duckdb.connect() on a 64-core host: +63 threads; second connect adds another +63 (per-connection, not shared).
Thread count scales with cores (confirmed by reproduction): ~28 channels →
32-core host: ~28 × 31 ≈ 870
64-core host: ~28 × 63 ≈ 1764
384-core host: ~28 × 383 ≈ 10,700
Proposed Solution
Cap threads at connection creation in DuckDBState.init:
self._con = duckdb.connect(config={"threads": N})
Verified on 64-core / duckdb 1.4.0: config={"threads": "4"} → +3 threads instead of +63 (in-memory form, no file needed). This makes the thread count independent of host cores.
Alternatives Considered
NA
Open questions
Best default N for these small current-value tables (likely 1–2 is plenty)? Worth making it configurable on the gateway.
Alternatively, share a single DuckDB connection across states — the per-connection comment at line 197 notes this caused pending-result issues; is that still true on duckdb 1.4.0?
Problem Statement
Each Struct-typed channel's State creates its own DuckDB connection with no thread limit, so DuckDB sizes its TaskScheduler to the host core count. Total worker threads ≈ (number of Struct channels) × (cores − 1). On large hosts this is thousands of mostly-idle threads and can exhaust the cgroup v2 pids.max (containers fail with RuntimeError: can't start new thread).
csp_gateway/server/gateway/csp/state.py:
_USE_DUCKDB_STATE = True(line 29) — DuckDB state is the default.DuckDBState.init (line 200):
self._con = duckdb.connect()— one connection per channel-state, no config={"threads": ...}.State.init (line 524) creates a DuckDBState for every channel whose record type is a Struct.
DuckDB defaults threads to the core count, so each connection starts cores − 1 workers.
Live container (gdb backtrace of all threads): 1764 / 1985 threads in duckdb::TaskScheduler::ExecuteForever.
Measured per duckdb.connect() on a 64-core host: +63 threads; second connect adds another +63 (per-connection, not shared).
Thread count scales with cores (confirmed by reproduction): ~28 channels →
32-core host: ~28 × 31 ≈ 870
64-core host: ~28 × 63 ≈ 1764
384-core host: ~28 × 383 ≈ 10,700
Proposed Solution
Cap threads at connection creation in DuckDBState.init:
self._con = duckdb.connect(config={"threads": N})Verified on 64-core / duckdb 1.4.0: config={"threads": "4"} → +3 threads instead of +63 (in-memory form, no file needed). This makes the thread count independent of host cores.
Alternatives Considered
NA
Open questions
Best default N for these small current-value tables (likely 1–2 is plenty)? Worth making it configurable on the gateway.
Alternatively, share a single DuckDB connection across states — the per-connection comment at line 197 notes this caused pending-result issues; is that still true on duckdb 1.4.0?