docs(tutorials): add tutorials, FAQ and troubleshooting guide

Doxyfile

@@ -100,7 +100,12 @@ INPUT                  = ./core \
                          ./src \
                          ./include \
                          ./examples \
-                         ./docs/mainpage.dox
+                         ./docs/mainpage.dox \
+                         ./docs/tutorial_threadpool.dox \
+                         ./docs/tutorial_dag.dox \
+                         ./docs/tutorial_lockfree.dox \
+                         ./docs/faq.dox \
+                         ./docs/troubleshooting.dox
 INPUT_ENCODING         = UTF-8
 FILE_PATTERNS          = *.c \
                          *.cc \

docs/faq.dox

@@ -0,0 +1,127 @@
+/**
+@page faq Frequently Asked Questions
+
+@tableofcontents
+
+This FAQ collects the most common questions from developers integrating Thread
+System into their projects. For longer walkthroughs, see the
+@ref tutorial_threadpool, @ref tutorial_dag, and @ref tutorial_lockfree pages.
+
+@section faq_threads How many threads should I use?
+
+Start with @c std::thread::hardware_concurrency() for CPU-bound work. For
+I/O-bound or mixed workloads, oversubscribe by a factor proportional to the
+average wait time over compute time — a 2x to 4x multiplier is a reasonable
+starting point. Avoid setting a fixed worker count without measuring; the
+optimal number depends on the workload, the host, and other processes
+contending for the same cores. The autoscaler can adjust the count over time
+based on queue depth and observed latency.
+
+@section faq_cancel How do I handle task cancellation?
+
+Use a @c kcenon::thread::cancellation_token. Pass the token to long-running
+jobs and check @c is_cancellation_requested() between work units. Cancellation
+is cooperative — the runtime cannot interrupt arbitrary code, so jobs must poll
+periodically. Tokens form a hierarchy: cancelling a parent cancels all linked
+children, which is useful for shutting down a request and all of its background
+fan-out.
+
+@code{.cpp}
+auto token = kcenon::thread::cancellation_token::create();
+pool->submit_task([token]() -> kcenon::thread::result_void {
+    while (!token->is_cancellation_requested()) {
+        // do a chunk of work
+    }
+    return {};
+});
+// later
+token->cancel();
+@endcode
+
+@section faq_async Thread pool vs. std::async — which is better?
+
+@c std::async on most implementations either spawns a new thread per call or
+uses an opaque process-wide pool with no scheduling controls. A dedicated
+thread pool gives you:
+- Bounded resource use.
+- Visibility into queue depth and latency.
+- Priority routing via @c typed_thread_pool.
+- Cooperative cancellation and structured shutdown.
+
+Use @c std::async only for one-off background work in small programs. Anything
+that runs in production with predictable load should use a thread pool.
+
+@section faq_monitoring How do I integrate with monitoring_system?
+
+Thread System exposes its diagnostic counters through the @c diagnostics and
+@c metrics modules. monitoring_system depends on Thread System and consumes
+these counters directly — you do not need to wire anything by hand. If you
+want to publish custom metrics, register a callback with
+@c thread_pool_diagnostics::observe(). Counters include queue depth, worker
+utilization, completed jobs, and per-priority latency histograms.
+
+@section faq_deadlock How do I avoid deadlocks when using the pool?
+
+A few rules prevent the most common cases:
+- Do not call @c future.get() from inside a job submitted to the same pool if
+  the awaited job depends on a worker from that pool — this can starve the
+  pool. Use the DAG scheduler for in-pool dependencies instead.
+- Acquire locks in a consistent global order across jobs.
+- Prefer @c std::scoped_lock to lock multiple mutexes atomically.
+- Avoid holding a mutex across @c submit_task — release first, then submit.
+- Watch for hidden blocking inside callbacks (logging, allocators, mutexes
+  used by other threads). The thread pool diagnostics can flag long-running
+  jobs that point at hidden contention.
+
+@section faq_perf How do I tune the pool for performance?
+
+1. Measure first. Use the included benchmarks or your own to capture baseline
+   throughput and latency.
+2. Right-size the worker count for your workload class.
+3. Try @c adaptive_job_queue (the default) before forcing a specific queue.
+4. Enable work stealing for highly imbalanced workloads only.
+5. Use @c typed_thread_pool to give latency-sensitive work a dedicated worker.
+6. Profile contention with @c perf or @c vtune to spot accidentally shared
+   state inside callbacks.
+
+@section faq_platform Are there platform differences I should know about?
+
+Thread System works on Linux, macOS, and Windows. Notable differences:
+- Linux supports NUMA-aware work stealing; macOS and Windows builds compile
+  the same code but treat the host as a single NUMA node.
+- Windows requires MSVC 2022 or newer for full @c std::format support.
+- macOS uses Grand Central Dispatch underneath @c std::thread, but the
+  thread pool itself does not depend on GCD.
+- ARMv7 and RISC-V are untested. AArch64 (Apple Silicon, Linux ARM64) is
+  supported and benchmarked.
+
+@section faq_memory How does memory management work for jobs and queues?
+
+Jobs are heap-allocated by the queue (typically as @c std::unique_ptr nodes).
+Lock-free queues defer node deletion until hazard pointer scanning confirms
+no thread is reading them. As a user you do not need to manage queue node
+lifetimes; just pass a callable into @c submit_task or build a typed job and
+the framework owns it from there. Avoid capturing very large objects by value
+in the lambda — capture by @c std::shared_ptr or move into the job.
+
+@section faq_errors How are errors propagated from jobs?
+
+Jobs return @c kcenon::thread::result_void or @c result<T>, which wrap the
+@c common_system Result type. Failures surface through the future returned by
+@c submit_task — calling @c .get() rethrows nothing; instead inspect the
+result and call @c get_error() (note: not @c error()) to read the failure
+detail. The DAG scheduler aggregates failures across nodes and exposes them
+through the run result.
+
+@section faq_testing How do I test code that uses the thread pool?
+
+- For unit tests, prefer the @c minimal_thread_pool example as a lightweight
+  fixture: it has predictable startup and shutdown costs.
+- Use a small worker count (1 or 2) so concurrency bugs surface deterministically.
+- Wait on futures returned from @c submit_task; never sleep-and-hope.
+- For races, run the test under TSan (@c cmake --preset tsan) and inside
+  Valgrind on Linux (@c valgrind --tool=helgrind).
+- The repository includes stress and sanitizer presets — use them in CI to
+  catch regressions in your downstream code as well.
+
+*/

docs/mainpage.dox

@@ -202,6 +202,19 @@ cd thread_system
 | **Adapters** | `adapters/` | common_executor_adapter.h, job_queue_adapter.h | Bridges to common_system IExecutor |
 | **Concepts** | `concepts/` | thread_concepts.h | C++20 concepts for thread domain |
 
+@section learning Learning Resources
+
+New to Thread System? Start here. These pages walk through the most common
+integration scenarios and answer the questions developers ask most often.
+
+| Resource | Description |
+|----------|-------------|
+| @ref tutorial_threadpool | Choosing between basic and typed pools, work-stealing, sizing |
+| @ref tutorial_dag | Defining task dependencies, execution order, error handling |
+| @ref tutorial_lockfree | MPMC vs SPSC selection, hazard pointers, performance |
+| @ref faq | Quick answers to the 10 most common integration questions |
+| @ref troubleshooting | Diagnosing deadlocks, leaks, hangs, and performance issues |
+
 @section examples Examples
 
 | Example | Source | Description |

docs/troubleshooting.dox

@@ -0,0 +1,147 @@
+/**
+@page troubleshooting Troubleshooting Guide
+
+@tableofcontents
+
+When something goes wrong with concurrent code, the symptoms are often vague —
+a hang, a crash without a stack trace, or a slowdown that only appears under
+load. This page collects the most frequent issues we see with Thread System and
+the diagnostic steps that resolve them.
+
+@section ts_deadlock 1. Deadlock detection
+
+**Symptoms**: The pool stops making progress, queue depth grows, futures never
+complete.
+
+**Common causes**:
+- A job calls @c future.get() on another job submitted to the same pool, but
+  every worker is blocked waiting on a future, so nothing can run the
+  awaited job.
+- Two jobs acquire the same locks in different orders.
+- A callback blocks on a mutex held elsewhere in user code.
+
+**Diagnostic steps**:
+1. Use @c kcenon::thread::diagnostics::dump_pool_state() (or your debugger) to
+   list worker stack traces. If every worker is parked in @c std::future::wait,
+   you have an in-pool dependency cycle.
+2. Run under @c gdb / @c lldb and inspect each worker's backtrace. Look for
+   matching mutex addresses across two threads.
+3. Enable thread sanitizer (@c cmake --preset tsan) and re-run the failing
+   test. TSan detects most lock-order inversions automatically.
+
+**Fix**: Use the DAG scheduler for in-pool dependencies. Adopt @c std::scoped_lock
+to lock multiple mutexes atomically. Never hold a mutex across @c submit_task.
+
+@section ts_leak 2. Memory leak with futures
+
+**Symptoms**: RSS grows steadily under load even though jobs complete.
+
+**Common causes**:
+- The pool returns a future from @c submit_task and the caller never reads or
+  drops it. The shared state stays alive until the future is destroyed.
+- Lambdas capture large objects by value; the lambda lives until the job
+  finishes, pinning the captures.
+- The hazard pointer retire list never reaches the reclamation threshold
+  because one thread rarely runs.
+
+**Diagnostic steps**:
+1. Run under @c valgrind --tool=memcheck or AddressSanitizer
+   (@c cmake --preset asan).
+2. Use @c heaptrack or @c jemalloc statistics to find the call site that
+   allocates the leaked memory.
+3. Check job lifetimes — long-running jobs delay cleanup of everything they
+   capture.
+
+**Fix**: Drop futures you do not need (or call @c .wait() and let them go).
+Capture large state by @c std::shared_ptr or move it. For hazard pointer
+buildup, occasionally call @c hazard_domain::scan_now() from a maintenance
+thread.
+
+@section ts_platform 3. Platform-specific threading issues
+
+**Symptoms**: The code runs on Linux but hangs on Windows, or works on x86_64
+but crashes on AArch64.
+
+**Common causes**:
+- Relying on a particular memory ordering that is enforced on x86 but not on
+  weakly ordered architectures.
+- Using thread-local storage that is destroyed in a different order on
+  different platforms during shutdown.
+- Differences in @c std::thread::hardware_concurrency() reporting (cgroups on
+  Linux containers, hybrid cores on Windows).
+
+**Diagnostic steps**:
+1. Reproduce on the failing platform under TSan if available.
+2. Audit any custom @c std::atomic usage; default to
+   @c std::memory_order_seq_cst until proven slow.
+3. On Linux containers, check @c /sys/fs/cgroup limits — they affect what
+   @c hardware_concurrency() returns.
+
+**Fix**: Use @c std::memory_order_seq_cst by default, then relax with care
+after profiling. Pin thread-local cleanup order using explicit
+@c thread_local destructors. Configure the pool's worker count from a runtime
+setting instead of trusting platform defaults.
+
+@section ts_perf 4. Performance problems
+
+**Symptoms**: Throughput is lower than expected, tail latency spikes, CPU
+utilization is high without forward progress.
+
+**Common causes**:
+- Workers spend most time in the kernel waking up from condition variables.
+- A single hot lock inside a callback bottlenecks every worker.
+- The job queue is mutex-backed under high contention; switching to lock-free
+  helps.
+- False sharing between counters or job control blocks placed on the same
+  cache line.
+
+**Diagnostic steps**:
+1. Run the @c benchmarks/thread_pool_benchmark and compare against the
+   shipped baseline numbers.
+2. Use @c perf record -F 999 -g to find the hottest functions.
+3. Check @c thread_pool_diagnostics::queue_depth and worker idle counters —
+   if workers are idle while the queue is non-empty, there is a wakeup or
+   stealing problem.
+4. Run the autoscaler in observation-only mode and compare its
+   recommendation to your static configuration.
+
+**Fix**: Switch to @c adaptive_job_queue, enable work stealing for skewed
+loads, split CPU and I/O work into separate pools, and align hot counters to
+cache lines (@c alignas(64)).
+
+@section ts_hang 5. Hang on shutdown
+
+**Symptoms**: The program reaches @c pool->stop() but never returns.
+
+**Common causes**:
+- Pending jobs that wait on a cancellation token nobody cancelled.
+- Futures still held by the caller; the pool destructor blocks until the
+  shared state is released.
+- A worker thread holds a hazard pointer to a node and never reaches the
+  reclamation point.
+- Nested pools — pool A's worker waits on a future from pool B, which is
+  already shutting down.
+
+**Diagnostic steps**:
+1. Attach a debugger and dump every thread's backtrace. Workers parked in
+   @c condition_variable::wait inside @c stop() indicate jobs that never
+   completed.
+2. Confirm cancellation tokens are actually cancelled before @c stop().
+3. Look for futures captured by other long-lived objects.
+
+**Fix**: Cancel any cancellation tokens before stopping the pool. Use
+@c stop(std::chrono::seconds(N)) to bound the wait. Shut down dependent pools
+in reverse dependency order — outermost first.
+
+@section ts_more More help
+
+If these steps do not isolate the issue, open a GitHub issue with:
+- The exact build configuration (compiler, version, CMake preset).
+- A minimal reproduction (the smaller the better).
+- Sanitizer output if available (TSan, ASan, UBSan).
+- @c thread_pool_diagnostics dump captured at the failure point.
+
+See also @ref faq for quick answers and @ref tutorial_threadpool for usage
+patterns that prevent these issues in the first place.
+
+*/