flux-config-exec(5) does not contain much detail about the kill timer

[garlick]

Problem: a question arose about the algorithm used by job-exec to kill processes, and flux-config-exec(5) wasn't super helpful.

Currently we have:

      kill-timeout
              (optional) The amount of time to wait after SIGTERM is sent to a
              job before sending SIGKILL.
      term-signal
              (optional) Specify an alternate signal to SIGTERM when terminat‐
              ing job tasks. Mainly used for testing.
      kill-signal
              (optional) Specify an alternate signal to SIGKILL  when  killing
              tasks and the job shell. Mainly used for testing.

Some observations

• default value for kill-timeout (5s) is missing
• max-kill-count is missing entirely (it has a default of 8)
• the kill algorithm isn't described

Maybe on the latter, we could tack on something like

JOB TERMINATION

When a job is canceled or its time limit is reached, jobs are terminated using the following sequence:

• The job shells are sent SIGTERM
• After kill-timeout, any remaining shells are sent SIGKILL
• This continues with an exponential back-off, with kill-timeout doubling after each attempt (capped at 300s)
• After max-kill-count attempts, any nodes still running processes are drained

Just enough to help people choose the right values for the configuration parameters.

[grondo]

For reference, PR #6101 added max-kill-count and failed to document it :-(

[garlick]

The man page also doesn't mention that kill-timeout is an FSD or that kill-signal and term-signal are strings rather than signal numbers.

I suppose it could also mention that SIGKILL is replaced with SIGUSR1 when the job is started by the IMP.

[grondo]

I suppose it could also mention that SIGKILL is replaced with SIGUSR1 when the job is started by the IMP.

That could be confusing because the net effect is that the job processes are sent SIGKILL?

[garlick]

Yeah if we go there it will require a bit more explanation I guess.