Expired jobs are not stopped

[HEYGUL]

We experienced issues with some long-running jobs which expirein property was set to a too short amount time.
Digging into our system we finally found out that the job was running several times concurrently which led to race conditions accessing our database.

What suprised us at first was the fact that the job was marked as expired.

After investigating pgboss source code we finally understood that an expired job is not cancel thus it is still running in the background and the retry process will start another occurence of same job.

As we have no event sent when marking a job as expired we cannot gracefully stop it on our own.

What would you advice us to do in such a case ?

[timgit]

I think this question is "how can I signal to a running promise to stop itself?". There are patterns, such as implementing AbortController, but it would be up to the job handler to listen to a signal or similar event. Another option would be adding a health check of some sort inside the job handler itself if this situation is not rare but expected.

[AAS97]

Hello there,
I agree that the current expiration behavior is misleading for developers and should be properly documented. The issue is potentially causing resource leaks, unexpected behavior and leave the job running without proper monitoring which is quite risky.
I'm in favor of exposing an AbortSignal as part of the job context as propose by @timgit. Willing developers can utilize it for mid-run cancellation and cleanup while maintaining backward compatibility (existing handler implementations will leave the job running).

If we agree on this solution I'd be happy to get my hands dirty.

[timgit]

I concur. Please be aware the state of the repo involves a near-term semver major to v11 (v11 branch). I'd recommend targeting that to keep the PR easier to merge.

[bcomnes]

I made an attempt at passing the AbortSignal to the jobs. #622