Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep finished jobs in the database for history, but skip it while processing the queue #255

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

sadpenguinn
Copy link

It's a useful feature to keep finished jobs in the database. Especially for production environment where this feature can help to discover bug cases in service logic by keeping failed and successful jobs. Clients can use it as well as logs.
Gue will ensure to keep finished jobs in the database using the job's 'SkipDelete' property. But client should handle and monitor the memory size that these jobs take up and reduce the size when it's necessary.

@sadpenguinn sadpenguinn changed the title Keep finished jobs in the database for history, but skip it while processing queue Keep finished jobs in the database for history, but skip it while processing the queue Jan 30, 2024
@vgarvardt
Copy link
Owner

Hooks were designed specifically for this purpose - you can set one with WithWorkerHooksJobDone or WithPoolHooksJobDone and store finished jobs to the log with all the additional required information. This will fulfill your requirement and keep the table and code untouched.

@kydemy-fran
Copy link

I'd like to comment on this issue.

Deleting the jobs is the right thing to do for performance and tidiness.
But it would be great if the library had the support out of the box to store the completed jobs:

  • Add a Option when configuring the pool: WithStoreDoneJobs
  • Add another migration to create a table: gue_jobs_completed
  • If the Option is set, automatically move the records to the other table.

I can help with a PR if that helps.
Thanks!

@vgarvardt
Copy link
Owner

I would not want to add this functionality to the core library, but I'm thinking about adding some sort of plugins system, so that some pieces of functionality could be easily plugged into the core library w/out changing the core itself. This could be the first candidate as core already have everything to implement this functionality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants