Merge remote-tracking branch 'origin/master' into stopping-state

* origin/master:
  Quote all versions in ci.yml
  Correct support versions in README
  Remove Django 2.2
  Add Django 4.0 and Python 3.10 to test matrix, remove unsupported versions
  Remove .python-version
  Reformat to match new Black version
  Blacken INSTALLED_APPS docs
  Bump version
  Blacken docs and reword slightly
  Clarify workspace docs
  Fix black formatting
  formatted w/black; added note about Windows support to README
  Test if `signal` has attribute `SIGQUIT`
  Add instructions for migrating

Author: j4mie

Diff for: .github/workflows/ci.yml

@@ -9,8 +9,13 @@ jobs:
 
     strategy:
       matrix:
-        python: [3.6, 3.7, 3.8, 3.9]
-        django: [3.1, 3.2]
+        python: ["3.6", "3.7", "3.8", "3.9", "3.10"]
+        django: ["3.2", "4.0"]
+        exclude:
+          - python: "3.6"
+            django: "4.0"
+          - python: "3.7"
+            django: "4.0"
         database_url:
           - postgres://runner:password@localhost/project
           - mysql://root:[email protected]/project

Diff for: .python-version

@@ -7,8 +7,8 @@ Simple database-backed job queue. Jobs are defined in your settings, and are pro
 Asynchronous tasks are run via a *job queue*. This system is designed to support multi-step job workflows.
 
 Supported and tested against:
-- Django 3.1 and 3.2
-- Python 3.6, 3.7, 3.8 and 3.9
+- Django 3.2 and 4.0
+- Python 3.6, 3.7, 3.8, 3.9 and 3.10
 
 ## Getting Started
 
@@ -20,10 +20,16 @@ Install from PIP
 
 Add `django_dbq` to your installed apps
 
-    INSTALLED_APPS = (
-        ...
-        'django_dbq',
-    )
+```python
+INSTALLED_APPS = [
+    ...,
+    "django_dbq",
+]
+```
+
+Run migrations
+
+    manage.py migrate
 
 ### Upgrading from 1.x to 2.x
 
@@ -36,6 +42,7 @@ In e.g. project.common.jobs:
 ```python
 import time
 
+
 def my_task(job):
     logger.info("Working hard...")
     time.sleep(10)
@@ -48,8 +55,8 @@ In project.settings:
 
 ```python
 JOBS = {
-    'my_job': {
-        'tasks': ['project.common.jobs.my_task']
+    "my_job": {
+        "tasks": ["project.common.jobs.my_task"],
     },
 }
 ```
@@ -65,16 +72,16 @@ A failure hook receives the failed `Job` instance along with the unhandled excep
 
 ```python
 def my_task_failure_hook(job, e):
-    # delete some temporary files on the filesystem
+    ...  # clean up after failed job
 ```
 
 To ensure this hook gets run, simply add a `failure_hook` key to your job config like so:
 
 ```python
 JOBS = {
-    'my_job': {
-        'tasks': ['project.common.jobs.my_task'],
-        'failure_hook': 'project.common.jobs.my_task_failure_hook'
+    "my_job": {
+        "tasks": ["project.common.jobs.my_task"],
+        "failure_hook": "project.common.jobs.my_task_failure_hook",
     },
 }
 ```
@@ -87,16 +94,16 @@ A creation hook receives your `Job` instance as its only argument. Here's an exa
 
 ```python
 def my_task_creation_hook(job):
-    # configure something before running your job
+    ...  # configure something before running your job
 ```
 
 To ensure this hook gets run, simply add a `creation_hook` key to your job config like so:
 
 ```python
 JOBS = {
-    'my_job': {
-        'tasks': ['project.common.jobs.my_task'],
-        'creation_hook': 'project.common.jobs.my_task_creation_hook'
+    "my_job": {
+        "tasks": ["project.common.jobs.my_task"],
+        "creation_hook": "project.common.jobs.my_task_creation_hook",
     },
 }
 ```
@@ -112,7 +119,7 @@ In another terminal:
 Using the name you configured for your job in your settings, create an instance of Job.
 
 ```python
-Job.objects.create(name='my_job')
+Job.objects.create(name="my_job")
 ```
 
 ### Prioritising jobs
@@ -121,10 +128,11 @@ important emails to users. However, once an hour, you may need to run a _really_
 of emails to be dispatched before it can begin.
 
 In order to make sure that an important job is run before others, you can set the `priority` field to an integer higher than `0` (the default). For example:
+
 ```python
-Job.objects.create(name='normal_job')
-Job.objects.create(name='important_job', priority=1)
-Job.objects.create(name='critical_job', priority=2)
+Job.objects.create(name="normal_job")
+Job.objects.create(name="important_job", priority=1)
+Job.objects.create(name="critical_job", priority=2)
 ```
 
 Jobs will be ordered by their `priority` (highest to lowest) and then the time which they were created (oldest to newest) and processed in that order.
@@ -133,7 +141,10 @@ Jobs will be ordered by their `priority` (highest to lowest) and then the time w
 If you'd like to create a job but have it run at some time in the future, you can use the `run_after` field on the Job model:
 
 ```python
-Job.objects.create(name='scheduled_job', run_after=timezone.now() + timedelta(minutes=10))
+Job.objects.create(
+    name="scheduled_job",
+    run_after=(timezone.now() + timedelta(minutes=10)),
+)
 ```
 
 Of course, the scheduled job will only be run if your `python manage.py worker` process is running at the time when the job is scheduled to run. Otherwise, it will run the next time you start your worker process after that time has passed.
@@ -150,24 +161,47 @@ The top-level abstraction of a standalone piece of work. Jobs are stored in the
 
 Jobs are processed to completion by *tasks*. These are simply Python functions, which must take a single argument - the `Job` instance being processed. A single job will often require processing by more than one task to be completed fully. Creating the task functions is the responsibility of the developer. For example:
 
-    def my_task(job):
-        logger.info("Doing some hard work")
-        do_some_hard_work()
+```python
+def my_task(job):
+    logger.info("Doing some hard work")
+    do_some_hard_work()
+```
 
 ### Workspace
 
-The *workspace* is an area that tasks within a single job can use to communicate with each other. It is implemented as a Python dictionary, available on the `job` instance passed to tasks as `job.workspace`. The initial workspace of a job can be empty, or can contain some parameters that the tasks require (for example, API access tokens, account IDs etc). A single task can edit the workspace, and the modified workspace will be passed on to the next task in the sequence. For example:
+The *workspace* is an area that can be used 1) to provide additional arguments to task functions, and 2) to categorize jobs with additional metadata. It is implemented as a Python dictionary, available on the `job` instance passed to tasks as `job.workspace`. The initial workspace of a job can be empty, or can contain some parameters that the tasks require (for example, API access tokens, account IDs etc).
+
+When creating a Job, the workspace is passed as a keyword argument:
+
+```python
+Job.objects.create(name="my_job", workspace={"key": value})
+```
+
+Then, the task function can access the workspace to get the data it needs to perform its task:
+
+```python
+def my_task(job):
+    cats_import = CatsImport.objects.get(pk=job.workspace["cats_import_id"])
+```
+
+Tasks within a single job can use the workspace to communicate with each other. A single task can edit the workspace, and the modified workspace will be passed on to the next task in the sequence. For example:
 
     def my_first_task(job):
         job.workspace['message'] = 'Hello, task 2!'
 
     def my_second_task(job):
         logger.info("Task 1 says: %s" % job.workspace['message'])
 
-When creating a Job, the workspace is passed as a keyword argument:
+The workspace can be queried like any [JSONField](https://docs.djangoproject.com/en/3.2/topics/db/queries/#querying-jsonfield). For instance, if you wanted to display a list of jobs that a certain user had initiated, add `user_id` to the workspace when creating the job:
 
 ```python
-Job.objects.create(name='my_job', workspace={'key': value})
+Job.objects.create(name="foo", workspace={"user_id": request.user.id})
+```
+
+Then filter the query with it in the view that renders the list:
+
+```python
+user_jobs = Job.objects.filter(workspace__user_id=request.user.id)
 ```
 
 ### Worker process
@@ -208,8 +242,8 @@ from django_dbq.models import Job
 
 ...
 
-Job.objects.create(name='do_work', workspace={})
-Job.objects.create(name='do_other_work', queue_name='other_queue', workspace={})
+Job.objects.create(name="do_work", workspace={})
+Job.objects.create(name="do_other_work", queue_name="other_queue", workspace={})
 
 queue_depths = Job.get_queue_depths()
 print(queue_depths)  # {"default": 1, "other_queue": 1}
@@ -244,6 +278,10 @@ jobs in the "NEW" or "READY" states will be returned.
 
 It may be necessary to supply a DATABASE_PORT environment variable.
 
+## Windows support
+
+Windows is supported on a best-effort basis only, and is not covered by automated or manual testing.
+
 ## Code of conduct
 
 For guidelines regarding the code of conduct when contributing to this repository please review [https://www.dabapps.com/open-source/code-of-conduct/](https://www.dabapps.com/open-source/code-of-conduct/)

Diff for: README.md

@@ -1 +1 @@
-__version__ = "2.1.0"
+__version__ = "2.1.1"

Diff for: django_dbq/__init__.py

@@ -16,7 +16,8 @@ def handle(self, *args, **options):
         queue_depths_string = " ".join(
             [
                 "{queue_name}={queue_depth}".format(
-                    queue_name=queue_name, queue_depth=queue_depths.get(queue_name, 0),
+                    queue_name=queue_name,
+                    queue_depth=queue_depths.get(queue_name, 0),
                 )
                 for queue_name in queue_names
             ]

Diff for: django_dbq/management/commands/queue_depth.py

@@ -25,7 +25,11 @@ def __init__(self, name, rate_limit_in_seconds):
 
     def init_signals(self):
         signal.signal(signal.SIGINT, self.shutdown)
-        signal.signal(signal.SIGQUIT, self.shutdown)
+
+        # for Windows, which doesn't support the SIGQUIT signal
+        if hasattr(signal, "SIGQUIT"):
+            signal.signal(signal.SIGQUIT, self.shutdown)
+
         signal.signal(signal.SIGTERM, self.shutdown)
 
     def shutdown(self, signum, frame):

Diff for: django_dbq/management/commands/worker.py

@@ -49,6 +49,8 @@ class Migration(migrations.Migration):
                     models.CharField(db_index=True, max_length=20, default="default"),
                 ),
             ],
-            options={"ordering": ["-created"],},
+            options={
+                "ordering": ["-created"],
+            },
         ),
     ]

Diff for: django_dbq/migrations/0001_initial.py

@@ -11,5 +11,8 @@ class Migration(migrations.Migration):
     ]
 
     operations = [
-        migrations.AlterModelOptions(name="job", options={"ordering": ["created"]},),
+        migrations.AlterModelOptions(
+            name="job",
+            options={"ordering": ["created"]},
+        ),
     ]

Diff for: django_dbq/migrations/0002_auto_20151016_1027.py

@@ -13,7 +13,8 @@ class Migration(migrations.Migration):
 
     operations = [
         migrations.AlterModelOptions(
-            name="job", options={"ordering": ["-priority", "created"]},
+            name="job",
+            options={"ordering": ["-priority", "created"]},
         ),
         migrations.AddField(
             model_name="job",

Diff for: django_dbq/migrations/0003_auto_20180713_1000.py

@@ -27,6 +27,8 @@ class Migration(migrations.Migration):
             ),
         ),
         migrations.AlterField(
-            model_name="job", name="workspace", field=models.JSONField(null=True),
+            model_name="job",
+            name="workspace",
+            field=models.JSONField(null=True),
         ),
     ]

Diff for: django_dbq/migrations/0004_auto_20210818_0247.py

@@ -103,7 +103,14 @@ def test_queue_depth_multiple_queues(self):
         )
 
         stdout = StringIO()
-        call_command("queue_depth", queue_name=("default", "testqueue",), stdout=stdout)
+        call_command(
+            "queue_depth",
+            queue_name=(
+                "default",
+                "testqueue",
+            ),
+            stdout=stdout,
+        )
         output = stdout.getvalue()
         self.assertEqual(output.strip(), "event=queue_depths default=2 testqueue=2")
 

Diff for: django_dbq/tests.py

@@ -3,4 +3,4 @@ freezegun==0.3.12
 mock==3.0.5
 dj-database-url==0.5.0
 psycopg2==2.8.4
-black==19.10b0
+black==21.12b0