Robust auto-cleaning functionality (#222)

Enables configuration options for auto cleaning functionality, as well as a management command to manually clean.

Author: Archmonger

Diff for: CHANGELOG.md

@@ -37,6 +37,11 @@ Using the following categories, list your changes in this order:
 ### Added
 
 -   Built-in cross-process communication mechanism via the `reactpy_django.hooks.use_channel_layer` hook.
+-   More robust control over ReactPy clean up tasks!
+    -   `settings.py:REACTPY_CLEAN_INTERVAL` to control how often ReactPy automatically performs cleaning tasks.
+    -   `settings.py:REACTPY_CLEAN_SESSIONS` to control whether ReactPy automatically cleans up expired sessions.
+    -   `settings.py:REACTPY_CLEAN_USER_DATA` to control whether ReactPy automatically cleans up orphaned user data.
+    -   `python manage.py clean_reactpy` command to manually perform ReactPy clean up tasks.
 
 ## [3.7.0] - 2024-01-30
 

Diff for: docs/mkdocs.yml

@@ -12,6 +12,7 @@ nav:
           - Utilities: reference/utils.md
           - Template Tag: reference/template-tag.md
           - Settings: reference/settings.md
+          - Management Commands: reference/management-commands.md
     - About:
           - Changelog: about/changelog.md
           - Contributor Guide:

Diff for: docs/src/reference/components.md

@@ -12,7 +12,7 @@ We supply some pre-designed that components can be used to help simplify develop
 
 Automatically convert a Django view into a component.
 
-At this time, this works best with static views that do not rely on HTTP methods other than `GET`.
+At this time, this works best with static views with no interactivity.
 
 Compatible with sync or async [Function Based Views](https://docs.djangoproject.com/en/dev/topics/http/views/) and [Class Based Views](https://docs.djangoproject.com/en/dev/topics/class-based-views/).
 
@@ -186,7 +186,7 @@ Compatible with sync or async [Function Based Views](https://docs.djangoproject.
 
     - No built-in method of signalling events back to the parent component.
     - All provided `#!python *args` and `#!python *kwargs` must be serializable values, since they are encoded into the URL.
-    - The `#!python iframe`'s contents will always load **after** the parent component.
+    - The `#!python iframe` will always load **after** the parent component.
     - CSS styling for `#!python iframe` elements tends to be awkward/difficult.
 
 ??? question "How do I use this for Class Based Views?"

Diff for: docs/src/reference/management-commands.md

@@ -0,0 +1,25 @@
+## Overview
+
+<p class="intro" markdown>
+
+ReactPy exposes Django management commands that can be used to perform various ReactPy-related tasks.
+
+</p>
+
+---
+
+## Clean ReactPy Command
+
+Command used to manually clean ReactPy data.
+
+When using this command without arguments, it will perform all cleaning operations. You can specify only performing specific cleaning operations through arguments such as `--sessions`.
+
+!!! example "Terminal"
+
+    ```bash linenums="0"
+    python manage.py clean_reactpy
+    ```
+
+??? example "See Interface"
+
+    Type `python manage.py clean_reactpy --help` to see the available options.

Diff for: docs/src/reference/settings.md

@@ -56,9 +56,10 @@ Set `#!python REACTPY_DEFAULT_QUERY_POSTPROCESSOR` to `#!python None` to disable
 
 Dotted path to the Django authentication backend to use for ReactPy components. This is only needed if:
 
-1. You are using `#!python AuthMiddlewareStack` and...
-2. You are using Django's `#!python AUTHENTICATION_BACKENDS` setting and...
-3. Your Django user model does not define a `#!python backend` attribute.
+1. You are using `#!python settings.py:REACTPY_AUTO_RELOGIN=True` and...
+2. You are using `#!python AuthMiddlewareStack` and...
+3. You are using Django's `#!python AUTHENTICATION_BACKENDS` setting and...
+4. Your Django user model does not define a `#!python backend` attribute.
 
 ---
 
@@ -84,7 +85,7 @@ This is useful to continuously update `#!python last_login` timestamps and refre
 
 **Example Value(s):** `#!python "my-reactpy-database"`
 
-Multiprocessing-safe database used by ReactPy, typically for session data.
+Multiprocessing-safe database used by ReactPy for database-backed hooks and features.
 
 If configuring this value, it is mandatory to enable our database router like such:
 
@@ -104,7 +105,7 @@ If configuring this value, it is mandatory to enable our database router like su
 
 Cache used by ReactPy, typically for caching disk operations.
 
-We recommend using [`redis`](https://docs.djangoproject.com/en/dev/topics/cache/#redis), [`python-diskcache`](https://grantjenks.com/docs/diskcache/tutorial.html#djangocache), or [`LocMemCache`](https://docs.djangoproject.com/en/dev/topics/cache/#local-memory-caching).
+We recommend using [`redis`](https://docs.djangoproject.com/en/dev/topics/cache/#redis), [`memcache`](https://docs.djangoproject.com/en/dev/topics/cache/#memcached), or [`local-memory caching`](https://docs.djangoproject.com/en/dev/topics/cache/#local-memory-caching).
 
 ---
 
@@ -211,8 +212,48 @@ Maximum number of reconnection attempts before the client gives up.
 
 **Example Value(s):** `#!python 0`, `#!python 60`, `#!python 96000`
 
-Maximum seconds to store ReactPy component sessions.
+Maximum seconds a ReactPy component session is valid for. Invalid sessions are deleted during [ReactPy clean up](#auto-clean-settings).
 
 ReactPy sessions include data such as `#!python *args` and `#!python **kwargs` passed into your `#!jinja {% component %}` template tag.
 
 Use `#!python 0` to not store any session data.
+
+---
+
+## Auto-Clean Settings
+
+---
+
+### `#!python REACTPY_CLEAN_INTERVAL`
+
+**Default:** `#!python 604800`
+
+**Example Value(s):** `#!python 0`, `#!python 3600`, `#!python 86400`, `#!python None`
+
+Minimum seconds between ReactPy automatic clean up operations.
+
+The server will check if the interval has passed after every component disconnection, and will perform a clean if needed.
+
+Set this value to `#!python None` to disable automatic clean up operations.
+
+---
+
+### `#!python REACTPY_CLEAN_SESSIONS`
+
+**Default:** `#!python True`
+
+**Example Value(s):** `#!python False`
+
+Configures whether ReactPy should clean up expired component sessions during automatic clean up operations.
+
+---
+
+### `#!python REACTPY_CLEAN_USER_DATA`
+
+**Default:** `#!python True`
+
+**Example Value(s):** `#!python False`
+
+Configures whether ReactPy should clean up orphaned user data during automatic clean up operations.
+
+Typically, user data does not become orphaned unless the server crashes during a `#!python User` delete operation.

Diff for: docs/src/reference/utils.md

@@ -84,7 +84,7 @@ Typically, this function is automatically called on all components contained wit
 
 This is the default postprocessor for the `#!python use_query` hook.
 
-This postprocessor is designed to avoid Django's `#!python SynchronousOnlyException` by recursively fetching all fields within a `#!python Model` or `#!python QuerySet` to prevent [lazy execution](https://docs.djangoproject.com/en/dev/topics/db/queries/#querysets-are-lazy).
+Since ReactPy is rendered within an `#!python asyncio` loop, this postprocessor is exists to prevent Django's `#!python SynchronousOnlyException` by recursively prefetching fields within a `#!python Model` or `#!python QuerySet`. This prefetching step works to eliminate Django's [lazy execution](https://docs.djangoproject.com/en/dev/topics/db/queries/#querysets-are-lazy) behavior.
 
 === "components.py"
 

Diff for: src/reactpy_django/checks.py

@@ -255,6 +255,15 @@ def reactpy_warnings(app_configs, **kwargs):
             )
         )
 
+    if getattr(settings, "REACTPY_CLEAN_SESSION", None):
+        warnings.append(
+            Warning(
+                "REACTPY_CLEAN_SESSION is not a valid property value.",
+                hint="Did you mean to use REACTPY_CLEAN_SESSIONS instead?",
+                id="reactpy_django.W019",
+            )
+        )
+
     return warnings
 
 
@@ -491,4 +500,43 @@ def reactpy_errors(app_configs, **kwargs):
             )
         )
 
+    if not isinstance(config.REACTPY_CLEAN_INTERVAL, (int, type(None))):
+        errors.append(
+            Error(
+                "Invalid type for REACTPY_CLEAN_INTERVAL.",
+                hint="REACTPY_CLEAN_INTERVAL should be an integer or None.",
+                id="reactpy_django.E023",
+            )
+        )
+
+    if (
+        isinstance(config.REACTPY_CLEAN_INTERVAL, int)
+        and config.REACTPY_CLEAN_INTERVAL < 0
+    ):
+        errors.append(
+            Error(
+                "Invalid value for REACTPY_CLEAN_INTERVAL.",
+                hint="REACTPY_CLEAN_INTERVAL should be a positive integer or None.",
+                id="reactpy_django.E024",
+            )
+        )
+
+    if not isinstance(config.REACTPY_CLEAN_SESSIONS, bool):
+        errors.append(
+            Error(
+                "Invalid type for REACTPY_CLEAN_SESSIONS.",
+                hint="REACTPY_CLEAN_SESSIONS should be a boolean.",
+                id="reactpy_django.E025",
+            )
+        )
+
+    if not isinstance(config.REACTPY_CLEAN_USER_DATA, bool):
+        errors.append(
+            Error(
+                "Invalid type for REACTPY_CLEAN_USER_DATA.",
+                hint="REACTPY_CLEAN_USER_DATA should be a boolean.",
+                id="reactpy_django.E026",
+            )
+        )
+
     return errors

Diff for: src/reactpy_django/clean.py

@@ -0,0 +1,141 @@
+from __future__ import annotations
+
+import logging
+from datetime import datetime, timedelta
+from typing import TYPE_CHECKING, Literal
+
+from django.contrib.auth import get_user_model
+from django.utils import timezone
+
+_logger = logging.getLogger(__name__)
+
+if TYPE_CHECKING:
+    from reactpy_django.models import Config
+
+CLEAN_NEEDED_BY: datetime = datetime(
+    year=1, month=1, day=1, tzinfo=timezone.now().tzinfo
+)
+
+
+def clean(
+    *args: Literal["all", "sessions", "user_data"],
+    immediate: bool = False,
+    verbosity: int = 1,
+):
+    from reactpy_django.config import (
+        REACTPY_CLEAN_SESSIONS,
+        REACTPY_CLEAN_USER_DATA,
+    )
+    from reactpy_django.models import Config
+
+    config = Config.load()
+    if immediate or is_clean_needed(config):
+        config.cleaned_at = timezone.now()
+        config.save()
+        sessions = REACTPY_CLEAN_SESSIONS
+        user_data = REACTPY_CLEAN_USER_DATA
+
+        if args:
+            sessions = any(value in args for value in {"sessions", "all"})
+            user_data = any(value in args for value in {"user_data", "all"})
+
+        if sessions:
+            clean_sessions(verbosity)
+        if user_data:
+            clean_user_data(verbosity)
+
+
+def clean_sessions(verbosity: int = 1):
+    """Deletes expired component sessions from the database.
+    As a performance optimization, this is only run once every REACTPY_SESSION_MAX_AGE seconds.
+    """
+    from reactpy_django.config import REACTPY_DEBUG_MODE, REACTPY_SESSION_MAX_AGE
+    from reactpy_django.models import ComponentSession
+
+    if verbosity >= 2:
+        print("Cleaning ReactPy component sessions...")
+
+    start_time = timezone.now()
+    expiration_date = timezone.now() - timedelta(seconds=REACTPY_SESSION_MAX_AGE)
+    session_objects = ComponentSession.objects.filter(
+        last_accessed__lte=expiration_date
+    )
+
+    if verbosity >= 2:
+        print(f"Deleting {session_objects.count()} expired component sessions...")
+
+    session_objects.delete()
+
+    if REACTPY_DEBUG_MODE or verbosity >= 2:
+        inspect_clean_duration(start_time, "component sessions", verbosity)
+
+
+def clean_user_data(verbosity: int = 1):
+    """Delete any user data that is not associated with an existing `User`.
+    This is a safety measure to ensure that we don't have any orphaned data in the database.
+
+    Our `UserDataModel` is supposed to automatically get deleted on every `User` delete signal.
+    However, we can't use Django to enforce this relationship since ReactPy can be configured to
+    use any database.
+    """
+    from reactpy_django.config import REACTPY_DEBUG_MODE
+    from reactpy_django.models import UserDataModel
+
+    if verbosity >= 2:
+        print("Cleaning ReactPy user data...")
+
+    start_time = timezone.now()
+    user_model = get_user_model()
+    all_users = user_model.objects.all()
+    all_user_pks = all_users.values_list(user_model._meta.pk.name, flat=True)  # type: ignore
+
+    # Django doesn't support using QuerySets as an argument with cross-database relations.
+    if user_model.objects.db != UserDataModel.objects.db:
+        all_user_pks = list(all_user_pks)  # type: ignore
+
+    user_data_objects = UserDataModel.objects.exclude(user_pk__in=all_user_pks)
+
+    if verbosity >= 2:
+        print(
+            f"Deleting {user_data_objects.count()} user data objects not associated with an existing user..."
+        )
+
+    user_data_objects.delete()
+
+    if REACTPY_DEBUG_MODE or verbosity >= 2:
+        inspect_clean_duration(start_time, "user data", verbosity)
+
+
+def is_clean_needed(config: Config | None = None) -> bool:
+    """Check if a clean is needed. This function avoids unnecessary database reads by caching the
+    CLEAN_NEEDED_BY date."""
+    from reactpy_django.config import REACTPY_CLEAN_INTERVAL
+    from reactpy_django.models import Config
+
+    global CLEAN_NEEDED_BY
+
+    if REACTPY_CLEAN_INTERVAL is None:
+        return False
+
+    if timezone.now() >= CLEAN_NEEDED_BY:
+        config = config or Config.load()
+        CLEAN_NEEDED_BY = config.cleaned_at + timedelta(seconds=REACTPY_CLEAN_INTERVAL)
+
+    return timezone.now() >= CLEAN_NEEDED_BY
+
+
+def inspect_clean_duration(start_time: datetime, task_name: str, verbosity: int):
+    clean_duration = timezone.now() - start_time
+
+    if verbosity >= 3:
+        print(
+            f"Cleaned ReactPy {task_name} in {clean_duration.total_seconds()} seconds."
+        )
+
+    if clean_duration.total_seconds() > 1:
+        _logger.warning(
+            "ReactPy has taken %s seconds to clean %s. "
+            "This may indicate a performance issue with your system, cache, or database.",
+            clean_duration.total_seconds(),
+            task_name,
+        )

Diff for: src/reactpy_django/components.py

@@ -259,7 +259,7 @@ def _django_js(static_path: str):
     return html.script(_cached_static_contents(static_path))
 
 
-def _cached_static_contents(static_path: str):
+def _cached_static_contents(static_path: str) -> str:
     from reactpy_django.config import REACTPY_CACHE
 
     # Try to find the file within Django's static files
@@ -272,7 +272,7 @@ def _cached_static_contents(static_path: str):
     # Fetch the file from cache, if available
     last_modified_time = os.stat(abs_path).st_mtime
     cache_key = f"reactpy_django:static_contents:{static_path}"
-    file_contents = caches[REACTPY_CACHE].get(
+    file_contents: str | None = caches[REACTPY_CACHE].get(
         cache_key, version=int(last_modified_time)
     )
     if file_contents is None:

Diff for: src/reactpy_django/config.py

@@ -118,3 +118,18 @@
     "REACTPY_AUTO_RELOGIN",
     False,
 )
+REACTPY_CLEAN_INTERVAL: int | None = getattr(
+    settings,
+    "REACTPY_CLEAN_INTERVAL",
+    604800,  # Default to 7 days
+)
+REACTPY_CLEAN_SESSIONS: bool = getattr(
+    settings,
+    "REACTPY_CLEAN_SESSIONS",
+    True,
+)
+REACTPY_CLEAN_USER_DATA: bool = getattr(
+    settings,
+    "REACTPY_CLEAN_USER_DATA",
+    True,
+)