From be327ea32eb061107e9932775a5784d516d9df18 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Tue, 15 Oct 2024 13:53:41 +0200 Subject: [PATCH 01/11] docs: use glossary as reference material --- docs/{concepts => reference}/glossary.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) rename docs/{concepts => reference}/glossary.rst (92%) diff --git a/docs/concepts/glossary.rst b/docs/reference/glossary.rst similarity index 92% rename from docs/concepts/glossary.rst rename to docs/reference/glossary.rst index 05046c86..f38d1242 100644 --- a/docs/concepts/glossary.rst +++ b/docs/reference/glossary.rst @@ -1,8 +1,7 @@ -Open edX filters glossary +Open edX Filters Glossary ########################## -This glossary provides definitions for some of the concepts needed to use the Open edX Filters library. - +This glossary provides definitions for some of the terms to ease the adoption of the Open edX Filters library. Pipelines --------- @@ -19,6 +18,11 @@ Open edX Filter An Open edX Filter is a Python class that inherits from `OpenEdXPublicFilter`, which is used for executing pipelines or list of functions in specific order. It implements a `run_filter` method that receives the data to be processed and returns the output of the pipeline. +Filter definition +----------------- + +It's the class that implements the `run_filter` method, usually implemented in this repository for community use. It's invoked by services to execute configured pipeline steps. + Open edX Filter signature ------------------------- From d8303e64116907a79c274ade96872f571ed22efe Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Tue, 15 Oct 2024 08:32:40 -0400 Subject: [PATCH 02/11] docs: use glossary directive --- docs/reference/glossary.rst | 146 ++++++++++++++++-------------------- 1 file changed, 63 insertions(+), 83 deletions(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index f38d1242..484adb2a 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -3,88 +3,68 @@ Open edX Filters Glossary This glossary provides definitions for some of the terms to ease the adoption of the Open edX Filters library. -Pipelines ---------- - -A pipeline is a list of functions that are executed in order. Each function receives the output of the previous function as input. The output of the last function is the output of the pipeline. - -Pipeline steps --------------- - -A pipeline step is a function that receives data, manipulates it and returns it. It can be used to transform data, to validate it, to filter it, to enrich it, etc. - -Open edX Filter ---------------- - -An Open edX Filter is a Python class that inherits from `OpenEdXPublicFilter`, which is used for executing pipelines or list of functions in specific order. It implements a `run_filter` method that receives the data to be processed and returns the output of the pipeline. - -Filter definition ------------------ - -It's the class that implements the `run_filter` method, usually implemented in this repository for community use. It's invoked by services to execute configured pipeline steps. - -Open edX Filter signature -------------------------- - -It's the signature of the `run_filter` method of each filter. It defines the input and output of the filter. The input is a dictionary with the data to be processed and the output is a dictionary with the processed data. - -Open edX Filters' pipeline steps --------------------------------- - -In the context of Open edX Filters, a pipeline step is a class that inherits from ``PipelineStep`` that implements the `run_filter` method which must match the Open edX Filter signature. - -Filter type ------------ - -It's the filter identifier. It's used to identify the filter in the configuration settings. When configuring the pipeline for a filter, the type is as an index for the filter configuration. - -Filter exceptions ------------------ - -Besides acting as a filter, an Open edX Filter can also raise exceptions. These exceptions are used to control the execution of the pipeline. If an exception is raised, the pipeline execution is stopped and the exception is raised again as the output of the pipeline. These exceptions are intentionally raised by the developer during the filter's execution when a condition is met. - -Filter configuration --------------------- - -The filter configuration is a dictionary with the configuration settings for the filter. It's used to configure the pipeline for a filter. The configuration settings are specific for each filter type. The dictionary looks like this: - -.. code-block:: python - - OPEN_EDX_FILTERS_CONFIG = { - "": { - "fail_silently": , - "pipeline": [ - "", - "", - ... - "", - ] - }, - } - -Where: - -- ```` is the filter type. -- ``fail_silently`` is a boolean value. - -If ``fail_silently`` is ``True``: when a pipeline step raises a runtime exception -- like ``ImportError`` or ``AttributeError`` exceptions which are not intentionally raised by the developer during the filter's execution; the exception won't be propagated and the pipeline execution will resume, i.e the next steps will be executed -If ``fail_silently`` is ``False``: the exception will be propagated and the pipeline execution will stop. - -For example, with this configuration: - -.. code-block:: python - - OPEN_EDX_FILTERS_CONFIG = { - "": { - "fail_silently": True, - "pipeline": [ - "non_existing_module.non_existing_function", - "existing_module.function_raising_attribute_error", - "existing_module.existing_function", - ] - }, - } - -The pipeline tooling will catch the ``ImportError`` exception raised by the first step and the ``AttributeError`` exception raised by the second step, then continue and execute the third step. Now, if ``fail_silently`` is ``False``, the pipeline tooling will catch the ``ImportError`` exception raised by the first step and propagate it, i.e the pipeline execution will stop. +.. glossary:: + + Pipeline + A pipeline is a list of functions that are executed in order. Each function receives the output of the previous function as input. The output of the last function is the output of the filter itself. + + Pipeline Steps + A pipeline step is a function that receives, manipulates, and returns data. It can be used to transform data, validate it, filter it, enrich it, etc. It's a class that inherits from ``PipelineStep`` that implements the ``run_filter`` method which must match the Open edX Filter signature. + + Filter Definition + A filter definition is the class that implements the ``run_filter`` method, which is usually implemented in this repository for community use. Services invoke it to execute configured pipeline steps. + + Open edX Filter signature + It's the signature of each filter's ``run_filter`` method in the class inheriting from ``OpenEdxPublicFilter``. The signature specifies the filter's input and output. + + Filter type + It's the filter identifier. It's used to identify the filter in the configuration settings. When configuring the pipeline for a filter, the type is as an index for the filter configuration. + + Filter exceptions + + Besides acting as a filter, an Open edX Filter can also raise exceptions. These exceptions are used to control the execution of the pipeline. If an exception is raised, the pipeline execution is stopped and the exception is raised again as the output of the pipeline. These exceptions are intentionally raised by the developer during the filter's execution when a condition is met. + + Filter configuration + + The filter configuration is a dictionary with the configuration settings for the filter. It's used to configure the pipeline for a filter. The configuration settings are specific for each filter type. The dictionary looks like this: + + .. code-block:: python + + OPEN_EDX_FILTERS_CONFIG = { + "": { + "fail_silently": , + "pipeline": [ + "", + "", + ... + "", + ] + }, + } + + Where: + + - ```` is the filter type. + - ``fail_silently`` is a boolean value. + + If ``fail_silently`` is ``True``: when a pipeline step raises a runtime exception -- like ``ImportError`` or ``AttributeError`` exceptions which are not intentionally raised by the developer during the filter's execution; the exception won't be propagated and the pipeline execution will resume, i.e the next steps will be executed + If ``fail_silently`` is ``False``: the exception will be propagated and the pipeline execution will stop. + + For example, with this configuration: + + .. code-block:: python + + OPEN_EDX_FILTERS_CONFIG = { + "": { + "fail_silently": True, + "pipeline": [ + "non_existing_module.non_existing_function", + "existing_module.function_raising_attribute_error", + "existing_module.existing_function", + ] + }, + } + + The pipeline tooling will catch the ``ImportError`` exception raised by the first step and the ``AttributeError`` exception raised by the second step, then continue and execute the third step. Now, if ``fail_silently`` is ``False``, the pipeline tooling will catch the ``ImportError`` exception raised by the first step and propagate it, i.e the pipeline execution will stop. - ``pipeline`` is list of paths for each pipeline step. Each path is a string with the following format: ``.``. The module path is the path to the module where the pipeline step class is defined and the class name is the name of the class that implements the ``run_filter`` method to be executed. From 17128178e39d4d6fc9158d57cbfc1a06558a55b3 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Tue, 15 Oct 2024 14:35:33 +0200 Subject: [PATCH 03/11] fix: add glossary to reference index --- docs/reference/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/reference/index.rst b/docs/reference/index.rst index de14b47f..41927696 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -5,5 +5,6 @@ References :maxdepth: 1 :caption: Contents: + glossary filters django-plugins-and-filters From 4a925464992ddd0cf2557b49afd5818c4416ce1c Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 31 Oct 2024 14:01:31 +0100 Subject: [PATCH 04/11] refactor: improve wording in glossary and reference it throughout docs --- docs/how-tos/create-new-filter.rst | 28 +++---- docs/how-tos/using-filters.rst | 10 +-- docs/reference/django-plugins-and-filters.rst | 4 +- docs/reference/glossary.rst | 74 +++++-------------- 4 files changed, 40 insertions(+), 76 deletions(-) diff --git a/docs/how-tos/create-new-filter.rst b/docs/how-tos/create-new-filter.rst index 3d879bf3..6a1e52d3 100644 --- a/docs/how-tos/create-new-filter.rst +++ b/docs/how-tos/create-new-filter.rst @@ -20,6 +20,8 @@ Assumptions * You have a basic understanding of Python and Django. * You understand the concept of filters or have reviewed the relevant :doc:`/concepts/index` docs. +* You are familiar with the terminology used in the project, such as + `:term: filter type`, `:term: filter signature`, etc. If not, you can review the :doc:`/reference/glossary` docs. Steps ***** @@ -34,10 +36,10 @@ Steps #. Place your filter in an architecture subdomain - As specified in the Architectural Decisions Record (ADR) filter naming and versioning, the filter definition needs an Open edX Architecture + As specified in the Architectural Decisions Record (ADR) filter naming and versioning, the `:term: filter definition` needs an Open edX Architecture Subdomain for: - - The type of the filter: ``{Reverse DNS}.{Architecture Subdomain}.{Subject}.{Action}.{Major Version}`` + - The `:term: type of the filter`: ``{Reverse DNS}.{Architecture Subdomain}.{Subject}.{Action}.{Major Version}`` - The package name where the definition will live, eg. ``learning/``. For those reasons, after studying your new filter purpose, you must place it in one of the subdomains already in use, or introduce a new subdomain: @@ -57,20 +59,20 @@ Steps Defining the filter's behavior includes: - - Defining the filter type for identification - - Defining the filter's signature + - Defining the `:term: filter type` for identification + - Defining the `:term: filter signature` - Defining the filter's behavior for stopping the process in which it is being used - The filter type is the name that will be used to identify the filter's and it'd help others identifying its purpose. For example, if you're creating a filter that will be used during the student registration process in the LMS, - according to the documentation, the filter type is defined as follows: + The `:term: filter type` is the name that will be used to identify the filter's and it'd help others identifying its purpose. For example, if you're creating a filter that will be used during the student registration process in the LMS, + according to the documentation, the `:term: filter type` is defined as follows: ``{Reverse DNS}.{Architecture Subdomain}.student.registration.requested.{Major Version}`` Where ``student`` is the subject and ``registration.requested`` the action being performed. The major version is the version of the filter, which will be incremented when a change is made to the filter that is not backwards compatible, as explained in the ADR. - Now that you have the filter type, you'll need to define the filter's signature and overall behavior. The filter's signature, which is the set of parameters that the filter will manipulate, depends on where the filter is located. For example, - if you're creating a filter that will be used during the student registration process in the LMS, the filter's signature will be the set of parameters available for that time for the user. In this case, the filter's signature will be the set of parameters that the registration form sends to the LMS. + Now that you have the `:term: filter type`, you'll need to define the `:term: filter signature` and overall behavior. The `:term: filter signature`, which is the set of parameters that the filter will manipulate, depends on where the filter is located. For example, + if you're creating a filter that will be used during the student registration process in the LMS, the `:term: filter signature` will be the set of parameters available for that time for the user. In this case, the `:term: filter signature` will be the set of parameters that the registration form sends to the LMS. You can ask yourself the following questions to help you figure out your filter's parameters: @@ -78,7 +80,7 @@ Steps - What parameters will the filter need to to that? (e.g. the email address) - Where in the registration process will the filter be used? (e.g. after the student submits the registration form but before anything else) - With that information, you can define the filter's signature: + With that information, you can define the `:term: filter signature`: - Arguments: ``email``. Since we want this filter to be broadly used, we'll add as much relevant information as possible for the user at that point. As we mentioned above, we can send more information stored in the registration form like ``name`` or ``username``. - Returns: since filters take in a set of parameters and return a set of parameters, we'll return the same set of parameters that we received. @@ -126,8 +128,8 @@ Steps Some things to note: - - The filter's type is defined in the ``filter_type`` class attribute. In this case, the filter type is ``org.openedx.learning.student.registration.requested.v1``. - - The filter's signature is defined in the ``run_filter`` method. In this case, the signature is the ``form_data`` parameter. + - The filter's type is defined in the ``filter_type`` class attribute. In this case, the `:term: filter type` is ``org.openedx.learning.student.registration.requested.v1``. + - The `:term: filter signature` is defined in the ``run_filter`` method. In this case, the signature is the ``form_data`` parameter. - The ``run_filter`` is a class method that returns the same set of parameters that it receives. - The ``run_filter`` class method calls the ``run_pipeline`` method, which is the method that executes the filter's logic. This method is defined in the ``OpenEdxPublicFilter`` class, which is the base class for all the filters in the library. This method returns a dictionary with the following structure: @@ -148,7 +150,7 @@ Steps "form_data": form_data, } - Where ``form_data`` is the same set of parameters that the filter receives, which is the accumulated output for the filter's pipeline. That is how ``run_filter`` should always look like. + Where ``form_data`` is the same set of parameters that the filter receives, which is the accumulated output for the `:term: filter pipeline`. That is how ``run_filter`` should always look like. - The filter's behavior for stopping the process is defined in the ``PreventRegistration`` exception which inherits from the ``OpenEdxFilterException`` base exception. In this case, the exception is raised when the filter stops the registration process. This is done in the service where the filter is being used, which in this case is the LMS. - The class name is the filter's type ``{Subject}.{Action}`` part in a camel case format. In this case, the filter's name is ``StudentRegistrationRequested``. @@ -196,7 +198,7 @@ Steps self.assertDictContainsSubset(attributes, exception.__dict__) .. note:: - Basically, we're testing the filter's signature and the filter's behavior for stopping the process. The first test is testing the filter's signature, which is the set of parameters that the filter receives and returns. The second test is testing the filter's behavior for stopping the process, which is the exception that is raised when the filter stops the process. + Basically, we're testing the `:term: filter signature` and the filter's behavior for stopping the process. The first test is testing the `:term: filter signature`, which is the set of parameters that the filter receives and returns. The second test is testing the filter's behavior for stopping the process, which is the exception that is raised when the filter stops the process. .. .. seealso:: diff --git a/docs/how-tos/using-filters.rst b/docs/how-tos/using-filters.rst index e3f71a02..51886d78 100644 --- a/docs/how-tos/using-filters.rst +++ b/docs/how-tos/using-filters.rst @@ -2,7 +2,7 @@ How to use Open edX Filters --------------------------- Using openedx-filters in your code is very straight forward. We can consider the -various use cases: implementing pipeline steps, attaching/hooking pipelines to filter, +various use cases: implementing `:term: Pipeline Steps`, attaching/hooking pipelines to filter, and triggering a filter. We'll also cover how to test the filters you create in your service. @@ -11,8 +11,8 @@ Implement pipeline steps Let's say you want to consult student's information with a third party service before generating the students certificate. This is a common use case for filters, -where the functions part of the filter's pipeline will perform the consulting tasks and -decide the execution flow for the application. These functions are the pipeline steps, +where the functions part of the `:term: filter pipeline` will perform the consulting tasks and +decide the execution flow for the application. These functions are the `:term: pipeline steps`, and can be implemented in an installable Python library: .. code-block:: python @@ -83,8 +83,8 @@ There's two key components to the implementation: Attach/hook pipeline to filter ****************************** -After implementing the pipeline steps, we have to tell the certificate creation -filter to execute our pipeline. +After implementing the `:term: pipeline steps`, we have to tell the certificate creation +filter to execute our `:term: pipeline`. .. code-block:: python diff --git a/docs/reference/django-plugins-and-filters.rst b/docs/reference/django-plugins-and-filters.rst index 5965e54c..5b241b0a 100644 --- a/docs/reference/django-plugins-and-filters.rst +++ b/docs/reference/django-plugins-and-filters.rst @@ -37,5 +37,5 @@ file. The dictionary has the following structure: Create pipeline steps ********************* -In your own plugin, you can create your own pipeline steps by inheriting from ``PipelineStep`` and implementing the -``run_filter`` method. You can find examples of pipeline steps in the ``openedx-filters-samples`` repository. See :doc:`/quickstarts/index` for more details. +In your own plugin, you can create your own `:term: pipeline steps` by inheriting from ``PipelineStep`` and implementing the +``run_filter`` method. You can find examples of `:term: pipeline steps` in the ``openedx-filters-samples`` repository. See :doc:`/quickstarts/index` for more details. diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index 484adb2a..1b49d3ed 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -1,70 +1,32 @@ Open edX Filters Glossary ########################## -This glossary provides definitions for some of the terms to ease the adoption of the Open edX Filters library. +A filter has multiple components, such as pipeline, pipeline steps, filter definition, filter signature, filter type, filter exceptions, filter configuration, etc. This glossary provides definitions for these components. + +Here's a brief explanation of each component: .. glossary:: - Pipeline - A pipeline is a list of functions that are executed in order. Each function receives the output of the previous function as input. The output of the last function is the output of the filter itself. + Pipeline + A pipeline is a list of functions executed in a specific order; these functions are known as pipeline steps. Each function in the pipeline takes the output of the previous function as its input, with the final function's output serving as the overall output of the filter. These pipelines are configured in the filter configuration and are executed in sequence. - Pipeline Steps - A pipeline step is a function that receives, manipulates, and returns data. It can be used to transform data, validate it, filter it, enrich it, etc. It's a class that inherits from ``PipelineStep`` that implements the ``run_filter`` method which must match the Open edX Filter signature. + Pipeline Step + A pipeline step is a function within a pipeline that receives, processes, and returns data. Each step may perform operations like transforming, validating, filtering, or enriching data. Pipeline steps are implemented as classes that inherit from a base step class and define specific logic within their ``run_filter`` method, which conforms to the filter's signature. Filter Definition - A filter definition is the class that implements the ``run_filter`` method, which is usually implemented in this repository for community use. Services invoke it to execute configured pipeline steps. - - Open edX Filter signature - It's the signature of each filter's ``run_filter`` method in the class inheriting from ``OpenEdxPublicFilter``. The signature specifies the filter's input and output. - - Filter type - It's the filter identifier. It's used to identify the filter in the configuration settings. When configuring the pipeline for a filter, the type is as an index for the filter configuration. + A filter definition is the class that defines the ``run_filter`` method for a filter, specifying the input and output behavior. This class, which inherits from a standard filter base, executes the configured pipeline steps, enabling custom processing within the defined filter. - Filter exceptions + Filter Signature + The filter signature consists of the specific parameters required by a filter's ``run_filter`` method. It defines the expected input and output structure for the filter, detailing the data the filter will process. - Besides acting as a filter, an Open edX Filter can also raise exceptions. These exceptions are used to control the execution of the pipeline. If an exception is raised, the pipeline execution is stopped and the exception is raised again as the output of the pipeline. These exceptions are intentionally raised by the developer during the filter's execution when a condition is met. + Filter Type + The filter type is a unique identifier for the filter, following a standardized format (e.g., reverse DNS style). This type is used as an index for configuring the filter pipeline and specifies which configuration settings apply to a given filter. - Filter configuration + Filter Exceptions + Filters can raise exceptions to control the flow of the pipeline. If a filter raises an exception, the pipeline halts, and the exception becomes the pipeline's output. Exceptions are typically raised when certain conditions specified in the filter's logic are met, signaling an event or state change. - The filter configuration is a dictionary with the configuration settings for the filter. It's used to configure the pipeline for a filter. The configuration settings are specific for each filter type. The dictionary looks like this: - - .. code-block:: python - - OPEN_EDX_FILTERS_CONFIG = { - "": { - "fail_silently": , - "pipeline": [ - "", - "", - ... - "", - ] - }, - } - - Where: - - - ```` is the filter type. - - ``fail_silently`` is a boolean value. - - If ``fail_silently`` is ``True``: when a pipeline step raises a runtime exception -- like ``ImportError`` or ``AttributeError`` exceptions which are not intentionally raised by the developer during the filter's execution; the exception won't be propagated and the pipeline execution will resume, i.e the next steps will be executed - If ``fail_silently`` is ``False``: the exception will be propagated and the pipeline execution will stop. - - For example, with this configuration: - - .. code-block:: python - - OPEN_EDX_FILTERS_CONFIG = { - "": { - "fail_silently": True, - "pipeline": [ - "non_existing_module.non_existing_function", - "existing_module.function_raising_attribute_error", - "existing_module.existing_function", - ] - }, - } - - The pipeline tooling will catch the ``ImportError`` exception raised by the first step and the ``AttributeError`` exception raised by the second step, then continue and execute the third step. Now, if ``fail_silently`` is ``False``, the pipeline tooling will catch the ``ImportError`` exception raised by the first step and propagate it, i.e the pipeline execution will stop. + Filter Configuration + Filter configuration is a dictionary that defines the pipeline settings for a filter. Each filter type has its own configuration, which includes settings like whether errors should fail silently or propagate, and the sequence of pipeline steps. Configurations specify the filter type, error-handling preferences, and a list of module paths for each pipeline step to be executed. -- ``pipeline`` is list of paths for each pipeline step. Each path is a string with the following format: ``.``. The module path is the path to the module where the pipeline step class is defined and the class name is the name of the class that implements the ``run_filter`` method to be executed. +.. note:: + In practice, "filter" is used to refer to the whole mechanism, including the pipeline steps, filter definition and so on. From d11e2c9ec1642ce7b2ada371dd23dbc29af8ae09 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 4 Nov 2024 14:05:37 +0100 Subject: [PATCH 05/11] fix: fix indentation for glossary entries --- docs/reference/glossary.rst | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index 1b49d3ed..e9146080 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -8,25 +8,25 @@ Here's a brief explanation of each component: .. glossary:: Pipeline - A pipeline is a list of functions executed in a specific order; these functions are known as pipeline steps. Each function in the pipeline takes the output of the previous function as its input, with the final function's output serving as the overall output of the filter. These pipelines are configured in the filter configuration and are executed in sequence. + A pipeline is a list of functions executed in a specific order; these functions are known as pipeline steps. Each function in the pipeline takes the output of the previous function as its input, with the final function's output serving as the overall output of the filter. These pipelines are configured in the filter configuration and are executed in sequence. - Pipeline Step - A pipeline step is a function within a pipeline that receives, processes, and returns data. Each step may perform operations like transforming, validating, filtering, or enriching data. Pipeline steps are implemented as classes that inherit from a base step class and define specific logic within their ``run_filter`` method, which conforms to the filter's signature. + Pipeline Step + A pipeline step is a function within a pipeline that receives, processes, and returns data. Each step may perform operations like transforming, validating, filtering, or enriching data. Pipeline steps are implemented as classes that inherit from a base step class and define specific logic within their ``run_filter`` method, which conforms to the filter's signature. - Filter Definition - A filter definition is the class that defines the ``run_filter`` method for a filter, specifying the input and output behavior. This class, which inherits from a standard filter base, executes the configured pipeline steps, enabling custom processing within the defined filter. + Filter Definition + A filter definition is the class that defines the ``run_filter`` method for a filter, specifying the input and output behavior. This class, which inherits from a standard filter base, executes the configured pipeline steps, enabling custom processing within the defined filter. - Filter Signature - The filter signature consists of the specific parameters required by a filter's ``run_filter`` method. It defines the expected input and output structure for the filter, detailing the data the filter will process. + Filter Signature + The filter signature consists of the specific parameters required by a filter's ``run_filter`` method. It defines the expected input and output structure for the filter, detailing the data the filter will process. - Filter Type - The filter type is a unique identifier for the filter, following a standardized format (e.g., reverse DNS style). This type is used as an index for configuring the filter pipeline and specifies which configuration settings apply to a given filter. + Filter Type + The filter type is a unique identifier for the filter, following a standardized format (e.g., reverse DNS style). This type is used as an index for configuring the filter pipeline and specifies which configuration settings apply to a given filter. - Filter Exceptions - Filters can raise exceptions to control the flow of the pipeline. If a filter raises an exception, the pipeline halts, and the exception becomes the pipeline's output. Exceptions are typically raised when certain conditions specified in the filter's logic are met, signaling an event or state change. + Filter Exceptions + Filters can raise exceptions to control the flow of the pipeline. If a filter raises an exception, the pipeline halts, and the exception becomes the pipeline's output. Exceptions are typically raised when certain conditions specified in the filter's logic are met, signaling an event or state change. - Filter Configuration - Filter configuration is a dictionary that defines the pipeline settings for a filter. Each filter type has its own configuration, which includes settings like whether errors should fail silently or propagate, and the sequence of pipeline steps. Configurations specify the filter type, error-handling preferences, and a list of module paths for each pipeline step to be executed. + Filter Configuration + Filter configuration is a dictionary that defines the pipeline settings for a filter. Each filter type has its own configuration, which includes settings like whether errors should fail silently or propagate, and the sequence of pipeline steps. Configurations specify the filter type, error-handling preferences, and a list of module paths for each pipeline step to be executed. .. note:: In practice, "filter" is used to refer to the whole mechanism, including the pipeline steps, filter definition and so on. From c7025e9164570a79b8d6459a52fd7289ae39f8ba Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 4 Nov 2024 14:15:19 +0100 Subject: [PATCH 06/11] fix: use glossary reference correctly --- docs/how-tos/create-new-filter.rst | 28 +++++++++---------- docs/how-tos/using-filters.rst | 10 +++---- docs/reference/django-plugins-and-filters.rst | 4 +-- 3 files changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/how-tos/create-new-filter.rst b/docs/how-tos/create-new-filter.rst index 6a1e52d3..41ce5cea 100644 --- a/docs/how-tos/create-new-filter.rst +++ b/docs/how-tos/create-new-filter.rst @@ -21,7 +21,7 @@ Assumptions * You understand the concept of filters or have reviewed the relevant :doc:`/concepts/index` docs. * You are familiar with the terminology used in the project, such as - `:term: filter type`, `:term: filter signature`, etc. If not, you can review the :doc:`/reference/glossary` docs. + :term:`filter type`, :term:`filter signature`, etc. If not, you can review the :doc:`/reference/glossary` docs. Steps ***** @@ -36,10 +36,10 @@ Steps #. Place your filter in an architecture subdomain - As specified in the Architectural Decisions Record (ADR) filter naming and versioning, the `:term: filter definition` needs an Open edX Architecture + As specified in the Architectural Decisions Record (ADR) filter naming and versioning, the :term:`filter definition` needs an Open edX Architecture Subdomain for: - - The `:term: type of the filter`: ``{Reverse DNS}.{Architecture Subdomain}.{Subject}.{Action}.{Major Version}`` + - The :term:`type of the filter`: ``{Reverse DNS}.{Architecture Subdomain}.{Subject}.{Action}.{Major Version}`` - The package name where the definition will live, eg. ``learning/``. For those reasons, after studying your new filter purpose, you must place it in one of the subdomains already in use, or introduce a new subdomain: @@ -59,20 +59,20 @@ Steps Defining the filter's behavior includes: - - Defining the `:term: filter type` for identification - - Defining the `:term: filter signature` + - Defining the :term:`filter type` for identification + - Defining the :term:`filter signature` - Defining the filter's behavior for stopping the process in which it is being used - The `:term: filter type` is the name that will be used to identify the filter's and it'd help others identifying its purpose. For example, if you're creating a filter that will be used during the student registration process in the LMS, - according to the documentation, the `:term: filter type` is defined as follows: + The :term:`filter type` is the name that will be used to identify the filter's and it'd help others identifying its purpose. For example, if you're creating a filter that will be used during the student registration process in the LMS, + according to the documentation, the :term:`filter type` is defined as follows: ``{Reverse DNS}.{Architecture Subdomain}.student.registration.requested.{Major Version}`` Where ``student`` is the subject and ``registration.requested`` the action being performed. The major version is the version of the filter, which will be incremented when a change is made to the filter that is not backwards compatible, as explained in the ADR. - Now that you have the `:term: filter type`, you'll need to define the `:term: filter signature` and overall behavior. The `:term: filter signature`, which is the set of parameters that the filter will manipulate, depends on where the filter is located. For example, - if you're creating a filter that will be used during the student registration process in the LMS, the `:term: filter signature` will be the set of parameters available for that time for the user. In this case, the `:term: filter signature` will be the set of parameters that the registration form sends to the LMS. + Now that you have the :term:`filter type`, you'll need to define the :term:`filter signature` and overall behavior. The :term:`filter signature`, which is the set of parameters that the filter will manipulate, depends on where the filter is located. For example, + if you're creating a filter that will be used during the student registration process in the LMS, the :term:`filter signature` will be the set of parameters available for that time for the user. In this case, the :term:`filter signature` will be the set of parameters that the registration form sends to the LMS. You can ask yourself the following questions to help you figure out your filter's parameters: @@ -80,7 +80,7 @@ Steps - What parameters will the filter need to to that? (e.g. the email address) - Where in the registration process will the filter be used? (e.g. after the student submits the registration form but before anything else) - With that information, you can define the `:term: filter signature`: + With that information, you can define the :term:`filter signature`: - Arguments: ``email``. Since we want this filter to be broadly used, we'll add as much relevant information as possible for the user at that point. As we mentioned above, we can send more information stored in the registration form like ``name`` or ``username``. - Returns: since filters take in a set of parameters and return a set of parameters, we'll return the same set of parameters that we received. @@ -128,8 +128,8 @@ Steps Some things to note: - - The filter's type is defined in the ``filter_type`` class attribute. In this case, the `:term: filter type` is ``org.openedx.learning.student.registration.requested.v1``. - - The `:term: filter signature` is defined in the ``run_filter`` method. In this case, the signature is the ``form_data`` parameter. + - The filter's type is defined in the ``filter_type`` class attribute. In this case, the :term:`filter type` is ``org.openedx.learning.student.registration.requested.v1``. + - The :term:`filter signature` is defined in the ``run_filter`` method. In this case, the signature is the ``form_data`` parameter. - The ``run_filter`` is a class method that returns the same set of parameters that it receives. - The ``run_filter`` class method calls the ``run_pipeline`` method, which is the method that executes the filter's logic. This method is defined in the ``OpenEdxPublicFilter`` class, which is the base class for all the filters in the library. This method returns a dictionary with the following structure: @@ -150,7 +150,7 @@ Steps "form_data": form_data, } - Where ``form_data`` is the same set of parameters that the filter receives, which is the accumulated output for the `:term: filter pipeline`. That is how ``run_filter`` should always look like. + Where ``form_data`` is the same set of parameters that the filter receives, which is the accumulated output for the :term:`filter pipeline`. That is how ``run_filter`` should always look like. - The filter's behavior for stopping the process is defined in the ``PreventRegistration`` exception which inherits from the ``OpenEdxFilterException`` base exception. In this case, the exception is raised when the filter stops the registration process. This is done in the service where the filter is being used, which in this case is the LMS. - The class name is the filter's type ``{Subject}.{Action}`` part in a camel case format. In this case, the filter's name is ``StudentRegistrationRequested``. @@ -198,7 +198,7 @@ Steps self.assertDictContainsSubset(attributes, exception.__dict__) .. note:: - Basically, we're testing the `:term: filter signature` and the filter's behavior for stopping the process. The first test is testing the `:term: filter signature`, which is the set of parameters that the filter receives and returns. The second test is testing the filter's behavior for stopping the process, which is the exception that is raised when the filter stops the process. + Basically, we're testing the :term:`filter signature` and the filter's behavior for stopping the process. The first test is testing the :term:`filter signature`, which is the set of parameters that the filter receives and returns. The second test is testing the filter's behavior for stopping the process, which is the exception that is raised when the filter stops the process. .. .. seealso:: diff --git a/docs/how-tos/using-filters.rst b/docs/how-tos/using-filters.rst index 51886d78..92adba67 100644 --- a/docs/how-tos/using-filters.rst +++ b/docs/how-tos/using-filters.rst @@ -2,7 +2,7 @@ How to use Open edX Filters --------------------------- Using openedx-filters in your code is very straight forward. We can consider the -various use cases: implementing `:term: Pipeline Steps`, attaching/hooking pipelines to filter, +various use cases: implementing :term:`Pipeline Steps`, attaching/hooking pipelines to filter, and triggering a filter. We'll also cover how to test the filters you create in your service. @@ -11,8 +11,8 @@ Implement pipeline steps Let's say you want to consult student's information with a third party service before generating the students certificate. This is a common use case for filters, -where the functions part of the `:term: filter pipeline` will perform the consulting tasks and -decide the execution flow for the application. These functions are the `:term: pipeline steps`, +where the functions part of the :term:`filter pipeline` will perform the consulting tasks and +decide the execution flow for the application. These functions are the :term:`pipeline steps`, and can be implemented in an installable Python library: .. code-block:: python @@ -83,8 +83,8 @@ There's two key components to the implementation: Attach/hook pipeline to filter ****************************** -After implementing the `:term: pipeline steps`, we have to tell the certificate creation -filter to execute our `:term: pipeline`. +After implementing the :term:`pipeline steps`, we have to tell the certificate creation +filter to execute our :term:`pipeline`. .. code-block:: python diff --git a/docs/reference/django-plugins-and-filters.rst b/docs/reference/django-plugins-and-filters.rst index 5b241b0a..b02a9b6c 100644 --- a/docs/reference/django-plugins-and-filters.rst +++ b/docs/reference/django-plugins-and-filters.rst @@ -37,5 +37,5 @@ file. The dictionary has the following structure: Create pipeline steps ********************* -In your own plugin, you can create your own `:term: pipeline steps` by inheriting from ``PipelineStep`` and implementing the -``run_filter`` method. You can find examples of `:term: pipeline steps` in the ``openedx-filters-samples`` repository. See :doc:`/quickstarts/index` for more details. +In your own plugin, you can create your own :term:`pipeline steps` by inheriting from ``PipelineStep`` and implementing the +``run_filter`` method. You can find examples of :term:`pipeline steps` in the ``openedx-filters-samples`` repository. See :doc:`/quickstarts/index` for more details. From ef3a9a007f2248ae1fa6c7771bf7ab420ad2c72e Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 4 Nov 2024 14:32:32 +0100 Subject: [PATCH 07/11] refactor: improve glossary description with usage for each component --- docs/reference/glossary.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index e9146080..c8758f2a 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -1,9 +1,7 @@ Open edX Filters Glossary ########################## -A filter has multiple components, such as pipeline, pipeline steps, filter definition, filter signature, filter type, filter exceptions, filter configuration, etc. This glossary provides definitions for these components. - -Here's a brief explanation of each component: +A filter has multiple components that are used to define, execute and handle filters, such as pipeline, pipeline steps, filter definition, filter signature, filter type, filter exceptions, filter configuration, etc. This glossary provides definitions for some of the terms to ease the adoption of the Open edX Filters library. .. glossary:: From 420cad04414a20ea91307cdb915ad4c8cdde3afb Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 4 Nov 2024 14:38:12 +0100 Subject: [PATCH 08/11] refactor: drop note about calling the mechanism filter to avoid confusion --- docs/reference/glossary.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index c8758f2a..56b75fc1 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -25,6 +25,3 @@ A filter has multiple components that are used to define, execute and handle fil Filter Configuration Filter configuration is a dictionary that defines the pipeline settings for a filter. Each filter type has its own configuration, which includes settings like whether errors should fail silently or propagate, and the sequence of pipeline steps. Configurations specify the filter type, error-handling preferences, and a list of module paths for each pipeline step to be executed. - -.. note:: - In practice, "filter" is used to refer to the whole mechanism, including the pipeline steps, filter definition and so on. From 9be58a186697789c0a016dff4509a899eee6f0c0 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 4 Nov 2024 23:40:31 +0100 Subject: [PATCH 09/11] refactor: address PR reviews --- docs/reference/glossary.rst | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index 56b75fc1..268c6b90 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -6,22 +6,30 @@ A filter has multiple components that are used to define, execute and handle fil .. glossary:: Pipeline - A pipeline is a list of functions executed in a specific order; these functions are known as pipeline steps. Each function in the pipeline takes the output of the previous function as its input, with the final function's output serving as the overall output of the filter. These pipelines are configured in the filter configuration and are executed in sequence. + A pipeline is a list of functions executed in a specific order; these functions are known as pipeline steps. Each function in the pipeline takes the output of the previous function as its input, with the final function's output serving as the overall output of the filter. The pipeline behavior was inspired by the `Python Social Auth accumulative pipeline`_, which is described in detail in the :doc:`/decisions/0003-hooks-filter-tooling-pipeline` ADR. These pipelines are configured in the filter configuration and are executed in sequence. Pipeline Step - A pipeline step is a function within a pipeline that receives, processes, and returns data. Each step may perform operations like transforming, validating, filtering, or enriching data. Pipeline steps are implemented as classes that inherit from a base step class and define specific logic within their ``run_filter`` method, which conforms to the filter's signature. + A pipeline step is a function within a pipeline that receives, processes, and returns data. Each step may perform operations like transforming, validating, filtering, or enriching data. Pipeline steps are implemented as classes that inherit from the base class `PipelineStep`_ and define specific logic within their `run_filter`_ method, which is executed by the pipeline tooling when the filter is triggered. Filter Definition - A filter definition is the class that defines the ``run_filter`` method for a filter, specifying the input and output behavior. This class, which inherits from a standard filter base, executes the configured pipeline steps, enabling custom processing within the defined filter. + A filter definition is a class that inherits from `OpenEdxPublicFilter`_ that implements the ``run_filter`` which defines the input and output behavior of the filter. This class executes the configured pipeline steps by calling the method `run_pipeline`_, passing down the input arguments, handling exceptions and returning the final output of the filter. Since the ``run_filter`` method is the entry point for the filter, the pipeline steps must have the same signature as the filter definition. Filter Signature - The filter signature consists of the specific parameters required by a filter's ``run_filter`` method. It defines the expected input and output structure for the filter, detailing the data the filter will process. + The filter signature consists of the specific parameters required by a filter's ``run_filter`` method. It defines the expected input and output structure for the filter, specifying the data the filter will process. The filter signature is used to ensure that all pipeline steps have the same input and output structure, enabling interchangeability between steps. Filter Type - The filter type is a unique identifier for the filter, following a standardized format (e.g., reverse DNS style). This type is used as an index for configuring the filter pipeline and specifies which configuration settings apply to a given filter. + The filter type is a unique identifier for the filter, following a standardized format following the :doc:`/decisions/0004-filters-naming-and-versioning`. This type is used as an index for configuring the filter pipeline and specifies which configuration settings apply to a given filter. Filter Exceptions - Filters can raise exceptions to control the flow of the pipeline. If a filter raises an exception, the pipeline halts, and the exception becomes the pipeline's output. Exceptions are typically raised when certain conditions specified in the filter's logic are met, signaling an event or state change. + Filters can raise exceptions to control the flow of the pipeline. If a filter raises an exception, the pipeline halts, and the exception becomes the pipeline's output. Exceptions are typically raised when certain conditions specified in the filter's logic are met, allowing the filter to control the application flow. Filter Configuration Filter configuration is a dictionary that defines the pipeline settings for a filter. Each filter type has its own configuration, which includes settings like whether errors should fail silently or propagate, and the sequence of pipeline steps. Configurations specify the filter type, error-handling preferences, and a list of module paths for each pipeline step to be executed. + +This glossary provides a high-level overview of the key concepts and components of the Open edX Filters library. Understanding these terms will help you implement filters in your application and leverage the filter tooling to control the flow of your application based on specific conditions. For a better illustration of these concepts, refer to the :doc:`/how-tos/using-filters` guide. + +.. _Python Social Auth accumulative pipeline: https://python-social-auth.readthedocs.io/en/latest/pipeline.html +.. _PipelineStep: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/filters.py#L10 +.. _run_filter: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/filters.py#L60 +.. _OpenEdxPublicFilter: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/tooling.py#L14 +.. _run_pipeline: https://github.com/openedx/openedx-filters/blob/main/openedx_filters/tooling.py#L164 From 0b70aa80f2123cc7ba5c40448b553bf84e10471e Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 7 Nov 2024 12:12:12 +0100 Subject: [PATCH 10/11] refactor: address PR reviews --- docs/how-tos/create-new-filter.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-tos/create-new-filter.rst b/docs/how-tos/create-new-filter.rst index 41ce5cea..0ab554c5 100644 --- a/docs/how-tos/create-new-filter.rst +++ b/docs/how-tos/create-new-filter.rst @@ -198,7 +198,7 @@ Steps self.assertDictContainsSubset(attributes, exception.__dict__) .. note:: - Basically, we're testing the :term:`filter signature` and the filter's behavior for stopping the process. The first test is testing the :term:`filter signature`, which is the set of parameters that the filter receives and returns. The second test is testing the filter's behavior for stopping the process, which is the exception that is raised when the filter stops the process. + In this example, we're testing the :term:`filter signature` and the filter's behavior for stopping the process. The first test is testing the :term:`filter signature`, specifically that the behavior works as expected when passed mock form data. The second test is testing the filter's behavior for stopping the process, which is the exception that is raised when the filter stops the process. .. .. seealso:: From 48b814f9c4b756f1734ad9f78585a3b09f7e99f5 Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Mon, 18 Nov 2024 06:16:37 -0400 Subject: [PATCH 11/11] docs: apply suggestions from code review Co-authored-by: Felipe Montoya --- docs/reference/glossary.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index 268c6b90..efbd9229 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -12,7 +12,7 @@ A filter has multiple components that are used to define, execute and handle fil A pipeline step is a function within a pipeline that receives, processes, and returns data. Each step may perform operations like transforming, validating, filtering, or enriching data. Pipeline steps are implemented as classes that inherit from the base class `PipelineStep`_ and define specific logic within their `run_filter`_ method, which is executed by the pipeline tooling when the filter is triggered. Filter Definition - A filter definition is a class that inherits from `OpenEdxPublicFilter`_ that implements the ``run_filter`` which defines the input and output behavior of the filter. This class executes the configured pipeline steps by calling the method `run_pipeline`_, passing down the input arguments, handling exceptions and returning the final output of the filter. Since the ``run_filter`` method is the entry point for the filter, the pipeline steps must have the same signature as the filter definition. + A filter definition is a class that inherits from `OpenEdxPublicFilter`_ that implements the ``run_filter`` method which defines the input and output behavior of the filter. This class executes the configured pipeline steps by calling the method `run_pipeline`_, passing down the input arguments, handling exceptions and returning the final output of the filter. Since the ``run_filter`` method is the entry point for the filter, the pipeline steps must have the same signature as the filter definition. Filter Signature The filter signature consists of the specific parameters required by a filter's ``run_filter`` method. It defines the expected input and output structure for the filter, specifying the data the filter will process. The filter signature is used to ensure that all pipeline steps have the same input and output structure, enabling interchangeability between steps.