From 647eca01bf8cd7e1310dc92cfde6505e498e05de Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Fri, 14 Feb 2025 15:52:40 +0100 Subject: [PATCH 01/10] docs: Add docs on how use custom Python processors --- .../custom-nars.adoc} | 4 +- .../custom-python-processors.adoc | 150 ++++++++++++++++++ .../usage_guide/custom-components/index.adoc | 9 ++ docs/modules/nifi/partials/nav.adoc | 4 +- 4 files changed, 165 insertions(+), 2 deletions(-) rename docs/modules/nifi/pages/usage_guide/{custom_processors.adoc => custom-components/custom-nars.adoc} (99%) create mode 100644 docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc create mode 100644 docs/modules/nifi/pages/usage_guide/custom-components/index.adoc diff --git a/docs/modules/nifi/pages/usage_guide/custom_processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-nars.adoc similarity index 99% rename from docs/modules/nifi/pages/usage_guide/custom_processors.adoc rename to docs/modules/nifi/pages/usage_guide/custom-components/custom-nars.adoc index d84bf8cc..78d5da2a 100644 --- a/docs/modules/nifi/pages/usage_guide/custom_processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-nars.adoc @@ -1,4 +1,5 @@ -= Loading custom components += Custom `.nar` files + :description: Load custom NiFi components by using custom Docker images or mounting external volumes with nar files for enhanced functionality. :nifi-docs-custom-components: https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#introduction @@ -36,6 +37,7 @@ spec: Also read the xref:guides:custom-images.adoc[Using customized product images] guide for additional information. == Using the official image + If you don't want to create a custom image or don't have access to an image registry, you can use the extra volume mount functionality to mount a volume containing your custom components and configure NiFi to read these from the mounted volumes. For this to work you'll need to prepare a PersistentVolumeClaim (PVC) containing your components. diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc new file mode 100644 index 00000000..242b34a2 --- /dev/null +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -0,0 +1,150 @@ += Custom Python processors + +In NiFi 2.0 support for custom processors written in Python have been added. +The Stackable images already contain the needed tools, such as - obviously - a supported Python version. + +For inspiration, here are some ways of adding these Python scripts into your NiFi stacklet: + +== General configuration + +[source,yaml] +---- +spec: + nodes: + configOverrides: + nifi.properties: + nifi.python.command: python3 + # This property needs to be specified (otherwise a NullPointerException occurs) + nifi.python.working.directory: /nifi-python-working-directory + # This is needed to detect the Controller.py location (internally used by NiFi) + nifi.python.framework.source.directory: /stackable/nifi/python/framework/ + # This is the folder where the Python scripts are sourced from + # We need to get the Python files in here + nifi.python.extensions.source.directory.custom: /nifi-python-extensions +---- + +== Getting Python scripts into NiFi + +TIP: NiFi should hot-reload the Python scripts. You might need to refresh your browser window to see the new processor. + +[#configmap] +=== 1. Mount as ConfigMap + +The easiest way is defining a ConfigMap as follows and mount that. +This way the Python processors are stored and versioned alongside your NiFiCluster itself. + +[source,yaml] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: nifi-python-extensions +data: + HelloWorldProcessor.py: | + from nifiapi.flowfiletransform import FlowFileTransform, FlowFileTransformResult + + class WriteHelloWorld(FlowFileTransform): + class Java: + implements = ['org.apache.nifi.python.processor.FlowFileTransform'] + class ProcessorDetails: + version = '0.0.1-SNAPSHOT' + + def __init__(self, **kwargs): + pass + + def transform(self, context, flowfile): + return FlowFileTransformResult(relationship = "success", contents = "Hello World", attributes = {"greeting": "hello"}) +---- + +You can add multiple Python scripts in the ConfigMap. +Afterwards we need to mount the Python scripts into `/nifi-python-extensions`: + +[source,yaml] +---- +spec: + nodes: + podOverrides: + spec: + containers: + - name: nifi + volumeMounts: + - name: nifi-python-extensions + mountPath: /nifi-python-extensions + - name: nifi-python-working-directory + mountPath: /nifi-python-working-directory + volumes: + - name: nifi-python-extensions + configMap: + name: nifi-python-extensions + - name: nifi-python-working-directory + emptyDir: {} +---- + +[#git-sync] +=== 2. Use git-sync + +As an alternative you can use `git-sync` to keep your Python processors up to date. +You need to add a sidecar using podOverrides that syncs into a shared volume between the `nifi` and `git-sync` container. + +The following snippet can serve as a starting point (the Git repo has the folder `processors` with the Python scripts inside). + +[source,yaml] +---- +spec: + nodes: + podOverrides: + spec: + containers: + - name: nifi + volumeMounts: + - name: nifi-python-extensions + mountPath: /nifi-python-extensions + - name: nifi-python-working-directory + mountPath: /nifi-python-working-directory + - name: git-sync + image: registry.k8s.io/git-sync/git-sync:v4.2.3 + args: + - --repo=https://github.com/stackabletech/nifi-talk + - --root=/nifi-python-extensions + - --period=10s + volumeMounts: + - name: nifi-python-extensions + mountPath: /nifi-python-extensions + volumes: + - name: nifi-python-extensions + emptyDir: {} + - name: nifi-python-working-directory + emptyDir: {} +---- + +Afterwards you need to update your source directory (you added previously) accordingly to point into the Git subfolder you have. + +[source,yaml] +---- +spec: + nodes: + configOverrides: + nifi.properties: + # Replace the property from the previous step + # Format is /nifi-python-extensions/// + nifi.python.extensions.source.directory.custom: /nifi-python-extensions/nifi-talk/processors/ +---- + +=== 3. Use PersistentVolume + +You can also mount a PVC below `/nifi-python-extensions` using podOverrides and shell into the NiFi Pod to make changes. +However, the <> or <> approach is recommended. + +== Check processors have been loaded + +NiFi logs every Python processor it found. +You can use that to check if the processors have been loaded. + +[source,console] +---- +$ kubectl logs nifi-2-0-0-node-default-0 -c nifi | grep -P 'Discovered Python Processor|Discovered or updated [0-9]+ Python Processors' +2025-02-14 14:40:20,694 INFO [main] o.a.n.n.StandardExtensionDiscoveringManager Discovered Python Processor PythonZgrepProcessor +2025-02-14 14:40:20,697 INFO [main] o.a.n.n.StandardExtensionDiscoveringManager Discovered Python Processor TransformOpenskyStates +2025-02-14 14:40:20,700 INFO [main] o.a.n.n.StandardExtensionDiscoveringManager Discovered Python Processor UpdateAttributeFileLookup +2025-02-14 14:40:20,700 INFO [main] o.a.n.n.StandardExtensionDiscoveringManager Discovered or updated 3 Python Processors in 60 millis +---- diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/index.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/index.adoc new file mode 100644 index 00000000..710b8812 --- /dev/null +++ b/docs/modules/nifi/pages/usage_guide/custom-components/index.adoc @@ -0,0 +1,9 @@ += Loading custom components +:description: Load custom NiFi components for enhanced functionality. + +You can develop or use custom components for Apache NiFi, typically custom processors, to extend its functionality. + +There are currently two types of custom components: + +1. xref:nifi:usage_guide/custom-components/custom-nars.adoc[] +2. Starting with NiFi 2.0 you can also use xref:nifi:usage_guide/custom-components/custom-python-processors.adoc[] diff --git a/docs/modules/nifi/partials/nav.adoc b/docs/modules/nifi/partials/nav.adoc index d3407296..cf5d74db 100644 --- a/docs/modules/nifi/partials/nav.adoc +++ b/docs/modules/nifi/partials/nav.adoc @@ -5,7 +5,6 @@ ** xref:nifi:usage_guide/listenerclass.adoc[] ** xref:nifi:usage_guide/zookeeper-connection.adoc[] ** xref:nifi:usage_guide/extra-volumes.adoc[] -** xref:nifi:usage_guide/custom_processors.adoc[] ** xref:nifi:usage_guide/external_ports.adoc[] ** xref:nifi:usage_guide/security.adoc[] ** xref:nifi:usage_guide/resource-configuration.adoc[] @@ -14,6 +13,9 @@ ** xref:nifi:usage_guide/updating.adoc[] ** xref:nifi:usage_guide/overrides.adoc[] ** xref:nifi:usage_guide/writing-to-iceberg-tables.adoc[] +** xref:nifi:usage_guide/custom-components/index.adoc[] +*** xref:nifi:usage_guide/custom-components/custom-nars.adoc[] +*** xref:nifi:usage_guide/custom-components/custom-python-processors.adoc[] ** xref:nifi:usage_guide/operations/index.adoc[] *** xref:nifi:usage_guide/operations/cluster-operations.adoc[] *** xref:nifi:usage_guide/operations/pod-placement.adoc[] From 366fc3f440dbbe7e45997ab3479aeaf352aa4c0f Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Fri, 14 Feb 2025 15:54:54 +0100 Subject: [PATCH 02/10] remove sentence --- .../usage_guide/custom-components/custom-python-processors.adoc | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc index 242b34a2..102e9cf9 100644 --- a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -3,8 +3,6 @@ In NiFi 2.0 support for custom processors written in Python have been added. The Stackable images already contain the needed tools, such as - obviously - a supported Python version. -For inspiration, here are some ways of adding these Python scripts into your NiFi stacklet: - == General configuration [source,yaml] From 0b76ecd54b57c8682e385ef4896f992cb8347fc8 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Wed, 19 Feb 2025 11:04:25 +0100 Subject: [PATCH 03/10] linter --- .../usage_guide/custom-components/custom-python-processors.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc index 102e9cf9..42f16bcd 100644 --- a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -79,7 +79,7 @@ spec: ---- [#git-sync] -=== 2. Use git-sync +=== 2. Use git-sync As an alternative you can use `git-sync` to keep your Python processors up to date. You need to add a sidecar using podOverrides that syncs into a shared volume between the `nifi` and `git-sync` container. From a1dcd607dd5ff0269242141c7851de099d41793d Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Wed, 19 Feb 2025 13:05:48 +0100 Subject: [PATCH 04/10] Apply suggestions from code review Co-authored-by: Siegfried Weber Co-authored-by: Malte Sander --- .../custom-components/custom-python-processors.adoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc index 42f16bcd..1efbb031 100644 --- a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -1,7 +1,7 @@ = Custom Python processors -In NiFi 2.0 support for custom processors written in Python have been added. -The Stackable images already contain the needed tools, such as - obviously - a supported Python version. +In NiFi 2.0, support for custom processors written in Python was added. +The Stackable images already contain the required tooling, such as - obviously - a supported Python version. == General configuration @@ -28,8 +28,8 @@ TIP: NiFi should hot-reload the Python scripts. You might need to refresh your b [#configmap] === 1. Mount as ConfigMap -The easiest way is defining a ConfigMap as follows and mount that. -This way the Python processors are stored and versioned alongside your NiFiCluster itself. +The easiest way is defining a ConfigMap and mounting it as follows. +This way, the Python processors are stored and versioned alongside your NifiCluster itself. [source,yaml] ---- @@ -115,7 +115,7 @@ spec: emptyDir: {} ---- -Afterwards you need to update your source directory (you added previously) accordingly to point into the Git subfolder you have. +Afterwards you need to update your source directory (the one you added previously) accordingly, to point into the Git subfolder you have. [source,yaml] ---- From e7d289fd90ab4334a26678a2e7325a47540917e6 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Wed, 19 Feb 2025 13:18:22 +0100 Subject: [PATCH 05/10] Update docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc Co-authored-by: Siegfried Weber --- .../custom-components/custom-python-processors.adoc | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc index 1efbb031..08abe57c 100644 --- a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -140,9 +140,10 @@ You can use that to check if the processors have been loaded. [source,console] ---- -$ kubectl logs nifi-2-0-0-node-default-0 -c nifi | grep -P 'Discovered Python Processor|Discovered or updated [0-9]+ Python Processors' -2025-02-14 14:40:20,694 INFO [main] o.a.n.n.StandardExtensionDiscoveringManager Discovered Python Processor PythonZgrepProcessor -2025-02-14 14:40:20,697 INFO [main] o.a.n.n.StandardExtensionDiscoveringManager Discovered Python Processor TransformOpenskyStates -2025-02-14 14:40:20,700 INFO [main] o.a.n.n.StandardExtensionDiscoveringManager Discovered Python Processor UpdateAttributeFileLookup -2025-02-14 14:40:20,700 INFO [main] o.a.n.n.StandardExtensionDiscoveringManager Discovered or updated 3 Python Processors in 60 millis +$ kubectl logs nifi-2-0-0-node-default-0 -c nifi \ + | grep 'Discovered.*Python Processor' +… INFO [main] … Discovered Python Processor PythonZgrepProcessor +… INFO [main] … Discovered Python Processor TransformOpenskyStates +… INFO [main] … Discovered Python Processor UpdateAttributeFileLookup +… INFO [main] … Discovered or updated 3 Python Processors in 64 millis ---- From d92f4da71731ab010cf5b1fad11706d7508466d8 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Wed, 26 Feb 2025 10:53:41 +0100 Subject: [PATCH 06/10] Update YAML comments --- .../custom-python-processors.adoc | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc index 08abe57c..a9ff0737 100644 --- a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -11,14 +11,20 @@ spec: nodes: configOverrides: nifi.properties: + # The command used to launch Python. + # This property must be set to enable Python-based processors. nifi.python.command: python3 - # This property needs to be specified (otherwise a NullPointerException occurs) - nifi.python.working.directory: /nifi-python-working-directory - # This is needed to detect the Controller.py location (internally used by NiFi) - nifi.python.framework.source.directory: /stackable/nifi/python/framework/ - # This is the folder where the Python scripts are sourced from - # We need to get the Python files in here + # The directory that NiFi should look in to find custom Python-based + # Processors. nifi.python.extensions.source.directory.custom: /nifi-python-extensions + # The directory that contains the Python framework for communicating + # between the Python and Java processes. + nifi.python.framework.source.directory: /stackable/nifi/python/framework/ + # The working directory where NiFi should store artifacts + # This property defaults to ./work/python but if you want to mount an + # emptyDir for the working directory then another directory has to be + # set to avoid ownership conflicts with ./work/nar. + nifi.python.working.directory: /nifi-python-working-directory ---- == Getting Python scripts into NiFi From a404661cd418a7c76f93579afb7b47480d5d9a66 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Wed, 26 Feb 2025 10:59:21 +0100 Subject: [PATCH 07/10] Update docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc Co-authored-by: Siegfried Weber --- .../custom-components/custom-python-processors.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc index a9ff0737..5d4c42ae 100644 --- a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -131,7 +131,8 @@ spec: nifi.properties: # Replace the property from the previous step # Format is /nifi-python-extensions/// - nifi.python.extensions.source.directory.custom: /nifi-python-extensions/nifi-talk/processors/ + nifi.python.extensions.source.directory.custom: > + /nifi-python-extensions/nifi-talk/processors/ ---- === 3. Use PersistentVolume From 110a26fdf5d7982efc61e92532810f291ebcc390 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Wed, 26 Feb 2025 11:26:04 +0100 Subject: [PATCH 08/10] Switch from HelloWorldProcessor to CreateFlowFileProcessor --- .../custom-python-processors.adoc | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc index 5d4c42ae..e470d1fd 100644 --- a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -44,22 +44,30 @@ kind: ConfigMap metadata: name: nifi-python-extensions data: - HelloWorldProcessor.py: | - from nifiapi.flowfiletransform import FlowFileTransform, FlowFileTransformResult + CreateFlowFileProcessor.py: | + from nifiapi.flowfilesource import FlowFileSource, FlowFileSourceResult - class WriteHelloWorld(FlowFileTransform): + class CreateFlowFile(FlowFileSource): class Java: - implements = ['org.apache.nifi.python.processor.FlowFileTransform'] + implements = ['org.apache.nifi.python.processor.FlowFileSource'] + class ProcessorDetails: version = '0.0.1-SNAPSHOT' + description = '''A Python processor that creates FlowFiles.''' def __init__(self, **kwargs): pass - def transform(self, context, flowfile): - return FlowFileTransformResult(relationship = "success", contents = "Hello World", attributes = {"greeting": "hello"}) + def create(self, context): + return FlowFileSourceResult( + relationship = 'success', + attributes = {'greeting': 'hello'}, + contents = 'Hello World!' + ) ---- +The python script is taken from https://nifi.apache.org/nifi-docs/python-developer-guide.html#flowfile-source[the offical NiFi Python developer guide]. + You can add multiple Python scripts in the ConfigMap. Afterwards we need to mount the Python scripts into `/nifi-python-extensions`: From ebb0a5bd6771fbc23c9c4569433d5cf9a204b9e0 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Wed, 26 Feb 2025 11:28:26 +0100 Subject: [PATCH 09/10] Set content type to python --- .../custom-components/custom-python-processors.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc index e470d1fd..e75bf340 100644 --- a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -37,7 +37,8 @@ TIP: NiFi should hot-reload the Python scripts. You might need to refresh your b The easiest way is defining a ConfigMap and mounting it as follows. This way, the Python processors are stored and versioned alongside your NifiCluster itself. -[source,yaml] +// Technically it's yaml, but the most content is Python +[source,python] ---- apiVersion: v1 kind: ConfigMap From e8bf2b2a4598c84c2e57623db23cc4f32aba15e7 Mon Sep 17 00:00:00 2001 From: Sebastian Bernauer Date: Wed, 26 Feb 2025 12:57:27 +0100 Subject: [PATCH 10/10] Update docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc Co-authored-by: Siegfried Weber --- .../usage_guide/custom-components/custom-python-processors.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc index e75bf340..99cffcc6 100644 --- a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -67,7 +67,7 @@ data: ) ---- -The python script is taken from https://nifi.apache.org/nifi-docs/python-developer-guide.html#flowfile-source[the offical NiFi Python developer guide]. +The Python script is taken from https://nifi.apache.org/nifi-docs/python-developer-guide.html#flowfile-source[the offical NiFi Python developer guide]. You can add multiple Python scripts in the ConfigMap. Afterwards we need to mount the Python scripts into `/nifi-python-extensions`: