diff --git a/docs/modules/nifi/pages/usage_guide/custom_processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-nars.adoc similarity index 99% rename from docs/modules/nifi/pages/usage_guide/custom_processors.adoc rename to docs/modules/nifi/pages/usage_guide/custom-components/custom-nars.adoc index d84bf8cc..78d5da2a 100644 --- a/docs/modules/nifi/pages/usage_guide/custom_processors.adoc +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-nars.adoc @@ -1,4 +1,5 @@ -= Loading custom components += Custom `.nar` files + :description: Load custom NiFi components by using custom Docker images or mounting external volumes with nar files for enhanced functionality. :nifi-docs-custom-components: https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#introduction @@ -36,6 +37,7 @@ spec: Also read the xref:guides:custom-images.adoc[Using customized product images] guide for additional information. == Using the official image + If you don't want to create a custom image or don't have access to an image registry, you can use the extra volume mount functionality to mount a volume containing your custom components and configure NiFi to read these from the mounted volumes. For this to work you'll need to prepare a PersistentVolumeClaim (PVC) containing your components. diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc new file mode 100644 index 00000000..99cffcc6 --- /dev/null +++ b/docs/modules/nifi/pages/usage_guide/custom-components/custom-python-processors.adoc @@ -0,0 +1,165 @@ += Custom Python processors + +In NiFi 2.0, support for custom processors written in Python was added. +The Stackable images already contain the required tooling, such as - obviously - a supported Python version. + +== General configuration + +[source,yaml] +---- +spec: + nodes: + configOverrides: + nifi.properties: + # The command used to launch Python. + # This property must be set to enable Python-based processors. + nifi.python.command: python3 + # The directory that NiFi should look in to find custom Python-based + # Processors. + nifi.python.extensions.source.directory.custom: /nifi-python-extensions + # The directory that contains the Python framework for communicating + # between the Python and Java processes. + nifi.python.framework.source.directory: /stackable/nifi/python/framework/ + # The working directory where NiFi should store artifacts + # This property defaults to ./work/python but if you want to mount an + # emptyDir for the working directory then another directory has to be + # set to avoid ownership conflicts with ./work/nar. + nifi.python.working.directory: /nifi-python-working-directory +---- + +== Getting Python scripts into NiFi + +TIP: NiFi should hot-reload the Python scripts. You might need to refresh your browser window to see the new processor. + +[#configmap] +=== 1. Mount as ConfigMap + +The easiest way is defining a ConfigMap and mounting it as follows. +This way, the Python processors are stored and versioned alongside your NifiCluster itself. + +// Technically it's yaml, but the most content is Python +[source,python] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: nifi-python-extensions +data: + CreateFlowFileProcessor.py: | + from nifiapi.flowfilesource import FlowFileSource, FlowFileSourceResult + + class CreateFlowFile(FlowFileSource): + class Java: + implements = ['org.apache.nifi.python.processor.FlowFileSource'] + + class ProcessorDetails: + version = '0.0.1-SNAPSHOT' + description = '''A Python processor that creates FlowFiles.''' + + def __init__(self, **kwargs): + pass + + def create(self, context): + return FlowFileSourceResult( + relationship = 'success', + attributes = {'greeting': 'hello'}, + contents = 'Hello World!' + ) +---- + +The Python script is taken from https://nifi.apache.org/nifi-docs/python-developer-guide.html#flowfile-source[the offical NiFi Python developer guide]. + +You can add multiple Python scripts in the ConfigMap. +Afterwards we need to mount the Python scripts into `/nifi-python-extensions`: + +[source,yaml] +---- +spec: + nodes: + podOverrides: + spec: + containers: + - name: nifi + volumeMounts: + - name: nifi-python-extensions + mountPath: /nifi-python-extensions + - name: nifi-python-working-directory + mountPath: /nifi-python-working-directory + volumes: + - name: nifi-python-extensions + configMap: + name: nifi-python-extensions + - name: nifi-python-working-directory + emptyDir: {} +---- + +[#git-sync] +=== 2. Use git-sync + +As an alternative you can use `git-sync` to keep your Python processors up to date. +You need to add a sidecar using podOverrides that syncs into a shared volume between the `nifi` and `git-sync` container. + +The following snippet can serve as a starting point (the Git repo has the folder `processors` with the Python scripts inside). + +[source,yaml] +---- +spec: + nodes: + podOverrides: + spec: + containers: + - name: nifi + volumeMounts: + - name: nifi-python-extensions + mountPath: /nifi-python-extensions + - name: nifi-python-working-directory + mountPath: /nifi-python-working-directory + - name: git-sync + image: registry.k8s.io/git-sync/git-sync:v4.2.3 + args: + - --repo=https://github.com/stackabletech/nifi-talk + - --root=/nifi-python-extensions + - --period=10s + volumeMounts: + - name: nifi-python-extensions + mountPath: /nifi-python-extensions + volumes: + - name: nifi-python-extensions + emptyDir: {} + - name: nifi-python-working-directory + emptyDir: {} +---- + +Afterwards you need to update your source directory (the one you added previously) accordingly, to point into the Git subfolder you have. + +[source,yaml] +---- +spec: + nodes: + configOverrides: + nifi.properties: + # Replace the property from the previous step + # Format is /nifi-python-extensions/// + nifi.python.extensions.source.directory.custom: > + /nifi-python-extensions/nifi-talk/processors/ +---- + +=== 3. Use PersistentVolume + +You can also mount a PVC below `/nifi-python-extensions` using podOverrides and shell into the NiFi Pod to make changes. +However, the <> or <> approach is recommended. + +== Check processors have been loaded + +NiFi logs every Python processor it found. +You can use that to check if the processors have been loaded. + +[source,console] +---- +$ kubectl logs nifi-2-0-0-node-default-0 -c nifi \ + | grep 'Discovered.*Python Processor' +… INFO [main] … Discovered Python Processor PythonZgrepProcessor +… INFO [main] … Discovered Python Processor TransformOpenskyStates +… INFO [main] … Discovered Python Processor UpdateAttributeFileLookup +… INFO [main] … Discovered or updated 3 Python Processors in 64 millis +---- diff --git a/docs/modules/nifi/pages/usage_guide/custom-components/index.adoc b/docs/modules/nifi/pages/usage_guide/custom-components/index.adoc new file mode 100644 index 00000000..710b8812 --- /dev/null +++ b/docs/modules/nifi/pages/usage_guide/custom-components/index.adoc @@ -0,0 +1,9 @@ += Loading custom components +:description: Load custom NiFi components for enhanced functionality. + +You can develop or use custom components for Apache NiFi, typically custom processors, to extend its functionality. + +There are currently two types of custom components: + +1. xref:nifi:usage_guide/custom-components/custom-nars.adoc[] +2. Starting with NiFi 2.0 you can also use xref:nifi:usage_guide/custom-components/custom-python-processors.adoc[] diff --git a/docs/modules/nifi/partials/nav.adoc b/docs/modules/nifi/partials/nav.adoc index d3407296..cf5d74db 100644 --- a/docs/modules/nifi/partials/nav.adoc +++ b/docs/modules/nifi/partials/nav.adoc @@ -5,7 +5,6 @@ ** xref:nifi:usage_guide/listenerclass.adoc[] ** xref:nifi:usage_guide/zookeeper-connection.adoc[] ** xref:nifi:usage_guide/extra-volumes.adoc[] -** xref:nifi:usage_guide/custom_processors.adoc[] ** xref:nifi:usage_guide/external_ports.adoc[] ** xref:nifi:usage_guide/security.adoc[] ** xref:nifi:usage_guide/resource-configuration.adoc[] @@ -14,6 +13,9 @@ ** xref:nifi:usage_guide/updating.adoc[] ** xref:nifi:usage_guide/overrides.adoc[] ** xref:nifi:usage_guide/writing-to-iceberg-tables.adoc[] +** xref:nifi:usage_guide/custom-components/index.adoc[] +*** xref:nifi:usage_guide/custom-components/custom-nars.adoc[] +*** xref:nifi:usage_guide/custom-components/custom-python-processors.adoc[] ** xref:nifi:usage_guide/operations/index.adoc[] *** xref:nifi:usage_guide/operations/cluster-operations.adoc[] *** xref:nifi:usage_guide/operations/pod-placement.adoc[]