Add new example for testing cupis on Decodable (#38)

* add example for testing custom flink jobs

* add new example entry to main repo README

* fix link to blog article

Author: hpgrahsl

.gitignore

@@ -25,7 +25,10 @@ config.*
 *.properties
 
 output
-*test*
+# unclear why this rule has been added initially, it would ignore
+# e.g. any test classes by default or complete examples which contain
+# the term 'test' in their name...
+#*test*
 .env
 api-key*
 *.properties

README.md

@@ -35,13 +35,14 @@ _Learn more [here](https://decodable.co), and [sign up for a free trial](https:/
 
 ### Data Pipelines
 
-| Example                                                 | Description                                                         |
-|---------------------------------------------------------|---------------------------------------------------------------------|
-| [Opinionated Data Pipelines](opinionated-pipelines)     | Building data pipelines with schema on write streams.               |
-| [Postman](postman)                                      | Building data pipelines with Postman.                               |
-| [Postgres to Snowflake](postgres-to-snowflake-with-cdc) | Getting data from Postgres to Snowflake using Decodable.            |
-| [Flink CDC](flink-cdc)                                  | Trying out Flink CDC and comparing it to Flink SQL.                 |
-| [Flink SQL and Custom Pipelines](sql-cupi-hybrid)       | Bridging Flink SQL and Custom Java Pipelines with the Decodable SDK |
+| Example                                                 | Description                                                                            |
+|---------------------------------------------------------|----------------------------------------------------------------------------------------|
+| [Opinionated Data Pipelines](opinionated-pipelines)     | Building data pipelines with schema on write streams.                                  |
+| [Postman](postman)                                      | Building data pipelines with Postman.                                                  |
+| [Postgres to Snowflake](postgres-to-snowflake-with-cdc) | Getting data from Postgres to Snowflake using Decodable.                               |
+| [Flink CDC](flink-cdc)                                  | Trying out Flink CDC and comparing it to Flink SQL.                                    |
+| [Flink SQL and Custom Pipelines](sql-cupi-hybrid)       | Bridging Flink SQL and Custom Java Pipelines with the Decodable SDK.                   |
+| [Testing Custom Pipelines on Decodable](testing-cupi)   | How to write more modular Flink jobs with pluggable components to improve testability. |
 
 
 ### PyFlink

testing-cupi/.gitignore

@@ -0,0 +1 @@
+!maven-wrapper.properties

testing-cupi/.mvn/wrapper/maven-wrapper.jar

@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.8.6/apache-maven-3.8.6-bin.zip
+wrapperUrl=https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.2.0/maven-wrapper-3.2.0.jar

testing-cupi/.mvn/wrapper/maven-wrapper.properties

@@ -0,0 +1,20 @@
+# Testing Custom Flink Jobs on Decodable
+
+Testing Flink jobs can be challenging for a number of reasons, one of them being the fact that there are typically external data systems or services involved.
+
+![high-level-example-overview](./docs/images/01_flink-job-high-level-showing-external-service-dependencies.webp)
+
+This example is the result of refactoring an [existing Flink job](https://github.com/decodableco/examples/blob/main/sql-cupi-hybrid/cupi/src/main/java/co/decodable/examples/FruitEnricherJob.java) to improve its testability. This alternative implementation is more modular by defining job-specific interfaces for the actual processing logic and by making the source and sink components which talk to external data systems pluggable. Doing so allows to not only unit test certain job components in isolation, but instead test the full processing logic of said jobs without the stringent necessity to always have access to any external services our Flink jobs almost certainly depend on.
+
+The accompanying blog post can be found [here](https://www.decodable.co/blog/testing-custom-flink-jobs-on-decodable). Find its outline below.
+
+## Outline
+
+* Quick Recap of the Flink Job
+* Assessing the Testability of the Existing Job
+* Refactoring for Better Job Testability
+  - Adapt Job Processing Logic
+  - Test New Job Processing Logic
+  - Adapt Job Setup
+  - Test New Job Setup
+  - Integration Testing the Job