You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- simplicity of data science tools (Python, Jupyter notebooks, NumPy, Pandas)
9
+
- simplicity of data science tools (Python, Jupyter notebooks, NumPy, pandas)
10
10
- powerful Machine Learning / Deep Learning frameworks (TensorFlow, Keras)
11
11
- reliable, scalable event-based streaming technology for production deployments (Apache Kafka, Kafka Connect, KSQL).
12
12
@@ -29,7 +29,7 @@ We will do the following:
29
29
30
30
1)*Data Integration (Kafka Connect)*: Integrate a stream of data from CSV file or continuous data stream (in real world you can connect directly to an existing Kafka stream from the Jupyter notebook). As alternative, you can create new events manually in command line
31
31
2)*Data Preprocessing (KSQL)*: Preprocess the data, e.g. filter, anonymize, aggreate / concatenate
32
-
3)*Machine Learning specific preprocessing (NumPy, Pandas, Scikit-learn): Normalize, split train / test data
32
+
3)*Machine Learning specific preprocessing (NumPy, pandas, Scikit-learn): Normalize, split train / test data
33
33
4)*Model Training (TensorFlow + Keras)*
34
34
5)*Model Deployment (KSQL + Tensorflow)*
35
35
6)*Monitoring of Model Behaviour (KSQL)* like accuracy and performance
@@ -48,7 +48,7 @@ Check out [this step-by-step guide](https://github.com/kaiwaehner/python-jupyter
48
48
49
49
### Separation between Apache Kafka Administration and KSQL Statements
50
50
51
-
You can either do everything within Jupyter or separate administration commands (e.g. starting backends, creating Kafka Topics) to command line and only run KSQL commands in Jupyter.
51
+
You can either do everything within Jupyter or separate administration commands such as starting backends or creating Kafka Topics to command line and only run KSQL commands in Jupyter.
52
52
53
53
Here is an example where even the Kafka administration in done in Jupyter:
54
54
@@ -68,23 +68,23 @@ The goal is to lose as little information as possible. This way we can use an au
68
68
69
69
## Hands-On with Python, TensorFlow, Keras, Apache Kafka and KSQL
70
70
71
-
We use KSQL for preprocessing, Numpy, Pandas and scikit-learn for ML-specific tasks like array shapes or splitting training and test data, TensorFlow + Keras for model training, and Kafka Streams or KSQL for model deployment and monitoring.
71
+
We use KSQL for preprocessing, Numpy, pandas and scikit-learn for ML-specific tasks like array shapes or splitting training and test data, TensorFlow + Keras for model training, and Kafka Streams or KSQL for model deployment and monitoring.
72
72
73
73
Here is a TensorBoard screenshot of the trained Autoencoder:
74
74
75
75

76
76
77
-
## Keras model (.h5) vs. TensorFlow model (.pb) - Workarounds needed?
77
+
## TensorFlow Model Serialisation - Keras model (.h5) vs. TensorFlow model (.pb)
78
78
79
79
Different model serialisation mechanisms exist. Also product vendors and cloud providers add additional features (like for any other standard). Therefore, your TensorFlow model might not work everywhere out-of-the-box.
80
80
81
-
This will probably get better with release of [TensorFlow 2.0](Standardizing on Keras: Guidance on High-level APIs in TensorFlow 2.0) in 2019, but as of today (January 2019), you need to think about where you want to deploy your model before you train and export it.
81
+
This will probably get better with release of [TensorFlow 2.0](https://medium.com/tensorflow/standardizing-on-keras-guidance-on-high-level-apis-in-tensorflow-2-0-bad2b04c819a) in 2019, but as of today (January 2019), you need to think about where you want to deploy your model before you train and export it.
82
82
83
83
This demo uses plain Keras API. This is fine e.g. if you want to load the model via Java API from a Java application (see e.g. my [Kafka Streams + Keras + TensorFlow example](https://github.com/kaiwaehner/kafka-streams-machine-learning-examples/blob/master/src/test/java/com/github/megachucky/kafka/streams/machinelearning/test/Kafka_Streams_TensorFlow_Keras_Example_IntegrationTest.java) where I load the H5 model file).
84
84
85
85
If you want to deploy the model in a specific TensorFlow infrastructure like Google ML Engine (based on TensorFlow Serving model server), it is best to train the model with GCP's tools as described in this [Google ML Getting Started] (https://cloud.google.com/ml-engine/docs/tensorflow/getting-started-training-prediction) guide.
86
86
87
-
Otherwise you need to convert the H5 Keras file to a TensorFlow Proto Buffers file and fulfil some more tasks, e.g. described in this [blog post](https://medium.com/google-cloud/serve-keras-models-using-google-cloud-machine-learning-services-910912238bf6).
87
+
Otherwise, you need to convert the H5 Keras file to a TensorFlow Proto Buffers file and fulfil some more tasks, e.g. described in this [blog post](https://medium.com/google-cloud/serve-keras-models-using-google-cloud-machine-learning-services-910912238bf6).
88
88
89
89
The Python tool [Keras to TensorFlow](https://github.com/amir-abdi/keras_to_tensorflow) is a good and simple solution:
0 commit comments