Zingg Compatibility with Spark/Databricks

[GeoPapang]

Hello team and thanks a lot for what you have produced!

I am just looking to test it in the latest spark versions or databricks runtimes. Do you have any code examples or guides? It seems that in the latest Databricks runtimes you cannot import a .jar file and this causes issues.. What is the latest runtime examples you have with the latest zingg version?

Thank you very much:)

[sonalgoyal]

Hi @GeoPapang, Zingg 0.5.0 is compatible with Spark 3.5.0 and Scala 2.12. So please choose the corresponding DBR version.

Regarding importing the jar, are you on a UC cluster? You should be able to add the jar to the UC volume and deploy it as a cluster library. HTH

[GeoPapang]

Thank you Sonal! I managed to install it through UC and it works! It also seems to learn after each training..which is a good sign.. However, when I try to predict Matching Records:
outputDF = spark.read.csv("dbfs:/FileStore/zingg/zingg05Trial29Sep_2_cleanPhone", schema_out)
colNames = ["z_minScore", "z_maxScore", "z_cluster", 'CompanyID', 'CompanyName', 'Email', 'Phone', 'Address1', 'LastName', 'FirstName', 'Address2', 'Country']
outputDF.toDF(*colNames).show(100)

I get: [UNABLE_TO_INFER_SCHEMA] Unable to infer schema for CSV. It must be specified manually. SQLSTATE: 42KD9
File , line 2
----> 2 outputDF = spark.read.csv("dbfs:/FileStore/zingg/zingg05Trial29Sep_2_cleanPhone")

So i do:
schema_out = "z_minScore float, z_maxScore float, z_cluster int, CompanyName string, CompanyID string, FirstName string, LastName string, Email string, Phone string, Country string, Address1 string, Address2 string"
outputDF = spark.read.csv("dbfs:/FileStore/zingg/zingg05Trial29Sep_2_cleanPhone")

but then i get an empty DF:
+----------+----------+---------+---------+-----------+-----+-----+--------+--------+---------+--------+-------+
|z_minScore|z_maxScore|z_cluster|CompanyID|CompanyName|Email|Phone|Address1|LastName|FirstName|Address2|Country|
+----------+----------+---------+---------+-----------+-----+-----+--------+--------+---------+--------+-------+
+----------+----------+---------+---------+-----------+-----+-----+--------+--------+---------+--------+-------+