java.lang.NoSuchMethodError on zingg 0.4.0

[Gabriel Medeiros (gmnmedeiros)]

I'm running Zingg 0.4.0 on a Glue Notebook and I'm having this error when matching data:

Py4JJavaError: An error occurred while calling o166.execute.
: java.lang.NoSuchMethodError: org.apache.spark.sql.Encoders.row(Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Encoder;
	at zingg.spark.core.util.SparkBlockingTreeUtil.getBlockHashes(SparkBlockingTreeUtil.java:47)
	at zingg.common.core.executor.Matcher.getBlocked(Matcher.java:45)
	at zingg.common.core.executor.Matcher.execute(Matcher.java:115)
	at zingg.common.client.Client.execute(Client.java:251)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:750)

Does anyone know what it means? Maybe something related to version?

[Sonal (sonalgoyal)]

can oyu please share your environment? Spark versions etc.

[Gabriel Medeiros (gmnmedeiros)]

Sure! Spark version: '3.3.0-amzn-1' and AWS Glue 4.0. Python 3.10.2. My sys.version command outputs 3.10.2 (main, Apr 25 2024, 14:08:54) [GCC 7.3.1 20180712 (Red Hat 7.3.1-17)]

[Gabriel Medeiros (gmnmedeiros)]

Here's the code I'm running:

from zingg.client import *
from zingg.pipes import *

df = spark.read.parquet("s3://mybucket/path/")

#build the arguments for zingg
args = Arguments()
#set field definitions
address_line_1 = FieldDefinition("address_line_1", "string", MatchType.FUZZY)
address_line_2 = FieldDefinition("address_line_2", "string", MatchType.FUZZY)
address_post_town = FieldDefinition("address_post_town", "string", MatchType.FUZZY)
address_county = FieldDefinition("address_county", "string", MatchType.FUZZY)
address_country = FieldDefinition("address_country", "string", MatchType.FUZZY)
incorporation_date = FieldDefinition("incorporation_date", "string", MatchType.FUZZY)
sic_code_text_1 = FieldDefinition("sic_code_text_1", "string", MatchType.FUZZY)
company_name = FieldDefinition("company_name", "string", MatchType.FUZZY)

# company_name.setStopWords('s3://<mybucketname>/models/1/stopWords/company_name.csv')
# address_line_1.setStopWords('s3://<mybucketname>/models/1/stopWords/address.csv')
# address_line_2.setStopWords('s3://<mybucketname>/models/1/stopWords/address.csv')

fieldDefs = [address_line_1, address_line_2, address_post_town, address_county, address_country, incorporation_date, sic_code_text_1, company_name]

args.setFieldDefinition(fieldDefs)
#set the modelid and the zingg dir
args.setModelId("2")
args.setZinggDir("s3://<mybucketname>/models/")
args.setNumPartitions(2000)

#reading dataset into inputPipe and settint it up in 'args'
#below line should not be required if you are reading from in memory dataset
#in that case, replace df with input df
schema = "company_name string, address_line_1 string, address_line_2 string, address_post_town string, address_county string, address_country string, incorporation_date string, sic_code_text_1 string, company_number string"
inputPipe = InMemoryPipe("uk_companies_extra_a_partition", df)
args.setData(inputPipe)
outputPipe = CsvPipe("resultUkCompanies", "s3://<mybucketname>/results/matchOuput3/")

args.setOutput(outputPipe)

options = ClientOptions([ClientOptions.PHASE,"match"])

#Zingg execution for the given phase
zingg = Zingg(args, options)
zingg.initAndExecute()

[Sonal (sonalgoyal)]

Can you please move to Spark 3.5.0 and let us know how it goes

[Gabriel Medeiros (gmnmedeiros)]

will try! if it's not possible to update spark in the glue notebook environment, would it be better for me to use zingg 0.3.4?

[Sonal (sonalgoyal)]

Sorry Zingg 0.4.0 needs Spark 3.5.0 - its been a while since Spark 3.5 was relased so it must be supported by Glue in all likelihood.

[Gabriel Medeiros (gmnmedeiros)]

Unfortunately it's not compatible yet...

https://repost.aws/questions/QUeUz49lE5QdWKumEDxfloDQ/spark-3-5-in-aws-glue

[Gabriel Medeiros (gmnmedeiros)]

I'll go ahead and use Zingg 0.3.4 on Glue. Thank you!