NoSuchMethodError in sparksql35-scalapb0_11 after update

[leoeareis]

Hi! I made some updates in my project from Spark 3.4.1 to Spark 3.5.0 and updated the scalapb dependency from sparksql34-scalapb0_11 to sparksql35-scalapb0_11. After this upgrade, I faced this error:

 java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke.<init>(Ljava/lang/Class;Lorg/apache/spark/sql/types/DataType;Ljava/lang/String;Lscala/collection/Seq;Lscala/collection/Seq;ZZZ)V
	at scalapb.spark.ToCatalystHelpers.fieldToCatalyst(ToCatalystHelpers.scala:165)
	at scalapb.spark.ToCatalystHelpers.fieldToCatalyst$(ToCatalystHelpers.scala:107)
	at scalapb.spark.Implicits$$anon$1.fieldToCatalyst(TypedEncoders.scala:123)
	at scalapb.spark.ToCatalystHelpers.$anonfun$messageToCatalyst$2(ToCatalystHelpers.scala:39)

I run my jobs in a Databricks environment using Runtime 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12) and my udf that performs the protobuf decoder is defined as

import org.apache.log4j.Logger
import org.apache.spark.sql.Column
import scalapb.spark.ProtoSQL
import example.root.root.{Event => RootEvent}
import scalapb.spark.Implicits.{messageTypedEncoder, typedEncoderToEncoder}

import scala.util.{Failure, Success, Try}

object ProtobufExample extends Serializable {

  val logger: Logger = org.apache.log4j.LogManager.getLogger(this.getClass.getSimpleName)
  val rootDecoderUdf: Column => Column = ProtoSQL.udf(Protobuf.decodeRootEvent)

  def decodeRootEvent(input: Array[Byte]): Option[RootEvent] = {
    val result = Try {
      RootEvent.parseFrom(input)
    }
    result match {
      case Success(value) => Some(value)
      case Failure(e) =>
        logger.error(s"Decode error:", e)
        None
    }
  }
}

Could you help me how to figure out this error?

[thesamet]

Let's try to isolate the problem (is it related to Databricks environment?) and also make it reproducible so I can confirm a certain solution works. Can you try to reproduce the problem outside of Databricks environment?

It would also be really helpful if you can prepare and share a minimal project (can be based on https://github.com/thesamet/sparksql-scalapb-test) and try to reproduce it both in and outside databricks. Since it will also include some specific protos that causes failure maybe that would provide another direction

[anamariavisan]

Hello @thesamet! I tried to update a service to Databricks 14.2 and above that uses the sparksql35-scalapb0_11_2.12 dependency and I got the following error:

java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke.<init>(Ljava/lang/Class;Lorg/apache/spark/sql/types/DataType;Ljava/lang/String;Lscala/collection/Seq;Lscala/collection/Seq;ZZZ)V
	at frameless.TypedEncoder$$anon$1.toCatalyst(TypedEncoder.scala:69)
	at frameless.RecordEncoder.$anonfun$toCatalyst$2(RecordEncoder.scala:155)
	at scala.collection.immutable.List.map(List.scala:293)
	at frameless.RecordEncoder.toCatalyst(RecordEncoder.scala:153)
	at frameless.TypedExpressionEncoder$.apply(TypedExpressionEncoder.scala:28)
	at scalapb.spark.Implicits.typedEncoderToEncoder(TypedEncoders.scala:119)
	at scalapb.spark.Implicits.typedEncoderToEncoder$(TypedEncoders.scala:116)
	at scalapb.spark.Implicits$.typedEncoderToEncoder(TypedEncoders.scala:122)

This doesn't happen locally. To your suggestion, I forked this repo https://github.com/thesamet/sparksql-scalapb-test/tree/master to see if the problem is related to the Databricks environment. The code can be found here: https://github.com/anamariavisan/sparksql-scalapb-test. To build the app I ran these commands:

curl -s "https://get.sdkman.io" | bash
sdk install java 11.0.24-zulu
sdk install sbt 1.6.2
sbt assembly

And to test it locally:

sdk install spark 3.5.0

spark-submit \
  --jars . \
  --class myexample.RunDemo \
  target/scala-2.12/sparksql-scalapb-test-assembly-1.0.0.jar

To test it in Databricks, I created a job and I uploaded the library target/scala-2.12/sparksql-scalapb-test-assembly-1.0.0.jar with the main class being myexample.RunDemo. I submitted the job locally and it worked, but in Databricks 14.2 and above, it failed with:

java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke.<init>(Ljava/lang/Class;Lorg/apache/spark/sql/types/DataType;Ljava/lang/String;Lscala/collection/Seq;Lscala/collection/Seq;ZZZ)V
	at scalapb.spark.ToCatalystHelpers.fieldToCatalyst(ToCatalystHelpers.scala:165)
	at scalapb.spark.ToCatalystHelpers.fieldToCatalyst$(ToCatalystHelpers.scala:107)
	at scalapb.spark.ProtoSQL$$anon$1$$anon$2.fieldToCatalyst(ProtoSQL.scala:84)
	at scalapb.spark.ToCatalystHelpers.$anonfun$messageToCatalyst$2(ToCatalystHelpers.scala:39)

I searched how to fix it and I found these issues that describe the same problem:

• Usage of "private" Spark APIs typelevel/frameless#300
• Spark 3.5 update typelevel/frameless#755
• #787 - Move encoder implementation details to external shim library (not dependent on the Spark 4 release) typelevel/frameless#800

I also left a comment on this issue on the frameless repo typelevel/frameless#787.

What is your action course on this matter for scalapb-sparksql?

[thesamet]

This is not actionable by sparksql-scalapb until there's a fix for frameless on Spark 3.5 and DBR 14.2.

[chris-twiner]

This is not actionable by sparksql-scalapb until there's a fix for frameless on Spark 3.5 and DBR 14.2.

fyi - the second stack is sparksql-scalapb internal and due to spark internal api usage rather than frameless itself. The proposed solution for frameless (#787) via shim could also be leveraged for the sparksql-scalapb api usage (tested across all supported DBRs for frameless usage at least).

[kewitschka]

Any updates on this how to fix this?

[thesamet]

1. Does it happen on most recent DBR?
2. I see that frameless still haven't adopted the shim solution (#787) - it doesn't seem we have a path forward until this happens.

[kewitschka]

1. Yes, this also happens on the latest Databricks Runtime 16.2 and following dependencies:

com.thesamet.scalapb:sparksql35-scalapb0_11_2.12:1.0.5
com.thesamet.scalapb:scalapb-runtime_2.12:0.11.17
com.thesamet.scalapb:scalapb-json4s_2.12:0.12.1
org.apache.spark:spark-core_2.12:3.5.5
org.apache.spark:spark-sql_2.12:3.5.5

2. Alright, thank you!

Until the issue is resolved, you can use Databricks Runtime 13.3, which runs on Spark 3.4.1, to avoid the error.

[chris-twiner]

It would be possible to migrate to the forked frameless which would fix one of the stack issues at least (versions of frameless are running on all tested DBRs ).

The other issues due to scalapbs own internal api would also require adopting sparksql-scalapb to use sparkutils shims/frameless, but I'd be happy to add support / accept PRs in shim for any extra shimming required as well.

Longer term (Spark 4+ / DBR 17.x+ only due to the awesome Connect/Classic api unification work) it's possible that most of sparksql-scalapb's use of internal api's could be replaced with AgnoticEncoders (TransformingEncoder probably) built on top of frameless derivation (this branch allows frameless derivation to work based on SPARK-49960). This should allow sparksql-scalapb to work on more runtimes without spark specific releases after 4+.