Skip to content

Fix spark-submit failures in MacOS in CI #174

@facundominguez

Description

@facundominguez

When working on #172, the following error was found

Exception in thread "main" java.lang.UnsatisfiedLinkError: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/sparkle-app-4130550189215234367/clodl-top0: dlopen(/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/sparkle-app-4130550189215234367/clodl-top0, 1): Library not loaded: @loader_path/libHSsparkle-lib-ghc9.0.2.dylib
  Referenced from: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/sparkle-app-4130550189215234367/libHSappsZShelloZShello-hs-ghc9.0.2.dylib
  Reason: image not found
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817)
	at java.lang.Runtime.load0(Runtime.java:782)
	at java.lang.System.load(System.java:1100)
	at io.tweag.sparkle.SparkleBase.loadApplication(SparkleBase.java:63)
	at io.tweag.sparkle.SparkleBase.<clinit>(SparkleBase.java:28)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:816)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/01/18 13:33:10 INFO util.ShutdownHookManager: Shutdown hook called
23/01/18 13:33:10 INFO util.ShutdownHookManager: Deleting directory /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/spark-8e2c2e74-029d-4f11-9c9b-99e9e1e719c3

This ticket is about diagnosing and fixing such failure.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions