Classpath stability for tests

[seancorfield]

Describe the bug
The classpath that poly test calculates is not stable/reproducible across machines, leading to unpredictable behavior of tests across multiple machines.

To Reproduce
I added debug output to the test orchestrator to print all-paths so that I could see the order being different on multiple machines. The Jetty libraries have changed their group/artifact over time but retained some of the same class names which means that a non-deterministic ordering of the classpath means that you can have tests pass on one machine but fail on another due to that ordering cause a different artifact to be found "first" on the classpath and an incompatible class used.

Expected behavior
tools.deps.alpha uses a specific sorting algorithm to ensure a reproducible classpath across all environments -- the ideal would be Polylith producing that exact same order (this is tricky because poly test has to construct a projection of the :test alias across all the bricks in order to construct the "basis"). Not so ideal but acceptable would at least be for the order of the classpath produced by poly test to be deterministic. That means not using sets or hash maps to store elements of the classpath.

[tengstrand]

This should be solved by #263 @seancorfield I think.

[seancorfield]

No. This is not fixed. Please reopen it.

[tengstrand]

I now sort the libs (line 17) in the issue-261 branch:

polylith/components/test-runner-orchestrator/src/polylith/clj/core/test_runner_orchestrator/core.clj

Lines 15 to 18 in ab63192

	(defn resolve-deps [{:keys [name] :as project} settings is-verbose color-mode]
	(try
	(sort (into #{} (mapcat #(-> % second :paths))
	(deps/resolve-deps project settings is-verbose)))

I'm not sure if that solves your problem @seancorfield. I couldn't figure out how to let the poly tool produce the exact same order as tools.deps (even though you tried to explain it).

[seancorfield]

That may make things worse -- but it does at least make things predictable (even if it ends up being predictably wrong).

I want to keep this open until I can spend some time looking into this problem. It's not going to affect most people, with smaller projects, but this sort of thing can have a really bad impact on projects of the scale we have at work.

[ieugen]

I think I've also encountered this bug.

In my case I get this error while running clojure -M:poly test

Caused by: java.sql.SQLException: No suitable driver
 at java.sql.DriverManager.getDriver (DriverManager.java:299)
    com.zaxxer.hikari.util.DriverDataSource.<init> (DriverDataSource.java:106)
    com.zaxxer.hikari.pool.PoolBase.initializeDataSource (PoolBase.java:326)
    com.zaxxer.hikari.pool.PoolBase.<init> (PoolBase.java:112)
    com.zaxxer.hikari.pool.HikariPool.<init> (HikariPool.java:93)
    com.zaxxer.hikari.HikariDataSource.getConnection (HikariDataSource.java:112)
    next.jdbc.connection$make_connection.invokeStatic (connection.clj:452)
    next.jdbc.connection$make_connection.invoke (connection.clj:436)
    next.jdbc.connection$eval22475$fn__22476.invoke (connection.clj:481)
    next.jdbc.protocols$eval22205$fn__22206$G__22196__22213.invoke (protocols.clj:25)
    next.jdbc$get_connection.invokeStatic (jdbc.clj:168)
    next.jdbc$get_connection.invoke (jdbc.clj:147)
    dre.semmed_ds.interface_test$fn__93117.invokeStatic (interface_test.clj:17)

I do have com.h2database/h2 {:mvn/version "2.1.214"} added to component class path, :test classpath, dev project classpath and :test classpath and also project test classpath .

I still get the above error.

Inside dev repl I can create the component that creates a hikari datasource using next jdbc component .
I can also run the code for the test and it works.

[tengstrand]

Is it possible to reproduce the problem in a smaller public repo and share it with me @ieugen?

[ieugen]

@tengstrand : I added a repro in this PR ieugen/poly-rcf#1 .

I run clojure -M:poly test .
I have component added to 2 projects.
First project tests are ran ok.
Second project fails with missing class.

Big questions are:

• Why are tests ran 2 times for the same component?
• Why is the classpath different for second run?

[wkok]

Any update on this issue?

We're encountering the same problem on some machines when running poly tests and at first we thought it's related to a specific dependency's problem as we've logged here:
https://discuss.xtdb.com/t/arrow-vector-conflicting-dependency/380/2

but then I discovered this issue which might explain better what we're seeing.

[tengstrand]

I have some ideas on how to solve this, so will have a look.