Dockerfile for building and running + SBT Fixes.

[jayunit100]

Quick pass at #110

[jayunit100]

@jkbradley @mengxr

the CMD is a little wacky, open to suggestions. Otherwise I'd love to merge this as a first pass.

I've also updated the SBT deps in a separate commit.

Do you think we could push this through? We can iterate against it very easily since the Dockerfile allows cross platform testing.

[jayunit100]

cc @tdas cc @JoshRosen

[rxin]

@jayunit100 this is pretty difficult to review given the size. Can you say which part is just standard sbt file versus code you did?

[jayunit100]

Hi... Yup ,

• I only added the dockerfile,
• everything else is just SBT.

I did this all in one PR because the two commits are self validating (the Dockerfile serves both as a runnable spark-perf driver, as well as a unit test for the SBT repo :) )

[jayunit100]

cc @willb @mattf

[jayunit100]

This CMD is just an example of how to invoke . Obviously people will customize it to their own preferences. It will run without issue when running docker build ./

[willb]

@jayunit100 you can just replace the bundled sbt runner (i.e. spark-perf/spark-tests/sbt/sbt) with the one from sbt-extras; you don't need the other stuff in the sbt-extras repository.

[jayunit100]

ah yeah good idea, o k testing nowww

[jayunit100]

ok, updated...

[jayunit100]

PTAL again, should be easier this time, removed tangential sbt-extras,
@rxin thanks :)

[jayunit100]

bump.. would like to start iterating against this

[rxin]

One thing is that we are considering just deprecating the existing code in spark-perf and replace it with the code in spark-sql-perf, since that one is better structured.

[jayunit100]

I figured as much since its been inactive.
Still though this pr at least will finalize the build for the repo. Right now it literally won't build as is with the bootstrapped

[nchammas]

@rxin - Does that mean we will no longer have a maintained suite of performance tests for the non-SQL parts of Spark?

[jayunit100]

@rxin maybe we could merge this anyway, since

• the migration to spark-sql-perf won't happen overnight, and anyway,
• people are relying on this repo (and this is a critical fix, w/o it, bootstrapping isnt working).

[willb]

I share the concern @nchammas raises about having maintained performance tests for Spark core, and agree with @jayunit100 that a test suite that builds is infinitely preferable to one that does not. (While I'm asking for things, it would be great to see the Spark 2.0 fixes mentioned in #108 incorporated as well, even if spark-sql-perf is the way of the future.)

[jayunit100]

layered another commit into this for spark release. i can squash but at this point probably not much need to squash since they are generally distinct

[jayunit100]

I will continue pushing WIP commits to this PR, since this seems to have languished.
Open to ping with comments when we are ready to merge.