-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dockerfile for building and running + SBT Fixes. #111
Conversation
the I've also updated the SBT deps in a separate commit. Do you think we could push this through? We can iterate against it very easily since the Dockerfile allows cross platform testing. |
cc @tdas cc @JoshRosen |
@jayunit100 this is pretty difficult to review given the size. Can you say which part is just standard sbt file versus code you did? |
Hi... Yup ,
I did this all in one PR because the two commits are self validating (the Dockerfile serves both as a runnable spark-perf driver, as well as a unit test for the SBT repo :) ) |
ADD . /opt/spark-perf/ | ||
WORKDIR /opt/spark-perf/ | ||
|
||
CMD if [ -n "$SPARK_MASTER_URL" ]; then echo "FAILED! Missing spark master url" && exit 1 ; fi ; export PATH=$PATH:/opt/sbt-extras/ && echo $SPARK_MASTER_URL > /root/spark-ec2/cluster-url && ./bin/run |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This CMD is just an example of how to invoke . Obviously people will customize it to their own preferences. It will run without issue when running docker build ./
@jayunit100 you can just replace the bundled sbt runner (i.e. |
ah yeah good idea, o k testing nowww |
1627d9c
to
7c3f4f0
Compare
ok, updated... |
PTAL again, should be easier this time, removed tangential sbt-extras, |
4cb2c8c
to
5a92642
Compare
bump.. would like to start iterating against this |
One thing is that we are considering just deprecating the existing code in spark-perf and replace it with the code in spark-sql-perf, since that one is better structured. |
I figured as much since its been inactive. |
@rxin - Does that mean we will no longer have a maintained suite of performance tests for the non-SQL parts of Spark? |
5a92642
to
a09ebc5
Compare
@rxin maybe we could merge this anyway, since
|
I share the concern @nchammas raises about having maintained performance tests for Spark core, and agree with @jayunit100 that a test suite that builds is infinitely preferable to one that does not. (While I'm asking for things, it would be great to see the Spark 2.0 fixes mentioned in #108 incorporated as well, even if spark-sql-perf is the way of the future.) |
c77b98e
to
6de8a76
Compare
layered another commit into this for spark release. i can squash but at this point probably not much need to squash since they are generally distinct |
…o avoid dupe functionality., Add example of how to customize tests from a local file
I will continue pushing WIP commits to this PR, since this seems to have languished. |
Quick pass at #110