MLE-23024 Bumping to Spark 4.1.0 preview1 #483
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Using constraints as well to ensure the pom.xml file contains the same dependency updates. Spark 4.1.0 is using io.netty 4.1.122 - not changing that for now, need to see what Black Duck thinks.
rjrudin
requested review from
BillFarber,
anu3990 and
stevebio
as code owners
|
Copyright Validation Results ⏭️ Skipped (Excluded) Files
✅ Valid Files
✅ All files have valid copyright headers! |
There was a problem hiding this comment.
Pull Request Overview
Upgrades Apache Spark from version 4.0.1 to 4.1.0-preview1 to leverage new features and improvements in the preview release.
- Updated Spark version to 4.1.0-preview1 in gradle.properties
- Updated test assertions to match new error message format in Spark 4.1.0
- Restructured build configuration to use constraints for dependency management
Reviewed Changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| gradle.properties | Updated sparkVersion from 4.0.1 to 4.1.0-preview1 |
| ImportParquetFilesTest.java | Updated error message assertion to match new Spark 4.1.0 format |
| ImportOrcFilesTest.java | Updated error message assertion and removed extra blank line |
| ImportAvroFilesTest.java | Updated error message assertion to match new Spark 4.1.0 format |
| OverrideSparkConfTest.java | Updated error message assertion to match new Spark 4.1.0 format |
| flux-cli/build.gradle | Restructured dependency management with constraints and removed distributionDependencies configuration |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| because "Force Spark's Hadoop dependencies from 3.4.1 to ${hadoopVersion}" | ||
| } | ||
| implementation("org.codehaus.janino:janino:3.1.12") { | ||
| because "Bumping from 3.1.9 (what Spark SQL 4.0.1 depends on) to 3.1.12 to minimize CVEs." |
There was a problem hiding this comment.
The comment mentions 'Spark SQL 4.0.1' but the version has been updated to 4.1.0-preview1. Update the comment to reflect the correct Spark version being used.
Suggested change
| because "Bumping from 3.1.9 (what Spark SQL 4.0.1 depends on) to 3.1.12 to minimize CVEs." | |
| because "Bumping from 3.1.9 (what Spark SQL 4.1.0-preview1 depends on) to 3.1.12 to minimize CVEs." |
Copilot uses AI. Check for mistakes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Using constraints as well to ensure the pom.xml file contains the same dependency updates.
Spark 4.1.0 is using io.netty 4.1.122 - not changing that for now, need to see what Black Duck thinks.