[SPARK-49701] Use JDK for Spark 3.5+ Docker image #66
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
dongjoon-hyun force-pushed the SPARK-49701 branch from ea60f78 to 6467280 Compare 
viirya approved these changes Sep 18, 2024
Member Author
| Thank you, @viirya . |
Member Author
| Merged to master. |
Member Author
| For the record, the fixed images are released. |
This was referenced Sep 19, 2024
dongjoon-hyun added a commit to apache/spark-kubernetes-operator that referenced this pull request Sep 19, 2024
### What changes were proposed in this pull request? This PR aims to propose to use `apache/spark` images instead of `spark` because `apache/spark` images are published first. For example, the following are only available in `apache/spark` as of now. - apache/spark-docker#66 - apache/spark-docker#67 - apache/spark-docker#68 ### Why are the changes needed? To apply the latest bits earlier. ### Does this PR introduce _any_ user-facing change? There is no change from `Apache Spark K8s Operator`. Only the underlying images are changed. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #128 from dongjoon-hyun/SPARK-49706. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
| Hi Team, think similar changes is required for Java 11 as well. At the moment we are using spark:3.5.2-scala2.12-java11-python3-ubuntu. We wanted to check heap dump but ended up into issue , It show something like below -: We went aheads and checked the Java installed within docker image to get Jmap by doing an |
Member Author
| To @anish97IND , please use the latest Spark 3.5.3 images instead of 3.5.2. FYI, there are correctness fixes in Apache Spark 3.5.3. |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aims to use JDK for Spark 3.5+ Docker image. Apache Spark Dockerfile are updated already.
17instead of17-jreimage in K8s Dockerfile spark#4576221instead of21-jreimage in K8s Dockerfile spark#45761Why are the changes needed?
Since Apache Spark 3.5.0, SPARK-44153 starts to use
jmaplike the following.Heap Histogramcolumn inExecutorstab spark#41709https://github.com/apache/spark/blob/c832e2ac1d04668c77493577662c639785808657/core/src/main/scala/org/apache/spark/util/Utils.scala#L2030
Does this PR introduce any user-facing change?
Yes, the user can use
Heap Histogramfeature.How was this patch tested?
Pass the CIs.