[SPARK-49703] Publish Java 21 Docker image for preview1 #69
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Member Author
| Thank you, @HyukjinKwon . |
viirya reviewed Sep 19, 2024
Member Author
| For the record, while preparing |
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Member Author
| Thank you, @viirya . |
Yikun reviewed Sep 19, 2024
Member Author
| Thank you, @Yikun . |
viirya approved these changes Sep 19, 2024
Member Author
| Thank you, @HyukjinKwon , @viirya , @Yikun . |
This was referenced Sep 19, 2024
dongjoon-hyun added a commit to apache/spark-kubernetes-operator that referenced this pull request Sep 19, 2024
### What changes were proposed in this pull request? This PR aims to add `Java 21`-based SparkPi example. ### Why are the changes needed? Apache Spark starts to publish Java 21-based image from Today (2024-09-19). This new example will illustrate how to use it. - apache/spark-docker#69 ### Does this PR introduce _any_ user-facing change? No, this is a new example. ### How was this patch tested? Manual review. ``` $ kubectl apply -f examples/pi-java21.yaml ``` ``` $ kubectl get sparkapp NAME CURRENT STATE AGE pi-java21 ResourceReleased 28s ``` ``` $ kubectl describe sparkapp pi-java21 Name: pi-java21 Namespace: default Labels: <none> Annotations: <none> API Version: spark.apache.org/v1alpha1 Kind: SparkApplication Metadata: Creation Timestamp: 2024-09-19T04:08:16Z Finalizers: sparkapplications.spark.apache.org/finalizer Generation: 2 Resource Version: 95294 UID: 2bc46e8d-6339-4867-9a28-6552c6c471e5 Spec: Application Tolerations: Application Timeout Config: Driver Ready Timeout Millis: 300000 Driver Start Timeout Millis: 300000 Executor Start Timeout Millis: 300000 Force Termination Grace Period Millis: 300000 Termination Requeue Period Millis: 2000 Instance Config: Init Executors: 0 Max Executors: 0 Min Executors: 0 Resource Retain Policy: OnFailure Restart Config: Max Restart Attempts: 3 Restart Backoff Millis: 30000 Restart Policy: Never Deployment Mode: ClusterMode Driver Args: Jars: local:///opt/spark/examples/jars/spark-examples.jar Main Class: org.apache.spark.examples.SparkPi Runtime Versions: Scala Version: 2.13 Spark Version: 4.0.0-preview1 Spark Conf: spark.dynamicAllocation.enabled: true spark.dynamicAllocation.maxExecutors: 3 spark.dynamicAllocation.shuffleTracking.enabled: true spark.kubernetes.authenticate.driver.serviceAccountName: spark spark.kubernetes.container.image: apache/spark:4.0.0-preview1-java21-scala spark.log.structuredLogging.enabled: false Status: Current Attempt Summary: Attempt Info: Id: 0 Current State: Current State Summary: ResourceReleased Last Transition Time: 2024-09-19T04:08:33.316041381Z State Transition History: 0: Current State Summary: Submitted Last Transition Time: 2024-09-19T04:08:16.584629470Z Message: Spark application has been created on Kubernetes Cluster. 1: Current State Summary: DriverRequested Last Transition Time: 2024-09-19T04:08:17.269457304Z Message: Requested driver from resource scheduler. 2: Current State Summary: DriverStarted Last Transition Time: 2024-09-19T04:08:17.809898304Z Message: Driver has started running. 3: Current State Summary: DriverReady Last Transition Time: 2024-09-19T04:08:17.810393971Z Message: Driver has reached ready state. 4: Current State Summary: RunningHealthy Last Transition Time: 2024-09-19T04:08:17.828526471Z Message: Application is running healthy. 5: Current State Summary: Succeeded Last Transition Time: 2024-09-19T04:08:33.241514089Z Message: Spark application completed successfully. 6: Current State Summary: ResourceReleased Last Transition Time: 2024-09-19T04:08:33.316041381Z Events: <none> ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #129 from dongjoon-hyun/SPARK-49714. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun added a commit to apache/spark-kubernetes-operator that referenced this pull request Sep 19, 2024
### What changes were proposed in this pull request? This PR aims to add `Java 21`-based `SparkCluster` example. ### Why are the changes needed? Apache Spark starts to publish Java 21-based image from Today (2024-09-19). This new example will illustrate how to use it. - apache/spark-docker#69 ### Does this PR introduce _any_ user-facing change? No, this is a new example. ### How was this patch tested? Manual review. ``` $ kubectl apply -f cluster-java21.yaml ``` ``` $ kubectl get sparkcluster NAME CURRENT STATE AGE cluster-java21 RunningHealthy 9s ``` ``` $ kubectl describe sparkcluster cluster-java21 Name: cluster-java21 Namespace: default Labels: <none> Annotations: <none> API Version: spark.apache.org/v1alpha1 Kind: SparkCluster Metadata: Creation Timestamp: 2024-09-19T04:25:30Z Finalizers: sparkclusters.spark.apache.org/finalizer Generation: 2 Resource Version: 96663 UID: 9421c957-380d-4a26-a266-379304bf83ee Spec: Cluster Tolerations: Instance Config: Init Workers: 3 Max Workers: 3 Min Workers: 3 Master Spec: Runtime Versions: Spark Version: 4.0.0-preview1 Spark Conf: spark.kubernetes.container.image: apache/spark:4.0.0-preview1-java21 spark.master.rest.enabled: true spark.master.rest.host: 0.0.0.0 spark.master.ui.title: Prod Spark Cluster (Java 21) spark.ui.reverseProxy: true Worker Spec: Status: Current Attempt Summary: Attempt Info: Id: 0 Current State: Current State Summary: RunningHealthy Last Transition Time: 2024-09-19T04:25:30.665095088Z Message: Cluster has reached ready state. State Transition History: 0: Current State Summary: Submitted Last Transition Time: 2024-09-19T04:25:30.640072963Z Message: Spark cluster has been submitted to Kubernetes Cluster. 1: Current State Summary: RunningHealthy Last Transition Time: 2024-09-19T04:25:30.665095088Z Message: Cluster has reached ready state. Events: <none> ``` ``` $ k get pod NAME READY STATUS RESTARTS AGE cluster-java21-master-0 1/1 Running 0 3m20s cluster-java21-worker-0 1/1 Running 0 3m20s cluster-java21-worker-1 1/1 Running 0 3m20s cluster-java21-worker-2 1/1 Running 0 3m20s spark-kubernetes-operator-778b9bbdc6-fqks9 1/1 Running 0 20m ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #130 from dongjoon-hyun/SPARK-49715. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Member Author
| For the record, Java 21 images are published successfully. |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aims to publish Java 21 Docker image for
preview1and will be extended forpreview2.Why are the changes needed?
Apache Spark supports Java 21 via SPARK-43831.
Does this PR introduce any user-facing change?
No, this is a new image.
How was this patch tested?
Pass the CIs.