[SPARK-49715] Add Java 21-based SparkCluster example

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to add Java 21-based SparkCluster example.

Why are the changes needed?

Apache Spark starts to publish Java 21-based image from Today (2024-09-19). This new example will illustrate how to use it.

• [SPARK-49703] Publish Java 21 Docker image for preview1 spark-docker#69

Does this PR introduce any user-facing change?

No, this is a new example.

How was this patch tested?

Manual review.

$ kubectl apply -f cluster-java21.yaml 

$ kubectl get sparkcluster NAME CURRENT STATE AGE cluster-java21 RunningHealthy 9s 

$ kubectl describe sparkcluster cluster-java21 Name: cluster-java21 Namespace: default Labels: <none> Annotations: <none> API Version: spark.apache.org/v1alpha1 Kind: SparkCluster Metadata: Creation Timestamp: 2024-09-19T04:25:30Z Finalizers: sparkclusters.spark.apache.org/finalizer Generation: 2 Resource Version: 96663 UID: 9421c957-380d-4a26-a266-379304bf83ee Spec: Cluster Tolerations: Instance Config: Init Workers: 3 Max Workers: 3 Min Workers: 3 Master Spec: Runtime Versions: Spark Version: 4.0.0-preview1 Spark Conf: spark.kubernetes.container.image: apache/spark:4.0.0-preview1-java21 spark.master.rest.enabled: true spark.master.rest.host: 0.0.0.0 spark.master.ui.title: Prod Spark Cluster (Java 21) spark.ui.reverseProxy: true Worker Spec: Status: Current Attempt Summary: Attempt Info: Id: 0 Current State: Current State Summary: RunningHealthy Last Transition Time: 2024-09-19T04:25:30.665095088Z Message: Cluster has reached ready state. State Transition History: 0: Current State Summary: Submitted Last Transition Time: 2024-09-19T04:25:30.640072963Z Message: Spark cluster has been submitted to Kubernetes Cluster. 1: Current State Summary: RunningHealthy Last Transition Time: 2024-09-19T04:25:30.665095088Z Message: Cluster has reached ready state. Events: <none> 

$ k get pod NAME READY STATUS RESTARTS AGE cluster-java21-master-0 1/1 Running 0 3m20s cluster-java21-worker-0 1/1 Running 0 3m20s cluster-java21-worker-1 1/1 Running 0 3m20s cluster-java21-worker-2 1/1 Running 0 3m20s spark-kubernetes-operator-778b9bbdc6-fqks9 1/1 Running 0 20m 

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

cc @viirya

[dongjoon-hyun]

Thank you. Merged to main.