Skip to content
Prev Previous commit
Next Next commit
bump versions
  • Loading branch information
maltesander committed Oct 10, 2023
commit fd85eef608c4b0ce32383264fc1caad1ee1892ca
2 changes: 1 addition & 1 deletion docs/modules/spark-k8s/examples/example-encapsulated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
version: "1.0"
sparkImage:
productVersion: 3.3.0 # <1>
productVersion: 3.5.0 # <1>
mode: cluster
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: /stackable/spark/examples/jars/spark-examples.jar # <2>
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/spark-k8s/examples/example-history-app.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
version: "1.0"
sparkImage:
productVersion: 3.3.0
productVersion: 3.5.0
pullPolicy: IfNotPresent
mode: cluster
mainClass: org.apache.spark.examples.SparkPi
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ metadata:
name: spark-history
spec:
image:
productVersion: 3.3.0
productVersion: 3.5.0
logFileDirectory: # <1>
s3:
prefix: eventlogs/ # <2>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ metadata:
spec:
version: "1.0"
sparkImage:
productVersion: 3.3.0
productVersion: 3.5.0
mode: cluster
mainApplicationFile: s3a://stackable-spark-k8s-jars/jobs/ny-tlc-report-1.1.0.jar # <3>
mainClass: tech.stackable.demo.spark.NYTLCReport
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ metadata:
spec:
version: "1.0"
sparkImage:
productVersion: 3.3.0
productVersion: 3.5.0
mode: cluster
mainApplicationFile: s3a://stackable-spark-k8s-jars/jobs/ny_tlc_report.py # <1>
args:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ spec:
version: "1.0"
image: docker.stackable.tech/stackable/ny-tlc-report:0.1.0 # <1>
sparkImage:
productVersion: 3.3.0
productVersion: 3.5.0
mode: cluster
mainApplicationFile: local:///stackable/spark/jobs/ny_tlc_report.py # <2>
args:
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/spark-k8s/examples/example-sparkapp-pvc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ metadata:
spec:
version: "1.0"
sparkImage:
productVersion: 3.3.0
productVersion: 3.5.0
mode: cluster
mainApplicationFile: s3a://stackable-spark-k8s-jars/jobs/ny-tlc-report-1.0-SNAPSHOT.jar # <1>
mainClass: org.example.App # <2>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
version: "1.0"
sparkImage:
productVersion: 3.3.0
productVersion: 3.5.0
mode: cluster
mainApplicationFile: s3a://my-bucket/spark-examples.jar # <1>
mainClass: org.apache.spark.examples.SparkPi # <2>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ metadata:
spec:
version: "1.0"
sparkImage:
productVersion: 3.3.0
productVersion: 3.5.0
mode: cluster
mainApplicationFile: local:///stackable/spark/examples/src/main/python/streaming/hdfs_wordcount.py
args:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ metadata:
spec:
version: "1.0"
sparkImage:
productVersion: 3.3.0
productVersion: 3.5.0
mode: cluster
mainApplicationFile: local:///stackable/spark/examples/src/main/python/pi.py
driver:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ metadata:
spec:
version: "1.0"
sparkImage:
productVersion: 3.3.0
productVersion: 3.5.0
mode: cluster
mainApplicationFile: local:///stackable/spark/examples/src/main/python/pi.py
driver:
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/spark-k8s/pages/crd-reference.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Below are listed the CRD fields that can be defined by the user:
|User-supplied image containing spark-job dependencies that will be copied to the specified volume mount

|`spec.sparkImage`
| Spark image which will be deployed to driver and executor pods, which must contain spark environment needed by the job e.g. `docker.stackable.tech/stackable/spark-k8s:3.3.0-stackable0.3.0`
| Spark image which will be deployed to driver and executor pods, which must contain spark environment needed by the job e.g. `docker.stackable.tech/stackable/spark-k8s:3.5.0-stackable0.0.0-dev`

|`spec.sparkImagePullPolicy`
| Optional Enum (one of `Always`, `IfNotPresent` or `Never`) that determines the pull policy of the spark job image
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ For a role group of the Spark history server, you can specify: `configOverrides`

The `security.properties` file is used to configure JVM security properties. It is very seldom that users need to tweak any of these, but there is one use-case that stands out, and that users need to be aware of: the JVM DNS cache.

The JVM manages it's own cache of successfully resolved host names as well as a cache of host names that cannot be resolved. Some products of the Stackable platform are very sensible to the contents of these caches and their performance is heavily affected by them. As of version 3.4.0, Apache Spark may perform poorly if the positive cache is disabled. To cache resolved host names, and thus speeding up queries you can configure the TTL of entries in the positive cache like this:
The JVM manages its own cache of successfully resolved host names as well as a cache of host names that cannot be resolved. Some products of the Stackable platform are very sensible to the contents of these caches and their performance is heavily affected by them. As of version 3.4.0, Apache Spark may perform poorly if the positive cache is disabled. To cache resolved host names, and thus speeding up queries you can configure the TTL of entries in the positive cache like this:

[source,yaml]
----
Expand Down
5 changes: 3 additions & 2 deletions docs/modules/spark-k8s/partials/supported-versions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,6 @@
// Stackable Platform documentation.
// Please sort the versions in descending order (newest first)

- 3.4.0 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11)
- 3.3.0 (Hadoop 3.3.3, Scala 2.12, Python 3.9, Java 11)
- 3.5.0 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11)
- 3.4.1 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11)
- 3.4.0 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11) (deprecated)
2 changes: 1 addition & 1 deletion rust/crd/src/affinity.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ mod test {
name: spark-history
spec:
image:
productVersion: 3.3.0
productVersion: 3.5.0
logFileDirectory:
s3:
prefix: eventlogs/
Expand Down