scala - How to convert unix timestamp to date in Spark

Scala - How to convert unix timestamp to date in Spark

In Spark, you can convert a Unix timestamp (which is the number of seconds since January 1, 1970) to a human-readable date format using the from_unixtime function. Here's how you can do it in Scala:

import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ object UnixTimestampToDate { def main(args: Array[String]): Unit = { // Create a SparkSession val spark = SparkSession.builder() .appName("UnixTimestampToDate") .getOrCreate() // Import implicits for DataFrame operations import spark.implicits._ // Sample DataFrame with Unix timestamps val df = Seq( (1, 1622908800), // June 5, 2021 (2, 1622995200) // June 6, 2021 ).toDF("id", "unix_timestamp") // Convert Unix timestamp to date val result = df.withColumn("date", from_unixtime($"unix_timestamp".cast("bigint"))) // Show the result result.show() // Stop the SparkSession spark.stop() } }

Explanation:

  • We start by importing necessary Spark components.
  • We create a SparkSession.
  • We import implicits to enable DataFrame operations.
  • We create a sample DataFrame (df) with two columns: id and unix_timestamp.
  • We use the withColumn function along with from_unixtime to convert the Unix timestamp to a human-readable date format. We cast the unix_timestamp column to a bigint before applying from_unixtime.
  • Finally, we show the result.

When you run this code, you will see the Unix timestamps converted to human-readable date format in the output DataFrame. Adjust the column names and formats according to your specific use case.

Examples

  1. "Spark Scala convert unix timestamp to date example"

    • Description: This query is likely searching for a straightforward example demonstrating how to convert Unix timestamps to human-readable dates using Spark and Scala.
    import org.apache.spark.sql.functions._ // Assuming your DataFrame has a column named 'unix_timestamp' containing Unix timestamps val dfWithDate = df.withColumn("date_column", from_unixtime(col("unix_timestamp")))

  2. "Scala Spark convert epoch to date"

    • Description: This query seeks information on how to convert epoch (Unix) timestamps to readable dates in a Spark Scala environment.
    import org.apache.spark.sql.functions._ // Assuming 'timestamp_col' is the name of the column containing Unix timestamps val dfWithDate = df.withColumn("date_column", from_unixtime(col("timestamp_col")))

  3. "Spark Scala convert unix timestamp to datetime"

    • Description: This query is looking for instructions on converting Unix timestamps to date-time representations within the Spark Scala framework.
    import org.apache.spark.sql.functions._ // Assuming 'unix_col' holds the Unix timestamps val dfWithDatetime = df.withColumn("datetime_column", from_unixtime(col("unix_col")))

  4. "Convert unix timestamp to human-readable date in Spark Scala"

    • Description: This query is asking for a solution to convert Unix timestamps into human-readable date formats specifically within Spark using Scala.
    import org.apache.spark.sql.functions._ // Assuming 'timestamp_col' contains Unix timestamps val dfWithReadableDate = df.withColumn("date_column", from_unixtime(col("timestamp_col")))

  5. "Scala Spark convert timestamp to date example"

    • Description: This query is likely seeking a practical example illustrating the conversion of timestamps to date formats in Spark using Scala.
    import org.apache.spark.sql.functions._ // Assuming 'ts_col' holds the timestamps val dfWithDate = df.withColumn("date_column", from_unixtime(col("ts_col")))

  6. "How to change unix timestamp to date in Spark Scala"

    • Description: This query is asking for a method to convert Unix timestamps to date formats specifically in a Spark Scala context.
    import org.apache.spark.sql.functions._ // Assuming 'timestamp_col' contains Unix timestamps val dfWithDate = df.withColumn("date_column", from_unixtime(col("timestamp_col")))

  7. "Scala Spark convert epoch milliseconds to date"

    • Description: This query seeks information on converting epoch milliseconds (Unix timestamps) into date formats within Spark using Scala.
    import org.apache.spark.sql.functions._ // Assuming 'epoch_millis_col' contains epoch milliseconds val dfWithDate = df.withColumn("date_column", from_unixtime(col("epoch_millis_col") / 1000))

  8. "Spark Scala convert timestamp to readable date"

    • Description: This query is asking for assistance in converting timestamps into readable date formats specifically within the Spark Scala environment.
    import org.apache.spark.sql.functions._ // Assuming 'timestamp_column' holds the timestamps val dfWithDate = df.withColumn("date_column", from_unixtime(col("timestamp_column")))

  9. "Scala Spark convert unix timestamp to string date"

    • Description: This query is looking for guidance on converting Unix timestamps into string representations of dates within Spark using Scala.
    import org.apache.spark.sql.functions._ // Assuming 'unix_time_col' contains Unix timestamps val dfWithStringDate = df.withColumn("string_date_column", date_format(from_unixtime(col("unix_time_col")), "yyyy-MM-dd HH:mm:ss"))

  10. "Spark Scala convert unix timestamp to local date"

More Tags

tidytext rerender paypal textview cmake cxf-codegen-plugin cluster-analysis angular2-observables file-sharing show

More Programming Questions

More Mortgage and Real Estate Calculators

More Pregnancy Calculators

More Entertainment Anecdotes Calculators

More Math Calculators