Spark on AWS Glue: Performance Tuning 2 (Glue DynamicFrame vs Spark DataFrame)

This is a continuation of my previous posts as follows.

1. Spark on AWS Glue: Performance Tuning 1 (CSV vs Parquet)

2. Spark on AWS Glue: Performance Tuning 2 (Glue DynamicFrame vs Spark DataFrame)

3. Spark on AWS Glue: Performance Tuning 3 (Impact of Partition Quantity)

4. Spark on AWS Glue: Performance Tuning 4 (Spark Join)

5. Spark on AWS Glue: Performance Tuning 5 (Using Cache)

Glue DynamicFrame vs Spark DataFrame

Let's compare them using the Parquet file which I created in the part 1.

Data Read Speed Comparison

We will read a single large Parquet file and a highly partitioned Parquet file.

with timer('df'): dyf = glueContext.create_dynamic_frame.from_options( "s3", { "paths": [ "s3://.../parquet-chunk-high/" ] }, "parquet", ) print(dyf.count()) with timer('df partition'): dyf = glueContext.create_dynamic_frame.from_options( "s3", { "paths": [ "s3:/.../parquet-partition-high/" ] }, "parquet", ) print(dyf.count()) 

324917265 [df] done in 125.9965 s 324917265 [df partition] done in 55.9798 s 

DynamicFrame is too slow...

Summary

• Based on the part 1 (Reading Speed Comparison), spark.read is 27.1 s (for single large file) and 36.3 s (for highly partitioned file), so DynamicFrame is quite slow.
• Interestingly, the speed of reading partitioned data is faster than single large Parquet file.