Skip to content

Commit 6c6dcd5

Browse files
SireInsectusSireInsectus
SireInsectus
authored and
SireInsectus
committed
Publishing v2.0.1
1 parent 314c715 commit 6c6dcd5

File tree

7 files changed

+27
-23
lines changed

7 files changed

+27
-23
lines changed

‎Apache-Spark-Programming-with-Databricks/ASP 2 - Spark Core/ASP 2.2 - Reader & Writer.py‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@
55
# MAGIC <img src="https://databricks.com/wp-content/uploads/2018/03/db-academy-rgb-1200px.png" alt="Databricks Learning" style="width: 600px">
66
# MAGIC </div>
77

8+
# COMMAND ----------
9+
10+
11+
812
# COMMAND ----------
913

1014
# MAGIC %md # Reader & Writer
@@ -32,9 +36,7 @@
3236
# MAGIC %md ## DataFrameReader
3337
# MAGIC Interface used to load a DataFrame from external storage systems
3438
# MAGIC
35-
# MAGIC ```
36-
# MAGIC spark.read.parquet("path/to/files")
37-
# MAGIC ```
39+
# MAGIC **`spark.read.parquet("path/to/files")`**
3840
# MAGIC
3941
# MAGIC DataFrameReader is accessible through the SparkSession attribute **`read`**. This class includes methods to load DataFrames from different external storage systems.
4042

‎Apache-Spark-Programming-with-Databricks/ASP 6 - Delta Lake/ASP 6.1 - Delta Lake.py‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@
3333

3434
# COMMAND ----------
3535

36-
events_df = spark.read.parquet(datasets_dir + "/events/events.parquet/")
36+
events_df = spark.read.format("parquet").load(f"{datasets_dir}/events/events.parquet")
3737
display(events_df)
3838

3939
# COMMAND ----------
@@ -43,7 +43,7 @@
4343

4444
# COMMAND ----------
4545

46-
delta_path = working_dir + "/delta-events"
46+
delta_path = f"{working_dir}/delta-events"
4747
events_df.write.format("delta").mode("overwrite").save(delta_path)
4848

4949
# COMMAND ----------
@@ -97,7 +97,7 @@
9797

9898
# COMMAND ----------
9999

100-
display(dbutils.fs.ls(delta_path + "/_delta_log/"))
100+
display(dbutils.fs.ls(f"{delta_path}/_delta_log/"))
101101

102102
# COMMAND ----------
103103

@@ -112,7 +112,7 @@
112112

113113
# COMMAND ----------
114114

115-
display(spark.read.json(delta_path + "/_delta_log/00000000000000000000.json"))
115+
display(spark.read.json(f"{delta_path}/_delta_log/00000000000000000000.json"))
116116

117117
# COMMAND ----------
118118

@@ -122,15 +122,15 @@
122122

123123
# COMMAND ----------
124124

125-
display(spark.read.json(delta_path + "/_delta_log/00000000000000000001.json"))
125+
display(spark.read.json(f"{delta_path}/_delta_log/00000000000000000001.json"))
126126

127127
# COMMAND ----------
128128

129129
# MAGIC %md Finally, let's take a look at the files inside one of the state partitions. The files inside corresponds to the partition commit (file 01) in the _delta_log directory.
130130

131131
# COMMAND ----------
132132

133-
display(dbutils.fs.ls(delta_path + "/state=CA/"))
133+
display(dbutils.fs.ls(f"{delta_path}/state=CA/"))
134134

135135
# COMMAND ----------
136136

@@ -167,7 +167,7 @@
167167

168168
# COMMAND ----------
169169

170-
display(dbutils.fs.ls(delta_path + "/state=CA/"))
170+
display(dbutils.fs.ls(f"{delta_path}/state=CA/"))
171171

172172
# COMMAND ----------
173173

‎Apache-Spark-Programming-with-Databricks/ASP 6 - Delta Lake/Labs/ASP 6.1L - Delta Lake Lab.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121

2222
# COMMAND ----------
2323

24-
sales_df = spark.read.parquet(datasets_dir + "/sales/sales.parquet/)
24+
sales_df = spark.read.parquet(datasets_dir + "/sales/sales.parquet")
2525
delta_sales_path = working_dir + "/delta-sales"
2626

2727
# COMMAND ----------

‎Apache-Spark-Programming-with-Databricks/Solutions/ASP 2 - Spark Core/ASP 2.2 - Reader & Writer.py‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@
55
# MAGIC <img src="https://databricks.com/wp-content/uploads/2018/03/db-academy-rgb-1200px.png" alt="Databricks Learning" style="width: 600px">
66
# MAGIC </div>
77

8+
# COMMAND ----------
9+
10+
11+
812
# COMMAND ----------
913

1014
# MAGIC %md # Reader & Writer
@@ -32,9 +36,7 @@
3236
# MAGIC %md ## DataFrameReader
3337
# MAGIC Interface used to load a DataFrame from external storage systems
3438
# MAGIC
35-
# MAGIC ```
36-
# MAGIC spark.read.parquet("path/to/files")
37-
# MAGIC ```
39+
# MAGIC **`spark.read.parquet("path/to/files")`**
3840
# MAGIC
3941
# MAGIC DataFrameReader is accessible through the SparkSession attribute **`read`**. This class includes methods to load DataFrames from different external storage systems.
4042

‎Apache-Spark-Programming-with-Databricks/Solutions/ASP 6 - Delta Lake/ASP 6.1 - Delta Lake.py‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@
3333

3434
# COMMAND ----------
3535

36-
events_df = spark.read.parquet(datasets_dir + "/events/events.parquet/")
36+
events_df = spark.read.format("parquet").load(f"{datasets_dir}/events/events.parquet")
3737
display(events_df)
3838

3939
# COMMAND ----------
@@ -43,7 +43,7 @@
4343

4444
# COMMAND ----------
4545

46-
delta_path = working_dir + "/delta-events"
46+
delta_path = f"{working_dir}/delta-events"
4747
events_df.write.format("delta").mode("overwrite").save(delta_path)
4848

4949
# COMMAND ----------
@@ -97,7 +97,7 @@
9797

9898
# COMMAND ----------
9999

100-
display(dbutils.fs.ls(delta_path + "/_delta_log/"))
100+
display(dbutils.fs.ls(f"{delta_path}/_delta_log/"))
101101

102102
# COMMAND ----------
103103

@@ -112,7 +112,7 @@
112112

113113
# COMMAND ----------
114114

115-
display(spark.read.json(delta_path + "/_delta_log/00000000000000000000.json"))
115+
display(spark.read.json(f"{delta_path}/_delta_log/00000000000000000000.json"))
116116

117117
# COMMAND ----------
118118

@@ -122,15 +122,15 @@
122122

123123
# COMMAND ----------
124124

125-
display(spark.read.json(delta_path + "/_delta_log/00000000000000000001.json"))
125+
display(spark.read.json(f"{delta_path}/_delta_log/00000000000000000001.json"))
126126

127127
# COMMAND ----------
128128

129129
# MAGIC %md Finally, let's take a look at the files inside one of the state partitions. The files inside corresponds to the partition commit (file 01) in the _delta_log directory.
130130

131131
# COMMAND ----------
132132

133-
display(dbutils.fs.ls(delta_path + "/state=CA/"))
133+
display(dbutils.fs.ls(f"{delta_path}/state=CA/"))
134134

135135
# COMMAND ----------
136136

@@ -167,7 +167,7 @@
167167

168168
# COMMAND ----------
169169

170-
display(dbutils.fs.ls(delta_path + "/state=CA/"))
170+
display(dbutils.fs.ls(f"{delta_path}/state=CA/"))
171171

172172
# COMMAND ----------
173173

‎Apache-Spark-Programming-with-Databricks/Solutions/ASP 6 - Delta Lake/Labs/ASP 6.1L - Delta Lake Lab.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121

2222
# COMMAND ----------
2323

24-
sales_df = spark.read.parquet(datasets_dir + "/sales/sales.parquet/)
24+
sales_df = spark.read.parquet(datasets_dir + "/sales/sales.parquet")
2525
delta_sales_path = working_dir + "/delta-sales"
2626

2727
# COMMAND ----------

‎Apache-Spark-Programming-with-Databricks/Version Info.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
# MAGIC
1212
# MAGIC * Name: **Apache Spark Programming with Databricks**
1313
# MAGIC * Version: **2.0.1**
14-
# MAGIC * Built On: **Jan 28, 2022 at 20:38:07 UTC**
14+
# MAGIC * Built On: **Jan 28, 2022 at 22:18:24 UTC**
1515

1616
# COMMAND ----------
1717

0 commit comments

Comments
 (0)