Publishing v2.0.0

Author: SireInsectus

Apache-Spark-Programming-with-Databricks/ASP 0 - Course Agenda.py

@@ -0,0 +1,77 @@
+# Databricks notebook source
+# MAGIC %md-sandbox
+# MAGIC 
+# MAGIC <div style="text-align: center; line-height: 0; padding-top: 9px;">
+# MAGIC <img src="https://databricks.com/wp-content/uploads/2018/03/db-academy-rgb-1200px.png" alt="Databricks Learning" style="width: 600px">
+# MAGIC </div>
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Apache Spark Programming with Databricks
+# MAGIC ## Course Agenda
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Introductions
+# MAGIC * [ASP 1.1 - Databricks Platform]($./ASP 1 - Introductions/ASP 1.1 - Databricks Platform)
+# MAGIC * [ASP 1.1L - Explore Datasets Lab]($./ASP 1 - Introductions/Labs/ASP 1.1L - Explore Datasets Lab)
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Spark Core
+# MAGIC * [ASP 2.1 - Spark SQL]($./ASP 2 - Spark Core/ASP 2.1 - Spark SQL)
+# MAGIC * [ASP 2.2 - Reader & Writer]($./ASP 2 - Spark Core/ASP 2.2 - Reader & Writer)
+# MAGIC * [ASP 2.2L - Spark SQL Lab]($./ASP 2 - Spark Core/Labs/ASP 2.2L - Spark SQL Lab)
+# MAGIC * [ASP 2.3 - DataFrame & Column]($./ASP 2 - Spark Core/ASP 2.3 - DataFrame & Column)
+# MAGIC * [ASP 2.3L - Ingesting Data Lab]($./ASP 2 - Spark Core/Labs/ASP 2.3L - Ingesting Data Lab)
+# MAGIC * [ASP 2.4L - Purchase Revenues Lab]($./ASP 2 - Spark Core/Labs/ASP 2.4L - Purchase Revenues Lab)
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Functions
+# MAGIC * [ASP 3.1 - Aggregation]($./ASP 3 - Functions/ASP 3.1 - Aggregation)
+# MAGIC * [ASP 3.1L - Revenue by Traffic Lab]($./ASP 3 - Functions/Labs/ASP 3.1L - Revenue by Traffic Lab)
+# MAGIC * [ASP 3.2 - Datetimes]($./ASP 3 - Functions/ASP 3.2 - Datetimes)
+# MAGIC * [ASP 3.2L - Active Users Lab]($./ASP 3 - Functions/Labs/ASP 3.2L - Active Users Lab)
+# MAGIC * [ASP 3.3 - Complex Types]($./ASP 3 - Functions/ASP 3.3 - Complex Types)
+# MAGIC * [ASP 3.4 - Additional Functions]($./ASP 3 - Functions/ASP 3.4 - Additional Functions)
+# MAGIC * [ASP 3.4L - Abandoned Carts Lab]($./ASP 3 - Functions/Labs/ASP 3.4L - Abandoned Carts Lab)
+# MAGIC * [ASP 3.5 - UDFs]($./ASP 3 - Functions/ASP 3.5 - UDFs)
+# MAGIC * [ASP 3.5L - Sort Day Lab]($./ASP 3 - Functions/Labs/ASP 3.5L - Sort Day Lab)
+# MAGIC 
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Performance
+# MAGIC * [ASP 4.1 - Query Optimization]($./ASP 4 - Performance/ASP 4.1 - Query Optimization)
+# MAGIC * [ASP 4.2 - Partitioning]($./ASP 4 - Performance/ASP 4.2 - Partitioning)
+# MAGIC * [ASP 4.3L - De-Duping Data Lab]($./ASP 4 - Performance/Labs/ASP 4.3L - De-Duping Data Lab)
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Streaming
+# MAGIC * [ASP 5.1 - Streaming Query]($./ASP 5 - Streaming/ASP 5.1 - Streaming Query)
+# MAGIC * [ASP 5.1L - Coupon Sales Lab]($./ASP 5 - Streaming/Labs/ASP 5.1L - Coupon Sales Lab)
+# MAGIC * [ASP 5.2L - Hourly Activity by Traffic Lab]($./ASP 5 - Streaming/Labs/ASP 5.2L - Hourly Activity by Traffic Lab)
+# MAGIC * [ASP 5.3L - Activity by Traffic Lab]($./ASP 5 - Streaming/Labs/ASP 5.3L - Activity by Traffic Lab)
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Delta Lake
+# MAGIC * [ASP 6.1 - Delta Lake]($./ASP 6 - Delta Lake/ASP 6.1 - Delta Lake)
+# MAGIC * [ASP 6.1L - Delta Lake Lab]($./ASP 6 - Delta Lake/Labs/ASP 6.1L - Delta Lake Lab)
+
+# COMMAND ----------
+
+# MAGIC %md-sandbox
+# MAGIC &copy; 2022 Databricks, Inc. All rights reserved.<br/>
+# MAGIC Apache, Apache Spark, Spark and the Spark logo are trademarks of the <a href="https://www.apache.org/">Apache Software Foundation</a>.<br/>
+# MAGIC <br/>
+# MAGIC <a href="https://databricks.com/privacy-policy">Privacy Policy</a> | <a href="https://databricks.com/terms-of-use">Terms of Use</a> | <a href="https://help.databricks.com/">Support</a>

Apache-Spark-Programming-with-Databricks/ASP 1.2 - Databricks Platform.py renamed to Apache-Spark-Programming-with-Databricks/ASP 1 - Introductions/ASP 1.1 - Databricks Platform.py

@@ -36,7 +36,7 @@
 
 # COMMAND ----------
 
-# MAGIC %run ./Includes/Classroom-Setup
+# MAGIC %run ../Includes/Classroom-Setup
 
 # COMMAND ----------
 
@@ -187,13 +187,39 @@
 
 # COMMAND ----------
 
+# MAGIC %md ## Our First Table
+# MAGIC 
+# MAGIC Is located in the path identfied by **`eventsPath`** (a variable we created for you).
+# MAGIC 
+# MAGIC We can see those files by running the following cell
+
+# COMMAND ----------
+
+files = dbutils.fs.ls(eventsPath)
+display(files)
+
+# COMMAND ----------
+
+# MAGIC %md ## But, Wait!
+# MAGIC I cannot use variables in SQL commands.
+# MAGIC 
+# MAGIC With the following trick you can!
+# MAGIC 
+# MAGIC Declare the python variable as a variable in the spark context which SQL commands can access:
+
+# COMMAND ----------
+
+spark.sql(f"SET c.events_path = {eventsPath}")
+
+# COMMAND ----------
+
 # MAGIC %md ## Create table
 # MAGIC Run <a href="https://docs.databricks.com/spark/latest/spark-sql/language-manual/index.html#sql-reference" target="_blank">Databricks SQL Commands</a> to create a table named `events` using BedBricks event files on DBFS.
 
 # COMMAND ----------
 
 # MAGIC %sql
-# MAGIC CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "/mnt/training/ecommerce/events/events.parquet");
+# MAGIC CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "${c.events_path}");
 
 # COMMAND ----------
 
@@ -294,154 +320,12 @@
 
 # COMMAND ----------
 
-# MAGIC %md
-# MAGIC # Explore Datasets Lab
-# MAGIC 
-# MAGIC We will use tools introduced in this lesson to explore the datasets used in this course.
-# MAGIC 
-# MAGIC ### BedBricks Case Study
-# MAGIC This course uses a case study that explores clickstream data for the online mattress retailer, BedBricks. 
-# MAGIC You are an analyst at BedBricks working with the following datasets: `events`, `sales`, `users`, and `products`.
-# MAGIC 
-# MAGIC ##### Tasks
-# MAGIC 1. View data files in DBFS using magic commands
-# MAGIC 1. View data files in DBFS using dbutils
-# MAGIC 1. Create tables from files in DBFS
-# MAGIC 1. Execute SQL to answer questions on BedBricks datasets
-
-# COMMAND ----------
-
-# MAGIC %md ### 1. List data files in DBFS using magic commands
-# MAGIC Use a magic command to display files located in the DBFS directory: **`/mnt/training/ecommerce`**
-# MAGIC 
-# MAGIC <img src="https://files.training.databricks.com/images/icon_hint_32.png" alt="Hint"> You should see four items: `events`, `products`, `sales`, `users`
-
-# COMMAND ----------
-
-# TODO
-
-# COMMAND ----------
-
-# MAGIC %md ### 2. List data files in DBFS using dbutils
-# MAGIC - Use **`dbutils`** to get the files at the directory above and save it to the variable **`files`**
-# MAGIC - Use the Databricks display() function to display the contents in **`files`**
-# MAGIC 
-# MAGIC <img src="https://files.training.databricks.com/images/icon_hint_32.png" alt="Hint"> You should see four items: `events`, `items`, `sales`, `users`
-
-# COMMAND ----------
-
-# TODO
-files = dbutils.FILL_IN
-display(files)
-
-# COMMAND ----------
-
-# MAGIC %md ### 3. Create tables below from files in DBFS
-# MAGIC - Create `users` table using files at location `"/mnt/training/ecommerce/users/users.parquet"` 
-# MAGIC - Create `sales` table using files at location `"/mnt/training/ecommerce/sales/sales.parquet"` 
-# MAGIC - Create `products` table using files at location `"/mnt/training/ecommerce/products/products.parquet"` 
-# MAGIC 
-# MAGIC (We created `events` table earlier using files at location `"/mnt/training/ecommerce/events/events.parquet"`)
-
-# COMMAND ----------
-
-# TODO
-
-# COMMAND ----------
-
-# MAGIC %md Use the data tab of the workspace UI to confirm your tables were created.
-
-# COMMAND ----------
-
-# MAGIC %md ### 4. Execute SQL to explore BedBricks datasets
-# MAGIC Run SQL queries on the `products`, `sales`, and `events` tables to answer the following questions. 
-# MAGIC - What products are available for purchase at BedBricks?
-# MAGIC - What is the average purchase revenue for a transaction at BedBricks?
-# MAGIC - What types of events are recorded on the BedBricks website?
-# MAGIC 
-# MAGIC The schema of the relevant dataset is provided for each question in the cells below.
-
-# COMMAND ----------
-
-# MAGIC %md 
-# MAGIC #### Q1: What products are available for purchase at BedBricks?
-# MAGIC 
-# MAGIC The **`products`** dataset contains the ID, name, and price of products on the BedBricks retail site.
-# MAGIC 
-# MAGIC | field | type | description
-# MAGIC | --- | --- | --- |
-# MAGIC | item_id | string | unique item identifier |
-# MAGIC | name | string | item name in plain text |
-# MAGIC | price | double | price of item |
-# MAGIC 
-# MAGIC Execute a SQL query that selects all from the **`products`** table. 
-# MAGIC 
-# MAGIC <img src="https://files.training.databricks.com/images/icon_hint_32.png" alt="Hint"> You should see 12 products.
-
-# COMMAND ----------
-
-# TODO
-
-# COMMAND ----------
-
-# MAGIC %md #### Q2: What is the average purchase revenue for a transaction at BedBricks?
-# MAGIC 
-# MAGIC The **`sales`** dataset contains order information representing successfully processed sales. 
-# MAGIC Most fields correspond directly with fields from the clickstream data associated with a sale finalization event.
-# MAGIC 
-# MAGIC | field | type | description|
-# MAGIC | --- | --- | --- |
-# MAGIC | order_id | long | unique identifier |
-# MAGIC | email | string | the email address to which sales configuration was sent |
-# MAGIC | transaction_timestamp | long | timestamp at which the order was processed, recorded in milliseconds since epoch |
-# MAGIC | total_item_quantity | long | number of individual items in the order |
-# MAGIC | purchase_revenue_in_usd | double | total revenue from order |
-# MAGIC | unique_items | long | number of unique products in the order |
-# MAGIC | items | array | provided as a list of JSON data, which is interpreted by Spark as an array of structs |
-# MAGIC 
-# MAGIC Execute a SQL query that computes the average **`purchase_revenue_in_usd`** from the **`sales`** table.
-# MAGIC 
-# MAGIC <img src="https://files.training.databricks.com/images/icon_hint_32.png" alt="Hint"> The result should be `1042.79`.
-
-# COMMAND ----------
-
-# TODO
-
-# COMMAND ----------
-
-# MAGIC %md #### Q3: What types of events are recorded on the BedBricks website?
-# MAGIC 
-# MAGIC The **`events`** dataset contains two weeks worth of parsed JSON records, created by consuming updates to an operational database. 
-# MAGIC Records are received whenever: (1) a new user visits the site, (2) a user provides their email for the first time.
-# MAGIC 
-# MAGIC | field | type | description|
-# MAGIC | --- | --- | --- |
-# MAGIC | device | string | operating system of the user device |
-# MAGIC | user_id | string | unique identifier for user/session |
-# MAGIC | user_first_touch_timestamp | long | first time the user was seen in microseconds since epoch |
-# MAGIC | traffic_source | string | referral source |
-# MAGIC | geo (city, state) | struct | city and state information derived from IP address |
-# MAGIC | event_timestamp | long | event time recorded as microseconds since epoch |
-# MAGIC | event_previous_timestamp | long | time of previous event in microseconds since epoch |
-# MAGIC | event_name | string | name of events as registered in clickstream tracker |
-# MAGIC | items (item_id, item_name, price_in_usd, quantity, item_revenue in usd, coupon)| array | an array of structs for each unique item in the user’s cart |
-# MAGIC | ecommerce (total_item_quantity, unique_items, purchase_revenue_in_usd) | struct | purchase data (this field is only non-null in those events that correspond to a sales finalization) |
-# MAGIC 
-# MAGIC Execute a SQL query that selects distinct values in **`event_name`** from the **`events`** table
-# MAGIC 
-# MAGIC <img src="https://files.training.databricks.com/images/icon_hint_32.png" alt="Hint"> You should see 23 distinct **`event_name`** values.
-
-# COMMAND ----------
-
-# TODO
-
-# COMMAND ----------
-
 # MAGIC %md ### Clean up classroom
+# MAGIC Clean up any temp files, tables and databases created by this lesson
 
 # COMMAND ----------
 
-# MAGIC %run ./Includes/Classroom-Cleanup
+classroom_cleanup()
 
 # COMMAND ----------