You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: dbt/dbt-udemy-course/models.md
+16-6Lines changed: 16 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -95,7 +95,7 @@ FROM
95
95
In seeds folder we can create static data by dropping csv files there. Goog for lookups data entered manually.
96
96
#### Sources
97
97
Add 1 more semantic layer in DBT, good for lineage.
98
-
Intead of reading from raw tables, can read from sources created in [sources.yml](./dbt_project/target/compiled/dbt_project/models/sources.yml) using the jinja macro for sources:
98
+
Intead of reading from raw tables, can read from sources created in [sources.yml](./dbt_project/models/sources.yml) using the jinja macro for sources:
99
99
```sql
100
100
SELECT*FROM
101
101
--AIRBNB.RAW.RAW_REVIEWS
@@ -107,7 +107,7 @@ dbt compile
107
107
```
108
108
109
109
### Source Freshness
110
-
Can set for each source the freshness to check last time the column was updated based on date column. Check [sources.yml](./dbt_project/target/compiled/dbt_project/models/sources.yml) to see how to implement, and then run the command:
110
+
Can set for each source the freshness to check last time the column was updated based on date column. Check [sources.yml](./dbt_project/models/sources.yml) to see how to implement, and then run the command:
111
111
```bash
112
112
dbt source freshness
113
113
```
@@ -119,7 +119,7 @@ Used to handle type 2 SCD.
119
119
* check: Any change in a set of columns (or all columns) will be picked up as an update.
120
120
121
121
#### Implementing
122
-
Snapshots live in their folder, like [scd_raw_listings.sql](./dbt_project/target/compiled/dbt_project/snapshots/scd_raw_listings.sql), and they ran over a different command:
122
+
Snapshots live in their folder, like [scd_raw_listings.sql](./dbt_project/snapshots/scd_raw_listings.sql), and they ran over a different command:
123
123
```bash
124
124
dbt snapshot
125
125
```
@@ -140,16 +140,26 @@ SELECT * FROM AIRBNB.DEV.SCD_RAW_LISTINGS WHERE ID=3176;
Create a file called [schema.yml](./dbt_project/target/compiled/dbt_project/models/schema.yml) in models. This is not mandatory, you can create sepatare files inside each model folder, as you wish to organize it. Test it with:
143
+
Create a file called [schema.yml](./dbt_project/models/schema.yml) in models. This is not mandatory, you can create sepatare files inside each model folder, as you wish to organize it. Test it with:
144
144
```bash
145
145
dbt test
146
146
```
147
147
Can also check the compiled queries in target folder to see the query created.
148
148
To check a failure, change one of the accepted values to a fake value and run the tests, it should break.
149
149
150
150
### Single Test
151
-
Live in the test folder and are queries, like [dim_listings_minimum_nights.sql](./dbt_project/target/compiled/dbt_project/tests/dim_listings_minimum_nights.sql).
151
+
Live in the test folder and are queries, like [dim_listings_minimum_nights.sql](./dbt_project/tests/dim_listings_minimum_nights.sql).
152
152
You can run as dbt test to execute all tests, or can execute just this test (same approach works to dbt run and other dbt commands).
153
153
```bash
154
154
dbt test --select dim_listings_cleansed
155
-
```
155
+
```
156
+
157
+
### Macros, Custom Tests and Packages
158
+
* Macros are jinja templates created in macro folder
159
+
* There are many built in macros
160
+
* Can be used in model definitions and tests
161
+
* A special macro called test can be used to create own generic tests
162
+
* Packages can be downloaded with more macros and tests
163
+
* in jinja there are control loops and so on, must the study jinja to do more advanced stuff
164
+
165
+
Create a new macro on macro folders as [no_nulls_in_columns.sql](./dbt_project/macros/no_nulls_in_columns.sql) and then call the macro in tests, creating a specific test like [no_nulls_in_dim_listings.sql](./dbt_project/tests/no_nulls_in_dim_listings.sql).
0 commit comments