python
youtube
Master data ingestion for data engineering with Python. Learn to tackle common pipeline failures like schema changes and API limits by adopting the mindset and practices of a senior platform engineer. This course covers essential techniques including extracting data from APIs, automatic schema management, incremental loading, and orchestrating scalable, automated workflows using modern tools.
Course developed by Alexey Grigorev & Adrian Brudaru.
💻 Code:
🏗️ dlthub.com provided a grant to make this course possible.
⭐️ Contents ⭐️
Alexey's part
0:00:00 1. Introduction
0:08:02 2. What is data ingestion
0:10:04 3. Extracting data: Data Streaming & Batching
0:14:00 4. Extracting data: Working with RestAPI
0:29:36 5. Normalizing data
0:43:41 6. Loading data into DuckDB
0:48:39 7. Dynamic schema management
0:56:26 8. What is next?
Adrian's part
0:56:36 1. Introduction
0:59:29 2. Overview
1:02:08 3. Extracting data with dlt: dlt RestAPI Client
1:08:05 4. dlt Resources
1:10:42 5. How to configure secrets
1:15:12 6. Normalizing data with dlt
1:24:09 7. Data Contracts
1:31:05 8. Alerting schema changes
1:33:56 9. Loading data with dlt
1:33:56 10. Write dispositions
1:37:34 11. Incremental loading
1:43:46 12. Loading data from SQL database to SQL database
1:47:46 13. Backfilling
1:50:42 14. SCD2
1:54:29 15. Performance tuning
2:03:12 16. Loading data to Data Lakes & Lakehouses & Catalogs
2:12:17 17. Loading data to Warehouses/MPPs,Staging
2:18:15 18. Deployment & orchestration
2:18:
|
2025/04/16
youtube
