Data Engineering Foundations: Pipelines, Warehouses, and ETL

The first time someone asks you to "set up a daily revenue dashboard from the production database," the honest answer is that you have no idea where to start. You can write the SQL. You've used a database. Maybe you've heard of dbt and Airflow. But the gap between "I can query data" and "I can build a pipeline that doesn't silently rot" is the gap this course is built to close. We walk through the data-engineering lifecycle from Reis and Housley's *Fundamentals of Data Engineering* — generation, storage, ingestion, transformation, serving — and the four pipeline archetypes most real analytical work collapses to: a daily batch ELT into a cloud warehouse, the same idea outsourced to managed connectors as the modern data stack, CDC and Kafka into a lakehouse for near-real-time analytics, and a streaming aggregator for the cases when seconds matter. For each, the lesson is the decisions, not the tool list: warehouse versus lake versus lakehouse, ETL versus ELT, dimensional model versus one-big-table, batch versus streaming. The dimensional canon — facts, dimensions, grain, slowly changing dimensions, conformed dimensions — comes from Kimball's *Data Warehouse Toolkit*. The storage-engine and stream-processing intuition comes from Kleppmann's *Designing Data-Intensive Applications*. None of it is vendor marketing. What separates a pipeline that lives in production from a script that ran once is reliability, so it isn't an afterthought here. You'll build idempotency into every write, add data-quality tests that fail loudly, define freshness SLOs that match what your consumers actually need, and learn an on-call posture that doesn't burn you out. By the end, given a new analytical request, you can defend a design on paper — sources, ingestion, storage, model, transformations, orchestration, monitoring — before writing a line of code. That's the point.