Experimental Design, AB Testing, and Causal Thinking

Most experiments fail silently. You run a test, see a 2% lift, ship the change, and assume it worked. But the lift was noise. Or confounding (high-value users adopted first, they spend more anyway). Or peeking bias (you checked results halfway and got lucky). Or your metric was too noisy to detect a real effect. Months later, the feature underperforms and you're left wondering what went wrong. Experiments are the gold standard for causal reasoning in business—they're how you answer "does this actually work?" instead of guessing. But running a valid experiment requires more than intuition. It requires framing a precise hypothesis, choosing a metric that actually measures impact, calculating how many users you need, randomizing correctly, resisting the urge to peek, analyzing results rigorously, and distinguishing correlation from causation. This course teaches the full lifecycle. You'll learn the statistical and causal foundations—why randomization breaks confounding, what p-values actually mean, how power and sample size relate. You'll design experiments that work: selecting metrics, planning sample sizes, choosing randomization schemes, and catching data quality issues before they sink your results. You'll execute without breaking things: monitoring safely, avoiding peeking bias, setting stopping rules. You'll analyze with rigor: interpreting effect sizes, spotting false positives, investigating heterogeneous effects. And you'll know when randomization isn't possible—observational methods like propensity score matching and causal diagrams are powerful tools when ethics or logistics forbid experimentation. The course emphasizes practical decision-making over pure theory. You'll learn to recognize common pitfalls—regression to the mean, multiple testing inflation, selection bias—and fix them. You'll build a mental model for when each method applies. And you'll develop the habit of pre-registering your analysis plan before you run, so data doesn't tempt you into p-hacking. By the end, you'll have the skills to plan, run, and interpret experiments that actually tell you what works. Not intuition. Not hunches. Evidence.