Bayesian Analytics
Ask the data how strongly it believes
Bayesian analysis combines prior belief with observed data to produce a posterior distribution over an unknown quantity. Instead of 'reject or not', it answers 'how likely is each effect size'.
What you will learn
- Explain prior, likelihood, and posterior in plain terms
- Distinguish a credible interval from a confidence interval
- Use Bayes for metric estimation and experiment interpretation
What is this?
Bayesian analysis treats unknown quantities (a conversion rate, an effect size) as distributions. You start with a prior reflecting what you already believe, update it with the likelihood of what you observed, and end with a posterior distribution. Claims come out as probabilities — 'there is a 95% probability the true rate is between 4% and 6%'.
Why do you need it?
- It gives a more honest-sounding answer to the questions stakeholders ask ('what are the odds?' ).
- Prior information can stabilize small samples and multiple comparisons.
- Posterior distributions compose naturally into business quantities like 'expected uplift in dollars'.
Mental model
A juror's growing certainty with each witness
Bayesian updating
The juror begins with prior beliefs, updates after each piece of evidence, and ends with a posterior — exactly the Bayes loop.
How it works
- Pick a prior distribution for the unknown rate or effect (often informed by history).
- Write the likelihood of your data given an assumed value.
- Combine prior × likelihood and normalize to get the posterior distribution.
- Summarize the posterior with a credible interval and probabilities of interest.
Prior belief narrows into a posterior after data is absorbed.
In production
- Experiment reviews present 'probability the variant beats control' plus a credible interval on lift.
- Early-stage products use weakly informative priors to get usable estimates from tiny samples.
- Anomaly monitoring uses posterior probability of a change rather than a p-value
Best practices
- State the prior explicitly and test how sensitive conclusions are to it.
- Report the full posterior summary (median and credible interval), not just a threshold.
- Pair Bayesian estimation with the same guard rails — priors can hide your assumptions from stakeholders.
Common mistakes
- Reading a 95% credible interval as a confidence interval (they answer different questions).
- Letting a strong prior dominate small samples silently.
- Reporting 'the probability of a difference' without the distribution of its size.
Debugging tips
- If the posterior barely moves with a large sample, the prior is too strong — soften it.
- If credible intervals look impossibly tight, check for a forgotten prior that is doing the work.
Interview checkpoint
Quick quiz
In Bayes' rule, the posterior combines
A 95% credible interval is a statement about