Probability, sampling, confidence intervals, hypothesis tests and A/B testing

Probability fundamentals

Probability P(A) ranges from 0 (impossible) to 1 (certain).

• Complement: P(not A) = 1 − P(A).
• Addition: P(A or B) = P(A) + P(B) − P(A and B).
• Independence: P(A and B) = P(A) × P(B) only if A and B are independent.
• Conditional probability: P(A | B) = P(A and B) ÷ P(B) — "the probability of A given B".

Example: 5% of transactions are fraud-flagged (F). A rule flags 90% of fraud and 2% of legitimate transactions. Of all flagged transactions, how many are actually fraud?

P(flag) = 0.9 × 0.05 + 0.02 × 0.95 = 0.045 + 0.019 = 0.064 P(F | flag) = 0.045 ÷ 0.064 ≈ 0.70

Even a "90% accurate" rule produces 30% false alarms when fraud is rare — Bayes' rule in practice, and why base rates matter.

Sampling and the central limit theorem

We usually observe a sample and want to learn about a population (all customers, all future visitors). Different samples give different estimates — sampling error. Good samples are random and representative; convenience samples (only customers who answer surveys) can be biased, and no formula fixes bias.

The central limit theorem: the distribution of a sample mean (or proportion) is approximately normal for large enough samples, even if the underlying data are skewed. Its spread is the standard error: SE = SD ÷ √n. Larger samples → smaller SE → more precise estimates (but halving the error needs 4× the data).

Confidence intervals

A 95% confidence interval: estimate ± 1.96 × SE. For a proportion p̂ from n observations, SE = √(p̂(1 − p̂) ÷ n).

Interpretation: if we repeated the sampling process many times, 95% of intervals built this way would contain the true value. It is not "95% probability the true value is in this particular interval" (in strict frequentist terms) — but it is a fair practical summary of uncertainty.

The Orion Shop A/B test

Orion Shop tested a new one-page checkout (B) against the existing checkout (A) from 1 September to 31 October 2025. Users were randomly assigned. Metric: checkout conversion = purchases ÷ sessions that started checkout.

import pandas as pd, numpy as np
from math import sqrt
from scipy import stats
s = pd.read_csv("shop/sessions.csv")
co = s[s["started_checkout"] == 1]
g = co.groupby("checkout_variant")["purchased"].agg(["sum", "count"])
g["rate"] = g["sum"] / g["count"]
print(g)
pA, nA = g.loc["A", "rate"], g.loc["A", "count"]
pB, nB = g.loc["B", "rate"], g.loc["B", "count"]
for name, p, n in [("A", pA, nA), ("B", pB, nB)]:
    se = sqrt(p * (1 - p) / n)
    print(f"{name}: {p:.3f}  95% CI [{p - 1.96*se:.3f}, {p + 1.96*se:.3f}]")
diff = pB - pA
se_diff = sqrt(pA * (1 - pA) / nA + pB * (1 - pB) / nB)
print(f"Difference B − A: {diff:.3f}  95% CI [{diff - 1.96*se_diff:.3f}, {diff + 1.96*se_diff:.3f}]")

                  sum  count      rate
checkout_variant                      
A                 768   1325  0.579623
B                 860   1332  0.645646
A: 0.580  95% CI [0.553, 0.606]
B: 0.646  95% CI [0.620, 0.671]
Difference B − A: 0.066  95% CI [0.029, 0.103]

Variant B converts about 6–7 percentage points better, and the confidence interval for the difference excludes zero.

Hypothesis testing and p-values

1. Null hypothesis H₀: no difference (p_B = p_A). Alternative H₁: there is a difference.
2. Compute a test statistic measuring how far the data are from H₀.
3. p-value: the probability of observing a difference at least this large if H₀ were true.
4. If p < α (commonly 0.05), reject H₀ — the result is statistically significant.

p_pool = (g["sum"].sum()) / (g["count"].sum())
se_pool = sqrt(p_pool * (1 - p_pool) * (1 / nA + 1 / nB))
z = diff / se_pool
p_value = 2 * (1 - stats.norm.cdf(abs(z)))
print(f"z = {z:.2f}, two-sided p-value = {p_value:.4f}")
chi2, p_chi, _, _ = stats.chi2_contingency(np.array([[g.loc['A','sum'], nA - g.loc['A','sum']], [g.loc['B','sum'], nB - g.loc['B','sum']]]), correction=False)
print(f"chi-square check: p = {p_chi:.4f}")

z = 3.49, two-sided p-value = 0.0005
chi-square check: p = 0.0005

The p-value is far below 0.05: a difference this large would be very unlikely if both checkouts truly converted equally.

What a p-value is not

• It is not the probability that H₀ is true.
• It is not the size or importance of the effect — a tiny effect can be "significant" with a huge sample.
• p = 0.06 is not "no effect"; it is weak evidence. Report the effect size with its confidence interval.

Type I error (false positive): rejecting a true H₀ — probability α. Type II error (false negative): missing a real effect. Power = 1 − P(Type II).

A/B test design and pitfalls

1. Decide the metric and minimum detectable effect before starting. Sample size per group for proportions ≈ 16 × p(1 − p) ÷ δ² for 80% power at α = 0.05 (δ = absolute difference you need to detect). For p = 0.58 and δ = 0.05: 16 × 0.58 × 0.42 ÷ 0.0025 ≈ 1,559 checkout sessions per variant. This test had about 1,330 per variant — slightly underpowered for a 5-point effect, but the observed effect was larger.
2. Randomise the unit (users, not sessions, so a user always sees the same variant) — Orion assigned users.
3. Don't peek and stop early when p dips below 0.05; it inflates false positives. Fix the duration (full weeks to cover weekday effects).
4. Check sample ratio: a 50/50 split should look 50/50; a mismatch signals a broken assignment.
5. Guardrail metrics: B must not hurt revenue per session, refunds or support contacts.
6. Novelty effects: users may react to newness; watch results over time.
7. Multiple comparisons: testing 20 metrics at α = 0.05 yields about one false "win" by chance.

Practice

Summary

• Conditional probability and base rates explain false alarms.
• Sampling error shrinks with √n; bias does not.
• CI = estimate ± 1.96 × SE; report effect sizes with intervals.
• p-value = probability of data this extreme if H₀ is true — not the probability H₀ is true.
• Good A/B tests fix metric, sample size and duration in advance, randomise properly and watch guardrails.