Every hypothesis test follows the same six steps. The diagram at the top fills in as you go: learn it once, and it works for the z-test, the t-test, and the F-test.
Under H0: D ~ N(0, 0.69²), so z = D / SE0 ~ N(0, 1)
H0 days0
SD of D–
CLT says0.69
The bars (simulation) follow the curve (CLT).
Our sample: z = D / SE0 = 1.5 / 0.69 = 2.17
4H1 distribution
If B truly adds δ points, how does D move?
H1 is not one world but many: one for each true gain δ. Pick one. Its bell has the same width, shifted to δ.
Under H1 with gain δ: D ~ N(δ, SE²)
In a real plan δ is chosen before the data: the smallest gain that matters. Here δ = 1.5.
5α and decision
α: how often we accept a false alarm
Definitionα = P( reject H0 | H0 true )In words: if B is really no better, how often our rule still says “B is better”. The false alarm rate of the rule.We fix it before the data. α = 0.05: about 5 false alarms in 100 tests of a useless page.Common misreadings
✗ “α is the chance that H0 is true.” α assumes H0 is true; it never asks whether it is.
✗ “We rejected, so it is a false alarm with chance 5%.” That is P(H0 true | reject). It depends on how many ideas really work: in ⑥(c) it is 44%.
✗ “α is the p-value.” α is fixed before the data; the p-value is computed from the data. We compare the two.
α sets the cutoff on the H0 bell (rust). Decide: reject H0 if |z| ≥ cutoff, i.e. if p ≤ α.
Our D (try other samples)
D
z2.17
cutoff (α 0.05)±1.96
p-value0.030
share of ③ days this far out–
6Power
Power: how often we catch a real gain
Definitionpower(δ) = P( reject H0 | θB − θA = δ )In words: if B really adds δ points, how often our rule says “B is better”. The catch rate. Miss: β = 1 − power.Built from ④ and ⑤: the share of the H1 bell beyond the cutoff that α set on the H0 bell. Computed before the data, from δ, n, and α.Common misreadings
✗ “Power is the chance that H1 is true.” Power assumes the gain is δ; it never asks whether it is.
✗ “Power says how strongly our data support H1.” Power never looks at our data: it is 0.58 whether we see D = +2.0 or D = +0.2 (table below).
✗ “A test has one power.” Power changes with δ, with the visitors n, and with α. Move the sliders.
✗ “Not significant, so there is no gain.” At δ = 1.0 with 4,000 per page, power is 0.30: a real gain is missed 7 times in 10.
cutoff (points)±1.36
α0.05
power0.58
miss β0.42
Power does not look at our data
If the test had seen
p-value
power at δ = 1.5
D = +2.0
0.004
0.58
D = +1.5 (ours)
0.030
0.58
D = +0.2
0.77
0.58
The p-value moves with the data. Power does not.
What power is for
(a) Planning: how many visitors?To catch δ with power 0.80:6,700n per page ≈ 2p(1−p)(z1−α/2 + z0.80)² / δ²
(b) Reading "not significant"
(c) Many studies: how many "hits" are real?44%of the "significant" results are false alarms.
Recap · the whole flow
③ and ⑤ give the decision. ④ and ⑤ give the power. The same six steps, for every test below.
Same six steps, other tests
t-test · changes ③, the H0 distribution
Small sample: the bell gets fatter tails
t cutoff2.03
z cutoff1.96
tail
F-test · changes ② and ③: a new statistic, a new H0 distribution
F = signal ÷ noise
signal748
noise158
F4.72
p-value0.013
p-hacking · breaks ⑤: α is no longer 0.05
Try until it works
The button does nothing. Keep testing new metrics anyway.
Tries0
P(≥1 false alarm)0%
Peeking · breaks ⑤: α is no longer 0.05
Peek every day, stop when lucky
A/A tests: B is the same page as A, so H0 is true. 20 days, 200 new visitors per page per day.