Bayes' theorem, worked as counts instead of formulas

An everyday example

A spam filter flags an email as junk. The filter is described as catching 95% of spam and wrongly flagging only 1% of real mail. That sounds close to certain, but it leaves out a crucial number: out of everything in your inbox, what share is spam to begin with. If spam is only a small fraction of your total mail, the 1% mistake rate applied to the much larger pile of real mail can still produce more false flags than true ones, even with a filter that is right the vast majority of the time on each individual message.

The filter's accuracy describes the filter. The question you actually want answered, given a flag, how likely is this one message to really be spam, needs the base rate folded in too, and Bayes' theorem is the rule for folding it in correctly.

Bayes' 1763 essay

Thomas Bayes was a minister and mathematician who did not publish the essay that carries his name in his own lifetime; his friend Richard Price found it among his papers after his death and had it read to the Royal Society and printed in 1763. The essay works through a geometrical thought experiment involving balls rolled across a table, using it to address a much more general problem: given a number of observed outcomes, what can be said about the underlying probability that produced them. That is the reverse of the more familiar question, given a known probability, what outcomes should be expected, which is why the result is sometimes called inverse probability. The rule that follows from Bayes' reasoning is the one now written as Bayes' theorem: the probability of a cause, given an effect, depends on the probability of the effect given that cause, weighted by how probable the cause was before any effect was observed.

The frequency-format finding

Gerd Gigerenzer and Ulrich Hoffrage's 1995 paper tested a different question from Bayes' own: not whether the theorem is correct, which is settled mathematics, but what form a problem needs to take before ordinary people, including trained professionals, can reason through it correctly without being taught the formula. They took a set of classic problems, including medical screening scenarios, and presented them to different groups either as probabilities and percentages, the form textbooks usually use, or as natural frequencies: whole numbers of cases out of a stated population, such as "10 out of 1,000" instead of "a 1% base rate." Recasting a problem as natural frequencies produced a large, consistent jump in the share of people who reached the mathematically correct answer, without any of them being taught Bayes' rule as a formula. The reasoning the theorem describes was there all along; the percentage format was hiding it.

Does it replicate?

Replication grade: Strong: the theorem is proven; the natural-frequency teaching effect is confirmed by meta-analysis

Bayes' theorem itself is a proven mathematical result, not an empirical claim, so there is nothing to replicate about the theorem: given its assumptions, the conclusion follows by logic alone, the way a geometric proof does. What can be, and has been, tested empirically is Gigerenzer and Hoffrage's claim that natural frequencies help people reason correctly under it. McDowell and Jacobs's 2017 meta-analysis gathered the results of dozens of published studies that had tested some version of the frequency-versus-probability comparison since 1995, across different problems, populations and countries, and found the advantage of natural frequencies held up as a real and fairly consistent effect across that whole body of work, not a result specific to Gigerenzer and Hoffrage's own experiments.

The meta-analysis is also clear about the limits: switching to natural frequencies raises the share of people who reach the correct answer several-fold, but it does not make the problem easy, and most people still do not reach the correct answer even in the easier, natural-frequency format. The format helps a great deal; it does not, by itself, make Bayesian reasoning routine.

The tree, worked in full

The diagram on this page is a worked example, invented for this page, not a real screening programme. It splits 1,000 cases first by the base rate, then by the test result, so the final answer, what share of flagged cases are actually real, can be read straight off the counts rather than computed from a formula.

Try it: the calculator

Enter a base rate, a hit rate and a false-alarm rate and the calculator turns them into counts of cases, drawn as an icon array, with every step of the arithmetic shown.

How to catch it

The formula is not the point; the habit of converting to counts is.

• Whenever you meet a percentage-based claim about evidence and a rare event, convert it into a count out of a round population, such as 1,000 or 10,000 people.
• Work out how many of that population actually have the thing in question, using the base rate, before working out how the test performs on them.
• Split the population into real and clean cases first, then split each of those by what the test says about them; four numbers should come out of this, not one.
• The answer to "how likely is a positive result to be real" is the true positives divided by all positives, not the test's raw accuracy figure.
• Remember that a highly accurate test can still produce mostly false alarms if the thing it is testing for is rare enough; accuracy and base rate both matter, and neither alone tells the whole story.