One memorable case is put in place of data on how often it actually happens.

Personal experience is a sample of size one, and it was not selected at random but because it stuck in the memory. Competing explanations are almost always at hand: coincidence; regression to the mean (help is sought at the worst point, and after the worst point things usually get better); survivorship bias — the cases where it did not work simply never get told. Experience is good as a source of a hypothesis and as a description of a mechanism, but not as an estimate of frequency: it has no denominator. More on small samples and selection in chapter 09.

At work. “At my last job Kafka lost messages on us, so we are not taking it” — one incident three years old, on someone else's version and someone else's configuration, against the experience of thousands of installations. Same in support: three loud tickets in one morning look like an epidemic until you work out the share — 3 reports per 40,000 sessions, that is, inside the normal background. Before you change the architecture or roll back a feature on the strength of a story, ask for the denominator, the version, the configuration and the period; one vivid case sets a hypothesis, but the decision is made on the rate.

What to answer. How many such cases were there in total, and how many of them ended differently? Do we know the rate — or only the one outcome that stuck in the memory?

The target is painted after the shot: the hypothesis is fitted to a random cluster already spotted in the data.

Any sufficiently large data set contains clumps — that is a property of randomness, not a signal. The fallacy arises when the segment, the window or the metric is chosen after looking at the result, and the find is presented as a confirmed prediction. There is only one competing explanation here, and it is settled by arithmetic: multiple comparisons. Twenty metrics across five segments in four windows is hundreds of tests, and at a level of 0.05 dozens of “significant” ones will turn up on their own. A hypothesis born out of the data can only be tested on new data.

At work. The A/B test showed no effect on the full sample, so the analyst slices the data by segment and finds +8% conversion for Android in Germany in the second week. There were about a hundred tests — five false positives are expected even when the effect is exactly zero. The right move: write the find down as a hypothesis and check it with a pre-registered test on new traffic, rather than shipping a release off a subgroup. Same with alerts and dashboards: if you watch 300 charts, several metrics “degrade” every day. Separate exploratory and confirmatory analysis explicitly, down to separate documents.

What to answer. Were this segment, this window and this metric chosen before we looked at the data? How many slices were tested in total, and what did the rest show?

From the fact that two quantities move together it is concluded that one causes the other.

Moving together is compatible with at least six structures: X causes Y; the reverse direction, Y causes X; a common cause Z acts on both X and Y; the association is manufactured by selection — by how the data got into the sample; feedback; and finally plain coincidence. Correlation narrows the list of hypotheses but does not pick from it. To speak of a cause you need a mechanism, a temporal order and a comparison group — the causal protocol of chapter 09.

At work. The dashboard shows it: services with frequent deploys have more incidents. The conclusion “frequent releases break prod” turns the picture upside down. The common cause is criticality and activity: loaded, actively developed services get more changes, more traffic and better observability, which simply means more recorded incidents. The arrow may also point the other way: after an incident you fix and ship more often. Before you introduce a release window, look at change failure rate per deploy rather than incident count per service, and compare against services of comparable criticality.

What to answer. What could have acted on both the supposed cause and the outcome at once? And could the arrow be pointing the other way round?

Past unrecoverable investment is offered as a reason to continue, although the decision depends only on the future.

Money, time and reputation already spent will not come back under any choice — which means they are identical in both branches and cannot tell them apart. What has to be compared is future benefit, future cost and the best alternative. The past is relevant exactly to the extent that it created an asset, an obligation, accumulated knowledge or an exit cost. A related mechanism is escalation of commitment: after a public decision it is easier to put in more than to admit a mistake; see chapter 10.

At work. An internal platform has been written for two years by six engineers, and the service migration has not moved. The argument “we have put in so much, we cannot drop it now” replaces the real question: if the platform did not exist today, would we start it at the current price of an off-the-shelf solution and the current requirements? The same conversation applies to a legacy rewrite, to killing an experiment with two months of collected data, and to the home-grown scheduler. The cure is not willpower but process: stop conditions and budget are fixed before the start, a kill review sits in the calendar, and the decision to continue is not made by the author of the hypothesis.

What to answer. If we ran into this opportunity today, having put nothing in yet — would we start? Which costs are still ahead of us rather than behind?

From the fact that B happened after A it is concluded that A caused B.

Temporal order is necessary for a cause but not sufficient. Competing explanations: coincidence; a common cause that set off both events; a process that started earlier and surfaced with a delay; regression to the mean — the intervention is usually made at the peak of the problem, and after the peak things get better by themselves; and finally, what changed was the measurement, not the thing measured. The closer the events are in time, the more convincing the illusion and the more a comparison group is needed.

At work. A release went out at 14:05, at 14:12 p99 tripled, it was rolled back — things got better, incident closed with the cause “the release”. Check: did the chart start climbing before 14:05? Did a nightly batch start in that window, did a dependency degrade, did a config change? Did the rollback coincide with the end of the peak hour? A rollback changes the entire state of the system at once, so “it got better after the rollback” is exactly the same post hoc as the original conclusion. A cause counts as confirmed when there is a mechanism and a reproduction (the same query on a copy of the data, say), not just a coincidence of timestamps.

What to answer. What else changed in the same window? Had the metric already started climbing before the event — and is there a comparable group where the event did not happen?

A property of a group, computed as an average, is carried over to each of its members.

Group-level statistics and individual-level statistics are different quantities; the association between them can differ in strength and even in sign. The competing explanation for an aggregate association usually lies not in people's behaviour but in the composition of the groups and the weights of the subgroups; the extreme case is Simpson's paradox, where the trend inside every group is the opposite of the trend in the union (chapter 09). The mirror-image fallacy is called atomistic: an individual-level regularity is carried over to groups, ignoring coordination, specialisation and the division of work.

At work. Teams with high test coverage take prod down less often — from which it is concluded that every engineer who raises coverage on their own module will cut their own incidents. The unit of analysis has been swapped: what works at the team level is code review, CI, on-call and which services the team was handed, not coverage on its own. The same slip in a performance review (“the team has high throughput” does not make every individual's throughput high) and in SLOs: 99.9% availability for the whole service does not mean 99.9% for every customer — a single tenant can absorb the entire error budget. Declare the unit of analysis — request, session, user, team — before you count, and do not mix levels inside one conclusion.

What to answer. What is the unit of analysis here — the group or the individual? Is the spread inside the groups no larger than the difference between them?

A past decision is judged with knowledge the person deciding did not have at the time.

What gets swapped is the information state: the outcome is already known, and the signals that led to it look singled out in hindsight, while the dozens of equally weighty signals that were drowning in the noise back then are invisible. The competing explanation is not “the person did not think it through” but “at that moment the information was indistinguishable from the background”. The cognitive lining of the fallacy is hindsight bias, the “I knew it all along” effect (chapter 10). An honest assessment asks what was available at the moment of the decision; the historian's fallacy asks why nobody guessed what we know now.

At work. In the post-mortem someone says: “but the alert fired at 3:40, why did the on-call not react”. At 3:40 that alert was one of seventeen, fourteen of which are noise every night; it became the salient one only after we learned the answer. The difference from an honest walkthrough is fundamental: a post-mortem reconstructs what information was available and what made it hard to see, and then adds a signal, a runbook or automation — that is, it changes the system; the historian's fallacy looks for who failed to guess — and teaches people to put fewer details into their reports. Hence the blameless review rule, a timeline built strictly from the data available at the time, and a decision log with the reasoning written down before the result.

What to answer. What did the person know at that minute — and what did it look like among the rest of the noise? Are we judging the decision, or the outcome we already know?

A comparative claim is presented without the second term of the comparison, without a baseline and without the conditions of measurement.

“Faster”, “cheaper”, “40% more efficient” are two-place relations. Until it is said than what, under which conditions and by which metric, they cannot be checked, and the listener fills in whichever version suits the speaker best. The competing explanations hide precisely in what is left unsaid: the baseline was chosen for convenience, the denominator was never named, a relative effect was shown instead of an absolute one, the window was picked to fit. Related moves are the denominator fallacy and the substitution of relative risk for absolute risk (chapter 09).

At work. “The new service is three times faster” — faster than what: than the old version on a cold cache, on a different machine, under a different load profile? Which metric — the mean or p99, on what share of requests, how long did the run last? A classic of the post-release report: “50% more errors” with no request count — with traffic doubled the error rate has actually gone down. Demand five things: the numerator, the denominator, the comparison baseline, the window, and absolute values with their spread. In benchmarks add the configuration of both sides — otherwise what is being compared is not the systems but the effort of whoever tuned them.

What to answer. Faster, cheaper, better — than what exactly, by which metric and under which conditions? What are the absolute numbers here, and the denominator?

An independent random process is expected to compensate for the run that came before.

If the trials are independent and the probability is constant, past outcomes do not change the next one: a coin has no memory. Frequencies even out not by paying back a debt but by diluting the run in the mass of new trials. The mirror-image extreme is belief in the “hot hand”, where a run is taken as proof of a regularity. Both versions place the same undeclared bet on a model, which is why the first thing to check is not the length of the run but the assumption of independence itself: a real process can be autocorrelated, worn out, or dependent on a shared resource.

At work. “Four quiet nights on call — tonight it is bound to blow up”, and its mirror image, “ten green builds in a row, the pipeline is stable, we can drop the checks”. Both conclusions take independence on faith. The opposite move is more useful: if a flaky test failed in 4 cases out of 100 and today it failed three times in an hour, the probability of that under independence is negligible — so the trials are dependent, and you should be looking for a shared resource, a neighbouring deploy or connection pool exhaustion. Same with retries: a run of timeouts is usually not bad luck but saturation. Work out the expected run length at your base rate and compare it with the observed one.

What to answer. Are the trials here really independent? If they are, the past does not affect the next outcome; if they are not, the run tells you the process has changed, not that chance owes you anything.

Only the confirming part of the available data is shown, and the equally weighty part that refutes it is passed over.

The selection makes the sample unrepresentative not by accident but on the criterion of agreeing with the thesis, so the conclusion is fixed by the procedure rather than by the data. The competing explanation is always the same one and always off-screen: the observations that were not selected. Particular forms are a convenient time window, showing only the cases that survived (survivorship bias), and p-hacking, where the flexibility of the analysis is used until the required result comes out. It differs from honest selection in that the criterion was formulated after looking at the data and was never disclosed.

At work. In the release report: conversion +12% over a chosen nine days. Off-screen: the window starts right after a disastrous Monday, a discount mailing landed inside it, retention and refunds are not shown, and out of twenty metrics two are quoted. Same in benchmarks: our system on our own load profile, theirs on the default config and without warm-up. The defence is procedural: the primary metric, the window, the guardrail metrics and the stopping rule are fixed before the experiment, negative results are published on the same footing as positive ones, and the raw data sits next to the conclusion.

What to answer. What data of the same quality did we not show? Were the metric and the window chosen before we saw the result?

From the fact that an event happened it is concluded that it could not have failed to happen.

The outcome that materialised looks like the only possible one because the alternative branches are no longer observable: we see a single trajectory and build a coherent explanation to fit it. Two mechanisms are at work — hindsight bias, which rewrites the memory of what was expected before, and the narrative pull towards explaining an outcome by a chain of steps that “led” to it. The competing explanation is simple: the probability was middling, and one branch out of several is the one that came true. The test is what estimates were given before the event, not how smoothly the story reads afterwards.

At work. After an incident the walkthrough sounds like “that architecture was bound to fall over sooner or later”. Maybe; but the same logic explains the three preceding years, when it did not fall over, just as smoothly. The damage is twofold: the team rewrites something that was not the main contributing factor, and at the same time loses the ability to tell genuinely high risk from an after-the-fact explanation. The antidote is to write forecasts down with probabilities in advance: a decision log when the choice is made, a premortem before launch (“imagine this fell over six months from now, name the reasons”), and after the incident compare against the records. Then you can see whether the failure was predicted or the inevitability was painted in afterwards.

What to answer. Do we have a record showing that before the event we thought this outcome likely? Which branches were possible then, and what exactly closed them off?

For an outcome with several interacting causes one is named, and the explanation is treated as finished.

Complex outcomes usually come out of a coincidence of conditions, none of which is sufficient on its own. Naming one factor does not rule out the rest — and, worse, it stops the search. Here the competing explanations are not alternatives but additions: part of the contribution belongs to a common cause, part to selection, part to coincidence. A distinct variety is reducing a multi-factor outcome to a culprit: that shifts the conversation from the mechanism to the person and usually ends the walkthrough before it has begun.

At work. “The cause of the incident was a heavy SQL query.” Four more conditions usually lie right next to it: the index did not make it over with the migration, retries without backoff, a connection pool of 20, and an alert with a threshold above the real SLO. The incident happened because all five coincided; removing one lowers the probability but does not close the failure class. That is why a post-mortem records contributing factors and chains of mechanisms rather than a single root cause: “five whys” down one branch gives a narrow fix and a false sense of closure. The same slip at a different scale — “the project failed because of the architecture”, with demand, deadlines, budget, dependencies and governance left off-screen.

What to answer. If this factor had been absent — would the outcome definitely not have happened? What other conditions had to coincide for it to come out exactly this way?