Academic and Engineering Walkthrough
After the incident, the team can build a valid argument from incomplete data,
because memory more easily retrieves vivid confirmations of a favored hypothesis.
Fixing the form does not correct evidence gathering. And conversely, the presence of bias does
not prove the falsity of a specific conclusion.
flowchart LR
A["Ожидание"] --> B["Направляет внимание"]
B --> C["Меняет сбор данных"]
C --> D["Формирует интерпретацию"]
D --> E["Решение"]
E --> F["Избирательная обратная связь"]
F --> A
G["Независимая оценка"] -. разрывает цикл .-> C
H["Заранее заданный критерий"] -. разрывает цикл .-> E
Chapter Rule
Do not diagnose a person by the name of a bias. Design a process in which
competing hypotheses, independent evaluations, and feedback become cheaper.
1. Bias, fallacy and noise
| Concept |
Level |
Example |
| Fallacy |
argument or dialogue structure |
asserting the consequent |
| Bias |
systematic direction of evaluation error |
anchoring |
| Noise |
random spread of judgments |
different evaluations of the same case |
One bias can lead to different fallacies. Confirmation bias leads to
cherry-picking, suppression of alternatives, and asymmetric evidence standards.
2. Be Careful with the “Fast and Slow System” Model
The distinction between fast automatic and slow controlled processes is useful,
but does not map to two literal brain departments. Expert intuition can
be fast and accurate in a stable environment with frequent feedback; prolonged
reasoning may rationalize a desired conclusion.
The question is not “did we think slowly?”, but:
- Is the environment predictable?
- Was there quality feedback?
- Is there an external criterion?
- Does the procedure allow for refutation?
3. Confirming Distortion
We are lighter:
- look for confirmations;
- interpret ambiguous in favor of the position;
- remember the consistent;
- check someone else’s hypothesis more strictly.
Engineering example:
Deciding that the incident cause is the network, the team looks for timeout and ignores
messages about schema mismatch.
Counteraction
Для каждой гипотезы:
что мы ожидаем увидеть, если она верна?
что ожидаем, если неверна?
какой результат заставит отказаться?
The best test distinguishes hypotheses rather than merely being compatible with the favored one.
4. Motivated Reasoning
The desired solution affects the proof standard:
- for a favorable outcome, one case is sufficient;
- for an unpleasant one, they require a perfect RCT;
- ambiguity is interpreted asymmetrically.
Sign:
What evidence would we require if the conclusion were the opposite?
Debiasing: fix in advance the decision criterion and the check owner.
5. Bias blind spot
We see distortions in others and consider our own introspection as proof of objectivity.
Я выбрал архитектуру по фактам; несогласные просто боятся нового.
Wanting to be objective does not replace procedures. ADRs, alternatives,
predictions, and a subsequent decision review are needed.
6. Accessibility
Probability is estimated by the ease of recall.
After a loud leak, it is precisely its scenario that is being reassessed; quiet, frequent access rights errors remain unnoticed.
Repair
- basic frequencies;
- incident registry;
- threat model;
- reference class;
- data instead of media brightness.
The availability heuristic is sometimes useful: an easily recallable event may be
frequent. A fallacy — not checking why it is available.
7. Anchoring
The first digit or frame shifts the subsequent estimates:
«Проект займёт год». Независимые оценки после этого группируются вокруг года.
Repair
- Gather independent estimates before discussion.
- Evaluate components and reference class.
- Reveal assumptions, do not average numbers without a model.
- Use ranges and distributions.
Display the anchor and then ask to “ignore it” — a weak defense.
8. Representativeness
The case seems similar to the prototype, and similarity substitutes probability.
Кандидат говорит как сильный архитектор.
Следовательно, вероятно, он сильный архитектор.
Basic frequency, feature validity, and structured check are needed.
Conjunction fallacy and base rate neglect often stem from this.
9. Framing
Same outcomes cause different decisions depending on the description:
90% релизов без инцидента
10% релизов с инцидентом
Repair
Show:
- gain and loss frame;
- absolute numbers;
- single horizon;
- default alternative;
- consequences of action and inaction.
Frame is inevitable. The task is to make the choice robust against reasonable reformulations.
10. Loss Aversion
Losing something existing feels stronger than gaining an equivalent. This can
support legacy even with a beneficial migration or, conversely, excessively
accelerate measures after recent damage.
Don’t reduce the solution to emotion:
expected value
tail risk
необратимость
ликвидность ресурсов
распределение ущерба между людьми
11. Status quo and default effect
The current option seems neutral, although continuation is also an action.
Мы не решили хранить данные бессрочно; просто не добавили удаление.
The absence of a deletion mechanism effectively chooses indefinite storage.
Make the default explicit and justify it.
12. Endowment effect
Ownership of a solution is valued higher than alternatives due to ownership:
Наш самописный scheduler гибче.
Ask:
If it didn’t exist today, would we choose to build it under current
requirements and costs?
13. Sunk cost
Sunk irreversible costs are used as a basis to continue:
Мы потратили два года, поэтому обязаны завершить.
The solution should compare future options:
будущая ценность - будущая стоимость - риск
Past costs are important only where they created a future asset, obligation,
training, or exit price.
14. Commitment Escalation
Once a person has made a public decision, it becomes difficult for them to admit a mistake; they invest further to justify the past.
Process:
- predefine stop conditions;
- separate the hypothesis author from the continuation decision owner;
- reward early termination of a bad experiment;
- conduct kill reviews on schedule.
15. Hindsight bias
After the event, it seems predictable:
It was obvious that the database would not hold up.
This breaks learning: the real uncertainty of the past disappears.
Store:
- forecasts until resolution;
- ranges;
- known data at that time;
- alternatives;
- confidence.
Postmortem compares the decision with the information available at that time, not just with
the outcome.
16. Outcome bias
The quality of the solution is evaluated based on one result:
Рискованный deploy прошёл — решение было хорошим.
Осторожный deploy упал — решение было плохим.
A good solution may yield a bad outcome in a probabilistic world. Check
the process, calibration, and repeatability.
17. Overconfidence
Manifestations:
- too narrow intervals;
- overestimated accuracy;
- low probability of self-failure;
- belief that we understand more of the mechanism than we can explain.
Calibration
Collect many predictions with probabilities. Among events rated at 70%, approximately
70% should occur. Without feedback, confidence is not learned.
18. Illusion of explanatory depth
It seems that we understand the system until we have to explain it step by step:
How exactly does the browser obtain the page after entering the URL?
Technique:
- ask to assess understanding;
- explain the mechanism step by step;
- point out gaps;
- assess again.
In architecture, require sequence diagram and failure path, not just familiar
words.
19. Planning fallacy
The deadline is estimated according to the internal ideal scenario and ignores the distribution of similar projects.
Inside view
список задач + оптимистичные длительности
Outside view
как распределялись сроки у похожих инициатив?
Use reference class forecasting, Monte Carlo by cycle time, buffers and
explicit dependencies.
20. Fundamental attribution error
The behavior of others is explained by their character, one’s own — by circumstances:
Он опоздал, потому что безответственный.
Я опоздал, потому что dependency задержала ответ.
In organizations, system constraints easily turn into a moral evaluation
of a person.
Repair:
- the same set of questions to oneself and others;
- system data;
- the ability to explain the context;
- separation of behavior, effect, and personality.
21. Halo and horns effect
One strong property colors the others:
Кандидат блестяще решил алгоритм → вероятно, хороший лидер.
Один плохой ответ → слабый инженер вообще.
Structured interview uses independent criteria and separate evaluations
before general discussion.
22. Authority bias and social proof
The senior participant position becomes an anchor; the rest do not present
alternatives.
Process:
- the leader speaks last;
- written pre-reads;
- independent voting;
- appointed challenger;
- anonymous risk collection, when psychological safety is needed.
23. Groupthink
The pursuit of consistency suppresses doubts:
- illusion of consensus;
- self-censorship;
- pressure on dissenters;
- rationalization;
- information “wardens”.
Not every consent is groupthink. The diagnosis requires a process that blocks
critical checking.
People repeat a position, considering public repetition as independent confirmation.
In reality, everyone refers to a single source.
Build the provenance graph:
10 статей → 2 обзора → 1 исходный неподтверждённый отчёт
The number of references is not equal to the number of independent testimonies.
25. Repetition Effect
A familiar statement seems more truthful. Repetition is useful for learning and dangerous if the source is of low quality.
Protection:
- visible sources;
- mark what has already been checked;
- do not repeat a myth in the headline without immediate correction;
- separate recognizability from evidence.
26. Dunning–Kruger: a careful caveat
The popular version “fools are always confident, wise doubt” — a caricature.
The original effect is related to the fact that low skill worsens both performance and
self-esteem, and the statistical picture is sensitive to measurement and regression to
the mean.
Practical takeaway is not to “diagnose a colleague”, but:
- provide an external criterion;
- teach self-checking;
- ensure frequent feedback;
- calibrate confidence on a series of tasks.
27. Curse of knowledge
The expert underestimates how much context is missing for the beginner:
«Просто подними окружение обычным способом».
Solutions:
- usability test of the documentation;
- onboarding a new person;
- glossary;
- explicit prerequisites;
- check “can it be done without the author”.
28. Design of the environment is stronger than the style guide
A list of 200 biases helps little at the moment of decision. More effective are built-in
constraints:
| Risk |
Process |
| Anchoring |
independent estimates before discussion |
| Confirmation |
competing hypotheses and falsification test |
| Hindsight |
decision log with forecast before event |
| Sunk cost |
predefined stop conditions |
| Groupthink |
leader speaks last, challenger |
| Planning fallacy |
reference class and Monte Carlo |
| Outcome bias |
review process separately from outcome |
| Availability |
base rates and registry |
| Halo |
independent scorecard criteria |
| Framing |
symmetric formulations of gain/loss |
29. Premortem
Imagine that the solution failed after a year:
Что именно произошло?
Какой ранний сигнал мы проигнорировали?
Какая предпосылка оказалась ложной?
Какой контроль отсутствовал?
Premortem legitimizes criticism before a public commitment. But it can also
create accessibility of scary scenarios; after generation, assess the probabilities.
30. Adversarial collaboration
Parties with different beliefs will agree in advance:
- precise question;
- data;
- criterion;
- model;
- possible outcomes;
- what will change each party’s opinion.
This is better than an infinite exchange of articles after the result.
31. Checklist
- Has the belief formation mechanism been confused with an argument fallacy?
- Are we seeking data that could disprove a favorite hypothesis?
- Is the standard the same for convenient and inconvenient conclusions?
- Is there a base rate and reference class?
- Were estimates made independently before the anchor?
- Are gain and loss frames shown?
- Are future costs separated from sunk cost?
- Is the decision quality separated from one outcome?
- Have predictions made before the event been preserved?
- Is there external calibration of confidence?
- Is a person not replacing a systematic explanation?
- Are recurring sources truly independent?
- Does the process allow safely disagreeing?
32. Tasks
- After one serious incident, the team buys expensive protection specifically from it. What data is needed?
- The architect first mentions the term “three months,” then the team estimates at 10–14 weeks. How to change the process?
- An experiment had a bad outcome, although the decision rule was followed. How to conduct a review without outcome bias?
- The team continues the product after three unsuccessful pilots because it has invested a year. What future parameters to compare?
- During an interview, one brilliant system design overshadowed weak collaboration data. How to redesign the scorecard?
- All ten articles refer to one report. Draw the provenance and recalculate the independent evidence.
Sources
- Daniel Kahneman. Thinking, Fast and Slow — useful map taking into account
subsequent replication and methodological discussions.
- Daniel Kahneman, Olivier Sibony, Cass Sunstein. Noise.
- Philip Tetlock, Dan Gardner. Superforecasting.
- Richard Thaler, Cass Sunstein. Nudge.
- Gary Klein. Sources of Power — conditions for quality expert intuition.
- NIST.
Towards a Standard for Identifying and Managing Bias in Artificial
Intelligence.
What’s next
We have collected fallacies of form, language, data, and thinking process. The next chapter
connects them into the scientific and engineering method: hypotheses, induction, abduction,
experiment, Mill’s methods, SDD, RFC, and incident investigation.
Scientific and Engineering Argumentation
