Quality
and Survivorship Bias: When Your Organization Studies Only Its Successes
— and the Lessons You Drew From Winning Become the Reason You Start
Losing
You’ve seen the diagram. A World War II bomber, its fuselage
pockmarked with red dots showing where returning aircraft took enemy
fire. The obvious conclusion: reinforce those areas. The brilliant
insight from Abraham Wald: reinforce everywhere else. The bullet holes
you’re looking at are on the planes that survived. The planes that
didn’t come back took hits in the places where you see no damage on the
survivors.
That’s survivorship bias. And it’s not just a historical curiosity
from military statistics. It’s the invisible hand shaping your quality
strategy right now.
Every day, quality organizations study their successes, benchmark
their best-performing lines, analyze their top suppliers, and
reverse-engineer their most profitable product launches. They build
playbooks from winners. They create best practices from champions. They
codify what worked.
And they systematically ignore everything that failed, disappeared,
or never made it past the first audit.
The result is a quality system built on half the data — the
flattering half — and an organization that confuses survival with
excellence.
The Mechanics of
Survivorship Bias in Quality
Survivorship bias enters your quality system through at least five
doors. Let’s walk through each one.
1. Best-Practice Codification
Your organization identifies its best-performing plant, line, or
shift. A team of engineers and quality managers visits. They document
everything: the standard work, the visual management boards, the team
huddles, the poka-yoke devices. They return to headquarters with a
120-page best-practice manual and roll it out across every facility.
Here’s what they missed.
The best-performing line might be winning despite certain
practices, not because of them. Maybe it has the newest equipment. Maybe
it runs the simplest product mix. Maybe it’s staffed by a team that has
worked together for eight years and has developed an intuitive
understanding of the process that no manual can capture. The practices
you observed and documented may be correlated with the success but
causally irrelevant — or even slightly detrimental, masked by other
advantages.
Meanwhile, the three lines that were shut down last year — the ones
that had similar practices but failed — are invisible. Nobody studies
dead lines. Nobody visits closed plants. Nobody reverse-engineers the
quality system that produced 14% scrap before the plug was pulled.
You end up with a best-practice manual that captures what the
survivors happened to be doing, not what actually made them survive.
2. Supplier Benchmarking
You rank your suppliers by quality performance. The top 20% get gold
status. You study their quality management systems, their inspection
protocols, their continuous improvement programs. You invite them to
present at your annual supplier conference. You hold them up as models
for the rest.
But consider: the suppliers who went bankrupt two years ago after a
catastrophic quality failure — did you study them? The supplier you
dropped because their parts caused a field failure that cost you €2
million — did you send a team to understand what went wrong in
their system before the relationship ended?
The suppliers you benchmark are the ones who survived your selection
process, your audits, and your market’s demands. They’re the bombers
that came back with bullet holes. Studying only survivors tells you what
tolerated stress looks like — not what catastrophic failure looks
like.
3. Product Success Analysis
Your organization has a flagship product. It’s been the revenue
leader for a decade. Quality is excellent — customer complaints are low,
field returns are minimal, process capability indices are consistently
above 1.67. The product team is asked to share the secrets of their
success.
What nobody mentions: the three similar products that were
discontinued. Each had comparable quality systems. Each followed the
same development process. Each failed — not because of poor quality, but
because the market shifted, a competitor undercut the price, or a
regulatory change made the product obsolete.
When you study only successful products, you conclude that your
development process is excellent. When you include the failures, you
discover that your process might be average and that success was driven
by factors your quality system neither controls nor measures.
4. Employee Performance
Reviews
Your best quality engineers get promoted. Your worst get reassigned.
The people left in the quality department are, by definition, the
survivors — those who adapted to the organizational culture, navigated
the politics, and delivered results that were recognized by the current
management system.
When you survey quality professionals about what skills matter most,
you’re surveying survivors. You’re not capturing the insights of the
people who left because they saw problems nobody wanted to hear about,
or the ones who were pushed out because they challenged the status quo
too aggressively.
The quality culture you think you have is the culture of the people
who remained. It may be a culture of conformity, not a culture of
excellence.
5. Customer Feedback
Your customer satisfaction scores are 92%. You survey your customers
annually. You track Net Promoter Score. You celebrate when the numbers
go up.
But who responds to your surveys? Satisfied customers. Customers who
are still your customers. The ones who left — the ones who experienced
the catastrophic quality failure that made them switch to your
competitor — they don’t fill out your survey. They’re not in your
database anymore.
You’re measuring the satisfaction of survivors. The customer base
that experienced your quality at its worst is invisible to your
measurement system.
Why
Survivorship Bias Is So Dangerous in Quality Management
Survivorship bias is particularly insidious in quality because
quality systems are designed to filter out failures. That’s
their job. SPC charts flag out-of-control conditions. Inspection
stations catch defects. Supplier scorecards identify underperformers.
Corrective action systems track problems to resolution.
The entire quality apparatus is built to make failures disappear —
either by preventing them, catching them, or resolving them. And when
failures disappear, so does the evidence that could teach you something
about why they happened.
This creates a paradox: the better your quality system works, the
less data you have about failures. And the less data you have about
failures, the more you rely on data from successes. And the more you
rely on data from successes, the more vulnerable you become to
survivorship bias.
It’s a feedback loop that makes you progressively less able to see
what could go wrong, precisely because your system is designed to
prevent things from going wrong.
The Bomber Diagram in Your
Factory
Let me make this concrete. Imagine your organization runs 20
production lines. Over five years:
- Line 3 was shut down after persistent quality problems.
- Line 7 was relocated to a new facility and never regained its
capability. - Line 14 had a catastrophic equipment failure that caused a six-month
shutdown. - Line 19 was repurposed for a different product family.
Today, you have 16 active lines. Your quality team studies these 16
lines. They benchmark the best performers. They identify patterns in the
data. They build models.
But those models are built on survivorship. Lines 3, 7, 14, and 19 —
the ones that failed — are invisible. Their data is archived, their
lessons are forgotten, their failure modes are not in your models.
Abraham Wald would tell you: the most important data about your
production system lives in the lines that don’t exist anymore. The
bullet holes on the survivors tell you what a plane can withstand. The
bullet holes on the planes that went down — the ones you can’t see —
tell you where the fatal vulnerabilities are.
How to
Counter Survivorship Bias in Your Quality System
Countering survivorship bias isn’t about eliminating it — you can’t.
It’s about recognizing it and building systematic countermeasures.
Conduct Post-Mortems on
Failures
When a line is shut down, a product is discontinued, a supplier is
dropped, or a project is cancelled, conduct a formal post-mortem. Not a
blame session — a learning exercise. Document what went wrong, what the
data showed before the failure, and what the organization could have
done differently.
Archive these post-mortems. Make them searchable. Reference them when
new projects face similar conditions.
Study the Near-Misses
Near-misses are failures that almost happened but didn’t. They’re the
bombers that limped home with damage to the areas Wald identified as
critical. Near-miss reporting systems are common in safety management
but rare in quality management.
Create a system where operators and engineers can report near-misses
without fear of punishment. Analyze near-misses with the same rigor
you’d apply to an actual failure. A near-miss is a free data point — the
failure happened, but the consequence didn’t. Don’t waste it.
Include Attrition Data
in Your Analysis
When you benchmark suppliers, include the ones you dropped. When you
analyze product quality, include discontinued products. When you study
line performance, include closed lines. When you review employee
performance, interview the people who left.
The cost of including attrition data is modest. The cost of excluding
it is survivorship bias.
Audit Your Best
Practices Against Failures
For every best practice you’ve codified, ask: did any organization
that failed also follow this practice? If the answer is yes — and it
usually is — then the practice may be necessary but not sufficient. It
may even be irrelevant.
A best practice that is common to both survivors and non-survivors is
not a best practice. It’s a baseline.
Build Red Team Exercises
A Red Team is a group charged with challenging assumptions and
finding vulnerabilities. In quality, a Red Team would examine your
quality strategy and ask: “What if everything we believe about our
quality system is based on incomplete data? What are we not seeing?”
Red Team exercises are uncomfortable. They’re supposed to be. Comfort
is the feeling of survivorship bias confirming itself.
Track Leading
Indicators From Failed Systems
When a line, product, or supplier fails, go back through the
historical data and identify the leading indicators that, in hindsight,
predicted the failure. Then check whether your current systems track
those indicators.
If Line 14 failed because of a specific vibration pattern in the
stamping press that nobody was monitoring, make vibration monitoring
part of the standard protocol for all stamping presses. The lesson from
the dead line protects the surviving lines.
The Leadership Challenge
Survivorship bias is ultimately a leadership problem. It takes
leadership to invest time and resources in studying failures when
successes are more pleasant to examine. It takes leadership to maintain
archives of discontinued products and closed lines when the organization
would rather move on. It takes leadership to ask “What are we not
seeing?” when the data dashboard shows green across the board.
The leaders who understand survivorship bias operate with a
fundamental humility: they know that the data they have is not the
complete picture. They know that the quality of their quality data
depends on what’s been filtered out — and that the most important data
might be the data that didn’t survive.
They’re the ones who look at the bomber diagram and, instead of
asking “Where should we add armor?” ask “Where did the planes that
didn’t come back get hit?”
That question — that shift in perspective — is the difference between
a quality system that learns from survival and one that learns from
reality.
The Uncomfortable Truth
Here’s the deepest cut: your organization’s entire quality narrative
is a survivorship story. You exist. Your products exist. Your customers
exist. You’re studying yourself — a successful organization — and
drawing conclusions about what makes organizations successful.
But the cemetery of failed companies is full of organizations that
had quality systems, ISO certifications, trained auditors, and
continuous improvement programs. They didn’t fail because they lacked
quality tools. They failed because survivorship bias made them confident
that the tools they had were sufficient.
The lesson isn’t that quality tools don’t work. The lesson is that
studying only successes teaches you what survival looks like — not what
failure looks like. And if you can’t see failure, you can’t prevent
it.
Abraham Wald saved bombers by looking at the blank spaces on the
diagram. Your quality system can save your organization by looking at
the blank spaces in your data — the lines that closed, the products that
failed, the suppliers that disappeared, the customers who left, and the
employees who walked away.
The most important quality data you’ll ever analyze is the data you
don’t have yet.
Peter Stasko is a Quality Architect with 25+ years of experience
transforming organizations across automotive, aerospace, and
pharmaceutical industries.
