Quality
and Survivorship Bias: When Your Organization Studies Only Its Successes
and Misses the Lessons Hidden in Its Failures — and the Products That
Survived Became the Only Data Anyone Trusted
Peter Stasko — Quality Architect, 25+ Years in Manufacturing
Excellence
The Bombs That Came Home
During World War II, the Allied forces faced a pressing problem:
their bombers were being shot down over Europe at devastating rates. The
military command gathered their best engineers and statisticians, spread
blueprints of the bombers across a table, and mapped every bullet hole
on every aircraft that made it back to base.
The pattern was clear. The wings, the fuselage, the tail section —
these areas were riddled with damage. The natural conclusion was equally
clear: reinforce those areas. Put more armor where the bullets are
hitting.
Then Abraham Wald, a mathematician at the Statistical Research Group,
pointed out something that changed everything.
The aircraft they were examining had survived. They were
looking at the bombers that made it home despite being hit. The bullet
holes they saw marked the places where a bomber could take
damage and still fly. The places with no bullet holes — the engines, the
cockpit — were the places where hits were fatal. Those aircraft never
came back to be examined.
Wald’s insight saved countless lives. It also gave the world one of
the most powerful warnings about the quality of our thinking:
the data you don’t have can be more important than the data you
do.
This is survivorship bias, and it is quietly undermining quality
decisions in manufacturing organizations every single day.
What Survivorship Bias Really
Is
Survivorship bias is a logical error that occurs when you focus only
on the things that survived a selection process — products that passed
inspection, suppliers that stayed in business, processes that survived
cost-cutting — while ignoring those that didn’t make it. The result is a
systematically distorted picture of reality.
You see the winners. You study the winners. You try to replicate the
winners. And you never ask the most important question: what
happened to the losers, and what do they know that the winners are
hiding?
In manufacturing and quality management, survivorship bias is
everywhere. It infects benchmarking studies, root cause analyses,
supplier evaluations, process optimization, and strategic planning. And
because it feels like data-driven decision-making — you’re studying real
products, real processes, real results — it is exceptionally difficult
to detect from the inside.
How Survivorship
Bias Infects Quality Systems
The Benchmarking Trap
A manufacturing company decides to benchmark its quality system
against industry leaders. It studies Toyota, Samsung, Siemens —
companies famous for their quality excellence. It documents their
practices, adopts their tools, and implements their frameworks.
What it doesn’t study are the hundreds of companies that implemented
those same tools and failed. Toyota’s quality system works within
Toyota’s culture, supply chain, workforce development model, and
historical context. You can copy the tools and still fail — many
organizations have. But you’ll never read case studies about the
failures because failed companies don’t publish white papers or give
keynote speeches.
The benchmarking data is already filtered through survival. You’re
studying the survivors, not the truth.
The Process That “Always
Worked”
“We’ve been doing it this way for fifteen years and it works.” How
many times have you heard this in a quality review?
Here’s the problem: the process survived, but did it survive because
it’s good, or did it survive because the failures it produced were
absorbed by downstream inspection, rework, or customer tolerance? The
process may have “worked” only because a hidden safety net caught its
failures — a safety net that might not exist tomorrow.
When you only study the process in its current, surviving state, you
miss the failures it quietly produced along the way. Those failures
contain the data you actually need.
The Supplier Evaluation
Illusion
Your organization evaluates suppliers based on quality performance.
You track defect rates, on-time delivery, and responsiveness. Over time,
the poor performers are dropped, and your supplier base becomes a
curated collection of high performers.
Now you use this supplier base to set quality benchmarks and
expectations. Your data shows that “typical” suppliers achieve 99.5%
quality rates. But that data excludes every supplier that failed, went
bankrupt, or was fired. Your baseline is inflated, your expectations are
skewed, and when a new supplier struggles, you judge them against a
standard that was constructed from survivors only.
The Successful Product
Post-Mortem
After a successful product launch, the team conducts a retrospective.
They document what went right: the robust design, the thorough testing,
the excellent supplier coordination. The retrospective becomes a
template for future launches.
What’s missing? The five products that failed during development and
never reached launch. Those products might have had robust designs and
thorough testing too — but they failed for reasons the surviving product
never exposed. By studying only successes, you build a theory of quality
that is incomplete and potentially dangerous.
The Hidden
Cost: What You Don’t Know You’re Missing
Survivorship bias doesn’t just give you an incomplete picture. It
gives you a confidently wrong picture. And that confidence is
what makes it so dangerous.
False Confidence in
Process Controls
When you study only the products that passed through your process
successfully, you conclude that your process controls are effective. But
the products that failed — the ones rejected, reworked, or scrapped —
may have failed precisely because of gaps in those same controls. By
excluding failures from your analysis, you systematically overestimate
the effectiveness of your quality system.
This is especially insidious in industries with high first-pass
yield. When 98% of products pass, it’s tempting to study the 98% and
optimize for them. But the 2% that failed contain more quality
intelligence than the 98% that passed — because they reveal the boundary
conditions, the edge cases, the vulnerabilities in your process.
Misallocated Improvement
Resources
Organizations allocate improvement resources based on what they can
see. If you can see successful processes, you invest in making them more
successful. If you can’t see the failures — because they were scrapped,
because the data was archived, because the team was reassigned — you
can’t invest in fixing them.
The result: over-engineered successes and under-addressed failures.
Your best processes get better while your worst processes quietly
deteriorate, invisible to a system that only measures what survives.
Strategic Blindness
At the strategic level, survivorship bias creates a false narrative
about what drives success. The company that survived three recessions
attributes its survival to strategic brilliance. But for every company
that survived with that strategy, how many perished with the same one?
Without studying the failures, you can’t distinguish strategy from luck
— and you can’t tell whether your quality approach is genuinely robust
or just fortunate.
Real-World Manufacturing
Examples
The Aerospace Supplier
An aerospace component manufacturer had a forging process that
produced exceptional quality for over a decade. The process parameters
were considered proprietary knowledge, passed down through operator
experience. When asked to document best practices, the team described
the process as it currently operated — successfully.
What they didn’t document was the eighteen-month debugging period in
the early years, when the forge produced a 30% scrap rate. The solutions
discovered during that period — specific temperature ramp rates, die
preheating protocols, material batch qualification procedures — were
embedded in operator behavior but never formally recorded. When
experienced operators retired and new operators followed the
“documented” process, scrap rates tripled.
The surviving process documentation was incomplete because it was
written by survivors who had internalized corrections that never made it
onto paper.
The Medical Device Company
A medical device company studied its most successful product launches
to create a quality playbook. The playbook emphasized design for
manufacturability, early supplier involvement, and extensive validation
testing. It was presented at industry conferences and adopted by other
divisions.
Two years later, a product developed using the same playbook failed
catastrophically in the field. The investigation revealed that the
successful products had shared an unstated advantage: they were all
designed by a small group of senior engineers who informally reviewed
each other’s work. The “playbook” documented the formal process but
missed the informal safety net that had actually prevented defects.
The playbook was a study of survivors. The real quality mechanism —
peer review by experienced engineers — was invisible because it was
never a formal part of the process that survived.
The Automotive Assembly
Plant
An automotive plant tracked quality by measuring defects per vehicle
at the end of the assembly line. Over two years, the defect rate dropped
steadily, and the plant received internal awards for quality
improvement.
Then a customer survey revealed that warranty claims had
increased during the same period. The investigation found that
assembly line workers had learned to “hide” defects that would be caught
at end-of-line inspection, fixing them informally before they were
recorded. The defects didn’t disappear — they just moved from the
tracked metric to the untracked one.
The “improving” quality data was survivorship bias in action: only
the defects that made it through the informal fixes were being measured,
and the real defect rate was hidden in the failures that the measurement
system couldn’t see.
How to Counter
Survivorship Bias in Quality
1. Study the Failures
Systematically
Create a formal process for analyzing failed products, rejected lots,
and scrapped components with the same rigor you apply to successful
ones. Failure analysis should not be a punishment exercise — it should
be a learning discipline.
Action: Implement a “failure retrospective” program
where every significant quality failure is documented, analyzed, and
archived with the same care as your best-practice case studies. Make
failure data as accessible as success data.
2. Look for the Missing Data
When reviewing quality data, always ask: “What’s not here?” If your
supplier performance data only shows current suppliers, you’re missing
the data from terminated ones. If your process capability study only
covers running production, you’re missing startup and shutdown
conditions. If your customer satisfaction data only comes from customers
who responded to the survey, you’re missing the silent ones.
Action: For every quality dataset, document what is
excluded and why. Then assess whether the exclusions could be
introducing survivorship bias.
3. Include
Near-Misses and Scrapped Data in Analysis
Near-misses — events that almost caused a defect but didn’t — contain
the same causal information as actual defects. Scrapped lots contain the
same process information as accepted ones. By including this data in
your analysis, you dramatically expand the sample size and reduce
survivorship bias.
Action: Track near-misses with the same discipline
as actual defects. Create a parallel tracking system for scrapped and
reworked material that feeds into your quality improvement process.
4. Seek Disconfirming Evidence
The human tendency is to seek data that confirms what we already
believe — and survivorship bias makes this easy, because the surviving
data often confirms that things are working. Actively seek data that
would disprove your quality assumptions.
Action: Assign a “devil’s advocate” role in quality
reviews. This person’s job is to find the data that contradicts the
prevailing narrative — the failures, the outliers, the cases that didn’t
survive.
5. Build Failure Libraries
Just as organizations build best-practice libraries, they should
build failure libraries — searchable databases of past failures, their
causes, and their resolutions. When engineers face a new quality
challenge, they should search the failure library alongside the
best-practice library.
Action: Create a structured failure database that
categorizes failures by type, cause, process area, and resolution. Make
it searchable and require that it be consulted during design reviews and
process changes.
6. Question “Proven” Processes
Every process that “has always worked” should be periodically
re-examined with fresh eyes. The conditions that made it work may have
changed. The invisible safety nets that caught its failures may have
been removed. The operators who knew its quirks may have retired.
Action: Conduct periodic “process archaeology”
reviews where cross-functional teams re-examine long-standing processes
to identify hidden assumptions, informal workarounds, and undocumented
knowledge that may be propping up apparent success.
The
Deeper Lesson: Humility in the Face of Incomplete Data
Survivorship bias teaches a lesson that goes beyond methodology. It
teaches humility. It reminds us that our data is always incomplete, that
our view is always partial, and that the things we can’t see may matter
more than the things we can.
In quality management, this humility is essential. Every control
chart, every inspection plan, every capability study is based on the
data we have — not the data that matters most. The most important
quality insights often live in the failures we didn’t record, the
products we didn’t study, and the processes that didn’t survive.
Abraham Wald didn’t just solve a military problem. He gave us a
permanent reminder: the absence of evidence is not evidence of
absence. In quality, the defects you don’t see, the failures
you don’t study, and the data you don’t collect are shaping your
outcomes just as powerfully as the data you have.
The organizations that understand this — that study their failures as
rigorously as their successes, that look for what’s missing as carefully
as what’s present, that build systems to capture the lessons hidden in
the data that didn’t survive — are the organizations that build quality
systems worthy of the name.
Everyone else is just reinforcing the wrong parts of the
airplane.
About the Author
Peter Stasko is a Quality Architect with over 25
years of hands-on experience in manufacturing excellence. He has worked
across automotive, aerospace, medical devices, and electronics
industries, helping organizations build quality systems that don’t just
survive audits — they survive reality. His approach combines deep
technical knowledge with an understanding of human behavior, because
he’s learned that the most expensive quality failures are rarely
technical — they’re human.
Connect with Peter at iaec.online
