The Bombs That Came Back
During World War II, the Allied forces faced a problem. Their bombers
were returning from missions riddled with bullet holes. The military
brass wanted to armor the planes, but armor is heavy — you can’t cover
everything. So they studied the patterns: the bullet holes clustered on
the fuselage, the outer wings, and the tail. The obvious answer was to
reinforce exactly those areas.
A mathematician named Abraham Wald said no. Reinforce the areas with
no bullet holes. The generals were confused. Wald explained: the planes
they were examining had survived. The bullet holes they saw marked the
places a plane could be hit and still fly home. The planes hit in the
engines or cockpit never came back. The absence of damage was the real
data.
The military listened. Wald was right. They armored the engines and
cockpit. Survival rates improved.
This is survivorship bias — the logical error of drawing conclusions
from only the things that survived a selection process while ignoring
everything that didn’t. And it is everywhere in manufacturing quality,
quietly warping decisions, strategies, and lessons learned in ways that
most organizations never detect.
What
Survivorship Bias Looks Like in Manufacturing
In a World War II bomber fleet, the selection filter was literal
destruction. In a modern factory, the filters are more subtle but
equally distorting.
Consider the commonplace: your best-performing production line. You
study it intensely. You document its practices, you benchmark other
lines against it, you build training programs around its methods. You
tell the story of Line 7 at every management review: “This is how we do
it when we do it right.”
What you don’t study is Line 3, which was shut down eighteen months
ago after chronic quality failures. Or the product line that was
discontinued because its defect rate was triple the target. Or the
supplier you fired because their parts never met spec. Those failures —
the ones that didn’t survive — contain at least as much instructional
value as Line 7’s success story. Probably more. But they’re gone,
forgotten, excluded from the dataset your organization uses to
understand itself.
Survivorship bias doesn’t just affect which lines you study. It
affects which metrics you trust, which suppliers you benchmark, which
processes you standardize, which engineers you promote, and which
strategies you double down on. Every time you look at what worked
without examining what failed, you are looking at the bullet holes on
the planes that came home.
The Best Practice Trap
“Best practices” might be the most dangerous two words in
manufacturing quality. Not because good practices don’t exist, but
because the process of identifying them is almost always contaminated by
survivorship bias.
Here’s how it typically works. Your company has twelve production
lines across three plants. Management decides to identify and share best
practices. They look at the metrics: scrap rate, cycle time, first-pass
yield, customer complaints. They find the top performer — let’s say it’s
Line 7 at Plant B. They send a team to study it. The team documents
everything: how operators start their shift, how tools are organized,
how the line leader runs meetings, the specific sequence of quality
checks. This becomes the “Line 7 Best Practice Package” and it gets
rolled out to all other lines.
Six months later, the results are mixed. Some lines improve
marginally. Others don’t change at all. A few actually get worse.
Management is frustrated. “We gave them the playbook. Why isn’t it
working?”
The answer is that the playbook was written from a survivor. Line 7
succeeded — but was it because of the documented practices, or despite
them? Maybe Line 7 has a uniquely skilled group of operators who
compensate for process weaknesses through tacit knowledge that no
benchmarking team could capture. Maybe Line 7’s product mix is simpler.
Maybe Line 7’s equipment is newer despite being the same model. Maybe
the raw materials from that plant’s local supplier happen to be more
consistent.
You don’t know, because you never studied the lines that failed.
Maybe Line 3 at Plant A had the exact same practices as Line 7 but
failed because of factors nobody documented. If you had studied both the
survivors and the casualties, you might have discovered that the
practices were irrelevant and the real differentiator was something
entirely different — supplier quality, or equipment maintenance history,
or operator experience levels.
The best practice trap is survivorship bias with a corporate budget.
You institutionalize the habits of the survivors without understanding
why they survived, and you miss the lessons buried in the failures.
The Customer Complaint
Mirror
Customer complaint data is one of the most survivorship-biased
datasets in manufacturing. Here’s why: you only hear from customers who
care enough to complain, who have the time to complain, who know how to
navigate your complaint process, and whose problems weren’t resolved by
the first point of contact.
The customers who received defective product and simply switched
suppliers? Silent. The customers whose defects were intermittent and
hard to characterize? Silent. The customers in markets where complaining
is culturally unusual? Silent. The customers whose defects created
safety issues that were intercepted before reaching the field? Silent —
unless a regulator got involved.
Your Pareto chart of top customer complaints looks authoritative. It
has real data, real frequencies, real descriptions. But it’s a chart of
the complaints that survived the filtration process of customer
behavior, corporate bureaucracy, and your own complaint intake system.
The real top complaints — the ones that caused the most damage, affected
the most customers, or carried the most risk — might not be on it at
all.
This has real consequences. Your quality improvement resources get
allocated based on complaint data. Your corrective actions target the
complaints you can see. Your management reviews celebrate progress
against the visible issues. And the invisible problems — the ones that
are silently destroying customer loyalty or creating latent safety risks
— continue unabated because they never made it onto a chart.
Wald would tell you to study the silence. The customers who don’t
complain. The defects that don’t get reported. The returns that come
back with vague descriptions. That’s where the engines and cockpit
are.
The Supplier Selection
Illusion
Your approved supplier list is a survivorship bias museum. The
suppliers on it survived your qualification process, your audits, your
performance reviews. The suppliers who failed are gone — delisted,
rejected, forgotten. You benchmark your current suppliers against each
other, rate them on scorecards, award business based on their relative
performance.
But consider what’s missing. The supplier you rejected during
qualification because their samples failed — what if their failure was
caused by a miscommunication in your specification, not a capability
gap? The supplier you delisted for delivery problems — what if the root
cause was in your own unpredictable ordering patterns? The supplier who
withdrew from the process because they found your payment terms
unacceptable — what if they were the most technically capable
option?
Every supplier selection decision narrows your visible universe. Over
time, you’re comparing an increasingly homogeneous pool and calling it a
competitive market. The real competitive insights — the ones that come
from understanding why suppliers fail, not just why they succeed — are
lost because the failures are excluded from your dataset.
The Successful Product
Fallacy
When your new product launch succeeds, everyone wants to take credit.
The design team points to their FMEA rigor. The manufacturing team
points to their process validation. The quality team points to their
control plan. The project team points to their APQP discipline. A
success story is constructed, documented, and presented at the next
company meeting as proof that the system works.
When your new product launch fails, the story is different. Blame is
assigned. A team is reorganized. The failed product is quietly
discontinued. And the lessons — the real, granular, specific lessons
about what went wrong and when — are buried because nobody wants to
document failure.
Over time, your organization’s collective knowledge base becomes a
library of success stories. Your new product development process is
based on what appeared to work in the past, without the corrective
ballast of understanding what actually caused the failures. You’re
building planes with armor on the wings and no armor on the engines,
wondering why some of them still don’t come back.
The Benchmarking Blindspot
Benchmarking is supposed to help you learn from others. You visit
best-in-class companies, study their processes, and bring back ideas.
But here’s the problem: you’re benchmarking companies that survived.
You’re not benchmarking the companies that went bankrupt, that lost
their market position, that were acquired at a discount because their
quality collapsed.
The companies you benchmark are happy to show you their successful
practices. They won’t show you their failures — those are hidden,
rationalized, or forgotten. You come away with a curated view of
excellence that omits the struggles, the dead ends, and the mistakes
that were just as instructive as the successes.
More subtly, you benchmark companies that are structurally different
from yours in ways you don’t appreciate. A best-in-class automotive
supplier’s quality system works in the context of their specific
customer demands, regulatory environment, workforce capabilities, and
capital investment. Extract one element and transplant it into your
context, and it may fail — not because it’s a bad practice, but because
it was part of a system that you didn’t copy in its entirety.
You never see the companies that tried to copy those same practices
and failed, because those companies aren’t hosting benchmarking visits.
They’re not writing case studies. They’re not presenting at conferences.
They’re the planes that didn’t come back.
The Promotion Filter
Your quality leaders — your directors, your managers, your senior
engineers — are professional survivors. They survived the political
landscape of your organization. They navigated the projects that
succeeded. Their track records, as visible to senior management, are
compilations of the programs and initiatives that turned out well.
This doesn’t mean they’re incompetent. It means their competence is
measured against a biased dataset. The quality director who successfully
launched three products — but whose career also includes two quiet
failures that were attributed to “market conditions” — appears to have a
perfect record. The engineer who identified a critical design flaw
before launch but whose prevention was never documented because the
product was simply delayed and fixed — that prevention is invisible.
When you promote based on visible success, you promote survivorship.
You elevate people who are skilled at navigating the system, but you may
overlook people who are skilled at preventing the failures that never
became visible. The quiet preventers — the people who stop problems
before they manifest — are systematically undervalued because the
evidence of their work is literally invisible.
How to
Counteract Survivorship Bias in Quality
Countering survivorship bias starts with recognizing that absence of
data is data. Here are practical approaches:
Study your failures systematically. Don’t just
conduct post-mortems on major failures — create a formal process for
documenting and analyzing every discontinued product, every lost
customer, every failed supplier, every shutdown production line. Build a
“failure library” that’s as accessible and well-organized as your best
practice database.
Include attrition analysis in your quality reviews.
When you review customer data, don’t just analyze the complaints you
received. Analyze the customers you lost and try to understand why. When
you review supplier performance, don’t just score your current suppliers
— examine the suppliers that left your approved list and understand the
full picture of why.
Blind yourself to outcomes during analysis. When
studying processes, try to evaluate practices without knowing whether
they succeeded or failed. This is harder than it sounds, but it forces
you to assess the inherent merit of a practice rather than being
influenced by knowing it “worked” or “didn’t work.”
Seek out negative examples deliberately. Actively
look for companies in your industry that failed, products that were
recalled, processes that produced catastrophic results. These negative
examples often contain more actionable intelligence than a dozen success
stories.
Question your “best” performers. When you identify a
top-performing line, plant, or team, don’t just study what they do. Ask:
“What would have to be true for these practices to produce bad results?”
This counterfactual thinking helps you identify hidden dependencies and
contextual factors that survivorship bias obscures.
Track the invisible. Monitor customer defection
rates, not just complaint rates. Track near-misses, not just escapes.
Measure process deviations that were caught internally, not just defects
that reached customers. The more you can illuminate the normally
invisible events, the less power survivorship bias has over your
decisions.
The Uncomfortable Truth
Survivorship bias is seductive because success feels like proof. When
something works, we want to believe we understand why. When we can tell
a clean story about cause and effect — “We did X, and Y improved” — we
feel in control. The messier reality is that success often involves
luck, timing, context, and factors we didn’t measure or even notice.
Wald’s insight wasn’t just statistical. It was epistemological. He
understood that the evidence you can see is only half the picture — and
it might be the less important half. The planes that didn’t return had
more to teach than the planes that did. The bullet holes you can see
tell you where you can afford to be hit. The absence of bullet holes
tells you where you can’t.
Your organization’s quality data works the same way. The successes
you can see are instructive but incomplete. The failures you can’t see —
the ones that were buried, forgotten, rationalized away, or never
documented — hold the lessons you actually need to learn. The question
isn’t whether survivorship bias exists in your quality system. It does.
The question is whether you have the discipline to look for the planes
that didn’t come back.
Peter Stasko is a Quality Architect with over 25
years of experience in manufacturing quality management, process
optimization, and quality systems design. He specializes in helping
organizations identify and correct the systematic thinking errors that
undermine their quality performance. His work focuses on bridging the
gap between quality theory and manufacturing reality.
