During World War II, Allied bombers returned from missions riddled with bullet holes. The military mapped the damage and planned to add armour to the fuselage and wings. A statistician named Abraham Wald disagreed. He told them to armour the areas with no holes.
Wald's logic was brutal. The planes returning with holes in the fuselage survived being hit there. The planes hit in the engine or cockpit never came back at all. The generals were studying survivors and mistaking them for the full population. They were about to reinforce the wrong areas.
Quality organisations repeat this error constantly. You study passed inspections, retained customers, and completed projects. You build dashboards from processes that survived. Meanwhile, the cancelled initiatives, lost suppliers, and abandoned processes carry the data that would prevent your next crisis.
How survivorship bias distorts quality metrics
In quality management, survivorship bias means concentrating on the processes that made it past a selection filter while ignoring those that did not, typically because they no longer generate data. A cancelled process has no SPC charts. A fired supplier has no monthly scorecard. The failure removes itself from the data stream.
I have audited plants that celebrated a 99% final inspection pass rate while ignoring what happened to the rejected 1%. Reworked parts pass on the second attempt and enter the yield calculation as successes. Parts shipped under concession blend into the approved batch. The true first-pass yield is buried inside a number that keeps the management dashboard green.
Your supplier quality program recognises top-performing vendors every quarter with metrics that look strong. What the program does not capture is the six suppliers dropped in the last two years for falsified material certificates, substituted materials, or paper-only quality management systems. The survivors are on your approved vendor list because they performed well, or simply because they have not yet been stress-tested.
The false competence of a curated portfolio
Organisations look at their current product portfolio, current supplier base, and current process roster, then conclude their quality system is robust. They are looking at survivors. The root causes that killed the failed initiatives still exist, but nobody studies them because the failure was quietly eliminated.
Consider a manufacturing plant that shuts down two chronic underperforming lines after years of high scrap and customer complaints. The next quarter, quality metrics improve dramatically. Scrap drops. Complaints fall. The management team reports a decisive improvement. The reality is that the improvement came from subtraction, not from root cause correction.
Eighteen months later, volume increases and the remaining smooth lines get pushed beyond their designed limits. Suddenly they exhibit the same failure modes that killed the shut-down lines. Nobody recognises the pattern because nobody performed a proper autopsy on the dead processes. The ghosts of past failures have returned to infect the survivors.
The customers who never complained
A pharmaceutical company I worked with tracked complaint resolution rates. Their data showed 92% of complaining customers were satisfied with how their issue was handled. The remaining 8% were archived as unresolvable. Management concluded the quality system was performing well.
The team had no data on non-complainers. No surveys, no follow-up calls, no win-back analysis. Their entire customer satisfaction picture was built from people who had already decided to engage. They had no visibility into customers who received defective product, noticed it, and simply left.
We commissioned a blind study of customers who had stopped ordering in the previous twelve months. Forty-three percent had experienced a quality issue and chose not to complain. They took their business elsewhere. Of those, 61% said the issue was recurring. The company had been studying its complaint data and concluding quality was strong while silently losing customers to systemic defects it never knew existed.
What a complaint resolution rate hides
Where survivorship bias hides in your QMS
Third-party audits measure whether your quality management system conforms to a standard such as ISO 9001 or AS9100. They do not measure whether your QMS is effective. An organisation can have perfectly documented procedures, meticulously maintained records, and a spotless audit history while producing mediocre product. The audit survives. The truth about process effectiveness does not surface in the nonconformity report.
Your improvement program tracks completed kaizen events and closed CAPAs. The dashboard shows impressive completion counts. What it does not show are the CAPAs closed without effective verification of corrective action, or the improvement initiatives shelved because the champion left the company. The completed projects generate confirming data. The abandoned ones hold the organisational learning you actually need.
Retained employees present the same trap. Your quality team has low turnover and management sees stability as a strength. But the young engineer who left for a competitor took knowledge of your PFMEA gaps with them. The remaining team does not know what is missing because the person who carried that awareness is gone. The survivors know what they know. They cannot know what departed.
Building a failure archive
The antidote to survivorship bias is systematic, deliberate documentation of failure. Not traditional 8D root cause analysis for individual defects, but an organisational practice of archiving everything that did not make it and forcing future teams to review that archive before launching new work.
A quality system that studies only its successes is a hospital that tracks only discharged patients.
When an improvement initiative is cancelled, conduct a post-mortem. Why was it cancelled? What was learned before cancellation? What assumptions were wrong? Store these in a searchable archive and require future project teams to review relevant cancelled initiatives before launching new ones. The post-mortem is not punitive. It is an investment in preventing the same mistakes.
When you drop a supplier, conduct an exit analysis. When a quality professional resigns, run a structured knowledge extraction interview focused on the risks they flagged that never got escalated and the improvements they proposed that never got approved. When a customer stops ordering, call them. A departing customer has no incentive to be polite and every incentive to be honest.
The failure documentation sequence
- 01TriggerA project, supplier, customer, or process is cancelled, dropped, or shut down
- 02DocumentRecord the root cause, timeline, early warning signs, and what was learned
- 03ArchiveStore the post-mortem in a searchable format indexed by process type and failure mode
- 04Mandate reviewRequire new project teams to read relevant past failures before approval
- 05ApplyUse archived lessons to adjust PFMEA, control plans, and supplier selection criteria
Balancing success metrics with failure metrics
For every success metric in your quality system, create a corresponding failure metric. Track first-pass yield alongside final inspection pass rate. Track lost customers alongside complaint resolution rate. Track abandoned projects alongside completed CAPAs. The success number tells you what survived. The failure number tells you where the next crisis will originate.
Map the missing data periodically. Ask what processes you are not measuring, what customers you are not hearing from, and what employees you are not listening to. The answers reveal where survivorship bias is most active in your organisation. A lesson extracted from a costly failed project is worth more than a success that nobody can explain.
Abraham Wald saved lives by looking at the same data everyone else had and asking a different question. He did not ask where the bullet holes were. He asked where they were not, and why. Your quality system needs that question. The survivors can show you where you are strong. Only the failures can show you where you are vulnerable.
Stop studying only the planes that came back. Build the archive. Conduct the autopsies. The most important quality lessons in your organisation are sitting in unmarked graves, waiting to be exhumed.