During World War II, Allied command examined bombers returning from missions over Europe and catalogued where bullet holes concentrated. The data pointed clearly to the fuselage, outer wings, and tail section. The obvious conclusion was to add armour precisely where the planes were getting hit.

Abraham Wald, a mathematician, reached the opposite conclusion. The planes under examination were the survivors. The bullet holes they carried marked areas where a bomber could sustain damage and still fly home. The planes hit in the engines and cockpit never returned — those were the fatal hits, hidden in the data the command did not have.

This is survivorship bias, and it quietly undermines quality management systems. Organizations systematically study their most successful product launches, top-performing suppliers, and most efficient production lines. They build best-practice playbooks from these successes. What they never study are the discontinued products, the fired suppliers, and the abandoned processes. The failures that carried the critical information are ignored entirely.

Success Playbooks Built on Incomplete Evidence

I have audited plants that documented impressive first-pass yields above 98 percent on flagship products. The quality teams meticulously recorded process parameters, inspection protocols, and training programmes. They distilled these into detailed success playbooks intended to replicate that performance across every new product launch.

When a new product line struggled to clear 85 percent first-pass yield, the team applied the playbook and remained baffled. Same process controls, same inspection frequency, same training structure. The playbook failed because it was built on survivorship bias. The flagship products had succeeded despite several suboptimal practices, not because of them.

Quality decisions are made at the process, not in the report that describes it afterwards. The missing data dictates the outcome.
Quality decisions are made at the process, not in the report that describes it afterwards. The missing data dictates the outcome.

Those successful products possessed forgiving designs, wide tolerance bands, and naturally consistent raw materials. The documented practices were coincidental. The real lessons regarding which parameters drove yield were hidden in the discontinued product lines from the previous year. Nobody had studied those failures with anything close to the analytical rigour applied to the successes.

Benchmarking Survivors Without Understanding Casualties

An automotive supplier spent two years implementing the quality management system of a decorated competitor. Leadership visited their facilities, attended their industry conference presentations, and consumed every available case study. They implemented the competitor's layered process audit system, statistical process control methods, and FMEA restructuring.

Eighteen months later, their defect rate had barely shifted. Quality costs increased by 30 percent, and production throughput dropped because the newly adopted systems were entirely unsuited to their specific volume and product complexity. The supplier had benchmarked a survivor without understanding the casualties.

What the supplier did not know was that three other companies in the same segment had tried to implement identical quality systems. One went bankrupt. One was acquired at a discount. One abandoned the approach after a catastrophic failure cost them their largest customer. Failed organizations do not publish case studies or present at conferences.

Benchmarking Reality vs Perception

What teams assume

  • The competitor's QMS directly caused their low defect rate.
  • Case studies represent the true effectiveness of the system.
  • Auditing and SPC frameworks are universally transferable.
  • Awards validate the underlying methodology without bias.

What actually happened

  • Market position and deep capital reserves masked inefficiencies.
  • Failures and abandoned implementations are never published.
  • System complexity only works for specific volume and product mix.
  • Multiple companies attempted the same system and went bankrupt.
The systems you study are shaped by market forces and capital reserves, not just quality engineering.

Mistaking Recovery Speed for Effective Prevention

A pharmaceutical manufacturer built a celebrated quality culture around thorough investigation and root cause analysis. Every significant event triggered an 8D investigation. Their CAPA closure rate sat at 95 percent. Management reviews highlighted story after story of teams mobilizing resources and implementing effective corrective actions.

When I analysed five years of their CAPA records, the survivorship trap became obvious. Nearly three-quarters of their significant quality events had identifiable precursors — earlier, smaller events containing the exact same root causes. These near-misses had been identified but dismissed because existing controls caught the defect before it reached the customer.

The organization had become excellent at firefighting and mistaken the speed of its fire department for the absence of fires. The near-misses were treated as evidence that the system worked, when they were actually evidence of systemic prevention failure. They were studying their successful recoveries and concluding their quality system was robust.

Applying the Wald Test to Quality Data

Recognizing survivorship bias is necessary but insufficient. Quality teams must systematically counteract it during data analysis. Before drawing conclusions from any quality dataset, you must inventory your missing data and ask what information is systematically absent from the room.

Apply what I call the Wald Test. Identify the exact population that generated your current dataset. Determine what population is systematically excluded from that data. Finally, ask how your conclusion would change if the excluded population exhibited entirely opposite characteristics.

Benchmarking survivors tells you what can coexist with success. It does not tell you what causes success.

An aerospace quality manager used this test on their supplier audit programme. The data showed that audited suppliers had fewer quality incidents than non-audited suppliers. The initial conclusion was that auditing directly reduces supplier defects. The Wald Test exposed the flaw: audited suppliers were selected because they were already higher-performing. The audit programme was selecting winners, not improving performance.

Conducting Retromortems on Historical Failures

A premortem imagines future failure before a project begins. A retromortem examines historical failure with the analytical rigour normally reserved for celebrating success. Select three to five significant failures from your organization's history and dissect the decision-making process that led to each collapse.

Identify exactly what data was available but ignored by leadership at the time. Document what the organization learned versus what it should have learned. A retromortem frequently reveals that current quality strategies were built on exactly half the available evidence — the successful half.

The Retromortem Sequence

  1. 01Select the failureIdentify a discontinued product, a lost customer, or a reverted process change.
  2. 02Map decision dataReconstruct the original decision-making process and isolate what metrics were used.
  3. 03Identify ignored signalsPinpoint the leading indicators or near-misses that leadership actively dismissed.
  4. 04Extract the true lessonCompare what the organization originally learned against the actual failure mechanism.
  5. 05Challenge current strategyTest existing playbooks against this newly surfaced failure data.
A structured methodology for extracting actionable data from discontinued products and failed processes.

Building Anti-Survivorship Quality Reviews

Standard management reviews are inherently biased toward positive results. The metrics that survive to be reported are the ones that look good. Restructuring these reviews requires deliberately injecting absent data through a dedicated failure portfolio and shadow metrics that track unresolved audit findings and overdue calibrations.

Organizations must actively normalize the study of failure. Make it standard practice to examine collapsed processes with the same engineering resources and attention given to flagship successes. Every best practice should be tagged as either a proven causal relationship, a mere correlation, or an assumption adopted through pure precedent.

I worked with a consumer electronics manufacturer that maintained a 99.7 percent outgoing quality rate for seven consecutive years. When a new product platform introduced unfamiliar technologies, their outgoing quality dropped to 94.2 percent in a single quarter. Their quality system had been optimized exclusively for a single, inherently robust product architecture. They spent seven years reinforcing the fuselage, totally blind to the vulnerabilities hidden in the planes that never came back.