During World War II, Allied command examined bombers returning from missions riddled with bullet holes. The fuselage, outer wings, and tail showed the heaviest damage concentrations. The obvious recommendation was to add armour to those areas.
Abraham Wald, a mathematician, stopped them. The planes they were examining had survived. The bullet holes they mapped were in the places a plane could be hit and still return. The planes hit in the engines or cockpit never came back. Wald argued that armour should go exactly where the surviving planes showed no damage.
This is survivorship bias, and it is one of the most dangerous structural traps in manufacturing quality. You study what passed inspection. You ignore what was scrapped. The strategies you develop from your surviving products and process runs become the very assumptions that guarantee future failures you will never examine.
The Anatomy of Invisible Process Distortion
In manufacturing, survivors are the products that passed, the runs that met targets, the suppliers that delivered, and the process changes that seemed to work. The casualties are the scrapped parts, the aborted runs, the rejected suppliers, and the abandoned initiatives. When you build your understanding of quality exclusively from the survivors, you construct a reality that cannot survive contact with uncontrolled variables.
I have audited plants where a new machining process showed a dropped defect rate over six months. The leadership team prepared a case study to replicate the approach across the organisation. They documented the steps, parameters, and training. Other facilities were expected to copy the playbook exactly.
What the case study did not capture was critical. It ignored the three months of aborted attempts before the process stabilized. It excluded the two operators who were reassigned because they could not adapt. It omitted the material batch quietly scrapped at a cost nobody reported to finance. The specific machine conditions—ambient temperature, tool wear state, coolant concentration—that aligned during the successful runs were never controlled as formal parameters.
The case study was the returning bomber. The failures were the planes that never came back. The documentation looked comprehensive but was dangerously incomplete.
Where Surviving Data Hides in ISO 9001 and IATF 16949 Systems
When engineers optimize a process, they typically study runs that produced conforming parts. They adjust parameters to replicate those conditions. But if runs producing scrap were discarded from the analysis—because the data was never captured or attributed to operator error—the optimization is based on an incomplete picture.
The result is a process window that looks robust but has hidden vulnerabilities. The parameters that caused failures are not understood because they were never studied. When those conditions reappear under a different shift, with a different material lot, or during a different season, the failures return. Nobody can explain why.
Supplier Selection and Root Cause Blind Spots
Organizations evaluate suppliers based on the ones they kept. A supplier on the approved list for five years is assumed to be good because it has delivered conforming material. But the three suppliers rejected during the same period are never studied. Were they rejected for legitimate quality reasons, or because their documentation format was unfamiliar and their pricing did not match expectations?
If you never study rejected suppliers, you cannot know whether your selection criteria predict quality performance or merely correlate with organizational comfort. You also cannot know whether a rejected supplier might have offered a superior solution to a problem you cannot solve with your current supply base.
Root cause analysis is equally vulnerable when applied only to defects that escape to the customer. Internal defects caught and reworked are treated as less significant, even when they represent the same underlying cause at an earlier stage. An organization that investigates only field failures is studying the bombers that returned with damage. The defects caught internally—those that never reached the customer—may represent entirely different failure modes.
The False Confidence of Benchmarking
Benchmarking is structurally built on survivorship bias. You visit the plant with the best quality metrics, document their practices, and attempt to replicate them. What you do not see are the plants that tried the same practices and failed. They invested in identical automation, adopted the same structure, and implemented the same QMS, but achieved completely different results.
You also miss the specific conditions that enabled the benchmark plant's success. A workforce with unusually low turnover. A customer base providing stable, predictable demand. A product design inherently forgiving of process variation. These contextual factors are never captured in the benchmarking report, yet they are the actual drivers of success.
The benchmark plant is a survivor. Studying it without understanding why other implementations failed is like mapping bullet holes on returning bombers. You are documenting the non-fatal hits and calling them the recipe for survival.
The Financial Cost of Ignoring Your Failures
The financial impact of survivorship bias is substantial but difficult to quantify, because the costs are incurred by failures that are never examined. Failed process improvements consume engineering time, training hours, material, and capacity. When these failures are dismissed as implementation problems rather than analyzed as data, the organization repeats the cycle with the same flawed assumptions.
Supplier failures causing line stoppages, premium freight, or emergency sourcing represent direct costs that often exceed the savings from the original sourcing decision. If the organization studies only successful supplier relationships, it cannot identify the criteria that predict failure.
Organizations that study only successes develop an inflated sense of competence. They believe they understand their processes better than they actually do.
This overconfidence leads to riskier decisions, larger bets on unproven approaches, and reduced investment in detection systems. The organization trusts a process capability index without questioning whether the data behind it represents the full range of operating conditions.
Two Approaches to Process Validation
Studying survivors only
- Analyzes conforming runs and ignores scrap data
- Attributes deviations to operator error without investigation
- Documents successful conditions as the standard playbook
- Reports Cpk from data that excludes failure modes
Studying the full picture
- Requires formal analysis of every aborted run and scrapped batch
- Investigates boundary conditions that caused near-misses
- Controls ambient variables formally, not as assumed constants
- Tests edges of the process window before declaring capability
How to Counter Survivorship Bias in Quality Management
Create a structured process for analyzing every significant failure, not just customer escapes. Maintain a database capturing conditions, root cause, corrective action, and cost. Include near-misses—failures caught by the detection system before they escalated. Near-misses are the bullet holes in the fuselage. They show you where the process is vulnerable and they are far more common than actual failures.
When you select a process parameter, a supplier, or a technology, document the alternatives that were rejected and the reasons why. Periodically review these rejected alternatives to determine whether the rejection was based on data or on assumptions that no longer hold. This is analogous to conducting a pre-mortem: before implementing a decision, ask what would cause it to fail.
Examine the conditions of success, not just the outcomes. When a process succeeds, investigate whether the enabling conditions were controlled and sustainable. A process producing conforming parts because ambient temperature happened to fall within a favorable range is not robust. It is lucky. Understanding the conditions of success allows you to control them and make the result reproducible.
Design experiments that deliberately test process boundaries. Run parameters at the edges of the established window, not just at the center. Test materials from alternative suppliers. Challenge the assumptions underpinning your PFMEA and control plans. Untested assumptions are functionally indistinguishable from survivorship bias.
Building a Failure-Inclusive Quality System
- 01Capture all failuresLog every internal reject, scrapped batch, aborted run, and near-miss in a structured database, not just customer returns.
- 02Document rejected alternativesRecord why suppliers, parameters, and technologies were rejected, and the data supporting that decision.
- 03Audit conditions of successVerify whether ambient variables, operator skill, and material lots were controlled or merely favorable.
- 04Test the process boundariesDeliberately run parameters at the edge of the established window to find where the process actually fails.
Wald's Lesson for the Quality Function
Every quality system is built on assumptions about what works. Those assumptions are derived from experience, but experience is inherently filtered through survival. The processes, suppliers, and practices in your organization are not necessarily the best. They are the ones that happened to work under specific conditions.
When those conditions change, the survivors will fail. If you have studied only the survivors, you will not understand why. The organizations achieving the highest quality are not the ones studying their successes most carefully. They are the ones studying their failures most honestly—including the failures they dismissed as anomalies and those hidden in blind spots created by the systems they trust.
Wald did not have data on the planes that did not return. But he had the intellectual honesty to recognize that the absence of data was itself the most important data point. In quality management, the failures you do not see are more dangerous than the ones you do, because the ones you see, you can fix. Stop looking only at where the bullet holes are. Start asking where they are not.