By Er. Rajan C. Mathew FIE (I)
Having spent a significant part of my professional life as an equipment and plant designer, I have watched the evolution of industrial and aerospace engineering with both fascination and growing concern. As human and machine systems grow larger, more intricate, and increasingly interconnected, there is a pervasive instinct among design teams to solve safety challenges by adding layers of redundancy.
If a primary valve might fail, add a secondary bypass valve. If a flight computer might glitch, add a second or third backup unit. If a sensor could misread, install a voting array of three sensors.
On paper, the mathematics of reliability engineering look impeccable: if the probability of a single component failing is low, the probability of two independent components failing simultaneously should be exponentially lower.
However, my own experience on the engineering floor taught me a very different lesson: redundancy in complex, tightly coupled systems eventually becomes unmanageable and invites catastrophic failure.
The Core Problem: The Redundancy Paradox!
In safety science, this phenomenon is often referred to as the Redundancy Paradox. While simple mechanical redundancy works brilliantly in isolated, independent systems—like having a twin hydraulic line on a mechanical steering gear—scaling that philosophy into modern, automated human-machine plants creates higher-order complexity.
When you add defensive layers, you do not simply reduce risk; you add new wiring, new software logic, new maintenance requirements, and new interfaces. Eventually, the safety system itself becomes so complex that it creates entirely new failure modes that no designer anticipated.
In his classic work on Normal Accidents Theory, sociologist Charles Perrow pointed out that when a system exhibits two key characteristics—high complexity (unintended or hidden interaction sequences) and tight coupling (where a failure in one section rapidly cascades to another without time buffers)—adding automated safety devices and redundant loops makes a major accident almost inevitable over time.
From a practical engineering and operational perspective, this breakdown happens for four major reasons:
1. The Fallacy of Independent Paths (Common-Cause Failures)
Designers frequently calculate reliability assuming Path A and Path B are independent. In the real world, they rarely are. They share physical spatial zones, software libraries, power buses, ambient thermal environments, or maintenance crews. When an extreme operational shock occurs, it routinely disables both the primary and the "independent" backup simultaneously.
2. Loss of the Operator’s Mental Model
Every auto-switch, interlock, or trip loop places a layer of opacity between the human operator and the physical reality of the plant. During a high-stress emergency, operators are forced to spend critical minutes diagnosing not just the root physical anomaly, but what the automated safety system is currently doing in response. When instrument displays reflect automated signals rather than true physical state, human judgment is severely compromised.
3. Latent Defects in Idle Systems
Redundant subsystems often spend 99% of their lifespan sitting idle. Because they operate only in emergencies, they are exceptionally difficult to test under true dynamic loads. Over time, these idle systems accumulate "latent defects"—hidden faults, misconfigurations, or degraded components—that lie dormant until the exact moment the primary system fails and dumps load onto a compromised backup.
4. Risk Compensation and Margin Erosion
There is a psychological trap in design and operation: when everyone knows a system is protected by triple redundancy, baseline risk perception shifts. Operating parameters are pushed closer to the red line, maintenance schedules are stretched, and safety margins are quietly eroded under the dangerous assumption that "the backup systems will catch it."
Real-World Proof: Four Classic Disasters
To understand how this plays out in practice, we only need to examine some of the most notable industrial and aerospace failures in history. In each case, safety systems or redundant architectures failed to prevent catastrophe—and in some cases, directly caused it.
1. Ariane 5 Flight 501 (1996) — Software Duplication Fallacy
Just 37 seconds into its maiden flight, the European Ariane 5 rocket veered off course and self-destructed.
The Setup: The rocket was equipped with two fully redundant Inertial Reference Systems (SRI 1 and SRI 2) running identical software inherited from Ariane 4.
The Interaction: A 64-bit floating-point number representing horizontal velocity caused an arithmetic overflow when converted into a 16-bit signed integer.
The Breakdown: SRI 1 crashed due to the unhandled software exception. Control immediately transferred to the backup SRI 2. However, because SRI 2 was running the exact same code, it processed the same data and crashed milliseconds later. Left without flight data, the main computer interpreted the system error dumps as physical movement, deflected the thrust vectoring nozzles to maximum angle, and structural loads tore the rocket apart. Redundant hardware was rendered useless by identical software logic.
2. Three Mile Island (1979) — The Masked Relief Valve
The nuclear accident in Pennsylvania demonstrates how a safety relief system can fail physically while simultaneously deceiving human operators.
The Setup: A mechanical trip on the main feedwater pumps caused heat and pressure to rise in the primary cooling circuit. An automated Pilot-Operated Relief Valve (PORV) opened as designed to relieve pressure.
The Interaction: Once pressure normalized, the control system commanded the PORV to close. The valve stuck open mechanically. However, the indicator light on the control console did not read actual valve position; it merely indicated that electrical power to the solenoid had been cut.
The Breakdown: Believing the light meant the valve was shut, operators assumed the relief loop had worked. In reality, reactor coolant was escaping through the open valve. Trusting the control panel status over conflicting temperature readings, operators manually reduced emergency cooling water injection, leading directly to a partial core meltdown.
3. Boeing 737 MAX MCAS (2018–2019) — Single Point of Truth in Dual Hardware
The crashes of Lion Air Flight 610 and Ethiopian Airlines Flight 302 highlight the danger of flawed software integration overriding redundant hardware.
The Setup: Larger, repositioned engines created a pitch-up tendency under certain flight conditions. Boeing introduced the Maneuver Characteristics Augmentation System (MCAS) to automatically command nose-down trim to prevent aerodynamic stall.
The Breakdown: Although the aircraft was fitted with two redundant Angle of Attack (AoA) sensors, MCAS was programmed to draw data from only one sensor during any given flight, switching inputs on alternating flights. When a single sensor failed in flight and sent erroneous high pitch data, MCAS repeatedly forced the nose down. The software system continuously fought the pilots' manual inputs until control was lost.
4. Deepwater Horizon (2010) — The Dormant Safety Net
The blowout in the Gulf of Mexico illustrates how complex, redundant emergency systems can collect hidden defects while sitting on standby.
The Setup: The primary defense against a well blowout was a 400-ton seabed Blowout Preventer (BOP) stack equipped with redundant hydraulic rams, blind shear rams, and control pods.
The Breakdown: When high-pressure hydrocarbons breached the wellhead and destroyed control lines, an automated emergency backup system (the "Deadman switch") was meant to fire the shear rams to cut the drill pipe and cap the well. Post-disaster investigation revealed that one control pod had a dead battery, while the second had a miswired solenoid valve that went undetected during routine maintenance checks. Furthermore, extreme pressure bowed the drill pipe out of position, causing the shear rams to jam.
The New Frontier: AI-Driven Autonomous Systems
As we move from deterministic software to machine learning and Artificial Intelligence, the Redundancy Paradox is not disappearing; it is changing shape and becoming harder to diagnose.
Engineers building autonomous vehicles, robotic manufacturing units, and smart grid managers often attempt to secure AI decision-making by placing "redundant" AI models in parallel—for example, using a primary deep learning neural network alongside a secondary "safety monitor" model trained on different data.
However, this creates novel interaction traps:
Correlated Hallucinations & Common Blindspots: If two redundant AI models are trained on real-world historical datasets, they inherently inherit the same underlying edge-case blind spots. When exposed to an unprecedented physical condition—such as extreme weather glare or rare sensor noise—both the primary AI and the backup "checker" AI are prone to misinterpret the scene in identical or complementary ways.
The "Fallback" Latency Dilemma: In high-speed physical systems (like autonomous driving or automated power balancing), switching control from a failing primary AI to a backup safety algorithm introduces a delay. During this split-second transition, the physical system is essentially unguided.
Opacities Layered on Opacities: Traditional deterministic systems are already difficult for human operators to interpret during an emergency. When you wrap a "black-box" neural network inside a secondary "black-box" monitoring system, diagnosing why a machine made a catastrophic choice in real time becomes nearly impossible for a human supervisor.
Adding AI layers on top of redundant hardware does not remove human-machine complexity; it amplifies it exponentially.
Where Plant and Equipment Design Must Head Next.
Reflecting on these events and my own decades in engineering design, it becomes obvious that we cannot engineer our way out of complexity simply by adding more control loops, extra sensors, and automatic trip routines.
Modern safety engineering framework models—such as MIT Professor Nancy Leveson’s STAMP (System-Theoretic Accident Model and Processes)—are reaching the same conclusion: safety is an emergent property of the whole system, not merely the sum of its redundant parts.
Industry practice needs to shift away from safety through secondary containment and move firmly toward:
Inherent Safety over Added Controls: Designing processes where hazards are physically eliminated at the physics level, rather than contained by dynamic control loops (e.g., using low-pressure fluid dynamics instead of ultra-high-pressure vessels equipped with complex relief arrays).
Decoupling Subsystems: Introducing physical, temporal, or spatial buffers so that a failure in one section cannot propagate instantaneously across the plant.
Simplicity Over Optimization: Accepting slightly lower theoretical performance or efficiency in exchange for a clean, transparent layout that a human operator can fully comprehend and manually manage during a crisis.
As designers, our ultimate goal should not be to build a system so complex that it requires a dozen safety nets to operate. The true mark of engineering excellence is building a system so straightforward, robust, and understandable that it rarely needs a net in the first place.
