Hypnotized in high places - Northwest Flight 188

( picture of vulcan cockpit from u07ch on flickr -- Click for larger view.)

So, yesteday, it seems that a Northwest flight #188 overflew its destination city as the FAA attempted desperately to reach it. According to the NY Daily News,

Crew members aboard Northwest Airlines Flight 188 told the Federal Aviation Administration they were distracted during an intense discussion over airline policy and lost track of their location in the bizarre Wednesday night error.

I wrote a comment to ABC News, after reading the other 200 or so comments, as follows (spacing put back in for clarity and ease of reading).

====

There are always multiple levels of contributing factors, from personal to procedural to crew interaction to cockpit design to job design to corporate policy to FAA policy.

For safety, versus lawsuits, it's worth looking at each level of that nested hierarchy of contexts to seek ways to reduce the odds of this type of thing recurring. If we do that, we don't need to know for sure what happened -- only what might have happened that we are now aware of is a gap in our current system that is relatively easy to fix without side effects.

I think the context of the discussion could be expanded in two ways, both of which involve asking "What other events is this event like?" in a much larger framework.For example, I note an uncanny resemblance between this situation and the behavior of CEO's and government regulators as the nation financial situation flew up to the red line, and past it, while thousands of people screamed and called for attention below, and those above seemed to be ... asleep? ... arguing?... out to lunch?

This is not just situational unawareness, it is unawareness or a shared-delusional-mesmerized state that cannot be broken into by repeated efforts from outside and below, in the corporate and governmental boardrooms.

I'm not saying that just to b####, although b####ing can be fun -- I'm saying that human beings, even those with superb qualifications in isolation, can manage, collectively, to get themselves set up so that those "above" are completely and thoroughly "cut off" from input and flying blind or simply not flying at all anymore.

Again, not as legal blame for this accident, but as a route to understanding "what goes wrong with human interactions", this event could spur us to look at that much larger question, asking seriously, "No, seriously, how could THIS KIND OF THING ever actually happen?"

The truth is, socially, it happens A LOT.

There is something structurally seriously wrong with our mental model of how a hierarchical command structure ACTUALLY functions versus how we IMAGINE it to function.The lives destroyed and lost on a corporate and national level from THIS KIND of error are far more than the lives lost in this latest incident (zero).

Wade_AA

====
(picture by aeneastudio on Flickr)

Other observations I've made about structural blindness and delusional-mesmerism in high places:

Why we have so much trouble seeing

Why are so many flights delayed? The circle of blame


Model induced blindness and FEMA

It's a year since Katrina made it obvious that people watching CNN knew more about what was going on top government officials.

We have to ask how that is even possible. It defies our intuition, although not our experience, which is interesting.

While the "blame-game" remains in high-gear, Systems Thinking leads us to discount the obvious "bad people" and look for deeper root-causes in the social structure. FEMA Director Brown has been replaced, but the systems problems are harder to see and may still be there....

The power of delusion

It is an astonishing fact of life, which the Times article reveals, that the desire for life to be simpler is so powerful that it can cause 10,000 "trained" scientists, with PhD's, to take 30 years to finally collectively observe what others outside their mutual-blindness-field already knew.

As I've said, textbooks such as "Feedback Control of Dynamic Systems" are in their 5th editions in Control System Engineering, but biologists, and much of public health's biomedical research community, discount that literature to the point of invisibility and effectively treat it with contempt. To them, this literature does not exist. When seen, it "comes as news to them", and is promptly forgotten, because it conflicts with the shared myth of their culture, and cultural myths always win out over boring contrary evidence.

The Way Things Are (The "Yarn Harlot" tells it like it is, beverage alert!)

There are some truths. Things that just are the way they are, and no amount of desperate human optimism will change them. Allow me to demonstrate.
The guys showed up with the new stove. I went out front to meet them....

OK, Seriously... WHY didn't we see it coming?

High-reliability human systems have been studied extensively, from Chernobyl to The Bay of Pigs to Challenger to aircraft cockpit teams to hospital surgical teams to the US Army Leadership Field Manual. The answer always comes down to the same thing -- dissenting views need to be heard, and dissenters need what Harvard Professor Amy Edmondson calls "Psychological Safety" or they will wilt and become ineffective. This is how humans always behave and unless steps are taken it always breaks along this fault line.

The right question then should be, who is going to take charge of seeing that those steps are taken and that level of social literacy achieved?

I can't emphasize enough how much more important this is than more math and science, in the absence of this. As T.S. Eliot said, we repeatedly get burned "dreaming of systems so perfect that no one will need to be good, but the man that is will shadow the man that pretends to be."

Much of my weblog is about what we really need to do to avoid such errors in judgment.

====
Various related posts:

My 40 page multilevel structural analysis of
the Crash of Comair 5191 crash in Lexington KY. August 2006
with extensive links to source materials

related webpost with links to Comair 5191 cockpit voice recorder transcripts.


Information on the investigation of the crash of Continental flight 3407 in Buffalo, NY Feb 13, 2009, from the Buffalo News.

On Thursday, Feb. 12, 2009 at 10:20 p.m., Continental flight 3407, en route from Newark, N.J., spun from the sky and crashed into a home as it made its approach to Buffalo Niagara International Airport. All 49 people on board the plane were killed, as was one man in the house in Clarence Center. It was the worst aviation accident in Western New York history.

...A moment later, the co-pilot, Rebecca Lynn Shaw, complained of her own inexperience.

"I've never seen icing conditions," she said. "I've never de-iced. I've never seen any. I've never experienced any of that. I don't want to have to experience that and make those kinds of calls. You know I'd 've freaked out. I'd have like seen this much ice and thought oh my gosh we were going to crash."

Moments later, the crew lowered the plane's flaps and landing gear, and the plane quickly encountered trouble.

Things we have to believe to see

Why men don't ask for directions

Pisa/OECD - Why our education stresses the wrong way of seeing

Failure is perhaps our most taboo subject (link to John Gall Systemantics)

Active strength through emergent synthesis

US - Economy of arrogance (and blindness)

Virtue drives the bottom line - secrets of high-reliability systems

High-Relability Organizations and asking for help

Secrets of High-Reliability Organizations (in depth, academic paper)

High-Reliability.org web site

Threat and Error Management - aviation and hospital safety

Failure is perhaps our most taboo subject (link to John Gall Systemantics)

The importance of social relationships.

Houston - we have another problem (on complexity and limits of one person's mind)

Institute of Medicine - Crossing the Quality Chasm and microsystems (small group teamwork)

Here's a few quotations from MIT Professor John Sterman's textbook "Business Dynamics".

Many advocate the development of systems thinking - the ability to see the world as a complex system, in which we understand that "you can't just do one thing" and that "everything is connected to everything else." (p4)

Such learning is difficult and rare because a variety of structural impediments thwart the feedback processes required for learning to be successful. (p5)

Quoting Lewis Thomas (1974):

When you are confronted by any complex social system, such as an urban center or a hamster, with things about it that you're dissatisfied with and anxious to fix, you cannot just step in and set about fixing things with much hope of helping. This realization is one of the sore discouragements of our century.... You cannot meddle with one part of a complex system from the outside without the almost certain risk of setting off disastrous events that you hadn't counted on in other, remote parts. If you want to fix something you are first obligated to understand ... the whole system ... Intervening is a way of causing trouble.

IN reality there are no side effects, there are just effects.

Unanticipated side effects arise because we too often act as if cause and effect were always closely linked in time and space. (p 11)

And, this crucial comment by Sterman, reflecting the same observation by persons such as John Maynard Keynes.

Most of us do not appreciate the ubiquity and invisibility of mental models, instead believing naively that our senses reveal the world as it is (p16).

My additional note on this crucial insight. The reality is that the world, as it shows up on the mental TV screen we watch, is NOT the world that is actually out there. It has been more than rose-tinted by our brains. It has had entire chunks of the scene edited out entirely, and other chunks that "should go there " put in their place. A whole set of things that have given us pain or conflict in the past have been summarily removed, without so much as a place-holder left where they were. A set of things we hope might be true have been "helpfully" added to the scene. People's behavior, where it deviated from what we expected, has been "corrected" to show us them acting "the way we KNOW the are", not the way they actually are.

We are, in other words, flying almost entirely blind. We have papered over the front and side windows of our cars with pictures of the way we WANT the road to be, and are driving and turning the steering wheel based on those internal delusions.

Throughout evolution, this has been useful to reduce the immense fire-hose of data to a smaller set we can live with -- and, if we do a bad job of managing it, heck, we just die off and don't reproduce and others who do better jobs have children and go on. No big deal.

The problem comes when those living in such delusional and self-confirming, often self-congratulory worlds are given the power to rule our communities, our corporations, or our governments and they continue onwards believing that what shows up on their mental TV screens IS in fact what is going on out there, and believing, therefore, that those voices of dissenting views are, in fact, some kind of misguided or enemy action that should best be suppressed, shut out by locked cockpit doors or isolated fortress war-rooms, etc.

I'm not saying that solving this problem of filtering the fire-hose of complexity down to a size we can comprehend and use as a guide for steering is an easy one -- but I am saying that it is the kind of hard, complex problem that can yield its secrets to methodical research and study, and it is THAT research we desperately need at this time in our lives on Earth.

This is where it is breaking.

This is where we need to fix it.

Well, at least, that's what MY internal mental TV is showing me right now as the "obvious truth".

Wade

New life forms from Synthetic DNA - Washington Post

The Washington Post today deals with "Synthetic DNA on the brink of Creating New Life Forms." Talk about children playing with matches... Rick Weiss begins " It has been 50 years since scientists first created DNA in a test tube..." I'd add - it has also been 50 years since Jay Forrester's classic piece on "unintended consequences."

Here was my reply:

wade2 wrote:

Bio-error indeed. Maybe error-gance is the bigger threat, and very real. Our social approach to low-odds of very-high-risk accidents, as Carl Sagan pointed out re return of samples from Mars, is completely overwhelmed by our normal intuition. At Los Alamos, the first atomic bomb was tested when only a minority of the scientists on the project (something like 6 of 14) thought it would detonate the earth's crust and explode the entire planet. No one was sure, so they tested it. Hmm.

Good books like "Lethal Arrogance" by Dumas and "Normal Accidents" by Perrow detail hundreds of examples of our tendency to run it till it breaks, and then, only then, stop to think.
The tools to even begin to think about the way coupled feedback-loops get their job done, such as System Dynamics, have languished for 50 years. MIT's John Sterman, in "Business Dynamics - Systems Thinking and Modeling for a Complex World" , details the lack of correct intuition, even for the MIT community, brighter than most. PhD's don't generally help, and most of us have less to work with.

So, at best we can model and simulate, which has been done at the Santa Fe Institute for the last few decades, with "artificial life" - virtual life and virtual DNA, genetic algorithms breeding and evolving, to see what happens. http://www.santafe.edu/ describes the work of many Nobel Prize winners.

In short (1) the little buggers are far smarter than we are and (2) parasitism evolves almost instantly in every case. The lesson of the movie Jurassic Park is a mild taste of the tenet "Life will find a way."

If the rest of our human affairs were measured and mature and stable, this would be a risky business. Having unstable tyrants convinced they must "master" this technology and use it to attack others, or defend from attack (exact same research), leads to the Russian model of stockpiling hundreds of tons of Anthrax or worse, in delusions that bio-warfare would be controllable or could be "won".

There are good odds the viruses and fungi and insects will win, not so good for humans.

Life is built with interactions with emergent properties on multiple levels, and we tend to think of "machines" at one level with only one function. But genes don't work like machines, they work like cooperative swarms.

Bio-warfare research has a "life of its own" that should already put us on alert that it is way easier to create things that "might as well be alive" than we think. Since we cannot stop it, we are committed to trying to get ahead of it and get the reins back, which means we should pour billions into understanding the world that the Santa Fe Institute has pioneered - massive interactions, how they go good, and how they go bad.

It becomes clear very quickly that, with complex systems, by the time you realize you "shouldn't have done that" it's too late. Experience is something that comes just after we need it.
For very high-stakes mistakes, that's too late. If we keep gambling with the whole planet on the table, sooner or later we'll lose one turn.

One is all it takes.

12/17/2007 6:07:22 AM
=========

Actually, all the research on high-reliability systems like nuclear power plant control rooms show that the maturity of the social system is what makes or breaks the technology-based system. Psychologically safe environments are needed for people to raise their hand, without fear of reprisal, and question what the heck is going on.

What we have instead is a whole culture used to using fear as a workplace and political context to "get things done", as described by Harvard Professor Amy Edmondson.

The Shuttle Columbia (picture at left) exploded because of an "o-ring" problem, that all the project engineers knew about, and had in fact gone in that day to tell the boss to tell the White House that it was too cold to launch safely. They all lost their nerve under workplace pressure to "deliver" so the Pres could talk to an orbiting teacher during the State of the Union address. She did, in fact, leave a message for us (picture at left) of what happens when we don't listen -- but, I guess we're still not learning that lesson.

Further reading

The classic paper in this field is Jay Forrester's congressional testimony:
"The Counterintutive Behavior of Social Systems",
https://mail.jhsph.edu/exchweb/bin/redir.asp?URL=http://web.mit.edu/sdg/www/D-4468-2.Counterintuitive.pdf

Quoting the abstract:

Society becomes frustrated as repeated attacks on deficiencies in social systems lead only to worse symptoms. Legislation is debated and passed with great hope, but many programs prove to be ineffective. Results are often far short of expectations Because dynamic behavior of social systems is not understood, government programs often cause exactly the reverse of desired results.

Another quote from the Washington Post article is this:

"We're heading into an era where people will be writing DNA programs like the early days of computer programming, but who will own these programs?" asked Drew Endy, a scientist at the Massachusetts Institute of Technology.

How true that is. I've been programming computers for over 40 years, and agree that the programs they write will be exactly like the "single-threaded" programs that mess up our airline reservations and everything else. In fact, a look inside some place like a hospital reveals the workings of the multiple legacy computer systems cobbled together in absence of any fundamental theory at all of how many interacting things should be structured in order to be reliable. Thirty years of research in computer science on "distributed operating systems" and how to build reliability in has had close to zero impact on the quick and dirty, cut-corners-now-and-we'll-debug-it-later model that vendors find locally profitable, but that always breaks down, producing, ta da!, more profitable rework. As a business model it's very popular; as a way of getting reliability, we all have seen the results. This is the culture we expect to "program" our genes? I'm not rushing to sign up.

The article quotes someone on the "unprecedented degree of control of creation" that the DNA technology gives us. Right. This is about the degree of "control" that a Labrador Retriever on your lap in the car at rush-hour has -- yes, it can turn the steering-wheel, but I wouldn't use the term "control" for what happens next. If you think our economy and business development and health care system are "under control", then maybe you would think genes could be "controlled" the same way - and they can, with about the same results.

Sadly, control requires maturity and depth of understanding, instead of simply strong muscles and a short attention span. I wish it were our strong suit as a nation, but see little evidence that it is, or even that it is valued or desired as a long-term goal.

We have instead young children playing with the cool gun they found in daddy's nightstand.

Oops.

======= Some after-thoughts:

Unlike the video games and computers this generation grew up with, life does not always have an "undo" button.

The core task of a civilization is to capture the wisdom we finally learn too late, and get it into a form that modifies the behavior of the next generation so those same lessons don't have to be learned all over again.

The hardest part of that task is that the next generation typically doesn't want to take advice from old people about situations the village elders seem way too concerned about - like, not going into debt over your head, you know, crazy stuff like that.

George Santayana said "Those who cannot learn from history are doomed to repeat it." I'd modify that slightly and add "Those who cannot learn from near-misses will someday not miss."

Each time we don't learn this, as a society, the costs go up. The biggest unknown in "the Drake Equation" about odds of there being other intelligent life in the galaxy that we could detect with radio is how long a civilization survives after it has gotten to the point where it has that much technology. The complete absence of any detectable signals from 100 trillion worlds "out there" suggests this is a pretty small number of years -- maybe under 200 years.

At the rate we're going, we're heading towards adding one more point to that data set.

Learning how to learn from our mistakes and our own past seems to be as important a problem as global warming, but actually more urgent, because time is running out a little faster on the 400,000 ways, besides global warming, that we can end human life on the planet.

Humans are remarkably inventive, and if every weapon and sharp object on the planet vanished, they'd find ways to attack each other with stones. Instead of tackling each symptom like global warming or genocide or terrorism, it would seem wiser to track further upstream and find the root-cause problem for why people are driven to fight, and fix that.

======================================

More further reading:

On High Reliablity organizations, which are sobering. They try really really hard to not have accidents, and still don't succeed from time to time:

http://www.highreliability.org/

I'm sure the US military tries very hard to keep nuclear weapons under control. Even that intense level of attention isn't enough to do the job 100% of the time, illustrating John Gall's law that "complex systems simply find complex ways of failing."

"Honey, I lost the nuclear weapons"

The US National Institutes of Medicine on how much the social relations of the front-line teams matter when your job is to get reliability in hospital care:

Crossing the Quality Chasm and other links

=========================
Photo credits :
Oops (car) by estherase
US Space Shuttle by Andrew Coulter Enright