Distal causality and the FBI, CIA, NSA, religion, business, public health

We wouldn't see it if it bit us. Most of our detection equipment is broken, and we don't even realize it. Let's start there.

Among all the tacit assumptions we hold, probably the worst one is that our detection equipment is actually functioning. We sort of know that, in the sense that no certified laboratory would ever do a run of samples through a piece of equipment without first calibrating the piece of equipment to prove it was actually working.

This problem was brought home to me and my wife a few years ago at a conference dealing with social support for "self regulation of health behaviors." Already, there's a problem. The issue is that about 2/3 of the USA's health care bill is due to lifestyle choices such as smoking, drinking, exercise, job stress. And, guess what, it's hard to get people to change. So there was a conference looking at what worked and what didn't, with the world's leading experts on the panel.

At the end of it, papers had been presented, and conclusions reached. I asked about the bizarre realization that every single one of the presented had related interpersonal interventions that worked the best, but only published personal actions. So, for example it was far more effective to pay a woman's children $10 per pound she lost, than to pay her. Why hadn't these top interventions made it into the papers? Well, came back teh answer, the researchers didn't know how to compute p-values for multiperson feedback looops, so they just erased that inconvenient data point and left it out of their papers. My jaw must have dropped to the floor.

The single most important, most effective interventions, for the most expensive problem our nation faces, and it was blipped because the researchers didn't have tools to deal with feedback systems, didn't want to talk to engineers to get them, and didn't want to be embarrassed. And we wonder why health care costs are running out of control.

It's the scientific paradigm that's broken here. That's what needs to be fixed. 50 years of systems dynamics still hasn't made it into biostatistical techniques as the new paradigm. This is amazing. Thomas Kuhn must be laughing.

Our problem is that the piece of equipment that's broken is our reliance on detection algorithms that in turn rely on the generalized linear model of causation. As soon as we get into non-linear feedback loops, those assumptions are all broken.

What this means is that we, as individuals and organizations, are almost completely blind to anything involving distal causality. It's hard enough for individuals to start seeing psychosocial factors behind depression, or enemy intent behind some far downstream event X; but then the peer-group resistance not only damps out that dim perception, but pushes back further into our own minds and discounts such perceptions before they even get to our consciousness. We become walking blind and totally unaware of it.

This is a shared problem across the board. In social epidemiology, those suggesting distal causality are more or less shunned as "making the problem too complicated' and "needing to focus." It is "obvious" that bleeding causes death, not bullets, not guns, not terrorists with guns, not a reservoir of anger causing men to become terrorists and obtain and use guns, etc. As soon as that is accepted as a "fact", all the great upstream, high-leverage intervention points fall off the map.

Judging from press releases, it appears that, say, George Bush absolutely cannot conceive of the core systems thinking idea that it is our own actions that are causing the increase in terrorism, our own attitudes coming back to haunt us.

Similarly, the huge social debate between "religion" and "science" is futile until one basic fact is recognized: the equipment we're using is broken. We wouldn't see "God" using linear, reductionistic, "scientific method" techniques if He/She was sitting across from us in the room. It's trivial to set up a basic multiloop feedback scenario and ask someone to make measurements of it and tell you what's going on, and see that, well, they can't do it.

The tool doesn't even work on tiny, easy, synthetic problems where we know the answer because we just designed it as a calibration test case.

So, there are two interlocked problems that we keep smashing our heads into:

1) The linear scientific method has a blind spot when it comes to analyzing complex, active, adaptive systems that involve thousands of cycles of feedback, and

2) The sociology of science, still stuck on the idea that Science has replaced God and is, in fact, godlike, is having a lot of trouble recognizing and admitting that their old model, like everything else in life, is obsolete and needs a new version and an update.

We're in the middle of a huge paradigm shift here, and it's not pretty. Jay Forrester laid out the basic principles of system dynamics 50 years ago, and they're only just now starting to actually get into practice. Part of that, of course, is that we needed huge computing power to manage the task, along with astoudingly good visual graphics, and that didn't arrive until the last decade.

There is, blocking the road, yet another fallen tree that needs to be dealt with, and that is the legacy of Turing and the concept of Turing machines and computability, the holy grail of some computer scientists. Turing modelled "all" computing problems as a little machine that could read an infinitely long tape that had ones and zeros on it.

3) Our problem in this reality is that the tape is noisy and biodegrades pretty rapidly with time and distance. The implications of that problem are pivotal. It means that no amount of processing of linear strings of symbols can possibly work, if accuracy matters. We have most of the best scientific minds of the world tied up trying to get one more decimal place of accuracy on "facts" that don't really matter, that can't really matter, because at least one of them is wrong, so the serial aggregation of them will be wrong. The algorithm sucks.

We already have the workaround for that in our hands, accelerated in hardware, namely, image processing techniques. Images, unlike symbol strings, are remarkably robust against most types of noise, at least when processed by humans. In other posts, I talk about how business has to operate this way in the real world, you know, the one scientists left "for later" when they shut their doors and poured their lives to looking down the microscope instead of out the window.

In any case, the paradigm is shifting, now at an accelerating rate. People are starting to accept that feedback is not only part of the equation we can't get rid of, but that feedback is the part we care about.

In fact, in many fields, such as general relativity, there are two ends of the world that can be examined that are relatively simple, and a middle region which is a mess. (See posts on near-field and far-field approximations of radiation patterns.)

There are "self-terms", where what we call "objects" do something we call "exist." Then there are interaction terms, where the effect of active interactions between objects are described.

The billiard ball, particle model of the world was great because it was relatively simple at the limit where interactions were reduced to a few "collisions", and mostly self-terms dominated. Then came waves, and worse, quantum mechanics, and relativity, and worse, general relativity.

In general relativity, there is feedback between context and content, and the two become smeared out in space and time. Uglier than the flat, linear approximation.

So as we get more complex, the math gets harder. But, just as with behaviors of of a gas or liquid, at the high end where there are billions of interactions, life gets easy again - it's just that we no longer even try to track individual molecules, we just look at bulk properties like "volume" and "pressure" and "flow", that mean nothing to individual molecules.

So, equivalently, we need to get past the ugly middle, and just skip over it, actually, and focus on the high-interaction end of life, where the "self-energy" terms can be dropped out of the equation, and the only thing left is the interaction terms, the synergy, the emergent properties that are, again, simple.

For hierarchies of living stuff, this seems to happen repeatedly across the vertical axis of scale, where, at the next larger scale, things close in on themselves and take on a life of their own, independent of the lower level substrate matrix upon which these upper level soliton waves now have an existence. One thing is sure, and that is that the universal basis of life, the regulatory feedback loop, will be found again at each level. We call the levels below us "alive", and call ourselves "alive." I

t gets really emotional to call levels above humans "alive", and goes off on tangents that distract us, so let me just call them systems with the "MAWBA" property - that is, they May As Well Be Alive. They consume energy, they have goals, they are goal seeking, they are aware of their environments, they have "controller" software that has mental models of the world and measures differences between sensed reality and desired reality and takes an action that, in that mental model, will tend to close that gap, etc.

Mathematically, the frustrating thing is that causality doesn't stay within one plane, but insists on having closed feedback loops vertically, across planes or levels of this hierarchy. If that is facilitated and made to work, we can solve the cybernetic 101 model of sensors feeding a mental model which controls actions which are sensed, and all is good. That's more or less what the Army leadership doctrine attempts to do - establish a safety culture where data challenging the model can flow upwards and make it to the brain.

If the synthesized cyernetic being is fragmented and these phase-lock loops are broken or made unidirectional (which would always be "top down"), the whole model collapses. We are walking blind.

What about the model of simply exchanging messages between silos, where the upper levels decide what is important based on "experts" at layers above who know all and see all? That can't possibly work either. The datastreams are too noisy. This is the Turing tape problem. If people can see all the facts for themselves, if the tank commander can see the whole battlefield, they can do sense-making despite the noise, and "see through" the noise. You can get the same effect with "adaptive contours" or "snakes" in 3-D MRI image analysis, where a coherent balloon type structure is "moved" through the pixel space and, by retaining it's own coherence, can see through amazing degrees of noise at any individual MRI plane and detect the contours of a tumor.

You cannot serialize this process. This process does not flatten. You cannot serialize this process. You do not end up with the same power if you ask each plane to detect the shape of its own contour, only, in isolation, and then have some "higher" power assemble those varying size circles, say, into the outline of a sphere. That does not work. That cannot work. Each layer needs the feedback, live, real-time, from the neighboring levels, to know whether to suspect that the spike it sees infront of it is noise or signal.

You cannot have each isolated subunit "find a dot" and then have the higher up experts "connect the dots." That algorithm is broken. That algorithm assumes that there is a high signal to noise ratio, which isn't true in practice.

OK, where's all that get us. We are faced in business, in intelligence operations, in epidemiology, in daily life, with trying to detect and respond to distal causality. Our current tools are demonstrably lousy, and sociologically stuck there by people who think civilization will end if a better version of the tools is released and they have to go back to school to learn it.

We should join forces and attack this problem together. Science, religion, business, intelligence, homeland security all care about the tools and techniques that will address it.

technorati tags:science, scientific, methods, methodology, feedback, systems, syststemsthinking, casuality, distalcausality, sensemaking, seeing, statisticalpower, business, religion, detection, Turing, computability, imageprocessing, symbolprocessing, noise, SNR