Mining the wisdom of crowds

While I've seen debates and books on "the wisdom of crowds", what I haven't seen is a systematic way to enhance the process. Drug companies develop drugs by finding something, anything, that has a detectable effect they want, then figuring out how to amplify or distill or concentrate that. I think we need to do this for crowd wisdom.

One of my favorite bugaboos is the latest fad in hositals, Computerized Physician Order Entry systems ("CPOE"). This kind of software, we are told, will let doctors enter their drug orders, or other orders, on-line into a handy computer terminal, perhaps a portable device like a high-end cell-phone, where the orders are then whisked off to be completely smartly without any problems due to legibility and doctors' infamously poor handwriting. Further we are told that many errors will be avoided by "decision support" that will help the doctors pick the latest and greatest drug to use, or detect that the doctor is considering a drug to which the patient is allergic, or which will interact badly with some other drug some other doctor has already prescribed for some other ailment. (And older people come in with multiple chronic ailments.) Finally, we are told, everyone wins because the doctor can be guided to select a generic instead of an expensive brand-name drug.

Well, sounds great! The reality is different, and the opinions about why the reality is different cover the spectrum. Many implementations crash and burn, and after a very expensive and ugly scene, the systems are "backed out" and removed. Some systems are installed and measures of outcomes, such as mortality, rise instead of falling. In general, getting staff to use the systems is an "uphill" struggle, which should say something about how much easier it is making their lives, at least in the short run.

Some administrators and vendors believe that "the problem" is "bad doctors" who are "contrary" and refuse to learn how to use a computer and need a swift kick in the pants or to be let go if they don't "get with the program." They see the solution as "bigger whip."

The assumption behind that view is that the software actually works.

And, as my readers will already suspect, I will define "works" as something that can only be specified if you have also specified the context in which this "working" occurs.

It is true, largely, but not entirely, that vendor CPOE software "works" in the sense that if you type in "27" and hit "enter" the system will record "27" somewhere, not "42". That much generally works, but it is remarkable how many cases can be found where even that much doesn't seem to have been tested by anyone before release of the product. Often some "bug" can be found in under 5 minutes of use. Articles in JAMA describe such systems, and generally note many inconvenient but "minor" issues which are often described as something that could "easily be fixed."

Nothing could be further from the truth, or more at odds with actual evidence. This mental model is entirely wrong.

These "small" imperfections are generally in a system that is running in dozens or more hospitals, and that have been running for at least 2-3 years. Yet, they remain unfixed.

Now, this should cause an alert observer to sit up and say, "Wait. Why aren't they fixed?"

The answer, and I base this on personal observation and 40 years in the IT field, is that these systems are largely un-fixable. Or, more precisely, the whole collection of software, vendor representatives, bug reports, bug-fix pricing, departmental prioritization, and hospital-wide prioritization means that 99% of such problems will never "make the cut". Ever.

There is a common flaw in priority scheduling for job shops that fails to account for the "age" of requests, and prioritizes purely on some other factor. The result is a sludge of lower-ranked requests that just grows and grows but never makes it up to the "top ten" or "top three" that will be selected. "We must focus on priorities" is the mantra, which I have attacked before as completely flawed, and largely for this reason.

So, yes, IF this bug fix were considered "top priority" by hospital administrators who pay the bill, or by the vendor, it could, in many but not all cases, be "easily fixed." The problem is that this "IF" will never happen. Never. Ever. So what we get is what we observe - hundreds of inconvenient but maybe not show-stopper problems, just below the "I can't tolerate this pain" threshold, but well above the "I feel the pain" threshold.

A second effect of this social process in which software evolution is embedded is that it quickly becomes clear that

• there are a lot of such bugs, and
• no one seems to care about them, and
• it is pointless to complain because no one ever listens, so
• one should just shut up and figure out some work around instead, while pretending to comply.

In some places people develop entire shadow systems to copy data off a screen onto paper, to do whatever they want using paper, then, when it is all done, come back and enter the results into the computer, bypassing the "helpful" part entirely. The reasons are many, but anyone who has ever fought with "helpful" paperclips or security features in Microsoft Office or Vista knows that this "help" can be a frustrating, distracting, pain in the ass.

Part of what is so frustrating about it is that the system cannot seem to learn anything. It remains as stupid on day 20 as it was on day 1, making the same dumb comments, and not reading or responding at all to my tone and body language and force with which I am smashing down the keys that "Yes, I know that. Any idiot knows that. Why do you keep telling me that?" And. ultimately, the "decision-support" features are shut off because they seem designed to support some kind of decision making that is not the kind we use here, or need.

The rest of the system remains like a stone in the shoe or a key that sticks on a keyboard, a constant nuisance, and a constant reminder that one's own good judgment has a net weight of exactly zero in the big scheme of things. Some novice programmer in Pakistan has more power to design my own work flow than I do, despite obviously never having done this job.

But I need to make a crucial point here. The problem is not that the system is wrong - it is that the system stays wrong and is impervious to social feedback. It is non-adapting.

Well, in some cases, the vendor happily chimes in, the system could be "customized" to fit better, at an enormous cost, which might address the "top priority" items, take 6 months to 2 years, by which time the needs will have changed but the work-order won't. After all that agony, the final result is no better fit than the unchanged system, so there is rapid learning socially to shut up and stop asking for changes.

If the social system of the hospital is sufficiently adaptive, and doctors are very clever, they can figure out work-arounds and make-do with this permanent new pipe across the living-room at a height of 4 feet that they now have to duck under when they remember, every time they cross the room to answer the phone.

It doesn't have to be that way. It's that way because we put up with it being that way.

In that sense, it's a metaphor for social systems and governmental agencies and regulations in general -- someone, somewhere, decided to impose something that might have worked for them, although that isn't clear, but that clearly is more pain than benefit now, but that no one can do anything about and you just exhaust yourself trying so shut up and put up with it.

But that, grasshopper, is not the Toyota Way.

The Toyota Way is "That's a pain in the ass, let's fix it!"

So, how did Toyota manage to do this allegedly impossible task, of fixing the internal legacy stupidities that were encrusting everything and smothering innovation and efficiency?

Not by central planning some huge new system, which would "fix everything."

It was done, to paraphrase the Institute of Medicine, by a billion $1 steps, not by one $billion step.

In other words, they figured out how to let people make very small changes, gasp, on their own authority and judgment call, to their own workplace and work flow. Not expert executives, not hotshot consultants, but, you know, regular people on the front line, "employees" and "staff", secretaries and the guys who pick up the trash and move boxes.

And a process of massively-parallel continuous improvement, in very small steps, compounded over time, took the company where no executive plan and no consultant could.

So, the problem we need to focus on with CPOE systems, or with anything else like that, is how to enable and empower the people at the front, who actually have to endure and work with both the real world and "the system", to modify "the system" to be easier to use. The crowd knows thing they don't realize they know, and can do things, if empowered, that no central planning office would ever think of or could do.

A second amazing feature of the Toyota Way is that there is no "implementation resistance" hurdle. The person making the change is making it because they want to and they think it might work to make completing their task easier, and because they want to try it and see. No one needs a whip to "get them" to "buy into" the change.

Third, the changes are very small, individually. The amount of "disruption" any single change causes is roughly zero, and that's if the change got put in backwards, which will be detected in a day or less, swapped around to frontwards. That is, "undo" is possible and not considered a "failure". And rather than "disruption" the change is, or should be, and can be, "anti-disruptive". The change actually makes things flow more smoothly, not less smoothly, not only for the person making it, but for everyone near them, in the social context in which employees care about the people near them because they have to work with them and people have memories. And, gasp, they might even be asked if this "works for them?" And, gasp, they might even be asked before making the change if this might work for them, or if not, what factor has been forgotten that I can learn right now about how you do your work, so I can take it into account from now on, and learn more about you at the same time, and things I never realized you had to take into account or do.

In other words, the changes are connected to a parallel process of social development of deeper friendships and working relationships with everyone upstream and downstream from my own neighborhood, treated with increasing respect as I understand more about why they do what they do the way they do it, and they learn the same about me.

OK, let's bring this home. Can enterprise-scale software ever possibly have that sort of property that, if you don't like it, you can actually adjust it so it is less painful to use? Or, gasp, more helpful than the designer intended?

Yes. I've done it. More precisely, a team I led evolved some software over time under continual advice and suggestions from users of, couldn't this field be up here instead of over there, because we always are doing this other thing at the same time and that window covers this, etc. The result was called "indispensable" by the users, who screamed if it was not available, but we were baffled as to the question by higher-ups of "what it did" and "what we called it." It had no name, and it was hard to describe exactly what it did, except that it had accumulated hundreds of things that were individually helpful in one place. It was a sort of a swiss-army-knife that dealt with issues that no one had realized were issues and couldn't describe easily in words if they had realized it, but it turned out we didn't need to describe it as a whole to build it, and we didn't need to figure out what the users needed because they told us, slowly, as they realized they actually had some say and control in what it did.

It was a "stone soup" system that kept getting better with additions that the users themselves supplied, or suggested but were "trivial" to add or adjust in under 5 minutes apiece.

It ran below the budget radar because it never got a name as a "project" or got funding.

There's a lesson here.

An important lesson.

And we need to turn back to John Gall's wisdom to see what's going on.

First, solutions that work to large problems are not created in whole cloth, nor do they spring full-grown from the head of Zeus. They start as small solutions to smaller problems, that actually work, and evolve slowly from there, always keeping the property that they actually still work. The reasons are myriad, but have to do with debriefing the social context about information we needed in words we lacked and concepts we lacked, but that could be easily done even if they couldn't be easily described in words. The context is important. And the lack of ability to express this in words is important.

This is not a pre-planned solution, developed in India, to a social context they've never met, with very real people already populating the seats. This is a home-grown solution that grew to meet the exact needs of these exact people doing this exact work.

Well, two problems immediately arise. One, what's the business model here? And two, how general is this concept? Won't the resulting software be totally messy, fragmented, and impossible to maintain or debug or explain to new staff? How can we validate or re-validate that it does what it is supposed to, and only that, when it keeps changing? Isn't this irresponsible and bad practice? And how can we sell the resulting software to anyone else and make money, when it is so specific to this environment? Aren't we just institutionalizing bad legacy practices in code instead of analyzing them and fixing them?

We'll get to those. The key point is that the "wisdom" needed to guide the development of a working system is located in the social context down where the company meets reality, on the front lines. It is not embodied in words, often. Sometimes it defies description in words, even though people do it -- like trying to describe how you crack an egg without making a mess.

Much of the wisdom can only be revealed and surfaced by actually trying something to see if it works or not. Great judgment is not required, only judgment of keeping the experiment small and keeping the "undo" button ready to push if we got it backwards.

This is similar to the successful strategy of Loeb of Loeb Rhodes, or whatever that stock broker is. He described a strategy that was "wrong" 80% of the time and made lots of money. He'd pick a stock he thought should go up, set expectations around that, buy it, and watch it like a hawk. If it turned out to start going the wrong way, he'd bail instantly, not "just wait until it comes back up before I sell it so I don't need to admit that I made a mistake and recognize a loss." So the 20% winners stayed on the table for a long time, and the 80% losers were cut off almost instantly, and the next result was an ever improving portfolio of stocks, chosen not by wisdom of perfect picking, but by the cybernetic wisdom of just keeping your eyes open and reacting to what is actually happening, not what you wish was happening. And small steps, so undo was possible.

This is a remarkably powerful strategy.

It's the strategy evolution has used to get our bodies and society evolved this far, without, apparently, a central planning committee.

Massively parallel, experimental at the edges, alert, aware, not only no hesitation to admit "an error" but spring-loaded to admit an error very very rapidly. No massive brilliance or Nobel prize winning "quants" required on the staff.

This is the power that the current process of CPOE, and for that matter most policy development and governmental development defeats, by trying to move in very large steps that are centrally planned, and frustrating the local wisdom where the concept meets reality so as to suppress it instead of enhance and encourage and harvest it.

This is the difference between successful marketing and product evolution based on real-time customer feedback, and trying to use a bigger whip and ad campaign or discount incentives to sell, say, GM cars that people don't actually want. It is a way to "listen" to the wisdom of the crowd", especially if you need the crowd to make things go.

OK, back to software. What might be done?

First, the whole budgeting process is in the way. Classic budget cycles are way too long, and way too chunky, and require detailed proposals of exactly what is to be done and how much it will cost, etc. There are only two worlds envisioned -- either the software is static, and simply being operated or "maintained" out of operating budgets, or there is some huge "project" attempting some huge change, out of "capital". There's nothing in-between where the sweet spot actually is. We need something that funds "maintenance plus a little"- daily incremental improvement in software, responsive to customer feedback.

Second, software design policies are in the way. Most companies do not use high quality design techniques to make structured, top-down, object-oriented systems that are easy to maintain and change with known results, and easy to revalidate, because that costs more to get the first one out the door than writing crap, and they weren't planning to make very may modifications to the software anyway -- just to ship 200 copies of exactly the same thing to 200 different contexts and suppress the dissent at every one of them, assuring management that one size fits all and, oh by the way, we have a special on larger whips today.

There are good tools to design good software that can be changed and revalidated in minutes, not months, but most people don't ramp up to use them. That, however, is where the investment would do the most good. Instead the money is spent trying to make changes to poorly designed software and revalidate the tangled mess that produces, which is a very expensive process and fraught with unexpected interactions and side effects and retractions, which then motivates the vendors to tell their software people "Just try to not change anything at all - we think maybe we have it working now."

So, I see a process effectively equivalent of this. All day, people use the software, that I picture as a huge object floating in space. As they work, they attach rubber bands to parts of it, pulling them slightly towards slightly different places than they are. At night, every night, while the users sleep, the software people come alive (maybe on the flip side of the planet where it's day), and assess the total net torque on the system produced by all those rubber bands, and first rotate everything to reduce the net torque to zero, then slightly shift the details to bring the gap between the system and individual requests towards zero, even if they can't all be met on this change. The total change is small, and revalidation can be done to be sure it's still on stable ground, since most parts won't have changed at all except around pivot points, which were previously validated.

The process should be, on a large scale, cost effective. The price of not fixing "small things" needs to include the costs of discouraging and squelching user interest in improving things more or any innovation at all, which changes the equation.

I'm sure there are all sorts of principles and guidelines that can be applied to define this process better. The point I'm making is that there is value in capitalizing on evolutionary design processes that actually work. This requires decentralizing the eyes, while retaining control of the central system, but in a way that is maximally responsive to the eyes. And it requires everyone repositioning themselves to assume the new world is dynamic, not static, so don't put your coffee cup right there cause that part rotates.

The invariants that are being invested in here, over time, are processes that capture massively parallel pressures and respond to them, a little bit, every day, in real time. We can get rid of the whips.

Once we admit that knowledge we need, about the social structure and environment, is actually distributed holographically and non-verbally in that environment, we can give up trying to have a three stage process of first having a massive effort to debrief everyone in words as to what they think they do, then going off to India and building some compromise least-common-denominator system, then coming back and forcing it down the throats of staff that are trying to say it doesn't fit. The debriefing and adjustments are done in a massively-parallel way, every day, and nothing needs to be forced on anyone, for the most part.

As with Toyota, much or most of the value here is actually having the people in the company have a structured way to begin to understand more and more what it is the other people are doing and why, and build respect and mutual support and collaboration in improvements.

The initial software that does this, like the rock in rock soup, doesn't need to be perfect, but it has to be small, not over-promise or be over sold, and be extremely dynamic in a stable way.

Dynamic stability is a real concept that airplanes and flywheels and cars need to have. Things change, but within limits so the overall thing doesn't crash.

As most people have found, if you delay repairs until huge changes are necessary, they will almost certainly not fit socially, and generate huge push-back that can be overcome, but at a huge cost. This "big project" way of debriefing social reality and adapting to it has a miserable track record. It institutionalizes the sins of the "we must prioritize!" approach to life's complex problems, which structurally guarantees that the needs of the people "on the bottom" will never be heard, let alone met.

And that forces competition and conflict as people try to push their way high enough up to have one of the few projects or needs or ideas that passes the "let's prioritize" cut-off point.

We need a new model, based on what's been working for a billion years all around us, a proven design pattern.

It has much less risk, much less cost, and gets more people what they need to do their jobs, while not requiring any of us to be Nobel prize winners, or in the clutches of the "suits" from huge consulting companies that promise to deliver "solutions" (made in India) to our "problems", as described by upper management, which by the nature of complexity and speed of change and "indescribabilty" of life is out of touch with the front lines. Oh yes, and "Prioritizing" because changes are so expensive that we can only do a few big ones and the rest will "have to wait." (Forever).

A living, real-time adaptive strategy doesn't need management to detect that the outside environment has changed, order a study, commission a planning group, prepare an implementation agony in order to respond. It can simply detect the change itself and pivot around to adjust to it, and management can read about it later.

The job of management is to realize that real-time control of huge complex adaptive social systems is not something they can micro-manage, and to put in place the kind of internal adaptive skeleton that is response to millions of pushes simultaneously, not a few huge pushes at the expense of everything else.

The world is past the point where "a few top priority items" is sufficient, and the world isn't going back to that point ever again. To cope with that we need to redesign corporate and governmental decision-making processes to be agile and parallel and responsive.

I fret about those with the mental model that "the problem" is all this "push back" and if a sufficiently strong top man were in place that gave and enforced orders with a big enough whip, why then everything would be back to running smoothly, you betcha.

Nope. There'd be no more painful delays in gathering information, and actions could be taken very rapidly -- with total assurance that they would be totally out of touch with today's new reality at the front lines, that wasn't even there yesterday but showed up in response to what we did yesterday, with some whole new set of things going on over here that don't make any sense at all, and half of the responses being, surprise, exactly the opposite of what we had planned -- except that we don't hear about those for a lag time of months, because of the strong dissent-suppression field we're using to get our way despite the potential push-back.

Are you my mommy? What shape AM I anyway?

I'm working on turning my thinking to the practical problem of the economy of Southeast Michigan, the area of the USA near the city of Detroit - on the Canadian border.

Actually, from Detroit, you cross the bridge Southeast to get to Canada, which everyone knows is to the North of the US. It's a perfect illustration of how things that are "true" at one scale can also be "false" at another scale at the same time. In general, on the world-sized map, yes, Canada is to the North of the USA (above it on the map for those with no sense of direction.) At the same time, on a city sized map, Canada is Southeast of Detroit.

Please try it now. Take this Mapquest map and slowly change to larger views by clicking successively lower buttons on the left. Watch as the city of Windsor (and the country of Canada) "move" from the south to the north of Detroit. Don't just think about it -- actualyl do it. Interacting helps the idea come forward in your mind which you'll need in a minute.

Both are true and not in conflict, because the "fact" varies with the size of your map. This kind of "fact" is common, but not discussed in school, and many people didn't pay attention in school anyway.

It seems to turn out to be a huge conceptual error to assume that things that are "true" at one scale must be "true" at every scale. Cultural, governmental, corporate, and personal mistakes due to this single, simple error are responsible for much of the misery we face in life.

I gave another example before, of the visual equivalent - a photograph of, well, either the female movie star of old, Marilyn Monroe, or of the male scientist Albert Einstein. Which it is depends on how far back from your computer screen you are. -- Up close, it is CLEARLY and OBVIOUSLY a picture of Einstein. Walk across the room and look back, and it is CLEARLY a picture of Marilyn Monroe. In between it is just confusing.

The original post was here: "The Sixth Discipline for Learning Organizations." And here's the picture:


(That picture is the work of researche rGregory T Huang, from New Scientist's 31 March 2007 issue at newscientist.com -- subscription required to get to it online.)

Now, imagine trying to achieve any meeting of the minds, or trade agreement, or corporate policy, or an end to conflict between groups sitting close to the screen and another group sitting far from the screen, that depended on what it was showing on the screen. "See, how we help you?" one group might say. "Help us? You're killing us!" the other group might say, and new fighting might ensue, or both groups could walk out of the room because the other group is "being unreasonable."

We've adapted to the fact that which building is larger needs to be adjusted for "perspective", so that now we automatically "see" a tiny skyscraper on the horizon as "being" much larger than a one-story house nearby, despite the fact that on a photograph of the scene or on a TV view of the screen, the house takes up most of the screen and the skyscraper is tiny.

Now, this is a really strange thing, if you stop and think about it, which we normally don't. It's not that our eyes "are broken" or "don't work" at all - our eyes are fine. There's a physical thing going on that changes what we see depending on where we are. The "one world" contains within it millions of "perspectives", as we try to reduce a real 3-dimensional world into a single two-dimensional image. The image is not the world. Same world has many different images, which may "appear" to conflict.

The only conflict is in our confusing the image with the world. The error is in "reducing" the complex world to a simple picture, which has a result that differs depending on where you are.

Note that it's not WHO you are. If you and the other person swapped places, you'd now see what they see, and vice versa. It's the same "you" near the screen with the picture of Einstein and across the room with the picture of Marilyn Monroe. The only thing that changed is where you are looking from.

Our language has terms "different perspective" and "different viewpoint" but they have become twisted around to mean that something inside the person is different, and that those viewpoints would persist even if the person swapped places with us. That's an error in thinking, responsible for much bloodshed.

So, with that discussion of Detroit and Marilyn Monroe in mind, let's turn and look at a recent news article , one of many recently on researchers discovering that "we" aren't at all shaped like what "we" thought "we" were.

And, again, this conceptual error is responsible for many failed policies across the spectrum.

The article is from the August 11-17, 2007 issue of New Scientist, an article by Chris Frith on Determining Free Will. This is a great subject for killing time and getting nowhere - back in the 70's I used to attend a conference out on Star Island of the group IRAS - Institute for Religion in an Age of Science, and we'd spend a week happily arguing about the existence or non-existence of "free will", or temptation, "demon possession", fate, etc.

So, anyway, the article goes on to talk about some presentations at a conference by the John Templeton Foundation on that subject. Frith discovers that in many cases, "we", our conscious selves, seem to actually just be inheriting actions that "our brain" decided for us long before, and sort of tells us about after the fact, when "we" swiftly pivot and act as if, then actually belive, that "we" made that decision and "took" that action.

I have noted, of course, a similar phenomenon among CEO's of companies, and Kings and Presidents, and the "pointy-haired boss" in the Cartoon "Dilbert", who act as if, and come to believe that they are "running things". They get all puffed up with their "greatness" and carried away with taking credit. Recently Forbes had a "Titans of Industry" article where some CEO's were astoundingly arrogant talking about why they, singlehandedly, had "done" such and such and why they, single-bank-accountedly, should indeed receive that bonus of $250,000,000 for the company's success.

Uh, actually, the other 350,000 people had something to do with the success.

But, again, this is the work of "perspective" and "viewpoint" and "scale" -- from the Office of the CEO, it really honestly LOOKS LIKE they are the one doing all the work, against the arrayed forces of decay and opposition and enemies and inertia. Of course, to the people actually building the cars with their hands and labor, it appears that the actual work is being done by them, and the CEO is some pompous windbag totally out of touch with reality.

Thus, management and labor, subject to the subtle and insidious power of "perspective", end up deciding each other must be some combination of demented, evil, uncaring, and stupid -- and productive work drops off, and the company becomes "non-competitive." Management then, to "lighten the burden" of all those excess useless moocher employees, lays off 10% or 20% of "the workforce" (a term whose meaning they have lost, as with the term "labor".) Wall Street applauds the great move and stock prices go up.

Then, curiously, output at the factory goes down for some reason. Management keeps on laying off more and more "workers" but, dammit, there STILL seems to be too much dead weight holding the company back from the efforts to lift it up that management is exerting.

In the extreme case, management lays off 100% of the "workers" trying to make the unit more "productive." I actually saw this happen at Cornell University, when times were rough and the Buildings and Grounds unit laid off 100% of the employees that they sent out to do actual work, for which B&G charged $60/hour, their only source of income. No one in management could understand why this still didn't improve the profitability of the unit.

The illusion at the top that "they" are in charge is very, very strong. It's not an "illusion", but a "perspective" effect, like the size of the buildings. To their eyes, their little 1-story house looks huge in comparison to the labor "skyscraper" that is very far away from them.

Well, it appears, as is so often the case, that important processes are the same across all scales. Some design patterns that work at one scale work at every scale, and those are the ones to focus on first -- and possibly the only ones we ever need to look at to understand what's going on.

So, "you" and "your body" and "people around you" are just the same as the "boss" and the company. In fact, yes, it turns out that most of what "you" end up doing was actually done by some other part of your mind or body, or by the people around you, or even your spouse or employees, when "you" weren't looking. And, in fact, it turns out that most of what "you" think "you" decided to do, because it was "obvious", is also the doing of all these other invisible actors that "you" tend to forget exist.

This repeated finding has startled and baffled some academic scientists, most of whom are deeply committed to the dogma of isolated experts being the only "actors" of importance.

This is a critical realization that impacts our health, the public health, the cost of health care in the USA, the competitiveness of General Motors, the productivity of Southeast Michigan, etc.

In health, for example, Johns Hopkins researchers have found that about 70% of the USA's health care bill is due to "life-style choices" - that is, what we decide to eat, or smoke, or ingest, or whether we decide to exercise or not, or whether we are "compliant" at taking those pills the doctor said we needed, etc.

Actually, the number is a lot higher than 70% if we include the downstream damage from bad choices at work, or "not feeling like" studying, or shutting off our friends and taking that "incredibly low cost mortgage" that now has turned into a nightmare.

While the focus for the last several thousand years has been on what we do and how we behave, and why we do things (belief, attitudes), that is now just at a "tipping point" and starting to change to go back again and revisit that word "we".

After many tens of thousands of efforts to change "people" or intervene in foreign cultures, it's becoming clear that, as soon as the effort stops, "people revert" to their old behaviors. It's also clear that this is due to the larger set of people who "have influence" over the first person, who we had always assumed up to now was the "actor" and the one "making the decision".

Now, that's all turning upside-down.

Now, it appears that many choices are already made for people by the group they are in, so, in fact, it appears that "we" have far less "free will" than assumed. In fact, yes, it appears that sometimes people are "not responsible" for what "they" do because they were "caught up in" a wave of humanity doing something else.

This, of course, as discussed before, totally messes up our concept of morality, and justice, and "criminal justice" and "blame". When Systems Dynamics demonstrates that problems in production aren't due to any person making mistakes, but due to "the system" making a mistake, that doesn't help the argument. When hospitals and airline companies talk about not punishing a person who "makes a mistake" because the "team made a mistake" it further confuses the issue. "What up?" we may well ask.

But, here again our reductionist thinking leads us astray. Yes, it is true that, at a given moment, we may find ourselves, to our surprise, unable to overcome our habit, or tendency to do something, or the impulse to do something that the herd or crowd around us is doing.

That does NOT mean that we are helpless victims, however, because if you change lenses, shift to a larger time frame and scale, you see that we did get to pick our friends, and our job, and the crowd we hang with, and which activities we wanted to pursue, so, in that larger scale we made the bed we are now sleeping in. So, we're responsible again.

But, no wait. It turns out that choice of housing is not a "free choice" but is limited by our language, our culture, our income, social discrimination that forces us to live in an immigrant ghetto, etc. It turns out that our employment is not a "free choice" - or is, more to the point, like "free choice" on network TV or "free choice" of who to vote for for President -- there's not much choice left by the time it's our turn to pick one.

But, no wait, those levels of discrimination are caused by actions that ... etc.

This apparent mess of seemingly contradictory arrows of causality can be quickly and neatly resolved if we simply recognize two things:

1) the near and far worlds are in a feedback loop, each influencing the other. There is no point where you can cut that loop and say "A" causes "B", because you will aso find that you left out the fact that "B" also causes "A". This is confusing to academics not familiar with loops, so they either leave it out of the analysis, or put it in a footnote, or go work on something else.

Sidebar: It's quite remarkable to watch as they erase a data point they don't like with some hand-waving justification for their total violation of intellectual honesty to get the "data to fit." My wife and I attended a conference on "Self-regulation of health behaviors" at the University of Michigan with all the big world-famous researchers - Prochaska, etc. - on a panel. It was clear as they talked that the most effective strategies each of them had found involved group interventions, not individual interventions. They would pay a woman's children $5 for each point she lost instead of paying her and it worked way better. But, I asked in the Q&A, am I imaginging it, or did you each find it's the group that matters, not hte perosn? Yes, after discussion, the panel agreed. Why, I persisted, is none of that in your published papers?
Oh, they said, none of us could figure out how to compute a p-value and do the analysis.

Oh. So, the most successful interventions for the largest health care cost in the country were left out on purpose, because they didn't make sense and fit the model of "an individual actor".

2) To make sense of it, the easiest thing to do is to redefine the shape of a human.

Now, maybe, this is not a profound thing to do. Maybe this is just practical, like treating the sun as if it goes around the earth, not vice versa, because the math is easier and, well, that's what it looks like from here anyway.

But, for "people", it appears that the relevant shape for "a person", or the unit that we're trying to change or understand, is actually a larger shape than their biological body.

In fact, "a person" can be thought of, for planning purposes, more like the combined human in the middle of the mix and the feedback loops out to include the other relevant persons, cultures, and organizations that influence "that person's" behavior and that, effectively, make the choices that we have "ascribed to " or "attributed to" that person.

What I'm trying to do is expand the circle of DNA included in "a person" to cover all the DNA that is involved in the "simple" act of choice or "making a decision." Clearly this includes more than just the DNA inside the skin of the person we're looking at. It includes the DNA of all people, present or past, near or remote, who had or are still having a "controlling" effect on the behavior, attitudes, and choices "this perosn" is making - as well as a "defining" effect on the set of things "this person" gets to choose FROM.

THAT entity, that larger glob of DNA spread out across space, time, and multiple "bodies", is the entity that is actually "making the decision" among "the remaining choices" that "person" has left to them at that time.

We need to stand way, way back in this analysis. Joe may have "no choice" about what happens on this road in life, except that he did select this road back at the last corner and that was his choice. OR was it. Or, more to the point, WHICH JOE are we talking about?

The little Joe, with one skin and one set of genes and DNA? Or the bigger META-Joe, with many people pushing for something, many hands on the steering wheel, many TV ads resonating through his head and shaping his perceptions, etc?

The bigger Meta-Joe seems to be the smallest unit that gets back to being a "causal" actor, but it's not clear that there is an edge here. Because everyone of THOSE other people is tied in a feedback loop to a larger set of people, etc. ,etc.

IN some very real sense, WE ALL are Joe, and Joe is US. The choices he makes, given the choices he has made, given the people pushing on him, given the resources he had available, given the choices he had to work with where he was, all come home to roost in the current "action" and "choices".

Many prior actions are now encoded in habits and patterns of thinking. Many are encoded and stored and persisted as the house or apartment or neighborhood he lives in. Many are now encoded in the friends he hangs with, his job, his boss, his education, his language and culture and sub-culture.

So, 2000 BC to 1900 AD or so, we had a clear model that people were little separate beings, who occasionally interacted. Yes, there was some strange anecdotal talking about marriage as being a "becoming one flesh" but that was just flowery language. Yes, there was talk of "possession" and temptation and not being able to control oneself, but that was ascribed to demons, devils, and maybe bad blood or bad company.

After World WAr II, the psychological warfare crowd realized that it was possible to change people's behavior by simply changing the advertising messages they received, and the unemployed psyop crowd became Madison Avenue, and started trying to control the shopping, eating, driving, buying, and voting habits of Americans - sometimes with startling success.

Finally, this century, we have the computing power and the concepts to start actually getting our hands around swarms of actors interacting, and stop discarding these data points, and look at how things actually come down.

The bad news is that it will appear, as with the New Scientist article, that much of our lives is determined outside "us". The good news is that, with another click of the microscope stage to a larger view, we see that, in fact, "we" get to pick our friends, our city, our job, what we watch on TV, whether to have a TV at all, what country to live in, etc. So, it's not a simple "either or", but a "both".

Yes much of what "we" do is influenced by others, if not actually determined 100% by others, or our subconscious, at this given moment. Longer term, though, we get our turn to determine which others are significant to "us". There is a feedback loop, which causes identities to merge, and we become more "waves" than "particles" -- there is no "me" and "you" there is only "us".

This isn't a "bug" -- it's a "feature". This is what will allow us to interact and build a new society based on the model our body uses to hold ten trillion cells together and act as one person.

It's also a terrifying thought for those who have a mental model of being "above the rabble" and higher class, not subject to this sort of weak-willed nonsense that the poor seem so prone to. As the housing crisis shows, foolishness spanned the rich and the poor equally.

It does suggest, however, that our prisons are filled with people who had less choice in their actions than our penal system considers. It suggests our CEO's are way over-paid, and should be more thankful for their labor force.

It suggests we can get either "active strength" from hanging with the right crowd, or "active weakness" by hanging with the wrong crowd.

It suggest that we don't live in this moment alone, but our second grade teacher's training on us is still operating today, which means, in that sense, she is still alive even though her body isn't.

This is, indeed, a "paradigm shift." It is inexorable now. Having seen it, it won't go back in the box. We're going to have to learn a whole lot more about getting along with "other people" because, it turns out, "they" are actually "us".

If we want to "control" how "we" act in the future, we do have levers to do that -- it's just that the levers look like changing the people around us and how "we" interact with "them".

Just because our internal self has only 5% control of what "we" do at any given moment doesn't mean that's not enough to get us anywhere we want to go. It just takes time and persistence and recognition that much of the "self" we need to change is outside our skin, not inside it.

So, bottom line - what is it? Are we "free" or not?
Well, it's like the Einstein/Monroe picture - - it's BOTH, depending on what SCALE you look at the problem with.

In the short run, over a few minutes or a day or week, perhaps, no, we are not very free, and have almost our whole life determined by "outside influences", constraints, and habits. In the long run, over years, we are pretty much free to determine our life, if we manage to get linked up with enough stability, perhaps a religious institution or a scientific one, so that we can actually make long-term plans and stick with them, despite local setbacks.

As motivational speaker Tony Robbins puts it -- "We overestimate what we can do in a year, but we underestimate what we can do in a decade. "

Amen.

We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time.
T.S. Eliot