Emergent solutions are good, but some are way better than others. This post is an academic musing on why that is and what solutions it suggests to many organizational problems.
The term emergence is used to describe large-scale results that aren't built in by the local rules, but that end up happening as a downstream effect of those rules being applied locally millions or billions of times. A tornado is an example.
Emergent solutions can be very powerful, and much more adaptive than consciously programmed solutions, as computer scientists are revealing.
To a large extent, the theoretical underpinnings of what economists call "free markets" are based on emergence, on "the invisible hand of Adam Smith", with the explicit assumption that billions of local transactions made by self-serving, self-centered, short-sighted individuals will add up to the best possible outcome.
Given the complexity of modern problems, it seems clear that "planned economies" or top-down driven organizations of any type are becoming increasingly extinct and incapable of responding or adapting to the speed of change around us. The questions are less "who" should be "at the top" as to whether there needs to be "a top", and, if so, what the role is of "the top."
It is clear that the role of the top management is not to know all things and have all wisdom. The old "Theory X" is being replaced by "Theory Y", although not without bloodshed. In large organizations, in hospitals, in the US Army, it is increasingly clear that knowledge of what's going on is increasingly at the front lines or the "bottom" of the organization, not at the top.
Still, we have only replicated on a larger scale the same problem of global versus local. Now the problem in large organizations is "silos", multiple internal departments that, gasp, try to follow short-sighted, self-centered rules or algorithms. It is clear to many in leadership that this type of operation has very clear limits.
So, while emergent solutions may be far better than planned ones, there may also be different kinds of emergent solutions, some of which are far better than others. How do we proceed?
One theme that is visible in corporations, or universities, is the constant cyclical battle between "centralization" and 'decentralization" of services, such as Information Technology (IT) which in universities tends to be about a 7 year pendulum swing from one "side'" to the "other."
Advance accounting systems have been invented that try to "make the global local" by altering the perceived price of goods internally, so that the short-sighted, self-centered ("greedy") decision makers will result, through emergence, in globally optimal solutions. That hasn't worked very well.
So, we have managed to get the planet to this point, but as expansion space runs out globally, we have on many scales the problems of silos - competing groups that each want to try to reshape the world in their own image, and that's resulted in figurative and literal wars.
What to do?
"Homogenization" has been proposed - the idea that, if everyone were like us, it would work. The problem is that the specialization evolved for a reason, and serves a purpose. There is great value for the species and long-term in diversity.
So, we need something that does both at once - a mystical unification of purposes on a global sense, while leaving diversity on a local scene. Are such solutions conceptually possible? Have we ever seen any examples? And how would we know?
One metric is looking at what class of difficulty cannot be solved by a particular set of rules, even used emergently. For example, the "greedy algorithm" solutions may not be able to solve problems such as "tragedy of the commons," where each actor would have to behave a little less selfishly locally in order to achieve a longer-term mutual benefit.
That problem, however, is precisely the problem we are faced with on all sides. The development of, say, an Electronic Health Record (EHR) requires that each department, with its own definitions of everything, will have to yield a little bit and take the cost hit of changing to a set of common definitions of terms, a change that, in the short run, in any one budget year, will be a pure cost with no tangible benefit. Further, much of the EHR seeks to achieve collaboration via dull-witted, least-common-denominator homogenization, precisely the evil that departments evolved specializations and specialized vocabularies to overcome.
So, if "overcoming silos" means everyone has to change to match silo X, you can forget it, unless we're silo X, in which case we agree.
What we need is more of a dynamic, almost fractal EHR format where the complexities of specialized areas are retained within the silo, and accessible within that area as the local working language and reference frame, but are ALSO summarized, and simplified, to something that makes sense, albeit with far less precision, and is shareable outside the silo.
The key issue that has to be solved is that the detailed view and the (over)simplified global view have to be linked, under the covers, so that if one changes, the other changes as well.
For mathematicians, it is similar to a metric where data tensors have rank N at each department, but are represented as reduced rank tensors at larger scale views. The internal complexity has to evaporate gracefully, leaving just the validly simplified essence at each higher level of scale. This is, of course, what we wish any management reporting system would do, but seldom does. In practice, there is over-simplification and distortion at each management reporting level, so that what makes it to the top is close to useless.
Emergent organizational solutions are required, because there is no reshuffling of reporting relationships that can fix that problem, and no restaffing plan with new competencies that will resolve it. What FEMA delivered in Kartrina is what top-down organizations are capable of, period. It doesn't get any better than that. The whole design is broken, the architecture doesn't work, the algorithm is too weak for the task.
So, because of bandwidth, speed, agility, response-time, adaptiveness, and robustness to broken units (i.e., corruption), decision making in organizations has to be pushed out and decentralized BUT, and this is a big but, it still has to have some kind of over-arching central unity and coherence, a coherence that doesn't come from billions of short-sighted, self-centered decisions.
The solution, it seems to me, comes from looking at what I've called the four levels of decision making that networks of mutually interacting feedback-controlled dynamic systems involve. They can battle over data, the first level. They can disagree on the model or framework in which the world is interpreted, the second level. They can disagree on their temporarily assigned goals and interests the third level. And they can disagree on their identity, the fourth level.
Safety cultures and highly-reliable organizations recognize that they have to get above disputing facts to challenging frameworks and mental models. So far so good. Books like "Getting to Yes" describe how to get past locked in "positions" and move to finding underlying common "Interests", which address the third level.
But, this fourth level I suggest is the critical one. For humans, there is a flexibility in identity itself. Some call this a feature, some call it a bug. A person can change, almost in an instant, from thinking of "me" to thinking of a larger "us", whether driven by protection of children, of family, of culture, of a nation in a patriotic effort, or defending the species against alien insect invaders. People can and do willingly sacrifice their body and life for a larger "us" that, at that moment, is really perceived as "me". This gets into the whole social evolution of altruism question, which I'm aware of but not going into.
The relevance is that the peak solutions for innovative, reliable, powerful groups of people require, unambigously, that the people be capable, at least at times, of working as one, of each person actually perceiving that the team as a whole is them, and the interests of the whole group are their own interests, so that it is no big deal, EVEN IF DRIVEN BY GREEDY SHORTSIGHTEDNESS, to give up some asset or resource or take some pain for a different protoplasmic unit (person) who is, at that moment, perceived to be as much me as my own hand or foot is.
This is the state of activity that Professor Kim Cameron and Positive Organizational Scholarshiop at the University of MIchigan are studying. This is a mode of operation that actually works in practice, and has saved billions of dollars on the bottom line when it hits critical mass and takes over.
We need to learn more about that. If there is a way to deal with silos, and to get what the Institute of Medicine calls "microsystems" to work together, it needs something like that, supported with the sort of fractal-tensor IT structure I also touch on above for sharing "data", especially data that are context-dependent, where we need to retain not only the data, but also the context, and where transmitting data involves transforming it validly to account for the different contexts of the viewer and the submitter of the data. We need a "context processor" not just a "data processing" solution, at a minimum.
What we really need is the whole four levels, a data processor, a context-processor, a goal-processor, and an identity processor. Then we can overcome silos without requiring homogenization, let people work in familiar worlds, but still communcate between silos and achieve not only collaboration, but actual ability to overcome "tragedy of the commons" level problems that current systems and approaches cannot.
technorati tags:organiztions, algorithms, organizationtheory, metrics, accounting, EHR, emergence, greed, unity, diversity, unityindiversity, systems, systemsthinking, systemsdynamics, multicellularism, connectionism
No comments:
Post a Comment