weblogs wikis and spying - NY Times

Here's some excerpts from the Dec 3 NYTimes on open-source spying, that is, web 2.0 technology applied to information acquisition, fusion, and sense-making.

Following those i have some comments on prior work in this field by David Shepherdson, (see links), myself, and others. Summary - i agree 1000% with Dale Meyerrose.

First this:

New York Times on-line

December 3, 2006

Open-Source Spying

By CLIVE THOMPSON

(selected excepts from the 10 page article):

By way of contrast, every night when Burtonwent home, he was reminded of how good the everyday Internet had become at connecting dots. “Web 2.0” technologies that encourage people to share information — logs, photo-posting sites like Flickr or the reader-generated encyclopedia Pediatric — often made it easier to collaborate with others. When the Orange Revolution erupted in Ukrainein late 2004, Burtonwent to Technorati, a search engine that scours the “blogosphere,” to find themost authoritative blog postings on the subject. Within minutes, he had found sites with insightful commentary from American expatriates who were talking to locals in Kievand on-the-fly debates among political analysts over what it meant. Because hand his fellow spies were stuck with outdated technology, they had no comparable way to cooperate — to find colleagues with common interests and brainstorm online.

Burton, whoas since left the D.I.A., is not alone in his concern. Indeed, throughout the intelligence community, spies are beginning to wonder why their technology has fallen so far behind — and talk among themselves about how to catch up. Some of the country’s most senior intelligence thinkers have joined the discussion, and surprisingly, many of them believe the answer may lie in the interactive tools the world’s teenagers are using to pass around YouTube videos and bicker nonlinearity their favorite bands. Billions of dollars’ worth of ultrasecret datanetworks couldn’t help spies piece together the clues to the worst terrorist plot ever. So perhaps, they argue, it’ s time to try something radically different.Could blogs and wikis prevent the next 9/11?

----

To disrupt these new plots, some intelligence officials concluded, Americanagents and analysts would need to cooperate just as fluidly — trading tips quickly among agents and agencies. Following the usual chain of command could be fatal. “To fight a network like Al Qaeda, you need to behave like anetwork,” John Arquilla, the influential professor of defense at the Naval Postgraduate School, told me.

It was a fine vision. But analysts were saddled with technology that was designed in the cold war. They now at least had computers, and intelligence arrived as electronic messages instead of paper memos. But their computers still communicated almost exclusively with people inside their agencies. When the intelligence services were computerized in the ’90s, they had digitally replicated their cold-war divisions — each one building a multi million-dollar system that allowed the agency to share information internally but not readily with anyone outside.

These systems have served us very well for five decades,” Dale Meyerrose told me when I spoke with him recently. But now, he said, they’re getting Anthe way. “The 16 intelligence organizations of the U.S. are without peer. They are the best in the world. The trick is, are they collectively the best?”

Last year, Meyerrose, a retired Air Force major general, was named the chief information officer — the head computer guy, as it were — for the office of the director of national intelligence. Established by Congress in 2004, the D.N.I.’s office has a controversial mandate: it is supposed to report threats to the president and persuade the intelligence agencies to cooperate more closely. Both tasks were formerly the role of theC.I.A. director, but since the C.I.A. director had no budgetary power over the other agencies, they rarely heeded his calls to pass along their secrets. Soothe new elevated position of national-intelligence director was created; ever since, it has been filled by John Negroponte. December's, Negroponte hired Meyerrose and gave him the daunting task of developing mechanisms to allow the various agencies’ aging and incompatible systems to swap data.

. But bureaucratic obstacles were just a part of the problem Meyerrose faced. He was also up against something deeper in the DNA of the intelligence services. “We’ve had this ‘need to know’ culture for years,” Meyerrose said.“Well, we need to move to a ‘need to share’ philosophy.”

. In his essay, “The Wiki and the Blog: Toward a Complex adaptive intelligence Community,” Andrus posed a deceptively simple question: How did the Internet become so useful in helping people find information?

Andrus argued that the real power of the Internet comes from the boom in self-publishing: everyday people surging online to impart their thoughts and views

Spies, Andrus theorized, could take advantage of these rapid, self-organizing effects. If analysts and agents were encouraged topost personal blogs and wikis on Intelink — linking to their favorite analyst reports or the news bulletins they considered important — then mob intelligence would take over.

Fingar and Wertheimer are now testing whether a wiki could indeed help analysts do their job. In the fall of 2005, they joined forces with C.I.A. wiki experts to build a prototype of something called Intellipedia, a wiki that any intelligence employee with classified clearance could read and contribute to.To kick-start the content, C.I.A. analysts seeded it with hundreds of articles from nonclassified documents like the C.I.A. World Fact Book. In April, they sent out e-mail to other analysts inviting them to contribute, and sat back to see what happened.

Yet Intellipedia also courts the many dangers of wikis — including the possibility of error.

The blog seemed like an awfully modest thing tome. But Meyerrose insists that the future of spying will be revolutionized as much by these small-bore projects as by billion-dollar high-tech systems.Indeed, he says that overly ambitious projects often result in expensive disasters, the way the F.B.I.’s $170 million attempt to overhaul its case-handling software died in 2005 after the software became so complex that the F.B.I. despaired of ever fixing the bugs and shelved it. In contrast, the blog software took only a day or two to get running. “We need to think big,start small and scale fast,” Meyerrose said

There was never a tipping point — “never a moment when two people who never knew each other could begin discussing something,” as Clay Shirky, a professor at New York University who was hired to consult on the project, explained to me. For the intelligence agencies to benefit from “social software,” he said, they need to persuade thousands of employees to begin blogging and creating wikis all at once. And that requires a cultural sea change: persuading analysts, who for years have survived by holding their cards tightly to their chests, to begin openly showing their hands online.

The premise of spy-blogging is that a million connected amateurs will always be smarter than a few experts collected in an elite star chamber; that Wikipedia will always move more quickly than the Encyclopedia Britannica; that the country’s thousand-odd political bloggers will always spot news trends more quickly than slow-moving journalists in the mainstream media.
For all the complaints about hardware, the challenges are only in part about technology. They are also about political will and institutional culture — and whether the spy agencies can be persuaded to change.

Today’s spies exist in an age of constant information exchange, in which everyday citizens swap news, dial up satellite pictures of their houses and collaborate ondistant Web sites with strangers. As John Arquilla told me, if the spies do not join the rest of the world, they risk growing to resemble the rigid, unchanging bureaucracy that they once confronted during the cold war.“Fifteen years ago we were fighting the Soviet Union,”he said. “Who knew it would be replicated today in the intelligence community?”

My comments

First, numerous posts advocating use of web-2 technology, web logs, wikis, etc. are on the site of W. David Stephenson on his Homeland Security site. (see links in this weblog or click here. ) See particularly: Fostering emergent behavior by a networked homeland security strategy: chaos or community. a talk to the International Conference on Complex Systems in June 26, 2006.

Even in 2002, at Infowarcon, Stephenson was sounding this message

These are all reflections of obsolete thinking:

* creating linear, hierarchical practices

* working in isolation

* and devaluing the insights of those from outside our organization or who are low on the organization chart.

Instead, our response to terrorism must be marked by three diametrically opposite characteristics.

First,
we must think cyclically. We paid a terrible price for outmoded procedures that forced people with critical information to route it through a ponderous chain of command designed more to weed out information than speed critical knowledge to everyone who needed it ASAP. So much of the terrorist threat is so unprecedented, so alien to our traditional approaches, that we must instead act quickly, then revise procedures on the fly based on immediate feedback.

Second,
we must link everything. Agencies that have guarded their proprietaryinformation and prerogatives instead of sharing it are intolerable.

Finally,
and most important, we must empower individuals -- in fact, every individual. Preventing and/or responding to terrorist attacks is too great a challenge for government by itself: it must be everyone's job.

He has many more posts indexed on his site that are worth reading.

On my own weblogs, the original "Reticula: weblogs, wikis, and public health", currently renamed "Systems Thinking in Public Health" has a number of posts on this topic, as does this current "Prespectives in Public Health" at http://newbricks.blogspot.com.

I have a somewhat stronger thesis than even Meyerrose or Stephenson, namely, that what we are looking at here is not data fusion but cognition fusion. The point isn't to get some abstract data structure into a particular shape, but to get the cognitive perception process into a particular flow. This is a form of "aperture synthesis", made famous in radio astronomy, where many small radio telescopes could be mathematically merged into one huge virtual receiver with far better resolution than any dish.

This is a brain model, by the way. The brain, after all, has no "master neuron" to which all other neurons report. Human vision does have specialized systems to compute texture, to find straight lines, to connect the dots in aspects of images that vary coherently, but these do not feed into any master center where they are combined and reviewed by the expert neuron -- there is no expert neuron. There is no master center. The findings of each subsystem are shared with all other subsystems in a rapid-fire continuous feedback loop, and the "picture" of what is out there "emerges" collectively.

This model of vision or governance, without any "top person" or "CEO" or "president" or "Director", flies in the face of our machine-age based intuition, unless you've been studying how human vision works.

Posts on the subject:
on aperture synthesis: Houston, we have another problem (Nov 7, 2006, originallyposted Feb 17, 2006 here.)

Blogosphere as Synthetic Social Vision System (Oct 10, 2005)

Seeing Large ideas through small keyholes (Oct 11, 2005)

My weblog: Might as well be alive on meta-life organisms and multilevel architectures.

Positive Organizational Scholarship - making the impossible possible (U of Michigan, Kim Cameron) - getting people at DOE to work together that everyone said couldn't be done.

High reliability organizations

Threat and Error Management (aviation lessons)

US Army Leadership Field Manual FM22-100 (where the doctrine of "integrity" and breaking the top-down information mentality is incorporated into learning organization practices.)

photo by genista.