Friday, July 21, 2006
Managing complexity
The number of people who have been without power in Queens for five days now is actually closer to 100,000, not the figure of 2,000 customers that officials of Consolidated Edison had cited in previous days.
At a press conference in Queens, Mayor Michael R. Bloomberg called the discrepancy annoying, and said that Con Edison apparently based its earlier count on the number of customers who complained to the utility company that they had no power, and not on any systematic assessment of the power outage.
Complex systems demand skilled analysis, but beyond a certain level of complexity the human mind can't cope. For this reason we create abstractions, ways of dealing with what we consider to be the most relevant variables. In this process, of course, information is lost.
It may be helpful to think of an abstraction as a contract that we attempt to make the system adhere to, whereby certain parts are governed according to specified rules without being visible to those outside, and only a subset of internal state is exposed in a carefully controlled way.
The advantage of this is the ability to manage a system that would normally be beyond the ability of humans to effectively deal with. No computers don't necessarily fix these problems. Humans have to program computers, and computers themselves are abstractions built on top of abstractions, or as a fellow programmer put it, "Our job is to magnetize quickly spinning magnetic disks at exactly the right places".
The disadvantage is that when something breaks you may have no idea why, or how to fix it. Even worse, you may have a leaky abstraction, if the underlying system violates the contract of the abstraction. And the more complex the abstracted system is, the more it will leak, and the more it will break.
So why do I write about this? Just a reminder that managing complexity is hard. Try it some time if you don't believe me. In our lives we try to build some sort of abstraction that we're going to get such and such a job and everything will be good, but ultimately we're going to have to dive into the muck underneeth and realize that you can't really impose that sort of contract on the world, or on God. Sometimes the power goes out and you have to crawl around under the streets until you figure out why your expensive equipment isn't working like it did in the brochure.