I have a rule in politics that has served me well: never attribute to malice or conspiracy what can be explained just as easily by incompetence.
You may remember this story from back in 2006, at the culmination of the bitter primary battle between Ned Lamont and I-can’t-believe-it’s-a-Democrat! Joe Lieberman: On the eve of the primary, Lieberman’s web site crashed… and the Lieberman campaign promptly blamed Lamont’s supporters for orchestrating an attack. (More than a few credulous media outlets blithely reported the site had been hacked, or was the target of a denial of service attack.)
In the immediate aftermath, speculation focused on a possible security vulnerability in the Joomla! content management system used for Lieberman’s site.
Well, an FBI investigation saw it differently. According to documents obtained in response to a newspaper’s freedom of information request, the problem was human error… and the humans in question were on Lieberman’s team:
According to the FBI memo, the site crashed because Lieberman officials continually exceeded a configured limit of 100 e-mails per hour the night before the primary.
“The system administrator misinterpreted the root cause,” the memo stated. “The system administrator finally declared the server was being attacked and the Lieberman campaign accused the Ned Lamont campaign. The news reported this on Aug. 8, 2006, causing additional Web traffic to visit the site.
“The additional Web traffic then overwhelmed the Web server. . . . Web traffic pattern analysis reports and Web logging that was available did not demonstrate traffic that was indicative of a denial of service attack.”
Look, sites go down. Sites under a lot of pressure go down more often. (And sites under a lot of pressure hosted in a shared environment… well, that wasn’t the brightest call his tech team could have made. A virtual private server could have saved everyone a lot of grief.)
When they do, and a crisis erupts, keeping a cool head is critical. That also means keeping an open mind; in times of high-voltage stress, people often seize the explanation that most suits their world-view, and stop considering other possibilities. It’s a hard instinct to fight in your own mind, and harder still to fight when the people around you have succumbed.
The quest for scapegoats (bloggers! hackers! our opponents! Joomla!) is always tempting. But ask yourself: how reliable is the information you’re getting? Does, say, your system admin have a vested interest in believing the situation involves an external attack instead of an internal mistake? Are your less ethical advisers sensing the opportunity to make political hay with outraged accusations, regardless of the real cause?
Maybe the answer is that you won’t have an answer – not for the next few hours, maybe not even for the next few days. The task at hand, then, is to restore your web presence as quickly and effectively as possible. As soon as it’s evident the problem is persistent, take your backups to another server and get up and running now… so you can keep communicating. (No luck? Launch a temporary WordPress.com blog to tell the world what’s happening. Start a Ning network. Just keep your online channel open.)
And then prepare for next time. Have an emergency plan ready. Make sure your host understands that the closer you get to election day, the more demand the server’s going to face. Ask them specifically what measures they’ll take as a precaution now, and which they’ll hold in reserve in case things get really hairy. If you aren’t satisfied with the answers, change hosts.
And if the site still goes down, hold off on the wild accusations. You’ll get to the bottom of the problem more quickly… and do a little to make the political atmosphere a bit less poisonous.
