Wednesday, April 18, 2007

Geek Wednesday: "The Dog Ate My Email"

Watch Sen. Patrick Leahy demand the truth on the deleted emails

As promised, today we are offering some technical perspective on exactly what might be required to perform a complete and total deletion of email in the volume being discussed by the Karl Rove leaf of the executive branch. We begin our discussion with the view of our own Nearly Redmond Nick, who in his day job is a fully-qualified systems architect. This means he knows intricately how a large network is set up to store, transmit, and protect data, including email. As we will see, the system is actually planned and configured to be "delete-proof" rather than "delete-friendly." First, though, here's NR Nick:

From what I understand, it wasn't even selective cleaning - it was all of Rove's emails over 4 years. Unfortunately for prosecutors, there are many ways this could happen. Settings on his account could be changed to not save sent emails (or received for that matter) for more than 90 days. Our company actually used to have a policy like that. They would periodically remind you to backup your emails, or move them to a folder other than the inbox. Anything older than 90 days in the inbox got deleted. But all that means to say is that the user couldn't access the email any more.

But wherever I've been in corporate America there was a process to retrieve "deleted" emails from tape backup. This is why so many Dems are crying foul, saying that in this day and age, you can't simply delete an email. To have these missing messages be totally unrecoverable would require a good deal of effort. First, Rove would have to clean out all his local copies of his email, and delete any messages stored on the server. Then the admins would have to wipe all the logs, since all inbound and outbound messages are usually recorded. Then all the backup tapes would have to be cleansed. Not a small amount of effort.

The worst part of all, is that on top of the 1978 act you mentioned, Rove had some other special "subpoena", for lack of a better word, on his emails. Supposedly, he was supposed to be watched more closely than most, and all emails saved (if I read the article right). Again, I have to dig into this a little more, but in sum, this was not a "whoops, I pressed the delete key" kind of action, but a much larger proect—you might even call it a conspiracy, for the effort and teams required to really make it happen.

.Mac (Apple Computer, Inc.)

That, ladies and gentlemen, is precisely the point: in order to wipe a network of servers, backup drives, and client machines totally clean of any trace of some five million emails would be an enormous effort of planning, execution, follow-up, and testing. Again, modern IT systems are designed in every respect to preserve data rather than destroy it. A large network like the RNC's would presumably be very much like a corporation's: there would be central servers with backup, and then there might even be additional backup, known as DR, or disaster recovery servers—usually in a remote location, so that if the main servers were lost, the DR servers would be unaffected.

And as Nick points out, log files would have to be searched and scoured on all the servers and backups mentioned. It would, in short, be an enormous effort involving multiple teams of system architects, developers (who might write the scripts used to automate such a large and complex deleting effort), testers who would verify that the desired data had been in fact wiped clean, and systems administrators who would ensure that no trace of system logs or other technical residue remained of the data to be erased. And we won't even go into a discussion of the differences among the varying levels of deletion, which also must be taken into account at a technical level.

So if I were a betting man, my money would be with Sen. Leahy: those emails are out there, magnetically alive and retrievable via any of the means mentioned above. Think of it as if you had thrown a pile of letters into a fireplace: they would burn, indeed; but they could also be recovered and read with the right technology and expertise.

What is required now is a full-fledged geek-body-right on this thing. It will have to start with Congressional subpoenas, maybe even an independent prosecutor to watchdog the affair, and most importantly, a team of system architects who know what to look for and where to find it. Given the time and the freedom to do so, they will—I'd bet my MacBook on it.


Blame the phone: As you might have heard, the release of Apple's OS X Leopard has been delayed by 4 months, until October. An Apple philosopher pointed out that "Life often presents tradeoffs," and the company has opted for diverting all its resources into delivering the iPhone on time, since it is such a "revolutionary and magical product." As for that dull, conformist, and thoroughly Muggle operating system, it will simply have to wait. When I posted a comment on Macworld's boards, wondering how a corporate behemoth like Apple couldn't afford engineers and QA resources for both products, I was treated with the appropriate condescension. How could anyone imagine that an OS poised to take a bite out of the Windows hegemony--in both the consumer and enterprise markets--might be as important as a telephone that plays music and does widgets? The very idea's enough to make a sane person pop his lorgnette!

Linux Update: So if you're looking for a new OS that has a chance at not making you blow lunch all over your keyboard, and you don't want to wait until Autumn, there's big news in Penguin-land. Tomorrow, Ubuntu releases version 7.04, The Feisty Fawn. It will feature advanced virtualization features, including the ability to use a technology called KVM to run guest OS's within Linux. There will be a fresh design of the UI in both the Gnome and KDE desktop environments (yeah, you do get to choose from alternate desktop styles), and more advanced compatibility with third-party drivers and software for peripherals and other devices. I'll be downloading it and will have a full report next week right here on GW.

World Bank, Local Dick: And finally our site of the week, BankSwirled*, an Onion-style sendup produced by anonymous World Bank employees as a tribute to their own boss, the war-starting, passion-playing Paul Wolfowitz. As the tattoo on the inside of Shaha Ali Riza's thighs says, "I greet you as a liberator."

*We're linking to the pdf version of the site, just in case its html counterpart is mysteriously shut down by a geo-financial wind of globalization.

No comments: