Imagine the sinking feeling of logging in to your Gmail account and finding it empty. Ben Treynor, Google's vice president of engineering and site reliability czar, invited us to picture that in our mind's eye.
But many Gmail users didn't have to use their imagination -- it actually happened. Indeed, Treynor reported that 0.02 percent (about 150 million) of Gmail users woke up Monday to find their Gmail account was empty. No important e-mail to answer. E-mail archives up in smoke. It wasn't a pretty picture. And Treynor is sorry.
"The good news is that e-mail was never lost and we've restored access for many of those affected," Treynor said. "Though it may take longer than we originally expected, we're making good progress and things should be back to normal for everyone soon."
What About Google Disaster Recovery?
That was the update as of 9:30 p.m. EST Monday night. After asking Gmail users to imagine the worst, Treynor suggested some of them were thinking: "How could this happen if Google has multiple copies of your data in multiple data centers?"
Indeed, Google made a pretty big deal about its disaster-recovery services about this time last year. In March 2010, the company promised that Google Apps customers wouldn't need to worry about disaster striking and used e-mail as a prime example. So what happened?
"Well, in some rare instances bugs can affect several copies of the data. That's what happened here. Some copies of mail were deleted, and we've been hard at work over the last 30 hours getting it back for the people affected by this issue," Treynor said.
Essentially, to protect Google Apps user information from these rare bugs, Google also backs up the data to tape. Since the tapes are offline, Treynor explained, they're protected from such software bugs. It follows, then, that restoring data from tapes also takes longer than transferring requests to another data center. That, Treynor said, is why it has taken Google so long to get the e-mail back into affected Gmail users' inboxes.
"So what caused this problem? We released a storage software update that introduced the unexpected bug, which caused 0.02 percent of Gmail users to temporarily lose access to their e-mail," Treynor said. "When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version."
Gmail's Definite Problems
Gmail has had some definite problems lately with performance, and now with disappearing e-mails. Greg Sterling, principal analyst at Sterling Market Intelligence, said he hasn't seen any empirical data to argue that satisfaction with Gmail is declining, but that's his anecdotal experience from talking to people.
"If glitches and problems like this continue, the grumbling and public criticism will grow louder," Sterling said. "However, it will probably take a good deal more to push people off Gmail into competitors' systems. E-mail switching costs are very high, especially if Gmail is your primary e-mail address."
The bottom line: Affected Gmail users probably will never see e-mail that was sent to them between 6:00 p.m. PST on Feb. 27 and 2:00 p.m. PST on Feb. 28. The good news is that senders received a notification that their messages weren't delivered and can try again.
Posted: 2011-09-26 @ 9:37pm PT
It is very important to backup your email on a regular basis for Disaster Recovery. If you ever lose your email by accidentally deleting or because of any other reason beyond your control, you can quickly restore your email from the backup using Beyond Inbox. Beyond Inbox is designed for convenience of taking secure backups of IMAP enabled email clients.
Posted: 2011-03-09 @ 12:11am PT
Is there any special facility for business accounts of renowned organization to get back their lost emails through any help line services?
Posted: 2011-03-02 @ 12:03am PT
The second-to-last paragraph sums it up well ... I lost nothing, but did experience an outage; and from time-to-time have experienced slowdowns that may not be related (could be host, provider, computer RAM overload, even mouse malfunctioning, how is a layman to know?). I have other e-mail online accounts, and still like Gmail best for its customizable labeling and sorting system.