News & Information for Technology Purchasers NewsFactor Sites:       NewsFactor.com     Enterprise Security Today     CRM Daily     Business Report     Sci-Tech Today  
   
This ad will display for the next 20 seconds. Click for more information, or
Home Enterprise I.T. Cloud Computing Applications Hardware More Topics...
Advertisement
Free Gartner Report:
Drive innovation & collaboration
with the "Everyone's IT" approach.

View the research report
Applications
Is your endpoint data protected?
Average Rating:
Rate this article:  
Google Blames Outage on Software Bug

Google Blames Outage on Software Bug
By Jennifer LeClaire

Share
Share on Facebook Share on Twitter Share on Linkedin Share on Google Plus

Isolated events like this are not a problem and users will forgive the Google outage. It becomes a problem when a pattern develops. If it were to happen multiple times it could become a problem for Google. Gmail has become a very strategic product and it's unlikely that Google will experience many more of these outages, said analyst Greg Sterling.
 

Related Topics

Google
Gmail
E-mail
Outage



If you are a hardcore Google user, you may have been tempted to pull out a few hairs last Friday as several of the company’s key services experienced a painful hiccup. Now, Google is shedding some light on the incident.

Specifically, Google users who use logged-in services like Gmail, Google+, Calendar and Documents were unable to access those services for about 25 minutes, according to Google vice president of Engineering Ben Treynor.

“For about 10 percent of users, the problem persisted for as much as 30 minutes longer,” he said on Friday. “Whether the effect was brief or lasted the better part of an hour, please accept our apologies -- we strive to make all of Google’s services available and fast for you, all the time, and we missed the mark today.”

What Really Happened?

Treynor reports that the issue has been resolved, and the company is now focused on correcting the bug that caused the outage, as well as putting more checks and monitors in place to ensure that this kind of problem doesn’t happen again. He then offered a technical explanation for what occurred and how it was fixed.

At 10:55 a.m. PST Friday morning, Treynor explained, an internal system that generates configurations -- essentially, information that tells other systems how to behave -- encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors.

“Users began seeing these errors on affected services at 11:02 a.m., and at that time our internal monitoring alerted Google’s Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time,” Treynor said. “By 11:30 a.m. the correct configuration was live everywhere and almost all users’ service was restored.”

Will Google See User Backlash?

With services once again working normally, Treynor said work is now focused on removing the source of failure that caused Friday’s outage, and speeding up recovery when a problem does occur. Google then took three more steps:

(1) Correcting the bug in the configuration generator to prevent recurrence, and auditing all other critical configuration generation systems to ensure they do not contain a similar bug; (2) adding additional input validation checks for configurations, so that a bad configuration generated in the future will not result in service disruption; and (3) adding additional targeted monitoring to more quickly detect and diagnose the cause of service failure.

We caught up with Greg Sterling, principal analyst at Sterling Market Intelligence, to get his take on the outage -- and its resolution. He told us because Google has such a strong reputation as a engineering-driven company when something like this happens it's surprising to many people.

“Again, isolated events like this are not a problem and users will forgive the outage,” Sterling said. “It becomes a problem when a pattern develops. If this were to happen multiple times it would start to become a problem for Google. Gmail has become a very strategic product for the company and it's unlikely that Google will experience many more of these outages.”
 

Tell Us What You Think
Comment:

Name:



IT departments are embracing cloud backup, but there's a lot you need to know before choosing a service provider. Learn all the critical things you need to know by accessing the white paper, "5 Things You Didn't Know About Cloud Backup". Access the White Paper now.


 Applications
1.   VMware Buys CloudVolumes' V-Apps
2.   Popular Mailbox App Comes to Mac
3.   9 Norton Security Products Are Now 1
4.   Infor Buys Cloud CRM App Saleslogix
5.   Foursquare: All Customer Experience


advertisement
China Puts Microsoft Under the Lens
Official anti-monopoly probe launched.
Average Rating:
Foursquare: All Customer Experience
App's new focus is on recommendations.
Average Rating:
Popular Mailbox App Comes to Mac
Takes to-do list approach to the inbox.
Average Rating:
Product Information and Resources for Technology You Can Use To Boost Your Business

Network Security Spotlight
Researchers Find Malicious Android Apps Can Hack Gmail
A new study shows that a weakness in the Android mobile operating system can be used to steal sensitive, personal info from unwitting users. Gmail proved to be the easiest app to attack; Amazon, the hardest.
 
UPS Stores in 24 States Hit by Data Breach
Big Brown has been breached. UPS said that about 105,000 customer transactions at 51 of its UPS Store locations in 24 states could have been compromised between January and August.
 
Cost of Target Data Breach: $148 Million Plus Loss of Trust
The now infamous Target data breach is still costing the company -- and its shareholders -- plenty. In fact, the retailing giant forecast the December 2013 incident cost shareholders $148 million.
 

Enterprise Hardware Spotlight
Acer's New Desktop Box Rides the Chrome OS Wave
Filling out its Chrome OS line, Acer is following the introduction of a larger Chromebook line earlier this month with a new tiny $180 desktop Chromebox and also a smaller Chromebook.
 
Feds OK $2.3 Billion IBM-Lenovo x86 Server Deal
IBM and Lenovo are celebrating U.S. approval of their x86-based server deal, having cleared some major security hurdles. The deal makes Lenovo a major player for enterprise data centers.
 
Three New Lenovo PCs Aimed at Business Users
With businesses wanting computing solutions that do more for less money, Lenovo has unveiled three new desktop PCs that it says offer solid computing at a budget-minded price.
 

Mobile Technology Spotlight
Screen Shortage Briefly Puts Brakes on iPhone 6
RAM? Check. Antenna switch? Check. Screen? Oops. Parts suppliers for Apple have found themselves facing a shortage of screens for the new iPhone 6 as next month's release date for the new smartphone looms.
 
Bounty Offered to Coders for Oculus Rift Bugs
Coders who find bugs in software for the Oculus Rift VR immersive headset could receive a reward of at least $500 under Facebook's White Hat bounty program. Facebook acquired Oculus in March.
 
Google Glass Adds Voice Access to Phone Contacts
The latest update to Google Glass will let users access their top 20 phone contacts with voice commands alone. A user can then choose a phone call, Google hangouts, e-mail or text messaging.
 

Navigation
NewsFactor Network
Home/Top News | Enterprise I.T. | Cloud Computing | Applications | Hardware | Mobile Tech | Big Data | Communications
World Wide Web | Network Security | Data Storage | CRM Systems | Microsoft/Windows | Apple/Mac | Linux/Open Source | Personal Tech
Press Releases
NewsFactor Network Enterprise I.T. Sites
NewsFactor Technology News | Enterprise Security Today | CRM Daily

NewsFactor Business and Innovation Sites
Sci-Tech Today | NewsFactor Business Report

NewsFactor Services
FreeNewsFeed | Free Newsletters

About NewsFactor Network | How To Contact Us | Article Reprints | Careers @ NewsFactor | Services for PR Pros | Top Tech Wire | How To Advertise

Privacy Policy | Terms of Service
© Copyright 2000-2014 NewsFactor Network. All rights reserved. Article rating technology by Blogowogo. Member of Accuserve Ad Network.