Newsletters
News & Information for Technology Purchasers NewsFactor Sites:       NewsFactor.com     Enterprise Security Today     CRM Daily     Business Report     Sci-Tech Today  
   
This ad will display for the next 20 seconds. Please click for more information, or scroll down to pass the ad, or Close Ad.
Home Enterprise I.T. Cloud Computing Applications Hardware More Topics...
Eliminate costly downtime!
Find out how with Free White Paper
& enter to win a Samsung Galaxy Note

www.apc.com
Applications
24/7/365 Network Uptime
Average Rating:
Rate this article:  
Google Blames Outage on Software Bug
Google Blames Outage on Software Bug

By Jennifer LeClaire
January 27, 2014 1:30PM

    Bookmark and Share
Isolated events like this are not a problem and users will forgive the Google outage. It becomes a problem when a pattern develops. If it were to happen multiple times it could become a problem for Google. Gmail has become a very strategic product and it's unlikely that Google will experience many more of these outages, said analyst Greg Sterling.
 

Related Topics

Google
Gmail
E-mail
Outage



If you are a hardcore Google user, you may have been tempted to pull out a few hairs last Friday as several of the company’s key services experienced a painful hiccup. Now, Google is shedding some light on the incident.

Specifically, Google users who use logged-in services like Gmail, Google+, Calendar and Documents were unable to access those services for about 25 minutes, according to Google vice president of Engineering Ben Treynor.

“For about 10 percent of users, the problem persisted for as much as 30 minutes longer,” he said on Friday. “Whether the effect was brief or lasted the better part of an hour, please accept our apologies -- we strive to make all of Google’s services available and fast for you, all the time, and we missed the mark today.”

What Really Happened?

Treynor reports that the issue has been resolved, and the company is now focused on correcting the bug that caused the outage, as well as putting more checks and monitors in place to ensure that this kind of problem doesn’t happen again. He then offered a technical explanation for what occurred and how it was fixed.

At 10:55 a.m. PST Friday morning, Treynor explained, an internal system that generates configurations -- essentially, information that tells other systems how to behave -- encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors.

“Users began seeing these errors on affected services at 11:02 a.m., and at that time our internal monitoring alerted Google’s Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time,” Treynor said. “By 11:30 a.m. the correct configuration was live everywhere and almost all users’ service was restored.”

Will Google See User Backlash?

With services once again working normally, Treynor said work is now focused on removing the source of failure that caused Friday’s outage, and speeding up recovery when a problem does occur. Google then took three more steps:

(1) Correcting the bug in the configuration generator to prevent recurrence, and auditing all other critical configuration generation systems to ensure they do not contain a similar bug; (2) adding additional input validation checks for configurations, so that a bad configuration generated in the future will not result in service disruption; and (3) adding additional targeted monitoring to more quickly detect and diagnose the cause of service failure.

We caught up with Greg Sterling, principal analyst at Sterling Market Intelligence, to get his take on the outage -- and its resolution. He told us because Google has such a strong reputation as a engineering-driven company when something like this happens it's surprising to many people.

“Again, isolated events like this are not a problem and users will forgive the outage,” Sterling said. “It becomes a problem when a pattern develops. If this were to happen multiple times it would start to become a problem for Google. Gmail has become a very strategic product for the company and it's unlikely that Google will experience many more of these outages.”
 

Tell Us What You Think
Comment:

Name:



APC has an established a reputation for solid products that virtually pay for themselves upon installation. Who has time to spend worrying about system downtime? APC makes it easy for you to focus on business growth instead of business downtime with reliable data center systems and IT solutions. Learn more here.


 Applications
1.   Silverpop: IBM Marketing Gets Personal
2.   VMware Horizon 6 Folds In AirWatch
3.   Cuban Twitter Creates New Hurdles
4.   Wedding in the Palm of Your Hand
5.   Last Fixes Tuesday for XP, Office 2003


advertisement
Last Fixes Tuesday for XP, Office 2003
Microsoft closing out support for two.
Average Rating:
VMware Horizon 6 Folds In AirWatch
Delivers both published apps, desktops.
Average Rating:
Cuban Twitter Creates New Hurdles
More work for bloggers, dissidents.
Average Rating:
Product Information and Resources for Technology You Can Use To Boost Your Business

Network Security Spotlight
IBM Offers Security, Disaster Recovery as SoftLayer Service
New disaster recovery and security services for SoftLayer clients are being added by IBM. Big Blue said the new capabilities will speed cloud adoption by alleviating concern over business continuity.
 
How To Beat the Heartbleed Bug
Heartbleed headlines continue as IT admins scramble for answers no one has. Early reports of stolen personal data, including 900 social insurance numbers in Canada, are starting to trickle in.
 
After Heartbleed, OpenSSL Calls for More Support
The president of the OpenSSL Foundation says more support is needed from companies and governments that use its software so that it can better spot and fix flawed pieces of code such as Heartbleed.
 

Enterprise Hardware Spotlight
Vaio Fit 11A Battery Danger Forces Recall by Sony
Using a Sony Vaio Fit 11A laptop? It's time to send it back to Sony. In fact, Sony is encouraging people to stop using the laptop after several reports of its Panasonic battery overheating.
 
Continued Drop in Global PC Shipments Slows
Worldwide shipments of PCs fell during the first three months of the year, but the global slump in PC demand may be easing, with a considerable slowdown from last year's drops.
 
Google Glass Finds a Home in Medical Education, Practice
Google Glass may find its first markets in verticals in which hands-free access to data is a boon. Medicine is among the most prominent of those, as seen in a number of Glass experiments under way.
 

Mobile Technology Spotlight
Amazon 3D Smartphone Pics Leaked
E-commerce giant Amazon is reportedly set to launch a smartphone after years of development. Photos of the phone, which may feature a unique 3D interface, were leaked by tech pub BGR.
 
Zebra Tech Buys Motorola Enterprise for $3.45B
Weeks after Lenovo bought Motorola Mobility’s assets from Google for $2.91 billion, Zebra Technologies is throwing down $3.45 billion for Motorola’s Enterprise business in an all-cash deal.
 
CTIA Caves, Volunteers Kill Switch Plan
After bucking against the concept of a smartphone kill switch, the CTIA just announced the “Smartphone Anti-Theft Voluntary Commitment” to thwart smartphone thefts in the U.S.
 

Navigation
NewsFactor Network
Home/Top News | Enterprise I.T. | Cloud Computing | Applications | Hardware | Mobile Tech | Big Data | Communications
World Wide Web | Network Security | Data Storage | Small Business | Microsoft/Windows | Apple/Mac | Linux/Open Source | Personal Tech
Press Releases
NewsFactor Network Enterprise I.T. Sites
NewsFactor Technology News | Enterprise Security Today | CRM Daily

NewsFactor Business and Innovation Sites
Sci-Tech Today | NewsFactor Business Report

NewsFactor Services
FreeNewsFeed | Free Newsletters | XML/RSS Feed

About NewsFactor Network | How To Contact Us | Article Reprints | Careers @ NewsFactor | Services for PR Pros | Top Tech Wire | How To Advertise

Privacy Policy | Terms of Service
© Copyright 2000-2014 NewsFactor Network. All rights reserved. Article rating technology by Blogowogo. Member of Accuserve Ad Network.