Newsletters
News & Information for Technology Purchasers NewsFactor Sites:       NewsFactor.com     Enterprise Security Today     CRM Daily     Business Report     Sci-Tech Today  
   
This ad will display for the next 20 seconds. Click for more information, or
Home Enterprise I.T. Cloud Computing Applications Hardware More Topics...
GET RECOGNIZED.
Let an ISACA® certification
elevate your career.

Register today and save
Network Security
Tame your scariest paperwork. Find Out How
Average Rating:
Rate this article:  
Google Blames Outage on Software Bug
Google Blames Outage on Software Bug

By Jennifer LeClaire
January 27, 2014 1:30PM

    Bookmark and Share
Isolated events like this are not a problem and users will forgive the Google outage. It becomes a problem when a pattern develops. If it were to happen multiple times it could become a problem for Google. Gmail has become a very strategic product and it's unlikely that Google will experience many more of these outages, said analyst Greg Sterling.
 

Related Topics

Google
Gmail
E-mail
Outage



If you are a hardcore Google user, you may have been tempted to pull out a few hairs last Friday as several of the company’s key services experienced a painful hiccup. Now, Google is shedding some light on the incident.

Specifically, Google users who use logged-in services like Gmail, Google+, Calendar and Documents were unable to access those services for about 25 minutes, according to Google vice president of Engineering Ben Treynor.

“For about 10 percent of users, the problem persisted for as much as 30 minutes longer,” he said on Friday. “Whether the effect was brief or lasted the better part of an hour, please accept our apologies -- we strive to make all of Google’s services available and fast for you, all the time, and we missed the mark today.”

What Really Happened?

Treynor reports that the issue has been resolved, and the company is now focused on correcting the bug that caused the outage, as well as putting more checks and monitors in place to ensure that this kind of problem doesn’t happen again. He then offered a technical explanation for what occurred and how it was fixed.

At 10:55 a.m. PST Friday morning, Treynor explained, an internal system that generates configurations -- essentially, information that tells other systems how to behave -- encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors.

“Users began seeing these errors on affected services at 11:02 a.m., and at that time our internal monitoring alerted Google’s Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time,” Treynor said. “By 11:30 a.m. the correct configuration was live everywhere and almost all users’ service was restored.”

Will Google See User Backlash?

With services once again working normally, Treynor said work is now focused on removing the source of failure that caused Friday’s outage, and speeding up recovery when a problem does occur. Google then took three more steps:

(1) Correcting the bug in the configuration generator to prevent recurrence, and auditing all other critical configuration generation systems to ensure they do not contain a similar bug; (2) adding additional input validation checks for configurations, so that a bad configuration generated in the future will not result in service disruption; and (3) adding additional targeted monitoring to more quickly detect and diagnose the cause of service failure.

We caught up with Greg Sterling, principal analyst at Sterling Market Intelligence, to get his take on the outage -- and its resolution. He told us because Google has such a strong reputation as a engineering-driven company when something like this happens it's surprising to many people.

“Again, isolated events like this are not a problem and users will forgive the outage,” Sterling said. “It becomes a problem when a pattern develops. If this were to happen multiple times it would start to become a problem for Google. Gmail has become a very strategic product for the company and it's unlikely that Google will experience many more of these outages.”
 

Tell Us What You Think
Comment:

Name:



APC has an established a reputation for solid products that virtually pay for themselves upon installation. Who has time to spend worrying about system downtime? APC makes it easy for you to focus on business growth instead of business downtime with reliable data center systems and IT solutions. Learn more here.


 Network Security
1.   Canada Says China Hacked Gov't
2.   Tor Working To Fix Security Exploit
3.   Wall Street Journal Hacked Again
4.   Dropbox for Business Boosts Security
5.   Hackers Breached StubHub Accounts


advertisement
Tor Working To Fix Security Exploit
Bug reportedly reveals ID of users
Average Rating:
New Technology Defeats Privacy Efforts
Study identifies 3 browser techniques.
Average Rating:
Banks Hit by Android-Skirting Malware
34 institutions, four European countries
Average Rating:
Product Information and Resources for Technology You Can Use To Boost Your Business

Network Security Spotlight
Canadian Government Charges China With Cyberattack
The government of Canada is not happy with China. Canadian officials have accused "a highly sophisticated Chinese state-sponsored actor" of launching a cyberattack on its National Research Council.
 
Researchers Working To Fix Tor Security Exploit
Developers for the Tor privacy browser are scrambling to fix a bug revealed Monday that researchers say could allow hackers, or government surveillance agencies, to track users online.
 
Wall Street Journal Hacked Again
Hacked again. That’s the story at the Wall Street Journal this week as the newspaper reports that the computer systems housing some of its news graphics were breached. Customers not affected -- yet.
 

Enterprise Hardware Spotlight
Apple Updates MacBook Pros, Cuts Prices Up to $100
The popular MacBook Pro laptop line just got an update and a price cut of as much as $100. The MacBook Pro with Retina display now includes faster processors and double the memory.
 
Watson Gets His First Customer Service Gig
Since appearing on Jeopardy, IBM's Watson supercomputer has been making a living using his super-intelligent knowledge base for business verticals. Now, Watson's been hired for his first customer service job.
 
Tablet Giants Apple and Samsung Feel the Heat
When a company saturates its home market with a once-hot product, expect it to pump up efforts elsewhere. Apple, for its part, is now pushing iPads to big corporations and the enterprise market.
 

Mobile Technology Spotlight
Android 'Fake ID' Puts Millions of Users at Risk
Having this fake ID is nothing to brag about, even if you are a minor. The “Fake ID” Android flaw drops malware into smartphone apps. It can steal credit card data and even take over your device.
 
FTC Wants Fix for 'Perfect Scam' of Mobile Cramming
The U.S. Federal Trade Commission has issued new guidelines to curb “mobile cramming,” a troublesome practice that adds unauthorized third-party charges to mobile phone bills.
 
Facebook: You Will Use Messenger, and You Will Like It
Starting this week, Facebook users with Android and iOS phones will be forced to use the separate Messenger app to send Facebook messages. Pending messages will still be visible in the main app.
 

Navigation
NewsFactor Network
Home/Top News | Enterprise I.T. | Cloud Computing | Applications | Hardware | Mobile Tech | Big Data | Communications
World Wide Web | Network Security | Data Storage | CRM Systems | Microsoft/Windows | Apple/Mac | Linux/Open Source | Personal Tech
Press Releases
NewsFactor Network Enterprise I.T. Sites
NewsFactor Technology News | Enterprise Security Today | CRM Daily

NewsFactor Business and Innovation Sites
Sci-Tech Today | NewsFactor Business Report

NewsFactor Services
FreeNewsFeed | Free Newsletters

About NewsFactor Network | How To Contact Us | Article Reprints | Careers @ NewsFactor | Services for PR Pros | Top Tech Wire | How To Advertise

Privacy Policy | Terms of Service
© Copyright 2000-2014 NewsFactor Network. All rights reserved. Article rating technology by Blogowogo. Member of Accuserve Ad Network.