News & Information for Technology Purchasers NewsFactor Sites:       NewsFactor.com     Enterprise Security Today     CRM Daily     Business Report     Sci-Tech Today  
   
Home Enterprise I.T. Cloud Computing Applications Hardware More Topics...
GET RECOGNIZED
Let an ISACA® certification elevate your career.
Register today and save
You are here: Home / Applications / Google Blames Gmail Outage on Glitch
DDoS Protection Powered By Verisign
Google Blames Outage on Software Bug
Google Blames Outage on Software Bug
By Jennifer LeClaire / NewsFactor Network Like this on Facebook Tweet this Link thison Linkedin Link this on Google Plus
PUBLISHED:
JANUARY
27
2014


If you are a hardcore Google user, you may have been tempted to pull out a few hairs last Friday as several of the company’s key services experienced a painful hiccup. Now, Google is shedding some light on the incident.

Specifically, Google users who use logged-in services like Gmail, Google+, Calendar and Documents were unable to access those services for about 25 minutes, according to Google vice president of Engineering Ben Treynor.

“For about 10 percent of users, the problem persisted for as much as 30 minutes longer,” he said on Friday. “Whether the effect was brief or lasted the better part of an hour, please accept our apologies -- we strive to make all of Google’s services available and fast for you, all the time, and we missed the mark today.”

What Really Happened?

Treynor reports that the issue has been resolved, and the company is now focused on correcting the bug that caused the outage, as well as putting more checks and monitors in place to ensure that this kind of problem doesn’t happen again. He then offered a technical explanation for what occurred and how it was fixed.

At 10:55 a.m. PST Friday morning, Treynor explained, an internal system that generates configurations -- essentially, information that tells other systems how to behave -- encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors.

“Users began seeing these errors on affected services at 11:02 a.m., and at that time our internal monitoring alerted Google’s Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time,” Treynor said. “By 11:30 a.m. the correct configuration was live everywhere and almost all users’ service was restored.”

Will Google See User Backlash?

With services once again working normally, Treynor said work is now focused on removing the source of failure that caused Friday’s outage, and speeding up recovery when a problem does occur. Google then took three more steps:

(1) Correcting the bug in the configuration generator to prevent recurrence, and auditing all other critical configuration generation systems to ensure they do not contain a similar bug; (2) adding additional input validation checks for configurations, so that a bad configuration generated in the future will not result in service disruption; and (3) adding additional targeted monitoring to more quickly detect and diagnose the cause of service failure.

We caught up with Greg Sterling, principal analyst at Sterling Market Intelligence, to get his take on the outage -- and its resolution. He told us because Google has such a strong reputation as a engineering-driven company when something like this happens it's surprising to many people.

“Again, isolated events like this are not a problem and users will forgive the outage,” Sterling said. “It becomes a problem when a pattern develops. If this were to happen multiple times it would start to become a problem for Google. Gmail has become a very strategic product for the company and it's unlikely that Google will experience many more of these outages.”

Read more on: Google, Gmail, E-mail, Outage
Tell Us What You Think
Comment:

Name:

Like Us on FacebookFollow Us on Twitter
TOP STORIES NOW
MAY BE OF INTEREST
Salesforce.com is the market and technology leader in Software-as-a-Service. Its award-winning CRM solution helps 82,400 customers worldwide manage and share business information over the Internet. Experience CRM success. Click here for a FREE 30-day trial.
MORE IN APPLICATIONS
Product Information and Resources for Technology You Can Use To Boost Your Business

Network Security Spotlight
Dairy Queen Latest Retailer To Report Hack
Known for its hot fries and soft-serve ice cream, Dairy Queen just made cyber history as the latest victim of a hack attack. The fast food chain said that customer data at some stores may be at risk.
 
Lessons from the JPMorgan Chase Cyberattack
JPMorgan Chase is investigating a likely cyberattack. The banking giant is cooperating with law enforcement, including the FBI, to understand what data hackers may have obtained.
 
Who Is the Hacker Group Lizard Squad?
Are they dangerous or just obnoxious? That’s what many are wondering about the hacker group Lizard Squad, which tweeted out a bomb threat that grounded a flight with a Sony exec aboard.
 

Enterprise Hardware Spotlight
Intel Intros Lightning-Fast PC Processors
Call it extreme. Intel just took the covers off its first-ever eight-core desktop processor, which is aimed at hardcore power users who expect more than the status quo from their computers.
 
HP Previews ProLiant Gen9 Data Center Servers
Because traditional data center and server architectures are “constraints” on businesses, HP is releasing new servers aimed at faster, simpler and more cost-effective delivery of computing services.
 
Apple Set To Release Largest iPad Ever
Tech giant Apple seems to have adopted the mantra “go big or go home.” The company is planning to introduce its largest iPad ever: a 12.9-inch behemoth that will dwarf its largest existing models.
 

Mobile Technology Spotlight
iWatch Watch: What Will Apple Ask Us To Wear?
There are still more questions than answers when it comes to details about the smart watch Apple seems poised to debut on Sept. 9. In fact, nobody seems completely sure that it will be a smart watch at all.
 
Samsung Maps Its Way with Nokia's 'Here' App for Galaxy Phones
Korean electronics giant Samsung has opted to license Here, Nokia’s mapping app -- formerly known as Nokia Maps -- for its Tizen-powered smart devices and Samsung Gear S wearable.
 
Google Successfully Tests Its Own Delivery Drone
While top technology companies are engaged in an "arms race" to develop drones that can quickly deliver goods to anyone anywhere, Google has revealed it successfully tested its own version.
 

Navigation
NewsFactor Network
Home/Top News | Enterprise I.T. | Cloud Computing | Applications | Hardware | Mobile Tech | Big Data | Communications
World Wide Web | Network Security | CRM Systems | Microsoft/Windows | Apple/Mac | Linux/Open Source | Personal Tech | Press Releases
NewsFactor Network Enterprise I.T. Sites
NewsFactor Technology News | Enterprise Security Today | CRM Daily

NewsFactor Business and Innovation Sites
Sci-Tech Today | NewsFactor Business Report

NewsFactor Services
FreeNewsFeed | Free Newsletters

About NewsFactor Network | How To Contact Us | Article Reprints | Careers @ NewsFactor | Services for PR Pros | Top Tech Wire | How To Advertise

Privacy Policy | Terms of Service
© Copyright 2000-2014 NewsFactor Network. All rights reserved. Article rating technology by Blogowogo. Member of Accuserve Ad Network.