Ride the Lightning

Google Global Outage Caused by Critical System Running Out of Storage

December 16, 2020

You read that headline and you are incredulous that this could happen. But it did.

Bleeping Computer reported on December 15 that the global Google services outage on December 14 was caused by the company's Identity Management System failing after a bug restricted its storage space. The failure prevented users from accessing Gmail, YouTube, Google Drive, Google Maps, Google Calendar, and other Google services.

During the outage, users could not send emails via Gmail mobile apps or receive email via POP3 for desktop clients. Also, YouTube visitors were seeing an error message stating, "There was a problem with the server (503) – Tap to retry."

According to a tweet and a Google status report, the outage was caused by the company's automated quota management system reducing the amount of storage available to Google's authentication system.

"Today, at 3.47AM PT Google experienced an authentication system outage for approximately 45 minutes due to an internal storage quota issue. This was resolved at 4:32AM PT, and all services are now restored," Google stated in a tweet from its Google Cloud account.

Google further clarified the cause of the outage in Google Cloud's status page, where it stated the reduced storage caused their identity management system (IdM) to fail.

"Google Cloud Platform and Google Workspace experienced a global outage affecting all services which require Google account authentication for a duration of 50 minutes. The root cause was an issue in our automated quota management system which reduced capacity for Google's central identity management system, causing it to return errors globally. As a result, we couldn't verify that user requests were authenticated and served errors to our users," Google's status page explained.

An identity management system is used to authenticate users and assign privileges when they log into a system.

After running out of storage, Google IdM began returning errors that prevented users from authenticating to Google's services, including Cloud Console, Cloud Storage, BigQuery, Google Kubernetes Engine, Gmail, Calendar, Meet, Docs, Drive, and YouTube.

To prevent these kinds of issues from occurring again, Google's automated quota management system has been disabled while they investigate the incident.

Google also said that the outages affected their internal users and tools, causing delays in the outage investigation and the reporting of status updates. It was, to quote from a famous book title, "a terrible, horrible, no good, very bad day." I realized something was happening as I was trying to correspond with multiple Gmail users. One would hope Google has a lot of exceptionally bright people figuring out how this could happen so it doesn't happen again.

Sharon D. Nelson, Esq., President, Sensei Enterprises, Inc.
3975 University Drive, Suite 225|Fairfax, VA 22030
Email: Phone: 703-359-0700
Digital Forensics/Cybersecurity/Information Technology
https://senseient.com
https://twitter.com/sharonnelsonesq
https://www.linkedin.com/in/sharondnelson
https://amazon.com/author/sharonnelson