Discussion – 


A Cautionary Tale in Cloud Management

In a week dominated by news from Google I/O, one major story emerged that has significant implications for cloud service users: Google Cloud accidentally deleted a major customer account. This account belonged to UniSuper, an Australian pension fund managing $135 billion and serving 647,000 members. The incident underscores the importance of multi-cloud strategies and robust data backup plans.

The Incident Unfolded

The Initial Outage

On May 2, UniSuper experienced an unprecedented disruption when their entire Google Cloud account, including all backups, was deleted. Fortunately, UniSuper had a contingency plan involving backups with a different provider, allowing for data recovery. However, the process was lengthy, with full service restoration not achieved until May 15.

Joint Statement from UniSuper and Google Cloud

In a joint statement on May 8, UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian explained the cause of the disruption. Kurian acknowledged an “unprecedented sequence of events” leading to the deletion of UniSuper’s Private Cloud subscription due to a misconfiguration during provisioning. He emphasized that this was an isolated incident, the first of its kind globally, and assured that measures were in place to prevent future occurrences.

The Duration of the Outage

Explaining the Extended Downtime

Despite having duplicate data storage across two geographies, the deletion affected both locations. This situation reveals a critical vulnerability: if a cloud provider’s system fails to safeguard against account-wide deletions, even redundant backups may be compromised. The restoration process was not merely about retrieving data but also involved processing pending requests and payments disrupted during the outage.

The Recovery Journey

Communication and Updates

Throughout the incident, UniSuper maintained transparency with its members. Initial updates identified the third-party provider as the source of the issue. By May 3, Google Cloud was publicly acknowledged, and both companies reassured users that the outage was not due to a cyberattack.

Progress and Apologies

By May 6, UniSuper CEO Peter Chun issued multiple statements to alleviate concerns, ensuring members that their accounts were safe, no data had been exposed, and pension payments were uninterrupted. He reiterated Google Cloud’s confirmation of the unique and unprecedented nature of the incident and their commitment to preventing recurrence.

Service Restoration Timeline

Signs of recovery appeared on May 9, with online account logins becoming available, though balances were initially outdated. Gradual restoration of website and mobile app functionalities continued, with full service reestablished by May 15.

Lessons Learned and Moving Forward

Importance of Multi-Cloud Strategies

UniSuper’s ability to recover was largely due to its multi-cloud strategy, highlighting the importance of diversifying cloud services to mitigate risks associated with provider-specific failures.

The Need for Transparent Post-Mortems

While UniSuper and Google Cloud provided regular updates, the technical details of the incident remain vague. A comprehensive post-mortem from Google Cloud would be beneficial for current and potential customers, offering insights into the failure and the steps taken to address it.

UniSuper’s experience with Google Cloud serves as a stark reminder of the potential risks in cloud service dependencies. It underscores the necessity for robust backup strategies and the importance of maintaining transparency and communication during crises. As cloud services become increasingly integral to business operations, learning from such incidents is crucial for enhancing reliability and customer trust.

At Oper8 Global, we understand the critical importance of secure and reliable cloud solutions. Our comprehensive suite of cybersecurity products, including CipherTrust Manager for key management, Payment HSMs for secure transaction processing, and CERT+ for certificate lifecycle management, ensures that your data remains protected across all environments. We are dedicated to helping you implement robust multi-cloud strategies and safeguard against potential disruptions, providing peace of mind and continuity for your business operations. Let us partner with you to enhance your cybersecurity posture and ensure your data is always secure.

In summary, Don’t bet everything on one Cloud option. Ensure your data is protected by diversifying your storage solutions and having contingency plans for worst-case scenarios.

Contact Oper8 Global

You May Also Like