On Jan 22, 2020, between the hours of 13:50 and 23:50 PDT, Snowflake customers in AWS - Asia Pacific (Sydney) could not provision warehouses and experienced a large rate of query failure incidents.
The root cause of this issue was the failure of the AWS data store responsible for storing the Virtual Private Cloud (VPC) network configuration impacting operations in AWS Sydney. As a result Snowflake services could not provision warehouses to execute queries.
First, our engineers responded to alerts on a burst of warehouse provisioning failures in the AWS - Pacific (Sydney) region. The log messages were apparent in indicating the source of the problem “Cannot connect using AWS API to provision warehouses.” Around this time, we were notified by AWS of the ongoing problem in AWS Sydney, and the incident was also posted on https://status.aws.amazon.com/
We also opened a case with AWS to track the progress of the resolution.
Ultimately, our cloud provider AWS took a series of steps to fix the data store problem that caused the AWS outage by restoring the VPC configuration data.
Finally, the data store was restored entirely by 20:49 PDT on Jan 22, 2020.
First, we apologize for the inconvenience caused by this incident.
AWS has outlined several improvements in its processes and code which are scheduled for completion by Feb 28, 2020.
The Snowflake engineering team is monitoring the progress of these improvements and following up on them in detail during our meetings with AWS engineering.
Note: The information contained in this report is confidential and is intended solely to promote safety and reduce customer risk.