AZURE - Southeast Asia (Singapore): MI-20220113
Incident Report for Snowflake
Postmortem

Dear Customer,

Snowflake Engineering has completed the postmortem of this major incident. A detailed Root Cause Analysis is made available on the Snowflake Community site.

https://community.snowflake.com/s/article/MI-20220113

We apologise for the major incident and the impact on your applications.

If you have any questions or difficulty accessing the Snowflake community site, please send feedback by submitting a support case or calling Snowflake Technical Support.

Thank you,

Snowflake Support

Posted Jan 26, 2022 - 13:31 PST

Resolved
The Service Incident is now closed.

A Root Cause Analysis (RCA) will be posted within the next ten business days.

We apologize for the inconvenience caused by this incident.

If you have any questions or experience any related issues, please open a support case via the Snowflake Community Site.
Posted Jan 13, 2022 - 23:02 PST
Monitoring
Our cloud service provider (CSP-Azure) have performed mitigation steps by rolling back a recent deployment and balanced requests to healthy role instances. Snowflake engineering validated and confirmed issue has been resolved.

Symptom(s): warehouses stuck trying to resume; replication job may experience delays
Incident Start Time: 11:55 PT January 13, 2022
Incident End Time: 21:00 PT January 13, 2022

We are now monitoring the system for any further recurrence of the problem. We will remain in this state for 60 minutes.

If you have any questions or experience any related issues, please open a support case via the Snowflake Community Site.
Posted Jan 13, 2022 - 22:02 PST
Identified
Our cloud service provider (CSP-Azure) have identified issues with their backend role instances leveraged by Azure Resource Manager. Currently, Azure are rolling back a recent deployment as a mitigation strategy and balancing requests to healthy role instances.



Customer Action:
None

Symptom(s): warehouses stuck trying to resume; replication job may experience delays
Incident Start Time: 11:55 PT January 13, 2022

We will update the status as soon as the issue is resolved or provide an update within 60 minutes.
Posted Jan 13, 2022 - 21:03 PST
Update
We are working with our cloud service provider (CSP - Azure) on troubleshooting the root cause of this issue. Azure teams are continuing to investigate the issue on their side at the highest priority.
Please refer to https://status.azure.com/en-gb/status/ for additional information.

Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022

Customer Action:
None

We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 60 minutes.
Posted Jan 13, 2022 - 20:00 PST
Update
We are continuing to investigate the issue with Snowflake services.

Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022

Customer Action:
None

We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 60 minutes.
Posted Jan 13, 2022 - 18:55 PST
Update
We are continuing to investigate the issue with Snowflake services.

Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022

Customer Action:
None

We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Posted Jan 13, 2022 - 18:24 PST
Update
We are continuing to investigate the issue with Snowflake services.

Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022

Customer Action:
None

We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Posted Jan 13, 2022 - 17:54 PST
Update
We are continuing to investigate this issue.
Posted Jan 13, 2022 - 17:52 PST
Update
We are continuing to investigate the issue with Snowflake services.

Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022

Customer Action:
None

We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Posted Jan 13, 2022 - 17:18 PST
Investigating
We are investigating an issue with one of the Snowflake services.

Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022

We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Posted Jan 13, 2022 - 16:49 PST
This incident affected: AZURE - Southeast Asia (Singapore) (Snowflake Data Warehouse (Database), Replication).