AZURE - Southeast Asia (Singapore): MI-20220113
Incident
Report for Snowflake
Postmortem
Dear Customer,
Snowflake Engineering has completed the postmortem of this major incident. A detailed Root Cause Analysis is made available on the Snowflake Community site.
https://community.snowflake.com/s/article/MI-20220113
We apologise for the major incident and the impact on your applications.
If you have any questions or difficulty accessing the Snowflake community site, please send feedback by submitting a support case or calling Snowflake Technical Support.
Thank you,
Snowflake Support
Resolved
The Service Incident is now closed.
A Root Cause Analysis (RCA) will be posted within the next ten business days.
We apologize for the inconvenience caused by this incident.
If you have any questions or experience any related issues, please open a support case via the Snowflake Community Site.
A Root Cause Analysis (RCA) will be posted within the next ten business days.
We apologize for the inconvenience caused by this incident.
If you have any questions or experience any related issues, please open a support case via the Snowflake Community Site.
Monitoring
Our cloud service provider (CSP-Azure) have performed mitigation steps by rolling back a recent deployment and balanced requests to healthy role instances. Snowflake engineering validated and confirmed issue has been resolved.
Symptom(s): warehouses stuck trying to resume; replication job may experience delays
Incident Start Time: 11:55 PT January 13, 2022
Incident End Time: 21:00 PT January 13, 2022
We are now monitoring the system for any further recurrence of the problem. We will remain in this state for 60 minutes.
If you have any questions or experience any related issues, please open a support case via the Snowflake Community Site.
Symptom(s): warehouses stuck trying to resume; replication job may experience delays
Incident Start Time: 11:55 PT January 13, 2022
Incident End Time: 21:00 PT January 13, 2022
We are now monitoring the system for any further recurrence of the problem. We will remain in this state for 60 minutes.
If you have any questions or experience any related issues, please open a support case via the Snowflake Community Site.
Identified
Our cloud service provider (CSP-Azure) have identified issues with their backend role instances leveraged by Azure Resource Manager. Currently, Azure are rolling back a recent deployment as a mitigation strategy and balancing requests to healthy role instances.
Customer Action:
None
Symptom(s): warehouses stuck trying to resume; replication job may experience delays
Incident Start Time: 11:55 PT January 13, 2022
We will update the status as soon as the issue is resolved or provide an update within 60 minutes.
Customer Action:
None
Symptom(s): warehouses stuck trying to resume; replication job may experience delays
Incident Start Time: 11:55 PT January 13, 2022
We will update the status as soon as the issue is resolved or provide an update within 60 minutes.
Update
We are working with our cloud service provider (CSP - Azure) on troubleshooting the root cause of this issue. Azure teams are continuing to investigate the issue on their side at the highest priority.
Please refer to https://status.azure.com/en-gb/status/ for additional information.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
Customer Action:
None
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 60 minutes.
Please refer to https://status.azure.com/en-gb/status/ for additional information.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
Customer Action:
None
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 60 minutes.
Update
We are continuing to investigate the issue with Snowflake services.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
Customer Action:
None
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 60 minutes.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
Customer Action:
None
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 60 minutes.
Update
We are continuing to investigate the issue with Snowflake services.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
Customer Action:
None
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
Customer Action:
None
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Update
We are continuing to investigate the issue with Snowflake services.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
Customer Action:
None
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
Customer Action:
None
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate the issue with Snowflake services.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
Customer Action:
None
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
Customer Action:
None
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Investigating
We are investigating an issue with one of the Snowflake services.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
Symptom(s): warehouses stuck trying to resume
Incident Start Time: 11:55 PT January 13, 2022
We will provide more information on the problem investigation status as soon as we have identified the problem or provide an update within 30 minutes.
This incident affected: Azure - UAE North (Dubai) (Snowflake Data Warehouse (Database)) and Azure - Southeast Asia (Singapore) (Replication).