Azure - East US 2 (Virginia): INC0130416
Incident
Report for Snowflake
Postmortem
A detailed Root Cause Analysis (RCA) is available on the Snowflake Community site: https://community.snowflake.com/s/article/INC0130416
Resolved
Current status: We've finished monitoring the environment and confirmed that all services are functioning properly. If you experience additional issues or have questions, please open a support case via Snowflake Community.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to execute queries. Affected customers might have encountered query failures.
Incident start time: 11:00 UTC April 03, 2025
Incident end time: 14:16 UTC April 03, 2025
Preliminary root cause: An increase in load to an internal system resulted in elevated CPU utilization on the Cloud Services layer. Specifically, the increase in CPU caused a backlog on the infrastructure responsible for query processing, and as a result, customers may have experienced long-running and/or failed queries.
A root cause analysis (RCA) document will be published within seven business days.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to execute queries. Affected customers might have encountered query failures.
Incident start time: 11:00 UTC April 03, 2025
Incident end time: 14:16 UTC April 03, 2025
Preliminary root cause: An increase in load to an internal system resulted in elevated CPU utilization on the Cloud Services layer. Specifically, the increase in CPU caused a backlog on the infrastructure responsible for query processing, and as a result, customers may have experienced long-running and/or failed queries.
A root cause analysis (RCA) document will be published within seven business days.
Monitoring
Current status: We've implemented the fix for this issue, and we'll continue to monitor the environment until we're confident all services are functioning properly. Please note that our telemetry indicated recovery began at 14:16 UTC; however, this time may have varied as the backlog of requests was processed.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to execute queries. Affected customers might have encountered query failures.
Incident start time: 11:00 UTC April 03, 2025
Incident end time: 14:16 UTC April 03, 2025
Preliminary root cause: An increase in load to an internal system resulted in elevated CPU utilization on the Cloud Services layer. Specifically, the increase in CPU caused a backlog on the infrastructure responsible for query processing, and as a result, customers may have experienced long-running and/or failed queries.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to execute queries. Affected customers might have encountered query failures.
Incident start time: 11:00 UTC April 03, 2025
Incident end time: 14:16 UTC April 03, 2025
Preliminary root cause: An increase in load to an internal system resulted in elevated CPU utilization on the Cloud Services layer. Specifically, the increase in CPU caused a backlog on the infrastructure responsible for query processing, and as a result, customers may have experienced long-running and/or failed queries.
Identified
Current status: The mitigation actions we have taken are showing signs of recovery. We're continuing to monitor the environment while further investigating the source of the issue. We'll provide another update within the next 60 minutes.
Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 11:00 UTC April 03, 2025
Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 11:00 UTC April 03, 2025
Update
Current status: Our investigation identified that there was a recent increase in utilization on the Cloud service layer. We have taken action to reduce CPU utilization on the affected infrastructure as a potential mitigation strategy. We're continuing to monitor the environment and investigate the source of the issue. We'll provide another update within the next 60 minutes.
Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 11:00 UTC April 03, 2025
Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 11:00 UTC April 03, 2025
Update
Current status: We've confirmed that this issue impacts service, and we're continuing to investigate to determine the source of the issue. We'll provide another update within 60 minutes.
Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 11:00 UTC April 03, 2025
Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 11:00 UTC April 03, 2025
Investigating
Current status: We're investigating an issue with Snowflake Data Cloud. We'll provide an update within 60 minutes.
Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 11:00 UTC April 03, 2025
Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 11:00 UTC April 03, 2025
This incident affected: Azure - East US 2 (Virginia) (Snowflake Data Warehouse (Database)).