Azure - East US 2 (Virginia): INC0078999
Incident
Report for Snowflake
Postmortem
Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis (RCA) is available on the Snowflake Community site:
https://community.snowflake.com/s/article/INC0078790-INC0078999
Resolved
Current status: We've implemented the fix for this issue and monitored the environment to confirm that service was restored. If you experience additional issues or have questions, please open a support case via Snowflake Community.
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again.
Incident start time: 15:00 UTC February 23, 2023
Incident end time: 00:54 UTC February 24, 2023
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again.
Incident start time: 15:00 UTC February 23, 2023
Incident end time: 00:54 UTC February 24, 2023
Monitoring
Current status: System latency levels have fully stabilized. We've implemented the fix for this issue, and we'll continue to monitor the environment until we're confident all services are functioning properly.
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again.
Incident start time: 15:00 UTC February 23, 2023
Incident end time: 00:54 UTC February 24, 2023
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again.
Incident start time: 15:00 UTC February 23, 2023
Incident end time: 00:54 UTC February 24, 2023
Update
Current status: System latency levels are continuing to improve but have not yet fully stabilized. We are continuing to investigate the underlying root cause of the increased system latency while we work to stabilize request volume across the environment. We'll provide another update within two hours.
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again, but will continue to experience degraded performance while we work to address the system latency.
Incident start time: 15:00 UTC February 23, 2023
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again, but will continue to experience degraded performance while we work to address the system latency.
Incident start time: 15:00 UTC February 23, 2023
Update
Current status: We have corrected an issue impacting a system responsible for helping request throttling within the environment. We are starting to observe an improvement in system latency levels; however, service health has not yet stabilized. We are continuing our efforts to mitigate the increased latency and normalize traffic across the impacted systems.
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again, but will continue to experience degraded performance while we work to address the system latency.
Incident start time: 15:00 UTC February 23, 2023
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again, but will continue to experience degraded performance while we work to address the system latency.
Incident start time: 15:00 UTC February 23, 2023
Update
Current status: We have identified a potential issue within a system responsible for helping request throttling within the environment, and we are investigating why the system is not functioning as expected. We'll provide another update within 60 minutes.
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again, but will continue to experience degraded performance while we work to address the system latency.
Incident start time: 15:00 UTC February 23, 2023
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again, but will continue to experience degraded performance while we work to address the system latency.
Incident start time: 15:00 UTC February 23, 2023
Update
Current status: We are continuing our efforts to mitigate the increased latency and normalize traffic across the impacted systems.
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again, but will continue to experience degraded performance while we work to address the system latency.
Incident start time: 15:00 UTC February 23, 2023
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using the Snowsight UI. Additionally, customers may observe that Snowpipe file ingestion and replication to and from the specified region is stopped or delayed. Between approximately 20:50 UTC and 21:15 UTC, some customers may have experienced a complete inability to access Snowflake. Affected customers should now be able to log in again, but will continue to experience degraded performance while we work to address the system latency.
Incident start time: 15:00 UTC February 23, 2023
Update
Current status: While we investigate the source of the issue, we are continuing to attempt multiple recovery operations such as reverting any recent changes within the environment, re-provisioning impacted systems, adding additional capacity to handle the increased load, and isolating and throttling impacted systems to help normalize traffic across the affected environments. Our primary investigation remains focused on identifying the cause of the increased latency within the metadata database layer, which is causing downstream impact to the Cloud Services layer and customer requests such as operations within the Snowsight web interface and queries.
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using Snowsight. Additionally, customers may observe that Snowpipe file ingestion is stopped or delayed.
Incident start time: 15:00 UTC February 23, 2023
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using Snowsight. Additionally, customers may observe that Snowpipe file ingestion is stopped or delayed.
Incident start time: 15:00 UTC February 23, 2023
Update
Current status: We are continuing to investigate an issue causing impact within our Cloud Services layer and preventing the impacted systems from auto recovering. We have further isolated the issue to systems within the metadata database layer that are experiencing an unexpected increase in latency. Due to the increased latency, requests are becoming throttled and may experience delays or timeouts. We are attempting to isolate the cause of the increased latency and recover the impacted systems to restore service.
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using Snowsight. Additionally, customers may observe that Snowpipe file ingestion is stopped or delayed.
Incident start time: 15:00 UTC February 23, 2023
Customer experience: Customers hosted in the specified regions may experience delays or failures while attempting to execute queries or access Snowflake services and features using Snowsight. Additionally, customers may observe that Snowpipe file ingestion is stopped or delayed.
Incident start time: 15:00 UTC February 23, 2023
Update
Current status: We've performed an initial series of service recovery operations, but impact from the issue persists. We're continuing to investigate to determine the source of the issue. We'll provide another update within 60 minutes.
Customer experience: Customers hosted in the specified regions may experience delays while attempting to access or use Snowflake services and features using Snowsight. Additionally, customers may observe that Snowpipe file ingestion is stopped or delayed.
Incident start time: 15:30 UTC February 23, 2023
Customer experience: Customers hosted in the specified regions may experience delays while attempting to access or use Snowflake services and features using Snowsight. Additionally, customers may observe that Snowpipe file ingestion is stopped or delayed.
Incident start time: 15:30 UTC February 23, 2023
Update
Current status: We're continuing to investigate the issue with Snowflake Data Cloud. We'll provide another update within 60 minutes.
Customer experience: Customers hosted in the specified regions may experience delays while attempting to access or use Snowflake services and features using Snowsight. Additionally, customers may observe that Snowpipe file ingestion is stopped or delayed.
Incident start time: 15:30 UTC February 23, 2023
Customer experience: Customers hosted in the specified regions may experience delays while attempting to access or use Snowflake services and features using Snowsight. Additionally, customers may observe that Snowpipe file ingestion is stopped or delayed.
Incident start time: 15:30 UTC February 23, 2023
Investigating
Current status: We're investigating a potential issue with Snowflake Data Cloud. We'll provide an update within 60 minutes or remove this message if we determine that there is no service issue.
Customer experience: Customers hosted in the specified regions may experience delays while attempting to access or use Snowflake services and features using Snowsight.
Incident start time: 16:08 UTC February 23, 2023
Customer experience: Customers hosted in the specified regions may experience delays while attempting to access or use Snowflake services and features using Snowsight.
Incident start time: 16:08 UTC February 23, 2023
This incident affected: Azure - East US 2 (Virginia) (Snowflake Data Warehouse (Database), Snowpipe (Data Ingestion), Replication, Snowsight).