5 Deployments Affected: INC0092585
Incident
Report for Snowflake
Postmortem
Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis (RCA) is available on the Snowflake Community site:
https://community.snowflake.com/s/article/INC0092585
Resolved
Current status: We've coordinated with our third-party service provider, who implemented a fix to restore service to the AWS - US West (Oregon) and AWS - US East (N. Virginia) regions. The environments have remained stable, which confirms that services were restored. If you experience additional issues or have questions, please open a support case via Snowflake Community.
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Incident end time: 04:06 UTC September 19, 2023
Preliminary root cause: A third-party service provider experienced a network issue, which caused performance degradation and errors with query execution.
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Incident end time: 04:06 UTC September 19, 2023
Preliminary root cause: A third-party service provider experienced a network issue, which caused performance degradation and errors with query execution.
Monitoring
Current status: We've coordinated with our third-party service provider who implemented a fix to restore service to the AWS - US West (Oregon) region as well as AWS - US East (N. Virginia). We will continue to monitor the environment until we're confident all services are functioning properly.
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Incident end time: 04:06 UTC September 19, 2023
Preliminary root cause: A third-party service provider experienced a network issue, which caused performance degradation and errors with query execution.
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Incident end time: 04:06 UTC September 19, 2023
Preliminary root cause: A third-party service provider experienced a network issue, which caused performance degradation and errors with query execution.
Update
Current status: We continue to coordinate with our third-party service provider who has identified the issue and continuing to implement a fix to restore full service on the AWS - US West (Oregon) region. Services on AWS - US East (N. Virginia) have been restored as of 21:52 UTC where a preliminary investigation indicated a networking issue within AWS Virtual Private Cloud service. AWS continues to work on the service restoration of the AWS US West 2 region and a fix has been applied to address the potential cause of the network issue. The fix continues to propagate to the rest of the region which should incrementally alleviate the latencies or performance degradation of impacted services. Our Engineering Teams continue to actively monitor the infrastructure to ensure that the number of servers in the free pool remains in line with demand. We'll provide another update within 120 minutes.
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Update
Current status: We continue to coordinate with our third-party service provider who has identified the issue and is now implementing a fix to restore full service on the AWS - US West (Oregon) region. Services on AWS - US East (N. Virginia) have been restored as of 21:52 UTC where preliminary investigation states a networking issue within AWS Virtual Private Cloud service. Our Engineering Teams continue to actively monitor the infrastructure to ensure that the number of servers in the free pool remains in line with demand. We'll provide another update within 120 minutes.
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Update
Current status: We continue to coordinate with our third-party service provider who have identified the issue and is now implementing a fix to restore service. Our Engineering Teams are actively monitoring the infrastructure to ensure that the number of servers in the free pool remains in line with demand. We'll provide another update within 120 minutes.
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Update
Current status: We're continuing to coordinate with our third-party service provider to develop and implement a fix to restore service. Our Engineering Teams are actively monitoring the infrastructure to ensure that the number of servers in the free pool remains in line with demand. We'll provide another update within 120 minutes.
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Update
Current status: We're continuing to coordinate with our third-party service provider to develop and implement a fix to restore service. We'll provide another update within 120 minutes.
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Identified
Current status: We've identified an issue with a third-party service provider, and we're coordinating with the provider to develop and implement a fix to restore service. We'll provide another update within 60 minutes.
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Customer experience: A subset of customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 17:05 UTC September 18, 2023
Investigating
Current status: We're investigating a potential issue with Snowflake Data Cloud afffecting a subset of customers. We'll provide an update within 60 minutes or remove this message if we determine that there is no service issue.
Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 16:47 UTC September 18, 2023
Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.
Incident start time: 16:47 UTC September 18, 2023
This incident affected: AWS - US East (N. Virginia) (Snowflake Data Warehouse (Database)) and AWS - US West (Oregon) (Snowflake Data Warehouse (Database)).