AWS - US East (N. Virginia): INC0098257
Incident
Report for Snowflake
Postmortem
Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis (RCA) is available on the Snowflake Community site:
https://community.snowflake.com/s/article/INC0098257
Resolved
Current status: We identified an issue that caused a subset of infrastructure (responsible for PrivateLink traffic management) to enter a degraded state, resulting in intermittent connectivity issues for some customers. As the issue is intermittent, subsequent connection retries would have been successful. While we initially implemented temporary mitigation at 21:21 UTC, which provided relief, a long-term fix of scaling up our infrastructure was carried out to ensure system stability.
Customer experience: Customers hosted in the specified regions may have experienced intermittent connectivity issues when connecting to Snowflake services and features over PrivateLink.
Incident start time: 10:29 UTC December 07, 2023
Incident end time: 21:21 UTC December 07, 2023
Preliminary root cause: Preliminary investigation suggests an insufficient resource issue, which gradually impacted our system's ability to process PrivateLink traffic, resulting in intermittent issues on logging to Snowflake system.
A root cause analysis (RCA) document will be published within seven business days.
Customer experience: Customers hosted in the specified regions may have experienced intermittent connectivity issues when connecting to Snowflake services and features over PrivateLink.
Incident start time: 10:29 UTC December 07, 2023
Incident end time: 21:21 UTC December 07, 2023
Preliminary root cause: Preliminary investigation suggests an insufficient resource issue, which gradually impacted our system's ability to process PrivateLink traffic, resulting in intermittent issues on logging to Snowflake system.
A root cause analysis (RCA) document will be published within seven business days.
Monitoring
Current status: We identified an issue that caused a subset of infrastructure (responsible for PrivateLink traffic management) to enter a degraded state, resulting in intermittent connectivity issues for some customers. As the issue is intermittent, subsequent connection retries would have been successful. While we initially implemented temporary mitigation to provide relief, a long-term fix of scaling up our infrastructure was carried out to ensure system stability.
Customer experience: Customers hosted in the specified regions may be experiencing intermittent connectivity issues when connecting to Snowflake services and features over PrivateLink.
Incident start time: 10:29 UTC December 07, 2023
Incident end time: 21:21 UTC December 07, 2023
Preliminary root cause: Preliminary investigation suggests an insufficient resource issue, which gradually impacted our system's ability to process PrivateLink traffic, resulting in intermittent issues on logging to Snowflake system.
Customer experience: Customers hosted in the specified regions may be experiencing intermittent connectivity issues when connecting to Snowflake services and features over PrivateLink.
Incident start time: 10:29 UTC December 07, 2023
Incident end time: 21:21 UTC December 07, 2023
Preliminary root cause: Preliminary investigation suggests an insufficient resource issue, which gradually impacted our system's ability to process PrivateLink traffic, resulting in intermittent issues on logging to Snowflake system.
Update
Current status: We identified an issue causing a subset of infrastructure that facilitates PrivateLink traffic management to enter a degraded state, resulting in intermittent connectivity issues for some customers. We have implemented a temporary mitigation to provide additional relief while we continue to address the underlying source of the issue. As the issue is intermittent in nature, subsequent connection retries may also be successful. We'll provide another update within 2 hours.
Customer experience: Customers hosted in the specified regions may be experiencing intermittent connectivity issues when connecting to Snowflake services and features over PrivateLink.
Incident start time: 10:29 UTC December 07, 2023
Customer experience: Customers hosted in the specified regions may be experiencing intermittent connectivity issues when connecting to Snowflake services and features over PrivateLink.
Incident start time: 10:29 UTC December 07, 2023
Identified
Current status: We've identified the source of the issue, and we're developing and implementing a fix to restore service. We'll provide another update within 60 minutes.
Customer experience: Customers hosted in the specified regions may be experiencing intermittent connectivity issues when connecting to Snowflake services and features over PrivateLink.
Incident start time: 10:29 UTC December 07, 2023
Customer experience: Customers hosted in the specified regions may be experiencing intermittent connectivity issues when connecting to Snowflake services and features over PrivateLink.
Incident start time: 10:29 UTC December 07, 2023
Investigating
Current status: We're investigating an issue with Snowflake Data Cloud. We'll provide an update within 90 minutes.
Customer experience: Customers hosted in the specified regions may be experiencing intermittent connectivity issues when connecting to Snowflake services and features over PrivateLink.
Incident start time: 10:29 UTC December 07, 2023
Customer experience: Customers hosted in the specified regions may be experiencing intermittent connectivity issues when connecting to Snowflake services and features over PrivateLink.
Incident start time: 10:29 UTC December 07, 2023
This incident affected: AWS - US East (N. Virginia) (Snowsight).