AWS - US East (N. Virginia): INC0111213
Incident
Report for Snowflake
Postmortem
Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis (RCA) is available on the Snowflake Community site:
https://community.snowflake.com/s/article/INC0110718-INC0110871-INC0111066-INC0111213
Resolved
Current status: We implemented the fix for this issue on July 03, 2024, at 16:44 UTC, and monitored the environment to confirm that service was restored. If you experience additional issues or have questions, please open a support case via Snowflake Community.
Customer experience: Users in the affected region(s) intermittently were unable to sign in or use the Snowflake service via Snowsight. As the issue was intermittent, attempting to reload the page or sign in again may have succeeded.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue recurred with the previously reported problems associated with incidents INC0110718, INC0110871, and INC0111066. Our investigation revealed an additional configuration change, which added resource pressure to the affected database infrastructure including memory usage. Our engineering teams actively monitored and analyzed all components of the affected database and confirmed we have addressed the contributing factors that resulted in intermittent increases in connection requests
A root cause analysis (RCA) document will be published within seven business days.
Customer experience: Users in the affected region(s) intermittently were unable to sign in or use the Snowflake service via Snowsight. As the issue was intermittent, attempting to reload the page or sign in again may have succeeded.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue recurred with the previously reported problems associated with incidents INC0110718, INC0110871, and INC0111066. Our investigation revealed an additional configuration change, which added resource pressure to the affected database infrastructure including memory usage. Our engineering teams actively monitored and analyzed all components of the affected database and confirmed we have addressed the contributing factors that resulted in intermittent increases in connection requests
A root cause analysis (RCA) document will be published within seven business days.
Update
Current status: Since implementing the latest fixes within the affected environment on July 03, 2024, at 16:44 UTC, there have been no additional periods of impact. We are continuing to monitor the environment closely and will provide another update by 23:00 UTC on July 9, 2024.
Customer experience: Users in the affected region(s) intermittently may have been unable to sign in or use the Snowflake service via Snowsight. As the issue was intermittent, attempting to reload the page or sign in again may have succeeded.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported problems associated with incidents INC0110718, INC0110871, and INC0111066. Our latest investigation revealed an additional configuration change, adding resource pressure to the affected database infrastructure. We have now reverted this configuration change. Meanwhile, our engineering teams are actively monitoring and analyzing all components of the affected database to ensure no other factors contribute to the intermittent increases in connection requests.
Customer experience: Users in the affected region(s) intermittently may have been unable to sign in or use the Snowflake service via Snowsight. As the issue was intermittent, attempting to reload the page or sign in again may have succeeded.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported problems associated with incidents INC0110718, INC0110871, and INC0111066. Our latest investigation revealed an additional configuration change, adding resource pressure to the affected database infrastructure. We have now reverted this configuration change. Meanwhile, our engineering teams are actively monitoring and analyzing all components of the affected database to ensure no other factors contribute to the intermittent increases in connection requests.
Update
Current status: There have been no further recurrences since implementing our latest configuration change at 16:44 UTC. We will continue to monitor the environment closely over the weekend and during peak traffic periods on Monday, July 8. We will provide another update by 23:00 UTC on July 8.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported problems associated with incidents INC0110718, INC0110871, and INC0111066. Our latest investigation revealed an additional configuration change, adding resource pressure to the affected database infrastructure. We have now reverted this configuration change. Meanwhile, our engineering teams are actively monitoring and analyzing all components of the affected database to ensure no other factors contribute to the intermittent increases in connection requests.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported problems associated with incidents INC0110718, INC0110871, and INC0111066. Our latest investigation revealed an additional configuration change, adding resource pressure to the affected database infrastructure. We have now reverted this configuration change. Meanwhile, our engineering teams are actively monitoring and analyzing all components of the affected database to ensure no other factors contribute to the intermittent increases in connection requests.
Update
Current status: There have been no further recurrences since implementing our latest configuration change at 16:44 UTC. We will continue to monitor the environment closely during peak traffic during business hours and provide another update within 24 hours.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported problems associated with incidents INC0110718, INC0110871, and INC0111066. Our latest investigation revealed an additional configuration change, adding resource pressure to the affected database infrastructure. We have now reverted this configuration change. Meanwhile, our engineering teams are actively monitoring and analyzing all components of the affected database to ensure no other factors contribute to the intermittent increases in connection requests.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported problems associated with incidents INC0110718, INC0110871, and INC0111066. Our latest investigation revealed an additional configuration change, adding resource pressure to the affected database infrastructure. We have now reverted this configuration change. Meanwhile, our engineering teams are actively monitoring and analyzing all components of the affected database to ensure no other factors contribute to the intermittent increases in connection requests.
Update
Current status: There have been no further recurrences since the implementation of our latest configuration change at 16:44 UTC. Further validations are ongoing to ensure system stability. We will continue to monitor the environment closely and provide another update within two hours.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported problems associated with incidents INC0110718, INC0110871, and INC0111066. Our latest investigation revealed an additional configuration change that was adding resource pressure to the affected database infrastructure. We have now reverted this configuration change. Meanwhile, our engineering teams are actively monitoring and analyzing all components of the affected database to ensure there are no other factors contributing to the intermittent increases in connection requests.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported problems associated with incidents INC0110718, INC0110871, and INC0111066. Our latest investigation revealed an additional configuration change that was adding resource pressure to the affected database infrastructure. We have now reverted this configuration change. Meanwhile, our engineering teams are actively monitoring and analyzing all components of the affected database to ensure there are no other factors contributing to the intermittent increases in connection requests.
Update
Current status: We continue to not see any further reoccurrence of impact since 16:44 UTC following our latest configuration change. We will monitor the environment closely and provide another update within two hours.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported impact associated with issues INC0110718, INC0110871, and INC0111066. The most recent investigation identified an additional configuration change contributing to resource pressure against the affected database infrastructure, and we have reverted the identified configuration change. In parallel, engineering teams are continuing to monitor and analyze all components of the affected database to ensure there are no additional factors contributing to the intermittent increases in connection requests.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported impact associated with issues INC0110718, INC0110871, and INC0111066. The most recent investigation identified an additional configuration change contributing to resource pressure against the affected database infrastructure, and we have reverted the identified configuration change. In parallel, engineering teams are continuing to monitor and analyze all components of the affected database to ensure there are no additional factors contributing to the intermittent increases in connection requests.
Monitoring
Current status: We have seen no further reoccurrence of impact since 16:44 UTC following our latest configuration change. We will continue to closely monitor the environment and provide another update within two hours.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported impact associated with issues INC0110718, INC0110871, and INC0111066. The most recent investigation identified an additional configuration change contributing to resource pressure against the affected database infrastructure, and we have reverted the identified configuration change. In parallel, engineering teams are continuing to monitor and analyze all components of the affected database to ensure there are no additional factors contributing to the intermittent increases in connection requests.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Incident end time: 16:44 UTC July 03, 2024
Preliminary root cause: This issue is a recurrence of the previously reported impact associated with issues INC0110718, INC0110871, and INC0111066. The most recent investigation identified an additional configuration change contributing to resource pressure against the affected database infrastructure, and we have reverted the identified configuration change. In parallel, engineering teams are continuing to monitor and analyze all components of the affected database to ensure there are no additional factors contributing to the intermittent increases in connection requests.
Update
Current status: Service health has remained stable since 16:44 UTC following our latest configuration change. We are continuing to closely monitor the environment and will provide another update in 60 minutes.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Update
Current status: We've completed implementing an emergency configuration change and are monitoring the affected infrastructure to ensure stability.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Identified
Current status: We are preparing an emergency configuration change which will cause an additional momentary disruption of five to ten minutes. We'll provide an update after the change is complete.
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Customer experience: Users in the affected region(s) intermittently may be unable to sign in or use the Snowflake service via Snowsight. As the issue is intermittent, attempting to reload the page or sign in again may succeed.
Incident start time: 15:22 UTC July 03, 2024
Investigating
Current status: We're investigating an issue with Snowflake Data Cloud. We'll provide an update within 60 minutes.
Customer experience: Customers hosted in the specified regions may be unable to access or use Snowflake services and features using Snowsight.
Customer experience: Customers hosted in the specified regions may be unable to access or use Snowflake services and features using Snowsight.
This incident affected: AWS - US East (N. Virginia) (Snowsight).