Azure - Japan East (Tokyo, Saitama): INC0104291
Incident
Report for Snowflake
Postmortem
Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis (RCA) is available on the Snowflake Community site:
https://community.snowflake.com/s/article/INC0104291
Resolved
Current status: We've implemented the fix for this issue and monitored the environment to confirm that service was restored. If you experience additional issues or have questions, please open a support case via Snowflake Community.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Incident end time: 02:27 UTC March 12, 2024
Preliminary root cause: Initial investigation revealed an underlying code issue leading to credential problems affecting ongoing operations in the mentioned deployment.
A root cause analysis (RCA) document will be published within seven business days.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Incident end time: 02:27 UTC March 12, 2024
Preliminary root cause: Initial investigation revealed an underlying code issue leading to credential problems affecting ongoing operations in the mentioned deployment.
A root cause analysis (RCA) document will be published within seven business days.
Update
Current status: We've deployed the fix for this issue at 2:27 AM UTC, but still a small subset of Cloud Services Layer has yet to receive the update. We will continue to monitor the environment until we're confident all services are functioning properly.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Incident end time: 02:27 UTC March 12, 2024
Preliminary root cause: Initial investigation revealed an underlying code issue leading to credential problems affecting ongoing operations in the mentioned deployment.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Incident end time: 02:27 UTC March 12, 2024
Preliminary root cause: Initial investigation revealed an underlying code issue leading to credential problems affecting ongoing operations in the mentioned deployment.
Update
Current status: The fix for this issue has been implemented. However, a small portion of the cloud services layer has not yet received the update. As a result, some customers may experience intermittent success when running their queries. We are actively monitoring the environment and will continue to do so until we are confident that all services are functioning properly.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Incident end time: 02:27 UTC March 12, 2024
Preliminary root cause: Initial investigation revealed an underlying code issue leading to credential problems affecting ongoing operations in the mentioned deployment.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Incident end time: 02:27 UTC March 12, 2024
Preliminary root cause: Initial investigation revealed an underlying code issue leading to credential problems affecting ongoing operations in the mentioned deployment.
Update
Current status: The fix for this issue has been implemented. However, a small portion of the cloud services layer has not yet received the update. As a result, some customers may experience intermittent success when running their queries. We are actively monitoring the environment and will continue to do so until we are confident that all services are functioning properly.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Incident end time: 02:27 UTC March 12, 2024
Preliminary root cause: Initial investigation revealed an underlying code issue leading to credential problems affecting ongoing operations in the mentioned deployment.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Incident end time: 02:27 UTC March 12, 2024
Preliminary root cause: Initial investigation revealed an underlying code issue leading to credential problems affecting ongoing operations in the mentioned deployment.
Monitoring
Current status: We've implemented the fix for this issue, and we'll continue to monitor the environment until we're confident all services are functioning properly.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Incident end time: 02:27 UTC March 12, 2024
Preliminary root cause: Initial investigation revealed an underlying code issue leading to credential problems affecting ongoing operations in the mentioned deployment.
Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Incident end time: 02:27 UTC March 12, 2024
Preliminary root cause: Initial investigation revealed an underlying code issue leading to credential problems affecting ongoing operations in the mentioned deployment.
Update
Current status: We're continuing to investigate to determine the source of the issue. We'll provide another update within 2 hour.
Customer experience: Customers hosted in the specified regions may experience delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Customer experience: Customers hosted in the specified regions may experience delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Investigating
Current status: We're investigating an issue with Snowflake Data Cloud. We'll provide an update within 60 minutes.
Customer experience: Customers hosted in the specified regions may experience delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
Customer experience: Customers hosted in the specified regions may experience delays while attempting to access or use Snowflake services and features.
Incident start time: 22:00 UTC March 11, 2024
This incident affected: Azure - Japan East (Tokyo, Saitama) (Snowflake Data Warehouse (Database), Snowpipe (Data Ingestion)).