AZURE - US East 2 (Virginia): SI-20200827
Incident
Report for Snowflake
Postmortem
Dear Customer,
Snowflake Engineering has completed the postmortem of this service incident, and a detailed Root Cause Analysis is made available on the Snowflake Community site.
https://community.snowflake.com/s/article/SI-20200827
We apologize for the service incident and the impact on your applications.
If you have any questions or difficulty accessing the Snowflake community site, please send feedback by submitting a support request via the Snowflake Community Site.
Thank you,
Snowflake Support
Resolved
All Snowflake Services are now healthy.
Update: As of 05:30 PT Sep 02, 2020
Query History is now available. Thank you for your patience.
We apologize for the inconvenience. If you have any questions or see any related issues, please submit a support request ticket via the Snowflake Community Site.
Thank you,
Snowflake Support
Update: As of 12:45 PT Sep 01, 2020
Snowflake Engineering completed testing the code required to extract the inaccessible query history data for dates before Aug 27, 2020. We are on track to make the data available by Sep 3, 2020.
We will provide another update on the progress of this task at 18:00 PT Sep 01, 2020.
Update: As of 11:45 PT Aug 31, 2020
Snowflake Engineering is testing the code required to extract the inaccessible query history data for dates before Aug 25, 2020, and we are expecting to make the data available by Sep 3, 2020.
We will provide another update on the progress of this task at 18:00 PT Sep 01, 2020.
Update: As of 17:15 PT Aug 28, 2020
Snowflake Engineering is working on the code that is required to extract the inaccessible query history data, and it is progressing at a healthy pace. We expect the release the code for testing over the weekend. We will provide another update on the progress of this task at 18:00 PT Aug 29, 2020.
Update: As of 18:00 PT Aug 27, 2020
Snowflake Engineering has identified the issue with the inaccessible query history data and is currently working on restoring access to the data. We will provide another update on the progress of this task at 18:00 PT Aug 28, 2020.
Here is a detailed update for the symptoms you may have experienced starting 17:00 PT Aug 25, 2020:
1. Orphaned table locks - This issue has been resolved, and all orphaned locks as of Aug 25, 2020, have been released.
2. Query History
a. Query History UI - This page now loads up correctly. The query history will show all queries executed after 05:00 PT Aug 27,
2020.
b. Query History by Information Schema - This command will show query history for all queries executed after 05:00 PT Aug
27, 2020.
c. Query History by Account Usage - The job history for any queries executed between 17:00 PT Aug 25, 2020, and 05:00 PT
Aug 27, 2020, is not captured.
We will post a Root Cause Analysis report within the next seven business days.
We apologize for the inconvenience. If you have any questions or see any related issues, please submit a support request ticket via the Snowflake Community Site.
Update: As of 05:30 PT Sep 02, 2020
Query History is now available. Thank you for your patience.
We apologize for the inconvenience. If you have any questions or see any related issues, please submit a support request ticket via the Snowflake Community Site.
Thank you,
Snowflake Support
Update: As of 12:45 PT Sep 01, 2020
Snowflake Engineering completed testing the code required to extract the inaccessible query history data for dates before Aug 27, 2020. We are on track to make the data available by Sep 3, 2020.
We will provide another update on the progress of this task at 18:00 PT Sep 01, 2020.
Update: As of 11:45 PT Aug 31, 2020
Snowflake Engineering is testing the code required to extract the inaccessible query history data for dates before Aug 25, 2020, and we are expecting to make the data available by Sep 3, 2020.
We will provide another update on the progress of this task at 18:00 PT Sep 01, 2020.
Update: As of 17:15 PT Aug 28, 2020
Snowflake Engineering is working on the code that is required to extract the inaccessible query history data, and it is progressing at a healthy pace. We expect the release the code for testing over the weekend. We will provide another update on the progress of this task at 18:00 PT Aug 29, 2020.
Update: As of 18:00 PT Aug 27, 2020
Snowflake Engineering has identified the issue with the inaccessible query history data and is currently working on restoring access to the data. We will provide another update on the progress of this task at 18:00 PT Aug 28, 2020.
Here is a detailed update for the symptoms you may have experienced starting 17:00 PT Aug 25, 2020:
1. Orphaned table locks - This issue has been resolved, and all orphaned locks as of Aug 25, 2020, have been released.
2. Query History
a. Query History UI - This page now loads up correctly. The query history will show all queries executed after 05:00 PT Aug 27,
2020.
b. Query History by Information Schema - This command will show query history for all queries executed after 05:00 PT Aug
27, 2020.
c. Query History by Account Usage - The job history for any queries executed between 17:00 PT Aug 25, 2020, and 05:00 PT
Aug 27, 2020, is not captured.
We will post a Root Cause Analysis report within the next seven business days.
We apologize for the inconvenience. If you have any questions or see any related issues, please submit a support request ticket via the Snowflake Community Site.
Update
We are continuing to investigate the issue with Snowflake services. Please be advised that the issue is being faced with a minimal number of accounts on the deployment.
We will post the next update as soon as the Job history issue is resolved or in 2 hours at the latest.
We sincerely apologize for the inconvenience. If you have any questions or see any related issues, please submit a support request ticket via the Snowflake Community Site.
We will post the next update as soon as the Job history issue is resolved or in 2 hours at the latest.
We sincerely apologize for the inconvenience. If you have any questions or see any related issues, please submit a support request ticket via the Snowflake Community Site.
Update
We are still continuing to investigate the issue with Snowflake services. We will provide an update as soon as we have more information. We plan to provide an update in 60 minutes.
We apologize for the inconvenience. If you have any questions or see any related issues, please submit a support request ticket via the Snowflake Community Site.
We apologize for the inconvenience. If you have any questions or see any related issues, please submit a support request ticket via the Snowflake Community Site.
Identified
We have identified the problem with the Snowflake Service that is interrupting the following services:
1. Job history is failing.
2. Statements like SHOW LOCKS that access job metadata are failing.
3. In few cases, Snowflake transactions are unable to commit/rollback since they were unable to access job data. These may block other transactions.
Workloads other than these should be working.
Incident Start Time: August 26, 2020, at 12:00 PDT
We will provide an update within 90 minutes or as soon as we have more details on the status of the issue.
1. Job history is failing.
2. Statements like SHOW LOCKS that access job metadata are failing.
3. In few cases, Snowflake transactions are unable to commit/rollback since they were unable to access job data. These may block other transactions.
Workloads other than these should be working.
Incident Start Time: August 26, 2020, at 12:00 PDT
We will provide an update within 90 minutes or as soon as we have more details on the status of the issue.
This incident affected: Azure - East US 2 (Virginia) (Databases, Tables, and Views, Data Loading and Unloading).