Missing Worksheets in Snowflake UI
Incident Report for Snowflake Computing
Postmortem

Article Body
Between Jan 28 5:45 am on Jan 28, 2019 and 1:01 am PST on Jan 29, 2019 Snowflake customers with deployments on AZURE - East US 2 and AZURE- West Europe regions were unable to retrieve or save worksheets in Snowflake UI.

Resolution Action

As soon as Snowflake found the issue with the worksheets we debugged the problem and narrowed it down to failure of the CORS (Cross-Origin Resource Sharing) allowed headers check in the communication between the browser and Azure storage, where worksheet data is stored. Snowflake escalated the issue with Microsoft Azure and simultaneously worked on making a temporary mitigation to storage configuration to accept the headers as sent by the browser. Microsoft Azure fixed the problem in Azure Storage configuration and started to roll out the code at 7:43 PM Jan 28, 2019. The patch rollout was completed to both regions at 1:01 am Jan 29, 2019. Snowflake then confirmed the fix was complete and rolled back our temporary mitigation.

Root Cause(s)

The connectivity disruption was caused by a bug in Microsoft Azure code where the CORS (Cross-Origin Resource Sharing) header “content-type” was rejected although “Content-Type” was listed as an allowed header. The Snowflake UI loaded in a web browser was sending the header tag as “content-type” (all lower case). Allowed headers should be, and always have been, matched case-insensitively, but Azure Storage bug began to match case-sensitively, causing all requests from browsers with “content-type” to be rejected as not matching CORS policy. This issue manifested in Chrome, Safari and Firefox. We are waiting for a full RCA from Microsoft Azure.

Improvements/Changes

1. Snowflake Engineering already collects telemetry data from Snowflake UI to gain insights into the quality of service. We will continue to make changes in this area.

2. We are now waiting for Microsoft Azure team to publish an RCA for the incident and will follow up with them on future enhancements to prevent service interruptions due to these kind of errors.

Posted 20 days ago. Jan 30, 2019 - 11:14 PST

Resolved
Dear Snowflake Customer,
Microsoft Azure has deployed a fix to address this issue and Snowflake has rolled back the workaround. Snowflake UI is working normally now. Please reach out to Snowflake support if you have follow up questions. Apologies for the inconvenience caused by this issue.

Kind Regards,
Snowflake Support
Posted 21 days ago. Jan 29, 2019 - 23:44 PST
This incident affected: AZURE - West Europe (Snowflake Data Warehouse (Database)) and AZURE - East US 2 (Snowflake Data Warehouse (Database)).