Partial outage

Incident Report for DataOps.live

Resolved

We have identified that the Snowflake Bundle 2023_05, automatically enabled on the 22nd August included a change in behaviour causing grant failures to stop a pipeline within DataOps.live. We believe to have found a permanent fix which we are currently testing and plan to release as a hotfix on Thursday 24th August. We will also provide instructions on how to enable the Snowflake Bundle afterwards.

As a workaround and with the ACCOUNTADMIN role within your Snowflake tenant you can disable the Snowflake Bundle:

select SYSTEM$DISABLE_BEHAVIOR_CHANGE_BUNDLE('2023_05');

After this the affected pipelines will start to run again.

Once the hotfix is released you can re-enable the Snowflake Bundle again.
Posted Aug 23, 2023 - 12:48 UTC

Update

We are continuing to monitor for any further issues.
Posted Aug 23, 2023 - 11:25 UTC

Monitoring

We working on an update to handle the 2023_05 Snowflake behaviour change.
Posted Aug 23, 2023 - 10:05 UTC

Identified

We believe that a Snowflake behaviour introduced by the Snowflake Bundle 2023_05 change is causing the issue. This change causes grant failures to stop the pipeline, whereas previously the pipeline would continue. Learn more about the bundle from the Snowflake release notes: https://docs.snowflake.com/en/release-notes/bcr-bundles/2023_05_bundle

A workaround as the ACCOUNTADMIN role within your Snowflake tenant:

select SYSTEM$DISABLE_BEHAVIOR_CHANGE_BUNDLE('2023_05');

---We are continuing to investigate and are working on an update to handle this behaviour change.
Posted Aug 23, 2023 - 09:27 UTC

Investigating

We are aware of some DataOps.live pipelines failing and are investigating.
Posted Aug 23, 2023 - 08:47 UTC
This incident affected: Orchestrators.