Partial outage
Incident Report for DataOps.live
Resolved
We have identified that the Snowflake Bundle 2023_05, automatically enabled on the 22nd August included a change in behaviour causing grant failures to stop a pipeline within DataOps.live. We believe to have found a permanent fix which we are currently testing and plan to release as a hotfix on Thursday 24th August. We will also provide instructions on how to enable the Snowflake Bundle afterwards.

As a workaround and with the ACCOUNTADMIN role within your Snowflake tenant you can disable the Snowflake Bundle:

select SYSTEM$DISABLE_BEHAVIOR_CHANGE_BUNDLE('2023_05');

After this the affected pipelines will start to run again.

Once the hotfix is released you can re-enable the Snowflake Bundle again.
Posted Aug 23, 2023 - 12:48 UTC
Update
We are continuing to monitor for any further issues.
Posted Aug 23, 2023 - 11:25 UTC
Monitoring
We working on an update to handle the 2023_05 Snowflake behaviour change.
Posted Aug 23, 2023 - 10:05 UTC
Identified
We believe that a Snowflake behaviour introduced by the Snowflake Bundle 2023_05 change is causing the issue. This change causes grant failures to stop the pipeline, whereas previously the pipeline would continue. Learn more about the bundle from the Snowflake release notes: https://docs.snowflake.com/en/release-notes/bcr-bundles/2023_05_bundle

A workaround as the ACCOUNTADMIN role within your Snowflake tenant:

select SYSTEM$DISABLE_BEHAVIOR_CHANGE_BUNDLE('2023_05');

---We are continuing to investigate and are working on an update to handle this behaviour change.
Posted Aug 23, 2023 - 09:27 UTC
Investigating
We are aware of some DataOps.live pipelines failing and are investigating.
Posted Aug 23, 2023 - 08:47 UTC
This incident affected: Orchestrators.