Page MenuHomeElementl

Fail harder when there's a config error generating a sensor run
ClosedPublic

Authored by dgibson on Jan 25 2021, 9:00 PM.
Tags
None
Referenced Files
Unknown Object (File)
Dec 13 2022, 5:16 AM
Unknown Object (File)
Nov 25 2022, 10:43 PM
Unknown Object (File)
Nov 11 2022, 7:48 PM
Unknown Object (File)
Nov 11 2022, 7:19 PM
Unknown Object (File)
Nov 11 2022, 3:13 PM
Unknown Object (File)
Nov 5 2022, 3:03 PM
Unknown Object (File)
Oct 29 2022, 3:42 AM
Unknown Object (File)
Oct 28 2022, 9:06 AM

Details

Summary

Right now if the sensor code succeeds but the subsequent RunConfig is misconfigured, we still create the run, just putting it immediately in a FAILED state (and taking up the run key). This is logic inherited from the scheduler, where it arguably makes sense to do this since there's usually exactly one run per tick but is likely not what we want for sensors - instead, we want to fix up the code and not take up the run key until a run is actually valid and ready to launch.

Should resolve https://github.com/dagster-io/dagster/issues/3593 and https://github.com/dagster-io/dagster/issues/3583 (kind of)

Test Plan

New BK coverage

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

I think this makes sense, but the way we're asking people to write sensors means this will fail every tick until a new code version is pushed.

This revision is now accepted and ready to land.Jan 25 2021, 11:35 PM

it will also retry any runs that don't have run keys and launch before the failure - all the more reason to use run keys i guess