Page MenuHomePhabricator

Throw an error on startup if there are running schedules that don't match the scheduler on the instance
ClosedPublic

Authored by dgibson on Tue, Jan 12, 12:00 AM.

Details

Summary

Right now if you change from one scheduler to another while there are running schedules, it can leave running orphaned cron jobs. Add a check for that specific case (using the new scheduler cfield on JobState added in https://dagster.phacility.com/D5919) prompting you to update.

Hopefully someday soon everybody will be on the daemon scheduler and this won't be a problem. Until then, detect this case and raise an error prompting you to go through the migration steps, which are to switch back to the old scheduler and run 'dagster schedule wipe'. The reason we need you to change the config back is that sometimes the wipe command uses the config (looking at you, K8sScheduler).

Test Plan

BK (writing now, seeing if approach seems reasonable)

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Tue, Jan 12, 1:06 AM
Harbormaster failed remote builds in B24129: Diff 29344!

Add a check to also prompt to wipe when there is anything inside the legacy schedules table (that's another way we could end up with hanging cron jobs)

prompt to migrate instead (will add the alembic migration and a test specifiically for the alembic migration path separately)

This revision is now accepted and ready to land.Tue, Jan 12, 4:35 PM