Page MenuHomeElementl

[sensors-7] external job storage for schedules

Authored by prha on Nov 14 2020, 12:32 AM.



Rewrites schedules to use generic job storage.

This diff is kind of beefy, but only has a couple conceptual components and is mostly mechanical:

  • Stored job ids are based on external origin, rather than origin, so we have to change how we fetch/reconcile to be based on external_origin_id
  • ScheduleState is now JobState, with job_specific_data taking ScheduleJobData or SensorJobData, depending on the job type.

This is a non-backward compatible change. It will require a schedule wipe and schedule up. I will land it only after we have split master / release branches.

Test Plan


Diff Detail

R1 dagster
Lint Not Applicable
Tests Not Applicable

Event Timeline

prha requested review of this revision.Nov 14 2020, 12:47 AM
dgibson requested changes to this revision.Nov 17 2020, 5:30 PM

ah sorry there's some overlap here with which I was *also* planning on landing after the branch cut. Is it worth rebasing on top of that for the origin piece once I rebase it and clean it up?

45–46 ↗(On Diff #25684)

imo we should just swap the implementation of id to use the new origin? dagit is a host process, externalId is more stable, it's confusing to have two IDs exposed




There's a perfectly good external_repository_origin right there already :)
origin = self._schedule_state.origin.external_repository_origin

(creating a new location handle involves spinning up a gRPC server)

197–199 ↗(On Diff #25684)

as above we don't want to do this, it spins up a new gRPC server. maybe rebase on top of instead?

This revision now requires changes to proceed.Nov 17 2020, 5:30 PM

a couple of somewhat important inlines of things to revert, but other than that this looks good. Will land the origin changes this is on top of ASAP


this is gone now no?

28–43 ↗(On Diff #26030)

this should be removed, external_id is just id now

190–225 ↗(On Diff #26030)

you should be able to revert this set of changes

This revision is now accepted and ready to land.Nov 20 2020, 4:08 PM
This revision was automatically updated to reflect the committed changes.