Page MenuHomeElementl

[crag] partition schedules migration guide

Authored by cdecarolis on Jul 20 2021, 10:08 PM.
Referenced Files
Unknown Object (File)
Sun, Mar 26, 10:21 PM
Unknown Object (File)
Fri, Mar 24, 8:20 AM
Unknown Object (File)
Tue, Mar 21, 10:06 AM
Unknown Object (File)
Tue, Mar 21, 8:58 AM
Unknown Object (File)
Thu, Mar 16, 3:10 PM
Unknown Object (File)
Wed, Mar 15, 9:01 AM
Unknown Object (File)
Sat, Mar 11, 4:08 AM
Unknown Object (File)
Sat, Mar 4, 4:39 AM



Title. I wish that there was a way to use partitioned config in tandem with config mapping, it would be a nice additional win if we could re-use a single partitioned config among multiple jobs.

Test Plan

adding unit tests

Diff Detail

R1 dagster
Lint Not Applicable
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 20 2021, 10:50 PM
Harbormaster failed remote builds in B33950: Diff 41938!

For consistency with tense in the other sections, avoid "would".


This is an important thing to point out, but relevant to the earlier schedule examples as well - maybe better to put it up with them?


The new graph APIs are based on the observation that a partitioned job exists independent of its schedule - i.e. it can be backfilled without its schedule - so it's confusing to attach the partitions to the schedule instead of the job. So I think it would be useful to emphasize that in some way. E.g. something like "with the new APIs, you first define a partitioning for the job, then you derive a schedule from that partitioning".


In other examples, I avoided naming things like "my_op" / "my_solid", so that the names could be the same across the new/old examples (hopefully making them easier to compare).

This revision is now accepted and ready to land.Jul 21 2021, 9:48 PM