Page MenuHomeElementl

[crag] partition schedules migration guide

Authored by cdecarolis on Jul 20 2021, 10:08 PM.
Referenced Files
Unknown Object (File)
Wed, May 24, 4:43 AM
Unknown Object (File)
Sat, May 20, 11:44 PM
Unknown Object (File)
Sat, May 20, 8:34 PM
Unknown Object (File)
Thu, May 18, 8:59 PM
Unknown Object (File)
Sun, May 14, 8:53 PM
Unknown Object (File)
Sat, May 13, 10:17 PM
Unknown Object (File)
May 1 2023, 5:17 AM
Unknown Object (File)
Mar 26 2023, 10:21 PM



Title. I wish that there was a way to use partitioned config in tandem with config mapping, it would be a nice additional win if we could re-use a single partitioned config among multiple jobs.

Test Plan

adding unit tests

Diff Detail

R1 dagster
Lint Not Applicable
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 20 2021, 10:50 PM
Harbormaster failed remote builds in B33950: Diff 41938!

For consistency with tense in the other sections, avoid "would".


This is an important thing to point out, but relevant to the earlier schedule examples as well - maybe better to put it up with them?


The new graph APIs are based on the observation that a partitioned job exists independent of its schedule - i.e. it can be backfilled without its schedule - so it's confusing to attach the partitions to the schedule instead of the job. So I think it would be useful to emphasize that in some way. E.g. something like "with the new APIs, you first define a partitioning for the job, then you derive a schedule from that partitioning".


In other examples, I avoided naming things like "my_op" / "my_solid", so that the names could be the same across the new/old examples (hopefully making them easier to compare).

This revision is now accepted and ready to land.Jul 21 2021, 9:48 PM