Page MenuHomePhabricator

Remove default partition selector from PartitionSetDefinition.create_schedule_definition

Authored by dgibson on Tue, Jan 5, 6:33 PM.



The old last_partition default here doesn't work very well wtih the new scheduler, which might be doing catchup and filling in multiple partitions. The schedule decorators pass in a smarter default in here - make th
at default more discoverable for users, and require a user who wants to create a schedule from a partition set to be more explicit about how each execution should match to a partition (by using that default function, or some other partition selector that they write).

Test Plan


Diff Detail

R1 dagster
Automatic diff as part of commit; lint not applicable.
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Tue, Jan 5, 6:50 PM
Harbormaster failed remote builds in B23647: Diff 28758!

Seems good to me, though we should make note of this in the release notes somewhere.

This revision is now accepted and ready to land.Wed, Jan 6, 4:50 PM

OK, I ended up changing this a bit since leaving people with no suggested function to use felt weird. I basically made the partition selectors that the built-in decorators use public (and a bit more user-friendly) and added comments / examples / put them in the apidocs. Would you mind taking another look?

fix docs formatting, add breaking change

reference the helpers in create_schedule_definition as well.


should we get rid of last_partition


should these selectors be exported from dagster?


sure, I can add this (as well as date_partition_range? that wasn't in there either)

This revision is now accepted and ready to land.Mon, Jan 11, 5:56 PM