Page MenuHomePhabricator

Remove default partition selector from PartitionSetDefinition.create_schedule_definition
ClosedPublic

Authored by dgibson on Tue, Jan 5, 6:33 PM.

Details

Summary

The old last_partition default here doesn't work very well wtih the new scheduler, which might be doing catchup and filling in multiple partitions. The schedule decorators pass in a smarter default in here - make th
at default more discoverable for users, and require a user who wants to create a schedule from a partition set to be more explicit about how each execution should match to a partition (by using that default function, or some other partition selector that they write).

Test Plan

BK

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Tue, Jan 5, 6:50 PM
Harbormaster failed remote builds in B23647: Diff 28758!

Seems good to me, though we should make note of this in the release notes somewhere.

This revision is now accepted and ready to land.Wed, Jan 6, 4:50 PM

OK, I ended up changing this a bit since leaving people with no suggested function to use felt weird. I basically made the partition selectors that the built-in decorators use public (and a bit more user-friendly) and added comments / examples / put them in the apidocs. Would you mind taking another look?

fix docs formatting, add breaking change

reference the helpers in create_schedule_definition as well.

python_modules/dagster/dagster/core/definitions/partition.py
32–33

should we get rid of last_partition

python_modules/dagster/dagster/utils/partitions.py
188

should these selectors be exported from dagster?

python_modules/dagster/dagster/utils/partitions.py
188

sure, I can add this (as well as date_partition_range? that wasn't in there either)

This revision is now accepted and ready to land.Mon, Jan 11, 5:56 PM