HomePhabricator

Introduce Partition API

Authored by sashank on Dec 3 2019, 12:16 AM.

Description

Introduce Partition API

Summary:
This diff introduces a new API for defining pipeline partitions.

The partition axis is completely disjoint from the scheduling intervals. Each ScheduleDefinition can now specify a PartitionSelector that can select a partition of the set of pipeline partitions for a given scheduled execution. This allows room for complicated logic (e.g. pick the next partition in reverse chronological order that does not have a successful run). It also allows for future plumbing to select multiple partitions, as long as they can be enqueued efficiently.

This diff does not generate config (TBD), but it does add a tag dagster/partition, for run grouping.

Test Plan: unit

Reviewers: alangenfeld, schrockn, prha

Reviewed By: prha

Subscribers: prha, alangenfeld

Differential Revision: https://dagster.phacility.com/D1430

Details

Committed
sashankDec 6 2019, 11:39 PM
Reviewer
prha
Differential Revision
D1430: Introduce Partition API
Parents
R1:d139d6453e4f: generate empty stats object
Branches
Unknown
Tags
Unknown