Details

Reviewers

• dgibson
• sidkmenon
• prha

Commits

R1:b697f6228565: [RFC] Construct partitions using serializable parameters

Summary

Depends on D7642.

Rather than construct the partitions using a user specified closure, we
want to capture all the relevant serializable information that can be used to
potentially generate the closure at run time when partition names are being retrieved.

We envision three classes of parameters to allow for this:

static
time based (e.g. a date partition)
dynamic (the current original behavior)

Test Plan

N/A

Diff Detail

Repository

R1 dagster

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

• rexledesma created this revision.Apr 27 2021, 7:34 AM

• rexledesma held this revision as a draft.

• rexledesma added inline comments.Apr 27 2021, 7:45 AM

python_modules/dagster/dagster/core/definitions/partition.py
82	from https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/python_modules/dagster/dagster/utils/partitions.py#L85:5-85:25

Harbormaster failed remote builds in B29544: Diff 36270!Apr 27 2021, 7:55 AM

add tests

Harbormaster failed remote builds in B29637: Diff 36390!Apr 28 2021, 7:02 AM

mypy

Harbormaster failed remote builds in B29661: Diff 36420!Apr 28 2021, 5:44 PM

mypy

Harbormaster completed remote builds in B29662: Diff 36422.Apr 28 2021, 6:15 PM

Harbormaster failed remote builds in B29695: Diff 36465!Apr 29 2021, 12:35 AM

invariant tests

Harbormaster completed remote builds in B29696: Diff 36466.Apr 29 2021, 1:01 AM

• rexledesma published this revision for review.Apr 29 2021, 1:05 AM

• rexledesma added a child revision: D7654: [3/n] Thread schedule decorator parameters to partition parameters.

• rexledesma edited the summary of this revision. (Show Details)

• rexledesma added a parent revision: D7642: Add exhaustive check for mypy.

• rexledesma retitled this revision from [RFC] Construct partitions using user specified parameters to [2/n][RFC] Construct partitions using user specified parameters.Apr 29 2021, 1:10 AM

Harbormaster completed remote builds in B29698: Diff 36469.Apr 29 2021, 2:05 AM

• prha accepted this revision.Apr 29 2021, 5:46 PM

• prha added inline comments.

python_modules/dagster/dagster/core/definitions/partition.py
89	nit: prefix with _, to show that it's a locally scoped function.

This revision is now accepted and ready to land.Apr 29 2021, 5:46 PM

might have to go to the drawing board to decouple partition sets from schedules... (i.e. how can we make partition sets that are not explicitly tied to one of our built in schedules?)

rebase

• rexledesma retitled this revision from [2/n][RFC] Construct partitions using user specified parameters to [RFC] Construct partitions using serializable parameters.May 17 2021, 9:46 PM

Harbormaster completed remote builds in B30687: Diff 37734.May 17 2021, 10:08 PM

• rexledesma added a child revision: D7976: Save partition offset in partition parameters.May 19 2021, 1:40 AM