Page MenuHomeElementl

[RFC] Construct partitions using serializable parameters
ClosedPublic

Authored by rexledesma on Apr 27 2021, 7:34 AM.

Details

Summary

Depends on D7642.

Rather than construct the partitions using a user specified closure, we
want to capture all the relevant serializable information that can be used to
potentially generate the closure at run time when partition names are being retrieved.

We envision three classes of parameters to allow for this:

  • static
  • time based (e.g. a date partition)
  • dynamic (the current original behavior)
Test Plan

N/A

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

rexledesma retitled this revision from [RFC] Construct partitions using user specified parameters to [2/n][RFC] Construct partitions using user specified parameters.Apr 29 2021, 1:10 AM
prha added inline comments.
python_modules/dagster/dagster/core/definitions/partition.py
89

nit: prefix with _, to show that it's a locally scoped function.

This revision is now accepted and ready to land.Apr 29 2021, 5:46 PM

might have to go to the drawing board to decouple partition sets from schedules... (i.e. how can we make partition sets that are not explicitly tied to one of our built in schedules?)

rexledesma retitled this revision from [2/n][RFC] Construct partitions using user specified parameters to [RFC] Construct partitions using serializable parameters.May 17 2021, 9:46 PM