Page MenuHomeElementl

[crag] partition schedules migration guide
ClosedPublic

Authored by cdecarolis on Jul 20 2021, 10:08 PM.
Tags
None
Referenced Files
Unknown Object (File)
Wed, May 24, 4:43 AM
Unknown Object (File)
Sat, May 20, 11:44 PM
Unknown Object (File)
Sat, May 20, 8:34 PM
Unknown Object (File)
Thu, May 18, 8:59 PM
Unknown Object (File)
Sun, May 14, 8:53 PM
Unknown Object (File)
Sat, May 13, 10:17 PM
Unknown Object (File)
May 1 2023, 5:17 AM
Unknown Object (File)
Mar 26 2023, 10:21 PM
Subscribers
None

Details

Summary

Title. I wish that there was a way to use partitioned config in tandem with config mapping, it would be a nice additional win if we could re-use a single partitioned config among multiple jobs.

Test Plan

adding unit tests

Diff Detail

Repository
R1 dagster
Branch
partition_schedules_migration_guide
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 20 2021, 10:50 PM
Harbormaster failed remote builds in B33950: Diff 41938!
docs/content/guides/dagster/graph_job_op.mdx
422

For consistency with tense in the other sections, avoid "would".

424

This is an important thing to point out, but relevant to the earlier schedule examples as well - maybe better to put it up with them?

452

The new graph APIs are based on the observation that a partitioned job exists independent of its schedule - i.e. it can be backfilled without its schedule - so it's confusing to attach the partitions to the schedule instead of the job. So I think it would be useful to emphasize that in some way. E.g. something like "with the new APIs, you first define a partitioning for the job, then you derive a schedule from that partitioning".

459

In other examples, I avoided naming things like "my_op" / "my_solid", so that the names could be the same across the new/old examples (hopefully making them easier to compare).

This revision is now accepted and ready to land.Jul 21 2021, 9:48 PM