Page MenuHomeElementl

[crag] partition schedules migration guide
ClosedPublic

Authored by cdecarolis on Jul 20 2021, 10:08 PM.

Details

Summary

Title. I wish that there was a way to use partitioned config in tandem with config mapping, it would be a nice additional win if we could re-use a single partitioned config among multiple jobs.

Test Plan

adding unit tests

Diff Detail

Repository
R1 dagster
Branch
partition_schedules_migration_guide
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 20 2021, 10:50 PM
Harbormaster failed remote builds in B33950: Diff 41938!
docs/content/guides/dagster/graph_job_op.mdx
415

For consistency with tense in the other sections, avoid "would".

417

This is an important thing to point out, but relevant to the earlier schedule examples as well - maybe better to put it up with them?

445

The new graph APIs are based on the observation that a partitioned job exists independent of its schedule - i.e. it can be backfilled without its schedule - so it's confusing to attach the partitions to the schedule instead of the job. So I think it would be useful to emphasize that in some way. E.g. something like "with the new APIs, you first define a partitioning for the job, then you derive a schedule from that partitioning".

452

In other examples, I avoided naming things like "my_op" / "my_solid", so that the names could be the same across the new/old examples (hopefully making them easier to compare).

This revision is now accepted and ready to land.Jul 21 2021, 9:48 PM