Page MenuHomeElementl

[crag] partition schedules migration guide
ClosedPublic

Authored by cdecarolis on Jul 20 2021, 10:08 PM.

Details

Summary

Title. I wish that there was a way to use partitioned config in tandem with config mapping, it would be a nice additional win if we could re-use a single partitioned config among multiple jobs.

Test Plan

adding unit tests

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 20 2021, 10:50 PM
Harbormaster failed remote builds in B33950: Diff 41938!
docs/content/guides/dagster/graph_job_op.mdx
422

For consistency with tense in the other sections, avoid "would".

424

This is an important thing to point out, but relevant to the earlier schedule examples as well - maybe better to put it up with them?

452

The new graph APIs are based on the observation that a partitioned job exists independent of its schedule - i.e. it can be backfilled without its schedule - so it's confusing to attach the partitions to the schedule instead of the job. So I think it would be useful to emphasize that in some way. E.g. something like "with the new APIs, you first define a partitioning for the job, then you derive a schedule from that partitioning".

459

In other examples, I avoided naming things like "my_op" / "my_solid", so that the names could be the same across the new/old examples (hopefully making them easier to compare).

This revision is now accepted and ready to land.Jul 21 2021, 9:48 PM