Check if all the border steps of the current run have non-in-memory asset stores for reexecution.
Details
Diff Detail
- Repository
- R1 dagster
- Lint
Automatic diff as part of commit; lint not applicable. - Unit
Automatic diff as part of commit; unit tests not applicable.
Event Timeline
python_modules/dagster/dagster/core/execution/memoization.py | ||
---|---|---|
57 | We already do this check in the outer method. If we want to check here, maybe better as an invariant? | |
60 | I'm probably missing something, because I see this is used in copy_required_intermediates_for_execution as well, but do we need to fetch events from the instance? Can we not compare step_keys_to_execute with the set of steps in the full plan or something? | |
72 | if we phrased this as if not asset_store or asset_store == mem_asset_store: return False we could avoid the continue | |
75 | Are we able to raise an informative error here? I think that "Cannot perform reexecution with non persistent intermediates manager" will likely confuse users in the asset store world. |
Looks good! It feels like there are some tricky edge cases here, e.g. if only some inputs of a solid come from a prior run, so maybe worth thinking through if there are tests to write that would give us confidence on them.
will revisit tests later in the stack https://github.com/dagster-io/dagster/issues/3320