Page MenuHomePhabricator

cleanup memoization copy_required_intermediates_for_execution

Authored by yuhan on Jan 13 2021, 1:45 AM.



as we are moving towards IO managers, we don't need copy_required_intermediates_for_execution to be run before step execution

this diff moves the the intermediate object copy check inside the intermediate object adaptor.
when loading inputs, the adaptor will check if the source is from upstream_output and source run id is the same as parent id, if so, it will copy the object before loading.

besides, similar to D5474, we switch CP_OBJECT event to context.log the file path as log messages, so then we can clean up ObjectStoreOperation events

Test Plan


Diff Detail

R1 dagster
Automatic diff as part of commit; lint not applicable.
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 13 2021, 2:19 AM
Harbormaster failed remote builds in B24221: Diff 29466!

use execution_plan.step_handles_to_execute as the source of truth for steps to execute, because one can execute a sub plan directly without pipeline_run.step_keys_to_execute like

yuhan requested review of this revision.Jan 13 2021, 2:48 AM

That was quick! Love the big red portion.

This revision is now accepted and ready to land.Jan 13 2021, 4:56 PM