HomePhabricator

cleanup memoization copy_required_intermediates_for_execution

Description

cleanup memoization copy_required_intermediates_for_execution

Summary:
as we are moving towards IO managers, we don't need copy_required_intermediates_for_execution to be run before step execution

this diff moves the the intermediate object copy check inside the intermediate object adaptor.
when loading inputs, the adaptor will check if the source is from upstream_output and source run id is the same as parent id, if so, it will copy the object before loading.

besides, similar to D5474, we switch CP_OBJECT event to context.log the file path as log messages, so then we can clean up ObjectStoreOperation events

Test Plan: bk

Reviewers: sandyryza, alangenfeld, cdecarolis

Reviewed By: sandyryza

Differential Revision: https://dagster.phacility.com/D5947

Details

Provenance
yuhanAuthored on Wed, Jan 13, 1:39 AM
Reviewer
sandyryza
Differential Revision
D5947: cleanup memoization copy_required_intermediates_for_execution
Parents
R1:4acaa7517c1d: [docs] mention IO manager docs in the migration guide #3455
Branches
Unknown
Tags
Unknown