Page MenuHomePhabricator

[NOT release-blocking] RFC convert OBJECT_STORE_OPERATION to LOADED_INPUT/HANDLED_OUTPUT
Needs ReviewPublic

Authored by yuhan on Thu, Jan 14, 12:00 AM.

Details

Summary

this applied to historical runs with OBJECT_STORE_OPERATION logs, where OBJECT_STORE_OPERATION log records would be converted to LOADED_INPUT/HANDLED_OUTPUT records.

tracking issue: https://github.com/dagster-io/dagster/issues/3351

I'd suggest to punt on converting it and keep showing OBJECT_STORE_OPERATION for historical runs until we have better ideas for the metadata events (open discussion https://elementl.quip.com/VfO1AeIprwVc/Events-Metadata-and-ObjectManagers). Because:

  • this conversion would eliminate historical runs' OBJECT_STORE_OPERATION event logs which showed useful and good-looking info, e.g. showing file path in key entries

  • this would result in this view for the same log -- it would provide less info via dagit because the new events LOADED_INPUT/HANDLED_OUTPUT don't capture as much info as the old OBJECT_STORE_OPERATION events. (we hacked it in io managers code paths by calling context.log to log the same info, but it won't work for this event backcompat code path)

  • we don't have a good way to convert CP_OBJECT logs, converting them to either LOADED_INPUT or HANDLED_OUTPUT would be confusing. But once we have metadata events, it'd make sense to convert to those.
Test Plan

na

Diff Detail

Repository
R1 dagster
Branch
arcpatch-D5988
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

yuhan added a reviewer: schrockn.
Harbormaster returned this revision to the author for changes because remote builds failed.Thu, Jan 14, 12:17 AM
Harbormaster failed remote builds in B24343: Diff 29628!
yuhan retitled this revision from RFC convert OBJECT_STORE_OPERATION to LOADED_INPUT/HANDLED_OUTPUT to [NOT release-blocking] RFC convert OBJECT_STORE_OPERATION to LOADED_INPUT/HANDLED_OUTPUT.Thu, Jan 14, 1:03 AM
yuhan edited the summary of this revision. (Show Details)
yuhan requested review of this revision.Thu, Jan 14, 1:05 AM