Page MenuHomeElementl

[NOT release-blocking] RFC convert OBJECT_STORE_OPERATION to LOADED_INPUT/HANDLED_OUTPUT
Changes PlannedPublic

Authored by yuhan on Jan 14 2021, 12:00 AM.

Details

Reviewers
None
Summary

this applied to historical runs with OBJECT_STORE_OPERATION logs, where OBJECT_STORE_OPERATION log records would be converted to LOADED_INPUT/HANDLED_OUTPUT records.

tracking issue: https://github.com/dagster-io/dagster/issues/3351

I'd suggest to punt on converting it and keep showing OBJECT_STORE_OPERATION for historical runs until we have better ideas for the metadata events (open discussion https://elementl.quip.com/VfO1AeIprwVc/Events-Metadata-and-ObjectManagers). Because:

  • this conversion would eliminate historical runs' OBJECT_STORE_OPERATION event logs which showed useful and good-looking info, e.g. showing file path in key entries

Screen Shot 2021-01-13 at 3.57.05 PM.png (1×4 px, 549 KB)

  • this would result in this view for the same log -- it would provide less info via dagit because the new events LOADED_INPUT/HANDLED_OUTPUT don't capture as much info as the old OBJECT_STORE_OPERATION events. (we hacked it in io managers code paths by calling context.log to log the same info, but it won't work for this event backcompat code path)

Screen Shot 2021-01-13 at 3.56.27 PM.png (1×4 px, 428 KB)

  • we don't have a good way to convert CP_OBJECT logs, converting them to either LOADED_INPUT or HANDLED_OUTPUT would be confusing. But once we have metadata events, it'd make sense to convert to those.
Test Plan

na

Diff Detail

Repository
R1 dagster
Branch
arcpatch-D5988
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

yuhan added a reviewer: schrockn.
Harbormaster returned this revision to the author for changes because remote builds failed.Jan 14 2021, 12:17 AM
Harbormaster failed remote builds in B24343: Diff 29628!
yuhan retitled this revision from RFC convert OBJECT_STORE_OPERATION to LOADED_INPUT/HANDLED_OUTPUT to [NOT release-blocking] RFC convert OBJECT_STORE_OPERATION to LOADED_INPUT/HANDLED_OUTPUT.Jan 14 2021, 1:03 AM
yuhan edited the summary of this revision. (Show Details)
yuhan requested review of this revision.Jan 14 2021, 1:05 AM

I agree that, given that converting from ObjectStoreOperation to HandledOutput / LoadedInput would be lossy, probably best to punt.

This revision now requires changes to proceed.Jan 19 2021, 4:37 PM
This revision now requires review to proceed.Feb 25 2021, 11:08 PM