Page MenuHomeElementl

input/output-address-operation-1 intermediate storage set/get from address
ClosedPublic

Authored by yuhan on Sep 28 2020, 11:47 PM.

Details

Summary

to support the memoization demo, we can use intermediate_storage (basically the object store) to set the data object to a given address and later we can get the data object from the address

Test Plan

unit

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

yuhan retitled this revision from input/output address operation via intermediate storage to input/output-address-operation-1 intermediate storage set/get from address.Sep 29 2020, 4:48 AM

LGTM! Thanks so much for doing this so fast.

python_modules/dagster/dagster/core/storage/intermediate_storage.py
226

I do think that it is a bit strange that address would be an optional param, but then the check doesn't reflect that optionality. However, I guess that isn't a huge issue seeing as we'll be merging this design into set_intermediate_object at some point, anyway.

This revision is now accepted and ready to land.Sep 29 2020, 2:15 PM
python_modules/dagster/dagster/core/storage/intermediate_storage.py
226

the merging happens in D4579 but im a bit hesitant on that one because it updates user-facing abstract methods and we are not sure about the change

python_modules/dagster/dagster/core/errors.py
472

Will all IOErrors mean that the address is invalid? There could be a failure to maintain a connection to some remote data source.

python_modules/dagster/dagster/core/storage/intermediate_storage.py
218

Nitpick: exprimental -> experimental. Also, it would be good to raise an experimental_fn_warning when this is used.

python_modules/dagster/dagster/core/errors.py
472

the call site of it will also pass in a detailed error message. maybe get rid of the default message and only surface the actual IO error message?

python_modules/dagster/dagster/core/storage/intermediate_storage.py
218

good call

yuhan marked 2 inline comments as done.

experimental + DagsterInvalidAddressForAsset -> DagsterAddressIOError