Page MenuHomePhabricator

input/output-address-operation-1 intermediate storage set/get from address
ClosedPublic

Authored by yuhan on Sep 28 2020, 11:47 PM.

Details

Summary

to support the memoization demo, we can use intermediate_storage (basically the object store) to set the data object to a given address and later we can get the data object from the address

Test Plan

unit

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

yuhan retitled this revision from input/output address operation via intermediate storage to input/output-address-operation-1 intermediate storage set/get from address.Sep 29 2020, 4:48 AM

LGTM! Thanks so much for doing this so fast.

python_modules/dagster/dagster/core/storage/intermediate_storage.py
225

I do think that it is a bit strange that address would be an optional param, but then the check doesn't reflect that optionality. However, I guess that isn't a huge issue seeing as we'll be merging this design into set_intermediate_object at some point, anyway.

This revision is now accepted and ready to land.Sep 29 2020, 2:15 PM
python_modules/dagster/dagster/core/storage/intermediate_storage.py
225

the merging happens in D4579 but im a bit hesitant on that one because it updates user-facing abstract methods and we are not sure about the change

python_modules/dagster/dagster/core/errors.py
471

Will all IOErrors mean that the address is invalid? There could be a failure to maintain a connection to some remote data source.

python_modules/dagster/dagster/core/storage/intermediate_storage.py
217

Nitpick: exprimental -> experimental. Also, it would be good to raise an experimental_fn_warning when this is used.

python_modules/dagster/dagster/core/errors.py
471

the call site of it will also pass in a detailed error message. maybe get rid of the default message and only surface the actual IO error message?

python_modules/dagster/dagster/core/storage/intermediate_storage.py
217

good call

yuhan marked 2 inline comments as done.

experimental + DagsterInvalidAddressForAsset -> DagsterAddressIOError