Page MenuHomeElementl

rfc: versioned_filesystem_io_manager default base_dir
AbandonedPublic

Authored by yuhan on Feb 23 2021, 1:02 AM.

Details

Reviewers
None
Summary

versioned_filesystem_io_manager defaults base_dir to use the instance.storage_dir.

initially, user-facing wise, i thought this would be a better experience to the end user because it would provide a better out-of-box experience as it doesn't require extra config.
however, after getting into the code, i think this diff is a fairly invasive change: it makes resolve_memoized_execution_plan depend on instance. as we haven't figured out https://github.com/dagster-io/dagster/issues/3302 (also see code [1]), i don't feel comfortable making the situation more complicated.

my proposal is to land D6637 for now and wait until we figure out [1] to loosen the constraint

Test Plan

unit

Diff Detail

Repository
R1 dagster
Branch
yuhan/io-basedir
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 23 2021, 1:19 AM
Harbormaster failed remote builds in B26312: Diff 32151!

v1: versioned_filesystem_io_manager defaults to instance.storage_dir and resolve_memoized_execution_plan(execution_plan, instance)

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 23 2021, 6:53 AM
Harbormaster failed remote builds in B26318: Diff 32159!
Harbormaster failed remote builds in B26319: Diff 32160!
yuhan retitled this revision from versioned_filesystem_io_manager default base_dir to rfc: versioned_filesystem_io_manager default base_dir.Feb 23 2021, 7:00 AM
yuhan edited the summary of this revision. (Show Details)
yuhan requested review of this revision.Feb 23 2021, 7:02 AM
yuhan edited the summary of this revision. (Show Details)
yuhan added inline comments.
python_modules/dagster/dagster/core/execution/resolve_versions.py
172–194

[1]