Page MenuHomePhabricator

[Dask] Put Dask execution in IEngine interface
ClosedPublic

Authored by natekupp on Jul 16 2019, 10:03 PM.

Details

Reviewers
max
Group Reviewers
Restricted Project
Commits
R1:c965fb956526: [Dask] Put Dask execution in IEngine interface
Summary

This diff breaks up execute_on_dask to split out a DaskEngine object; it also uses the executor config to construct the engine, so the core can invoke the DaskEngine without needing to know about Dask via the executor_config property on RunConfig

I'll use this in a follow-up diff which will drop execute_on_dask entirely in favor of execute_pipeline w/ RunConfig.

Test Plan

unit

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

natekupp created this revision.Jul 16 2019, 10:03 PM
natekupp edited the summary of this revision. (Show Details)Jul 16 2019, 10:08 PM
natekupp edited the summary of this revision. (Show Details)
natekupp added a reviewer: Restricted Project.
max accepted this revision.Jul 16 2019, 10:13 PM
max added a subscriber: max.

๐Ÿ‘

python_modules/dagster/dagster/core/engine/engine_multiprocess.py
83

why?

This revision is now accepted and ready to land.Jul 16 2019, 10:13 PM
natekupp added inline comments.Jul 16 2019, 10:15 PM
python_modules/dagster/dagster/core/engine/engine_multiprocess.py
83

We use MultiprocessExecutorConfig all over the place, wanted to make this consistent with that. I don't feel strongly about which to standardize, but Multiprocess is shorter so ยฏ\_(ใƒ„)_/ยฏ