Page MenuHomeElementl

Incorporate run config into choosing the executor in host mode run worker flow
ClosedPublic

Authored by dgibson on Apr 6 2021, 1:30 PM.

Details

Summary

Instead of just passing in the Executor to execute_run_host_mode, let callsites pass in a function that lets you specify the allowed ExecutorDefinitions (as well as their names). The run worker then uses them to create the executor.

Still missing from this is something to enforce config validation in callsites that will be using this run worker (for example, the instance could define the set of allowed executors and the set of default executors,, and at pipeline snapshot creation time and plan creation time we could throw if the pipeline isn't configured correctly)

Test Plan

BK

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

dgibson retitled this revision from Incorporate run config into chooseing the executor in host mode run worker flow to Incorporate run config into choosing the executor in host mode run worker flow.Apr 6 2021, 1:30 PM
dgibson added a subscriber: johann.
Harbormaster returned this revision to the author for changes because remote builds failed.Apr 6 2021, 1:50 PM
Harbormaster failed remote builds in B28471: Diff 34939!
alangenfeld added inline comments.
python_modules/dagster/dagster/core/execution/context_creation_pipeline.py
319–476

I think it would be nice to move all this host mode stuff to its own file

This revision is now accepted and ready to land.Apr 6 2021, 2:57 PM