Page MenuHomeElementl

[crag] Ignore executor config when using execute_in_process
ClosedPublic

Authored by cdecarolis on Aug 2 2021, 10:08 PM.

Details

Summary

Currently, we foist the original config from the job into execute_in_process. This means that we might end up with executor config that doesn't make any sense. This diff makes execute_in_process ignore that extra config.

Test Plan

Added test to ensure executor config is ingored

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

python_modules/dagster/dagster/core/definitions/run_config.py
98

This is sus, but I think it's just that pylint can't infer execute_in_process_executor is an ExecutorDefinition.

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 2 2021, 10:51 PM
Harbormaster failed remote builds in B34564: Diff 42743!
alangenfeld added inline comments.
python_modules/dagster/dagster/core/definitions/executor.py
235

maybe expand that it triggers special behavior in the config system

python_modules/dagster/dagster/core/system_config/objects.py
191โ€“203

why do we need this? Is it solvable by changing the config schema on the in process executor?

if not add a clear comment here

This revision is now accepted and ready to land.Aug 3 2021, 2:52 PM
python_modules/dagster/dagster/core/system_config/objects.py
191โ€“203

We might be able to get around it, but I think a config schema change would be more confusing. Essentially I think we would have to change the name of the executor to be whatever the name of the expected executor is, which would be a rough experience, and harder to explain than this IMO.