Page MenuHomePhabricator

(dagster-cli-api-run-launcher-3) Add CliApiRunLauncher
ClosedPublic

Authored by schrockn on Wed, May 20, 8:57 PM.

Details

Summary

Adds the CliApiRunLauncher. This is largely insprired
(copied) from the PipelineExecutionManager, which will
eventually be deleted. This does the same monitoring and
running of pipelines as that, but uses python.threading
instead of gevent, subprocess.pOpen instead of multiprocessing, etc.

Depends on D3009

Test Plan

BK

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

schrockn created this revision.Wed, May 20, 8:57 PM
schrockn requested review of this revision.Wed, May 20, 9:13 PM
schrockn updated this revision to Diff 14321.Wed, May 20, 9:16 PM
schrockn added reviewers: alangenfeld, max.

up

schrockn updated this revision to Diff 14341.Wed, May 20, 10:21 PM

adding sashank's windows fixes

alangenfeld added inline comments.Wed, May 20, 10:40 PM
python_modules/dagster/dagster/core/launcher/cli_api_run_launcher.py
33–34
alangenfeld requested changes to this revision.Wed, May 20, 10:43 PM

looks reasonable so far - think its worth updating this diff til it functions as a proper run launcher

This revision now requires changes to proceed.Wed, May 20, 10:43 PM
schrockn updated this revision to Diff 14358.Wed, May 20, 10:47 PM

rebase and alex feedback

schrockn updated this revision to Diff 14366.Wed, May 20, 11:04 PM

change temp dir utility

schrockn planned changes to this revision.Wed, May 20, 11:04 PM
schrockn removed reviewers: alangenfeld, max, sashank.

removed reviewers while debugging some BK-only bugs

schrockn updated this revision to Diff 14372.Wed, May 20, 11:29 PM

doing another run to see if it worked

schrockn updated this revision to Diff 14379.Thu, May 21, 1:00 AM

doing another run to see if it worked

schrockn updated this revision to Diff 14432.Thu, May 21, 6:30 PM
schrockn edited the summary of this revision. (Show Details)
schrockn added reviewers: alangenfeld, max, sashank.

up

alangenfeld accepted this revision.Thu, May 21, 6:57 PM

limecat

python_modules/dagster/dagster/core/launcher/cli_api_run_launcher.py
86–91

i wonder if we ever race the write with the read here

186

nice - hopefully this has no issues on windows

This revision is now accepted and ready to land.Thu, May 21, 6:57 PM
This revision was landed with ongoing or failed builds.Thu, May 21, 7:02 PM
This revision was automatically updated to reflect the committed changes.