Page MenuHomePhabricator

Add CliApiRunLauncher [redux]
ClosedPublic

Authored by schrockn on Thu, May 21, 9:01 PM.

Details

Summary

Putting this up again because I want this to go through
proper dev and qa post rebase.

Adds the CliApiRunLauncher. This is largely insprired
(copied) from the PipelineExecutionManager, which will
eventually be deleted. This does the same monitoring and
running of pipelines as that, but uses python.threading
instead of gevent, subprocess.pOpen instead of multiprocessing, etc.

Test Plan

BK

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

schrockn updated this revision to Diff 14464.Thu, May 21, 9:01 PM
schrockn created this revision.

up

Harbormaster completed remote builds in B11760: Diff 14464.
schrockn requested review of this revision.Thu, May 21, 9:16 PM
alangenfeld accepted this revision.Thu, May 21, 9:19 PM
alangenfeld added inline comments.
python_modules/dagster/dagster/core/launcher/cli_api_run_launcher.py
124–126

could use local artifact storage component on instance to get place to put file instead of sys temp

This revision is now accepted and ready to land.Thu, May 21, 9:19 PM
schrockn added inline comments.Thu, May 21, 9:51 PM
python_modules/dagster/dagster/core/launcher/cli_api_run_launcher.py
127

hmmm would need to think about that. not sure we want to do an s3 write in every case (when that is a thing)

This revision was automatically updated to reflect the committed changes.