Page MenuHomePhabricator

[run launcher] RemoteDagitRunLauncher
ClosedPublic

Authored by alangenfeld on Oct 22 2019, 4:50 PM.

Details

Summary

Launches runs to another dagit server

Test Plan

With default instance:
run dagit -p 3333

In another terminal with DAGSTER_HOME set to instance config

run_launcher:
  module: dagster.core.launcher.remote
  class: RemoteDagitRunLauncher
  config:
    address: http://localhost:3333

Run

dagster-graphql -p launchPipelineExecution -v '{"executionParams": {"selector": {"name": "sleepy_pipeline"}, "mode": "default"}}'

Observe run happens successfully on terminal 1

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

alangenfeld created this revision.Oct 22 2019, 4:50 PM
schrockn requested changes to this revision.Oct 22 2019, 10:46 PM
schrockn added a subscriber: schrockn.

So this to me seeings like it should be a flag to a command line as opposed to an instance config. If the run queue is remote then none of the other config elements really apply. Only the remote instance.

This revision now requires changes to proceed.Oct 22 2019, 10:46 PM

So this to me seeings like it should be a flag to a command line as opposed to an instance config. If the run queue is remote then none of the other config elements really apply. Only the remote instance.

I think I see what you are getting at, and agree we should think this all through carefully. I'm not sure this shouldn't be able to work like this though. One motivating use case is being able to have N RemoteDagitRunQueues underneath a DelegatingRunQueue.

One other thought is that for the use case you are describing it's not clear why you wouldn't just open dagit on that server directly or do dagster-graphql --remote.

I don't think its unreasonable to have this desired set up:

  • I am running dagit locally
  • I click start to run things on my machine
  • I click enqueue to run things remotely
  • my local dagit run history is just my local runs
  • I have to open the remote run history page to see the "prod" runs

There is a question of assumed code sync in that set up - but I think its worth considering.

alangenfeld requested review of this revision.Oct 24 2019, 4:53 PM

back to review queue since I'm not convinced we don't want this at this point

i have some time today. can chat once i am in

alangenfeld planned changes to this revision.Oct 28 2019, 4:25 PM
alangenfeld retitled this revision from [RunQueue] RemoteDagitRunQueue to [execution server] RemoteDagitExecutionServer.Fri, Nov 15, 9:55 PM
alangenfeld edited the summary of this revision. (Show Details)
alangenfeld edited the test plan for this revision. (Show Details)
alangenfeld removed reviewers: Restricted Project, schrockn.
alangenfeld planned changes to this revision.Tue, Nov 19, 8:53 PM
alangenfeld updated this revision to Diff 6768.Thu, Nov 21, 5:13 PM

run launcher version

alangenfeld retitled this revision from [execution server] RemoteDagitExecutionServer to [run launcher] RemoteDagitRunLauncher.Thu, Nov 21, 5:15 PM
alangenfeld edited the summary of this revision. (Show Details)
alangenfeld edited the test plan for this revision. (Show Details)
This revision is now accepted and ready to land.Fri, Nov 22, 12:08 AM

move to dagster-graphql package

might make sense for it to be in the dagit package or be named differently - something we can change later

This revision was automatically updated to reflect the committed changes.