Details

Reviewers

• johann
alangenfeld

Commits

R1:b55f3aed8ccc: Refactor launch_run and submit_run to take in an IWorkspace rather than an…

Summary

This allows the run queue daemon to launch runs without needing to reload the gRPC server for each repository, which is nice both for speed, but more importantly it can still operate even if the gRPC server is down (instead of throwing

For forward-compat, RunLauncher and RunCoordinator now take in a Context object rather than specific arguments.

Test Plan

BK, Integration

Diff Detail

Repository

R1 dagster

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

• dgibson created this revision.Jun 28 2021, 4:44 AM

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 28 2021, 5:01 AM

Harbormaster failed remote builds in B32715: Diff 40297!

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 28 2021, 3:41 PM

Harbormaster failed remote builds in B32732: Diff 40317!

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 28 2021, 5:15 PM

Harbormaster failed remote builds in B32739: Diff 40333!

Harbormaster failed remote builds in B32740: Diff 40334!

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 28 2021, 7:13 PM

Harbormaster failed remote builds in B32753: Diff 40349!

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 28 2021, 8:55 PM

Harbormaster failed remote builds in B32760: Diff 40357!

upup

Harbormaster failed remote builds in B32763: Diff 40360!Jun 28 2021, 9:31 PM

Harbormaster failed remote builds in B32766: Diff 40362!Jun 28 2021, 9:38 PM

Harbormaster failed remote builds in B32762: Diff 40359!Jun 28 2021, 9:42 PM

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 28 2021, 9:59 PM

Harbormaster failed remote builds in B32768: Diff 40365!

• dgibson published this revision for review.Jun 28 2021, 10:13 PM

Harbormaster failed remote builds in B32774: Diff 40372!Jun 28 2021, 10:36 PM

I think this is fine but lots of little changes through-out. Anything you think deserves a closer inspection?

not really? everything is pretty rote once the APIs. Maybe look at one representative run launcher and run coordinator, the top level RunLauncher and RunCoordinator APIs, and the piece inside DagsterInstance?

Harbormaster failed remote builds in B32977: Diff 40646!Jul 1 2021, 3:48 PM

Harbormaster failed remote builds in B32998: Diff 40673!Jul 1 2021, 6:02 PM

Harbormaster completed remote builds in B33010: Diff 40686.Jul 1 2021, 7:23 PM

muk

python_modules/dagster/dagster/cli/pipeline.py
82–173	its touch unfortunate that you need an instance for these cli commands that don't interact with it very niche use case tho so I don't think worth standing against
python_modules/dagster/dagster/cli/workspace/cli_target.py
171–176	[1] theoretically we could create a different flavor `IWorkspace` here that doesn't need the instance
592–643	might be worth commenting that `instance` and `version` are only required here to use a shared impl for `IWorkspace` [1] and not fundamentally required to produce the `External` artifact
python_modules/dagster/dagster/core/instance/__init__.py
1366–1367	update dis

This revision is now accepted and ready to land.Jul 2 2021, 9:29 PM

alangenfeld mentioned this in D8524: [crag] execute_job API.Jul 2 2021, 9:38 PM

add comment to the get_xx_from_kwargs method where the instance isn't strictly required (share your feelings on that but ultimately decided it wasn't worth the trouble)

Harbormaster failed remote builds in B33165: Diff 40882!Jul 2 2021, 10:19 PM

This revision was landed with ongoing or failed builds.Jul 3 2021, 2:49 AM

Closed by commit R1:b55f3aed8ccc: Refactor launch_run and submit_run to take in an IWorkspace rather than an… (authored by • dgibson). · Explain Why

This revision was automatically updated to reflect the committed changes.

• dgibson added a commit: R1:b55f3aed8ccc: Refactor launch_run and submit_run to take in an IWorkspace rather than an….

Harbormaster failed remote builds in B33196: Diff 40918!Jul 3 2021, 2:54 AM

Harbormaster completed remote builds in B33197: Diff 40919.Jul 3 2021, 2:58 AM

Refactor launch_run and submit_run to take in an IWorkspace rather than an ExternalPipelineClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 40921

docs/content/guides/dagster/run-attribution.mdx

examples/docs_snippets/docs_snippets/guides/dagster/run-attribution/custom_run_coordinator_skeleton.py

examples/run_attribution_example/run_attribution_example/custom_run_coordinator.py

examples/run_attribution_example/run_attribution_example_tests/test_custom_run_coordinator.py

integration_tests/test_suites/celery-k8s-integration-test-suite/test_daemon_scheduler.py

integration_tests/test_suites/celery-k8s-integration-test-suite/test_integration.py

integration_tests/test_suites/celery-k8s-integration-test-suite/test_queued_run_coordinator.py

integration_tests/test_suites/daemon-test-suite/conftest.py

integration_tests/test_suites/daemon-test-suite/test_daemon.py

integration_tests/test_suites/daemon-test-suite/test_queued.py

integration_tests/test_suites/daemon-test-suite/utils.py

integration_tests/test_suites/k8s-integration-test-suite/test_executor.py

integration_tests/test_suites/k8s-integration-test-suite/test_integration.py

integration_tests/test_suites/k8s-integration-test-suite/test_job_spec.py

python_modules/dagster-graphql/dagster_graphql/implementation/execution/backfill.py

python_modules/dagster-graphql/dagster_graphql/implementation/execution/launch_execution.py

python_modules/dagster-graphql/dagster_graphql_tests/graphql/setup.py

python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_all_snapshot_ids.py

python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_sync_run_launcher.py

python_modules/dagster-test/dagster_test/test_project/__init__.py

python_modules/dagster-test/dagster_test_tests/test_test_project.py

python_modules/dagster/dagster/cli/pipeline.py

python_modules/dagster/dagster/cli/schedule.py

python_modules/dagster/dagster/cli/sensor.py

python_modules/dagster/dagster/cli/workspace/cli_target.py

python_modules/dagster/dagster/core/execution/backfill.py

python_modules/dagster/dagster/core/instance/__init__.py

python_modules/dagster/dagster/core/launcher/__init__.py

python_modules/dagster/dagster/core/launcher/base.py

python_modules/dagster/dagster/core/launcher/default_run_launcher.py

python_modules/dagster/dagster/core/launcher/sync_in_memory_run_launcher.py

python_modules/dagster/dagster/core/run_coordinator/__init__.py

python_modules/dagster/dagster/core/run_coordinator/base.py

python_modules/dagster/dagster/core/run_coordinator/default_run_coordinator.py

python_modules/dagster/dagster/core/run_coordinator/queued_run_coordinator.py

python_modules/dagster/dagster/core/test_utils.py

python_modules/dagster/dagster/core/workspace/context.py

python_modules/dagster/dagster/daemon/backfill.py

python_modules/dagster/dagster/daemon/cli/__init__.py

python_modules/dagster/dagster/daemon/run_coordinator/queued_run_coordinator_daemon.py

python_modules/dagster/dagster/daemon/sensor.py

python_modules/dagster/dagster/scheduler/scheduler.py

python_modules/dagster/dagster/utils/external.py

python_modules/dagster/dagster_tests/api_tests/utils.py

python_modules/dagster/dagster_tests/cli_tests/command_tests/test_launch_command.py

python_modules/dagster/dagster_tests/cli_tests/command_tests/test_list_command.py

python_modules/dagster/dagster_tests/cli_tests/command_tests/test_print_command.py

python_modules/dagster/dagster_tests/cli_tests/workspace_tests/test_pipeline_load.py

python_modules/dagster/dagster_tests/cli_tests/workspace_tests/test_repository_load.py

python_modules/dagster/dagster_tests/core_tests/instance_tests/test_instance.py

python_modules/dagster/dagster_tests/core_tests/launcher_tests/test_default_run_launcher.py

python_modules/dagster/dagster_tests/core_tests/launcher_tests/test_persistent_grpc_run_launcher.py

python_modules/dagster/dagster_tests/core_tests/run_coordinator_tests/test_default_run_coordinator.py

python_modules/dagster/dagster_tests/core_tests/run_coordinator_tests/test_queued_run_coordinator.py

python_modules/dagster/dagster_tests/daemon_tests/test_queued_run_coordinator_daemon.py

python_modules/libraries/dagster-aws/dagster_aws/ecs/launcher.py

python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/conftest.py

python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/test_launching.py

python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/test_termination.py

python_modules/libraries/dagster-celery-k8s/dagster_celery_k8s/launcher.py

python_modules/libraries/dagster-celery-k8s/dagster_celery_k8s_tests/test_launcher.py

python_modules/libraries/dagster-docker/dagster_docker/docker_run_launcher.py

python_modules/libraries/dagster-docker/dagster_docker_tests/test_launch_docker.py

python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py

python_modules/libraries/dagster-k8s/dagster_k8s_tests/unit_tests/test_launcher.py

Refactor launch_run and submit_run to take in an IWorkspace rather than an ExternalPipeline
ClosedPublic
Actions

Revision Contents
Changeset List

python_modules/dagster-test/dagster_test/test_project/init.py

python_modules/dagster/dagster/core/instance/init.py

python_modules/dagster/dagster/core/launcher/init.py

python_modules/dagster/dagster/core/run_coordinator/init.py

python_modules/dagster/dagster/daemon/cli/init.py