Refactor launch_run and submit_run to take in an IWorkspace rather than an…
b55f3aed8ccc
Actions

Description

Refactor launch_run and submit_run to take in an IWorkspace rather than an ExternalPipeline

Summary:
This allows the run queue daemon to launch runs without needing to reload the gRPC server for each repository, which is nice both for speed, but more importantly it can still operate even if the gRPC server is down (instead of throwing

For forward-compat, RunLauncher and RunCoordinator now take in a Context object rather than specific arguments.

Test Plan: BK, Integration

Reviewers: johann, alangenfeld

Reviewed By: alangenfeld

Differential Revision: https://dagster.phacility.com/D8557

Details

Provenance

• dgibson

Authored on Jun 26 2021, 2:49 AM

Reviewer

alangenfeld

Differential Revision

D8557: Refactor launch_run and submit_run to take in an IWorkspace rather than an ExternalPipeline

Parents

R1:a8e598f4852b: Add 'pipeline_code_origin' to PipelineRun

Branches

Unknown

Tags

Unknown

Event Timeline

• dgibson committed R1:b55f3aed8ccc: Refactor launch_run and submit_run to take in an IWorkspace rather than an… (authored by • dgibson).Jul 3 2021, 2:48 AM

• dgibson added an edge: D8557: Refactor launch_run and submit_run to take in an IWorkspace rather than an ExternalPipeline.

Changes (65)

Path

Size

docs/

content/

guides/

dagster/

run-attribution.mdx

examples/

docs_snippets/

guides/

dagster/

run-attribution/

custom_run_coordinator_skeleton.py

run_attribution_example/

custom_run_coordinator.py

run_attribution_example_tests/

test_custom_run_coordinator.py

integration_tests/

test_suites/

celery-k8s-integration-test-suite/

test_daemon_scheduler.py

test_integration.py

test_queued_run_coordinator.py

daemon-test-suite/

k8s-integration-test-suite/

test_executor.py

test_integration.py

test_job_spec.py

python_modules/

dagster-graphql/

dagster_graphql/

implementation/

execution/

backfill.py

launch_execution.py

dagster_graphql_tests/

graphql/

setup.py

test_all_snapshot_ids.py

test_sync_run_launcher.py

dagster-test/

dagster_test/

test_project/

__init__.py

dagster_test_tests/

test_test_project.py

dagster/

cli/

workspace/

core/

execution/

instance/

launcher/

default_run_launcher.py

sync_in_memory_run_launcher.py

run_coordinator/

__init__.py

base.py

default_run_coordinator.py

queued_run_coordinator.py

workspace/

daemon/

cli/

run_coordinator/

queued_run_coordinator_daemon.py

sensor.py

scheduler/

scheduler.py

utils/

external.py

dagster_tests/

api_tests/

utils.py

cli_tests/

command_tests/

test_launch_command.py

test_list_command.py

test_print_command.py

workspace_tests/

test_pipeline_load.py

test_repository_load.py

core_tests/

instance_tests/

test_instance.py

launcher_tests/

test_default_run_launcher.py

test_persistent_grpc_run_launcher.py

run_coordinator_tests/

test_default_run_coordinator.py

test_queued_run_coordinator.py

daemon_tests/

test_queued_run_coordinator_daemon.py

libraries/

dagster-aws/

dagster_aws/

ecs/

launcher.py

dagster_aws_tests/

ecs_tests/

launcher_tests/

conftest.py

test_launching.py

test_termination.py

dagster-celery-k8s/

dagster_celery_k8s/

launcher.py

dagster_celery_k8s_tests/

test_launcher.py

dagster-docker/

dagster_docker/

docker_run_launcher.py

dagster_docker_tests/

test_launch_docker.py

dagster-k8s/

dagster_k8s/

launcher.py

dagster_k8s_tests/

unit_tests/

test_launcher.py

R1:b55f3aed8ccc

View Options

docs/content/guides/dagster/run-attribution.mdx

View Options

examples/docs_snippets/docs_snippets/guides/dagster/run-attribution/custom_run_coordinator_skeleton.py

View Options

examples/run_attribution_example/run_attribution_example/custom_run_coordinator.py

View Options

examples/run_attribution_example/run_attribution_example_tests/test_custom_run_coordinator.py

View Options

integration_tests/test_suites/celery-k8s-integration-test-suite/test_daemon_scheduler.py

View Options

integration_tests/test_suites/celery-k8s-integration-test-suite/test_integration.py

View Options

integration_tests/test_suites/celery-k8s-integration-test-suite/test_queued_run_coordinator.py

View Options

integration_tests/test_suites/daemon-test-suite/conftest.py

View Options

integration_tests/test_suites/daemon-test-suite/test_daemon.py

View Options

integration_tests/test_suites/daemon-test-suite/test_queued.py

View Options

integration_tests/test_suites/daemon-test-suite/utils.py

View Options

integration_tests/test_suites/k8s-integration-test-suite/test_executor.py

View Options

integration_tests/test_suites/k8s-integration-test-suite/test_integration.py

View Options

integration_tests/test_suites/k8s-integration-test-suite/test_job_spec.py

View Options

python_modules/dagster-graphql/dagster_graphql/implementation/execution/backfill.py

View Options

python_modules/dagster-graphql/dagster_graphql/implementation/execution/launch_execution.py

View Options

python_modules/dagster-graphql/dagster_graphql_tests/graphql/setup.py

View Options

python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_all_snapshot_ids.py

View Options

python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_sync_run_launcher.py

View Options

python_modules/dagster-test/dagster_test/test_project/init.py

View Options

python_modules/dagster-test/dagster_test_tests/test_test_project.py

View Options

python_modules/dagster/dagster/cli/pipeline.py

View Options

python_modules/dagster/dagster/cli/schedule.py

View Options

python_modules/dagster/dagster/cli/sensor.py

View Options

python_modules/dagster/dagster/cli/workspace/cli_target.py

View Options

python_modules/dagster/dagster/core/execution/backfill.py

View Options

python_modules/dagster/dagster/core/instance/init.py

View Options

python_modules/dagster/dagster/core/launcher/init.py

View Options

python_modules/dagster/dagster/core/launcher/base.py

View Options

python_modules/dagster/dagster/core/launcher/default_run_launcher.py

View Options

python_modules/dagster/dagster/core/launcher/sync_in_memory_run_launcher.py

View Options

python_modules/dagster/dagster/core/run_coordinator/init.py

View Options

python_modules/dagster/dagster/core/run_coordinator/base.py

View Options

python_modules/dagster/dagster/core/run_coordinator/default_run_coordinator.py

View Options

python_modules/dagster/dagster/core/run_coordinator/queued_run_coordinator.py

View Options

python_modules/dagster/dagster/core/test_utils.py

View Options

python_modules/dagster/dagster/core/workspace/context.py

View Options

python_modules/dagster/dagster/daemon/backfill.py

View Options

python_modules/dagster/dagster/daemon/cli/init.py

View Options

python_modules/dagster/dagster/daemon/run_coordinator/queued_run_coordinator_daemon.py

View Options

python_modules/dagster/dagster/daemon/sensor.py

View Options

python_modules/dagster/dagster/scheduler/scheduler.py

View Options

python_modules/dagster/dagster/utils/external.py

View Options

python_modules/dagster/dagster_tests/api_tests/utils.py

View Options

python_modules/dagster/dagster_tests/cli_tests/command_tests/test_launch_command.py

View Options

python_modules/dagster/dagster_tests/cli_tests/command_tests/test_list_command.py

View Options

python_modules/dagster/dagster_tests/cli_tests/command_tests/test_print_command.py

View Options

python_modules/dagster/dagster_tests/cli_tests/workspace_tests/test_pipeline_load.py

View Options

python_modules/dagster/dagster_tests/cli_tests/workspace_tests/test_repository_load.py

View Options

python_modules/dagster/dagster_tests/core_tests/instance_tests/test_instance.py

View Options

python_modules/dagster/dagster_tests/core_tests/launcher_tests/test_default_run_launcher.py

View Options

python_modules/dagster/dagster_tests/core_tests/launcher_tests/test_persistent_grpc_run_launcher.py

View Options

python_modules/dagster/dagster_tests/core_tests/run_coordinator_tests/test_default_run_coordinator.py

View Options

python_modules/dagster/dagster_tests/core_tests/run_coordinator_tests/test_queued_run_coordinator.py

View Options

python_modules/dagster/dagster_tests/daemon_tests/test_queued_run_coordinator_daemon.py

View Options

python_modules/libraries/dagster-aws/dagster_aws/ecs/launcher.py

View Options

python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/conftest.py

View Options

python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/test_launching.py

View Options

python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/test_termination.py

View Options

python_modules/libraries/dagster-celery-k8s/dagster_celery_k8s/launcher.py

View Options

python_modules/libraries/dagster-celery-k8s/dagster_celery_k8s_tests/test_launcher.py

View Options

python_modules/libraries/dagster-docker/dagster_docker/docker_run_launcher.py

View Options

python_modules/libraries/dagster-docker/dagster_docker_tests/test_launch_docker.py

View Options

python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py

View Options

python_modules/libraries/dagster-k8s/dagster_k8s_tests/unit_tests/test_launcher.py

Refactor launch_run and submit_run to take in an IWorkspace rather than an…b55f3aed8cccActions

Description

Details

Event Timeline

Changes (65)

R1:b55f3aed8ccc

docs/content/guides/dagster/run-attribution.mdx

examples/docs_snippets/docs_snippets/guides/dagster/run-attribution/custom_run_coordinator_skeleton.py

examples/run_attribution_example/run_attribution_example/custom_run_coordinator.py

examples/run_attribution_example/run_attribution_example_tests/test_custom_run_coordinator.py

integration_tests/test_suites/celery-k8s-integration-test-suite/test_daemon_scheduler.py

integration_tests/test_suites/celery-k8s-integration-test-suite/test_integration.py

integration_tests/test_suites/celery-k8s-integration-test-suite/test_queued_run_coordinator.py

integration_tests/test_suites/daemon-test-suite/conftest.py

integration_tests/test_suites/daemon-test-suite/test_daemon.py

integration_tests/test_suites/daemon-test-suite/test_queued.py

integration_tests/test_suites/daemon-test-suite/utils.py

integration_tests/test_suites/k8s-integration-test-suite/test_executor.py

integration_tests/test_suites/k8s-integration-test-suite/test_integration.py

integration_tests/test_suites/k8s-integration-test-suite/test_job_spec.py

python_modules/dagster-graphql/dagster_graphql/implementation/execution/backfill.py

python_modules/dagster-graphql/dagster_graphql/implementation/execution/launch_execution.py

python_modules/dagster-graphql/dagster_graphql_tests/graphql/setup.py

python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_all_snapshot_ids.py

python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_sync_run_launcher.py

python_modules/dagster-test/dagster_test/test_project/__init__.py

python_modules/dagster-test/dagster_test_tests/test_test_project.py

python_modules/dagster/dagster/cli/pipeline.py

python_modules/dagster/dagster/cli/schedule.py

python_modules/dagster/dagster/cli/sensor.py

python_modules/dagster/dagster/cli/workspace/cli_target.py

python_modules/dagster/dagster/core/execution/backfill.py

python_modules/dagster/dagster/core/instance/__init__.py

python_modules/dagster/dagster/core/launcher/__init__.py

python_modules/dagster/dagster/core/launcher/base.py

python_modules/dagster/dagster/core/launcher/default_run_launcher.py

python_modules/dagster/dagster/core/launcher/sync_in_memory_run_launcher.py

python_modules/dagster/dagster/core/run_coordinator/__init__.py

python_modules/dagster/dagster/core/run_coordinator/base.py

python_modules/dagster/dagster/core/run_coordinator/default_run_coordinator.py

python_modules/dagster/dagster/core/run_coordinator/queued_run_coordinator.py

python_modules/dagster/dagster/core/test_utils.py

python_modules/dagster/dagster/core/workspace/context.py

python_modules/dagster/dagster/daemon/backfill.py

python_modules/dagster/dagster/daemon/cli/__init__.py

python_modules/dagster/dagster/daemon/run_coordinator/queued_run_coordinator_daemon.py

python_modules/dagster/dagster/daemon/sensor.py

python_modules/dagster/dagster/scheduler/scheduler.py

python_modules/dagster/dagster/utils/external.py

python_modules/dagster/dagster_tests/api_tests/utils.py

python_modules/dagster/dagster_tests/cli_tests/command_tests/test_launch_command.py

python_modules/dagster/dagster_tests/cli_tests/command_tests/test_list_command.py

python_modules/dagster/dagster_tests/cli_tests/command_tests/test_print_command.py

python_modules/dagster/dagster_tests/cli_tests/workspace_tests/test_pipeline_load.py

python_modules/dagster/dagster_tests/cli_tests/workspace_tests/test_repository_load.py

python_modules/dagster/dagster_tests/core_tests/instance_tests/test_instance.py

python_modules/dagster/dagster_tests/core_tests/launcher_tests/test_default_run_launcher.py

python_modules/dagster/dagster_tests/core_tests/launcher_tests/test_persistent_grpc_run_launcher.py

python_modules/dagster/dagster_tests/core_tests/run_coordinator_tests/test_default_run_coordinator.py

python_modules/dagster/dagster_tests/core_tests/run_coordinator_tests/test_queued_run_coordinator.py

python_modules/dagster/dagster_tests/daemon_tests/test_queued_run_coordinator_daemon.py

python_modules/libraries/dagster-aws/dagster_aws/ecs/launcher.py

python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/conftest.py

python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/test_launching.py

python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/test_termination.py

python_modules/libraries/dagster-celery-k8s/dagster_celery_k8s/launcher.py

python_modules/libraries/dagster-celery-k8s/dagster_celery_k8s_tests/test_launcher.py

python_modules/libraries/dagster-docker/dagster_docker/docker_run_launcher.py

python_modules/libraries/dagster-docker/dagster_docker_tests/test_launch_docker.py

python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py

python_modules/libraries/dagster-k8s/dagster_k8s_tests/unit_tests/test_launcher.py

Refactor launch_run and submit_run to take in an IWorkspace rather than an…
b55f3aed8ccc
Actions

python_modules/dagster-test/dagster_test/test_project/init.py

python_modules/dagster/dagster/core/instance/init.py

python_modules/dagster/dagster/core/launcher/init.py

python_modules/dagster/dagster/core/run_coordinator/init.py

python_modules/dagster/dagster/daemon/cli/init.py