Details

Reviewers

catherinewu
schrockn

Commits

R1:cf0eddfb37f9: [core] StepHandle

Summary

Born from the ashes of D5360, extracted from the prototype D4259 comes StepHandle

This new object for interacting with steps in an ExecutionPlan sets us up to be able to handle more complex "key" structures which is needed for dynamic orchestration.

In addition to introducing this handle for keying in to step_dict, this diff attempts to clean up ExecutionPlan and ExecutionStep a bit.

Test Plan

buildkite
integration

Diff Detail

Repository

R1 dagster

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

alangenfeld created this revision.Dec 8 2020, 11:32 PM

alangenfeld edited the test plan for this revision. (Show Details)Dec 8 2020, 11:36 PM

test fixees

Harbormaster failed remote builds in B22443: Diff 27277!Dec 8 2020, 11:48 PM

fix commits

Harbormaster failed remote builds in B22444: Diff 27278!Dec 9 2020, 12:04 AM

Harbormaster completed remote builds in B22445: Diff 27279.Dec 9 2020, 12:13 AM

alangenfeld requested review of this revision.Dec 9 2020, 12:13 AM

alangenfeld added reviewers: catherinewu, schrockn.Dec 9 2020, 3:51 PM

remove extraneous code

lint

Harbormaster failed remote builds in B22474: Diff 27318!Dec 9 2020, 4:24 PM

Harbormaster completed remote builds in B22475: Diff 27319.Dec 9 2020, 4:42 PM

small cleanup

Harbormaster completed remote builds in B22483: Diff 27329.Dec 9 2020, 5:31 PM

rm mapping_key

Harbormaster failed remote builds in B22501: Diff 27359!Dec 9 2020, 7:53 PM

rebase, snapshot

Harbormaster completed remote builds in B22513: Diff 27371.Dec 9 2020, 9:00 PM

Macro looksgood:

python_modules/dagster/dagster/core/execution/plan/handle.py
9–14	This is docblock worthy given that it is a bit odd that it only container a solid_handle atm
16	remember to rebase :-)
21	can we do anything to future proof this for non-solid-compute steps in the future?

This revision is now accepted and ready to land.Dec 10 2020, 4:43 PM

rebase

Harbormaster failed remote builds in B22558: Diff 27423!Dec 10 2020, 5:42 PM

rebase

Harbormaster failed remote builds in B22565: Diff 27431!Dec 10 2020, 6:41 PM

post rebase fixup

alangenfeld added inline comments.Dec 10 2020, 6:48 PM

python_modules/dagster/dagster/core/execution/plan/handle.py
9–14	might just catch this in the upcoming diff that fleshes this out
21	as long as we are smart about how we encode them i think its fine - ie having a special delimiter or use a reserved keyword

Harbormaster completed remote builds in B22570: Diff 27436.Dec 10 2020, 7:05 PM

Closed by commit R1:cf0eddfb37f9: [core] StepHandle (authored by alangenfeld). · Explain WhyDec 10 2020, 7:09 PM

This revision was automatically updated to reflect the committed changes.

alangenfeld added a commit: R1:cf0eddfb37f9: [core] StepHandle.

alangenfeld mentioned this in D5540: fix get_steps_to_execute_in_topo_order.Dec 10 2020, 11:18 PM

alangenfeld mentioned this in R1:41f4e9f1cc17: fix get_steps_to_execute_in_topo_order.Dec 11 2020, 9:31 PM

[core] StepHandle
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 27440

python_modules/dagster/dagster/core/execution/memoization.py

python_modules/dagster/dagster/core/execution/plan/active.py

python_modules/dagster/dagster/core/execution/plan/compute.py

python_modules/dagster/dagster/core/execution/plan/execute_step.py

python_modules/dagster/dagster/core/execution/plan/handle.py

python_modules/dagster/dagster/core/execution/plan/objects.py

python_modules/dagster/dagster/core/execution/plan/plan.py

python_modules/dagster/dagster/core/execution/resolve_versions.py

python_modules/dagster/dagster/core/execution/resources_init.py

python_modules/dagster/dagster_tests/core_tests/execution_plan_tests/test_execution_plan.py

python_modules/dagster/dagster_tests/core_tests/execution_plan_tests/test_external_step.py

python_modules/dagster/dagster_tests/core_tests/test_external_execution_plan.py

python_modules/dagster/dagster_tests/core_tests/test_nothing_dependencies.py

python_modules/libraries/dagster-airflow/dagster_airflow/compile.py

python_modules/libraries/dagster-dask/dagster_dask/executor.py

[core] StepHandleClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 27440

python_modules/dagster/dagster/core/execution/memoization.py

python_modules/dagster/dagster/core/execution/plan/active.py

python_modules/dagster/dagster/core/execution/plan/compute.py

python_modules/dagster/dagster/core/execution/plan/execute_step.py

python_modules/dagster/dagster/core/execution/plan/handle.py

python_modules/dagster/dagster/core/execution/plan/objects.py

python_modules/dagster/dagster/core/execution/plan/plan.py

python_modules/dagster/dagster/core/execution/resolve_versions.py

python_modules/dagster/dagster/core/execution/resources_init.py

python_modules/dagster/dagster_tests/core_tests/execution_plan_tests/test_execution_plan.py

python_modules/dagster/dagster_tests/core_tests/execution_plan_tests/test_external_step.py

python_modules/dagster/dagster_tests/core_tests/test_external_execution_plan.py

python_modules/dagster/dagster_tests/core_tests/test_nothing_dependencies.py

python_modules/libraries/dagster-airflow/dagster_airflow/compile.py

python_modules/libraries/dagster-dask/dagster_dask/executor.py

[core] StepHandle
ClosedPublic
Actions

Revision Contents
Changeset List