Page MenuHomeElementl

[examples] Adding a Run Attribution Example via a Custom Run Coordinator
ClosedPublic

Authored by sidkmenon on May 22 2021, 6:07 AM.
Tags
None
Referenced Files
Unknown Object (File)
Tue, Mar 21, 11:51 PM
Unknown Object (File)
Tue, Mar 21, 1:17 PM
Unknown Object (File)
Fri, Mar 17, 12:38 PM
Unknown Object (File)
Fri, Mar 17, 9:32 AM
Unknown Object (File)
Thu, Mar 16, 8:17 AM
Unknown Object (File)
Wed, Mar 15, 11:28 AM
Unknown Object (File)
Wed, Mar 15, 11:26 AM
Unknown Object (File)
Wed, Mar 15, 7:37 AM
Subscribers
None

Details

Summary

As part of an ask by a user, this is an example of how one would perform Run Attribution by reading off of HTTP headers in the Run Coordinator.
Depends on D8030, D8187

Test Plan

Header parsing logic was tested both manually (by replacing the jwt_token in test with the one provided by the user) and by creating a similar X-Amzn-Oidc-Data header to the one used by the user.

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

sidkmenon created this revision.
sidkmenon held this revision as a draft.

lint

Adding logic more in line with user ask

sidkmenon edited the test plan for this revision. (Show Details)

Will this example be explicitly mentioned in the docs somewhere?

examples/run_attribution/run_attribution/custom_run_coordinator.py
14–24 ↗(On Diff #38721)

nit: prefer early returns to avoid nesting hell

examples/run_attribution/run_attribution_tests/test_custom_run_coordinator.py
34–38 ↗(On Diff #38721)

to avoid nesting

examples/run_attribution/setup.py
4 ↗(On Diff #38721)
sidkmenon edited the test plan for this revision. (Show Details)

restructuring to address @rexledesma's comments as well as rebasing

rename run_attribution -> run_attribution_example

This example needs to show up in the docs somewhere, probably in https://docs.dagster.io/deployment/run-coordinator. Similar to https://docs.dagster.io/deployment/guides/docker#example, we should highlight the fact that a custom run coordinator can be used, and that an example is a run coordinator that injects run tags.

This revision now requires changes to proceed.Jun 2 2021, 1:39 AM

rebasing & prefixing test class with Test** to adhere to pytest conventions

This example needs to show up in the docs somewhere, probably in https://docs.dagster.io/deployment/run-coordinator. Similar to https://docs.dagster.io/deployment/guides/docker#example, we should highlight the fact that a custom run coordinator can be used, and that an example is a run coordinator that injects run tags

Added a separate docs diff at D8206!

looks fine, maybe remove the patch on warnings?

examples/run_attribution_example/run_attribution_example_tests/test_custom_run_coordinator.py
25–27

patching the warnings out feels weird.. this shouldn't affect the test behavior right?

This revision is now accepted and ready to land.Jun 3 2021, 6:24 PM