Page MenuHomeElementl

Prototype: Add test utility for dagstermill solids
Needs ReviewPublic

Authored by yuhan on Oct 7 2020, 2:03 PM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Mar 17, 8:21 PM
Unknown Object (File)
Tue, Mar 14, 5:36 AM
Unknown Object (File)
Tue, Mar 14, 5:36 AM
Unknown Object (File)
Tue, Mar 14, 5:36 AM
Unknown Object (File)
Fri, Mar 10, 12:59 PM
Unknown Object (File)
Feb 10 2023, 4:05 AM
Unknown Object (File)
Feb 9 2023, 12:19 PM
Unknown Object (File)
Feb 8 2023, 3:25 AM
Subscribers
This revision needs review, but all specified reviewers are disabled or inactive.

Details

Summary

Goal: users expect to test dagstermill solids in an easy way, i.e. without writing boilerplates that touch a lot of internal implementations.
Ideal API: execute_dagstermill_solid(solid_def, mode_def)

This this diff is not there yet. The current API:

def solid_callable():
    return hello_world

def mode_callable():
    return ModeDefinition(resource_defs={"file_manager": local_file_manager})

def test_execute():
    res = execute_dagstermill_solid(solid_callable, mode_callable=mode_callable)
  • most args are the same as execute_solid, except it takes solid_callable not solid_def and mode_callable not mode_def, which is bad for approachability.
  • the difference between execute_dagstermill_solid and execute_solid is executing dagstermill solid requires reconstructable pipeline. In order to create a ReconstructablePipeline from the given user python code (solid def and mode def), there are many boilerplates needed, and this diff is a prototype that shows those boilerplates. - Re: Submit-API
Test Plan

Unit

Diff Detail

Repository
R1 dagster
Branch
arcpatch-D4698
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 7 2020, 2:23 PM
Harbormaster failed remote builds in B19250: Diff 23389!
yuhan retitled this revision from Add test utility for dagstermill solids to RFC: Add test utility for dagstermill solids.Jul 30 2021, 12:12 AM
yuhan edited the summary of this revision. (Show Details)
yuhan retitled this revision from RFC: Add test utility for dagstermill solids to Prototype: Add test utility for dagstermill solids.Jul 30 2021, 12:14 AM

My initial reaction is that this is a pretty small improvement for a very specialized utility function. Its hard for me to make a strong case that this is any better than reconstructable in terms of having to discover some special tool when the normal testing flow fails on a dagstermill solid.

directions i think worth exploring:

  • a solution that works for direct invocation since thats our new testing story
  • explore more clever of use solid_fn.__module__ and sys.modules