Page MenuHomeElementl

[crag] unbound sensors
AbandonedPublic

Authored by sandyryza on May 6 2021, 4:09 PM.

Details

Reviewers
None
Summary

The idea here is that, if someone has a pipeline with different sets of resources for execution in different environments, and they want a corresponding sensor for each of those environments, they don't need to include multiple identical implementations of their sensor function, and they don't need to define a sensor factory.

We could potentially extend this to enable having the sensor itself require resources.

Test Plan

bk

Diff Detail

Repository
R1 dagster
Branch
crag-examples (branched from master)
Lint
Lint Skipped
Unit
No Test Coverage

Event Timeline

sandyryza retitled this revision from abstract sensors to [crag] abstract sensors.May 6 2021, 4:13 PM
sandyryza retitled this revision from [crag] abstract sensors to [crag] unbound sensors.
alangenfeld added inline comments.
python_modules/dagster-test/dagster_test/toys/crag/abstract_sensor.py
35–47

i dont have a great reaction to binding resources on the sensor creating affecting the pipeline - something like this makes more sense to me

@sensor
def my_pipeline_sensor():
    yield RunRequest(None, run_config={"solids": {"my_solid": {"config": {7}}}})


my_pipeline_sensor_staging = my_pipeline_sensor.target(
   my_pipeline.bind({"my_resource": ResourceDefinition.hardcoded_resource(5)}, suffix="staging")
)


my_pipeline_sensor_prod = my_pipeline_sensor.target(
   my_pipeline.bind({"my_resource": ResourceDefinition.hardcoded_resource(5)}, suffix="prod")
)

or even just having to define the sensor twice. Can always factor the body of the sensor out if you need shared impl

I think it makes sense to enforce whatever rules well have about what pipelines can be in a repo to what pipelines a schedule/sensor can target

@alangenfeld I updated the diff with your suggestion.

I suppose when question for deciding between the two is: do we imagine that the combination of "abstract pipeline + sensor" would be a useful artifact on its own? I.e. might someone want to test that the sensor appropriately fills out the solid config, even before they've bound resources?