Page MenuHomeElementl

[crag] unbound sensors

Authored by sandyryza on May 6 2021, 4:09 PM.



The idea here is that, if someone has a pipeline with different sets of resources for execution in different environments, and they want a corresponding sensor for each of those environments, they don't need to include multiple identical implementations of their sensor function, and they don't need to define a sensor factory.

We could potentially extend this to enable having the sensor itself require resources.

Test Plan


Diff Detail

R1 dagster
crag-examples (branched from master)
Lint Skipped
No Test Coverage

Event Timeline

sandyryza retitled this revision from abstract sensors to [crag] abstract sensors.May 6 2021, 4:13 PM
sandyryza retitled this revision from [crag] abstract sensors to [crag] unbound sensors.
alangenfeld added inline comments.

i dont have a great reaction to binding resources on the sensor creating affecting the pipeline - something like this makes more sense to me

def my_pipeline_sensor():
    yield RunRequest(None, run_config={"solids": {"my_solid": {"config": {7}}}})

my_pipeline_sensor_staging =
   my_pipeline.bind({"my_resource": ResourceDefinition.hardcoded_resource(5)}, suffix="staging")

my_pipeline_sensor_prod =
   my_pipeline.bind({"my_resource": ResourceDefinition.hardcoded_resource(5)}, suffix="prod")

or even just having to define the sensor twice. Can always factor the body of the sensor out if you need shared impl

I think it makes sense to enforce whatever rules well have about what pipelines can be in a repo to what pipelines a schedule/sensor can target

@alangenfeld I updated the diff with your suggestion.

I suppose when question for deciding between the two is: do we imagine that the combination of "abstract pipeline + sensor" would be a useful artifact on its own? I.e. might someone want to test that the sensor appropriately fills out the solid config, even before they've bound resources?