add a toy sensor that fires a pipeline execution every time a file in given directory
I think it would be good to structure this a function that returns a sensor and that takes a folder argument to make it more generic, since this is an example
error message not accurate"??
also for our code examples would strongly prefer limiting exception-as-control-flow as much as possible. Can we just check up front if the directory exists? Or contain the try catch to a more tightly scoped function?
This error message looks inaccurate
hmm. does this object need to be specialized to sensors?
it is the execution key that make this idempotent correct?
yeah. One thing i've been debating is if we should make execution_key required (but maybe none-able for some of the non-partition-y cases @sandyryza presented). Seems like you should opt-out of idempotency/run-exactly-once rather than opt-in
This is kind of why I have these two sensor implementations, showing the two approaches:
The first relies on the user doing the bookkeeping, by using the last evaluation time, to find the delta.
I don't think the second is necessarily totally incompatible with the first? we could still have a last_successful_execution_time or cursor that you pass in, so that you don't have to return the full set of SensorRunParams every time the sensor runs - it would just be important that that time be the checkpoint time after which we know every run fired
this seems good to get in and iterate on, we should definitely bring back the execution key for our example though?
this comment seems not right?
where did the execution_key go :(
To keep harping on this, I think execution_key should be required (but Noneable)