Page MenuHomeElementl
Differential D5022

[sensors-4] add sensors to scheduler event loop
ClosedPublic
Actions

Authored by prha on Nov 3 2020, 10:04 PM.

Details

Reviewers
schrockn
dgibson
sandyryza
Commits
R1:1c105eaad5b6: [sensors-4] add sensors to scheduler event loop
Summary

1st pass at evaluating sensors in the schedule event loop.

There's a fair amount of code duplication, but can start combining if this ends up being the right approach.

This doesn't allow for customization of the evaluation interval or provide for threading to mitigate for latency from one sensor evaluation to affect other sensors / schedules.

Test Plan

bk

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

prha created this revision.Nov 3 2020, 10:04 PM
Harbormaster completed remote builds in B20757: Diff 25160.Nov 3 2020, 10:21 PM
prha requested review of this revision.Nov 3 2020, 10:21 PM
prha edited the summary of this revision. (Show Details)Nov 3 2020, 10:51 PM
prha added reviewers: schrockn, dgibson, sandyryza.
dgibson requested changes to this revision.Nov 4 2020, 3:06 AM

this seems reasonable, time for tests?

python_modules/dagster/dagster/scheduler/sensor.py
56

update copy

83–85

this will be using external repository origin to work if you make the changes from the previous diff so I won't belabor that

99

setting the timestamp here doesn't mess up future checks if it fails does it? like if it suddenly crashed after doing this write and started up again, would it be idempotent?

If it would that's probably undesirable and maybe we should filter be completed ticks when determining the checkpoint time?

This revision now requires changes to proceed.Nov 4 2020, 3:06 AM
prha updated this revision to Diff 25273.Nov 6 2020, 7:07 AM

update, add tests

Harbormaster completed remote builds in B20847: Diff 25273.Nov 6 2020, 7:30 AM
prha updated this revision to Diff 25281.Nov 6 2020, 4:25 PM
prha marked 2 inline comments as done.

lint

Harbormaster completed remote builds in B20854: Diff 25281.Nov 6 2020, 4:47 PM
dgibson added a comment.Nov 17 2020, 3:08 PM

will keep working through the stack - seems like this sensor format is not the final version since it just returns a boolean? and isn't currently offering any idempotency/run-exactly-once guarantees right?

python_modules/dagster/dagster/scheduler/scheduler.py
109–116 ↗(On Diff #25281)

this is going to need a rebase as this method no longer exists- should probably be a new SensorDaemon in the dagster daemon loop in dagster/daemon/daemon.py? Should be functionally the same though

python_modules/dagster/dagster/scheduler/sensor.py
44–51

Probably worth chatting about this part IRL, having some trouble totally following it - seems hard to guarantee idempotence/exactly-one-run-per-sensor-execution for? what if it fails in the middle of doing this?

Do we want to maintain the idempotence guarantees for sensors?

106–113

i had assumed the cursor would be based on the execution keys of the created runs, not just a string version of the current time? For idempotence purposes?

146–147

are we eventually going to include the cursor here? seems like it would need to be part of this in order for it to be useful?

python_modules/dagster/dagster_tests/scheduler_tests/test_sensor_run.py
36–46

this formatting is going to be updated right?

158

rename

dgibson added a comment.Nov 17 2020, 3:09 PM

ah right, the API changes happen in https://dagster.phacility.com/D5026

dgibson accepted this revision.Nov 17 2020, 3:10 PM

The scaffolding here seems fine, we will need to sort out the exact details of the recovery-from-failure rules but it seems like that is happening later in the stack where we will actually use the execution_key and cursor params?

Will have a bit of a painful rebase though

This revision is now accepted and ready to land.Nov 17 2020, 3:10 PM
prha updated this revision to Diff 26205.Nov 23 2020, 3:29 PM

rebase

prha updated this revision to Diff 26206.Nov 23 2020, 3:32 PM

rebase

prha updated this revision to Diff 26210.Nov 23 2020, 3:46 PM

undo bad rebase

Harbormaster completed remote builds in B21599: Diff 26205.Nov 23 2020, 3:47 PM
This revision was landed with ongoing or failed builds.Nov 23 2020, 3:49 PM
Closed by commit R1:1c105eaad5b6: [sensors-4] add sensors to scheduler event loop (authored by prha). · Explain Why
This revision was automatically updated to reflect the committed changes.
prha added a commit: R1:1c105eaad5b6: [sensors-4] add sensors to scheduler event loop.
Harbormaster completed remote builds in B21600: Diff 26206.Nov 23 2020, 3:50 PM
Harbormaster completed remote builds in B21602: Diff 26210.Nov 23 2020, 4:03 PM

Revision Contents
Changeset List

PathSize
python_modules/
dagster/
dagster/
daemon/
4 lines
6 lines
scheduler/
242 lines
dagster_tests/
daemon_tests/
16 lines
scheduler_tests/
260 lines

Diff 26212

View Options

python_modules/dagster/dagster/daemon/controller.py

Loading...
View Options

python_modules/dagster/dagster/daemon/daemon.py

Loading...
View Options

python_modules/dagster/dagster/scheduler/sensor.py

Loading...
View Options

python_modules/dagster/dagster_tests/daemon_tests/test_dagster_daemon.py

Loading...
View Options

python_modules/dagster/dagster_tests/scheduler_tests/test_sensor_run.py

Loading...