Page MenuHomePhabricator

Add start and resume methods to Scheduler
AbandonedPublic

Authored by sashank on Sep 11 2019, 8:03 AM.

Details

Reviewers
None
Group Reviewers
Restricted Project
Summary

Adds stop and resume methods to Scheduler implementations.

Previously, "ending" a scheduler would simply delete the schedule. By adding a separate "stop" method, we can pause and resume schedules. In the case of the SystemCronScheduler:

  • start creates metadata files (.json and .sh) and creates a cron tab
  • stop simply removes the cron tab
  • resume puts the cron tab back
  • end deletes all metadata files and removes the crontab

Motivation: https://github.com/dagster-io/dagster/issues/1740

Test Plan

unit

$ dagster schedules list
Repository experimental_repository
**********************************
Schedule: log_spew_hourly
Cron Schedule: 0 0 * * *
********************************
Schedule: many_events_every_min
Cron Schedule: * * * * *
************************************
Schedule: pandas_hello_world_hourly
Cron Schedule: * * * * *

$ dagster schedules start log_spew_hourly
Started schedule log_spew_hourly with ID 0e378a83-1a9d-447c-bd74-9cae2b250308

$ dagster schedules list
Repository experimental_repository
**********************************
Schedule: log_spew_hourly [RUNNING]
Cron Schedule: 0 0 * * *
********************************
Schedule: many_events_every_min
Cron Schedule: * * * * *
************************************
Schedule: pandas_hello_world_hourly
Cron Schedule: * * * * *

$ dagster schedules stop log_spew_hourly
Stopped schedule log_spew_hourly with ID 0e378a83-1a9d-447c-bd74-9cae2b250308

$ dagster schedules list
Repository experimental_repository
**********************************
Schedule: log_spew_hourly [STOPPED]
Cron Schedule: 0 0 * * *
********************************
Schedule: many_events_every_min
Cron Schedule: * * * * *
************************************
Schedule: pandas_hello_world_hourly
Cron Schedule: * * * * *

$ dagster schedules resume log_spew_hourly
Resumed schedule log_spew_hourly with ID 0e378a83-1a9d-447c-bd74-9cae2b250308

$ dagster schedules list
Repository experimental_repository
**********************************
Schedule: log_spew_hourly [RUNNING]
Cron Schedule: 0 0 * * *
********************************
Schedule: many_events_every_min
Cron Schedule: * * * * *
************************************
Schedule: pandas_hello_world_hourly
Cron Schedule: * * * * *

$ dagster schedules end log_spew_hourly
Ended schedule log_spew_hourly with ID 0e378a83-1a9d-447c-bd74-9cae2b250308

$ dagster schedules list
Repository experimental_repository
**********************************
Schedule: log_spew_hourly
Cron Schedule: 0 0 * * *
********************************
Schedule: many_events_every_min
Cron Schedule: * * * * *
************************************
Schedule: pandas_hello_world_hourly
Cron Schedule: * * * * *

Diff Detail

Repository
R1 dagster
Branch
suspend-resume-schedules
Lint
Lint OK
Unit
No Unit Test Coverage

Event Timeline

sashank created this revision.Sep 11 2019, 8:03 AM
sashank edited the summary of this revision. (Show Details)Sep 11 2019, 8:12 AM
sashank updated this revision to Diff 4597.Sep 11 2019, 8:15 AM

Fix filters

sashank updated this revision to Diff 4598.Sep 11 2019, 8:24 AM

Rename TestScheduler to MockScheduler and update implementation

sashank updated this revision to Diff 4599.Sep 11 2019, 8:28 AM

Fix MockSystemCronScheduler

alangenfeld added inline comments.
python_modules/dagster/dagster/core/scheduler/scheduler.py
26–37

wowow

schrockn added inline comments.
python_modules/dagster/dagster/core/scheduler/scheduler.py
37

yeah i think we should talk about this to see if it needs to be this complicated

sashank abandoned this revision.Sep 11 2019, 10:32 PM