Page MenuHomePhabricator
Differential D798

Add Scheduler to dagit and DagsterGraphQLContext
ClosedPublic
Actions

Authored by sashank on Aug 12 2019, 6:45 PM.

Details

Reviewers
alangenfeld
Group Reviewers
Restricted Project
Commits
R1:a8adcc7d7e40: Add Scheduler to dagit and DagsterGraphQLContext
Summary

Add Scheduler to the DagsterGraphQLContext and the option to specify the storage directory for schedules in the dagit cli. By default, the storage directory is $DAGSTER_HOME/schedules

Test Plan

run dagit with SCHEDULER feature enabled, unit

dagster.cgf:

[FEATURES]
scheduler=

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

sashank created this revision.Aug 12 2019, 6:45 PM
sashank added a parent revision: D791: Create Scheduler API.Aug 12 2019, 6:46 PM
Harbormaster failed remote builds in B2881: Diff 3602!Aug 12 2019, 6:52 PM
sashank updated this revision to Diff 3604.Aug 12 2019, 7:00 PM

Fix typo

Harbormaster failed remote builds in B2882: Diff 3604!Aug 12 2019, 7:07 PM
sashank updated this revision to Diff 3605.Aug 12 2019, 8:31 PM

Rebase

Harbormaster completed remote builds in B2883: Diff 3605.Aug 12 2019, 8:45 PM
alangenfeld requested changes to this revision.Aug 12 2019, 9:23 PM
alangenfeld added a subscriber: alangenfeld.
alangenfeld added inline comments.
python_modules/dagster/dagster/utils/__init__.py
272–310

lets set up something a little better here - we should at bare minimum use an enum. Could be nice to set up something cute
being able to do Features.SCHEDULER.is_enabled() for example just to spitball an api

This revision now requires changes to proceed.Aug 12 2019, 9:23 PM
sashank updated this revision to Diff 3631.Aug 13 2019, 5:04 AM

Update to use new Scheduler

Create Features class:

  • Check if feature is enabled using Features.FEATURENAME.is_enabled
sashank retitled this revision from Add schedule storage to dagit to Add Scheduler to dagit and DagsterGraphQLContext.Aug 13 2019, 5:10 AM
sashank edited the summary of this revision. (Show Details)
Harbormaster failed remote builds in B2904: Diff 3631!Aug 13 2019, 5:11 AM
sashank edited the test plan for this revision. (Show Details)Aug 13 2019, 5:11 AM
sashank added a child revision: D804: Create Scheduler GraphQL interface.Aug 13 2019, 9:23 AM
alangenfeld accepted this revision.Aug 13 2019, 3:13 PM
alangenfeld added inline comments.
python_modules/dagit/dagit/cli.py
73

hm wonder if this will be useful at all

python_modules/dagster/dagster/utils/__init__.py
272–310

_ prefix these since they shouldnt be used outside this file

This revision is now accepted and ready to land.Aug 13 2019, 3:13 PM
sashank updated this revision to Diff 3651.Aug 13 2019, 6:14 PM

Make get_enabled_features() private

Harbormaster failed remote builds in B2921: Diff 3651!Aug 13 2019, 6:21 PM
sashank updated this revision to Diff 3653.Aug 13 2019, 6:22 PM

Use six.raise_from in SystemCronScheduler (from last diff)

sashank updated this revision to Diff 3654.Aug 13 2019, 6:26 PM

make graphql

Harbormaster failed remote builds in B2922: Diff 3653!Aug 13 2019, 6:27 PM
Harbormaster completed remote builds in B2923: Diff 3654.Aug 13 2019, 6:32 PM
sashank updated this revision to Diff 3658.Aug 13 2019, 6:54 PM

Rerun tests

Harbormaster completed remote builds in B2927: Diff 3658.Aug 13 2019, 7:01 PM
Closed by commit R1:a8adcc7d7e40: Add Scheduler to dagit and DagsterGraphQLContext (authored by sashank). · Explain WhyAug 13 2019, 8:34 PM
This revision was automatically updated to reflect the committed changes.
Sashank Thupukari <s@thupukari.com> added a commit: R1:a8adcc7d7e40: Add Scheduler to dagit and DagsterGraphQLContext.

Revision Contents
Changeset List

PathSize
js_modules/
dagit/
src/
python_modules/
dagit/
dagit/
30 lines
25 lines
dagit_tests/
27 lines
dagster-graphql/
dagster_graphql/
implementation/
15 lines
scheduler/
17 lines
dagster/
dagster/
utils/
32 lines

Diff 3663

View Options

js_modules/dagit/src/filteredSchema.generated.json

Loading...
View Options

python_modules/dagit/dagit/app.py

Loading...
View Options

python_modules/dagit/dagit/cli.py

Loading...
View Options

python_modules/dagit/dagit_tests/test_app.py

Loading...
View Options

python_modules/dagster-graphql/dagster_graphql/implementation/context.py

Loading...
View Options

python_modules/dagster-graphql/dagster_graphql/implementation/scheduler/system_cron_scheduler.py

Loading...
View Options

python_modules/dagster/dagster/utils/__init__.py

Loading...