Modified schema, added BK checks, & added listen/notify skeleton
AbandonedPublicDraft
Actions

Authored by sidkmenon on Feb 9 2021, 5:47 PM.

Details

Reviewers: None

Summary

Listen/Notify skeleton for MySQL, as well as schema changes + BK checks to ensure future MySQL compatability

Note also that the listen/notify logic is not finished yet; a cursor seems to be the best way to go about the polling logic but having trouble finding the correct info in SQLAlchemy documentation.

Still TODO: cursor vs. query modification in SQL query, alembic migrations stuff, as well as testing & perf testing

Depends on https://dagster.phacility.com/D5710

Test Plan

N/A yet (work in progress)

Diff Detail

Repository

R1 dagster

Branch

mysql-backend (branched from master)

Lint

Lint Passed

Unit

No Test Coverage

Event Timeline

sidkmenon created this revision.Feb 9 2021, 5:47 PM

sidkmenon edited the summary of this revision. (Show Details)Feb 9 2021, 5:49 PM

sidkmenon added inline comments.Feb 9 2021, 5:51 PM

python_modules/libraries/dagster-mysql/dagster_mysql/event_log/event_log.py
287–303	This is the area that remains as the primary TODO - basically how I can keep a cursor alive between queries (or whether that's a sane thing to do at all). TBD - will update this diff as I make more progress.

Harbormaster failed remote builds in B25489: Diff 31101!Feb 9 2021, 6:05 PM

alangenfeld added a subscriber: alangenfeld.Feb 9 2021, 10:36 PM

alangenfeld added inline comments.

python_modules/libraries/dagster-mysql/dagster_mysql/event_log/event_log.py
287–303	I would expect the `sqlite` impl to be a relevant example https://dagster.phacility.com/source/dagster/browse/master/python_modules/dagster/dagster/core/storage/event_log/sqlite/sqlite_event_log.py$272-298