Details

Reviewers: None

Summary

This passes all tests with the exception of listen/notify stuff. To get fully working, will need to:

Investigate alternatives (MySQL-specific column types?) to the schema changes so that we don't need to incur a migration for existing users
Add an event listener/poller for MySQL that periodically polls the event logs table for new events by tracking the last event timestamp we saw

Test Plan

WIP

Diff Detail

Repository

R1 dagster

Branch

dagster_mysql

Lint

Lint Passed

Unit

No Test Coverage

Event Timeline

• nate created this revision.Dec 19 2020, 10:09 PM

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 19 2020, 10:28 PM

Harbormaster failed remote builds in B23231: Diff 28248!

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 20 2020, 12:26 AM

Harbormaster failed remote builds in B23232: Diff 28249!

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 20 2020, 1:40 AM

Harbormaster failed remote builds in B23233: Diff 28250!

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 20 2020, 6:38 AM

Harbormaster failed remote builds in B23234: Diff 28251!

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 20 2020, 7:50 PM

Harbormaster failed remote builds in B23237: Diff 28254!

Harbormaster failed remote builds in B23235: Diff 28252!Dec 20 2020, 7:57 PM

Harbormaster failed remote builds in B23236: Diff 28253!Dec 20 2020, 8:04 PM

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 21 2020, 5:45 AM

Harbormaster failed remote builds in B23239: Diff 28256!

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 21 2020, 6:11 AM

Harbormaster failed remote builds in B23240: Diff 28257!

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 21 2020, 6:26 AM

Harbormaster failed remote builds in B23241: Diff 28258!

• nate retitled this revision from WIP db changes to Add MySQL support.Dec 26 2020, 11:06 PM

• nate edited the summary of this revision. (Show Details)

rebase

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 4 2021, 10:51 PM

Harbormaster failed remote builds in B23598: Diff 28694!

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 14 2021, 6:49 PM

Harbormaster failed remote builds in B24417: Diff 29721!

catherinewu added a subscriber: catherinewu.Jan 22 2021, 5:49 AM

sidkmenon mentioned this in D6381: Modified schema, added BK checks, & added listen/notify skeleton.Feb 9 2021, 5:47 PM

• nate abandoned this revision.Feb 9 2021, 6:29 PM

sidkmenon mentioned this in D6392: RFC: Added listen/notify skeleton for MySQL (dagit UI updates correctly) + schema changes + BK checks.Feb 9 2021, 9:29 PM

sidkmenon added a child revision: D6392: RFC: Added listen/notify skeleton for MySQL (dagit UI updates correctly) + schema changes + BK checks.Feb 9 2021, 9:51 PM

sidkmenon mentioned this in D6410: Add scaffolding for MySQL support.Feb 10 2021, 6:50 PM

sidkmenon mentioned this in D6617: [dagster-mysql] Add dagster-mysql to libraries.Feb 19 2021, 10:50 PM

Sidharth Menon <menon.sid.k@gmail.com> mentioned this in R1:7eff7a85a436: [dagster-mysql] Add dagster-mysql to libraries.Mar 1 2021, 8:10 PM

Add MySQL supportAbandonedPublicDraftActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 28254

docs/sections/api/apidocs/libraries/dagster_mysql.rst

python_modules/dagster/dagster/core/storage/event_log/schema.py

python_modules/dagster/dagster/core/storage/runs/schema.py

python_modules/dagster/dagster/core/storage/schedules/schema.py

python_modules/dagster/dagster/utils/test/mysql_instance.py

python_modules/libraries/dagster-mysql/.coveragerc

python_modules/libraries/dagster-mysql/LICENSE

python_modules/libraries/dagster-mysql/MANIFEST.in

python_modules/libraries/dagster-mysql/README.md

python_modules/libraries/dagster-mysql/dagster_mysql/__init__.py

python_modules/libraries/dagster-mysql/dagster_mysql/event_log/__init__.py

python_modules/libraries/dagster-mysql/dagster_mysql/event_log/alembic/README

python_modules/libraries/dagster-mysql/dagster_mysql/event_log/alembic/alembic.ini

python_modules/libraries/dagster-mysql/dagster_mysql/event_log/alembic/env.py

python_modules/libraries/dagster-mysql/dagster_mysql/event_log/alembic/script.py.mako

python_modules/libraries/dagster-mysql/dagster_mysql/event_log/event_log.py

python_modules/libraries/dagster-mysql/dagster_mysql/pynotify.py

python_modules/libraries/dagster-mysql/dagster_mysql/run_storage/__init__.py

python_modules/libraries/dagster-mysql/dagster_mysql/run_storage/alembic/README

python_modules/libraries/dagster-mysql/dagster_mysql/run_storage/alembic/alembic.ini

python_modules/libraries/dagster-mysql/dagster_mysql/run_storage/alembic/env.py

python_modules/libraries/dagster-mysql/dagster_mysql/run_storage/alembic/script.py.mako

python_modules/libraries/dagster-mysql/dagster_mysql/run_storage/run_storage.py

python_modules/libraries/dagster-mysql/dagster_mysql/schedule_storage/__init__.py

python_modules/libraries/dagster-mysql/dagster_mysql/schedule_storage/alembic/README

python_modules/libraries/dagster-mysql/dagster_mysql/schedule_storage/alembic/alembic.ini

python_modules/libraries/dagster-mysql/dagster_mysql/schedule_storage/alembic/env.py

python_modules/libraries/dagster-mysql/dagster_mysql/schedule_storage/alembic/script.py.mako

python_modules/libraries/dagster-mysql/dagster_mysql/schedule_storage/schedule_storage.py

python_modules/libraries/dagster-mysql/dagster_mysql/utils.py

python_modules/libraries/dagster-mysql/dagster_mysql/version.py

python_modules/libraries/dagster-mysql/dagster_mysql_tests/__init__.py

python_modules/libraries/dagster-mysql/dagster_mysql_tests/conftest.py

python_modules/libraries/dagster-mysql/dagster_mysql_tests/docker-compose-multi.yml

python_modules/libraries/dagster-mysql/dagster_mysql_tests/docker-compose.yml

python_modules/libraries/dagster-mysql/dagster_mysql_tests/test_event_log.py

python_modules/libraries/dagster-mysql/dagster_mysql_tests/test_instance.py

python_modules/libraries/dagster-mysql/dagster_mysql_tests/test_run_storage.py

python_modules/libraries/dagster-mysql/dagster_mysql_tests/test_schedule_storage.py

python_modules/libraries/dagster-mysql/dagster_mysql_tests/test_version.py

python_modules/libraries/dagster-mysql/setup.cfg

python_modules/libraries/dagster-mysql/setup.py

python_modules/libraries/dagster-mysql/tox.ini

Add MySQL support
AbandonedPublicDraft
Actions

Revision Contents
Changeset List

python_modules/libraries/dagster-mysql/dagster_mysql/init.py

python_modules/libraries/dagster-mysql/dagster_mysql/event_log/init.py

python_modules/libraries/dagster-mysql/dagster_mysql/run_storage/init.py

python_modules/libraries/dagster-mysql/dagster_mysql/schedule_storage/init.py

python_modules/libraries/dagster-mysql/dagster_mysql_tests/init.py