Page MenuHomePhabricator
Differential D5982

Don't always run create_all during storage creation, add stamps to postgres storage
ClosedPublic
Actions

Authored by dgibson on Jan 13 2021, 10:49 PM.

Details

Reviewers
alangenfeld
prha
max
Commits
R1:0b06d69e093c: Don't always run create_all during storage creation, add stamps to postgres…
Summary

Still testing this, but this is a more invasive migration solution that should leave our alembic stamps consistent .

Instead of relying on create_all to create the new tables we added in 0.10.0, we only create new tables during the initial load, and during migration.

Test Plan

BK

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

dgibson created this revision.Jan 13 2021, 10:49 PM
dgibson updated this revision to Diff 29612.Jan 13 2021, 11:00 PM

remove a test involving migrating from a time when there was no ennet log, since that was incorrectly stamping (is it safe to assume that 0.7.6 => 0.10.0 migrations are not a thing?)

Harbormaster failed remote builds in B24331: Diff 29610!Jan 13 2021, 11:05 PM
Harbormaster returned this revision to the author for changes because remote builds failed.Jan 13 2021, 11:16 PM
Harbormaster failed remote builds in B24332: Diff 29612!
dgibson updated this revision to Diff 29616.Jan 13 2021, 11:20 PM

up

dgibson published this revision for review.Jan 13 2021, 11:20 PM
dgibson mentioned this in D5926: Raise a migration required exception when you load a pre-0.10.0 schedule storage.
dgibson updated this revision to Diff 29618.Jan 13 2021, 11:33 PM

keep the skip_migration_checks stuff out of DagsterInstance.get()

Harbormaster failed remote builds in B24336: Diff 29616!Jan 13 2021, 11:36 PM
Harbormaster failed remote builds in B24337: Diff 29618!Jan 13 2021, 11:50 PM
dgibson updated this revision to Diff 29629.Jan 14 2021, 12:02 AM

result of chat with phil - take out places where we were dropping the "schedules" table in past migrations, so that we don't lose our clue in the 0.10.0 migration that we're in the schedule DB. Instead, drop the schedules table at the end of hte 0.10.0 migration.

Harbormaster failed remote builds in B24344: Diff 29629!Jan 14 2021, 12:18 AM
prha added inline comments.Jan 14 2021, 12:19 AM
python_modules/dagster/dagster/core/storage/schedules/sqlite/alembic/versions/140198fdfe65_wipe_schedules_table_for_0_10_0.py
38–39

maybe we should remove these too.

python_modules/dagster/dagster/core/storage/schedules/sqlite/alembic/versions/b22f16781a7c_.py
27–28

maybe comment these out instead of removing? and add a comment about why?

python_modules/libraries/dagster-postgres/dagster_postgres/run_storage/alembic/versions/5c18fd3c2957_scheduler_update.py
27–28

same, maybe comment here instead?

dgibson updated this revision to Diff 29645.Jan 14 2021, 2:07 AM

phil's feedback

dgibson updated this revision to Diff 29648.Jan 14 2021, 2:22 AM

up

dgibson updated this revision to Diff 29649.Jan 14 2021, 2:24 AM

fix test

Harbormaster failed remote builds in B24357: Diff 29645!Jan 14 2021, 2:25 AM
Harbormaster failed remote builds in B24359: Diff 29649!Jan 14 2021, 2:41 AM
Harbormaster failed remote builds in B24358: Diff 29648!
prha accepted this revision.Jan 14 2021, 4:53 AM

push_n_pray

This revision is now accepted and ready to land.Jan 14 2021, 4:53 AM
dgibson updated this revision to Diff 29676.Jan 14 2021, 3:03 PM

up

This revision was landed with ongoing or failed builds.Jan 14 2021, 3:22 PM
Closed by commit R1:0b06d69e093c: Don't always run create_all during storage creation, add stamps to postgres… (authored by dgibson). · Explain Why
This revision was automatically updated to reflect the committed changes.
dgibson added a commit: R1:0b06d69e093c: Don't always run create_all during storage creation, add stamps to postgres….
Harbormaster completed remote builds in B24384: Diff 29676.Jan 14 2021, 3:27 PM
alangenfeld mentioned this in D5996: only connect on init when needed.Jan 14 2021, 4:03 PM
alangenfeld mentioned this in R1:b14a5d08d927: only connect on init when needed.Jan 14 2021, 4:25 PM

Revision Contents
Changeset List

PathSize
python_modules/
dagster/
dagster/
cli/
12 lines
core/
instance/
16 lines
storage/
event_log/
sqlite/
alembic/
versions/
22 lines
4 lines
4 lines
migration/
96 lines
runs/
8 lines
sqlite/
alembic/
versions/
22 lines
9 lines
schedules/
sqlite/
alembic/
versions/
14 lines
22 lines
14 lines
4 lines
dagster_tests/
general_tests/
compat_tests/
10 lines
libraries/
dagster-postgres/
dagster_postgres/
event_log/
alembic/
versions/
22 lines
22 lines
14 lines
22 lines
14 lines
15 lines
run_storage/
alembic/
versions/
22 lines
22 lines
14 lines
22 lines
14 lines
29 lines
schedule_storage/
alembic/
versions/
22 lines
22 lines
14 lines
22 lines
14 lines
17 lines
7 lines
dagster_postgres_tests/
compat_tests/
183 lines

Diff 29681

View Options

python_modules/dagster/dagster/cli/instance.py

Loading...
View Options

python_modules/dagster/dagster/core/instance/__init__.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/event_log/sqlite/alembic/versions/942138e33bf9_0_10_0_create_new_event_log_tables.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/event_log/sqlite/consolidated_sqlite_event_log.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/event_log/sqlite/sqlite_event_log.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/migration/utils.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/runs/sql_run_storage.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/runs/sqlite/alembic/versions/f5bac2c36fad_0_10_0_create_new_run_tables.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/runs/sqlite/sqlite_run_storage.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/schedules/sqlite/alembic/versions/140198fdfe65_wipe_schedules_table_for_0_10_0.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/schedules/sqlite/alembic/versions/8ccbed5060b8_0_10_0_create_new_schedule_tables.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/schedules/sqlite/alembic/versions/b22f16781a7c_.py

Loading...
View Options

python_modules/dagster/dagster/core/storage/schedules/sqlite/sqlite_schedule_storage.py

Loading...
View Options

python_modules/dagster/dagster_tests/general_tests/compat_tests/test_back_compat.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/event_log/alembic/versions/493871843165_0_10_0_create_new_schedule_tables.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/event_log/alembic/versions/5ae53cd67d0c_0_10_0_create_new_run_tables.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/event_log/alembic/versions/5c18fd3c2957_scheduler_update.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/event_log/alembic/versions/a0234163e0e3_0_10_0_create_new_event_log_tables.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/event_log/alembic/versions/b32a4f3036d2_wipe_schedules_table_for_0_10_0.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/event_log/event_log.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/run_storage/alembic/versions/493871843165_0_10_0_create_new_schedule_tables.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/run_storage/alembic/versions/5ae53cd67d0c_0_10_0_create_new_run_tables.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/run_storage/alembic/versions/5c18fd3c2957_scheduler_update.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/run_storage/alembic/versions/a0234163e0e3_0_10_0_create_new_event_log_tables.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/run_storage/alembic/versions/b32a4f3036d2_wipe_schedules_table_for_0_10_0.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/run_storage/run_storage.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/schedule_storage/alembic/versions/493871843165_0_10_0_create_new_schedule_tables.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/schedule_storage/alembic/versions/5ae53cd67d0c_0_10_0_create_new_run_tables.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/schedule_storage/alembic/versions/5c18fd3c2957_scheduler_update.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/schedule_storage/alembic/versions/a0234163e0e3_0_10_0_create_new_event_log_tables.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/schedule_storage/alembic/versions/b32a4f3036d2_wipe_schedules_table_for_0_10_0.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/schedule_storage/schedule_storage.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres/utils.py

Loading...
View Options

python_modules/libraries/dagster-postgres/dagster_postgres_tests/compat_tests/test_back_compat.py

Loading...