Page MenuHomePhabricator

Use WAL for sqlite-backed run storages as well
ClosedPublic

Authored by max on Dec 19 2019, 7:53 PM.

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

max created this revision.Dec 19 2019, 7:53 PM

test plan Manual

should we use the backfill stuff from @prha to verify that this makes a difference?

given the reading I've done I believe this change is fine at a fundamental level - so I'm torn on whether to do the investigation on whether it address our issues here at diff time or not.

alangenfeld accepted this revision.Dec 19 2019, 8:20 PM

I will defer to you on sequencing

This revision is now accepted and ready to land.Dec 19 2019, 8:20 PM
max added a comment.Dec 19 2019, 10:12 PM

can't repro @prha's backfill issue -- including before the WAL change on the event logs

max updated this revision to Diff 8055.Dec 19 2019, 10:13 PM

Rebase

max added a comment.Dec 19 2019, 10:32 PM

but @prha can still repro

but @prha can still repro

to be clear he can repro with this diff patched?

looking at https://github.com/dagster-io/dagster/issues/2021 too many open files is not an issue this would address - this would only fix lock contention