Page MenuHomePhabricator

Refactor PipelineRunStorage
ClosedPublic

Authored by max on Aug 7 2019, 6:53 PM.

Details

Reviewers
alangenfeld
Group Reviewers
Restricted Project
Commits
R1:524361d9a76d: Refactor PipelineRunStorage
Summary

Preliminary to further work on distributed logging

Test Plan

Unit

Diff Detail

Repository
R1 dagster
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

max retitled this revision from Wip to Refactor PipelineRunStorage.Aug 8 2019, 6:54 PM
max edited the summary of this revision. (Show Details)
max added a reviewer: Restricted Project.
alangenfeld added inline comments.
python_modules/dagit/dagit/cli.py
55–63

for your consideration in follow ups - --log is a bit goofy of a set up for determining storage, we should come up with a more explicit scheme. dagit --storage fs or something like that

python_modules/dagster-graphql/dagster_graphql/implementation/pipeline_run_storage.py
32–39

kinda feel like we can drop pipeline here, RunStorage communicates whats going on just as effectively imo.

123

this is used by both dagster-graphql and dagit now so this err message is stale. Also could do a better job communicating that its just moving on instead of failing outright

python_modules/dagster/dagster/core/events/log.py
35

while a small code change, this is a pretty big system change - can you explain what the motivation here is? Just want to make sure we think through this since I reckon theres no going back as something will depend on these ids quickly.

to your queue

This revision now requires changes to proceed.Aug 8 2019, 7:15 PM
python_modules/dagit/dagit/cli.py
55–63

yep

python_modules/dagster-graphql/dagster_graphql/implementation/pipeline_run_storage.py
32–39

ok

123

ok

python_modules/dagster/dagster/core/events/log.py
35

yep, i am trying to make sure we have a way to tell if we've seen an event record before or not. i think this will simplify the case where we have two dagit processes writing/reading from a shared filesystem run storage

alangenfeld added inline comments.
python_modules/dagster/dagster/core/events/log.py
35

I'm inclined to punt this until we know the exact constraints of what we are trying to solve, for example I can imagine choosing a different scheme than uuid

This revision is now accepted and ready to land.Aug 9 2019, 4:45 PM
python_modules/dagster/dagster/core/events/log.py
35

what do you have in mind? i think hashing is going to suck

This revision was automatically updated to reflect the committed changes.