Page MenuHomeElementl

Add a base IRepositoryData class that can be used to implement repositories with different caching rules
ClosedPublic

Authored by dgibson on Jul 19 2021, 8:44 PM.
Tags
None
Referenced Files
F2894373: D8953.diff
Sun, Mar 26, 9:26 PM
Unknown Object (File)
Fri, Mar 10, 8:57 PM
Unknown Object (File)
Feb 15 2023, 1:54 AM
Unknown Object (File)
Feb 14 2023, 10:14 PM
Unknown Object (File)
Feb 14 2023, 5:29 PM
Unknown Object (File)
Feb 12 2023, 12:30 AM
Unknown Object (File)
Feb 11 2023, 4:10 PM
Unknown Object (File)
Jan 31 2023, 11:03 AM
Subscribers
None

Details

Summary

The suggestion in the docs to subclass RepositoryData if you want to implement your own pipeline loading and caching logic doesn't really work - the constructor of RepositoryData makes a bunch of assumptions about caching. Instead, add a base class (essentially an interface, open to naming suggestions) with a bare-bones implemention of get_all_pipelines, and update the example to use it.

Test Plan

New test that verifies that you can use this RepositoryData to implement a repo that changes its definitions every time it is called

Diff Detail

Repository
R1 dagster
Branch
xu
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

python_modules/dagster/dagster/core/definitions/repository.py
549–563

I don't think anything was using this (other than for the validation check which I kept)

ill give some time for @max to take a peak

maybe rename the existing one to CachingRepositoryData and keep RepositoryData as the base class

also we should add a top level export if there isnt one already and double check how this renders in the API docs

i like alex's renaming suggestion

python_modules/dagster/dagster/core/definitions/decorators/repository.py
122

can we be sure to expose this in the docs

This revision is now accepted and ready to land.Jul 21 2021, 5:20 PM
This revision was landed with ongoing or failed builds.Jul 21 2021, 7:50 PM
This revision was automatically updated to reflect the committed changes.