Page MenuHomeElementl

Add a base IRepositoryData class that can be used to implement repositories with different caching rules

Authored by dgibson on Jul 19 2021, 8:44 PM.
Referenced Files
Unknown Object (File)
Tue, Jan 31, 11:03 AM
Unknown Object (File)
Wed, Jan 25, 6:50 AM
Unknown Object (File)
Wed, Jan 25, 6:50 AM
Unknown Object (File)
Mon, Jan 16, 10:40 AM
Unknown Object (File)
Dec 25 2022, 8:38 AM
Unknown Object (File)
Nov 18 2022, 1:23 AM
Unknown Object (File)
Nov 18 2022, 1:23 AM
Unknown Object (File)
Nov 18 2022, 1:22 AM



The suggestion in the docs to subclass RepositoryData if you want to implement your own pipeline loading and caching logic doesn't really work - the constructor of RepositoryData makes a bunch of assumptions about caching. Instead, add a base class (essentially an interface, open to naming suggestions) with a bare-bones implemention of get_all_pipelines, and update the example to use it.

Test Plan

New test that verifies that you can use this RepositoryData to implement a repo that changes its definitions every time it is called

Diff Detail

R1 dagster
Lint Not Applicable
Tests Not Applicable

Event Timeline


I don't think anything was using this (other than for the validation check which I kept)

ill give some time for @max to take a peak

maybe rename the existing one to CachingRepositoryData and keep RepositoryData as the base class

also we should add a top level export if there isnt one already and double check how this renders in the API docs

i like alex's renaming suggestion


can we be sure to expose this in the docs

This revision is now accepted and ready to land.Jul 21 2021, 5:20 PM
This revision was landed with ongoing or failed builds.Jul 21 2021, 7:50 PM
This revision was automatically updated to reflect the committed changes.