Page MenuHomeElementl

Add a base IRepositoryData class that can be used to implement repositories with different caching rules
ClosedPublic

Authored by dgibson on Jul 19 2021, 8:44 PM.

Details

Summary

The suggestion in the docs to subclass RepositoryData if you want to implement your own pipeline loading and caching logic doesn't really work - the constructor of RepositoryData makes a bunch of assumptions about caching. Instead, add a base class (essentially an interface, open to naming suggestions) with a bare-bones implemention of get_all_pipelines, and update the example to use it.

Test Plan

New test that verifies that you can use this RepositoryData to implement a repo that changes its definitions every time it is called

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

python_modules/dagster/dagster/core/definitions/repository.py
549–563

I don't think anything was using this (other than for the validation check which I kept)

dgibson published this revision for review.Jul 19 2021, 8:49 PM

ill give some time for @max to take a peak

maybe rename the existing one to CachingRepositoryData and keep RepositoryData as the base class

also we should add a top level export if there isnt one already and double check how this renders in the API docs

i like alex's renaming suggestion

python_modules/dagster/dagster/core/definitions/decorators/repository.py
122

can we be sure to expose this in the docs

This revision is now accepted and ready to land.Jul 21 2021, 5:20 PM
This revision was landed with ongoing or failed builds.Jul 21 2021, 7:50 PM
This revision was automatically updated to reflect the committed changes.