IntermediateStore acts as a layer of indirection between IntermediateStorage and ObjectStore, but it more obfuscates how they are related than helps. This change aims to deprecate and remove IntermediateStore entirely by moving its functionality a layer up, to the IntermediateStorage layer.
Modified IntermediateStore tests to instead test the new IntermediateStorage functionality. Propagated the removal of IntermediateStore to a ton of tests.
There's a function build_intermediate_storage_from_object_store. Would we be able to use that in places where we'd use this? It seems like there's a decent bit of duplicated code between the IntermediateStore / IntermediateStorage implementations for S3, GCS, and ADLS2.
Does this need to be kept around? Does it get used anywhere?
Would it make sense to call this intermediate_storage?
I'll take a look and see what I can do
To my knowledge it's been entirely deprecated, but I wanted to have a separate diff that handled all the removal of these classes / complete deprecation of IntermediateStore.
Good point, fits the new naming schema better.
This looks great! My one lingering concern is about whether we need to maintain compatibility with the TypeStoragePlugin interface for now.
I think this is the right direction. However, this is technically a public API that we would be breaking, so I think it probably makes sense to at least separate this part out to a separate change.
If that's a big pain to do at this point, then best would be to post in slack to find out whether anyone thinks it's important to preserve compatibility.
That makes sense. While it would be possible to separate out this part of the change, I do think it would end up being quite an involved one. I will post in slack and find out what people think.