HomePhabricator

have IntermediateStorage determine version addresses

Description

have IntermediateStorage determine version addresses

Summary:
Depends on D4733.

This diff, aimed getting version-based memoization truly working for a user who's trying it out, takes a step back in exchange for a step forward:

  • It breaks version-based step selection when the user specifies an address in the compute_fn. I believe getting that working again will depend on work Yuhan is doing to pass addresses yielded from one solid to downstream solids.
  • It enables version-based memoization and step selection for users who don't specify an address in the compute_fn.

It does this by putting the IntermediateStorage in charge of determining the address that's associated with a particular versioned output.

Doing this required moving versions and addresses from step output events to object store operation events. We probably should have done this in the first place. I wasn't aware of these kinds of events.

Test Plan: updated tests

Reviewers: yuhan, cdecarolis, alangenfeld

Reviewed By: cdecarolis

Differential Revision: https://dagster.phacility.com/D4738

Details