Page MenuHomeElementl

[RFC] with_resources

Authored by cdecarolis on Apr 28 2021, 10:06 PM.



This diff adds the with_resources method, which adds a way of providing resources to a solid for invocation.

Test Plan

Review, unit tests

Diff Detail

R1 dagster
Lint Passed
No Test Coverage

Event Timeline

Do we want with_resources to work in a composition context too?
If yes:

  • Where does the config for that resource go?
  • Would that create a new SolidDefinition like configured? New name?
  • If I provide the same ResourceDefinition instance in multiple places - do those share the same instantiated resource if executed in the same plan?
  • How do I tell which resources my solids are using in my pipeline?

If no:

  • How do you explain to a user why not? What is the difference between with_resources, with_hooks, and configured?

These are some of the questions that came up when discussing with_resources on pipeline / graph with @sandyryza .
I think you can work through things down either path, but I don't personally *feel* great about either so far.

It surely has its own problems, but one benefit of APIs that orient around building and passing a value for context is that its clear that they only work for direct invocation since context is explicitly absent during composition.

Following discussion with @alangenfeld and @sandyryza, we're going to move forward with the direction of directly passing in a context, and punt on this API for now.