Finish type annotations

respin to allow backcompat with dbt 0.18.x users

Up

Wrap iterator in tighter error bound, fix some messages

works for me !

Add more tests for multi-output case, change arg names

fixing outputs

In D7832#206156, @sandyryza wrote:

It seems like this is a strict improvement over what we currently have, so we should probably merge. That said, do we have an understanding of what happened to this data that we're now missing? Did dbt stop producing it entirely? Is there a different JSON file that contains it?

Update examples, remove lambda_solid from API docs

Fix multi-output, add tests for retry, failure, add asset materialization recording, be more strict about when context is provided.

q mgmt

This is now ready for another round of review

Update on top of resource object changes

I_Contains_Generator -> IContainsGenerator

Change to use isinstance checks to determine generator-ness

Up

Respun using a fxn/context manager to build out a solid execution context, and on top of optional-context changes

Up

Up

q mgmt for now

q mgmt for now

q mgmt for now

Up

Add comments, simplify test

Up

Up

Update test_decorators.py

Docs and errors pass

revert most recent change and stack into invocation diff

Added a "context arg provided" attribute to solid definition. We will use this in invocation to determine whether or not a context arg is provided, rather than magically going based on requirements.

Ensure that we don't tear down a provided instance, only the one that we construct.

Might be better to wait for context-optional, bc we might want to change it again for that.

🙏

Thought of another issue: We only want to tear down an instance that we construct, not one that is passed in.

Move teardown to __del__ if not in context manager scope. This ensures that teardown will always occur before the end of the program.

Think I'm gonna add a follow up / breaking change diff s.t. we're a bit more strict about name validation, ie if context doesn't work, neither should context_ or _context.

Test battery, fix inference

Adding battery of tests

Respin after input arg validation refactor

Respin to have clearer functionality

Respinning to be more clear about functionality

Wait for refactor

There is still a possibility that we don't tear down properly here - if someone initializes a context manager but never tries to actually use anything from it, then we won't perform teardown. Trying to think of a place to inject teardown to ensure that it always occurs (if not in context manager, and not already torn down), I was thinking del?

Update to check whether any resource is a context manager, and if it is, ensure that resources are being accessed in context.

Respin with clearer functionality

Respin with separate fxns for validation and getting extra args

Deque for now

After discussion with @alangenfeld and @sandyryza , gonna work on this in parallel with another approachability change: making solids not necessarily require context arg

Following discussion with @alangenfeld and @sandyryza, we're going to move forward with the direction of directly passing in a context, and punt on this API for now.

Respin with better context manager error support, wait to see how build_resources changes land

Use more descriptive disable expressions

Respin with DirectSolidExecutionContext

Fix kwargs logic, tests for asyncio and multiple outputs

Up

Removing reviewers until this is ready for review again

Up

up

Up

Remove invokable stuff

In D7601#199770, @alangenfeld wrote:

can we add a test for a context manager based resource? what happens? does it just silently not tear down ?

It makes sense to have nice APIs for the simple common case - but I think we should be intentional about what the behavior is when these get used with the complex rare case. I am suspect that just ignoring it is what we want

This discussion may be more appropriate to settle on the diff below

In D7526#199930, @sandyryza wrote:

The main source of uncertainty for me here is what we call the invokable / resourced. A proposal so we can make progress:

• Take out invokable from this diff.
• Make this functionality experimental.
• Merge this diff (it will work on solids without resources).
• Open up a separate diff to propose the API for binding in resources. My guess is that a name like with_resources will be more future-proof than a name like invokable for the reasons in my earlier comment. Also could be used on resources to close resource-to-resource dependencies. @schrockn @alangenfeld maybe worth a direct discussion to get this moving?

In D7596#199654, @sandyryza wrote:

Does the standard Python library need to pull this kind of trickery for open?

In D7596#199552, @alangenfeld wrote:

hmm this is a lot of trickery to have one API be able to do both things.

Its not clear to me that learning that this thing can be used as a context manager is better than learning there is a separate function ie managed_resources for the context managed case

Up and fix worse type sig

Up!!