Page MenuHomeElementl

cdecarolis (Christopher DeCarolis)
User

Projects

User does not belong to any projects.

User Details

User Since
Jul 20 2020, 3:42 PM (42 w, 3 d)

Recent Activity

Yesterday

cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Finish type annotations

Thu, May 13, 7:35 PM
cdecarolis planned changes to D7832: [dagster-dbt] re-enable asset materializations when running `dbt cli` solids.

respin to allow backcompat with dbt 0.18.x users

Thu, May 13, 7:30 PM
cdecarolis updated the summary of D7526: [RFC] Directly invoke solids.
Thu, May 13, 7:00 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Up

Thu, May 13, 6:59 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Wrap iterator in tighter error bound, fix some messages

Thu, May 13, 6:17 PM
cdecarolis accepted D7747: add max_completion_wait_time_seconds option to databricks_pyspark_step_launcher.

works for me !

Thu, May 13, 1:41 PM

Wed, May 12

cdecarolis added inline comments to D7526: [RFC] Directly invoke solids.
Wed, May 12, 9:10 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Add more tests for multi-output case, change arg names

Wed, May 12, 9:05 PM
cdecarolis planned changes to D7526: [RFC] Directly invoke solids.

fixing outputs

Wed, May 12, 3:32 PM
cdecarolis added inline comments to D7526: [RFC] Directly invoke solids.
Wed, May 12, 3:21 PM
cdecarolis retitled D7771: Add a flag for whether resources object contains a generator from Make the constructed resources object not break mypy and add a flag for whether a resource is a generator to Add a flag for whether resources object contains a generator.
Wed, May 12, 2:31 PM
cdecarolis added a comment to D7832: [dagster-dbt] re-enable asset materializations when running `dbt cli` solids.

It seems like this is a strict improvement over what we currently have, so we should probably merge. That said, do we have an understanding of what happened to this data that we're now missing? Did dbt stop producing it entirely? Is there a different JSON file that contains it?

Wed, May 12, 2:30 PM

Mon, May 10

cdecarolis requested review of D7843: Add test for ensuring config schema not applied if resource not used.
Mon, May 10, 9:49 PM
cdecarolis retitled D7832: [dagster-dbt] re-enable asset materializations when running `dbt cli` solids from [dagster-dbt] make node result match v1 of results.json to [dagster-dbt] re-enable asset materializations when running `dbt cli` solids.
Mon, May 10, 9:09 PM
cdecarolis requested review of D7832: [dagster-dbt] re-enable asset materializations when running `dbt cli` solids.
Mon, May 10, 8:58 PM
cdecarolis updated the diff for D7677: [Depends on D7674] Refactor lambda_solid as a simple wrapper around _Solid.

Update examples, remove lambda_solid from API docs

Mon, May 10, 8:54 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Fix multi-output, add tests for retry, failure, add asset materialization recording, be more strict about when context is provided.

Mon, May 10, 6:28 PM
cdecarolis added inline comments to D7526: [RFC] Directly invoke solids.
Mon, May 10, 4:59 PM

Fri, May 7

cdecarolis abandoned D7596: make build_resources usable as a function..

q mgmt

Fri, May 7, 8:06 PM
cdecarolis requested review of D7526: [RFC] Directly invoke solids.
Fri, May 7, 7:57 PM
cdecarolis planned changes to D7526: [RFC] Directly invoke solids.

This is now ready for another round of review

Fri, May 7, 7:57 PM
cdecarolis requested review of D7778: Update solid examples.
Fri, May 7, 7:36 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Update on top of resource object changes

Fri, May 7, 7:09 PM
cdecarolis updated the diff for D7771: Add a flag for whether resources object contains a generator.

I_Contains_Generator -> IContainsGenerator

Fri, May 7, 7:06 PM
cdecarolis updated the diff for D7771: Add a flag for whether resources object contains a generator.

Change to use isinstance checks to determine generator-ness

Fri, May 7, 4:25 PM

Thu, May 6

cdecarolis added a reviewer for D7771: Add a flag for whether resources object contains a generator: sandyryza.
Thu, May 6, 8:08 PM
cdecarolis updated the diff for D7677: [Depends on D7674] Refactor lambda_solid as a simple wrapper around _Solid.

Up

Thu, May 6, 8:06 PM
cdecarolis published D7771: Add a flag for whether resources object contains a generator for review.
Thu, May 6, 6:50 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Respun using a fxn/context manager to build out a solid execution context, and on top of optional-context changes

Thu, May 6, 4:24 PM
cdecarolis updated the diff for D7596: make build_resources usable as a function..

Up

Thu, May 6, 4:22 PM
cdecarolis updated the diff for D7596: make build_resources usable as a function..

Up

Thu, May 6, 2:24 PM

Wed, May 5

cdecarolis abandoned D7194: [execute_in_process_improvements 1/n] Add method to convert pipeline definition to composite solid definition.

q mgmt for now

Wed, May 5, 9:32 PM
cdecarolis abandoned D6822: [execute_in_process improvements 2/n] provide resource instances, split out config and inputs..

q mgmt for now

Wed, May 5, 9:32 PM
cdecarolis abandoned D7601: [depends on D7596] build_resource.

q mgmt for now

Wed, May 5, 9:31 PM
cdecarolis updated the diff for D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

Up

Wed, May 5, 9:23 PM
cdecarolis updated the diff for D7596: make build_resources usable as a function..

Add comments, simplify test

Wed, May 5, 9:22 PM
cdecarolis updated the diff for D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

Up

Wed, May 5, 7:48 PM
cdecarolis updated the diff for D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

Up

Wed, May 5, 7:44 PM
cdecarolis updated the diff for D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

Update test_decorators.py

Wed, May 5, 5:54 PM
cdecarolis updated the test plan for D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.
Wed, May 5, 5:01 PM
cdecarolis updated the diff for D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

Docs and errors pass

Wed, May 5, 4:46 PM
cdecarolis planned changes to D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

revert most recent change and stack into invocation diff

Wed, May 5, 3:25 PM
cdecarolis updated the diff for D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

Added a "context arg provided" attribute to solid definition. We will use this in invocation to determine whether or not a context arg is provided, rather than magically going based on requirements.

Wed, May 5, 3:02 PM

Tue, May 4

cdecarolis updated the diff for D7596: make build_resources usable as a function..

Ensure that we don't tear down a provided instance, only the one that we construct.

Tue, May 4, 7:20 PM
cdecarolis added a comment to D7729: remove lambda solid from pipeline apidoc.

Might be better to wait for context-optional, bc we might want to change it again for that.

Tue, May 4, 6:17 PM
cdecarolis accepted D7729: remove lambda solid from pipeline apidoc.

🙏

Tue, May 4, 6:16 PM
cdecarolis planned changes to D7596: make build_resources usable as a function..

Thought of another issue: We only want to tear down an instance that we construct, not one that is passed in.

Tue, May 4, 6:13 PM
cdecarolis updated the diff for D7596: make build_resources usable as a function..

Move teardown to __del__ if not in context manager scope. This ensures that teardown will always occur before the end of the program.

Tue, May 4, 4:09 PM
cdecarolis added a comment to D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

Think I'm gonna add a follow up / breaking change diff s.t. we're a bit more strict about name validation, ie if context doesn't work, neither should context_ or _context.

Tue, May 4, 3:49 PM
cdecarolis added inline comments to D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.
Tue, May 4, 3:47 PM

Mon, May 3

cdecarolis updated the diff for D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

Test battery, fix inference

Mon, May 3, 8:56 PM
cdecarolis planned changes to D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

Adding battery of tests

Mon, May 3, 7:14 PM
cdecarolis updated the diff for D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

Respin after input arg validation refactor

Mon, May 3, 6:54 PM
cdecarolis accepted D7691: fix Dict & Set inner_types.

Macro aerodactyl:

Mon, May 3, 4:15 PM
cdecarolis updated the diff for D7670: Refactor the validation step for positional args in decorators.

Respin to have clearer functionality

Mon, May 3, 4:12 PM
cdecarolis planned changes to D7670: Refactor the validation step for positional args in decorators.

Respinning to be more clear about functionality

Mon, May 3, 3:42 PM
cdecarolis planned changes to D7677: [Depends on D7674] Refactor lambda_solid as a simple wrapper around _Solid.

Wait for refactor

Mon, May 3, 3:31 PM
cdecarolis added a comment to D7596: make build_resources usable as a function..

There is still a possibility that we don't tear down properly here - if someone initializes a context manager but never tries to actually use anything from it, then we won't perform teardown. Trying to think of a place to inject teardown to ensure that it always occurs (if not in context manager, and not already torn down), I was thinking del?

Mon, May 3, 3:30 PM
cdecarolis added a comment to D7607: [mypy] resource definition.
Mon, May 3, 3:09 PM
cdecarolis requested review of D7607: [mypy] resource definition.
Mon, May 3, 3:05 PM
cdecarolis updated the diff for D7596: make build_resources usable as a function..

Update to check whether any resource is a context manager, and if it is, ensure that resources are being accessed in context.

Mon, May 3, 2:59 PM

Fri, Apr 30

cdecarolis updated the diff for D7670: Refactor the validation step for positional args in decorators.

Respin with clearer functionality

Fri, Apr 30, 7:36 PM
cdecarolis planned changes to D7670: Refactor the validation step for positional args in decorators.

Respin with separate fxns for validation and getting extra args

Fri, Apr 30, 6:07 PM
cdecarolis added inline comments to D7670: Refactor the validation step for positional args in decorators.
Fri, Apr 30, 6:06 PM
cdecarolis planned changes to D7194: [execute_in_process_improvements 1/n] Add method to convert pipeline definition to composite solid definition.

Deque for now

Fri, Apr 30, 4:41 PM
cdecarolis requested review of D7677: [Depends on D7674] Refactor lambda_solid as a simple wrapper around _Solid.
Fri, Apr 30, 4:38 PM
cdecarolis added inline comments to D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.
Fri, Apr 30, 4:20 PM
cdecarolis added inline comments to D7646: RFC solid RetryPolicy.
Fri, Apr 30, 2:10 PM
cdecarolis updated the summary of D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.
Fri, Apr 30, 12:00 AM

Thu, Apr 29

cdecarolis requested review of D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.
Thu, Apr 29, 11:59 PM
cdecarolis added inline comments to D7670: Refactor the validation step for positional args in decorators.
Thu, Apr 29, 11:16 PM
cdecarolis added inline comments to D7670: Refactor the validation step for positional args in decorators.
Thu, Apr 29, 11:15 PM
cdecarolis requested review of D7670: Refactor the validation step for positional args in decorators.
Thu, Apr 29, 11:11 PM
cdecarolis planned changes to D7526: [RFC] Directly invoke solids.

After discussion with @alangenfeld and @sandyryza , gonna work on this in parallel with another approachability change: making solids not necessarily require context arg

Thu, Apr 29, 5:35 PM
cdecarolis abandoned D7648: [RFC] with_resources.

Following discussion with @alangenfeld and @sandyryza, we're going to move forward with the direction of directly passing in a context, and punt on this API for now.

Thu, Apr 29, 5:33 PM

Wed, Apr 28

cdecarolis requested review of D7648: [RFC] with_resources.
Wed, Apr 28, 10:27 PM
cdecarolis planned changes to D7601: [depends on D7596] build_resource.

Respin with better context manager error support, wait to see how build_resources changes land

Wed, Apr 28, 9:30 PM
cdecarolis updated the summary of D7526: [RFC] Directly invoke solids.
Wed, Apr 28, 9:01 PM
cdecarolis added reviewers for D7526: [RFC] Directly invoke solids: sandyryza, schrockn.
Wed, Apr 28, 9:01 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Use more descriptive disable expressions

Wed, Apr 28, 8:52 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Respin with DirectSolidExecutionContext

Wed, Apr 28, 8:51 PM

Tue, Apr 27

cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Fix kwargs logic, tests for asyncio and multiple outputs

Tue, Apr 27, 8:43 PM
cdecarolis added inline comments to D7526: [RFC] Directly invoke solids.
Tue, Apr 27, 7:58 PM
cdecarolis added inline comments to D7526: [RFC] Directly invoke solids.
Tue, Apr 27, 5:34 PM
cdecarolis added inline comments to D7526: [RFC] Directly invoke solids.
Tue, Apr 27, 5:32 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Up

Tue, Apr 27, 5:22 PM
cdecarolis removed reviewers for D7526: [RFC] Directly invoke solids: alangenfeld, sandyryza, schrockn, prha.

Removing reviewers until this is ready for review again

Tue, Apr 27, 5:09 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Up

Tue, Apr 27, 4:46 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

up

Tue, Apr 27, 4:43 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Up

Tue, Apr 27, 4:41 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Remove invokable stuff

Tue, Apr 27, 4:40 PM
cdecarolis added a comment to D7601: [depends on D7596] build_resource.

can we add a test for a context manager based resource? what happens? does it just silently not tear down ?

It makes sense to have nice APIs for the simple common case - but I think we should be intentional about what the behavior is when these get used with the complex rare case. I am suspect that just ignoring it is what we want

This discussion may be more appropriate to settle on the diff below

Tue, Apr 27, 2:52 PM
cdecarolis added a comment to D7526: [RFC] Directly invoke solids.

The main source of uncertainty for me here is what we call the invokable / resourced. A proposal so we can make progress:

  • Take out invokable from this diff.
  • Make this functionality experimental.
  • Merge this diff (it will work on solids without resources).
  • Open up a separate diff to propose the API for binding in resources. My guess is that a name like with_resources will be more future-proof than a name like invokable for the reasons in my earlier comment. Also could be used on resources to close resource-to-resource dependencies. @schrockn @alangenfeld maybe worth a direct discussion to get this moving?
Tue, Apr 27, 2:43 PM
cdecarolis accepted D7610: s3 io manager tiny cleanup.
Tue, Apr 27, 2:17 PM

Mon, Apr 26

cdecarolis added a comment to D7596: make build_resources usable as a function..

Does the standard Python library need to pull this kind of trickery for open?

Mon, Apr 26, 8:28 PM
cdecarolis requested review of D7605: [Docs] Fix header of Solids/Pipeline testing.
Mon, Apr 26, 8:05 PM
cdecarolis added a comment to D7596: make build_resources usable as a function..

hmm this is a lot of trickery to have one API be able to do both things.

Its not clear to me that learning that this thing can be used as a context manager is better than learning there is a separate function ie managed_resources for the context managed case

Mon, Apr 26, 6:36 PM
cdecarolis requested review of D7601: [depends on D7596] build_resource.
Mon, Apr 26, 6:05 PM
cdecarolis updated the diff for D7596: make build_resources usable as a function..

Up and fix worse type sig

Mon, Apr 26, 5:44 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Up!!

Mon, Apr 26, 5:41 PM