Page MenuHomeElementl
Feed Advanced Search

Jul 22 2021

alangenfeld added a comment to D8942: [crag] Job description.

I think its a reasonable approach to land this and change the defaulting-from-graph behavior later if we decide its the right thing to do

Jul 22 2021, 3:13 PM
alangenfeld accepted D8998: check.is_dict.

whynot

Jul 22 2021, 3:10 PM

Jul 21 2021

alangenfeld added reviewers for D8981: add `from_failure` bool arg to rexecute pipeline apis: sandyryza, owen.

but the cost is to expose a new top-level api and the api being verbose

Jul 21 2021, 10:02 PM
alangenfeld added a comment to D8550: [op] permit op config entry.

expect to see some tests in python_modules/dagster/dagster_tests/core_tests/config_types_tests/ for validation, hashing, etc

Jul 21 2021, 8:21 PM
alangenfeld added inline comments to D8550: [op] permit op config entry.
Jul 21 2021, 8:20 PM
alangenfeld added a comment to D8981: add `from_failure` bool arg to rexecute pipeline apis.

I would like to discuss alternative ways to expose this. My hesitation towards this approach comes from using a bool argument that conflicts with another argument [1]. I think this represents an ergonomic / feel issue as well as point at a lack of extensibility if we desire other times of re-execution where steps are selected by the system.

Jul 21 2021, 7:43 PM
alangenfeld closed D8990: pull description from doc string on all decorators.
Jul 21 2021, 5:50 PM
alangenfeld committed R1:9ff4feee97a2: pull description from doc string on all decorators (authored by alangenfeld).
pull description from doc string on all decorators
Jul 21 2021, 5:50 PM
alangenfeld accepted D8991: [dagit] Fix JobMetadataQuery for non-crag.
Jul 21 2021, 5:04 PM
alangenfeld requested review of D8990: pull description from doc string on all decorators.
Jul 21 2021, 5:00 PM
alangenfeld updated the diff for D8942: [crag] Job description.

rebase

Jul 21 2021, 4:30 PM
alangenfeld added reviewers for D8942: [crag] Job description: prha, owen.
Jul 21 2021, 4:29 PM
alangenfeld added reviewers for D8943: [crag] fix snapshots handling composed @graph: prha, sandyryza.
Jul 21 2021, 4:27 PM
alangenfeld resigned from D8979: [crag] partition schedules migration guide.
Jul 21 2021, 4:24 PM
alangenfeld added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

Do we envision memoization to always be opt-in? Or could we at some point make it opt-out?

Jul 21 2021, 4:09 PM

Jul 20 2021

alangenfeld added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

what is the right way to opt-in to memoization behavior? Is it a toggle on the Job? Something else? I don't know if I have a good answer.

Jul 20 2021, 9:35 PM
alangenfeld added a reviewer for D8963: [memoization 3/n] move core memoization logic to live on the execution plan: yuhan.

cc @yuhan I feel like this is pretty related to the load-from-other-run-in-the-run-group look-ups we are doing inefficiently right now. Seems like there might be a way for this and that to work the same or be modeled the same

Jul 20 2021, 9:27 PM
alangenfeld accepted D8967: fix postgres backcompat test flakiness (stable set comparison).
Jul 20 2021, 9:12 PM
alangenfeld closed D8974: remove bad use of 'in'.
Jul 20 2021, 9:09 PM
alangenfeld committed R1:60c2561c5ed6: remove bad use of 'in' (authored by alangenfeld).
remove bad use of 'in'
Jul 20 2021, 9:09 PM
alangenfeld requested review of D8974: remove bad use of 'in'.

flake

Jul 20 2021, 8:57 PM
alangenfeld accepted D8939: Better tests for dbt integration guide, and slightly modify example.

cubone

Jul 20 2021, 8:54 PM
alangenfeld requested changes to D8920: [memoization improvements 1/n] get rid of step_keys_to_execute parameter on create_run_for_pipeline, simplify implementation of create_run_for_pipeline..

gonna round trip this one since im a little weary of that new check run for all creates

Jul 20 2021, 8:02 PM
alangenfeld accepted D8923: [memoization improvements 2/n] allow dagster execution with no steps..

feels like there should be a few more tests but im struggling to think of what cases make sense to test. reexecute_pipeline with explicit empty step selection?

Jul 20 2021, 8:01 PM
alangenfeld added inline comments to D8920: [memoization improvements 1/n] get rid of step_keys_to_execute parameter on create_run_for_pipeline, simplify implementation of create_run_for_pipeline..
Jul 20 2021, 7:59 PM
alangenfeld added inline comments to D8550: [op] permit op config entry.
Jul 20 2021, 5:47 PM
alangenfeld requested changes to D8550: [op] permit op config entry.
  • I think field_aliases is better than field_substitutions since we are just creating a second name that the same information is available under
    • related - I think you want to just swap the order since you are flipping it in like all the call sites
  • use this approach to handle storage compat
  • add tests at the Shape level in the config package
  • add tests for conflicts (error I assume)
  • take a minute and look at the context around each change to verify the impl feels right. I think the dict get with default trick you used was good to get it working, but there may be cleaner approaches.
Jul 20 2021, 5:45 PM
alangenfeld closed D8898: [mypy] update check.inst to not blank out types.
Jul 20 2021, 5:29 PM
alangenfeld committed R1:f613452a3b32: [mypy] update check.inst to not blank out types (authored by alangenfeld).
[mypy] update check.inst to not blank out types
Jul 20 2021, 5:29 PM
alangenfeld updated the diff for D8898: [mypy] update check.inst to not blank out types.

rebase, deserialize_as typing

Jul 20 2021, 4:22 PM
alangenfeld added inline comments to D8942: [crag] Job description.
Jul 20 2021, 3:47 PM
alangenfeld added a comment to D8942: [crag] Job description.

Im not sure. I think the graph page having the graph desc is clear.

Jul 20 2021, 3:43 PM
alangenfeld accepted D8903: Add solid as parameter to build_hook_context.

soclean

Jul 20 2021, 3:29 PM
alangenfeld accepted D8918: improvements to graph/job/op migration guide.

thumbsup

Jul 20 2021, 3:01 PM
alangenfeld added inline comments to D8903: Add solid as parameter to build_hook_context.
Jul 20 2021, 2:59 PM
alangenfeld closed D8945: [crag] jobs entry in @repository dict form.
Jul 20 2021, 2:35 PM
alangenfeld committed R1:24f8b4ed31b1: [crag] jobs entry in @repository dict form (authored by alangenfeld).
[crag] jobs entry in @repository dict form
Jul 20 2021, 2:35 PM

Jul 19 2021

alangenfeld requested review of D8943: [crag] fix snapshots handling composed @graph.
Jul 19 2021, 10:40 PM
alangenfeld closed D8947: unpin docstring-parser.
Jul 19 2021, 10:19 PM
alangenfeld committed R1:17544b6ca229: unpin docstring-parser (authored by alangenfeld).
unpin docstring-parser
Jul 19 2021, 10:19 PM
alangenfeld closed D8928: increase test_process_killed_after_client_finished timeout.
Jul 19 2021, 10:15 PM
alangenfeld committed R1:8fa0c1041367: increase test_process_killed_after_client_finished timeout (authored by alangenfeld).
increase test_process_killed_after_client_finished timeout
Jul 19 2021, 10:15 PM
alangenfeld added a comment to D8947: unpin docstring-parser.

can we make sure the make build in docs still works - iirc we had some lib version related issue when building the api docs. i could be wrong but just wanted to make sure unpinning it wont break the docs build.

Jul 19 2021, 10:10 PM
alangenfeld added a comment to D8550: [op] permit op config entry.

If we want to go back to the previous direction, we could walk / refer to the pipeline [1] structure when transforming the config.

Jul 19 2021, 10:05 PM
alangenfeld added a comment to D8953: Add a base IRepositoryData class that can be used to implement repositories with different caching rules.

also we should add a top level export if there isnt one already and double check how this renders in the API docs

Jul 19 2021, 9:14 PM
alangenfeld added a comment to D8953: Add a base IRepositoryData class that can be used to implement repositories with different caching rules.

ill give some time for @max to take a peak

Jul 19 2021, 9:13 PM
alangenfeld requested review of D8947: unpin docstring-parser.
Jul 19 2021, 6:05 PM
alangenfeld requested review of D8945: [crag] jobs entry in @repository dict form.
Jul 19 2021, 5:41 PM
alangenfeld requested review of D8942: [crag] Job description.
Jul 19 2021, 3:54 PM

Jul 16 2021

alangenfeld closed D8933: NodeHandle serdes adjustments.
Jul 16 2021, 10:31 PM
alangenfeld committed R1:332561414f25: NodeHandle serdes adjustments (authored by alangenfeld).
NodeHandle serdes adjustments
Jul 16 2021, 10:31 PM
alangenfeld added inline comments to D8933: NodeHandle serdes adjustments.
Jul 16 2021, 10:21 PM
alangenfeld added reviewers for D8933: NodeHandle serdes adjustments: prha, yuhan.
Jul 16 2021, 9:57 PM
alangenfeld updated the diff for D8933: NodeHandle serdes adjustments.

snapshots

Jul 16 2021, 8:51 PM
alangenfeld published D8933: NodeHandle serdes adjustments for review.
Jul 16 2021, 7:48 PM
alangenfeld requested review of D8928: increase test_process_killed_after_client_finished timeout.
Jul 16 2021, 6:30 PM
alangenfeld added a comment to D8903: Add solid as parameter to build_hook_context.

I think an ephemeral graph isnt so bad - honestly could simplify how we make the Solid too

Jul 16 2021, 6:27 PM
alangenfeld closed D8926: PipelineDefinition.all_solid_defs -> all_node_defs.
Jul 16 2021, 5:51 PM
alangenfeld committed R1:68e2c371d617: PipelineDefinition.all_solid_defs -> all_node_defs (authored by alangenfeld).
PipelineDefinition.all_solid_defs -> all_node_defs
Jul 16 2021, 5:51 PM
alangenfeld requested review of D8926: PipelineDefinition.all_solid_defs -> all_node_defs.
Jul 16 2021, 5:26 PM
alangenfeld closed D8911: bugfix: catch more dynamic output restrictions with graphs.
Jul 16 2021, 5:02 PM
alangenfeld committed R1:dddd2d2699a2: bugfix: catch more dynamic output restrictions with graphs (authored by alangenfeld).
bugfix: catch more dynamic output restrictions with graphs
Jul 16 2021, 5:02 PM
alangenfeld closed D8895: GraphDefinition.solid_dict -> node_dict.
Jul 16 2021, 4:54 PM
alangenfeld committed R1:56d3644f5c06: GraphDefinition.solid_dict -> node_dict (authored by alangenfeld).
GraphDefinition.solid_dict -> node_dict
Jul 16 2021, 4:54 PM
alangenfeld closed D8715: Solid -> Node.
Jul 16 2021, 4:54 PM
alangenfeld committed R1:5c4286f60822: Solid -> Node (authored by alangenfeld).
Solid -> Node
Jul 16 2021, 4:54 PM
alangenfeld closed D8714: SolidHandle -> NodeHandle.
Jul 16 2021, 4:53 PM
alangenfeld committed R1:7de455e40c62: SolidHandle -> NodeHandle (authored by alangenfeld).
SolidHandle -> NodeHandle
Jul 16 2021, 4:53 PM
alangenfeld added inline comments to D8903: Add solid as parameter to build_hook_context.
Jul 16 2021, 3:43 PM

Jul 15 2021

alangenfeld requested review of D8911: bugfix: catch more dynamic output restrictions with graphs.
Jul 15 2021, 9:27 PM
alangenfeld added inline comments to D8840: convert local compute log manager.
Jul 15 2021, 4:37 PM
alangenfeld added inline comments to D8714: SolidHandle -> NodeHandle.
Jul 15 2021, 3:51 PM

Jul 14 2021

alangenfeld added inline comments to D8895: GraphDefinition.solid_dict -> node_dict.
Jul 14 2021, 9:42 PM
alangenfeld requested review of D8895: GraphDefinition.solid_dict -> node_dict.
Jul 14 2021, 9:32 PM
alangenfeld requested review of D8898: [mypy] update check.inst to not blank out types.
Jul 14 2021, 9:05 PM
alangenfeld added a comment to D8714: SolidHandle -> NodeHandle.

^ last failure passed on retry

Jul 14 2021, 8:44 PM
alangenfeld requested review of D8715: Solid -> Node.
Jul 14 2021, 5:32 PM
alangenfeld updated the summary of D8714: SolidHandle -> NodeHandle.
Jul 14 2021, 4:51 PM
alangenfeld updated the diff for D8714: SolidHandle -> NodeHandle.

fix test

Jul 14 2021, 4:40 PM
alangenfeld updated the diff for D8714: SolidHandle -> NodeHandle.

add desc

Jul 14 2021, 4:32 PM
alangenfeld updated the diff for D8714: SolidHandle -> NodeHandle.

rebase

Jul 14 2021, 4:04 PM
alangenfeld closed D8869: mypy DagsterInstance.
Jul 14 2021, 3:58 PM
alangenfeld committed R1:3a8709b2e71f: mypy DagsterInstance (authored by alangenfeld).
mypy DagsterInstance
Jul 14 2021, 3:57 PM
alangenfeld closed D8876: mention positionals in Nothing example.
Jul 14 2021, 3:52 PM
alangenfeld committed R1:25e3dbf9afcf: mention positionals in Nothing example (authored by alangenfeld).
mention positionals in Nothing example
Jul 14 2021, 3:52 PM

Jul 13 2021

alangenfeld requested review of D8876: mention positionals in Nothing example.
Jul 13 2021, 11:03 PM
alangenfeld requested changes to D8385: add captured log manager interface.

to your queue for bunch of thoughts

Jul 13 2021, 10:25 PM
alangenfeld requested review of D8869: mypy DagsterInstance.
Jul 13 2021, 9:41 PM
alangenfeld requested changes to D8550: [op] permit op config entry.

I am pretty sure this current approach isn't going to work. We may need to add custom support to the config system to handle this flexibility only at the nodes in the config tree where we should

Jul 13 2021, 8:33 PM
alangenfeld accepted D8861: Fix coroutine bug for solid invocation.

behold

Jul 13 2021, 8:26 PM
alangenfeld added inline comments to D8840: convert local compute log manager.
Jul 13 2021, 8:08 PM
alangenfeld added inline comments to D8845: have AzureComputeLogManager implement CapturedLogManager interface.
Jul 13 2021, 8:04 PM
alangenfeld added inline comments to D8840: convert local compute log manager.
Jul 13 2021, 8:02 PM
alangenfeld added inline comments to D8385: add captured log manager interface.
Jul 13 2021, 7:58 PM
alangenfeld requested review of D8846: [dag-star] download debug file.
Jul 13 2021, 6:04 PM
alangenfeld requested review of D8810: [dag-star] graphql websocket endpoint.
Jul 13 2021, 6:02 PM
alangenfeld added a reverting change for D8800: remove storage accessors from instance: R1:daa67efb7169: Revert "remove storage accessors from instance".
Jul 13 2021, 5:49 PM
alangenfeld added a reverting change for R1:0e5e3563eb9a: remove storage accessors from instance: R1:daa67efb7169: Revert "remove storage accessors from instance".
Jul 13 2021, 5:49 PM
alangenfeld committed R1:daa67efb7169: Revert "remove storage accessors from instance" (authored by alangenfeld).
Revert "remove storage accessors from instance"
Jul 13 2021, 5:49 PM
alangenfeld requested changes to D8678: [RFC] time-based partitioned config and schedule from partitioned config.

^

Jul 13 2021, 5:43 PM