Page MenuHomeElementl

alangenfeld (Alex Langenfeld)
UserAdministrator

Projects

User does not belong to any projects.

User Details

User Since
Mar 20 2019, 8:25 PM (130 w, 4 d)
Roles
Administrator

Recent Activity

Tue, Aug 24

alangenfeld accepted D9221: update type annotations on context properties.
Tue, Aug 24, 4:42 PM
alangenfeld accepted D9224: Add docstring and type annotation to merger.merge_dicts.
Tue, Aug 24, 4:15 PM
alangenfeld accepted D9225: better error when wrong type supplied in deps dictionary.
Tue, Aug 24, 4:13 PM

Aug 19 2021

alangenfeld added reviewers for D9220: [Memoization 12/13] Enable memoization on all out-of-the-box executors: dgibson, johann.
Aug 19 2021, 5:18 PM

Aug 12 2021

alangenfeld added a reverting change for R1:7636c9a1ee1d: dagster PEP 561 - expose mypy typing (#4504): R1:271d1fd70c9e: Revert "dagster PEP 561 - expose mypy typing (#4504)".
Aug 12 2021, 9:27 PM
alangenfeld committed R1:271d1fd70c9e: Revert "dagster PEP 561 - expose mypy typing (#4504)" (authored by alangenfeld).
Revert "dagster PEP 561 - expose mypy typing (#4504)"
Aug 12 2021, 9:27 PM
alangenfeld committed R1:c89fab337e03: Update azure-pipelines.yml for Azure Pipelines (authored by alangenfeld).
Update azure-pipelines.yml for Azure Pipelines
Aug 12 2021, 9:13 PM
alangenfeld requested changes to D9220: [Memoization 12/13] Enable memoization on all out-of-the-box executors.
Aug 12 2021, 7:46 PM
alangenfeld added a comment to D9220: [Memoization 12/13] Enable memoization on all out-of-the-box executors.

what about all the other executors? Just fixing it for multiprocess seems wrong

Aug 12 2021, 7:43 PM
alangenfeld accepted D9218: [Memoization 11/13] Make resource versions optional, even when resources are provided..

I think the asymettery between solids and resources and their memoization behavior is unfortunate, but as long as its clearly communicated it should be fine

Aug 12 2021, 7:29 PM
alangenfeld accepted D9139: [mypy] definitions/events.py.
Aug 12 2021, 3:05 PM

Aug 11 2021

alangenfeld added inline comments to D9219: [memoization 13/13] revamp memoization docs.
Aug 11 2021, 4:01 PM

Aug 9 2021

alangenfeld added inline comments to D9085: [memoization 8/n] make default fs io manager memoizable.
Aug 9 2021, 7:31 PM

Aug 4 2021

alangenfeld accepted D9195: add composite solids to graph/job/op migration guide.

limecat

Aug 4 2021, 7:15 PM
alangenfeld added a comment to D9202: [dagit] Lazy-load partition set info for Schedules.

I think Bens "Show Coverage" suggestion is better the "load" ones

Aug 4 2021, 3:50 PM
alangenfeld accepted D9201: [crag] add GraphDefinition to the public API.

whynot

Aug 4 2021, 3:00 PM
alangenfeld added a comment to D9202: [dagit] Lazy-load partition set info for Schedules.

we can build and show the link to the partitions page with [1] right?

Aug 4 2021, 3:00 PM
alangenfeld requested changes to D9195: add composite solids to graph/job/op migration guide.

I think its very valuable for this example to include input & output mapping

Aug 4 2021, 2:45 PM

Aug 3 2021

alangenfeld accepted D9191: test execute_in_process with aliasing.
Aug 3 2021, 3:01 PM
alangenfeld accepted D9190: [crag] Add validation step to default config buildup for to_job..
Aug 3 2021, 2:56 PM
alangenfeld accepted D9189: [crag] Ignore executor config when using execute_in_process.
Aug 3 2021, 2:52 PM

Aug 2 2021

alangenfeld accepted D9181: allow step failure events to have no serialized error.
Aug 2 2021, 5:02 PM
alangenfeld added inline comments to D9085: [memoization 8/n] make default fs io manager memoizable.
Aug 2 2021, 3:55 PM
alangenfeld added a comment to D9181: allow step failure events to have no serialized error.

make sure we update the mypy type on StepFailureData which is also not set as Optional

Aug 2 2021, 2:46 PM
alangenfeld added a comment to D9181: allow step failure events to have no serialized error.

what condition did we add where we have a step failure with no exception?

Aug 2 2021, 2:36 PM
alangenfeld added a comment to D4698: Prototype: Add test utility for dagstermill solids.

My initial reaction is that this is a pretty small improvement for a very specialized utility function. Its hard for me to make a strong case that this is any better than reconstructable in terms of having to discover some special tool when the normal testing flow fails on a dagstermill solid.

Aug 2 2021, 2:21 PM
alangenfeld accepted D9171: [crag] allow non-string tag values in to_job.

add tests for valid non-str tag and invalid non-str tag

Aug 2 2021, 2:01 PM
alangenfeld accepted D9175: Carry over field aliases to default config for job.
Aug 2 2021, 1:59 PM

Jul 29 2021

alangenfeld added inline comments to D9061: [memoization 4/n][rfc] memoization strategy for jobs.
Jul 29 2021, 7:38 PM
alangenfeld added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

the inline comments for the tags bit is still an open area of concern

Jul 29 2021, 7:29 PM
alangenfeld closed D9101: move dagster execution_tests in to seperate tox target.
Jul 29 2021, 6:05 PM
alangenfeld committed R1:97a29c62c546: move dagster execution_tests in to seperate tox target (authored by alangenfeld).
move dagster execution_tests in to seperate tox target
Jul 29 2021, 6:05 PM
alangenfeld added inline comments to D9061: [memoization 4/n][rfc] memoization strategy for jobs.
Jul 29 2021, 5:22 PM
alangenfeld added inline comments to D9061: [memoization 4/n][rfc] memoization strategy for jobs.
Jul 29 2021, 4:16 PM
alangenfeld accepted D9141: Add custom serializer to config type snapshot.

^ above comments worth considering

Jul 29 2021, 2:52 PM

Jul 28 2021

alangenfeld accepted D9130: [mypy] typing.NamedTuple for dependency.py.

thumbsup

Jul 28 2021, 7:26 PM
alangenfeld requested review of D9101: move dagster execution_tests in to seperate tox target.
Jul 28 2021, 6:27 PM
alangenfeld accepted D9110: dagstermill - yield_result output path account for aliased solids and composite solids.

agh - should have caught this in review

Jul 28 2021, 5:58 PM
alangenfeld accepted D9114: [mypy] leftover definition_config_schema.
Jul 28 2021, 5:50 PM
alangenfeld accepted D9121: Smarter dbt default argument handling.

magmar

Jul 28 2021, 5:50 PM
alangenfeld accepted D8550: [op] permit op config entry.

push_n_pray

Jul 28 2021, 5:47 PM
alangenfeld closed D9094: [RFC][grpc] turn on gzip and set higher receive limit.
Jul 28 2021, 5:16 PM
alangenfeld committed R1:c52b1d50af57: [RFC][grpc] turn on gzip and set higher receive limit (authored by alangenfeld).
[RFC][grpc] turn on gzip and set higher receive limit
Jul 28 2021, 5:16 PM
alangenfeld added a comment to D9061: [memoization 4/n][rfc] memoization strategy for jobs.

This strategy only really works for solids. A follow up diff will address the resources situation. Not sure if the best solution there is to keep the version argument on the resource decorator and make it not required to use memoization, or to add a resource_versioning_strategy argument (also not required) as well.

Jul 28 2021, 3:58 PM
alangenfeld updated the diff for D9094: [RFC][grpc] turn on gzip and set higher receive limit.

add env, test via 'DAGSTER_GRPC_MAX_RX_BYTES=5000000 dagit --grpc-port 5000'

Jul 28 2021, 3:46 PM

Jul 27 2021

alangenfeld added inline comments to D9094: [RFC][grpc] turn on gzip and set higher receive limit.
Jul 27 2021, 9:37 PM
alangenfeld added a comment to D9094: [RFC][grpc] turn on gzip and set higher receive limit.

this looks good, but is back-compat a concern? if you upgrade dagit and don't upgrade the server will things break?

Jul 27 2021, 9:36 PM
alangenfeld requested review of D9094: [RFC][grpc] turn on gzip and set higher receive limit.
Jul 27 2021, 9:26 PM
alangenfeld closed D8846: [dag-star] download debug file.
Jul 27 2021, 9:15 PM
alangenfeld committed R1:1d8379e84c87: [dag-star] download debug file (authored by alangenfeld).
[dag-star] download debug file
Jul 27 2021, 9:15 PM
alangenfeld closed D8810: [dag-star] graphql websocket endpoint.
Jul 27 2021, 9:14 PM
alangenfeld committed R1:8de70acb42bc: [dag-star] graphql websocket endpoint (authored by alangenfeld).
[dag-star] graphql websocket endpoint
Jul 27 2021, 9:14 PM
alangenfeld closed D9082: update stress tests for grpc limit repro.
Jul 27 2021, 5:36 PM
alangenfeld committed R1:badd30dc5735: update stress tests for grpc limit repro (authored by alangenfeld).
update stress tests for grpc limit repro
Jul 27 2021, 5:36 PM
alangenfeld requested review of D9082: update stress tests for grpc limit repro.
Jul 27 2021, 4:29 PM
alangenfeld accepted D9053: dagstermill - fix: yield_result temp path includes solid name.

looking forward to a more thorough revamp of this stuff

Jul 27 2021, 4:13 PM

Jul 26 2021

alangenfeld closed D8808: [dag-star] graphql endpoint.
Jul 26 2021, 9:50 PM
alangenfeld committed R1:835fafac3195: [dag-star] graphql endpoint (authored by alangenfeld).
[dag-star] graphql endpoint
Jul 26 2021, 9:50 PM
alangenfeld closed D8809: [dag-star] index html and static resources.
Jul 26 2021, 9:50 PM
alangenfeld committed R1:9cc482ca6ebd: [dag-star] index html and static resources (authored by alangenfeld).
[dag-star] index html and static resources
Jul 26 2021, 9:50 PM
alangenfeld closed D8807: [dag-star] initial starlette app for dagit.
Jul 26 2021, 9:50 PM
alangenfeld committed R1:097308c71ae0: [dag-star] initial starlette app for dagit (authored by alangenfeld).
[dag-star] initial starlette app for dagit
Jul 26 2021, 9:50 PM
alangenfeld added a reviewer for D8810: [dag-star] graphql websocket endpoint: dgibson.
Jul 26 2021, 8:49 PM
alangenfeld added inline comments to D9045: [crag] Add fields not brought about in default config, fix bug with execute_in_process and default config, catch bad config earlier on ..
Jul 26 2021, 8:38 PM
alangenfeld added inline comments to D9059: [memoization 2/n] provide default base dir for versioned fs io manager.
Jul 26 2021, 5:58 PM
alangenfeld accepted D9057: dagit default app handler tweaks.
Jul 26 2021, 5:04 PM
alangenfeld added a comment to D8550: [op] permit op config entry.

Additionally, having trouble getting the snapshot tests to cooperate. pytest --snapshot-update does not seem to be doing the trick.

Jul 26 2021, 4:55 PM
alangenfeld accepted D9046: Use grpc rather than ipc to decide if a grpc server is running.

the broad except + assumptions spooks me a bit, so at least make sure the assumptions are clearly commented in-case someone bumps in to this debugging

Jul 26 2021, 4:27 PM
alangenfeld added a comment to D9046: Use grpc rather than ipc to decide if a grpc server is running.

ah ok so it is the module load - the error condition you describe makes sense then but definitely worth commenting in [1] that we are assuming any exception that occurs is coming from the target code and due to python module loading behavior we capture, keep, and always re-throw that original error.

Jul 26 2021, 4:25 PM
alangenfeld resigned from D8969: [caprisun] @asset and build_assets_job.

current recommendation is that, in their IOManager, users switch on the Python type attached to the InputDefinition

Jul 26 2021, 4:13 PM
alangenfeld added inline comments to D9046: Use grpc rather than ipc to decide if a grpc server is running.
Jul 26 2021, 3:50 PM

Jul 23 2021

alangenfeld accepted D9041: Load execution plan from the snapshot on the run when resume retrying.

coolbeans

Jul 23 2021, 6:53 PM
alangenfeld requested changes to D9045: [crag] Add fields not brought about in default config, fix bug with execute_in_process and default config, catch bad config earlier on ..
Jul 23 2021, 6:36 PM
alangenfeld added inline comments to D9045: [crag] Add fields not brought about in default config, fix bug with execute_in_process and default config, catch bad config earlier on ..
Jul 23 2021, 6:36 PM
alangenfeld updated the diff for D8810: [dag-star] graphql websocket endpoint.

rebase, review feedback

Jul 23 2021, 6:21 PM
alangenfeld added inline comments to D8810: [dag-star] graphql websocket endpoint.
Jul 23 2021, 5:24 PM
alangenfeld accepted D9043: handle wiped assets for asset index migration.

arcanine

Jul 23 2021, 5:07 PM
alangenfeld closed D9033: rm dagster dep from automation.
Jul 23 2021, 5:06 PM
alangenfeld committed R1:eed9cea7b1e3: rm dagster dep from automation (authored by alangenfeld).
rm dagster dep from automation
Jul 23 2021, 5:06 PM
alangenfeld accepted D9038: feat(helm): set compute log manager default as noop.

ya this is the right call

Jul 23 2021, 4:22 PM
alangenfeld requested changes to D8684: style: adopt pre-commit for Python black formatting.

Well given that installing the pre-commit hooks is a choice, I think its safe to add the config file that meets our projects standards to ease that process for those that want it.

Jul 23 2021, 3:25 PM

Jul 22 2021

alangenfeld accepted D8145: Quieten experimental warnings on example pipeline.

vileplume

Jul 22 2021, 9:05 PM
alangenfeld requested changes to D8550: [op] permit op config entry.

round trip since there are failures any way - I think these last set of test adjustments should be it

Jul 22 2021, 8:44 PM
alangenfeld added inline comments to D8550: [op] permit op config entry.
Jul 22 2021, 8:44 PM
alangenfeld added a comment to D9027: limit azure runs to main branch.

welp - this wasnt quite right. Followed up with
https://github.com/dagster-io/dagster/commit/66a185240a7b7137d8fb97beaabd3b3d93ae9b4e
where i just disabled PRs as well for now

Jul 22 2021, 8:11 PM
alangenfeld committed R1:66a185240a7b: Update azure-pipelines.yml for Azure Pipelines (authored by alangenfeld).
Update azure-pipelines.yml for Azure Pipelines
Jul 22 2021, 8:11 PM
alangenfeld closed D9027: limit azure runs to main branch.
Jul 22 2021, 8:09 PM
alangenfeld committed R1:89d8baa79988: limit azure runs to main branch (authored by alangenfeld).
limit azure runs to main branch
Jul 22 2021, 8:09 PM
alangenfeld added a comment to D9027: limit azure runs to main branch.

PRs might make that noisy-ish still but we may have to disable those as well

Jul 22 2021, 8:08 PM
alangenfeld requested review of D9027: limit azure runs to main branch.
Jul 22 2021, 7:59 PM
alangenfeld closed D9014: support get_mapping_key in direct invocation.
Jul 22 2021, 6:27 PM
alangenfeld committed R1:e95742d174df: support get_mapping_key in direct invocation (authored by alangenfeld).
support get_mapping_key in direct invocation
Jul 22 2021, 6:27 PM
alangenfeld requested review of D9014: support get_mapping_key in direct invocation.
Jul 22 2021, 6:21 PM
alangenfeld closed D8942: [crag] Job description.
Jul 22 2021, 4:46 PM
alangenfeld committed R1:cf74b1eaf237: [crag] Job description (authored by alangenfeld).
[crag] Job description
Jul 22 2021, 4:46 PM
alangenfeld closed D8943: [crag] fix snapshots handling composed @graph.
Jul 22 2021, 4:46 PM
alangenfeld committed R1:6af4c12ea15e: [crag] fix snapshots handling composed @graph (authored by alangenfeld).
[crag] fix snapshots handling composed @graph
Jul 22 2021, 4:45 PM
alangenfeld accepted D8920: [memoization improvements 1/n] get rid of step_keys_to_execute parameter on create_run_for_pipeline, simplify implementation of create_run_for_pipeline..

thanks! very clean change now

Jul 22 2021, 4:33 PM
alangenfeld added a comment to D8969: [caprisun] @asset and build_assets_job.

Everything thats here seems reasonable from my perspective

Jul 22 2021, 3:37 PM
alangenfeld updated the diff for D8943: [crag] fix snapshots handling composed @graph.

rebase

Jul 22 2021, 3:14 PM