Page MenuHomePhabricator

alangenfeld (Alex Langenfeld)
UserAdministrator

Projects

User does not belong to any projects.

User Details

User Since
Mar 20 2019, 8:25 PM (44 w, 1 d)
Roles
Administrator

Recent Activity

Today

alangenfeld added a comment to D1863: [engine] shared step iteration.

Exposing a configurable priority key is actually pretty risky and will result in a lot of replicated config.

Thu, Jan 23, 9:18 PM
alangenfeld added a comment to D1863: [engine] shared step iteration.

So the crux of it is:

  • we do dagster/priority and it works for the core executors but (probably?) none of the others which leads to an unfortunate expectation mis-match in a not clearly communicated way (keys just ignored)
  • we do separate keys for each executor to align expectations ie dagster/in-process/priority, dagster-in-process/priority, whatever but thats aesthetically unfortunate and leads to having to set all keys to support different execution modes
Thu, Jan 23, 9:15 PM
alangenfeld added a comment to D1863: [engine] shared step iteration.

what should the predefined keys be for the in process and multi process engines? I wasn't happy with what I came up with which motivated me to punt it out to config

Thu, Jan 23, 9:09 PM
alangenfeld planned changes to D1853: handle differing arg and input def ordering.
Thu, Jan 23, 8:39 PM
alangenfeld updated the diff for D1863: [engine] shared step iteration.

move sorting out to individual engines. use config to drive what key to use for prioirty on default engines

Thu, Jan 23, 8:18 PM
alangenfeld resigned from D1874: Bay Bikes Weather Pipeline refactor.
Thu, Jan 23, 6:04 PM
alangenfeld committed R1:ddcfd23617b6: Fix issue with pyspark_solid decorator not handling inputs (authored by nate).
Fix issue with pyspark_solid decorator not handling inputs
Thu, Jan 23, 5:39 PM
alangenfeld closed D1765: Fix issue with pyspark_solid decorator not handling inputs.
Thu, Jan 23, 5:38 PM
alangenfeld accepted D1873: Working K8sRunLauncher.
Thu, Jan 23, 5:38 PM
alangenfeld added a comment to D1873: Working K8sRunLauncher.

changes here all seem reasonable to me

Thu, Jan 23, 5:38 PM
alangenfeld added inline comments to D1873: Working K8sRunLauncher.
Thu, Jan 23, 5:38 PM
alangenfeld accepted D1871: Combine start/launch buttons into a toggleable execute button group.

I think this is better

Thu, Jan 23, 5:22 PM
alangenfeld updated the diff for D1765: Fix issue with pyspark_solid decorator not handling inputs.

rebase

Thu, Jan 23, 5:19 PM
alangenfeld added a comment to D1864: RFC: is_optional --> is_required.

dont have strong feelings - but the complexity cost of warn here feels pretty minimal so i maybe lean that way

Thu, Jan 23, 5:10 PM
alangenfeld added a comment to D1864: RFC: is_optional --> is_required.

can you explain automated comment. i think i am missing something

Thu, Jan 23, 4:54 PM

Yesterday

alangenfeld accepted D1869: Upgrade yarn.

whynot

Wed, Jan 22, 11:30 PM
alangenfeld updated the diff for D1868: [buildkite] use snapshot for caching in test_project.

different approach

Wed, Jan 22, 11:21 PM
alangenfeld accepted D1857: BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores.
Wed, Jan 22, 11:09 PM
alangenfeld created D1868: [buildkite] use snapshot for caching in test_project.
Wed, Jan 22, 11:04 PM
alangenfeld added a comment to D1864: RFC: is_optional --> is_required.

flipping the bool does make the change not quite automated - not sure where that nets out on hard break vs deprecate

Wed, Jan 22, 10:30 PM
alangenfeld accepted D1861: add collapsed tags view in RunTable.

personal opinion: the blue color is a bit agro i would try to tone it down a bit

Wed, Jan 22, 10:25 PM
alangenfeld updated the diff for D1853: handle differing arg and input def ordering.

revert change to other code

Wed, Jan 22, 10:24 PM
alangenfeld updated the diff for D1853: handle differing arg and input def ordering.

fix old code and add comments to tests

Wed, Jan 22, 10:12 PM
alangenfeld committed R1:442669519f97: dont use integration as base for test_pipeline (authored by alangenfeld).
dont use integration as base for test_pipeline
Wed, Jan 22, 9:50 PM
alangenfeld closed D1859: dont use integration as base for test_pipeline.
Wed, Jan 22, 9:50 PM
alangenfeld updated the diff for D1859: dont use integration as base for test_pipeline.

different approach to handle mock issue

Wed, Jan 22, 9:36 PM
alangenfeld added inline comments to D1866: Move dagster_graphql_test context utils to dagster_graphql.
Wed, Jan 22, 9:26 PM
alangenfeld added inline comments to D1865: Fix example schedules.
Wed, Jan 22, 9:26 PM
alangenfeld accepted D1866: Move dagster_graphql_test context utils to dagster_graphql.

illallowit

Wed, Jan 22, 9:23 PM
alangenfeld added inline comments to D1857: BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores.
Wed, Jan 22, 8:20 PM
alangenfeld accepted D1860: Basic composite expansion in the Dagit explore tab.

I'm cool with this if 'handle-like' names are actually the same as the handles - if not it would be good to get an explanation of what exactly is going on there

Wed, Jan 22, 8:13 PM
alangenfeld added inline comments to D1860: Basic composite expansion in the Dagit explore tab.
Wed, Jan 22, 8:12 PM
alangenfeld requested changes to D1857: BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores.
Wed, Jan 22, 8:02 PM
alangenfeld committed R1:fbb1b47698cc: [nightly] re-align nightly versions (authored by alangenfeld).
[nightly] re-align nightly versions
Wed, Jan 22, 7:38 PM
alangenfeld closed D1862: [nightly] re-align nightly versions.
Wed, Jan 22, 7:38 PM
alangenfeld updated the diff for D1863: [engine] shared step iteration.

more ordered dict

Wed, Jan 22, 7:34 PM
alangenfeld updated the diff for D1863: [engine] shared step iteration.

cant repro locally add some logging for BK

Wed, Jan 22, 7:10 PM
alangenfeld updated the diff for D1863: [engine] shared step iteration.

yaytests

Wed, Jan 22, 6:52 PM
alangenfeld added a comment to D1865: Fix example schedules.

proper way to test this? This error only comes up at either schedule page load time or schedule execution time because that's when we run environment_dict_fn.

Wed, Jan 22, 6:48 PM
alangenfeld retitled D1863: [engine] shared step iteration from [engine] introduce ActivePlan to [engine] shared step iteration.
Wed, Jan 22, 6:47 PM
alangenfeld updated the diff for D1863: [engine] shared step iteration.

py2 fixes?

Wed, Jan 22, 6:37 PM
alangenfeld updated the summary of D1863: [engine] shared step iteration.
Wed, Jan 22, 6:12 PM
alangenfeld updated the diff for D1863: [engine] shared step iteration.

priority test

Wed, Jan 22, 6:10 PM
alangenfeld updated the diff for D1863: [engine] shared step iteration.

ordered dict

Wed, Jan 22, 5:37 PM
alangenfeld created D1863: [engine] shared step iteration.
Wed, Jan 22, 5:12 PM
alangenfeld created D1862: [nightly] re-align nightly versions.
Wed, Jan 22, 4:38 PM

Tue, Jan 21

alangenfeld updated the diff for D1859: dont use integration as base for test_pipeline.

set mock to None when not found

Tue, Jan 21, 10:06 PM
alangenfeld accepted D1855: Gaant-style execution plan visualization behind feature flag.

makeitso

Tue, Jan 21, 9:58 PM
alangenfeld updated the summary of D1859: dont use integration as base for test_pipeline.
Tue, Jan 21, 9:55 PM
alangenfeld committed R1:20c0028bfd2b: [codemod] yaml.load -> yaml.safe_load (authored by alangenfeld).
[codemod] yaml.load -> yaml.safe_load
Tue, Jan 21, 9:51 PM
alangenfeld closed D1856: [codemod] yaml.load -> yaml.safe_load.
Tue, Jan 21, 9:51 PM
alangenfeld updated the diff for D1859: dont use integration as base for test_pipeline.

does the non-slim have gcc?

Tue, Jan 21, 9:48 PM
alangenfeld added inline comments to D1856: [codemod] yaml.load -> yaml.safe_load.
Tue, Jan 21, 9:41 PM
alangenfeld created D1859: dont use integration as base for test_pipeline.
Tue, Jan 21, 9:34 PM
alangenfeld updated the diff for D1856: [codemod] yaml.load -> yaml.safe_load.

apply workaround

Tue, Jan 21, 9:32 PM
alangenfeld requested changes to D1857: BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores.
Tue, Jan 21, 9:28 PM
alangenfeld updated the diff for D1856: [codemod] yaml.load -> yaml.safe_load.

more debug

Tue, Jan 21, 5:05 PM
alangenfeld updated the diff for D1856: [codemod] yaml.load -> yaml.safe_load.

back to debugging

Tue, Jan 21, 4:29 PM
alangenfeld updated the diff for D1856: [codemod] yaml.load -> yaml.safe_load.

remove debug

Tue, Jan 21, 3:44 PM
alangenfeld added a comment to D1856: [codemod] yaml.load -> yaml.safe_load.

seems fine, although isn't this a slightly different fix than the warning text indicates?

Tue, Jan 21, 3:42 PM

Sat, Jan 18

alangenfeld updated the diff for D1856: [codemod] yaml.load -> yaml.safe_load.

canary

Sat, Jan 18, 2:32 AM
alangenfeld committed R1:30f95699bb68: [k8s] dont hardcode service account name (authored by alangenfeld).
[k8s] dont hardcode service account name
Sat, Jan 18, 2:22 AM
alangenfeld closed D1858: [k8s] dont hardcode service account name.
Sat, Jan 18, 2:22 AM
alangenfeld updated the diff for D1858: [k8s] dont hardcode service account name.

set service account name

Sat, Jan 18, 2:09 AM
alangenfeld added a comment to D1857: BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores.

I would just look through the other tests in dagster_graphql - python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_run_cancellation.py is probably a relevant example

Sat, Jan 18, 1:31 AM
alangenfeld updated the diff for D1858: [k8s] dont hardcode service account name.

god damn yaml templates

Sat, Jan 18, 12:30 AM
alangenfeld updated the diff for D1858: [k8s] dont hardcode service account name.

rebase

Sat, Jan 18, 12:20 AM
alangenfeld added inline comments to D1857: BUGFIX: Ensure SubprocessExecutionManager isn't leaking POSIX semaphores.
Sat, Jan 18, 12:19 AM

Fri, Jan 17

alangenfeld added inline comments to D1849: Improve local test story.
Fri, Jan 17, 11:45 PM
alangenfeld created D1858: [k8s] dont hardcode service account name.
Fri, Jan 17, 11:14 PM
alangenfeld created D1856: [codemod] yaml.load -> yaml.safe_load.
Fri, Jan 17, 8:17 PM
alangenfeld added a comment to D1855: Gaant-style execution plan visualization behind feature flag.

just spitballing ideas one thing we could do is keep the box pinned but to the left but change where the shading starts? What i want visually communicated when these things are waiting on eachother for not-data reasons is:

Fri, Jan 17, 8:04 PM
alangenfeld added a comment to D1855: Gaant-style execution plan visualization behind feature flag.

from use - if you execute the sleepy toy pipeline with the default config which uses inprocess engine - the four parallel steps happen sequentially which is not communicated in the waterfall timed view at all since we pin no the left alignment. Is this something worth addressing now or in a subsequent diff?

Fri, Jan 17, 7:35 PM
alangenfeld added inline comments to D1855: Gaant-style execution plan visualization behind feature flag.
Fri, Jan 17, 6:27 PM

Thu, Jan 16

alangenfeld accepted D1832: RFC: Update TypeStoragePluginRegistry to take list of tuples instead of dict.

this seems fine, double check if you need to add the check stuff

Thu, Jan 16, 10:53 PM
alangenfeld committed R1:64eb38facc20: remove deprecated ReexecutionConfig (authored by alangenfeld).
remove deprecated ReexecutionConfig
Thu, Jan 16, 10:02 PM
alangenfeld closed D1852: remove deprecated ReexecutionConfig.
Thu, Jan 16, 10:02 PM
alangenfeld closed D1847: [dagit] handle engine event errors.
Thu, Jan 16, 9:56 PM
alangenfeld committed R1:826f50a6639e: [dagit] handle engine event errors (authored by alangenfeld).
[dagit] handle engine event errors
Thu, Jan 16, 9:56 PM
alangenfeld committed R1:53f858f49b85: [celery] emit engine event on failed broker connection (authored by alangenfeld).
[celery] emit engine event on failed broker connection
Thu, Jan 16, 9:56 PM
alangenfeld closed D1846: [celery] emit engine event on failed broker connection.
Thu, Jan 16, 9:56 PM
alangenfeld updated the diff for D1853: handle differing arg and input def ordering.

special case singurlar input_def

Thu, Jan 16, 9:47 PM
alangenfeld planned changes to D1853: handle differing arg and input def ordering.
Thu, Jan 16, 6:50 PM
alangenfeld added a reviewer for D1765: Fix issue with pyspark_solid decorator not handling inputs: max.
Thu, Jan 16, 6:46 PM
alangenfeld updated the diff for D1765: Fix issue with pyspark_solid decorator not handling inputs.

rebaesw

Thu, Jan 16, 6:46 PM
alangenfeld commandeered D1765: Fix issue with pyspark_solid decorator not handling inputs.
Thu, Jan 16, 6:45 PM
alangenfeld created D1853: handle differing arg and input def ordering.
Thu, Jan 16, 6:33 PM
alangenfeld updated the diff for D1852: remove deprecated ReexecutionConfig.

rm print()

Thu, Jan 16, 5:49 PM
alangenfeld updated the diff for D1852: remove deprecated ReexecutionConfig.

update serdes and seven to better handle py2

Thu, Jan 16, 5:38 PM
alangenfeld accepted D1849: Improve local test story.

nice - just make sure to cleanup unused stuff

Thu, Jan 16, 5:08 PM
alangenfeld created D1852: remove deprecated ReexecutionConfig.
Thu, Jan 16, 4:29 PM
alangenfeld updated the diff for D1847: [dagit] handle engine event errors.

rebase + comment

Thu, Jan 16, 4:17 PM

Wed, Jan 15

alangenfeld added inline comments to D1844: RFC: Eliminate ConfigMappingContext.
Wed, Jan 15, 10:19 PM
alangenfeld added a comment to D1844: RFC: Eliminate ConfigMappingContext.

it seems like something bad is happening in changed.md thats causing the autoformatter to go nuts

Wed, Jan 15, 10:10 PM
alangenfeld added inline comments to D1846: [celery] emit engine event on failed broker connection.
Wed, Jan 15, 10:05 PM
alangenfeld created D1847: [dagit] handle engine event errors.
Wed, Jan 15, 10:01 PM
alangenfeld added a comment to D1846: [celery] emit engine event on failed broker connection.

Wed, Jan 15, 10:01 PM
alangenfeld created D1846: [celery] emit engine event on failed broker connection.
Wed, Jan 15, 9:59 PM
alangenfeld committed R1:e9e17639631b: [core] rework engine requirments checks (authored by alangenfeld).
[core] rework engine requirments checks
Wed, Jan 15, 4:05 PM
alangenfeld closed D1842: [core] rework engine requirments checks.
Wed, Jan 15, 4:05 PM