Page MenuHomePhabricator
Feed Advanced Search

Dec 6 2019

schrockn updated the diff for D1555: Use hashed names for all system dictionaries and selectors.

upmessage

Dec 6 2019, 6:05 PM
schrockn updated the diff for D1555: Use hashed names for all system dictionaries and selectors.

up

Dec 6 2019, 5:50 PM
schrockn updated the diff for D1555: Use hashed names for all system dictionaries and selectors.

up

Dec 6 2019, 5:42 PM
schrockn created D1555: Use hashed names for all system dictionaries and selectors.
Dec 6 2019, 5:19 PM
schrockn accepted D1553: [tutorial] fix resources.py.
Dec 6 2019, 4:53 PM
schrockn accepted D1549: [dagit] Fix wrapping in sidebar, disable reflow of fan-out.
Dec 6 2019, 3:46 PM
schrockn updated the summary of D1549: [dagit] Fix wrapping in sidebar, disable reflow of fan-out.
Dec 6 2019, 3:39 PM
schrockn accepted D1550: Replace current two-step zoom with smooth zoom.

nice

Dec 6 2019, 1:26 AM
schrockn accepted D1551: [core] remove redundant step level calls.
Dec 6 2019, 1:13 AM
schrockn added a comment to D1549: [dagit] Fix wrapping in sidebar, disable reflow of fan-out.

My only concern is that this is too tailored towards our current examples and will have an adverse effect on dags where there is no correlation between naming and distance

Dec 6 2019, 1:11 AM
schrockn added a comment to D1542: Considerably reduce volume of type data fetched by ConfigEditor.

Safe now that https://dagster.phacility.com/D1544 landed

Dec 6 2019, 12:12 AM
schrockn accepted D1542: Considerably reduce volume of type data fetched by ConfigEditor.
Dec 6 2019, 12:11 AM

Dec 5 2019

schrockn committed R1:662d59cf51a3: Construct all config types in environment schema (authored by schrockn).
Construct all config types in environment schema
Dec 5 2019, 11:55 PM
schrockn closed D1544: Construct all config types in environment schema.
Dec 5 2019, 11:55 PM
schrockn updated the diff for D1544: Construct all config types in environment schema.

actually delete test

Dec 5 2019, 9:41 PM
schrockn updated the diff for D1544: Construct all config types in environment schema.

upmessage

Dec 5 2019, 9:40 PM
schrockn updated the diff for D1544: Construct all config types in environment schema.

up

Dec 5 2019, 9:01 PM
schrockn updated the diff for D1544: Construct all config types in environment schema.

up

Dec 5 2019, 8:38 PM
schrockn updated the diff for D1544: Construct all config types in environment schema.

up

Dec 5 2019, 7:53 PM
schrockn added a comment to D1542: Considerably reduce volume of type data fetched by ConfigEditor.

@bengotow I think this might be the breaking point for doing a more fundamental server side rearchitecture of the config type system. there are some really problematic core aspects of it that this is exposing. I did throw up a hack that gets this working but not the best at all https://dagster.phacility.com/D1544

Dec 5 2019, 7:47 PM
schrockn created D1544: Construct all config types in environment schema.
Dec 5 2019, 6:29 PM
schrockn added a comment to D1542: Considerably reduce volume of type data fetched by ConfigEditor.

Dec 5 2019, 6:14 PM
schrockn requested changes to D1542: Considerably reduce volume of type data fetched by ConfigEditor.

I'm actually seeing crashes now in the config editor with this applied

Dec 5 2019, 6:13 PM
schrockn accepted D1542: Considerably reduce volume of type data fetched by ConfigEditor.

excellent

Dec 5 2019, 5:33 PM

Dec 4 2019

schrockn committed R1:398b955c0ef5: Print process on dagit load (authored by schrockn).
Print process on dagit load
Dec 4 2019, 11:34 PM
schrockn closed D1538: Print process on dagit load.
Dec 4 2019, 11:34 PM
schrockn created D1538: Print process on dagit load.
Dec 4 2019, 9:15 PM
schrockn accepted D1536: Make dagit typeaheads match on value substrings, not just prefixes.
Dec 4 2019, 6:48 PM

Dec 3 2019

schrockn added a comment to D1533: Core dagster_celery engine.

will look more later, but this is really smooth

Dec 3 2019, 7:17 PM
schrockn added inline comments to D1521: [EMR 1/N] Extract EMR implementation.
Dec 3 2019, 6:57 PM
schrockn added inline comments to D1521: [EMR 1/N] Extract EMR implementation.
Dec 3 2019, 6:57 PM
schrockn added a comment to D1524: Attempt to go to strict mode universally.

I may opt-in all these pipelines into the strict mode, however

Dec 3 2019, 6:31 PM
schrockn planned changes to D1524: Attempt to go to strict mode universally.

Not going to make this is the default for now.

Dec 3 2019, 6:30 PM
schrockn committed R1:2f9beabb7f38: Hang created EnvironmentSchemas off of pipeline definition (authored by schrockn).
Hang created EnvironmentSchemas off of pipeline definition
Dec 3 2019, 3:31 AM
schrockn closed D1529: Hang created EnvironmentSchemas off of pipeline definition.
Dec 3 2019, 3:31 AM
schrockn updated the diff for D1529: Hang created EnvironmentSchemas off of pipeline definition.

go back to old diff (also differential is so good)

Dec 3 2019, 3:23 AM
schrockn updated the diff for D1529: Hang created EnvironmentSchemas off of pipeline definition.

fancy functools.lru_cache

Dec 3 2019, 3:13 AM
schrockn committed R1:f7f85e5a52ef: Config schema building perf improvements (authored by schrockn).
Config schema building perf improvements
Dec 3 2019, 3:03 AM
schrockn closed D1528: Config schema building perf improvements.
Dec 3 2019, 3:03 AM
schrockn added a comment to D1494: Add strict mode to pipelines and only initialized required resources.

πŸ‘πŸ» @alangenfeld agree let's chat when you are in tmrw

Dec 3 2019, 12:38 AM
schrockn updated the summary of D1529: Hang created EnvironmentSchemas off of pipeline definition.
Dec 3 2019, 12:01 AM
schrockn created D1529: Hang created EnvironmentSchemas off of pipeline definition.
Dec 3 2019, 12:00 AM

Dec 2 2019

schrockn updated the summary of D1528: Config schema building perf improvements.
Dec 2 2019, 11:44 PM
schrockn retitled D1528: Config schema building perf improvements from Schema building improvements to Config schema building perf improvements.
Dec 2 2019, 11:42 PM
schrockn created D1528: Config schema building perf improvements.
Dec 2 2019, 11:41 PM
schrockn retitled D1494: Add strict mode to pipelines and only initialized required resources from Add strict mod to pipelines and only initialized required resources to Add strict mode to pipelines and only initialized required resources.
Dec 2 2019, 10:40 PM
schrockn added a comment to D1523: Move database interactions from airline demo to dagster-postgres and dagster-aws.

agree with alex. i don't think these have hit the level of utility yet to warrant incorporation to a library that we are on the hook supporting

Dec 2 2019, 10:38 PM
schrockn added inline comments to D1494: Add strict mode to pipelines and only initialized required resources.
Dec 2 2019, 10:35 PM
schrockn added inline comments to D1494: Add strict mode to pipelines and only initialized required resources.
Dec 2 2019, 10:26 PM
schrockn planned changes to D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.

I'm going to pop this off folks' queues for now. Let's see if we actually need this.

Dec 2 2019, 10:24 PM
schrockn committed R1:c3994b5e927a: Check-in expected failure repros for #1950 and #1949 (authored by schrockn).
Check-in expected failure repros for #1950 and #1949
Dec 2 2019, 10:23 PM
schrockn closed D1505: Check-in expected failure repros for #1950 and #1949.
Dec 2 2019, 10:23 PM
schrockn retitled D1505: Check-in expected failure repros for #1950 and #1949 from Check-in skipped repros for #1950 and #1949 to Check-in expected failure repros for #1950 and #1949.
Dec 2 2019, 10:21 PM
schrockn updated the diff for D1505: Check-in expected failure repros for #1950 and #1949.

change from skip to xfail

Dec 2 2019, 10:14 PM
schrockn added a comment to D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.

@alangenfeld i agree it's a concern and there's no rush to merge. in fact totally willing to wait for more instances of user feedback since it does muddy the model and injects complexity (as is evidenced by all the error conditions I needed to test for)

Dec 2 2019, 7:26 PM
schrockn committed R1:e6b7b2013ee5: Eliminate selector argument from PipelineDefinition (authored by schrockn).
Eliminate selector argument from PipelineDefinition
Dec 2 2019, 5:29 PM
schrockn closed D1501: Eliminate selector argument from PipelineDefinition.
Dec 2 2019, 5:28 PM
schrockn updated the diff for D1501: Eliminate selector argument from PipelineDefinition.

rebase

Dec 2 2019, 5:18 PM

Dec 1 2019

schrockn accepted D1516: Disable Re-execute and Retry when pipeline is unavailabe in repo (#1759).

yeah let's go ahead and merge this.

Dec 1 2019, 7:43 PM
schrockn added a comment to D1494: Add strict mode to pipelines and only initialized required resources.

Migrating the test suite is a pretty modest change: https://dagster.phacility.com/D1524

Dec 1 2019, 7:17 PM
schrockn updated the diff for D1524: Attempt to go to strict mode universally.

up

Dec 1 2019, 7:11 PM
schrockn updated the diff for D1524: Attempt to go to strict mode universally.

up

Dec 1 2019, 7:02 PM
schrockn updated the diff for D1524: Attempt to go to strict mode universally.

up

Dec 1 2019, 6:51 PM
schrockn created D1524: Attempt to go to strict mode universally.
Dec 1 2019, 6:38 PM
schrockn added inline comments to D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.
Dec 1 2019, 6:34 PM
schrockn added inline comments to D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.
Dec 1 2019, 6:27 PM
schrockn updated the diff for D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.

up

Dec 1 2019, 6:26 PM
schrockn updated the diff for D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.

upmessage

Dec 1 2019, 6:25 PM

Nov 30 2019

schrockn updated the diff for D1501: Eliminate selector argument from PipelineDefinition.

rebase

Nov 30 2019, 2:26 AM
schrockn abandoned D1447: WIP metas.
Nov 30 2019, 2:23 AM
schrockn abandoned D1462: Return all things.
Nov 30 2019, 2:22 AM
schrockn accepted D1513: Add input and output labels to invocations panel in dagit (#1503).
Nov 30 2019, 2:16 AM
schrockn requested changes to D1516: Disable Re-execute and Retry when pipeline is unavailabe in repo (#1759).

We should also consider the case where the configuration has changed and can no longer be parsed.

Nov 30 2019, 2:16 AM
schrockn accepted D1517: Enable pyspark tests.

Failure is unrelated

Nov 30 2019, 2:14 AM
schrockn updated the summary of D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.
Nov 30 2019, 2:02 AM
schrockn updated the summary of D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.
Nov 30 2019, 2:01 AM
schrockn added a comment to D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.

Also a tip of the hat to one @alangenfeld for the excellent implementation of the dep dsl. This was very straightforward to add. Vast, vast majority of the time was spent writing tests and delineating error states and helpful error messages. The core functionality is implemented in only 3 one-liner methods.

Nov 30 2019, 1:59 AM
schrockn updated the summary of D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.
Nov 30 2019, 1:55 AM
schrockn updated the diff for D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.

upmessage

Nov 30 2019, 1:53 AM

Nov 29 2019

schrockn updated the diff for D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.

py27

Nov 29 2019, 8:39 PM
schrockn abandoned D1370: RFC: chain function for building strings of solids.

funny enough trying the bitwise or approach over here https://dagster.phacility.com/D1512

Nov 29 2019, 8:11 PM
schrockn created D1512: Add pipe dep operator to easily express serial deps of single input/single output solids.
Nov 29 2019, 7:34 PM
schrockn accepted D1508: Fix invalid + unrecoverable subselection state in Dagit (#1924).

πŸ‘πŸ»

Nov 29 2019, 6:31 PM

Nov 28 2019

schrockn accepted D1510: Show invocations of composite solids when they’re expanded (#1836).

lovely

Nov 28 2019, 5:06 PM
schrockn accepted D1509: Show autocompletions in initial empty config state, mention Ctrl-Space (#1576).

this is lovely

Nov 28 2019, 5:02 PM
schrockn requested changes to D1508: Fix invalid + unrecoverable subselection state in Dagit (#1924).

Great. Please add the comment so that we have a record for why the network-only is there.

Nov 28 2019, 4:47 PM
schrockn added inline comments to D1508: Fix invalid + unrecoverable subselection state in Dagit (#1924).
Nov 28 2019, 4:27 PM
schrockn updated the diff for D1501: Eliminate selector argument from PipelineDefinition.

upmessage

Nov 28 2019, 12:56 AM
schrockn updated the diff for D1501: Eliminate selector argument from PipelineDefinition.

upmessage

Nov 28 2019, 12:54 AM
schrockn updated the diff for D1494: Add strict mode to pipelines and only initialized required resources.

upmessage

Nov 28 2019, 12:52 AM
schrockn updated the diff for D1494: Add strict mode to pipelines and only initialized required resources.

upmessage

Nov 28 2019, 12:47 AM
schrockn created D1505: Check-in expected failure repros for #1950 and #1949.
Nov 28 2019, 12:28 AM
schrockn accepted D1504: Add verbose flag to twine.
Nov 28 2019, 12:28 AM

Nov 27 2019

schrockn resigned from D1430: Introduce Partition API.

resigning until sashank takes a looksie. if changes are substantial plz readd me

Nov 27 2019, 11:44 PM
schrockn updated the diff for D1494: Add strict mode to pipelines and only initialized required resources.

up

Nov 27 2019, 11:43 PM
schrockn updated the diff for D1494: Add strict mode to pipelines and only initialized required resources.

upmessage

Nov 27 2019, 11:39 PM
schrockn updated the diff for D1501: Eliminate selector argument from PipelineDefinition.

upmessage

Nov 27 2019, 11:35 PM
schrockn created D1501: Eliminate selector argument from PipelineDefinition.
Nov 27 2019, 11:09 PM
schrockn updated the diff for D1494: Add strict mode to pipelines and only initialized required resources.

up

Nov 27 2019, 10:40 PM
schrockn updated the diff for D1494: Add strict mode to pipelines and only initialized required resources.

up

Nov 27 2019, 10:12 PM