Page MenuHomePhabricator

schrockn (Nick Schrock)
UserAdministrator

Projects

User does not belong to any projects.

User Details

User Since
Mar 20 2019, 8:23 PM (47 w, 6 d)
Roles
Administrator

Recent Activity

Yesterday

schrockn added inline comments to D2074: Disable dask and gcp tests..
Tue, Feb 18, 11:46 PM
schrockn accepted D2074: Disable dask and gcp tests..

Commenting out body of functions seems unnecessary. Thrashes commit history (will all blame to you) and a refactor might be missed.

Tue, Feb 18, 11:38 PM
schrockn updated the diff for D1883: warmup.

up

Tue, Feb 18, 11:28 PM
schrockn abandoned D2037: Fix k8s tests post-release.
Tue, Feb 18, 8:11 PM
schrockn accepted D2067: ActiveExecution cleanup.

thumbsup

Tue, Feb 18, 8:11 PM
schrockn added a comment to D2062: Move tutorial into main dagster package.

I'd be more open to making the tutorials available to dagit. I just want to keep dagster core as lean as possible.

Tue, Feb 18, 3:34 PM

Mon, Feb 17

schrockn requested changes to D2062: Move tutorial into main dagster package.

I'm not convinced that this is what we want. A simpler solution would seem to be to have a separate repo for examples that is up-to-date with the latest public version. Implementing a tutorial downloader in the dagster core seems very odd to me.

Mon, Feb 17, 11:56 PM
schrockn added a comment to D2062: Move tutorial into main dagster package.

Just feels a bit wrong to force people to deploy the tutorial the world over

Mon, Feb 17, 10:40 PM
schrockn added a comment to D2062: Move tutorial into main dagster package.

Is there prior art in other systems here? I'm a little wary of putting this in core. How about making a separate installable dagster-tutorial module?

Mon, Feb 17, 10:40 PM

Thu, Feb 13

schrockn updated the diff for D2037: Fix k8s tests post-release.

respin tests

Thu, Feb 13, 12:39 AM
schrockn added a comment to D2037: Fix k8s tests post-release.

Issue https://github.com/dagster-io/dagster/issues/2156

Thu, Feb 13, 12:36 AM
schrockn created D2037: Fix k8s tests post-release.
Thu, Feb 13, 12:30 AM

Wed, Feb 12

schrockn committed R1:708af820c01e: Config migration guide and some improved error messages (authored by schrockn).
Config migration guide and some improved error messages
Wed, Feb 12, 11:42 PM
schrockn closed D2029: Config migration guide and some improved error messages.
Wed, Feb 12, 11:42 PM
schrockn abandoned D1937: restructure resolve_dagster_type.
Wed, Feb 12, 11:34 PM
schrockn abandoned D1973: unpin werkzeug.
Wed, Feb 12, 11:34 PM
schrockn added a reviewer for D2029: Config migration guide and some improved error messages: themissinghlink.
Wed, Feb 12, 11:34 PM
schrockn added a comment to D2029: Config migration guide and some improved error messages.

inconsistency of DagsterInvalidDefinitionError versus DagsterInvariantViolationError is unfortunate. however I think that we need a pretty substantial refactor of both the dagster and config type resolution codepath and would not like to embark on that in the next 30 mins

Wed, Feb 12, 11:33 PM
schrockn accepted D2032: Add input hydration and output materialization configs to custom dataframes..

cool

Wed, Feb 12, 11:28 PM
schrockn updated the diff for D2029: Config migration guide and some improved error messages.

good call on tests

Wed, Feb 12, 11:27 PM
schrockn accepted D2031: solid metadata -> tags.

Please include better summary but lgtm

Wed, Feb 12, 11:23 PM
schrockn added inline comments to D2028: Prevent default Gaant options from being mutated when merged with provided options [2148].
Wed, Feb 12, 10:53 PM
schrockn accepted D2028: Prevent default Gaant options from being mutated when merged with provided options [2148].

clefable

Wed, Feb 12, 10:51 PM
schrockn committed R1:0282e592016f: 0.7.0 Changelog reorg + schrockn additions (authored by schrockn).
0.7.0 Changelog reorg + schrockn additions
Wed, Feb 12, 10:47 PM
schrockn closed D2022: 0.7.0 Changelog reorg + schrockn additions.
Wed, Feb 12, 10:47 PM
schrockn created D2029: Config migration guide and some improved error messages.
Wed, Feb 12, 10:42 PM
schrockn committed R1:8fd9edd70c75: Per live discussions, remove usable types from libraries (Verbose bay bikes… (authored by schrockn).
Per live discussions, remove usable types from libraries (Verbose bay bikes…
Wed, Feb 12, 10:04 PM
schrockn closed D2009: Per live discussions, remove usable types from libraries (Verbose bay bikes version).
Wed, Feb 12, 10:04 PM
schrockn added inline comments to D2022: 0.7.0 Changelog reorg + schrockn additions.
Wed, Feb 12, 9:29 PM
schrockn updated the diff for D2009: Per live discussions, remove usable types from libraries (Verbose bay bikes version).

rebase

Wed, Feb 12, 9:23 PM
schrockn abandoned D1996: Per live discussions, remove usable types from libraries.

abandoning in favor of https://dagster.phacility.com/D2009

Wed, Feb 12, 9:19 PM
schrockn updated the diff for D2009: Per live discussions, remove usable types from libraries (Verbose bay bikes version).

rebase

Wed, Feb 12, 9:18 PM
schrockn added a reviewer for D2022: 0.7.0 Changelog reorg + schrockn additions: sashank.
Wed, Feb 12, 9:15 PM
schrockn created D2022: 0.7.0 Changelog reorg + schrockn additions.
Wed, Feb 12, 9:14 PM
schrockn committed R1:63301820e7cf: Dagster Type Guide (authored by schrockn).
Dagster Type Guide
Wed, Feb 12, 8:49 PM
schrockn closed D1967: Dagster Type Guide.
Wed, Feb 12, 8:49 PM
schrockn updated the diff for D1967: Dagster Type Guide.

up

Wed, Feb 12, 8:03 PM
schrockn updated the diff for D1967: Dagster Type Guide.

up

Wed, Feb 12, 7:52 PM
schrockn retitled D1967: Dagster Type Guide from Dagster Type Guide Stab at dagster docs to Dagster Type Guide.
Wed, Feb 12, 7:51 PM
schrockn committed R1:b1edb7ca2f07: Add type check to migration guide (authored by schrockn).
Add type check to migration guide
Wed, Feb 12, 6:38 PM
schrockn closed D2016: Add type check to migration guide.
Wed, Feb 12, 6:38 PM
schrockn added a comment to D2009: Per live discussions, remove usable types from libraries (Verbose bay bikes version).

@alangenfeld do you think we should change the airline_demo to be the verbose variant as well

Wed, Feb 12, 6:28 PM
schrockn accepted D2014: add prha changes 0.7.0, migrations.
Wed, Feb 12, 6:27 PM
schrockn added a comment to D2006: Document the Celery executor.

looks great. merge away

Wed, Feb 12, 6:26 PM
schrockn created D2016: Add type check to migration guide.
Wed, Feb 12, 6:17 PM
schrockn resigned from D2013: add should_overwrite flags and tests for make_python_type_usable_as_dagster_type.

q mgmt

Wed, Feb 12, 6:07 PM
schrockn added a comment to D2013: add should_overwrite flags and tests for make_python_type_usable_as_dagster_type.

The issue with an overwrite strategy is now the library could be counting on the mapping existing and then the user has blown it away and can do so at any time.

Wed, Feb 12, 4:51 PM
schrockn added a comment to D2013: add should_overwrite flags and tests for make_python_type_usable_as_dagster_type.

removing alex and max for now to avoid noise for now.

Wed, Feb 12, 4:45 PM
schrockn removed reviewers for D2013: add should_overwrite flags and tests for make_python_type_usable_as_dagster_type: alangenfeld, max.
Wed, Feb 12, 4:41 PM
schrockn committed R1:91b59dcb6730: Fix up config docblocks and extract config out of core types documentation (authored by schrockn).
Fix up config docblocks and extract config out of core types documentation
Wed, Feb 12, 2:51 AM
schrockn closed D2003: Fix up config docblocks and extract config out of core types documentation.
Wed, Feb 12, 2:51 AM
schrockn added a comment to D2003: Fix up config docblocks and extract config out of core types documentation.

confirmed no new indentation warnings

Wed, Feb 12, 2:50 AM
schrockn committed R1:b2fa0d4785d6: Fix types.rst issues from merge (authored by schrockn).
Fix types.rst issues from merge
Wed, Feb 12, 2:48 AM
schrockn closed D2007: Fix types.rst issues from merge.
Wed, Feb 12, 2:48 AM
schrockn updated the diff for D2007: Fix types.rst issues from merge.

up

Wed, Feb 12, 2:28 AM
schrockn added a comment to D2007: Fix types.rst issues from merge.

no worries

Wed, Feb 12, 2:28 AM
schrockn committed R1:82d7ccf496ea: 0.7.0 Migration Guide (authored by schrockn).
0.7.0 Migration Guide
Wed, Feb 12, 2:27 AM
schrockn closed D2004: 0.7.0 Migration Guide.
Wed, Feb 12, 2:27 AM
schrockn added a comment to D1997: Rework Gaant for performant rendering on large DAGs, add sidebar, polish.

My only other feedback would be that it would be great to vertically scroll to the node that is currently executing when you single click on it in the table of contents

Wed, Feb 12, 2:23 AM
schrockn added a comment to D2009: Per live discussions, remove usable types from libraries (Verbose bay bikes version).

let's sleep on it and I think @themissinghlink wanted to stew on it a bit

Wed, Feb 12, 1:58 AM
schrockn added a comment to D1997: Rework Gaant for performant rendering on large DAGs, add sidebar, polish.

So I think a reasonable resolution to "undo" the selection of *foo*: If you double click again it toggles back to "*"

Wed, Feb 12, 1:54 AM
schrockn accepted D1964: Dagster Pandas Guide Docs.

Great! Please heed final comments, especially on the first sentence (some of @max 's feedback was unaddressed) Great job! Fun to see this documented

Wed, Feb 12, 12:09 AM

Tue, Feb 11

schrockn updated the diff for D2009: Per live discussions, remove usable types from libraries (Verbose bay bikes version).

spin tests

Tue, Feb 11, 11:54 PM
schrockn added inline comments to D2009: Per live discussions, remove usable types from libraries (Verbose bay bikes version).
Tue, Feb 11, 11:43 PM
schrockn added a comment to D2009: Per live discussions, remove usable types from libraries (Verbose bay bikes version).

This was thrown up as an alternative/basis of discussion on https://dagster.phacility.com/D1996

Tue, Feb 11, 11:42 PM
schrockn added a comment to D1996: Per live discussions, remove usable types from libraries.

Here's an example of what it would look like by using dagster types only in InputDefinitions and OutputDefinitions, and unmapped python types only in type annotations:

Tue, Feb 11, 11:42 PM
schrockn added a comment to D1996: Per live discussions, remove usable types from libraries.

The advantage of moving this out is that it is quite "presumptuous" for a library to claim a fundamental type and map it to the dagster type. It could cause some quite magical behavior. Imagine a user who wants to use mypy but then they forget to use an InputDefinition or OutputDefinition with their specialized dagster types. The system would magically coerce it to the library's dagster type.

Tue, Feb 11, 11:41 PM
schrockn created D2009: Per live discussions, remove usable types from libraries (Verbose bay bikes version).
Tue, Feb 11, 11:41 PM
schrockn added a comment to D1996: Per live discussions, remove usable types from libraries.

You definitely don't have to do this. You only do this *if* you want to be able to annotate your solids with the *python* type and have dagster associate the dagster type with it.

Tue, Feb 11, 11:22 PM
schrockn created D2007: Fix types.rst issues from merge.
Tue, Feb 11, 11:20 PM
schrockn added inline comments to D2004: 0.7.0 Migration Guide.
Tue, Feb 11, 11:12 PM
schrockn updated the diff for D1996: Per live discussions, remove usable types from libraries.

upmessage

Tue, Feb 11, 11:05 PM
schrockn updated the diff for D1996: Per live discussions, remove usable types from libraries.

up

Tue, Feb 11, 11:00 PM
schrockn accepted D1768: [core] add context to type_checks.

looks good

Tue, Feb 11, 10:39 PM
schrockn requested changes to D2006: Document the Celery executor.

Looking nice. Thanks for doing this.

Tue, Feb 11, 10:36 PM
schrockn added inline comments to D1964: Dagster Pandas Guide Docs.
Tue, Feb 11, 10:21 PM
schrockn updated the diff for D1967: Dagster Type Guide.

feedback; next step is to switch to code literal includes

Tue, Feb 11, 10:20 PM
schrockn added inline comments to D1967: Dagster Type Guide.
Tue, Feb 11, 10:17 PM
schrockn added inline comments to D1964: Dagster Pandas Guide Docs.
Tue, Feb 11, 10:04 PM
schrockn updated the diff for D2004: 0.7.0 Migration Guide.

feedback

Tue, Feb 11, 9:46 PM
schrockn updated the diff for D2003: Fix up config docblocks and extract config out of core types documentation.

upmessage

Tue, Feb 11, 9:43 PM
schrockn added inline comments to D2004: 0.7.0 Migration Guide.
Tue, Feb 11, 9:40 PM
schrockn committed R1:00d7788d2361: Fix types tutorial to comport with new API (authored by schrockn).
Fix types tutorial to comport with new API
Tue, Feb 11, 9:39 PM
schrockn closed D1989: Fix types tutorial to comport with new API.
Tue, Feb 11, 9:39 PM
schrockn requested changes to D1768: [core] add context to type_checks.

q mgmt. good call @max

Tue, Feb 11, 9:38 PM
schrockn requested changes to D1964: Dagster Pandas Guide Docs.

almost there. thanks for working through this!

Tue, Feb 11, 9:38 PM
schrockn updated the diff for D1989: Fix types tutorial to comport with new API.

rebase

Tue, Feb 11, 9:20 PM
schrockn updated subscribers of D2004: 0.7.0 Migration Guide.

Including the folks who I believe are responsible for breaking API changes this time around. @themissinghlink @nate if you have some api changes that require migration guide here you go

Tue, Feb 11, 9:10 PM
schrockn updated the diff for D2004: 0.7.0 Migration Guide.

upmessage

Tue, Feb 11, 9:09 PM
schrockn created D2004: 0.7.0 Migration Guide.
Tue, Feb 11, 9:05 PM
schrockn added inline comments to D2003: Fix up config docblocks and extract config out of core types documentation.
Tue, Feb 11, 8:49 PM
schrockn updated the diff for D2003: Fix up config docblocks and extract config out of core types documentation.

upmessage

Tue, Feb 11, 8:47 PM
schrockn updated the diff for D2003: Fix up config docblocks and extract config out of core types documentation.

upmessage

Tue, Feb 11, 8:47 PM
schrockn created D2003: Fix up config docblocks and extract config out of core types documentation.
Tue, Feb 11, 8:43 PM
schrockn updated the diff for D1996: Per live discussions, remove usable types from libraries.

up

Tue, Feb 11, 6:00 PM
schrockn committed R1:44c84a276212: Update dagster type docblocks (authored by schrockn).
Update dagster type docblocks
Tue, Feb 11, 5:50 PM
schrockn closed D1995: Update dagster type docblocks.
Tue, Feb 11, 5:50 PM
schrockn updated the diff for D1989: Fix types tutorial to comport with new API.

up

Tue, Feb 11, 5:43 PM
schrockn added a comment to D1997: Rework Gaant for performant rendering on large DAGs, add sidebar, polish.

This is pretty slick. A few product things:

Tue, Feb 11, 5:38 PM
schrockn updated the diff for D1989: Fix types tutorial to comport with new API.

up

Tue, Feb 11, 1:30 AM