Page MenuHomePhabricator

schrockn (Nick Schrock)
UserAdministrator

Projects

User does not belong to any projects.

User Details

User Since
Mar 20 2019, 8:23 PM (101 w, 5 d)
Roles
Administrator

Recent Activity

Fri, Feb 19

schrockn added a comment to D6572: [dagit] Use Python event types in log filter view.

screenshot looks good. some of those events are deprecated and we should consider not rendering them

Fri, Feb 19, 7:17 PM

Thu, Feb 18

schrockn added a comment to D6416: website top-level headlines update.

"My interpretation of the the feedback we got was that "data flow between reusable, logical components" is too abstract to convey useful information. I also think we should downplay reusability in our core messaging - in most cases, writing reusable solids is not the right pattern."

Thu, Feb 18, 6:53 PM

Mon, Feb 15

schrockn closed D6484: Update sensor docstring.
Mon, Feb 15, 8:38 PM
schrockn committed R1:c304b494c4cc: Update sensor docstring (authored by schrockn).
Update sensor docstring
Mon, Feb 15, 8:38 PM
schrockn requested review of D6484: Update sensor docstring.
Mon, Feb 15, 5:22 PM

Tue, Feb 9

schrockn added a comment to D6017: initial async solids support.

mixing await and yield is a bit odd. is that normative in python?

Tue, Feb 9, 9:28 PM

Tue, Feb 2

schrockn added a comment to D6244: Run each DagsterDaemon subclass in its own thread.

did you consider process-level isolation?

Tue, Feb 2, 3:09 PM

Mon, Feb 1

schrockn added a comment to D6226: Update the headers on the run table to call out Run ID vs Snapshot ID, pipeline more clearly.

Did we ever consider breaking tags into its own column?

Mon, Feb 1, 5:38 PM

Jan 29 2021

schrockn added a comment to D6164: Remove Dauphin.

Sent via Superhuman ( https://sprh.mn/?vip=schrockn@elementl.com )

Jan 29 2021, 2:32 AM
schrockn added a comment to D6164: Remove Dauphin.

definitely think we should keep prefix like @dgibson. grepability is good

Jan 29 2021, 12:06 AM
schrockn resigned from D6164: Remove Dauphin.

Very much support getting rid of the registry. Thanks for doing this! Let's just try to mitigate risk as much as possible with a diff this large

Jan 29 2021, 12:05 AM

Jan 27 2021

schrockn added a comment to D6177: WIP: dagster new-repo Generated Output.

I meant code artifact, like "solid", "pipeline", "resource" etc

Jan 27 2021, 10:51 PM
schrockn added a comment to D6177: WIP: dagster new-repo Generated Output.

Agree with nearly all of sandy's and rex's feedback.

Jan 27 2021, 8:26 PM

Jan 24 2021

schrockn resigned from D6132: UI support for dynamic outputs, Run view state cleanup.

Very excited to see this land. Will defer to dish on the deets of the js.

Jan 24 2021, 12:12 AM

Jan 23 2021

schrockn added a comment to D6029: Flip asset materialization UI so newest time/partition is on the left.

adding @sandyryza here. I think this makes a lot of sense and I don't think having the events be left-to-right and the graphs right-to-left is a big problem but open to other's perspective

Jan 23 2021, 11:58 PM
schrockn added a reviewer for D6029: Flip asset materialization UI so newest time/partition is on the left: sandyryza.
Jan 23 2021, 11:57 PM
schrockn added a comment to D5115: don't error on solid type annotations that don't resolve to dagster types.

"I'm pretty resistant to directions that involve using type annotations for anything other than annotating the actual Python type that's expected somewhere. It's fairly abusive of a Python language feature, and it causes non-trivial trouble for anyone who's trying to use that language feature in the correct way."

Jan 23 2021, 7:22 PM
schrockn added a comment to D5115: don't error on solid type annotations that don't resolve to dagster types.

We could also get clever and try to reuse input_defs and output_defs supporting a few different variants:

Jan 23 2021, 1:30 PM
schrockn added a comment to D5115: don't error on solid type annotations that don't resolve to dagster types.

I'm concerned about too many breaking changes and too much thrash. But I think that the critical thing here would be coming up with a more condense syntax for inputs and outputs. Right now if we forced users to InputDefinition and OuptutDefinition for the dagster-type-only case the code explosion would be brutal.

Jan 23 2021, 1:27 PM

Jan 22 2021

schrockn added a comment to D5115: don't error on solid type annotations that don't resolve to dagster types.

Yeah my first instinct is stress and terror given our past forays into magically commingling the python and dagster type system.

Jan 22 2021, 11:36 PM

Jan 19 2021

schrockn added a comment to D6036: Make type_check_fn optional on DagsterType.

Here's a thought:

Jan 19 2021, 7:56 PM
schrockn abandoned D5321: Different scenarios to consider when doing input/output management.
Jan 19 2021, 4:39 PM

Jan 14 2021

schrockn accepted D6018: update partitions and backfills doc with screenshots of new UI.
Jan 14 2021, 10:57 PM
schrockn published D6018: update partitions and backfills doc with screenshots of new UI for review.
Jan 14 2021, 10:57 PM

Jan 13 2021

schrockn accepted D5951: [dagit] Add "Scheduled" tab to Runs page.
Jan 13 2021, 8:41 PM
schrockn requested changes to D5951: [dagit] Add "Scheduled" tab to Runs page.

can you display pipeline name in addition to schedule name?

Jan 13 2021, 7:27 PM
schrockn accepted D5721: DynamicOutput DSL map approach.

excited to see this land. I will let @prha look at the lint ish as I have little context on that

Jan 13 2021, 7:23 PM
schrockn added a comment to D5951: [dagit] Add "Scheduled" tab to Runs page.

could you throw up a screenshot?

Jan 13 2021, 4:06 PM

Jan 12 2021

schrockn added inline comments to D5829: Modernize dagstermill file handling.
Jan 12 2021, 11:11 PM
schrockn accepted D5834: RFC: fix result.result_for_solid when using mem_io_manager + default to mem_io_manager.

"hardcoded_resource_instances" is a pretty rough name since they aren't really hardcoded. they are not fixed in code. maybe "resource_instances_to_override" or something?

Jan 12 2021, 1:43 PM
schrockn resigned from D5474: ObjectStoreOperation/AssetStoreOperation -> HandledOutput/LoadedInput.

Seems good. Will let others do a detailed review!

Jan 12 2021, 1:35 PM

Jan 8 2021

schrockn accepted D5858: Update instance docs to reflect 0.10.0 changes.

startreknod

Jan 8 2021, 10:41 PM
schrockn accepted D5812: Add initial overview docs for sensors.
Jan 8 2021, 1:02 PM
schrockn requested changes to D5858: Update instance docs to reflect 0.10.0 changes.
Jan 8 2021, 12:59 PM
schrockn accepted D5862: Run launcher docs refresh for 0.10.0.
Jan 8 2021, 12:57 PM
schrockn requested changes to D5843: root input manager.

just doing q mgmt. any other alternative names?

Jan 8 2021, 12:55 PM
schrockn added inline comments to D5843: root input manager.
Jan 8 2021, 12:55 PM
schrockn requested changes to D5721: DynamicOutput DSL map approach.

q mgmt.

Jan 8 2021, 12:50 PM
schrockn requested changes to D5613: RFC DynamicOutput DSL __iter__ approach.

is this still a relevant diff? req'ing changes for q mgmt

Jan 8 2021, 12:50 PM
schrockn requested changes to D5834: RFC: fix result.result_for_solid when using mem_io_manager + default to mem_io_manager.

So just brainstorming here a bit. Another approach might be special-casing at a different spot.

Jan 8 2021, 12:49 PM
schrockn added a comment to D5843: root input manager.

How fixed are we on the term "root"? I'm trying to recall other words that we have used or would use to describe inputs that have no upstream. I just want to make sure this is the best we have come up with.

Jan 8 2021, 12:45 PM

Jan 7 2021

schrockn requested changes to D5721: DynamicOutput DSL map approach.

lgtm. just want to know about the linting issue

Jan 7 2021, 10:39 PM
schrockn requested changes to D5721: DynamicOutput DSL map approach.

let's kill forEach

Jan 7 2021, 9:36 PM
schrockn resigned from D5632: DynamicOutput re-execution.

Very happy with the tracking changes

Jan 7 2021, 9:35 PM
schrockn accepted D5542: DynamicOutput execution support.
Jan 7 2021, 9:31 PM
schrockn added a comment to D5542: DynamicOutput execution support.

Excellent. Really exciting move forward. I had some minor comments around places where you can be a little more defensive, but I think this ended up at a nice place

Jan 7 2021, 9:30 PM
schrockn added inline comments to D5834: RFC: fix result.result_for_solid when using mem_io_manager + default to mem_io_manager.
Jan 7 2021, 9:14 PM
schrockn added inline comments to D5834: RFC: fix result.result_for_solid when using mem_io_manager + default to mem_io_manager.
Jan 7 2021, 8:59 PM
schrockn accepted D5844: Troubleshooting page for schedules.

this is great. very clearly written

Jan 7 2021, 4:08 PM
schrockn accepted D5755: DagsterEvent step_handle.

This is nice

Jan 7 2021, 3:26 PM
schrockn resigned from D5852: mypy everything (almost).

will let you guys handle this

Jan 7 2021, 3:23 PM

Jan 6 2021

schrockn added a comment to D5834: RFC: fix result.result_for_solid when using mem_io_manager + default to mem_io_manager.

oof ok I didn't think about that deeply enough. that definitely needs to work

Jan 6 2021, 10:18 PM
schrockn added inline comments to D5834: RFC: fix result.result_for_solid when using mem_io_manager + default to mem_io_manager.
Jan 6 2021, 7:51 PM
schrockn resigned from D5840: RFC: Add migration to pull partition columns to the run table.
Jan 6 2021, 7:26 PM
schrockn resigned from D5337: s3 sensor toy pipeline.

looking good! will let someone else do final accept

Jan 6 2021, 7:25 PM
schrockn accepted D5341: add toy asset sensor.

This really is lovely. Excited to see it land.

Jan 6 2021, 6:57 PM

Jan 5 2021

schrockn accepted D5747: Type hints for check.
Jan 5 2021, 10:52 PM
schrockn resigned from D5764: Run mypy on dagster core.

cool stuff. looking forward to this landing!

Jan 5 2021, 9:54 PM

Jan 4 2021

schrockn added inline comments to D5744: RFC: handle type-based loading with input managers.
Jan 4 2021, 10:05 PM
schrockn added a comment to D5778: Replace namedtuple with typing.NamedTuple in outputs.py.

to be clear i love the new syntax.

Jan 4 2021, 5:12 PM
schrockn requested changes to D5778: Replace namedtuple with typing.NamedTuple in outputs.py.

I don't think we should eliminate the check calls until we turn on mypy typing

Jan 4 2021, 5:12 PM
schrockn added a comment to D5757: Make DagsterDaemonScheduler the default scheduler.

seems like we could make a UI affordance the use case where you turn on the schedule without a daemon process running

Jan 4 2021, 5:10 PM
schrockn requested changes to D5764: Run mypy on dagster core.

Please see comment on https://dagster.phacility.com/D5768 I would like to have a plan and messaging in place for the team before we merge this.

Jan 4 2021, 4:48 PM
schrockn requested changes to D5768: mypy.

So before you and @max start to do this I would like to see 1) a plan for adopting this and goals around it and 2) language that we will announce to the team around expectations here. E.g. is all new code typed? who is responsible for migrating? what is the overall plan?

Jan 4 2021, 4:47 PM

Dec 28 2020

schrockn added inline comments to D5768: mypy.
Dec 28 2020, 7:22 PM
schrockn added inline comments to D5768: mypy.
Dec 28 2020, 6:48 PM

Dec 23 2020

schrockn added a comment to D5747: Type hints for check.

Never assume anything is happening locally if it doesn’t happen in the ci/cd pipeline :-)

Dec 23 2020, 1:33 AM

Dec 22 2020

schrockn added a comment to D5747: Type hints for check.

Yeah would like to have a clear plan for actually turning on mypy. Adding typehints without automated checks is worse than having no typehints imo

Dec 22 2020, 10:10 PM
schrockn abandoned D5728: Revert "Eliminate pipeline_def from InitResourceContext".
Dec 22 2020, 12:49 PM
schrockn accepted D5649: RFC object_manager_from_intermediate_storage.
Dec 22 2020, 12:48 PM
schrockn accepted D5696: Fix re-execution in Dagit.
Dec 22 2020, 12:48 PM
schrockn requested review of D5728: Revert "Eliminate pipeline_def from InitResourceContext".
Dec 22 2020, 2:42 AM
schrockn added a reverting change for D5709: Eliminate pipeline_def from InitResourceContext: D5728: Revert "Eliminate pipeline_def from InitResourceContext".
Dec 22 2020, 2:17 AM
schrockn added a reverting change for R1:682da0c179bb: Eliminate pipeline_def from InitResourceContext: D5728: Revert "Eliminate pipeline_def from InitResourceContext".
Dec 22 2020, 2:17 AM
schrockn added a comment to D5649: RFC object_manager_from_intermediate_storage.

I can either revert it or we can add the backwards compat variant. Your call!

Dec 22 2020, 2:00 AM

Dec 21 2020

schrockn closed D5709: Eliminate pipeline_def from InitResourceContext.
Dec 21 2020, 10:22 PM
schrockn committed R1:682da0c179bb: Eliminate pipeline_def from InitResourceContext (authored by schrockn).
Eliminate pipeline_def from InitResourceContext
Dec 21 2020, 10:22 PM
schrockn updated the diff for D5709: Eliminate pipeline_def from InitResourceContext.

up

Dec 21 2020, 9:51 PM
schrockn accepted D5712: [Issue 3463] Remove versioning code paths from main execution path..
Dec 21 2020, 8:02 PM
schrockn added a comment to D5712: [Issue 3463] Remove versioning code paths from main execution path..

Since you already have the profiling set up, do you have a notion what the hot paths are for the remaining 9 seconds?

Dec 21 2020, 8:01 PM
schrockn closed D5320: Make Noneable types make fields become optional by default.
Dec 21 2020, 8:00 PM
schrockn committed R1:6cc97b7dcf82: Make Noneable types make fields become optional by default (authored by schrockn).
Make Noneable types make fields become optional by default
Dec 21 2020, 8:00 PM
schrockn updated the diff for D5320: Make Noneable types make fields become optional by default.

up

Dec 21 2020, 7:19 PM
schrockn updated the diff for D5320: Make Noneable types make fields become optional by default.

rebase

Dec 21 2020, 7:18 PM
schrockn added a comment to D5709: Eliminate pipeline_def from InitResourceContext.

I would rather eliminate it as I anticipate that we will add resources other artifacts like sensors to enable local dev and testing. This change makes this abstraction applicable across other axes than just graphs

Dec 21 2020, 4:43 PM
schrockn added a comment to D5428: [RFC] Resource keys re-mapping.

I've been thinking about this a bunch and name issue continues to gnaw at me. configured, while elegant, expose this problem but it is this diff that in particularly makes it more acute.

Dec 21 2020, 4:39 PM
schrockn requested changes to D5632: DynamicOutput re-execution.

Very concerned about using the step key DSL in _update_tracking_dict and friends. I think should be persisting the structured step key information DagsterEvent to reduce the surface area of our reliance on the DSL

Dec 21 2020, 4:05 PM
schrockn requested changes to D5542: DynamicOutput execution support.

Thanks for your patience in terms of me getting to this:

Dec 21 2020, 3:51 PM
schrockn added a comment to D5637: Add an HTML metadata entry type, allow JSON and HTML to be inlined into logs.

I have fairly grave concerns about this. I think we open ourselves to a lot of chicanery.

Dec 21 2020, 3:23 PM
schrockn added a comment to D5650: Implemented request/retry/failure for input/output managers.

Definitely agree that these events (Failure, RetryRequested) should apply to the entire step

Dec 21 2020, 3:21 PM

Dec 19 2020

schrockn requested review of D5709: Eliminate pipeline_def from InitResourceContext.
Dec 19 2020, 7:31 PM

Dec 15 2020

schrockn accepted D5612: DynamicOutput enforce constraints.

cool. plz consider final comments

Dec 15 2020, 5:14 PM
schrockn added a comment to D5607: Keep recent sensor ticks, regardless of status and add tick garbage collection to sensor loop.

Yeah I was thinking like a week of historical data or something.

Dec 15 2020, 3:53 PM
schrockn accepted D5604: mappable -> dynamic.
Dec 15 2020, 2:54 AM
schrockn added inline comments to D5522: object managers docs overview.
Dec 15 2020, 2:31 AM

Dec 14 2020

schrockn accepted D5522: object managers docs overview.

I think this is excellent. I have some fairly minor comment which I'd love to see considered. Please check them out prior to landing.

Dec 14 2020, 8:34 PM
schrockn added a comment to D5522: object managers docs overview.

arc patch works great thanks!

Dec 14 2020, 7:18 PM
schrockn closed D5589: Make DefinitionConfigSchema backwards compatibile with ConfigField.
Dec 14 2020, 7:14 PM
schrockn committed R1:913c2c357d31: Make DefinitionConfigSchema backwards compatibile with ConfigField (authored by schrockn).
Make DefinitionConfigSchema backwards compatibile with ConfigField
Dec 14 2020, 7:14 PM
schrockn closed D5567: Be able to use the full spectrum of config schema in composites.
Dec 14 2020, 7:13 PM