Page MenuHomeElementl

schrockn (Nick Schrock)
UserAdministrator

Projects

User does not belong to any projects.

User Details

User Since
Mar 20 2019, 8:23 PM (123 w, 5 d)
Roles
Administrator

Recent Activity

Jun 21 2021

schrockn added a comment to D8435: In and Out.

@sandyryza Yeah I actually find that reasoning pretty compelling. Do you think we should support the old format whenever?

Jun 21 2021, 9:48 PM

Jun 19 2021

schrockn added a comment to D8435: In and Out.

So a couple things:

Jun 19 2021, 12:12 AM

Jun 18 2021

schrockn added a comment to D8435: In and Out.

Definitely want to kick-test the dict-style passing for dogfooding at a minimum. I have meetings starting in about 5 mins but I do have a preference for the dict-style approach for a few reasons. Will follow up later

Jun 18 2021, 4:53 PM

Jun 11 2021

schrockn added a comment to D8344: [crag] job name.

IMO if the purpose to support "pipeline_name: mode_name" in dagit displays we could do that at the dagit/graphql layer exclusively rather than inject the notion of a suffix into the greenfield job concept

Jun 11 2021, 6:56 PM
schrockn added a comment to D8344: [crag] job name.

It seems odd to provide this and not just a name param

Jun 11 2021, 5:20 PM

Jun 10 2021

schrockn added a comment to D8283: [crag] partitions on modes.

Super exciting stuff. Sandy Ryza: The Ontology Slayer

Jun 10 2021, 10:33 PM

Jun 9 2021

schrockn added a comment to D8261: [crag] config mapping via mode.

If this is just a temporary exigency to enable CRAG and we will eventually remove mode, maybe the argument to the ModeDefinition constructor should __config_mapping_for_job_compat to make it clear that we're not support it as a public API and people shouldn't worry about the weird implications of it on shared modes and other issues?

Jun 9 2021, 7:55 PM

Jun 3 2021

schrockn added a comment to D8222: [RFC] make_log_handler_resource.

I'm a bit more wary of allowing users to do that. This is "special" since it is a "system" resource that is not directly reference by a context. (i.e., the user always goes through the intermediate logging API rather than direct against the resource)

Jun 3 2021, 8:55 PM
schrockn added a comment to D8222: [RFC] make_log_handler_resource.

Yeah I think with this one might need some special rule for "logger resources" that they are automatically added to every solid in an execution

Jun 3 2021, 6:29 PM
schrockn added a comment to D8209: [RFC] send python logging messages to the DagsterLogManager.

One nice thing we could do here would be to make it easy to filter down log messages in the structured viewer to specific loggers. That could actually be a pretty nice feature.

Jun 3 2021, 6:22 PM

May 26 2021

schrockn added inline comments to D7999: build_init_resource_context.
May 26 2021, 7:09 PM
schrockn added a comment to D7999: build_init_resource_context.

I think this is a fine incremental improvement. To me the biggest question (which this begins to address in some ways) is testing complex resources in a nice way. Would love to see how this feels on dogfooding/demo pipelines

May 26 2021, 4:02 PM

May 24 2021

schrockn requested changes to D8065: approachability blog post.

can you spin this on github so we get vercel previews?

May 24 2021, 9:30 PM

May 7 2021

schrockn added a comment to D7677: [Depends on D7674] Refactor lambda_solid as a simple wrapper around _Solid.

I think we can just remove it from all docs and examples and leave it as undeprecated since the cost of keeping it around is low

May 7 2021, 5:03 PM

May 5 2021

schrockn closed D7734: Ease up on language and change -- to em dashes.
May 5 2021, 1:11 AM
schrockn committed R1:6997dc91c809: Ease up on language and change -- to em dashes (authored by schrockn).
Ease up on language and change -- to em dashes
May 5 2021, 1:11 AM
schrockn requested review of D7734: Ease up on language and change -- to em dashes.
May 5 2021, 12:30 AM

May 4 2021

schrockn resigned from D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

I'll let @alangenfeld accept, but I'm very supportive of this change and excited to see it land!

May 4 2021, 8:18 PM

Apr 30 2021

schrockn added a comment to D7680: [RFC] DagsterAuthManager for run attribution.

Actually duh that is in the daemon process rather than dagit.

Apr 30 2021, 8:57 PM
schrockn added a comment to D7680: [RFC] DagsterAuthManager for run attribution.

Re: th

Apr 30 2021, 8:04 PM
schrockn added a comment to D7674: [Depends on D7670] Allow solid to wrap fxn without context arg.

I think this is potentially an exciting improvement (despite some initial hesitancy about the magic) and not just for testing. There are other benefits as well:

Apr 30 2021, 4:24 PM
schrockn added a comment to D7619: RFC: add basic event metadata using plain dictionary.

One last question on this one: what happens if someone passes a non-json serializable value in the dictionary that gets automatically coerced to JsonMetadataEntryData. Do we have a clear error message? A test of this would be good.

Apr 30 2021, 2:30 PM

Apr 29 2021

schrockn accepted D7619: RFC: add basic event metadata using plain dictionary.

This looks great. A huge improvement. One minor thing is that I think we should add a union or marker interface identify all entry data classes for more concise documentation (probably some code wins too?).

Apr 29 2021, 3:55 PM

Apr 28 2021

schrockn added a comment to D7630: tone down testing concepts section.

Convo with Sandy on slack:

Apr 28 2021, 4:10 PM
schrockn accepted D7630: tone down testing concepts section.

Chatted with sandy briefly and got to agreement. This content should eventually be massaged and resurfaced in a guide or post. I archived it here: https://docs.google.com/document/d/1B7fEOM8Ga03NbDwSu29LF7--JDri8JPGV1YBeE4rIIE/edit

Apr 28 2021, 4:08 PM
schrockn added inline comments to D7630: tone down testing concepts section.
Apr 28 2021, 1:31 AM

Apr 27 2021

schrockn accepted D7484: RFC make_values_resource.

No problem on the thrash. think I'm the one that started (or at least continued) the bikeshedding party! We should probably use the term "values" instead of "variables" in examples, tests, and docs, but I think this is good.

Apr 27 2021, 10:19 PM
schrockn requested changes to D7484: RFC make_values_resource.

Cool. Agree with your reasoning on the _def suffix! Was just spitballing. Req'ing for q mgmt.

Apr 27 2021, 8:22 PM
schrockn added a comment to D7484: RFC make_values_resource.

Yeah that name in particular I find very confusing. Config is the way that you specify how something is constructed, rather than the output of that construction process. It also makes it sound like a property or feature of the config system, rather than the other way around. I'm concerned about the reaction of "Oh a config resource. I guess this means the config system has a resource concept as well?"

Apr 27 2021, 5:09 PM
schrockn added a comment to D7619: RFC: add basic event metadata using plain dictionary.

Oh I misinterpreted/read. So you are proposing introducing a new class EventMetadata?

Apr 27 2021, 5:07 PM
schrockn resigned from D7526: [RFC] Directly invoke solids.

removing myself to reduce noise. please re-add once this is ready for review!

Apr 27 2021, 5:03 PM
schrockn added a comment to D7619: RFC: add basic event metadata using plain dictionary.

Given that we're introducing the short form and the assumption is that the long form will used way way less, I don't think that renaming the long form is worth the thrash.

Apr 27 2021, 4:57 PM
schrockn added a comment to D7484: RFC make_values_resource.

Reflecting on this I think the major con of this approach is that the term "variable" (1) makes it feel like an integrated feature as opposed to a user-space feature that a user could easily implement on their own, (2) it occupies the term "variable" which could potentially be used for a feature in the future and (3) it could cause confusion with Airflow variables.

Apr 27 2021, 2:57 PM

Apr 26 2021

schrockn added a comment to D7484: RFC make_values_resource.

@sandyryza @yuhan What do you think of this approach?

Apr 26 2021, 9:57 PM
schrockn added a comment to D7484: RFC make_values_resource.

Yes effectively that with a little sugar and an approachable framing.

Apr 26 2021, 5:57 PM
schrockn added a comment to D7484: RFC make_values_resource.

Not suggesting a reserved keyword. This is mostly about naming and norms. I think people are likely to understand intuitively what a variables_resource is used for and the config variant is harder to understand. This also allows the use of the variables resource without specifying a config schema.

Apr 26 2021, 5:21 PM
schrockn added a comment to D7484: RFC make_values_resource.

This name is definitely a bit tough. If I wanted to get global variables into the pipeline make_config_resource wouldn't really lead me to it.

Apr 26 2021, 4:41 PM

Apr 23 2021

schrockn added a comment to D6648: [RFC] pipeline contains a graph.

whats the plan with this?

Apr 23 2021, 8:24 PM
schrockn added a comment to D7572: [dagit] Link to pipeline from Run table.

seems like a good compromise

Apr 23 2021, 3:45 PM

Apr 22 2021

schrockn added a comment to D7526: [RFC] Directly invoke solids.

should be really straightforward to implement invokable on top of this

Apr 22 2021, 6:22 PM
schrockn added a comment to D7526: [RFC] Directly invoke solids.

One other thing worth noting here is that since we are passing inputs directly and getting out outputs directly resource requirements from i/o managers should be ignored

Apr 22 2021, 5:00 PM
schrockn added a comment to D7526: [RFC] Directly invoke solids.

I think it's a great direction and solves the problem quite elegantly.

Apr 22 2021, 4:18 PM
schrockn added a comment to D7526: [RFC] Directly invoke solids.

Agree with alex that we should focus on direct solid execution only

Apr 22 2021, 3:37 PM
schrockn added a comment to D7384: RFC: remove the airline demo.

I'd like to have a nice dagstermill example in the repo, and this was the best one. Other than that I support deleting.

Apr 22 2021, 1:23 AM

Apr 21 2021

schrockn accepted D7438: Revamp the configuration concept section.

great stuff. consider final comment but big improvement

Apr 21 2021, 10:24 PM
schrockn added inline comments to D7438: Revamp the configuration concept section.
Apr 21 2021, 10:23 PM
schrockn added a comment to D7438: Revamp the configuration concept section.

that did it! thanks

Apr 21 2021, 2:57 PM
schrockn requested changes to D7438: Revamp the configuration concept section.

This is a great overall. Just a few suggestions.

Apr 21 2021, 2:31 PM
schrockn added a comment to D7438: Revamp the configuration concept section.

The preview link looks out-of-date?

Apr 21 2021, 2:08 PM

Apr 20 2021

schrockn added a comment to D7515: merge inferred input props in to declared.

Yes I think this is a good idea

Apr 20 2021, 4:47 PM
schrockn added inline comments to D7515: merge inferred input props in to declared.
Apr 20 2021, 3:06 PM

Apr 19 2021

schrockn added a comment to D7493: change run view to show compute log panel with log type toggle.

This is a great strat.

Apr 19 2021, 6:48 PM
schrockn added a comment to D7492: allow partial InputDefinition list.

per-offline discussion agree with decision that InputProps or similar doesn't clear the cost-benefit bar right now

Apr 19 2021, 6:04 PM
schrockn added a comment to D7492: allow partial InputDefinition list.

One reason I like the named "InputProps" is that it felt additive, rather than the single source of truth for definitions. My concern here is that input defs would no longer be a single source of truth. The current state of things is definitely annoying so this is not a veto or anything. Just raising the concern.

Apr 19 2021, 4:10 PM
schrockn updated subscribers of D7446: default config schema to Any.

We also need to determine the fate of the dagit "config nub" with this change. cc: @bengotow

Apr 19 2021, 3:06 PM

Apr 15 2021

schrockn requested changes to D7438: Revamp the configuration concept section.

minor changes but bouncing back to your queue

Apr 15 2021, 4:24 PM
schrockn accepted D7407: advanced tutorial changes.

👍🏻

Apr 15 2021, 4:17 PM
schrockn resigned from D7374: execution context naming revamp, remove resources and intermediate storage from run worker.

Super excited for this change. Big step forward. Thanks @cdecarolis for working through this. Will let others approve.

Apr 15 2021, 4:16 PM
schrockn closed D7452: Remove pyrsistent.
Apr 15 2021, 2:52 PM
schrockn committed R1:d506da7796c5: Remove pyrsistent (authored by schrockn).
Remove pyrsistent
Apr 15 2021, 2:52 PM
schrockn updated the diff for D7452: Remove pyrsistent.

up

Apr 15 2021, 2:37 PM
schrockn requested review of D7452: Remove pyrsistent.
Apr 15 2021, 2:28 PM
schrockn closed D7448: Move upload_logs to dagit.
Apr 15 2021, 2:02 PM
schrockn committed R1:efe944ac9b4e: Move upload_logs to dagit (authored by schrockn).
Move upload_logs to dagit
Apr 15 2021, 2:02 PM
schrockn updated the diff for D7448: Move upload_logs to dagit.

up

Apr 15 2021, 1:36 PM
schrockn updated the diff for D7448: Move upload_logs to dagit.

up

Apr 15 2021, 1:18 PM
schrockn requested review of D7448: Move upload_logs to dagit.
Apr 15 2021, 1:06 PM

Apr 14 2021

schrockn accepted D7376: make extra resources config optional.

I'd love to get this in for tmrw's release. Feel free to take stab at my proposed refactor or not. Your discretion.

Apr 14 2021, 4:59 PM

Apr 13 2021

schrockn added a comment to D7406: [dagstermill] support Failure and RetryRequested.

To be clear in vanilla solids you still have to raise these right?

Apr 13 2021, 5:12 PM

Apr 12 2021

schrockn added a comment to D7376: make extra resources config optional.

Very exciting.

Apr 12 2021, 4:28 PM

Apr 10 2021

schrockn closed D7385: Unpin pandas.
Apr 10 2021, 6:50 PM
schrockn committed R1:75fde1e43c9d: Unpin pandas (authored by schrockn).
Unpin pandas
Apr 10 2021, 6:49 PM
schrockn requested review of D7385: Unpin pandas.
Apr 10 2021, 3:27 PM

Apr 4 2021

schrockn closed D7296: Remove no-op input defs.
Apr 4 2021, 7:47 PM
schrockn committed R1:ee999454282c: Remove no-op input defs (authored by schrockn).
Remove no-op input defs
Apr 4 2021, 7:47 PM
schrockn requested review of D7296: Remove no-op input defs.
Apr 4 2021, 7:11 PM

Mar 30 2021

schrockn accepted D7212: [dagit] Repo filter feedback.

lgtm. thanks!

Mar 30 2021, 3:48 PM

Mar 25 2021

schrockn requested review of D7146: Fix typos and alter some working in the Solids concept section.
Mar 25 2021, 7:14 PM
schrockn closed D7141: Unpin GE.
Mar 25 2021, 5:31 PM
schrockn committed R1:425e9f8acb76: Unpin GE (authored by schrockn).
Unpin GE
Mar 25 2021, 5:31 PM
schrockn added a comment to D7141: Unpin GE.

From https://github.com/dagster-io/dagster/issues/3319

Mar 25 2021, 2:07 PM
schrockn updated the diff for D7141: Unpin GE.

up

Mar 25 2021, 2:06 PM
schrockn requested review of D7141: Unpin GE.
Mar 25 2021, 1:53 PM

Mar 22 2021

schrockn accepted D7063: infer solid description from fn docstring if description is not set.

Slightly concerned about possible weird aesthetics of whitespace depending on how the docstrings are slurped and how we handle whitespace on the client. But given that we do this for inputs it makes sense

Mar 22 2021, 8:28 PM

Mar 19 2021

schrockn accepted D7078: [dynamic] fix multiprocess retries bug.
Mar 19 2021, 10:05 PM

Mar 18 2021

schrockn closed D7030: Make pass at concepts section.
Mar 18 2021, 4:04 PM
schrockn committed R1:aad48112714b: Make pass at concepts section (authored by schrockn).
Make pass at concepts section
Mar 18 2021, 4:04 PM
schrockn published D7031: Clean up integrations copy and add disclaimer about Airflow integration for review.
Mar 18 2021, 3:49 PM
schrockn added a comment to D6822: [execute_in_process improvements 2/n] provide resource instances, split out config and inputs..

Having both feels very painful. Why can't they be consolidated?

Mar 18 2021, 3:42 PM
schrockn requested review of D7030: Make pass at concepts section.
Mar 18 2021, 3:39 PM

Mar 17 2021

schrockn accepted D5115: don't error on solid type annotations that don't resolve to dagster types.

I think this is good and we should highlight it in the release notes. "You no longer need to use DagsterPythonObjectType or make_dagster_type_usable_as_WAIT_WHY_DO_I_NEED_TO_TYPE_THIS"

Mar 17 2021, 10:06 PM
schrockn added a comment to D6702: add evaluate_sensor util function for testing sensors.

ah ok. curious to hear @prha's thoughts on the diff btwn those two

Mar 17 2021, 6:58 PM
schrockn requested changes to D6400: type_based_io_manager.

q mgmt. this is very stale. if it is still in reviewable state bounce back and i'll review

Mar 17 2021, 6:57 PM
schrockn added a comment to D6702: add evaluate_sensor util function for testing sensors.

I think the context construction pathways are so intricate and tied to internal implementations that they should be considered private

Mar 17 2021, 6:52 PM
schrockn added a comment to D6702: add evaluate_sensor util function for testing sensors.

Yeah I think the core point of contention is whether or not the constructor for contexts should be considered private. I think they should. We could even go the point where we change *Context classes to be abstract classes only.

Mar 17 2021, 6:51 PM
schrockn requested changes to D6017: initial async solids support.

I support landing it. Can we add docs in the same diff (or another one concurrently?)

Mar 17 2021, 6:44 PM
schrockn requested changes to D6702: add evaluate_sensor util function for testing sensors.

I think this is good. My final question is what we should default the instance to

Mar 17 2021, 6:43 PM
schrockn requested changes to D6648: [RFC] pipeline contains a graph.

q mgmt

Mar 17 2021, 6:41 PM
schrockn accepted D6951: Collect support in annotations on the DAG and solid sidebar.

not perfect but better than nothing

Mar 17 2021, 6:41 PM
schrockn accepted D6970: new-repo -> new-project.
Mar 17 2021, 6:40 PM