Page MenuHomePhabricator

schrockn (Nick Schrock)
UserAdministrator

Projects

User does not belong to any projects.

User Details

User Since
Mar 20 2019, 8:23 PM (78 w, 4 d)
Roles
Administrator

Recent Activity

Tue, Sep 15

schrockn accepted D4432: Disable flaky termination test.
Tue, Sep 15, 7:35 PM

Mon, Sep 14

schrockn requested changes to D4427: pyspark blog post.

send this up to github so we get the preview

Mon, Sep 14, 8:30 PM

Fri, Sep 11

schrockn updated the diff for D4373: RFC: Check that all definition names are invalid and warn.

up

Fri, Sep 11, 11:50 PM
schrockn requested changes to D4401: dagstermill context.solid_config has non-deterministic behavior.

Let's add a test for this?

Fri, Sep 11, 2:59 PM
schrockn added a comment to D4379: (config-editor-2) Add button to automatically remove extra config.

I think this feature might be a good one to put in front of dish next week. While this is incrementally good, I bet we can do a deeper rethink to implement this more elegantly.

Fri, Sep 11, 12:58 PM

Thu, Sep 10

schrockn updated the diff for D4373: RFC: Check that all definition names are invalid and warn.

up

Thu, Sep 10, 11:47 PM
schrockn added a comment to D4384: #2887 enable subset reexecution when part of selection failed.

This is cool

Thu, Sep 10, 11:41 PM
schrockn updated the diff for D4373: RFC: Check that all definition names are invalid and warn.

up

Thu, Sep 10, 11:28 PM
schrockn updated the diff for D4373: RFC: Check that all definition names are invalid and warn.

up

Thu, Sep 10, 11:19 PM
schrockn updated the diff for D2155: Warm the BK cluster.

up

Thu, Sep 10, 11:11 PM
schrockn updated the diff for D4373: RFC: Check that all definition names are invalid and warn.

up

Thu, Sep 10, 11:04 PM
schrockn updated the diff for D4373: RFC: Check that all definition names are invalid and warn.

up

Thu, Sep 10, 11:02 PM
schrockn updated the diff for D4373: RFC: Check that all definition names are invalid and warn.

up

Thu, Sep 10, 10:28 PM
schrockn resigned from D4332: selector-2 reexecute_pipeline support DSL.

seems quite reasonable to add. will let @alangenfeld handle his objections :-)

Thu, Sep 10, 9:42 PM
schrockn accepted D4255: add long description to setup.py.
Thu, Sep 10, 9:40 PM
schrockn resigned from D4199: RFC: @dagster_type decorator.
Thu, Sep 10, 9:40 PM
schrockn resigned from D3848: lakehouse generate asset materializations.

clearing my queue. readd if you start pushing on this again

Thu, Sep 10, 9:40 PM
schrockn abandoned D4115: Remove notebook_solid wrapper.
Thu, Sep 10, 9:39 PM
schrockn added a comment to D4373: RFC: Check that all definition names are invalid and warn.

yup. should we do a warning for a bit?

Thu, Sep 10, 9:38 PM
schrockn accepted D4389: [hotfix] fix create_databricks_job_solid.
Thu, Sep 10, 6:12 PM
schrockn published D4389: [hotfix] fix create_databricks_job_solid for review.
Thu, Sep 10, 6:12 PM

Wed, Sep 9

schrockn added a comment to D4374: Make DagsterInstance a contextmanager for resource cleanup.

I would also strongly support renaming DagsterInstance.get to deter its usage.

Wed, Sep 9, 2:15 PM
schrockn requested review of D4373: RFC: Check that all definition names are invalid and warn.
Wed, Sep 9, 1:26 PM

Wed, Sep 2

schrockn accepted D4319: [buildkite] update images.
Wed, Sep 2, 3:23 PM

Tue, Sep 1

schrockn accepted D4315: Mark pandas structured validation APIs as experimental.

yeah agree that testing for every experiment seems like a bit much, although we could write a test helper that would make the trivial

Tue, Sep 1, 3:27 PM
schrockn accepted D4314: readme top-level messaging.
Tue, Sep 1, 3:22 PM

Fri, Aug 28

schrockn added inline comments to D4285: [trigger-3] Add graphql endpoint to trigger execution based off trigger name.
Fri, Aug 28, 7:00 PM

Wed, Aug 26

schrockn added inline comments to D4285: [trigger-3] Add graphql endpoint to trigger execution based off trigger name.
Wed, Aug 26, 2:32 AM
schrockn added a comment to D4283: [trigger-1] define triggered execution in repository.

So one thing here. 1) I don't think we should include this in the top-level includes and 2) I think we should mark it as experimenetal

Wed, Aug 26, 2:31 AM

Tue, Aug 25

schrockn accepted D4280: disable core_tests on py2+windows.
Tue, Aug 25, 6:22 PM

Mon, Aug 24

schrockn accepted D4216: [docs] Releases and deprecations policy.

cool. please look at final comments!

Mon, Aug 24, 7:13 PM

Aug 20 2020

schrockn added a comment to D4231: Add GraphQL API Docs.

Ha no problem. I'm the worst when it comes to typos.

Aug 20 2020, 2:14 PM
schrockn accepted D4231: Add GraphQL API Docs.

I would copy and paste into google docs or something for a spell check

Aug 20 2020, 2:08 PM

Aug 19 2020

schrockn accepted D4217: update top-level messaging in README.
Aug 19 2020, 9:19 PM
schrockn accepted D4214: Don't include internal tags in dagster-graphql.

much more clear thank

Aug 19 2020, 9:15 PM
schrockn added a comment to D4216: [docs] Releases and deprecations policy.

What about language like "we will introduce deprecations every four releases (or so), and there will be a minimum of four releases between deprecation and removal" or something to that effect?

Aug 19 2020, 9:15 PM
schrockn resigned from D4225: Add more out-of-process execution coverage to dagster-graphql tests.
Aug 19 2020, 8:40 PM
schrockn requested changes to D4216: [docs] Releases and deprecations policy.

Big point of discussion is are we going to unconditionally up the minor version every 4 weeks. Or should we just mark/determine that approximately every 4 dot releases we can introduce deprecations?

Aug 19 2020, 8:40 PM
schrockn requested changes to D4214: Don't include internal tags in dagster-graphql.

time to bikeshed!

Aug 19 2020, 8:36 PM

Aug 18 2020

schrockn requested changes to D4214: Don't include internal tags in dagster-graphql.

makes sense. test?

Aug 18 2020, 9:57 PM
schrockn added a comment to D4198: [partition-backfill-6] ui for partitions backfill.

A few pieces of feedback/questions:

Aug 18 2020, 7:10 PM
schrockn added a comment to D4188: RFC: Composable Types with Materializers as first example.

I can't wait for the project that deems itself a platform for domain-specific package managers.

Aug 18 2020, 3:55 AM
schrockn added a comment to D4188: RFC: Composable Types with Materializers as first example.

How does Spark dynamically register new things? E.g. if you "install" delta lake does it dynamically register a new format?

Aug 18 2020, 3:07 AM

Aug 17 2020

schrockn requested changes to D4171: first run of hooks on expectations.

This is a case where I think our "list" approach fails us and we may want to consider a different type of hook and processes one event at a time. For hooks of this nature I'm concerned about long-running computations where getting the events only at the end will feel like a bug. We've spent a bunch of effort making the whole core streaming so want to keep that going.

Aug 17 2020, 2:31 PM

Aug 14 2020

schrockn requested review of D4188: RFC: Composable Types with Materializers as first example.
Aug 14 2020, 6:09 PM

Aug 13 2020

schrockn accepted D4172: update messaging in setup.py.
Aug 13 2020, 8:02 PM
schrockn accepted D3955: [docs] intro page with code example and screenshots.
Aug 13 2020, 5:31 PM

Aug 12 2020

schrockn added a comment to D4163: Turn on auto-formatting for string literals in black.

yeah you're right. i vote for switching. i like autoformatting so much that i sometimes purposefully screw it up so it autofixes and I get the emotional satisfication

Aug 12 2020, 5:05 PM
schrockn added inline comments to D4151: Throw from sync_get_external_execution_plan rather than relying on callsites to check error data.
Aug 12 2020, 12:41 AM

Aug 7 2020

schrockn accepted D4133: make ipc errors easier to read.
Aug 7 2020, 6:25 PM
schrockn accepted D4135: Disable tests currently breaking Windows.
Aug 7 2020, 4:25 PM

Aug 6 2020

schrockn added a comment to D4119: [docs] examples/hooks.

I might be wrong here since the hardcoded slack token won't work anyways hmmm

Aug 6 2020, 9:34 PM
schrockn accepted D4036: migration guide.
Aug 6 2020, 9:31 PM
schrockn added a comment to D4121: Just for looking: This is an example of some helpers we could use to help mock resources.

Sent via Superhuman ( https://sprh.mn/?vip=schrockn@elementl.com )

Aug 6 2020, 9:24 PM
schrockn added inline comments to D4119: [docs] examples/hooks.
Aug 6 2020, 9:21 PM
schrockn added a comment to D4121: Just for looking: This is an example of some helpers we could use to help mock resources.

Mock is new and hardcoded is not. However I do think we should consider porting to resource for improved ergonomics, although it is more non-standard and documentable so there are tradeoffs

Aug 6 2020, 8:47 PM
schrockn accepted D4102: default working directory for python file targets.
Aug 6 2020, 6:37 PM
schrockn added a comment to D4121: Just for looking: This is an example of some helpers we could use to help mock resources.

@yuhan feel free to commandeer if that is helpful

Aug 6 2020, 4:37 PM
schrockn added a comment to D4114: Add retries for flakey integration tests.

🙏🏻

Aug 6 2020, 4:25 PM
schrockn added inline comments to D4119: [docs] examples/hooks.
Aug 6 2020, 4:24 PM
schrockn requested review of D4121: Just for looking: This is an example of some helpers we could use to help mock resources.
Aug 6 2020, 1:58 PM
schrockn added inline comments to D4119: [docs] examples/hooks.
Aug 6 2020, 1:07 PM
schrockn requested review of D4115: Remove notebook_solid wrapper.
Aug 6 2020, 12:36 AM
schrockn committed R1:1e89eec6e59e: rm gevent from dagster (authored by alangenfeld).
rm gevent from dagster
Aug 6 2020, 12:19 AM
schrockn closed D4074: rm gevent from dagster.
Aug 6 2020, 12:19 AM

Aug 5 2020

schrockn added a comment to D4102: default working directory for python file targets.

Cool. I think it's a toss up so let's go with that one and see what user feedback is

Aug 5 2020, 10:51 PM
schrockn accepted D4061: Move launch_pipeline CLI commands to not run any user code.

makes sense to me

Aug 5 2020, 9:16 PM
schrockn added inline comments to D4107: step-selection-1 parse step selection.
Aug 5 2020, 7:09 PM
schrockn added a comment to D4102: default working directory for python file targets.

The following problem made me consider the defaults when I was prepping a new workspace.yaml to load in several examples at the same time.

Aug 5 2020, 6:29 PM
schrockn accepted D4106: Update CHANGES.md about config removal.
Aug 5 2020, 6:18 PM
schrockn added a comment to D4074: rm gevent from dagster.

@alangenfeld any real risk with this one? if not can @prha, @dgibson or @max accept? thanks

Aug 5 2020, 6:07 PM
schrockn commandeered D4074: rm gevent from dagster.
Aug 5 2020, 6:06 PM
schrockn accepted D4014: solid-hook-3 hooks can decorate pipeline defs.

Great stuff. Please consider final comments as I think it makes code clearer. But your call!

Aug 5 2020, 4:55 PM
schrockn accepted D4071: Move schedule backfill command to use External classes instead of user code classes.

cool.

Aug 5 2020, 4:36 PM
schrockn requested changes to D4061: Move launch_pipeline CLI commands to not run any user code.

This looks very good overall. I just wanted to make sure you saw the feedback around getting rid of DagsterInstance.get() calls. I could be missing something, but seems possible to eliminate them here

Aug 5 2020, 4:23 PM
schrockn added a comment to D4102: default working directory for python file targets.

Also for python_file targets, what do you think about defaulting to the directory where that python files lives?

Aug 5 2020, 1:43 PM
schrockn added a comment to D4102: default working directory for python file targets.

Can you give a little more context on what the user-behavior facing ends up being in the diff?

Aug 5 2020, 1:43 PM

Aug 4 2020

schrockn added a comment to D4091: Full support for Python 3.8.

❤️

Aug 4 2020, 11:43 PM
schrockn accepted D3979: solid-hook-2 apply hooks on solid instances.

excellent

Aug 4 2020, 10:27 PM
schrockn accepted D3978: RFC: add origin information into the repository picker.
Aug 4 2020, 9:13 PM
schrockn requested changes to D3979: solid-hook-2 apply hooks on solid instances.
Aug 4 2020, 8:25 PM
schrockn accepted D4083: Split test_cli_commands.py into its own BK test.
Aug 4 2020, 8:19 PM
schrockn accepted D3977: solid-hook-1 @hook and HookDefinition.

Great stuff. Just a couple minor comments but excited to land this

Aug 4 2020, 8:18 PM
schrockn added a comment to D4014: solid-hook-3 hooks can decorate pipeline defs.

Per in-person convo:

Aug 4 2020, 6:51 PM
schrockn requested changes to D4054: experimental warning.

Third option seems fine. "Users" is just going to be the team and core contributors, so it seems pretty straightfoward to manage and to easily change. Back to your queue

Aug 4 2020, 4:49 PM
schrockn added inline comments to D3977: solid-hook-1 @hook and HookDefinition.
Aug 4 2020, 4:12 PM
schrockn requested changes to D3979: solid-hook-2 apply hooks on solid instances.

Code is great. Just question on naming

Aug 4 2020, 4:10 PM
schrockn requested changes to D4014: solid-hook-3 hooks can decorate pipeline defs.

Super clean API. This turned out very nicely.

Aug 4 2020, 4:08 PM
schrockn accepted D4054: experimental warning.

please heed final comments

Aug 4 2020, 4:00 PM
schrockn requested changes to D3977: solid-hook-1 @hook and HookDefinition.

ok almost there. most critically we definitely don't want to be mutating the context.

Aug 4 2020, 3:56 PM

Aug 3 2020

schrockn committed R1:e25fa6599359: Undeprecate create_dagster_pandas_dataframe_type (authored by schrockn).
Undeprecate create_dagster_pandas_dataframe_type
Aug 3 2020, 11:47 PM
schrockn closed D4075: Undeprecate create_dagster_pandas_dataframe_type.
Aug 3 2020, 11:47 PM
schrockn requested review of D4075: Undeprecate create_dagster_pandas_dataframe_type.
Aug 3 2020, 11:29 PM
schrockn added inline comments to D4022: refactor configured implementation + more docs.
Aug 3 2020, 6:26 PM
schrockn accepted D4056: pull back dagster_type arg to asset materializations.
Aug 3 2020, 5:26 PM

Jul 31 2020

schrockn added inline comments to D4054: experimental warning.
Jul 31 2020, 9:46 PM
schrockn added inline comments to D4054: experimental warning.
Jul 31 2020, 9:45 PM
schrockn added a comment to D4054: experimental warning.

I was also thinking that we would have some config or some way to disable the experimentation warnings for those want to opt into this stuff

Jul 31 2020, 8:59 PM
schrockn added a comment to D4054: experimental warning.

__init__ seems fine

Jul 31 2020, 8:58 PM
schrockn added a comment to D4054: experimental warning.

for classes would be annotate the __init__ method?

Jul 31 2020, 6:59 PM