Page MenuHomePhabricator

schrockn (Nick Schrock)
UserAdministrator

Projects

User does not belong to any projects.

User Details

User Since
Mar 20 2019, 8:23 PM (22 w, 1 d)
Roles
Administrator

Recent Activity

Yesterday

schrockn accepted D844: Initial pass at custom modification of the Dagre layout for wide pipelines.

accepting because of fun tests in other diff

Thu, Aug 22, 9:44 PM
schrockn added a comment to D875: Add test coverage of the pipeline to svg process using svg snapshots.
  1. We should definitely configure phab to hide those huge data files
  2. This is really cool! Can you post a screenshot of the visual diff? Really curious what that looks like.
Thu, Aug 22, 9:04 PM
schrockn added a comment to D874: RFC: Remove lambda_solid from tutorial.

Yeah agree that it's not a slam dunk

Thu, Aug 22, 9:01 PM
schrockn created D874: RFC: Remove lambda_solid from tutorial.
Thu, Aug 22, 6:53 PM
schrockn accepted D872: Fix FilesystemRunStorage instantiations.
Thu, Aug 22, 6:15 PM
schrockn accepted D870: [storage] handle missing event log files.

Actually I'm quite wrong given that the EventLogStorage class in question has context about the underlying store.

Thu, Aug 22, 5:44 PM
schrockn requested changes to D870: [storage] handle missing event log files.
Thu, Aug 22, 5:43 PM
schrockn requested changes to D851: RFC: Add support for specifying Dask resources.
Thu, Aug 22, 3:13 AM
schrockn added a comment to D851: RFC: Add support for specifying Dask resources.

agree on deciding on some sort of namespacing scheme here. i bias towards {'dask': {'resource_requirements': {...}}} option because it is inline with our current config system. but it does say something the software people like (e.g. vscode and k8s) using the in-key namespacing scheme

Thu, Aug 22, 3:13 AM

Wed, Aug 21

schrockn accepted D861: [RFC] simpler user defined types section.

Sauce

Wed, Aug 21, 11:01 PM

Tue, Aug 20

schrockn accepted D849: Fix SFTP solid test.
Tue, Aug 20, 10:31 PM
schrockn added a comment to D841: Consolidate double implementation of run storage.

This is a solid step forward. I'd love for you to comment on this diff re: what next steps are in terms of reining it in. I assume we want to get rid of the PipelineRun hierarchy?

Tue, Aug 20, 9:27 PM
schrockn accepted D842: Create ScheduleDefinition and dagster CLI scheduler commands.

This is looking good. Per discussion in follow ons let's extract dagster-cron (or whatever we call it) into its own module so that the core library is not tied to the cron scheduler.

Tue, Aug 20, 9:17 PM
schrockn added inline comments to D842: Create ScheduleDefinition and dagster CLI scheduler commands.
Tue, Aug 20, 9:12 PM
schrockn accepted D847: [dagster] fix composition bug.

ya this seems nicer

Tue, Aug 20, 9:09 PM
schrockn accepted D848: Add systemd service for dagit.

Exciting stuff. I barely know anything about systems-level linux issues, so I can't really give substantive feedback on high level stuff, but this all seems reasonable.

Tue, Aug 20, 9:07 PM
schrockn requested changes to D848: Add systemd service for dagit.

This is really exciting to see. Just have the request about restart latency.

Tue, Aug 20, 9:02 PM
schrockn requested changes to D844: Initial pass at custom modification of the Dagre layout for wide pipelines.

I'd also love some screenshots from pipelines generated with https://dagster.phacility.com/D846

Tue, Aug 20, 8:59 PM
schrockn accepted D847: [dagster] fix composition bug.

per IRL discussion let's do some better names and comments because this is complicated, but looks good

Tue, Aug 20, 8:05 PM
schrockn committed R1:bb56afdc0522: Configurable fan-in/fan-out pipeline for testing rendering (authored by schrockn).
Configurable fan-in/fan-out pipeline for testing rendering
Tue, Aug 20, 7:44 PM
schrockn closed D846: Configurable fan-in/fan-out pipeline for testing rendering.
Tue, Aug 20, 7:44 PM
schrockn updated the diff for D846: Configurable fan-in/fan-out pipeline for testing rendering.

up

Tue, Aug 20, 6:21 PM
schrockn added a comment to D846: Configurable fan-in/fan-out pipeline for testing rendering.

Result of construct_level_pipeline('fan_in_fan_out_pipeline', levels=10, fanout=50)

Tue, Aug 20, 6:18 PM
schrockn created D846: Configurable fan-in/fan-out pipeline for testing rendering.
Tue, Aug 20, 6:17 PM

Mon, Aug 19

schrockn accepted D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

Let's get this merged. Please heed final feedback

Mon, Aug 19, 11:38 PM
schrockn added a comment to D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.

seems good now. will let @alangenfeld approve

Mon, Aug 19, 11:20 PM
schrockn committed R1:1c91bd265ff4: Temporarily pin ge to 0.7.6 or less (authored by schrockn).
Temporarily pin ge to 0.7.6 or less
Mon, Aug 19, 9:04 PM
schrockn closed D840: Temporarily pin ge to 0.7.6 or less.
Mon, Aug 19, 9:04 PM
schrockn abandoned D839: Revert "Add initial sqllite run storage".
Mon, Aug 19, 8:58 PM
schrockn created D840: Temporarily pin ge to 0.7.6 or less.
Mon, Aug 19, 8:56 PM
schrockn added a reverting change for R1:b938ef97e0c8: Add initial sqllite run storage: D839: Revert "Add initial sqllite run storage".
Mon, Aug 19, 8:46 PM
schrockn added a reverting change for D830: Add initial sqllite run storage: D839: Revert "Add initial sqllite run storage".
Mon, Aug 19, 8:46 PM
schrockn created D839: Revert "Add initial sqllite run storage".
Mon, Aug 19, 8:46 PM
schrockn committed R1:b938ef97e0c8: Add initial sqllite run storage (authored by schrockn).
Add initial sqllite run storage
Mon, Aug 19, 8:31 PM
schrockn closed D830: Add initial sqllite run storage.
Mon, Aug 19, 8:31 PM
schrockn accepted D835: [dagster] whitelisted namedtuple serdes.

decorator is nice

Mon, Aug 19, 8:29 PM
schrockn accepted D829: [dagster] remove event_callback from RunConfig.
Mon, Aug 19, 8:25 PM

Sun, Aug 18

schrockn added inline comments to D830: Add initial sqllite run storage.
Sun, Aug 18, 7:24 PM
schrockn updated the diff for D830: Add initial sqllite run storage.

conn instead of provider

Sun, Aug 18, 6:46 PM

Sat, Aug 17

schrockn added inline comments to D826: Implement start_schedule and end_schedule for SystemCronTabScheduler.
Sat, Aug 17, 2:41 PM

Fri, Aug 16

schrockn committed R1:69f8dcb78787: Add dbt test solid (authored by schrockn).
Add dbt test solid
Fri, Aug 16, 9:59 PM
schrockn closed D821: Add dbt test solid.
Fri, Aug 16, 9:59 PM
schrockn added a comment to D831: RFC: push config obj instantiation into `reconstitute_pipeline_context`.

lovely

Fri, Aug 16, 4:26 PM
schrockn added a comment to D833: Update Flask dependency.

So will limit what versions of airflow we can integrate with? Iā€™m a little hazy on how this all coheres to a flexible sane strategy around deps

Fri, Aug 16, 2:57 PM
schrockn updated subscribers of D831: RFC: push config obj instantiation into `reconstitute_pipeline_context`.

This looks great to me. I'll let @max approve to resolve your question re: the use/non-use of raise_from.

Fri, Aug 16, 1:41 PM
schrockn accepted D833: Update Flask dependency.
Fri, Aug 16, 1:38 PM
schrockn updated the diff for D821: Add dbt test solid.

up

Fri, Aug 16, 1:36 PM

Thu, Aug 15

schrockn updated the diff for D830: Add initial sqllite run storage.

self review

Thu, Aug 15, 10:51 PM
schrockn updated the diff for D830: Add initial sqllite run storage.

py2 floats

Thu, Aug 15, 10:48 PM
schrockn updated the diff for D830: Add initial sqllite run storage.

py2 floats

Thu, Aug 15, 10:45 PM
schrockn updated the diff for D830: Add initial sqllite run storage.

up

Thu, Aug 15, 10:41 PM
schrockn added a comment to D830: Add initial sqllite run storage.

@alangenfeld curious about what you think should be our standard for passing around connection strings given the py2-only bs you ran into.

Thu, Aug 15, 10:40 PM
schrockn created D830: Add initial sqllite run storage.
Thu, Aug 15, 10:39 PM
schrockn accepted D824: [dagster] EventSink.

This is quite lovely. @max should take a look prior to merging but this looks great.

Thu, Aug 15, 9:01 PM
schrockn added a comment to D815: Promote some step events to INFO.

cool

Thu, Aug 15, 5:06 PM
schrockn accepted D822: fix required resource boolean.
Thu, Aug 15, 5:05 PM
schrockn added a comment to D815: Promote some step events to INFO.

can you screenshot output of hello world tutorial?

Thu, Aug 15, 4:58 PM
schrockn accepted D815: Promote some step events to INFO.
Thu, Aug 15, 4:58 PM
schrockn updated the diff for D821: Add dbt test solid.

up

Thu, Aug 15, 4:36 PM
schrockn created D821: Add dbt test solid.
Thu, Aug 15, 4:29 PM
schrockn committed R1:dfa0e8210378: Throwing up example yaml-driven deps (authored by schrockn).
Throwing up example yaml-driven deps
Thu, Aug 15, 12:28 AM
schrockn closed D813: Throwing up example yaml-driven deps.
Thu, Aug 15, 12:28 AM

Wed, Aug 14

schrockn updated the diff for D813: Throwing up example yaml-driven deps.

rebase

Wed, Aug 14, 11:30 PM
schrockn updated the diff for D813: Throwing up example yaml-driven deps.

comment

Wed, Aug 14, 11:29 PM
schrockn accepted D809: Display required_resources in dagit.

this magically fixed itself. still not sure what happened.

Wed, Aug 14, 11:22 PM
schrockn accepted D820: fix misc JS console warnings.
Wed, Aug 14, 10:36 PM
schrockn requested changes to D809: Display required_resources in dagit.

I'm getting this error in a dev build with this diff patched.

Wed, Aug 14, 6:11 PM
schrockn added a comment to D815: Promote some step events to INFO.

what does dagster cli look like on hello world?

Wed, Aug 14, 6:06 PM
schrockn added a reviewer for D813: Throwing up example yaml-driven deps: Restricted Project.
Wed, Aug 14, 6:04 PM
schrockn updated the diff for D813: Throwing up example yaml-driven deps.

black

Wed, Aug 14, 4:52 PM
schrockn updated the diff for D813: Throwing up example yaml-driven deps.

py2 and test

Wed, Aug 14, 4:50 PM
schrockn added a comment to D815: Promote some step events to INFO.

Maybe we should have some be debug and some be info

Wed, Aug 14, 3:42 PM
schrockn requested changes to D815: Promote some step events to INFO.

Hmmm. I think maybe we should upgrade the emitted system events to INFO instead?

Wed, Aug 14, 3:22 PM
schrockn accepted D814: Keep the abspath to repository.yaml around in ExecutionTargetHandle.

would be cool to add a test that catches this

Wed, Aug 14, 3:20 PM
schrockn created D813: Throwing up example yaml-driven deps.
Wed, Aug 14, 2:09 PM

Tue, Aug 13

schrockn requested changes to D809: Display required_resources in dagit.

So think we might to have this be a row-per required resource. That way we can add things like descriptions or metadata to the resource requirement.

Tue, Aug 13, 7:37 PM
schrockn accepted D808: Actually preserve the original config dict.

nice very surgical

Tue, Aug 13, 7:36 PM
schrockn added inline comments to D791: Create Scheduler API.
Tue, Aug 13, 6:18 PM
schrockn added inline comments to D791: Create Scheduler API.
Tue, Aug 13, 6:16 PM
schrockn updated the summary of D795: Initial dagster-dbt prototype.
Tue, Aug 13, 6:06 PM
schrockn added a comment to D795: Initial dagster-dbt prototype.

Here is a video of this in action.

Tue, Aug 13, 6:06 PM
schrockn added inline comments to D791: Create Scheduler API.
Tue, Aug 13, 6:01 PM
schrockn accepted D759: Create script to generate library.

cool

Tue, Aug 13, 5:50 PM
schrockn added inline comments to D759: Create script to generate library.
Tue, Aug 13, 5:50 PM
schrockn accepted D801: Initial k8s operator and split out of shared functionality.

great let's move forward. seems like next critical thing here will be to set up an integration test

Tue, Aug 13, 5:49 PM
schrockn committed R1:1894d3397d81: disable coverage (authored by schrockn).
disable coverage
Tue, Aug 13, 3:09 PM
schrockn closed D806: disable coverage.
Tue, Aug 13, 3:09 PM
schrockn updated the diff for D806: disable coverage.

up

Tue, Aug 13, 2:53 PM
schrockn created D806: disable coverage.
Tue, Aug 13, 2:51 PM
schrockn accepted D805: Add dagster-dbt to install_dev_python_modules.

thx

Tue, Aug 13, 2:47 PM
schrockn requested changes to D801: Initial k8s operator and split out of shared functionality.

this looks reasonable overall. if python modules were easier to manage for us i might push for a separate module for this but meh for now.

Tue, Aug 13, 3:23 AM
schrockn resigned from D791: Create Scheduler API.

just popping off of my queue

Tue, Aug 13, 3:12 AM
schrockn added a comment to D791: Create Scheduler API.

seems like a good start. will let @alangenfeld do final approval as you two have been discussing

Tue, Aug 13, 3:12 AM
schrockn accepted D802: Update docs and changelog for 0.5.6.

makeitso

Tue, Aug 13, 3:08 AM
schrockn committed R1:f545529d3bdb: Initial dagster-dbt prototype (authored by schrockn).
Initial dagster-dbt prototype
Tue, Aug 13, 2:05 AM
schrockn closed D795: Initial dagster-dbt prototype.
Tue, Aug 13, 2:05 AM

Mon, Aug 12

schrockn accepted D800: markdown support in event metadata.

šŸ‘šŸ»

Mon, Aug 12, 9:42 PM
schrockn updated the diff for D795: Initial dagster-dbt prototype.

up

Mon, Aug 12, 9:38 PM
schrockn updated the diff for D795: Initial dagster-dbt prototype.

up

Mon, Aug 12, 9:29 PM
schrockn added inline comments to D791: Create Scheduler API.
Mon, Aug 12, 9:25 PM