Page MenuHomeElementl
Feed Advanced Search

Thu, Apr 15

schrockn requested changes to D7438: Revamp the configuration concept section.

minor changes but bouncing back to your queue

Thu, Apr 15, 4:24 PM
schrockn accepted D7407: advanced tutorial changes.

👍🏻

Thu, Apr 15, 4:17 PM
schrockn resigned from D7374: execution context naming revamp, remove resources and intermediate storage from run worker.

Super excited for this change. Big step forward. Thanks @cdecarolis for working through this. Will let others approve.

Thu, Apr 15, 4:16 PM
schrockn closed D7452: Remove pyrsistent.
Thu, Apr 15, 2:52 PM
schrockn committed R1:d506da7796c5: Remove pyrsistent (authored by schrockn).
Remove pyrsistent
Thu, Apr 15, 2:52 PM
schrockn updated the diff for D7452: Remove pyrsistent.

up

Thu, Apr 15, 2:37 PM
schrockn requested review of D7452: Remove pyrsistent.
Thu, Apr 15, 2:28 PM
schrockn closed D7448: Move upload_logs to dagit.
Thu, Apr 15, 2:02 PM
schrockn committed R1:efe944ac9b4e: Move upload_logs to dagit (authored by schrockn).
Move upload_logs to dagit
Thu, Apr 15, 2:02 PM
schrockn updated the diff for D7448: Move upload_logs to dagit.

up

Thu, Apr 15, 1:36 PM
schrockn updated the diff for D7448: Move upload_logs to dagit.

up

Thu, Apr 15, 1:18 PM
schrockn requested review of D7448: Move upload_logs to dagit.
Thu, Apr 15, 1:06 PM

Wed, Apr 14

schrockn accepted D7376: make extra resources config optional.

I'd love to get this in for tmrw's release. Feel free to take stab at my proposed refactor or not. Your discretion.

Wed, Apr 14, 4:59 PM

Tue, Apr 13

schrockn added a comment to D7406: [dagstermill] support Failure and RetryRequested.

To be clear in vanilla solids you still have to raise these right?

Tue, Apr 13, 5:12 PM

Mon, Apr 12

schrockn added a comment to D7376: make extra resources config optional.

Very exciting.

Mon, Apr 12, 4:28 PM

Sat, Apr 10

schrockn closed D7385: Unpin pandas.
Sat, Apr 10, 6:50 PM
schrockn committed R1:75fde1e43c9d: Unpin pandas (authored by schrockn).
Unpin pandas
Sat, Apr 10, 6:49 PM
schrockn requested review of D7385: Unpin pandas.
Sat, Apr 10, 3:27 PM

Sun, Apr 4

schrockn closed D7296: Remove no-op input defs.
Sun, Apr 4, 7:47 PM
schrockn committed R1:ee999454282c: Remove no-op input defs (authored by schrockn).
Remove no-op input defs
Sun, Apr 4, 7:47 PM
schrockn requested review of D7296: Remove no-op input defs.
Sun, Apr 4, 7:11 PM

Tue, Mar 30

schrockn accepted D7212: [dagit] Repo filter feedback.

lgtm. thanks!

Tue, Mar 30, 3:48 PM

Thu, Mar 25

schrockn requested review of D7146: Fix typos and alter some working in the Solids concept section.
Thu, Mar 25, 7:14 PM
schrockn closed D7141: Unpin GE.
Thu, Mar 25, 5:31 PM
schrockn committed R1:425e9f8acb76: Unpin GE (authored by schrockn).
Unpin GE
Thu, Mar 25, 5:31 PM
schrockn added a comment to D7141: Unpin GE.

From https://github.com/dagster-io/dagster/issues/3319

Thu, Mar 25, 2:07 PM
schrockn updated the diff for D7141: Unpin GE.

up

Thu, Mar 25, 2:06 PM
schrockn requested review of D7141: Unpin GE.
Thu, Mar 25, 1:53 PM

Mon, Mar 22

schrockn accepted D7063: infer solid description from fn docstring if description is not set.

Slightly concerned about possible weird aesthetics of whitespace depending on how the docstrings are slurped and how we handle whitespace on the client. But given that we do this for inputs it makes sense

Mon, Mar 22, 8:28 PM

Fri, Mar 19

schrockn accepted D7078: [dynamic] fix multiprocess retries bug.
Fri, Mar 19, 10:05 PM

Mar 18 2021

schrockn closed D7030: Make pass at concepts section.
Mar 18 2021, 4:04 PM
schrockn committed R1:aad48112714b: Make pass at concepts section (authored by schrockn).
Make pass at concepts section
Mar 18 2021, 4:04 PM
schrockn published D7031: Clean up integrations copy and add disclaimer about Airflow integration for review.
Mar 18 2021, 3:49 PM
schrockn added a comment to D6822: [execute_in_process improvements 2/n] provide resource instances, split out config and inputs..

Having both feels very painful. Why can't they be consolidated?

Mar 18 2021, 3:42 PM
schrockn requested review of D7030: Make pass at concepts section.
Mar 18 2021, 3:39 PM

Mar 17 2021

schrockn accepted D5115: don't error on solid type annotations that don't resolve to dagster types.

I think this is good and we should highlight it in the release notes. "You no longer need to use DagsterPythonObjectType or make_dagster_type_usable_as_WAIT_WHY_DO_I_NEED_TO_TYPE_THIS"

Mar 17 2021, 10:06 PM
schrockn added a comment to D6702: add evaluate_sensor util function for testing sensors.

ah ok. curious to hear @prha's thoughts on the diff btwn those two

Mar 17 2021, 6:58 PM
schrockn requested changes to D6400: type_based_io_manager.

q mgmt. this is very stale. if it is still in reviewable state bounce back and i'll review

Mar 17 2021, 6:57 PM
schrockn added a comment to D6702: add evaluate_sensor util function for testing sensors.

I think the context construction pathways are so intricate and tied to internal implementations that they should be considered private

Mar 17 2021, 6:52 PM
schrockn added a comment to D6702: add evaluate_sensor util function for testing sensors.

Yeah I think the core point of contention is whether or not the constructor for contexts should be considered private. I think they should. We could even go the point where we change *Context classes to be abstract classes only.

Mar 17 2021, 6:51 PM
schrockn requested changes to D6017: initial async solids support.

I support landing it. Can we add docs in the same diff (or another one concurrently?)

Mar 17 2021, 6:44 PM
schrockn requested changes to D6702: add evaluate_sensor util function for testing sensors.

I think this is good. My final question is what we should default the instance to

Mar 17 2021, 6:43 PM
schrockn requested changes to D6648: [RFC] pipeline contains a graph.

q mgmt

Mar 17 2021, 6:41 PM
schrockn accepted D6951: Collect support in annotations on the DAG and solid sidebar.

not perfect but better than nothing

Mar 17 2021, 6:41 PM
schrockn accepted D6970: new-repo -> new-project.
Mar 17 2021, 6:40 PM
schrockn accepted D6978: [docs] make Integrations a top-level section + rename lib to `dagster-*`.

huge fan thanks

Mar 17 2021, 6:40 PM
schrockn requested changes to D5126: RFC use custom yaml loader.

clearing out queue. what is the plan with this?

Mar 17 2021, 6:39 PM
schrockn added a comment to D5115: don't error on solid type annotations that don't resolve to dagster types.

I'd like to see docs changes in this diff or another one up simultaneously

Mar 17 2021, 6:39 PM
schrockn created Image Macro "caveman".
Mar 17 2021, 3:52 AM
schrockn requested changes to D5115: don't error on solid type annotations that don't resolve to dagster types.

I really want to land this. Just wanted to surface my concerns so you could address quickly

Mar 17 2021, 3:48 AM
schrockn accepted D6940: [graphql] Input.isDynamicCollect & Solid.isDynamicMapped.
Mar 17 2021, 1:17 AM
schrockn accepted D6902: [dynamic orchestration] collect support.

i think this turned out quite nicely.

Mar 17 2021, 1:12 AM

Mar 16 2021

schrockn added a comment to D6970: new-repo -> new-project.

makes sense to me

Mar 16 2021, 6:32 PM
schrockn accepted D6964: PipelineDefinition requires name.

noice

Mar 16 2021, 1:48 PM

Mar 15 2021

schrockn accepted D6953: [RFC] rename STEP_MATERIALIZATION to ASSET_MATERIALIZATION.

yes please this has been driving me crazy

Mar 15 2021, 10:49 PM
schrockn added inline comments to D6822: [execute_in_process improvements 2/n] provide resource instances, split out config and inputs..
Mar 15 2021, 3:26 PM

Mar 12 2021

schrockn added inline comments to D6822: [execute_in_process improvements 2/n] provide resource instances, split out config and inputs..
Mar 12 2021, 10:39 PM
schrockn requested changes to D6902: [dynamic orchestration] collect support.

This is relatively clear overall. My big concern is naming here. In particular there are a couple overlapping terms here that confuse dynamic collect and static list fan in.

Mar 12 2021, 10:33 PM

Mar 11 2021

schrockn added inline comments to D6822: [execute_in_process improvements 2/n] provide resource instances, split out config and inputs..
Mar 11 2021, 3:28 PM

Mar 5 2021

schrockn accepted D6807: CHANGES.md for unbreaking dagster-postgres.
Mar 5 2021, 3:10 PM

Mar 4 2021

schrockn added a comment to D6792: [dagit] Separate pipeline and snapshot links in Run header.

Nice. Would love this in the run list viewer too!

Mar 4 2021, 8:20 PM

Feb 19 2021

schrockn added a comment to D6572: [dagit] Use Python event types in log filter view.

screenshot looks good. some of those events are deprecated and we should consider not rendering them

Feb 19 2021, 7:17 PM

Feb 18 2021

schrockn added a comment to D6416: website top-level headlines update.

"My interpretation of the the feedback we got was that "data flow between reusable, logical components" is too abstract to convey useful information. I also think we should downplay reusability in our core messaging - in most cases, writing reusable solids is not the right pattern."

Feb 18 2021, 6:53 PM

Feb 15 2021

schrockn closed D6484: Update sensor docstring.
Feb 15 2021, 8:38 PM
schrockn committed R1:c304b494c4cc: Update sensor docstring (authored by schrockn).
Update sensor docstring
Feb 15 2021, 8:38 PM
schrockn requested review of D6484: Update sensor docstring.
Feb 15 2021, 5:22 PM

Feb 9 2021

schrockn added a comment to D6017: initial async solids support.

mixing await and yield is a bit odd. is that normative in python?

Feb 9 2021, 9:28 PM

Feb 2 2021

schrockn added a comment to D6244: Run each DagsterDaemon subclass in its own thread.

did you consider process-level isolation?

Feb 2 2021, 3:09 PM

Feb 1 2021

schrockn added a comment to D6226: Update the headers on the run table to call out Run ID vs Snapshot ID, pipeline more clearly.

Did we ever consider breaking tags into its own column?

Feb 1 2021, 5:38 PM

Jan 29 2021

schrockn added a comment to D6164: Remove Dauphin.

Sent via Superhuman ( https://sprh.mn/?vip=schrockn@elementl.com )

Jan 29 2021, 2:32 AM
schrockn added a comment to D6164: Remove Dauphin.

definitely think we should keep prefix like @dgibson. grepability is good

Jan 29 2021, 12:06 AM
schrockn resigned from D6164: Remove Dauphin.

Very much support getting rid of the registry. Thanks for doing this! Let's just try to mitigate risk as much as possible with a diff this large

Jan 29 2021, 12:05 AM

Jan 27 2021

schrockn added a comment to D6177: WIP: dagster new-repo Generated Output.

I meant code artifact, like "solid", "pipeline", "resource" etc

Jan 27 2021, 10:51 PM
schrockn added a comment to D6177: WIP: dagster new-repo Generated Output.

Agree with nearly all of sandy's and rex's feedback.

Jan 27 2021, 8:26 PM

Jan 24 2021

schrockn resigned from D6132: UI support for dynamic outputs, Run view state cleanup.

Very excited to see this land. Will defer to dish on the deets of the js.

Jan 24 2021, 12:12 AM

Jan 23 2021

schrockn added a comment to D6029: Flip asset materialization UI so newest time/partition is on the left.

adding @sandyryza here. I think this makes a lot of sense and I don't think having the events be left-to-right and the graphs right-to-left is a big problem but open to other's perspective

Jan 23 2021, 11:58 PM
schrockn added a reviewer for D6029: Flip asset materialization UI so newest time/partition is on the left: sandyryza.
Jan 23 2021, 11:57 PM
schrockn added a comment to D5115: don't error on solid type annotations that don't resolve to dagster types.

"I'm pretty resistant to directions that involve using type annotations for anything other than annotating the actual Python type that's expected somewhere. It's fairly abusive of a Python language feature, and it causes non-trivial trouble for anyone who's trying to use that language feature in the correct way."

Jan 23 2021, 7:22 PM
schrockn added a comment to D5115: don't error on solid type annotations that don't resolve to dagster types.

We could also get clever and try to reuse input_defs and output_defs supporting a few different variants:

Jan 23 2021, 1:30 PM
schrockn added a comment to D5115: don't error on solid type annotations that don't resolve to dagster types.

I'm concerned about too many breaking changes and too much thrash. But I think that the critical thing here would be coming up with a more condense syntax for inputs and outputs. Right now if we forced users to InputDefinition and OuptutDefinition for the dagster-type-only case the code explosion would be brutal.

Jan 23 2021, 1:27 PM

Jan 22 2021

schrockn added a comment to D5115: don't error on solid type annotations that don't resolve to dagster types.

Yeah my first instinct is stress and terror given our past forays into magically commingling the python and dagster type system.

Jan 22 2021, 11:36 PM

Jan 19 2021

schrockn added a comment to D6036: Make type_check_fn optional on DagsterType.

Here's a thought:

Jan 19 2021, 7:56 PM
schrockn abandoned D5321: Different scenarios to consider when doing input/output management.
Jan 19 2021, 4:39 PM

Jan 14 2021

schrockn accepted D6018: update partitions and backfills doc with screenshots of new UI.
Jan 14 2021, 10:57 PM
schrockn published D6018: update partitions and backfills doc with screenshots of new UI for review.
Jan 14 2021, 10:57 PM

Jan 13 2021

schrockn accepted D5951: [dagit] Add "Scheduled" tab to Runs page.
Jan 13 2021, 8:41 PM
schrockn requested changes to D5951: [dagit] Add "Scheduled" tab to Runs page.

can you display pipeline name in addition to schedule name?

Jan 13 2021, 7:27 PM
schrockn accepted D5721: DynamicOutput DSL map approach.

excited to see this land. I will let @prha look at the lint ish as I have little context on that

Jan 13 2021, 7:23 PM
schrockn added a comment to D5951: [dagit] Add "Scheduled" tab to Runs page.

could you throw up a screenshot?

Jan 13 2021, 4:06 PM

Jan 12 2021

schrockn added inline comments to D5829: Modernize dagstermill file handling.
Jan 12 2021, 11:11 PM
schrockn accepted D5834: RFC: fix result.result_for_solid when using mem_io_manager + default to mem_io_manager.

"hardcoded_resource_instances" is a pretty rough name since they aren't really hardcoded. they are not fixed in code. maybe "resource_instances_to_override" or something?

Jan 12 2021, 1:43 PM
schrockn resigned from D5474: ObjectStoreOperation/AssetStoreOperation -> HandledOutput/LoadedInput.

Seems good. Will let others do a detailed review!

Jan 12 2021, 1:35 PM

Jan 8 2021

schrockn accepted D5858: Update instance docs to reflect 0.10.0 changes.

startreknod

Jan 8 2021, 10:41 PM
schrockn accepted D5812: Add initial overview docs for sensors.
Jan 8 2021, 1:02 PM
schrockn requested changes to D5858: Update instance docs to reflect 0.10.0 changes.
Jan 8 2021, 12:59 PM
schrockn accepted D5862: Run launcher docs refresh for 0.10.0.
Jan 8 2021, 12:57 PM
schrockn requested changes to D5843: root input manager.

just doing q mgmt. any other alternative names?

Jan 8 2021, 12:55 PM
schrockn added inline comments to D5843: root input manager.
Jan 8 2021, 12:55 PM
schrockn requested changes to D5721: DynamicOutput DSL map approach.

q mgmt.

Jan 8 2021, 12:50 PM
schrockn requested changes to D5613: RFC DynamicOutput DSL __iter__ approach.

is this still a relevant diff? req'ing changes for q mgmt

Jan 8 2021, 12:50 PM