Page MenuHomeElementl
Feed All Stories

Today

sashank accepted D7899: Fix race condition when reload workspace is called from a background thread.
Thu, May 13, 6:49 PM
sashank published D7899: Fix race condition when reload workspace is called from a background thread for review.
Thu, May 13, 6:48 PM
dish requested review of D7897: [dagit] Use Yarn v2.
Thu, May 13, 6:47 PM
Harbormaster failed remote builds in B30523: Diff 37522 for D7526: [RFC] Directly invoke solids!
Thu, May 13, 6:38 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Wrap iterator in tighter error bound, fix some messages

Thu, May 13, 6:17 PM
prha closed D7613: Add sensor cursor (independent from run_key), to skip unnecessary sensor evaluation.
Thu, May 13, 5:43 PM
prha committed R1:945e175ce891: Add sensor cursor (independent from run_key), to skip unnecessary sensor… (authored by prha).
Add sensor cursor (independent from run_key), to skip unnecessary sensor…
Thu, May 13, 5:43 PM
jordansanders closed D7887: Pip install twice.
Thu, May 13, 5:13 PM
jordansanders committed R1:99ce9c8ba0bb: Pip install twice (authored by jordansanders).
Pip install twice
Thu, May 13, 5:13 PM
sandyryza added a comment to D7476: monitor sensor prototype.

I'm ambivalent about pipeline_failure_sensor. I think it's nice that it exposes the underlying implementation, but not sure if we want to lock in the expectation that we support the same signature for the user function as you would a regular sensor evaluation function.

Thu, May 13, 5:11 PM
prha added a comment to D7476: monitor sensor prototype.

I'm ambivalent about pipeline_failure_sensor. I think it's nice that it exposes the underlying implementation, but not sure if we want to lock in the expectation that we support the same signature for the user function as you would a regular sensor evaluation function.

Thu, May 13, 4:56 PM
prha added inline comments to D7476: monitor sensor prototype.
Thu, May 13, 4:54 PM
sandyryza added a comment to D7476: monitor sensor prototype.

What about @pipeline_failure_sensor? @alangenfeld might also have opinions.

Thu, May 13, 4:53 PM
prha added a comment to D7476: monitor sensor prototype.

This is looking pretty good to me. Not sure about the naming, but don't have great alternatives (pipeline_failure_watcher? watch_pipeline_failure?).

Thu, May 13, 4:47 PM
prha added a comment to D7895: monitor sensor 4/[dagit] show origin runs on the sensor's page.

Yep, I agree with @sandyryza... It's nice that we can distinguish between skip ticks (no runs failed) vs successfully ran monitor function ticks (reported into failed run), but it'd be nice to be able to do that without using the same UI that other sensors use for run requests. What would the UI look like if we did a retry request?

Thu, May 13, 3:49 PM
johann closed D7892: Put run coordinator instance yaml under test.
Thu, May 13, 3:38 PM
johann committed R1:8e3f761525f7: Put run coordinator instance yaml under test (authored by johann).
Put run coordinator instance yaml under test
Thu, May 13, 3:38 PM
sandyryza added a comment to D7895: monitor sensor 4/[dagit] show origin runs on the sensor's page.

It seems like these runs could be interpreted as runs that were triggered by the sensor - how do we expect users to tell the difference?

Thu, May 13, 3:10 PM
sandyryza closed D7747: add max_completion_wait_time_seconds option to databricks_pyspark_step_launcher.
Thu, May 13, 3:04 PM
sandyryza committed R1:836b564e1436: add max_completion_wait_time_seconds option to databricks_pyspark_step_launcher (authored by sandyryza).
add max_completion_wait_time_seconds option to databricks_pyspark_step_launcher
Thu, May 13, 3:04 PM
alangenfeld closed D7855: [mypy] system_config/objects.py.
Thu, May 13, 3:02 PM
alangenfeld committed R1:aa5d02eee9a2: [mypy] system_config/objects.py (authored by alangenfeld).
[mypy] system_config/objects.py
Thu, May 13, 3:02 PM
alangenfeld added inline comments to D7855: [mypy] system_config/objects.py.
Thu, May 13, 3:02 PM
alangenfeld requested changes to D7526: [RFC] Directly invoke solids.

maybe add a DagsterInvalidInvocationError for when the wrong info is passed?

Thu, May 13, 3:00 PM
alangenfeld committed R1:72d98d17d3b1: docs snapshot (authored by alangenfeld).
docs snapshot
Thu, May 13, 2:54 PM
johann closed D7891: Allow dagster/priority tag on definitions.
Thu, May 13, 2:41 PM
johann committed R1:7e121ab1e2f3: Allow dagster/priority tag on definitions (authored by johann).
Allow dagster/priority tag on definitions
Thu, May 13, 2:41 PM
alangenfeld accepted D7891: Allow dagster/priority tag on definitions.
Thu, May 13, 2:28 PM
jordansanders accepted D7891: Allow dagster/priority tag on definitions.
Thu, May 13, 2:02 PM
cdecarolis accepted D7747: add max_completion_wait_time_seconds option to databricks_pyspark_step_launcher.

works for me !

Thu, May 13, 1:41 PM
GitHub <noreply@github.com> committed R1:5c9dde0ed0f8: fix `deploing` -> `deploying` (authored by catherinewu).
fix `deploing` -> `deploying`
Thu, May 13, 8:02 AM
yuhan updated the diff for D7476: monitor sensor prototype.

rebase

Thu, May 13, 6:33 AM
yuhan retitled D7878: monitor sensor 1/ rename cursor to before_cursor, add arg after_cursor and ascending to RunStorage.get_runs from monitor sensor 1/ add arg after_cursor to RunStorage.get_runs to monitor sensor 1/ rename cursor to before_cursor, add arg after_cursor and ascending to RunStorage.get_runs.
Thu, May 13, 6:28 AM
yuhan updated the diff for D7878: monitor sensor 1/ rename cursor to before_cursor, add arg after_cursor and ascending to RunStorage.get_runs.

cursor -> before_cursor
+ ascending=False

Thu, May 13, 6:22 AM
yuhan requested review of D7895: monitor sensor 4/[dagit] show origin runs on the sensor's page.
Thu, May 13, 5:36 AM
yuhan updated the summary of D7476: monitor sensor prototype.
Thu, May 13, 5:17 AM
dgibson requested review of D7894: Include container_image in recon_pipeline_from_origin and recon_repository_from_origin.
Thu, May 13, 5:09 AM
dgibson closed D7883: Add an event before the create_namespaced_job call in the K8sRunLauncher.
Thu, May 13, 4:50 AM
dgibson committed R1:6d1459bac7a0: Add an event before the create_namespaced_job call in the K8sRunLauncher (authored by dgibson).
Add an event before the create_namespaced_job call in the K8sRunLauncher
Thu, May 13, 4:50 AM
yuhan accepted D7855: [mypy] system_config/objects.py.
Thu, May 13, 4:20 AM
yuhan updated the diff for D7476: monitor sensor prototype.

mypy

Thu, May 13, 4:10 AM
Harbormaster failed remote builds in B30493: Diff 37484 for D7476: monitor sensor prototype!
Thu, May 13, 2:32 AM
yuhan added a comment to D7476: monitor sensor prototype.

naming spitballing:

  • pipeline_failure_sensor / pipeline_sensor which means it watches pipelines - accurate, no new noun
  • pipeline_failure_monitor / pipeline_monitor which creates a new noun "monitor" but the name "monitor" is more intuitive as it describes the use case not the machinery
Thu, May 13, 2:28 AM
sashank accepted D7892: Put run coordinator instance yaml under test.
Thu, May 13, 2:28 AM
Harbormaster failed remote builds in B30492: Diff 37483 for D7476: monitor sensor prototype!
Thu, May 13, 2:17 AM
yuhan updated the diff for D7476: monitor sensor prototype.

base decorator

Thu, May 13, 2:11 AM
yuhan added inline comments to D7476: monitor sensor prototype.
Thu, May 13, 2:05 AM
yuhan updated the summary of D7476: monitor sensor prototype.
Thu, May 13, 1:57 AM
yuhan updated the summary of D7476: monitor sensor prototype.
Thu, May 13, 1:56 AM
yuhan updated the diff for D7476: monitor sensor prototype.

+ MonitorRequest

Thu, May 13, 1:55 AM
sandyryza added a comment to D7476: monitor sensor prototype.

This looks pretty nice. A couple naming questions:

  • If a monitor ultimately boils down to a sensor, is it worth introducing a new noun? I.e. should we just call this pipeline_failure_sensor?
  • We probably will want to add a version that monitors / senses failures for a particular pipeline or set of pipelines. How do we imagine the name of that fitting in with the name of this API?
Thu, May 13, 1:27 AM
johann requested review of D7892: Put run coordinator instance yaml under test.
Thu, May 13, 1:03 AM
yuhan updated the diff for D7880: monitor sensor 2/ allow nullable pipeline_name in sensor def.

+ is_monitor_sensor flag

Thu, May 13, 12:29 AM

Yesterday

dgibson added a comment to D7878: monitor sensor 1/ rename cursor to before_cursor, add arg after_cursor and ascending to RunStorage.get_runs.

That would work too - the most important thing to me is that you’re able to
paginate in a forwards direction with a limit

Wed, May 12, 11:52 PM
johann requested review of D7891: Allow dagster/priority tag on definitions.
Wed, May 12, 11:09 PM
rexledesma closed D7868: Add Dagster alert events.
Wed, May 12, 9:44 PM
rexledesma committed R1:a2181ff36e38: Add Dagster alert events (authored by rexledesma).
Add Dagster alert events
Wed, May 12, 9:44 PM
alangenfeld added inline comments to D7526: [RFC] Directly invoke solids.
Wed, May 12, 9:27 PM
alangenfeld added inline comments to D7526: [RFC] Directly invoke solids.
Wed, May 12, 9:25 PM
dish requested review of D7889: [dagit] Extract AppCache to top level, consolidate TestProvider.
Wed, May 12, 9:18 PM
cdecarolis added inline comments to D7526: [RFC] Directly invoke solids.
Wed, May 12, 9:10 PM
rexledesma updated the diff for D7868: Add Dagster alert events.

rebase

Wed, May 12, 9:06 PM
cdecarolis updated the diff for D7526: [RFC] Directly invoke solids.

Add more tests for multi-output case, change arg names

Wed, May 12, 9:05 PM
GitHub <noreply@github.com> committed R1:2d003a222e6e: Fix schedule docs typo (#4169) (authored by Doron Grinzaig <grdoron@gmail.com>).
Fix schedule docs typo (#4169)
Wed, May 12, 8:47 PM
yuhan added a comment to D7878: monitor sensor 1/ rename cursor to before_cursor, add arg after_cursor and ascending to RunStorage.get_runs.

to make sure im understanding it correctly, say we have runs = [1,2,3,4,5,6,7,8,9,10] and we will get:
get_runs(cursor=5, ascending=True, limit=2) = [6,7] where cursor is an after cursor
get_runs(cursor=5, ascending=False, limit=2) = [2,1] where cursor is a before cursor

Wed, May 12, 7:18 PM
yuhan accepted D7868: Add Dagster alert events.

per offline discussion with @rexledesma - these events won't be persisted in OSS for now so should be safe to land - if we ever need to update we can run migration on our end.
i'd recommend stack the call site diff and make sure the new events are working properly before landing :)

Wed, May 12, 6:56 PM
yuhan added inline comments to D7880: monitor sensor 2/ allow nullable pipeline_name in sensor def.
Wed, May 12, 6:53 PM
alangenfeld accepted D7887: Pip install twice.

thumbsup

Wed, May 12, 6:11 PM
rexledesma added a comment to D7868: Add Dagster alert events.

@yuhan The call site would be in the monitoring daemon. Initially this was for the prototype, which is why ALERT_FAILURE was not included, but will do so now.

Wed, May 12, 5:50 PM
jordansanders published D7887: Pip install twice for review.
Wed, May 12, 5:25 PM
alangenfeld closed D7875: [dagit] fix composites in sidebar.
Wed, May 12, 4:57 PM
alangenfeld committed R1:55b822b40c23: [dagit] fix composites in sidebar (authored by alangenfeld).
[dagit] fix composites in sidebar
Wed, May 12, 4:57 PM
johann accepted D7883: Add an event before the create_namespaced_job call in the K8sRunLauncher.
Wed, May 12, 4:46 PM
dgibson requested review of D7883: Add an event before the create_namespaced_job call in the K8sRunLauncher.
Wed, May 12, 4:45 PM
prha added inline comments to D7880: monitor sensor 2/ allow nullable pipeline_name in sensor def.
Wed, May 12, 4:18 PM
prha closed D7848: add metadata types for dagster pipeline runs, assets.
Wed, May 12, 4:12 PM
prha committed R1:169f49436901: add metadata types for dagster pipeline runs, assets (authored by prha).
add metadata types for dagster pipeline runs, assets
Wed, May 12, 4:12 PM
prha accepted D7875: [dagit] fix composites in sidebar.
Wed, May 12, 4:12 PM
alangenfeld requested review of D7875: [dagit] fix composites in sidebar.
Wed, May 12, 4:10 PM
dgibson requested changes to D7878: monitor sensor 1/ rename cursor to before_cursor, add arg after_cursor and ascending to RunStorage.get_runs.

Goal here seems very reasonable. Question for you and other reviewers - Would adding an ascending=True/False flag be another way to deal with this? (i.e. the same cursor could be either a before_cursor or after_cursor depending on the order that you want the results)?

Wed, May 12, 3:51 PM
cdecarolis planned changes to D7526: [RFC] Directly invoke solids.

fixing outputs

Wed, May 12, 3:32 PM
alangenfeld requested review of D7855: [mypy] system_config/objects.py.
Wed, May 12, 3:32 PM
cdecarolis added inline comments to D7526: [RFC] Directly invoke solids.
Wed, May 12, 3:21 PM
dgibson accepted D7869: [dagit] Rename auth header (1/3).
Wed, May 12, 3:07 PM
Chris DeCarolis <cdguitar817@gmail.com> closed D7843: Add test for ensuring config schema not applied if resource not used.
Wed, May 12, 3:06 PM
Chris DeCarolis <cdguitar817@gmail.com> committed R1:1e452e369867: Add test for ensuring config schema not applied if resource not used (authored by Chris DeCarolis <cdguitar817@gmail.com>).
Add test for ensuring config schema not applied if resource not used
Wed, May 12, 3:06 PM
Chris DeCarolis <cdguitar817@gmail.com> closed D7677: [Depends on D7674] Refactor lambda_solid as a simple wrapper around _Solid.
Wed, May 12, 3:04 PM
Chris DeCarolis <cdguitar817@gmail.com> committed R1:ba4aeb147494: [Depends on D7674] Refactor lambda_solid as a simple wrapper around _Solid (authored by Chris DeCarolis <cdguitar817@gmail.com>).
[Depends on D7674] Refactor lambda_solid as a simple wrapper around _Solid
Wed, May 12, 3:04 PM
dish requested review of D7881: [dagit] Delete src/main.
Wed, May 12, 2:38 PM
cdecarolis retitled D7771: Add a flag for whether resources object contains a generator from Make the constructed resources object not break mypy and add a flag for whether a resource is a generator to Add a flag for whether resources object contains a generator.
Wed, May 12, 2:31 PM
cdecarolis added a comment to D7832: [dagster-dbt] re-enable asset materializations when running `dbt cli` solids.

It seems like this is a strict improvement over what we currently have, so we should probably merge. That said, do we have an understanding of what happened to this data that we're now missing? Did dbt stop producing it entirely? Is there a different JSON file that contains it?

Wed, May 12, 2:30 PM
yuhan updated the test plan for D7880: monitor sensor 2/ allow nullable pipeline_name in sensor def.
Wed, May 12, 6:38 AM
yuhan requested review of D7880: monitor sensor 2/ allow nullable pipeline_name in sensor def.
Wed, May 12, 6:12 AM
yuhan updated the diff for D7476: monitor sensor prototype.

black

Wed, May 12, 5:49 AM
yuhan retitled D7476: monitor sensor prototype from pipeline hook prototype to monitor sensor prototype.
Wed, May 12, 5:45 AM
Harbormaster failed remote builds in B30458: Diff 37438 for D7476: monitor sensor prototype!
Wed, May 12, 5:30 AM
Harbormaster failed remote builds in B30457: Diff 37437 for D7476: monitor sensor prototype!
Wed, May 12, 5:28 AM
yuhan updated the diff for D7476: monitor sensor prototype.

up

Wed, May 12, 5:08 AM
yuhan updated the diff for D7476: monitor sensor prototype.

TODO:

  • allow query multiple types: +DagsterEventType.PIPELINE_INIT_FAILURE
  • yield something to indicate the monitor success/error instead of "skipping"
Wed, May 12, 5:06 AM
yuhan requested review of D7878: monitor sensor 1/ rename cursor to before_cursor, add arg after_cursor and ascending to RunStorage.get_runs.
Wed, May 12, 4:35 AM