Page MenuHomeElementl
Feed All Stories

Today

sandyryza added a comment to D8362: rename SensorExecutionContext to SensorContext.

I like SensorEvaluationContext and ScheduleEvaluationContext too.

Tue, Jun 15, 5:46 AM
yuhan requested review of D8372: expose PipelineRunStatus as a top-level API.
Tue, Jun 15, 4:32 AM
dgibson requested changes to D8211: feat: retrieve partition set names from external partition set in schedule based partitions.

overall this makes sense to me - tweak some names and remove some extra timezone fields and i think we're good

Tue, Jun 15, 4:09 AM
yuhan requested review of D8371: fix images on Pipelines page.
Tue, Jun 15, 4:09 AM
sandyryza closed D8347: [crag] a set of pipelines and schedules to prototype cragit UI on top of.
Tue, Jun 15, 1:24 AM
sandyryza committed R1:6f0520630762: [crag] a set of pipelines and schedules to prototype cragit UI on top of (authored by sandyryza).
[crag] a set of pipelines and schedules to prototype cragit UI on top of
Tue, Jun 15, 1:24 AM
sandyryza added inline comments to D8356: don't require context arg when solid has required resources.
Tue, Jun 15, 1:23 AM
sandyryza closed D8344: [crag] job name.
Tue, Jun 15, 1:22 AM
sandyryza committed R1:e09712d42073: [crag] job name (authored by sandyryza).
[crag] job name
Tue, Jun 15, 1:22 AM
yuhan closed D7895: monitor sensor 4/[dagit] show origin runs on the sensor's page.
Tue, Jun 15, 12:26 AM
yuhan committed R1:99c6d4e27dc1: monitor sensor 4/[dagit] show origin runs on the sensor's page (authored by yuhan).
monitor sensor 4/[dagit] show origin runs on the sensor's page
Tue, Jun 15, 12:26 AM
prha accepted D7895: monitor sensor 4/[dagit] show origin runs on the sensor's page.
Tue, Jun 15, 12:06 AM
yuhan requested changes to D8259: [guide] file processing.

requesting changes for q management

Tue, Jun 15, 12:03 AM
yuhan accepted D8360: [docs] respin io manager testing section.

👏

Tue, Jun 15, 12:02 AM

Yesterday

yuhan added inline comments to D7895: monitor sensor 4/[dagit] show origin runs on the sensor's page.
Mon, Jun 14, 11:56 PM
yuhan updated the diff for D7895: monitor sensor 4/[dagit] show origin runs on the sensor's page.

up

Mon, Jun 14, 11:56 PM
dish added a comment to D8363: [dagit] Crag: Flattened left nav jobs.

Ideally we'll just lift the valuable things off those pages onto the Job overview.

Mon, Jun 14, 10:04 PM
yuhan requested review of D8365: correctly process configs of io managers on solids inside composite.
Mon, Jun 14, 9:57 PM
cdecarolis requested review of D8360: [docs] respin io manager testing section.
Mon, Jun 14, 9:53 PM
alangenfeld added inline comments to D8356: don't require context arg when solid has required resources.
Mon, Jun 14, 9:30 PM
cdecarolis accepted D8356: don't require context arg when solid has required resources.

Yea this makes sense to me. The step launcher case already made me a bit uneasy.

Mon, Jun 14, 9:19 PM
bengotow accepted D8363: [dagit] Crag: Flattened left nav jobs.

This looks great! I think the iconography + colors are nice and I'm a fan of not having the icons click through to the sensor / schedule pages. Ideally we'll just lift the valuable things off those pages onto the Job overview.

Mon, Jun 14, 9:11 PM
dish added inline comments to D8363: [dagit] Crag: Flattened left nav jobs.
Mon, Jun 14, 8:56 PM
sandyryza requested review of D8356: don't require context arg when solid has required resources.
Mon, Jun 14, 8:51 PM
sandyryza accepted D8331: [docs] add resource testing section.
Mon, Jun 14, 8:40 PM
yuhan updated the diff for D8272: error when solids inside composite have root_manager_key.

move _check_io_managers_on_composite_solid to _CompositeSolid so we error as soon as the composite solid is defined

Mon, Jun 14, 8:35 PM
sandyryza closed D8355: mark to_job as experimental.
Mon, Jun 14, 8:34 PM
sandyryza committed R1:5469ba2fee4f: mark to_job as experimental (authored by sandyryza).
mark to_job as experimental
Mon, Jun 14, 8:33 PM
dish requested review of D8363: [dagit] Crag: Flattened left nav jobs.
Mon, Jun 14, 8:22 PM
cdecarolis added a comment to D8362: rename SensorExecutionContext to SensorContext.
In D8362#218480, @prha wrote:

+1 on moving way from ExecutionContext.

I think SensorContext is fine... did you also consider SensorEvaluationContext/ ScheduleEvaluationContext? I think we've been pretty good in the docs about "evaluating" sensors & schedules and "executing" pipelines.

Mon, Jun 14, 7:44 PM
prha added a comment to D8362: rename SensorExecutionContext to SensorContext.

+1 on moving way from ExecutionContext.

Mon, Jun 14, 7:40 PM
cdecarolis requested review of D8362: rename SensorExecutionContext to SensorContext.
Mon, Jun 14, 7:11 PM
GitHub <noreply@github.com> committed R1:1582b6e99ae2: Fix pyspark_resource example (#4290) (authored by Andrew-Crosby <37139900+Andrew-Crosby@users.noreply.github.com>).
Fix pyspark_resource example (#4290)
Mon, Jun 14, 7:03 PM
dish closed D8303: [dagit] More optimistic switch behavior for schedules/sensors.
Mon, Jun 14, 6:58 PM
dish committed R1:e7997d675c43: [dagit] More optimistic switch behavior for schedules/sensors (authored by dish).
[dagit] More optimistic switch behavior for schedules/sensors
Mon, Jun 14, 6:58 PM
yuhan added inline comments to D8324: get_stored_events and get_stored_runs.
Mon, Jun 14, 6:31 PM
alangenfeld added inline comments to D8324: get_stored_events and get_stored_runs.
Mon, Jun 14, 5:43 PM
prha accepted D8303: [dagit] More optimistic switch behavior for schedules/sensors.
Mon, Jun 14, 5:22 PM
prha added inline comments to D8324: get_stored_events and get_stored_runs.
Mon, Jun 14, 5:22 PM
rexledesma closed D8349: feat: add flag to disable all warnings when running dagit.
Mon, Jun 14, 5:21 PM
rexledesma committed R1:a9bf380fa399: feat: add flag to disable all warnings when running dagit (authored by rexledesma).
feat: add flag to disable all warnings when running dagit
Mon, Jun 14, 5:21 PM
alangenfeld added inline comments to D8272: error when solids inside composite have root_manager_key.
Mon, Jun 14, 4:37 PM
alangenfeld accepted D8283: [crag] partitions on modes.

should add a test for lazy job construction with partition too

Mon, Jun 14, 4:35 PM
Harbormaster failed remote builds in B32042: Diff 39477 for D8275: Make solid invocation directly call the underlying decorated fxn!
Mon, Jun 14, 4:20 PM
Harbormaster failed remote builds in B32041: Diff 39475 for D8275: Make solid invocation directly call the underlying decorated fxn!
Mon, Jun 14, 4:15 PM
Harbormaster failed remote builds in B32040: Diff 39474 for D8275: Make solid invocation directly call the underlying decorated fxn!
Mon, Jun 14, 4:13 PM
cdecarolis updated the diff for D8275: Make solid invocation directly call the underlying decorated fxn.

Up

Mon, Jun 14, 3:56 PM
prha closed D8354: add option to skip uploading empty files to improve perf with minio.
Mon, Jun 14, 3:53 PM
prha committed R1:288c4c072ea4: add option to skip uploading empty files to improve perf with minio (authored by prha).
add option to skip uploading empty files to improve perf with minio
Mon, Jun 14, 3:53 PM
cdecarolis updated the diff for D8275: Make solid invocation directly call the underlying decorated fxn.

up

Mon, Jun 14, 3:51 PM
cdecarolis updated the diff for D8275: Make solid invocation directly call the underlying decorated fxn.

Rearrange implementation around flag for decorators requiring context arguments, fix tests

Mon, Jun 14, 3:48 PM
alangenfeld accepted D8355: mark to_job as experimental.
Mon, Jun 14, 3:32 PM
alangenfeld accepted D8354: add option to skip uploading empty files to improve perf with minio.

works for me

Mon, Jun 14, 3:32 PM
rexledesma published D8211: feat: retrieve partition set names from external partition set in schedule based partitions for review.
Mon, Jun 14, 10:04 AM
GitHub <noreply@github.com> committed R1:e646625a687d: docs clean up 0.11.13rcx (#4288) (authored by yuhan).
docs clean up 0.11.13rcx (#4288)
Mon, Jun 14, 5:03 AM

Sat, Jun 12

dgibson closed D8358: remove 0.11.14 RCs from docs.
Sat, Jun 12, 1:27 AM
dgibson committed R1:afe5e08c7c3c: remove 0.11.14 RCs from docs (authored by dgibson).
remove 0.11.14 RCs from docs
Sat, Jun 12, 1:26 AM
elementl-devtools committed R1:2df4b60f29a5: Automation: versioned docs for 0.11.14rc7 (authored by elementl-devtools).
Automation: versioned docs for 0.11.14rc7
Sat, Jun 12, 1:03 AM

Fri, Jun 11

sandyryza requested review of D8355: mark to_job as experimental.
Fri, Jun 11, 11:52 PM
sandyryza updated the diff for D8283: [crag] partitions on modes.

up

Fri, Jun 11, 11:34 PM
prha requested review of D8354: add option to skip uploading empty files to improve perf with minio.
Fri, Jun 11, 11:30 PM
Harbormaster failed remote builds in B32023: Diff 39452 for D8283: [crag] partitions on modes!
Fri, Jun 11, 11:27 PM
sandyryza closed D8191: make input and output definition metadata unexperimental.
Fri, Jun 11, 11:19 PM
sandyryza committed R1:632895150a62: make input and output definition metadata unexperimental (authored by sandyryza).
make input and output definition metadata unexperimental
Fri, Jun 11, 11:19 PM
sandyryza updated the diff for D8347: [crag] a set of pipelines and schedules to prototype cragit UI on top of.

up

Fri, Jun 11, 11:08 PM
sandyryza updated the diff for D8283: [crag] partitions on modes.

up

Fri, Jun 11, 11:04 PM
yuhan updated the diff for D8272: error when solids inside composite have root_manager_key.

up

Fri, Jun 11, 10:59 PM
yuhan updated the summary of D8272: error when solids inside composite have root_manager_key.
Fri, Jun 11, 10:58 PM
yuhan updated the diff for D8272: error when solids inside composite have root_manager_key.

error on root_manager_key and io_manager_key on composite solids

Fri, Jun 11, 10:56 PM
sandyryza planned changes to D8283: [crag] partitions on modes.
Fri, Jun 11, 10:21 PM
sandyryza requested review of D8353: Dockerfile for hacker news example.
Fri, Jun 11, 10:08 PM
sandyryza retitled D8283: [crag] partitions on modes from [RFC] [crag] partitions on modes to [crag] partitions on modes.
Fri, Jun 11, 9:56 PM
sandyryza updated the diff for D8283: [crag] partitions on modes.

up

Fri, Jun 11, 9:56 PM
elementl-devtools committed R1:e71da6bcea13: Automation: versioned docs for 0.11.14rc5 (authored by elementl-devtools).
Automation: versioned docs for 0.11.14rc5
Fri, Jun 11, 9:54 PM
Harbormaster failed remote builds in B32015: Diff 39443 for D8347: [crag] a set of pipelines and schedules to prototype cragit UI on top of!
Fri, Jun 11, 9:41 PM
yuhan requested review of D8272: error when solids inside composite have root_manager_key.

I think the ideal behavior is to ban io_manager_key and root_input_manager on @composite_solid but allow these to be set on the inner @solids (which is quite the opposite of this diff). reasons are:

  1. users can still specify io managers for solids inside composite - imo composite is a way to better organizing solids but limit existing functionalities of its inner solids.
  2. we can keep solids' reusability - a solid that requires custom io managers can still be included in a composite.
  3. the dagster machinery can still treat composite as a container - meaning a composite has nothing to do with how io works and it would just defer to its inner solids' set up.
Fri, Jun 11, 9:08 PM
elementl-devtools committed R1:d8b0055fde57: Automation: versioned docs for 0.11.14rc4 (authored by elementl-devtools).
Automation: versioned docs for 0.11.14rc4
Fri, Jun 11, 9:01 PM
sandyryza updated the diff for D8347: [crag] a set of pipelines and schedules to prototype cragit UI on top of.

up

Fri, Jun 11, 9:00 PM
sandyryza added a comment to D8344: [crag] job name.

I had a conversation with @schrockn and I am receptive to his argument that it's more elegant for jobs to have names than suffixes. Here's a re-spin that does that.

Fri, Jun 11, 8:36 PM
sandyryza retitled D8344: [crag] job name from [crag] job suffix to [crag] job name.
Fri, Jun 11, 8:35 PM
sandyryza updated the diff for D8344: [crag] job name.

up

Fri, Jun 11, 8:35 PM
dgibson closed D8339: Be less strict in dagster image build CLI about the environment matching the passed in version.
Fri, Jun 11, 8:34 PM
dgibson committed R1:7c9a752e9a82: Be less strict in dagster image build CLI about the environment matching the… (authored by dgibson).
Be less strict in dagster image build CLI about the environment matching the…
Fri, Jun 11, 8:34 PM
sandyryza accepted D8349: feat: add flag to disable all warnings when running dagit.

lgtm!

Fri, Jun 11, 8:33 PM
elementl-devtools committed R1:967a1bcd5036: Automation: versioned docs for 0.11.14rc3 (authored by elementl-devtools).
Automation: versioned docs for 0.11.14rc3
Fri, Jun 11, 7:46 PM
alangenfeld updated the diff for D6648: [RFC] pipeline contains a graph.

lint

Fri, Jun 11, 7:42 PM
rexledesma updated the diff for D8349: feat: add flag to disable all warnings when running dagit.

+sandy

Fri, Jun 11, 7:02 PM
schrockn added a comment to D8344: [crag] job name.

IMO if the purpose to support "pipeline_name: mode_name" in dagit displays we could do that at the dagit/graphql layer exclusively rather than inject the notion of a suffix into the greenfield job concept

Fri, Jun 11, 6:56 PM
sandyryza requested changes to D8349: feat: add flag to disable all warnings when running dagit.
Fri, Jun 11, 6:45 PM
rexledesma requested review of D8349: feat: add flag to disable all warnings when running dagit.
Fri, Jun 11, 6:32 PM
sandyryza requested review of D8335: mlflow CI.
Fri, Jun 11, 6:15 PM
Harbormaster failed remote builds in B32001: Diff 39428 for D6648: [RFC] pipeline contains a graph!
Fri, Jun 11, 6:15 PM
alangenfeld added a comment to D8344: [crag] job name.

It seems odd to provide this and not just a name param

Fri, Jun 11, 6:09 PM
Harbormaster failed remote builds in B31996: Diff 39420 for D8347: [crag] a set of pipelines and schedules to prototype cragit UI on top of!
Fri, Jun 11, 5:54 PM
alangenfeld updated the diff for D6648: [RFC] pipeline contains a graph.

test, lint, mypy

Fri, Jun 11, 5:53 PM
Harbormaster failed remote builds in B31994: Diff 39418 for D8347: [crag] a set of pipelines and schedules to prototype cragit UI on top of!
Fri, Jun 11, 5:45 PM
dish closed D8342: [dagit] Upgrade to Typescript 4.3.2.
Fri, Jun 11, 5:44 PM
dish committed R1:c1924a615c90: [dagit] Upgrade to Typescript 4.3.2 (authored by dish).
[dagit] Upgrade to Typescript 4.3.2
Fri, Jun 11, 5:44 PM
dish closed D8334: [dagit] Clean up package.jsons.
Fri, Jun 11, 5:44 PM
dish committed R1:715b0b9761ea: [dagit] Clean up package.jsons (authored by dish).
[dagit] Clean up package.jsons
Fri, Jun 11, 5:44 PM