Page MenuHomeElementl
Feed All Stories

Today

dish requested review of D9185: [dagit] Add Permissions query.
Mon, Aug 2, 9:46 PM
sandyryza closed D9184: split up test_decorators.
Mon, Aug 2, 9:34 PM
sandyryza committed R1:72ba29df076e: split up test_decorators (authored by sandyryza).
split up test_decorators
Mon, Aug 2, 9:34 PM
cdecarolis accepted D9172: [crag] remove job arg from partitioned schedule decorators.

See comment

Mon, Aug 2, 9:18 PM
cdecarolis accepted D9184: split up test_decorators.

yay

Mon, Aug 2, 9:16 PM
cdecarolis updated the diff for D9061: [memoization 4/n][rfc] memoization strategy for jobs.

Get rid of code example

Mon, Aug 2, 9:13 PM
sandyryza closed D9171: [crag] allow non-string tag values in to_job.
Mon, Aug 2, 8:25 PM
sandyryza committed R1:1f936709ec27: [crag] allow non-string tag values in to_job (authored by sandyryza).
[crag] allow non-string tag values in to_job
Mon, Aug 2, 8:24 PM
sandyryza retitled D9171: [crag] allow non-string tag values in to_job from allow non-string tag values in to_job to [crag] allow non-string tag values in to_job.
Mon, Aug 2, 8:23 PM
sandyryza retitled D9173: [crag] [RFC] accept job as a string when constructing schedules from [RFC] accept job as a string when constructing schedules to [crag] [RFC] accept job as a string when constructing schedules.
Mon, Aug 2, 8:22 PM
sandyryza retitled D9172: [crag] remove job arg from partitioned schedule decorators from remove job arg from partitioned schedule decorators to [crag] remove job arg from partitioned schedule decorators.
Mon, Aug 2, 8:21 PM
sandyryza retitled D7722: [crag] cragitize the IO manager concepts page from cragitize the IO manager concepts page to [crag] cragitize the IO manager concepts page.
Mon, Aug 2, 8:21 PM
sandyryza retitled D9070: [caprisun] RepositoryDefinition.asset_definition_graph from RepositoryDefinition.asset_definition_graph to [caprisun] RepositoryDefinition.asset_definition_graph.
Mon, Aug 2, 8:20 PM
sandyryza retitled D9074: [caprisun] Run ops after assets from Run ops after assets to [caprisun] Run ops after assets.
Mon, Aug 2, 8:20 PM
sandyryza retitled D8969: [caprisun] @asset and build_assets_job from @asset and build_assets_job to [caprisun] @asset and build_assets_job.
Mon, Aug 2, 8:14 PM
GitHub <noreply@github.com> committed R1:c97480de8c16: [docs-infra] yarn build-master disables versioning (#4432) (authored by yuhan).
[docs-infra] yarn build-master disables versioning (#4432)
Mon, Aug 2, 8:11 PM
sandyryza retitled D9156: [caprisun] asset in from asset in to [caprisun] asset in.
Mon, Aug 2, 8:00 PM
cdecarolis updated the diff for D9061: [memoization 4/n][rfc] memoization strategy for jobs.

Rebase + allow tag to toggle memoization off

Mon, Aug 2, 7:45 PM
GitHub <noreply@github.com> committed R1:757dd5025ccd: feat(helm): add serviceAccount.name to the user deployment subchart and schema… (authored by Michel Rouly <michel@rouly.net>).
feat(helm): add serviceAccount.name to the user deployment subchart and schema…
Mon, Aug 2, 6:58 PM
dgibson closed D9181: allow step failure events to have no serialized error.
Mon, Aug 2, 5:39 PM
dgibson committed R1:6b8d319f822a: allow step failure events to have no serialized error (authored by dgibson).
allow step failure events to have no serialized error
Mon, Aug 2, 5:39 PM
dgibson added a comment to D9181: allow step failure events to have no serialized error.

@dish before:

before.png (86×2 px, 36 KB)
after:
after.png (70×2 px, 18 KB)

Mon, Aug 2, 5:38 PM
Chris DeCarolis <cdguitar817@gmail.com> closed D9175: Carry over field aliases to default config for job.
Mon, Aug 2, 5:31 PM
Chris DeCarolis <cdguitar817@gmail.com> committed R1:3751d986cf70: Carry over field aliases to default config for job (authored by Chris DeCarolis <cdguitar817@gmail.com>).
Carry over field aliases to default config for job
Mon, Aug 2, 5:31 PM
dgibson updated the diff for D9181: allow step failure events to have no serialized error.

up

Mon, Aug 2, 5:29 PM
GitHub <noreply@github.com> committed R1:7fe311babd48: Update dbt_example to use dbt_cli_resource (#4430) (authored by owen).
Update dbt_example to use dbt_cli_resource (#4430)
Mon, Aug 2, 5:26 PM
GitHub <noreply@github.com> committed R1:f1a30ab34630: PipelineRunsFilter can have multiple filters (#4425) (authored by jordansanders).
PipelineRunsFilter can have multiple filters (#4425)
Mon, Aug 2, 5:22 PM
yuhan accepted D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

my hunch is later in the stack (after D9085), this shouldn't be too large of a change, e.g. could potentially toggle is on io manager config?

Mon, Aug 2, 5:18 PM
dish added inline comments to D9181: allow step failure events to have no serialized error.
Mon, Aug 2, 5:11 PM
sandyryza requested review of D9172: [crag] remove job arg from partitioned schedule decorators.
Mon, Aug 2, 5:07 PM
Harbormaster failed remote builds in B34550: Diff 42721 for D9181: allow step failure events to have no serialized error!
Mon, Aug 2, 5:07 PM
alangenfeld accepted D9181: allow step failure events to have no serialized error.
Mon, Aug 2, 5:02 PM
sandyryza requested review of D9184: split up test_decorators.
Mon, Aug 2, 4:56 PM
bengotow accepted D9161: [dagit] RFC: useSubscription instead of DirectGraphQLSubscription.
Mon, Aug 2, 4:05 PM
alangenfeld added inline comments to D9085: [memoization 6/n] make default fs io manager memoizable.
Mon, Aug 2, 3:55 PM
sandyryza requested review of D9173: [crag] [RFC] accept job as a string when constructing schedules.
Mon, Aug 2, 3:48 PM
sandyryza closed D9174: type check failures often include two adjacent periods.
Mon, Aug 2, 3:47 PM
sandyryza committed R1:5e59e859a2cd: type check failures often include two adjacent periods (authored by sandyryza).
type check failures often include two adjacent periods
Mon, Aug 2, 3:47 PM
Harbormaster failed remote builds in B34546: Diff 42714 for D9182: Add an error boundary within get_pipeline_run_observable to prevent dagit hanging on message parsing failures!
Mon, Aug 2, 3:44 PM
dgibson updated the diff for D9181: allow step failure events to have no serialized error.

mypy

Mon, Aug 2, 3:41 PM
sandyryza updated the diff for D9171: [crag] allow non-string tag values in to_job.

up

Mon, Aug 2, 3:39 PM
dish updated the diff for D9161: [dagit] RFC: useSubscription instead of DirectGraphQLSubscription.

Fix ref error in useViewport

Mon, Aug 2, 3:31 PM
dgibson closed D9179: speed up test project builds.
Mon, Aug 2, 3:06 PM
dgibson committed R1:39fefebd7dcd: speed up test project builds (authored by dgibson).
speed up test project builds
Mon, Aug 2, 3:06 PM
dgibson published D9182: Add an error boundary within get_pipeline_run_observable to prevent dagit hanging on message parsing failures for review.

i'm not actually sure this is right despite fixing the hangs (in that it will cause batches of logs to just be skipped - if there was some way to still raise the exception, but give the observable time to clean up using its standard ways of doing that, that would be ideal)

Mon, Aug 2, 3:05 PM
alangenfeld added a comment to D9181: allow step failure events to have no serialized error.

make sure we update the mypy type on StepFailureData which is also not set as Optional

Mon, Aug 2, 2:46 PM
dgibson added a comment to D9181: allow step failure events to have no serialized error.

step failure event with no serialized error info: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/python_modules/libraries/dagster-k8s/dagster_k8s/executor.py?L233:42

Mon, Aug 2, 2:42 PM
jordansanders accepted D9179: speed up test project builds.
Mon, Aug 2, 2:36 PM
alangenfeld added a comment to D9181: allow step failure events to have no serialized error.

what condition did we add where we have a step failure with no exception?

Mon, Aug 2, 2:36 PM
alangenfeld added a comment to D4698: Prototype: Add test utility for dagstermill solids.

My initial reaction is that this is a pretty small improvement for a very specialized utility function. Its hard for me to make a strong case that this is any better than reconstructable in terms of having to discover some special tool when the normal testing flow fails on a dagstermill solid.

Mon, Aug 2, 2:21 PM
dish closed D9170: [dagit] Stop overfetching schedule/sensor in JobMetadata.
Mon, Aug 2, 2:04 PM
dish committed R1:e93b23ef47e9: [dagit] Stop overfetching schedule/sensor in JobMetadata (authored by dish).
[dagit] Stop overfetching schedule/sensor in JobMetadata
Mon, Aug 2, 2:04 PM
dgibson updated the diff for D9057: dagit default app handler tweaks.

up

Mon, Aug 2, 2:04 PM
dish added inline comments to D9161: [dagit] RFC: useSubscription instead of DirectGraphQLSubscription.
Mon, Aug 2, 2:03 PM
alangenfeld accepted D9171: [crag] allow non-string tag values in to_job.

add tests for valid non-str tag and invalid non-str tag

Mon, Aug 2, 2:01 PM
alangenfeld accepted D9175: Carry over field aliases to default config for job.
Mon, Aug 2, 1:59 PM
dgibson requested review of D9181: allow step failure events to have no serialized error.
Mon, Aug 2, 1:49 PM
dgibson updated the summary of D9179: speed up test project builds.
Mon, Aug 2, 1:16 PM
dgibson added a reviewer for D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them: johann.
Mon, Aug 2, 12:57 PM
dgibson updated the summary of D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them.
Mon, Aug 2, 12:57 PM
Harbormaster failed remote builds in B34537: Diff 42703 for D9179: speed up test project builds!
Mon, Aug 2, 12:57 AM
dgibson requested review of D9179: speed up test project builds.
Mon, Aug 2, 12:51 AM
dgibson requested review of D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them.
Mon, Aug 2, 12:18 AM

Yesterday

dgibson closed D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"".
Sun, Aug 1, 1:25 PM
dgibson added a reverting change for D9137: Revert "Load execution plan from the snapshot on the run when resume retrying": R1:b33469677b47: Bring back "Load execution plan from the snapshot on the run when resume….
Sun, Aug 1, 1:25 PM
dgibson added a reverting change for R1:bfb90e8b0b44: Revert "Load execution plan from the snapshot on the run when resume retrying": R1:b33469677b47: Bring back "Load execution plan from the snapshot on the run when resume….
Sun, Aug 1, 1:25 PM
dgibson committed R1:b33469677b47: Bring back "Load execution plan from the snapshot on the run when resume… (authored by dgibson).
Bring back "Load execution plan from the snapshot on the run when resume…
Sun, Aug 1, 1:25 PM

Sat, Jul 31

cdecarolis requested review of D9175: Carry over field aliases to default config for job.
Sat, Jul 31, 7:28 PM
sandyryza requested review of D9171: [crag] allow non-string tag values in to_job.
Sat, Jul 31, 5:32 PM
Harbormaster failed remote builds in B34522: Diff 42687 for D9174: type check failures often include two adjacent periods!
Sat, Jul 31, 2:17 AM
sandyryza updated the diff for D9174: type check failures often include two adjacent periods.

up

Sat, Jul 31, 1:12 AM
sandyryza updated the diff for D9174: type check failures often include two adjacent periods.

up

Sat, Jul 31, 1:10 AM
dgibson updated the diff for D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"".

up

Sat, Jul 31, 12:15 AM
dgibson closed D9168: Insert solid_selection into pipeline snapshots in sorted order.
Sat, Jul 31, 12:12 AM
dgibson committed R1:2f84b74c2953: Insert solid_selection into pipeline snapshots in sorted order (authored by dgibson).
Insert solid_selection into pipeline snapshots in sorted order
Sat, Jul 31, 12:12 AM
prha accepted D9170: [dagit] Stop overfetching schedule/sensor in JobMetadata.
Sat, Jul 31, 12:02 AM
prha accepted D9174: type check failures often include two adjacent periods.
Sat, Jul 31, 12:01 AM
sandyryza requested review of D9174: type check failures often include two adjacent periods.
Sat, Jul 31, 12:00 AM

Fri, Jul 30

cdecarolis added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.
In D8963#237434, @yuhan wrote:

in terms of user experience, i think it's fine to implicitly add a tag to a run - some other features work like that, e.g. we implicitly add parent_run_id and root_run_id as tags to new runs, and solid/step selection too. besides, it's also good that users can view those info in the Runs page such as:

image.png (802×2 px, 185 KB)

to clarify, i agree that tags are less robust as it doesn't have good checks and could get lost - that may be the trade-off we are making or config may be a better alternative bc it's schematized.

Fri, Jul 30, 11:47 PM
dish requested review of D9170: [dagit] Stop overfetching schedule/sensor in JobMetadata.
Fri, Jul 30, 10:14 PM
prha closed D9152: fix run stats for backfill progress.
Fri, Jul 30, 10:02 PM
prha committed R1:6d7ed0a2db01: fix run stats for backfill progress (authored by prha).
fix run stats for backfill progress
Fri, Jul 30, 10:02 PM
Harbormaster failed remote builds in B34513: Diff 42675 for D9169: skip failing windows test for grpc/ipc failures on empty dir!
Fri, Jul 30, 10:00 PM
prha accepted D9161: [dagit] RFC: useSubscription instead of DirectGraphQLSubscription.
Fri, Jul 30, 9:59 PM
prha updated the diff for D9169: skip failing windows test for grpc/ipc failures on empty dir.

update

Fri, Jul 30, 9:58 PM
prha published D9169: skip failing windows test for grpc/ipc failures on empty dir for review.
Fri, Jul 30, 9:56 PM
prha accepted D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"".
Fri, Jul 30, 9:53 PM
prha accepted D9168: Insert solid_selection into pipeline snapshots in sorted order.
Fri, Jul 30, 9:51 PM
dgibson requested review of D9168: Insert solid_selection into pipeline snapshots in sorted order.
Fri, Jul 30, 9:50 PM
prha added a comment to D9161: [dagit] RFC: useSubscription instead of DirectGraphQLSubscription.

This makes sense to me... thanks for cleaning this up.

Fri, Jul 30, 9:49 PM
dgibson published D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"" for review.
Fri, Jul 30, 9:08 PM
Chris DeCarolis <cdguitar817@gmail.com> closed D9166: [bug] fix dynamic outputs issue with adls2 io manager.
Fri, Jul 30, 9:03 PM
Chris DeCarolis <cdguitar817@gmail.com> committed R1:f20b2704cdda: [bug] fix dynamic outputs issue with adls2 io manager (authored by Chris DeCarolis <cdguitar817@gmail.com>).
[bug] fix dynamic outputs issue with adls2 io manager
Fri, Jul 30, 9:03 PM
prha accepted D9166: [bug] fix dynamic outputs issue with adls2 io manager.

oddish

Fri, Jul 30, 8:49 PM
cdecarolis updated the diff for D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

Fix plumbing

Fri, Jul 30, 8:00 PM
dish updated the diff for D9161: [dagit] RFC: useSubscription instead of DirectGraphQLSubscription.

Commented line

Fri, Jul 30, 7:42 PM
yuhan added a comment to D8963: [memoization 3/n] move core memoization logic to live on the execution plan.

in terms of user experience, i think it's fine to implicitly add a tag to a run - some other features work like that, e.g. we implicitly add parent_run_id and root_run_id as tags to new runs, and solid/step selection too. besides, it's also good that users can view those info in the Runs page such as:

image.png (802×2 px, 185 KB)

Fri, Jul 30, 7:13 PM
dgibson added a reverting change for D9137: Revert "Load execution plan from the snapshot on the run when resume retrying": D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"".
Fri, Jul 30, 7:11 PM
dgibson added a reverting change for R1:bfb90e8b0b44: Revert "Load execution plan from the snapshot on the run when resume retrying": D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"".
Fri, Jul 30, 7:11 PM
cdecarolis requested review of D9166: [bug] fix dynamic outputs issue with adls2 io manager.
Fri, Jul 30, 7:03 PM