Page MenuHomeElementl

dgibson (Daniel Gibson)
User

Projects

User does not belong to any projects.

User Details

User Since
Jul 6 2020, 12:49 PM (73 w, 4 h)

Recent Activity

Sep 8 2021

dgibson accepted D9220: [Memoization 12/13] Enable memoization on all out-of-the-box executors.

see inline - esp the contextmanager thing. I'll sleep easier when this passes your new back-compat integration tests, and you'll want to have an internal diff ready to go before landing this i think

Sep 8 2021, 3:28 PM

Aug 20 2021

dgibson closed D9203: Dockerfile/deployment instructions for deploying hacker-news to k8s.
Aug 20 2021, 7:08 PM
dgibson committed R1:6086d580226c: Dockerfile/deployment instructions for deploying hacker-news to k8s (authored by dgibson).
Dockerfile/deployment instructions for deploying hacker-news to k8s
Aug 20 2021, 7:08 PM

Aug 19 2021

dgibson retitled D9203: Dockerfile/deployment instructions for deploying hacker-news to k8s from WIP: Dockerfile/deployment instructions for deploying hacker-news to k8s to Dockerfile/deployment instructions for deploying hacker-news to k8s.
Aug 19 2021, 8:04 PM
dgibson updated the diff for D9203: Dockerfile/deployment instructions for deploying hacker-news to k8s.

update after succesfully running

Aug 19 2021, 7:38 PM

Aug 11 2021

dgibson closed D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them.
Aug 11 2021, 9:40 PM
dgibson committed R1:dd67332137c2: add secrets/config maps from the executor config to the run launcher config… (authored by dgibson).
add secrets/config maps from the executor config to the run launcher config…
Aug 11 2021, 9:40 PM
dgibson added a comment to D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them.

(was more relevant when it would replace the launcher config when it was set though - since in that case None was distinct from [])

Aug 11 2021, 9:39 PM
dgibson added inline comments to D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them.
Aug 11 2021, 9:37 PM
dgibson removed a reviewer for D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them: johann.
Aug 11 2021, 9:36 PM
dgibson abandoned D9182: Add an error boundary within get_pipeline_run_observable to prevent dagit hanging on message parsing failures.
Aug 11 2021, 3:04 PM
dgibson added a comment to D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them.

@alangenfeld or @rexledesma if you wouldn't mind reviewing this since johann is out that week I would be very grateful

Aug 11 2021, 3:03 PM
dgibson updated the diff for D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them.

up

Aug 11 2021, 3:01 PM
dgibson updated the diff for D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them.

johann's feedback, beefed up the test coverage quite a bit

Aug 11 2021, 2:56 PM

Aug 9 2021

dgibson closed D9193: Tests for permissions query in graphql.
Aug 9 2021, 7:40 PM
dgibson committed R1:3b7cafe9523b: Tests for permissions query in graphql (authored by dgibson).
Tests for permissions query in graphql
Aug 9 2021, 7:40 PM

Aug 4 2021

dgibson added inline comments to D9203: Dockerfile/deployment instructions for deploying hacker-news to k8s.
Aug 4 2021, 2:51 AM
dgibson published D9203: Dockerfile/deployment instructions for deploying hacker-news to k8s for review.
Aug 4 2021, 2:49 AM

Aug 3 2021

dgibson accepted D9196: [dagit] Perms followup.
Aug 3 2021, 4:32 PM
dgibson published D9193: Tests for permissions query in graphql for review.
Aug 3 2021, 1:50 PM
dgibson added reviewers for D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them: rexledesma, alangenfeld.
Aug 3 2021, 1:43 PM
dgibson added a comment to D9182: Add an error boundary within get_pipeline_run_observable to prevent dagit hanging on message parsing failures.

I think @alangenfeld is in the process of unearthing a better fix here (fixing the root cause of the deadlock vs. preventing errors from getting thrown)

Aug 3 2021, 1:41 PM
dgibson accepted D9185: [dagit] Add Permissions query.

tests in https://dagster.phacility.com/D9193 - lets make sure we have a change ready for internal though if this is breaking there

Aug 3 2021, 1:19 PM
dgibson added a comment to D9185: [dagit] Add Permissions query.

this looks good, one big thing that was missing from the diff this is patched on is python tests though - I may be able to tackle that this morning, given that permissions are fairly sensitive i think its important to have some coverage there

Aug 3 2021, 12:46 PM
dgibson closed D9192: Make a new port in the gRPC test.
Aug 3 2021, 12:43 PM
dgibson committed R1:a230cafc3bcd: Make a new port in the gRPC test (authored by dgibson).
Make a new port in the gRPC test
Aug 3 2021, 12:43 PM
dgibson requested changes to D9169: skip failing windows test for grpc/ipc failures on empty dir.
Aug 3 2021, 12:43 PM
dgibson requested review of D9192: Make a new port in the gRPC test.
Aug 3 2021, 2:02 AM

Aug 2 2021

dgibson closed D9181: allow step failure events to have no serialized error.
Aug 2 2021, 5:39 PM
dgibson committed R1:6b8d319f822a: allow step failure events to have no serialized error (authored by dgibson).
allow step failure events to have no serialized error
Aug 2 2021, 5:39 PM
dgibson added a comment to D9181: allow step failure events to have no serialized error.

@dish before:

before.png (86×2 px, 36 KB)
after:
after.png (70×2 px, 18 KB)

Aug 2 2021, 5:38 PM
dgibson updated the diff for D9181: allow step failure events to have no serialized error.

up

Aug 2 2021, 5:29 PM
dgibson updated the diff for D9181: allow step failure events to have no serialized error.

mypy

Aug 2 2021, 3:41 PM
dgibson closed D9179: speed up test project builds.
Aug 2 2021, 3:06 PM
dgibson committed R1:39fefebd7dcd: speed up test project builds (authored by dgibson).
speed up test project builds
Aug 2 2021, 3:06 PM
dgibson published D9182: Add an error boundary within get_pipeline_run_observable to prevent dagit hanging on message parsing failures for review.

i'm not actually sure this is right despite fixing the hangs (in that it will cause batches of logs to just be skipped - if there was some way to still raise the exception, but give the observable time to clean up using its standard ways of doing that, that would be ideal)

Aug 2 2021, 3:05 PM
dgibson added a comment to D9181: allow step failure events to have no serialized error.

step failure event with no serialized error info: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/python_modules/libraries/dagster-k8s/dagster_k8s/executor.py?L233:42

Aug 2 2021, 2:42 PM
dgibson updated the diff for D9057: dagit default app handler tweaks.

up

Aug 2 2021, 2:04 PM
dgibson requested review of D9181: allow step failure events to have no serialized error.
Aug 2 2021, 1:49 PM
dgibson updated the summary of D9179: speed up test project builds.
Aug 2 2021, 1:16 PM
dgibson added a reviewer for D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them: johann.
Aug 2 2021, 12:57 PM
dgibson updated the summary of D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them.
Aug 2 2021, 12:57 PM
dgibson requested review of D9179: speed up test project builds.
Aug 2 2021, 12:51 AM
dgibson requested review of D9178: add secrets/config maps from the executor config to the run launcher config, rather than replacing them.
Aug 2 2021, 12:18 AM

Aug 1 2021

dgibson closed D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"".
Aug 1 2021, 1:25 PM
dgibson added a reverting change for D9137: Revert "Load execution plan from the snapshot on the run when resume retrying": R1:b33469677b47: Bring back "Load execution plan from the snapshot on the run when resume….
Aug 1 2021, 1:25 PM
dgibson added a reverting change for R1:bfb90e8b0b44: Revert "Load execution plan from the snapshot on the run when resume retrying": R1:b33469677b47: Bring back "Load execution plan from the snapshot on the run when resume….
Aug 1 2021, 1:25 PM
dgibson committed R1:b33469677b47: Bring back "Load execution plan from the snapshot on the run when resume… (authored by dgibson).
Bring back "Load execution plan from the snapshot on the run when resume…
Aug 1 2021, 1:25 PM

Jul 31 2021

dgibson updated the diff for D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"".

up

Jul 31 2021, 12:15 AM
dgibson closed D9168: Insert solid_selection into pipeline snapshots in sorted order.
Jul 31 2021, 12:12 AM
dgibson committed R1:2f84b74c2953: Insert solid_selection into pipeline snapshots in sorted order (authored by dgibson).
Insert solid_selection into pipeline snapshots in sorted order
Jul 31 2021, 12:12 AM

Jul 30 2021

dgibson requested review of D9168: Insert solid_selection into pipeline snapshots in sorted order.
Jul 30 2021, 9:50 PM
dgibson published D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"" for review.
Jul 30 2021, 9:08 PM
dgibson added a reverting change for D9137: Revert "Load execution plan from the snapshot on the run when resume retrying": D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"".
Jul 30 2021, 7:11 PM
dgibson added a reverting change for R1:bfb90e8b0b44: Revert "Load execution plan from the snapshot on the run when resume retrying": D9167: Bring back "Load execution plan from the snapshot on the run when resume retrying"".
Jul 30 2021, 7:11 PM
dgibson requested changes to D9162: Move termination status checks from launcher to instance.

i like the idea of implementing the k8s termination check an a way that's akin to the other launchers, vs. affecting the other launchers

Jul 30 2021, 6:21 PM
dgibson requested changes to D9116: [Easy] Adding serialization decorators & convenience methods.
Jul 30 2021, 1:24 PM
dgibson added a comment to D9116: [Easy] Adding serialization decorators & convenience methods.

as per convo on the internal diff I think we're going to end up wanting to use ExternalXXXOrigins for this

Jul 30 2021, 1:24 PM
dgibson accepted D9155: show schedules/sensors that are not in storage.
Jul 30 2021, 1:20 PM

Jul 29 2021

dgibson accepted D9138: enable backfills to recover from failed partitioned solid selection runs.
Jul 29 2021, 5:43 PM
dgibson closed D9141: Add custom serializer to config type snapshot.
Jul 29 2021, 3:25 PM
dgibson committed R1:d6777854326a: Add custom serializer to config type snapshot (authored by dgibson).
Add custom serializer to config type snapshot
Jul 29 2021, 3:24 PM
dgibson updated the diff for D9141: Add custom serializer to config type snapshot.

add test case, make logic clearer

Jul 29 2021, 3:00 PM
dgibson published D9141: Add custom serializer to config type snapshot for review.
Jul 29 2021, 2:48 PM
dgibson accepted D9142: feat(helm): configure environment variables for K8sRunLauncher.

optionally something in the run launcher integration test that makes use of the env_vars field would be nice for e2e coverage, but we do have integration tests on the various pieces so that seems pretty good

Jul 29 2021, 2:33 PM
dgibson accepted D9140: feat: allow environment variables to be passed into K8sRunLauncher.

sweet, let's get this into the release today for hebo.

Jul 29 2021, 2:10 PM
dgibson added a comment to D9137: Revert "Load execution plan from the snapshot on the run when resume retrying".

Or I wonder if we had the same issue in the non-backfill case.. I’ll need to check

Jul 29 2021, 2:30 AM
dgibson accepted D9137: Revert "Load execution plan from the snapshot on the run when resume retrying".

No worries. If there’s a test you’re adding to backfills for the thing you’re working on that caught this, that would break if I ever tried to I revert this later, that would be cool.

Jul 29 2021, 2:29 AM

Jul 28 2021

dgibson accepted D9101: move dagster execution_tests in to seperate tox target.
Jul 28 2021, 9:05 PM
dgibson closed D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.
Jul 28 2021, 5:34 PM
dgibson committed R1:4ccdbb18c493: delegate k8s executor config to run launcher if it isn't set, instead of… (authored by dgibson).
delegate k8s executor config to run launcher if it isn't set, instead of…
Jul 28 2021, 5:34 PM
dgibson retitled D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default from RFC: delegate k8s executor config to run launcher if it isn't set, instead of assuming default to delegate k8s executor config to run launcher if it isn't set, instead of assuming default.
Jul 28 2021, 4:22 PM
dgibson closed D9117: Make ttl and backoff limit customizable.
Jul 28 2021, 4:20 PM
dgibson committed R1:3ae6cfd90b23: Make ttl and backoff limit customizable (authored by dish).
Make ttl and backoff limit customizable
Jul 28 2021, 4:20 PM
dgibson requested review of D9117: Make ttl and backoff limit customizable.
Jul 28 2021, 4:00 PM
dgibson abandoned D9120: Make run storage test a bit less strict.
Jul 28 2021, 3:53 PM
dgibson published D9120: Make run storage test a bit less strict for review.
Jul 28 2021, 2:50 PM
dgibson abandoned D8726: zlib compress repos in gRPC server.
Jul 28 2021, 2:50 PM
dgibson accepted D9105: Return false if describe_tasks isn't consistent.
Jul 28 2021, 1:06 PM

Jul 27 2021

dgibson accepted D9094: [RFC][grpc] turn on gzip and set higher receive limit.
Jul 27 2021, 9:37 PM
dgibson added a comment to D9094: [RFC][grpc] turn on gzip and set higher receive limit.

this looks good, but is back-compat a concern? if you upgrade dagit and don't upgrade the server will things break?

Jul 27 2021, 9:29 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

lint

Jul 27 2021, 8:35 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

up

Jul 27 2021, 8:01 PM
dgibson requested review of D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.
Jul 27 2021, 7:58 PM
dgibson requested review of D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.
Jul 27 2021, 7:58 PM
dgibson planned changes to D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

back to queue since this is a much more comprehensive change now

Jul 27 2021, 7:57 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

add test case

Jul 27 2021, 7:47 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

up

Jul 27 2021, 7:26 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

up

Jul 27 2021, 6:32 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

up

Jul 27 2021, 5:44 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

up

Jul 27 2021, 4:57 PM
dgibson accepted D9082: update stress tests for grpc limit repro.
Jul 27 2021, 4:52 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

up

Jul 27 2021, 4:50 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

up

Jul 27 2021, 4:45 PM
dgibson closed D9081: Use grpc rather than ipc to decide if a grpc server is running" - with integration test fix.
Jul 27 2021, 4:27 PM
dgibson added a reverting change for R1:5b6e23b470c5: Revert "Use grpc rather than ipc to decide if a grpc server is running": R1:29797eb9c6c9: Use grpc rather than ipc to decide if a grpc server is running" - with….
Jul 27 2021, 4:27 PM
dgibson added a reverting change for D9079: Revert "Use grpc rather than ipc to decide if a grpc server is running": R1:29797eb9c6c9: Use grpc rather than ipc to decide if a grpc server is running" - with….
Jul 27 2021, 4:27 PM
dgibson committed R1:29797eb9c6c9: Use grpc rather than ipc to decide if a grpc server is running" - with… (authored by dgibson).
Use grpc rather than ipc to decide if a grpc server is running" - with…
Jul 27 2021, 4:27 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

up

Jul 27 2021, 4:18 PM
dgibson updated the diff for D9078: delegate k8s executor config to run launcher if it isn't set, instead of assuming default.

add the rest of fields, which surfaces a bit of image pull policy inconsistency with the celery k8s config / run launcher..

Jul 27 2021, 4:13 PM