Page MenuHomeElementl
Feed Advanced Search

Today

dgibson added a comment to D7901: Refactor step handler to only use ExecuteStepArgs.

"I think the abstraction is actually more useful if we limit it's parameters." - maybe this is the part I'm not following - what becomes more useful here? Is the idea that lots of step handlers will involve executing an execute_step CLI call in some environment, so it reduces boilerplate to have that all packaged up already?

Fri, May 14, 2:23 AM
dgibson accepted D7903: Load new image into existing kind cluster.
Fri, May 14, 2:18 AM
dgibson added a comment to D7901: Refactor step handler to only use ExecuteStepArgs.

I was under the impression that the main goal of ExecuteStepArgs was to have a serializable input to the execute_steps CLI command (something that you can JSON serialize), so its a little surprising to me to see it given this new job as well (the primary interface into step handlers in the new executor). Is the idea that this same class will be usable on the other side of a process boundary that takes in an ExecuteStepArgs?

Fri, May 14, 2:16 AM

Yesterday

dgibson updated the diff for D7894: Include container_image in recon_pipeline_from_origin and recon_repository_from_origin,.

up

Thu, May 13, 10:16 PM
dgibson planned changes to D7894: Include container_image in recon_pipeline_from_origin and recon_repository_from_origin,.
Thu, May 13, 10:14 PM
dgibson updated the diff for D7894: Include container_image in recon_pipeline_from_origin and recon_repository_from_origin,.

up

Thu, May 13, 9:53 PM
dgibson updated the summary of D7894: Include container_image in recon_pipeline_from_origin and recon_repository_from_origin,.
Thu, May 13, 9:42 PM
dgibson added reviewers for D7894: Include container_image in recon_pipeline_from_origin and recon_repository_from_origin,: johann, sashank, catherinewu.
Thu, May 13, 9:40 PM
dgibson updated the diff for D7894: Include container_image in recon_pipeline_from_origin and recon_repository_from_origin,.

up

Thu, May 13, 9:39 PM
dgibson closed D7905: Fix bad assert in workspace cleanup.
Thu, May 13, 8:59 PM
dgibson committed R1:3bb6cc08aee0: Fix bad assert in workspace cleanup (authored by dgibson).
Fix bad assert in workspace cleanup
Thu, May 13, 8:59 PM
dgibson published D7905: Fix bad assert in workspace cleanup for review.
Thu, May 13, 8:31 PM
dgibson closed D7899: Fix race condition when reload workspace is called from a background thread.
Thu, May 13, 7:23 PM
dgibson committed R1:6f32cec16233: Fix race condition when reload workspace is called from a background thread (authored by dgibson).
Fix race condition when reload workspace is called from a background thread
Thu, May 13, 7:23 PM
dgibson updated the diff for D7899: Fix race condition when reload workspace is called from a background thread.

can't hurt!

Thu, May 13, 7:22 PM
dgibson requested review of D7894: Include container_image in recon_pipeline_from_origin and recon_repository_from_origin,.
Thu, May 13, 5:09 AM
dgibson closed D7883: Add an event before the create_namespaced_job call in the K8sRunLauncher.
Thu, May 13, 4:50 AM
dgibson committed R1:6d1459bac7a0: Add an event before the create_namespaced_job call in the K8sRunLauncher (authored by dgibson).
Add an event before the create_namespaced_job call in the K8sRunLauncher
Thu, May 13, 4:50 AM

Wed, May 12

dgibson added a comment to D7878: monitor sensor 1/ rename cursor to before_cursor, add arg after_cursor and ascending to RunStorage.get_runs.

That would work too - the most important thing to me is that you’re able to
paginate in a forwards direction with a limit

Wed, May 12, 11:52 PM
dgibson requested review of D7883: Add an event before the create_namespaced_job call in the K8sRunLauncher.
Wed, May 12, 4:45 PM
dgibson requested changes to D7878: monitor sensor 1/ rename cursor to before_cursor, add arg after_cursor and ascending to RunStorage.get_runs.

Goal here seems very reasonable. Question for you and other reviewers - Would adding an ascending=True/False flag be another way to deal with this? (i.e. the same cursor could be either a before_cursor or after_cursor depending on the order that you want the results)?

Wed, May 12, 3:51 PM
dgibson accepted D7869: [dagit] Rename auth header (1/3).
Wed, May 12, 3:07 PM
dgibson accepted D7879: pin jinja2 to fix tests.

awesome, thx for digging into this

Wed, May 12, 2:15 AM
dgibson added a comment to D7879: pin jinja2 to fix tests.

would you mind referencing this issue in a comment next to the pins, so we can track unpinnning? https://github.com/dagster-io/dagster/issues/4167

Wed, May 12, 2:08 AM
dgibson requested changes to D7879: pin jinja2 to fix tests.

small thing, otherwise looks good

Wed, May 12, 2:04 AM

Tue, May 11

dgibson accepted D7862: fix sql event log cursors.

see question inline about efficiency? thanks for resolving https://github.com/dagster-io/dagster/issues/3621 though, thought i was going crazy

Tue, May 11, 9:19 PM
dgibson added reviewers for D7869: [dagit] Rename auth header (1/3): sashank, max.
Tue, May 11, 9:00 PM
dgibson accepted D7857: [Easy] Remove unneccesary verify_complete calls.
Tue, May 11, 3:50 AM

Mon, May 10

dgibson closed D7846: Expose Workspace on WorkspaceProcessContext.
Mon, May 10, 9:42 PM
dgibson committed R1:181200464014: Expose Workspace on WorkspaceProcessContext (authored by dgibson).
Expose Workspace on WorkspaceProcessContext
Mon, May 10, 9:42 PM
dgibson published D7846: Expose Workspace on WorkspaceProcessContext for review.
Mon, May 10, 9:37 PM

Fri, May 7

dgibson accepted D7807: Add volume mounts to dagster k8s jobs.
Fri, May 7, 6:43 PM
dgibson accepted D7787: [dagit] Upgrade apollo.
Fri, May 7, 2:10 PM
dgibson added a comment to D7688: [dagit] Make Non-Subscription GraphQL ops use HTTP.

I haven't been following the full saga of this diff closely, but what's here seems reasonable - any blockers left here now that the release has shipped?

Fri, May 7, 1:35 PM

Thu, May 6

dgibson accepted D7781: [dagit] Add a few more main exports.
Thu, May 6, 11:04 PM
dgibson published D7781: [dagit] Add a few more main exports for review.
Thu, May 6, 11:04 PM
dgibson accepted D7624: [RFC] Celery-less k8s executor.

sweet!

Thu, May 6, 3:40 PM
dgibson accepted D7756: [dagit] Repair a type import.
Thu, May 6, 2:14 PM

Tue, May 4

dgibson closed D7732: [easy] fix postgres snapshots.
Tue, May 4, 11:36 PM
dgibson committed R1:edb23a912131: [easy] fix postgres snapshots (authored by dgibson).
[easy] fix postgres snapshots
Tue, May 4, 11:36 PM
dgibson published D7732: [easy] fix postgres snapshots for review.
Tue, May 4, 10:28 PM
dgibson closed D7731: [easy] Add can_watch() to TestEventLogStorage.
Tue, May 4, 9:59 PM
dgibson committed R1:2cd951628542: [easy] Add can_watch() to TestEventLogStorage (authored by dgibson).
[easy] Add can_watch() to TestEventLogStorage
Tue, May 4, 9:59 PM
dgibson requested review of D7731: [easy] Add can_watch() to TestEventLogStorage.
Tue, May 4, 9:56 PM
dgibson closed D7719: Allow TestEventLogStorage subclasses that cannot wipe storage.
Tue, May 4, 9:06 PM
dgibson committed R1:c732623344c8: Allow TestEventLogStorage subclasses that cannot wipe storage (authored by dgibson).
Allow TestEventLogStorage subclasses that cannot wipe storage
Tue, May 4, 9:06 PM
dgibson requested changes to D7624: [RFC] Celery-less k8s executor.

for that round of feedback

Tue, May 4, 6:58 PM
dgibson added a comment to D7714: Add documentation for manually scheduling partition, adding to a repo.

Thank you for doing this, this was a big hole - just wondering if there a more plausible example than random_partition though? Maybe, like.. take the day mod 3 or something? (Something that demonstrates that a likely input into the selector will be context.scheduled_execution_time would be nice)

Tue, May 4, 4:59 PM
dgibson added a comment to D7624: [RFC] Celery-less k8s executor.

nice, I think this turned out quite clean. My only real comments are about the initialize() method and maybe some naming. @alangenfeld probably has some good thoughts there as well

Tue, May 4, 3:15 PM
dgibson closed D7705: Move grpc server watcher to workspace rather than repo location.
Tue, May 4, 3:09 PM
dgibson committed R1:61ad55fad153: Move grpc server watcher to workspace rather than repo location (authored by dgibson).
Move grpc server watcher to workspace rather than repo location
Tue, May 4, 3:09 PM
dgibson added a comment to D7705: Move grpc server watcher to workspace rather than repo location.

(also fixed reconnect_loop to run forever until it reconnects, good call)

Tue, May 4, 2:59 PM
dgibson updated the diff for D7705: Move grpc server watcher to workspace rather than repo location.

up

Tue, May 4, 2:47 PM
dgibson abandoned D7717: Allow TestEventLogStorage subclasses that cannot wipe.
Tue, May 4, 2:36 PM
dgibson added a reviewer for D7719: Allow TestEventLogStorage subclasses that cannot wipe storage: jordansanders.
Tue, May 4, 2:09 PM
dgibson added a comment to D7594: Allow the set of default executors to use when none are specified to be customized.

right, so we'd need to also separate out the object that Dagit operates against to generate the config schema to be separate from just a field on the pipeline snapshot (and at that stage we could bring in the instance) - I don't have a great sense into how messy that would be

Tue, May 4, 2:06 PM
dgibson requested review of D7719: Allow TestEventLogStorage subclasses that cannot wipe storage.
Tue, May 4, 1:42 PM
dgibson requested review of D7717: Allow TestEventLogStorage subclasses that cannot wipe.
Tue, May 4, 3:42 AM
dgibson added a reviewer for D7705: Move grpc server watcher to workspace rather than repo location: alangenfeld.
Tue, May 4, 2:39 AM
dgibson updated the diff for D7705: Move grpc server watcher to workspace rather than repo location.

add a lock to Workspace since the background thread might be updating the various location dicts while the main thread is using them

Tue, May 4, 2:34 AM
dgibson updated the diff for D7705: Move grpc server watcher to workspace rather than repo location.

up

Tue, May 4, 1:34 AM
dgibson accepted D7699: [dagit] Split apart Instance Health query.

thanks!

Tue, May 4, 12:46 AM
dgibson planned changes to D7705: Move grpc server watcher to workspace rather than repo location.
Tue, May 4, 12:40 AM

Mon, May 3

dgibson added a comment to D7707: Fix daemon liveness check command.

(and thanks for fixing)

Mon, May 3, 10:20 PM
dgibson accepted D7707: Fix daemon liveness check command.

accepting b/c i have to step away for a bit, but see instructions inline - sorry for breaking

Mon, May 3, 10:19 PM
dgibson added a comment to D7707: Fix daemon liveness check command.

Seems like our liveness test coverage is not so great

Mon, May 3, 10:18 PM
dgibson requested changes to D7707: Fix daemon liveness check command.

oof, sorry, this my bad - you want to remove the option altogether, not add the argument

Mon, May 3, 10:17 PM
dgibson published D7705: Move grpc server watcher to workspace rather than repo location for review.

Here is the stack trace I get when a server goes down: https://dagster.phacility.com/P197

Mon, May 3, 10:02 PM
dgibson added a comment to D7594: Allow the set of default executors to use when none are specified to be customized.

lose not use, sorry

Mon, May 3, 9:29 PM
dgibson added a comment to D7594: Allow the set of default executors to use when none are specified to be customized.

the big thing you use there is config playground typeahead support in dagit right?

Mon, May 3, 9:29 PM
dgibson accepted D7692: [easy] Fix bad merge for location subscriber.
Mon, May 3, 4:17 PM
dgibson accepted D7673: test solid_selection and manual partition schedule.
Mon, May 3, 2:27 PM
dgibson resigned from D7665: Scaffold networking and load balancer for example ECS deployment.

will defer to others on the specifics of the CloudFormation, one question about placement inline

Mon, May 3, 2:25 PM

Sat, May 1

dgibson closed D7678: Make daemon heartbeat failure tolerance configurable and less aggressive by default.
Sat, May 1, 1:47 AM
dgibson committed R1:53108093a59c: Make daemon heartbeat failure tolerance configurable and less aggressive by… (authored by dgibson).
Make daemon heartbeat failure tolerance configurable and less aggressive by…
Sat, May 1, 1:47 AM
dgibson updated the diff for D7678: Make daemon heartbeat failure tolerance configurable and less aggressive by default.

up

Sat, May 1, 1:15 AM
dgibson updated the diff for D7678: Make daemon heartbeat failure tolerance configurable and less aggressive by default.

incorporate into liveness/health checks

Sat, May 1, 12:55 AM

Fri, Apr 30

dgibson requested review of D7678: Make daemon heartbeat failure tolerance configurable and less aggressive by default.
Fri, Apr 30, 10:03 PM
dgibson accepted D7682: Re-rache repository locations on WorkspaceProcessContext everytime a request is made.
Fri, Apr 30, 5:57 PM
dgibson requested changes to D7682: Re-rache repository locations on WorkspaceProcessContext everytime a request is made.

this seems better, the reload_workspace path has a problem now though

Fri, Apr 30, 5:44 PM
dgibson requested changes to D7682: Re-rache repository locations on WorkspaceProcessContext everytime a request is made.

as per slack convo can this diff remove WorkspaceProcessContext.repository_locations altogether and always delegate to Workspace.repository_locations

Fri, Apr 30, 5:30 PM
dgibson closed D7664: Add ability for a partition selector to return multiple partitions.
Fri, Apr 30, 2:10 PM
dgibson committed R1:d3b8e0b6bf8c: Add ability for a partition selector to return multiple partitions (authored by dgibson).
Add ability for a partition selector to return multiple partitions
Fri, Apr 30, 2:09 PM

Thu, Apr 29

dgibson requested review of D7664: Add ability for a partition selector to return multiple partitions.
Thu, Apr 29, 7:50 PM
dgibson accepted D7642: [1/n] Add exhaustive check for mypy.

ah nice, I would use this frequently

Thu, Apr 29, 1:28 PM

Wed, Apr 28

dgibson accepted D7644: Fix typo in monthly schedule decorator.
Wed, Apr 28, 9:45 PM

Tue, Apr 27

dgibson accepted D7627: [dagster-k8s] retry on 500.
Tue, Apr 27, 8:11 PM
dgibson abandoned D7123: Don't throw when two simultaneous alembic stamps result in a unique key violation.
Tue, Apr 27, 1:44 AM
dgibson updated the diff for D7123: Don't throw when two simultaneous alembic stamps result in a unique key violation.

bringing this back since i think its causing daemon integration test failures

Tue, Apr 27, 1:42 AM
dgibson accepted D7603: [dagster-postgres] dont swallow exc in watcher_thread.
Tue, Apr 27, 1:25 AM
dgibson closed D7600: Heartbeat right away in the daemon on startup.
Tue, Apr 27, 1:17 AM
dgibson committed R1:1541ec43f343: Heartbeat right away in the daemon on startup (authored by dgibson).
Heartbeat right away in the daemon on startup
Tue, Apr 27, 1:16 AM

Mon, Apr 26

dgibson requested review of D7600: Heartbeat right away in the daemon on startup.
Mon, Apr 26, 5:28 PM
dgibson added inline comments to D7594: Allow the set of default executors to use when none are specified to be customized.
Mon, Apr 26, 3:20 PM
dgibson abandoned D7593: Allow the set of default executors to be customized in different instances/entry points.
Mon, Apr 26, 2:53 PM
dgibson added a comment to D7594: Allow the set of default executors to use when none are specified to be customized.

nearly all of this diff is passing things down from [2] and [3] so that they can be accessed in [1] (and customized in the near future)

Mon, Apr 26, 2:40 PM
dgibson published D7594: Allow the set of default executors to use when none are specified to be customized for review.

spurious coverall test failure, will investigate separately

Mon, Apr 26, 2:30 PM
dgibson accepted D7588: Add directions for running integration tests locally.
Mon, Apr 26, 1:54 PM

Sun, Apr 25

dgibson requested review of D7593: Allow the set of default executors to be customized in different instances/entry points.
Sun, Apr 25, 8:42 PM

Fri, Apr 23

dgibson accepted D7574: throw interrupt error in incomplete plans.

for extra credit, add additional logic to test_terminate_pipeline_on_celery that covers the termination message and/or exception class

Fri, Apr 23, 5:44 PM