HomePhabricator

[RFC] DagsterInstance.optimize_for_dagit

Description

[RFC] DagsterInstance.optimize_for_dagit

Summary:
This diff introduces a call on the DagsterInstance to allow it to optimize for being the central copy used in the long running dagit process. This allows us to control two behaviors

  • connection re-use / pooling
  • statement timeouts

that we only really *know* what we want if its the copy used by dagit. We don't want long running connections on random instances of DagsterInstance, and we don't want an aggressive statement_timeout breaking things like instance migrations when used in the CLI.

Since its difficult to thread this option in at construction time due to the existing DagsterInstance / configurable class / instance ref machinery, adding this opt-in function was my best idea. Open to other suggestions.

Test Plan:
added tests
loaded dagit runs page with an rds database in us-west from my laptop here in MN, time drops from 24 seconds to 4 seconds since it can re-use connections.

Reviewers: max, nate, sandyryza, schrockn, dgibson, prha

Reviewed By: sandyryza, schrockn, prha

Differential Revision: https://dagster.phacility.com/D4736

Details

Provenance
alangenfeldAuthored on Mon, Oct 12, 9:56 PM
Reviewer
sandyryza
Differential Revision
D4736: [RFC] DagsterInstance.optimize_for_dagit
Parents
R1:00705e5c2465: grpc generated code
Branches
Unknown
Tags
Unknown