Page MenuHomePhabricator

Conslidate location reloading into single thread

Authored by sashank on Dec 9 2020, 9:36 PM.


Test Plan


Diff Detail

R1 dagster
Lint OK
No Unit Test Coverage

Event Timeline

dgibson added a subscriber: alangenfeld.
dgibson added inline comments.

as discussed over slack I think we're best served by finding a way to run this event loop on the main thread, which might involve something with gevent or flask so that we can hook into the existing event loop?

@alangenfeld for context, we realized this background gRPC server update stuff is running on a background thread rather than the main thread, but dagit isn't set up to be multi-threaded. So we're trying to find a way to keep these callbacks firing on the background thread, but have the actual work that they fire execute on the main thread. Dunno if you're aware of any way to do that by any chance?


i talked with a sashank a bit yesterday about this and the idea we came to was to change up how DagsterGraphQLContext works and set things up such that we have a request scoped context object that grabs whatever the current workspace is at the top of the request. We need that kind fo set up for when these requests actually cooperatively multi task and should work with the pre-request bound object maybe changing from another thread.

will try to implement alex's suggestion today