Page MenuHomeElementl

Add a default gRPC timeout to better handle slow/hanging sensors in the daemon
ClosedPublic

Authored by dgibson on Apr 16 2021, 3:58 PM.

Details

Summary

Rather than hanging forever when the gRPC server executes user code that hangs or takes forever, timeout after 60 seconds. This won't solve every problem with sensors that hang, but at least will give some indication to the user that something is wrong with a particular sensor (rather than crashing the entire daemon because of one bad sensor)

Test Plan

BK

Diff Detail

Repository
R1 dagster
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 16 2021, 4:17 PM
Harbormaster failed remote builds in B28989: Diff 35580!

skip tests from the soon-to-be-removed cron scheduler that don't play nicely with timeouts = freezing time to 2019

I think this is good. We might eventually want to fork a process for each evaluation, and then change the timeout based on the min interval? But better to make this change and then think through the ramifications of the other stuff.

This revision is now accepted and ready to land.Apr 16 2021, 5:00 PM