HomeElementl

Log heartbeats during the first daemon iteration (while keeping the first error…

Description

Log heartbeats during the first daemon iteration (while keeping the first error heartbeat)

Summary: I realized that to get the full benefits of https://dagster.phacility.com/D6641 (avoiding the situation where a bunch of schedules/sensors cause an iteration to take more than 2 minutes and trigger a heartbeat failure), we need to be heartbeating more often on the first iteration as well. To still accomplish the goal of not incorrectly saying that the daemon is healthy, I added logic to ensure we log a heartbeat with an error the first time one comes up. This could lead us to incorrectly saying the first iteration is healthy, but I think that's better than the daemon crashing due to a long first iteration.

Test Plan: Integration, BK (see channes to error test)

Reviewers: johann, prha

Reviewed By: johann

Differential Revision: https://dagster.phacility.com/D6654

Details

Provenance
dgibsonAuthored on Feb 24 2021, 5:01 PM
Reviewer
johann
Differential Revision
D6654: Log heartbeats during the first daemon iteration (while keeping the first error heartbeat)
Parents
R1:f0e36c6b6d8f: Add Slack badge to README
Branches
Unknown
Tags
Unknown