feat(notifications): add a failed-check threshold for node.down alerts (#6768)

One lost heartbeat on a lossy link flipped a node offline and sent node.down,
and the next probe sent node.up, so a flaky node flooded Telegram, email and
Discord with alert pairs.

A new nodeDownThreshold setting (default 1, today's behaviour) works like
outboundDownThreshold: node.down fires once the failed-check streak reaches it,
once per outage, and node.up only after a node.down was sent. The announcement
is persisted in nodes.down_notified, so a restart inside the threshold window
neither drops the outage's node.down nor re-announces nodes already reported
offline; nodes offline at upgrade are seeded as announced.
This commit is contained in:
Grigoriy
2026-10-24 05:26:13 +03:00
committed by GitHub
parent c3037fc716
commit 2f62dadd25
34 changed files with 379 additions and 35 deletions
+2
View File
@@ -878,6 +878,8 @@ type Node struct {
// save that grows the selection zeroes it so reconcile waits before sweeping.
InboundsAdoptedAt int64 `json:"-" gorm:"column:inbounds_adopted_at;default:0"`
DownNotified bool `json:"-" gorm:"column:down_notified;default:false"`
InboundCount int `json:"inboundCount" gorm:"-" example:"5"`
ClientCount int `json:"clientCount" gorm:"-" example:"27"`
OnlineCount int `json:"onlineCount" gorm:"-" example:"3"`