feat(notifications): add a failed-check threshold for node.down alerts (#6768)

One lost heartbeat on a lossy link flipped a node offline and sent node.down,
and the next probe sent node.up, so a flaky node flooded Telegram, email and
Discord with alert pairs.

A new nodeDownThreshold setting (default 1, today's behaviour) works like
outboundDownThreshold: node.down fires once the failed-check streak reaches it,
once per outage, and node.up only after a node.down was sent. The announcement
is persisted in nodes.down_notified, so a restart inside the threshold window
neither drops the outage's node.down nor re-announces nodes already reported
offline; nodes offline at upgrade are seeded as announced.
This commit is contained in:
Grigoriy
2026-10-24 05:26:13 +03:00
committed by GitHub
parent c3037fc716
commit 2f62dadd25
34 changed files with 379 additions and 35 deletions
+6
View File
@@ -1064,6 +1064,12 @@ func (s *NodeService) ClearNodeDirty(id int, dirtyAt int64) error {
Update("config_dirty", false).Error
}
func (s *NodeService) SetDownNotified(id int, notified bool) error {
return database.GetDB().Model(model.Node{}).
Where("id = ?", id).
Update("down_notified", notified).Error
}
func (s *NodeService) MarkNodeInboundsAdopted(id int) error {
if id <= 0 {
return nil
+10
View File
@@ -238,6 +238,8 @@ var defaultValueMap = map[string]string{
// Consecutive failed observatory probes before an outbound.down event fires
"outboundDownThreshold": "3",
"nodeDownThreshold": "1",
// Email (SMTP) notifications
"smtpEnable": "false",
"smtpHost": "",
@@ -1641,6 +1643,14 @@ func (s *SettingService) SetOutboundDownThreshold(value int) error {
return s.setInt("outboundDownThreshold", value)
}
func (s *SettingService) GetNodeDownThreshold() (int, error) {
return s.getInt("nodeDownThreshold")
}
func (s *SettingService) SetNodeDownThreshold(value int) error {
return s.setInt("nodeDownThreshold", value)
}
// SecretClears marks redacted secrets the user explicitly emptied. Without a
// flag, a blank submitted secret means "unchanged" (the field is always served
// blank to the browser) and the stored value is preserved.