feat(notifications): add a failed-check threshold for node.down alerts (#6768)

One lost heartbeat on a lossy link flipped a node offline and sent node.down,
and the next probe sent node.up, so a flaky node flooded Telegram, email and
Discord with alert pairs.

A new nodeDownThreshold setting (default 1, today's behaviour) works like
outboundDownThreshold: node.down fires once the failed-check streak reaches it,
once per outage, and node.up only after a node.down was sent. The announcement
is persisted in nodes.down_notified, so a restart inside the threshold window
neither drops the outage's node.down nor re-announces nodes already reported
offline; nodes offline at upgrade are seeded as announced.
This commit is contained in:
Grigoriy
2026-10-24 05:26:13 +03:00
committed by GitHub
parent c3037fc716
commit 2f62dadd25
34 changed files with 379 additions and 35 deletions
+5
View File
@@ -174,6 +174,7 @@ export class AllSetting {
smtpCpu = 80;
smtpMemory = 80;
outboundDownThreshold = 3;
nodeDownThreshold = 1;
hasTgBotToken = false;
hasTwoFactorToken = false;
hasLdapPassword = false;
@@ -216,6 +217,10 @@ export class AllSetting {
this.outboundDownThreshold = Number.isFinite(threshold)
? Math.min(100, Math.max(1, threshold))
: 3;
const nodeThreshold = Math.round(Number(this.nodeDownThreshold));
this.nodeDownThreshold = Number.isFinite(nodeThreshold)
? Math.min(100, Math.max(1, nodeThreshold))
: 1;
}
equals(other: AllSetting): boolean {