fix(node): don't delete clients when a node reports an empty snapshot

A node snapshot that comes back with zero clients for an inbound the hub
still has clients on was treated as authoritative: SyncInbound strips
every link for that inbound, the orphan sweep marks the now-linkless
clients, and ReapSyncOrphans hard-deletes them once the grace period
elapses. But a zero-client snapshot is indistinguishable from a degraded
node — one that was just deleted, reset, restarted, or answered before
its config loaded. On 2026-10-04 this deleted clients across the whole
hub when a single node was removed.

Treat a zero-client snapshot for an inbound that still has clients as
non-authoritative: skip the link rebuild and the orphan sweep for that
inbound (the same handling a failed SyncInbound already gets) and wait
for a snapshot that carries clients. Removing a node's last client is
done from the hub (which updates links and pushes); a node still serving
other clients prunes a removed one through the existing partial path.

The two orphan tests that drove removal via an empty snapshot now drive
it via a partial snapshot (node alive, still serving another client),
the authoritative path. New tests in node_degraded_snapshot_test.go
cover the guard and its narrowness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Yuriy Khachaturian
2026-10-04 23:49:36 +03:00
parent 815c9c5772
commit 4295ee8a28
3 changed files with 191 additions and 9 deletions
+21
View File
@@ -1228,6 +1228,27 @@ func (s *InboundService) setRemoteTrafficLocked(nodeID int, snap *runtime.Traffi
applyMasterClientLifecycle(&clients[i], existing, csPtr)
filtered = append(filtered, clients[i])
}
// Empty-snapshot guard. A node that comes back reporting zero clients for
// an inbound the hub still has clients on is indistinguishable from a
// genuine "every client was removed" — and in practice it is almost always
// the former: the node was just deleted, reset, restarted, or returned a
// snapshot before its config loaded. Acting on it is destructive and hard
// to undo: SyncInbound makes the inbound's links match the set exactly, so
// an empty set deletes every link for the inbound; a client shared across
// nodes then loses its links one node at a time and, once the last one
// goes, is orphan-marked and hard-deleted by ReapSyncOrphans — the
// 2026-10-04 outage, where removing one node deleted clients that still
// lived on the others. Treat a zero-client snapshot as non-authoritative:
// skip the link rebuild and the orphan sweep for this inbound (the same
// handling a failed SyncInbound gets) and wait for a snapshot that carries
// clients. Removing the last client from a node is instead done from the
// hub (which updates links and pushes); a node that still serves other
// clients prunes a removed one authoritatively through the partial path.
if len(filtered) == 0 && len(oldEmailsRows) > 0 {
logger.Warningf("setRemoteTraffic: node snapshot for tag %q reported zero clients while the hub has %d attached — treating as a degraded snapshot, skipping link rebuild and orphan sweep for this inbound", snapIb.Tag, len(oldEmailsRows))
syncFailedInbounds[c.Id] = struct{}{}
continue
}
localEmails := make([]string, 0, len(filtered))
for i := range filtered {
if filtered[i].Email != "" {