63b46cd6 made a multi-inbound client create apply its inbounds concurrently,
and d34ec97f did the same for the single-client update, delete and detach. The
bulk operations were never converted, so they still walked their inbounds in a
plain sequential loop with a node RPC in each iteration — and those are what the
panel actually calls for a multi-select delete or an enable/disable, which is
why editing and deleting still felt slow on a master with several nodes.
Measured with a node runtime injecting 100ms per RPC, one client per node:
nodes=1 update=101ms bulkSetEnable=101ms bulkAdjust=101ms bulkDelete=101ms
nodes=3 update=102ms bulkSetEnable=302ms bulkAdjust=304ms bulkDelete=303ms
nodes=5 update=203ms bulkSetEnable=504ms bulkAdjust=504ms bulkDelete=504ms
after, all of them track the single-client ops:
nodes=3 bulkSetEnable=103ms bulkAdjust=101ms bulkDelete=101ms
nodes=5 bulkSetEnable=202ms bulkAdjust=202ms bulkDelete=202ms
Generalize the fanout into fanoutInboundResults over an arbitrary per-inbound
result type and route six loops through it: BulkDelete, BulkSetEnable,
BulkAdjust, BulkDetach, BulkAttach, BulkCreate, plus applyClientFieldByEmail —
the field edit behind the Telegram bot's enable/limit/expiry buttons and the
LDAP job. Each keeps its preparation sequential and overlaps only the node
pushes, inheriting the same concurrency cap and per-inbound panic recovery.
Two ordering details the sequential loops got for free and the fanout must do
itself: the three loops that ranged a map now walk sortedInboundIds, so which
inbound wins a per-email skip reason is the lowest id instead of whatever the
map yielded; and BulkAttach de-duplicates a repeated inbound id up front,
because the second pass used to see the client the first pass had just added.
The allocating paths stay serial when a tunnel inbound is involved. WireGuard
and AmneziaWG pick a free peer address by reading every inbound's used-set
before they write, so two overlapping allocations hand out the same address and
the in-transaction re-check refuses the loser — a bulk create of two clients
onto two wg inbounds returned created=1. addFanoutLimit drops those batches back
to one at a time; every other protocol keeps the full cap.
Eight tests: seven barrier tests that a sequential caller cannot satisfy (peak
pushes in flight is 1 without the change, 4 with it), and one that pins the
tunnel allocation.
Editing, deleting or detaching a client on a master with several nodes took
one node round-trip per node, added end to end. Create and Attach already
fanned their per-inbound applies out through fanoutInboundClientAdds, but
Update, Delete, Detach and DeleteByEmail's record-less fallback still walked
their inbounds in a plain sequential loop, and each iteration blocks on a
node RPC (10s timeout, more when a node is slow or has just gone unreachable
and the heartbeat has not marked it offline yet).
Measured with a node runtime injecting 100ms per RPC, before:
nodes=1 create=101ms update=101ms delete=101ms
nodes=3 create=102ms update=303ms delete=302ms
nodes=5 create=202ms update=504ms delete=504ms
after, all three track create:
nodes=3 create=102ms update=102ms delete=101ms
nodes=5 create=203ms update=203ms delete=203ms
Generalize the existing helper into fanoutInboundApplies over an inboundApply
list and route the four remaining loops through it, so they inherit the same
concurrency cap, per-inbound panic recovery and joined errors. Each caller
still builds its payloads sequentially first: fillProtocolDefaults mints the
shared credentials on the first inbound and every later one reuses them, so
that order has to stay deterministic. Only the applies overlap; their DB work
still serializes through the single traffic writer, and the per-inbound
mutation lock is unchanged, which is exactly what Create has relied on.
Behaviour change: one failing inbound no longer aborts the remaining ones,
matching what Create already does. The error still names each failed inbound
and the record-level writes are still skipped when any inbound failed.
The snapshot merge on the same serialized writer was measured as a second
suspect and cleared: ~43ms per node at 500 clients, an order of magnitude
below the RPC serialization.
Sanaei