backfillClientStats loaded every column of every client once per inbound,
although enrichment only reads UUID, email and subscription ID. Each inbound
list read (panel list, the master's node traffic poll, Xray config builds,
bots) repeated that wide join per inbound. Identities now come from one narrow
clients/client_inbounds projection per 400 inbounds, honouring the supplied
transaction; stats output, sibling-traffic append order and the #6436
clients-table source of truth are unchanged.
Closes#6926
Most tests opened a throwaway panel DB with database.InitDB, which runs the
full AutoMigrate + seed on an empty file every time: ~230ms, and ~850ms under
-race because GORM's reflection-heavy migration is what the detector slows
most. internal/web/service does this in ~550 of its 830 tests, so the CI race
job spent ~10 of its ~14.6 minutes re-migrating empty databases.
internal/database/dbtest.InitDB migrates once per test process, then hands
each test its own copy of that file (~130ms under -race) and registers the
CloseDB cleanup. The copy then goes through InitDB like a panel restart, so
every test still starts from the state a fresh install has. Tests that reopen
an existing file, migrate a hand-built legacy DB or target Postgres keep
calling database.InitDB.
Locally under -race: internal/web/service 626s (last CI run) -> 114s,
internal/sub 246s -> 35s.
Replace strings.Split loops with strings.SplitSeq iterators in the CSV
parsers (reality_scan and the scale-test helpers) and swap a manual map
copy for maps.Copy in the MTProto traffic collector. No behavior change;
these are the fixes the modernize analyzer reports.
The paths that run continuously in production had no scale coverage: the
5s traffic poll (AddTraffic with its auto-renew and depleted scans), the
websocket snapshot the job broadcasts while a browser is connected, the
10s ip-limit job (hasLimitIp LIKE scan + per-email settings parse), a
subscription fetch inside a huge inbound, and the full Xray config build.
New benchmarks reuse the XUI_SCALE_TEST / XUI_DB_TYPE gating and stay
log-only. Sizes default to 10k/100k; XUI_SCALE_SIZES=500000 raises the
ladder without editing code. seedScaleDataset writes inbounds, clients,
client_inbounds and client_traffics directly in one transaction instead
of SyncInbound, so a 500k seed takes seconds. XUI_SCALE_DB_PATH persists
the seeded SQLite file for manual smoke runs against a live panel.
Local hot paths:
- autoRenewClients: replace the O(clients x expired) inner scan with an
email->traffic map lookup (quadratic at scale).
- node traffic sync: scope the client_traffics email-membership query to the
snapshot's emails instead of plucking the whole table every poll.
- add a (expiry_time, reset) index for the per-tick auto-renew filter.
- SQLite: add cache_size/mmap_size/temp_store pragmas (env-tunable); keep the
single-file DELETE journal and synchronous=FULL defaults.
- scale benchmarks now run on SQLite too via XUI_SCALE_TEST=1 (shared
setupScaleDB/resetScaleTables helpers), not just Postgres.
Node paths:
- bulk add/delete/adjust on a node-attached inbound folded one HTTP RPC per
client; above nodeBulkPushThreshold (32) mark the node dirty and let one
ReconcileNode push converge it instead of O(M) sequential round-trips.
Small ops keep the live per-client path. Also hoist nodePushPlan out of the
per-email delete loop.
- ReconcileNode skips inbounds whose wire payload is unchanged (per-tag
fingerprint on Remote), guarded by node-side tag presence so a restarted
node is still re-seeded.
Tests: auto-renew multi-inbound correctness, node-path dispatch (large ops
fold to dirty, small ops push live) via a manager runtime override seam, and
reconcile delta-skip.
Hamed
MHSanaei