Commit Graph

2 Commits

Author SHA1 Message Date
Diego Rodrigues de Sa e Souza
86fc1aade2 fix(ops): judge the canary install by the SHA on disk, not npm's exit code (#10699)
`npm install -g <tarball>` on the .17 gateway writes the whole package and then fails
renaming the old tree into its staging directory (ENOTEMPTY, exit 217). The canary read
that non-zero exit as "install failed", aborted before the restart, and discarded npm's
stderr through execFileSync throwing — so on 2026-08-18 the deploy stopped half-done
twice, each time leaving new files on disk under an old running process, with no clue in
the log.

The exit code is not trustworthy in either direction: the 2026-08-14 outage installed a
package built from the wrong branch and exited 0. classifyInstallOutcome() therefore
decides on the BUILD_SHA read back from the installed package, and fails closed when it
is absent or does not match — a zero exit with the wrong artifact is still a failure.

npm reuses the same staging directory name, so the orphan blocks the next install with
the same error; orphanStagingDirFromStderr() surfaces the exact path. It is not removed
automatically — that is an rm -rf under /usr/lib, not something a deploy script should
decide on its own.

Refs #10429

Co-authored-by: Xiangzhe <bakryun0718@proton.me>
2026-08-18 21:51:01 -03:00
Diego Rodrigues de Sa e Souza
1287b6a75d feat(ops): canary deploy with provenance gate, real smoke and rollback anchor (#10446)
Deploying the internal gateway was a manual build/pack/scp/npm-i/pm2-restart sequence with no record of what landed and no proof it served traffic. On 2026-08-14 that shipped a package built from a branch predating #10373: the process came up, health said 'healthy', and every request returned 502 until a human hit it.

scripts/ops/deployCanary.ts holds the policy as pure functions — refuse an artifact that is not traceable to the release line (reusing #10427), and grade the deploy on health PLUS at least one real completion. Zero probes fails: 'no probe ran' must never read as 'everything is fine', which is exactly how a broken egress path hides behind a green health check. Remote steps are argv arrays, never shell strings (Hard Rule #13), ordered so the rollback anchor is captured before the install overwrites it.

scripts/ops/deploy-canary.mjs performs the side effects, supports --dry-run, and prints the rollback command when the smoke fails.

Closes #10429
2026-08-15 03:22:09 -03:00