feat(amneziawg): add opt-in Linux kernel datapath (#6877)

* feat(amneziawg): add opt-in Linux kernel datapath

Add the kernel backend as an explicit per-inbound option while keeping the embedded backend as the default. Include host preflight, UI/API support, the bundled installer CLI, lifecycle and accounting integration, regression tests, and upstream-facing documentation. Kernel mode is NAT-only and rejects unsupported routing, IPv6, forwarding, and pinned-listen configurations. Local saves check the local host; remote nodes check their own host during apply.

Preserve adopted interface accounting across panel restarts, retain manager state after failed teardown so the job can retry, prevent embedded startup while a kernel interface remains live, and prune collector state for removed peers. Tested the restart accounting path on Debian 13 with XanMod 7.1.11 on x86_64; Docker and cross-kernel coverage are not claimed.

* fix(amneziawg): align readiness and endpoint reporting

Expire endpoint sightings using the latest handshake and sweep stale addresses for the polled interface. Keep the readiness API schema, generated client types, and ready-host empty blocker list consistent; watch the nested form field and defer local readiness checks for remote targets. Document safe rollback before using a build without kernel support.

Regression evidence: the idle endpoint test failed before the collector change because the old address remained visible after 31 minutes; endpoint tests pass after the fix. The form component suite passes 3 tests; restoring the invalid array watch made its off-mode severity test fail, while the corrected dot-path watch passes. Targeted Go regressions across amneziawgkern, job, and service pass.

* style(amneziawg): format readiness test

* test: reproduce kernel AWG client validation gaps

* test(service): cover kernel AWG shared client fanout

* fix(amneziawg): validate kernel client fanout and config values

* test(amneziawgkern): cover owned interface lifecycle recovery

* fix(amneziawgkern): recover owned interfaces on restart

* test(amneziawgkern): retain visibility after recovered cleanup failure

* fix(amneziawgkern): report recovered interfaces while cleanup retries

* style(amneziawgkern): satisfy static analysis

* test(amneziawgkern): cover long rendered peer comments

* fix(amneziawgkern): accept long generated config comments

* test(amneziawgkern): refuse foreign occupied interfaces

Capture the current Ensure ordering failure with real temporary config paths and manager side-effect boundaries.

* test(amneziawgkern): cover stale interface collisions

Extend the Ensure regression across absent links, mismatched live interfaces, cached manager state, and Remove teardown.

* fix(amneziawgkern): verify interface ownership before mutation

Refuse to replace or tear down links unless the canonical config is a valid panel-owned file and its key matches the live interface. Preserve normal adoption and key rotation, and document migration of legacy unmarked configs.

Validation: go test ./internal/amneziawgkern -count=1; go test -race ./internal/amneziawgkern -count=1; golangci-lint run ./internal/amneziawgkern/...; git diff --check.

* build: update Go toolchain to 1.27.3

* test(amneziawg): reproduce unsafe key rotation recovery

Keep the previous owned config authoritative until teardown succeeds, and require fresh managers to rebuild structural key changes. These regressions expose the stuck live-key/disk-key mismatch after teardown failure.

* fix(amneziawg): recover interrupted key rotations

Keep the old owned config on disk until a structural rebuild has confirmed the previous interface is down. Fresh managers only adopt when the live config and desired structural identity match, so retries and removals retain the key needed to prove ownership.

* fix(scanner): align Go image with module version

* fix(docs): pin pnpm release with bootstrap fix

pnpm 12.9.1 restores a Node launcher before its postinstall replaces the target with ELF; pnpm 11 then runs that binary as JavaScript. Pin 12.10.0, which includes the shim fix, so the docs CI bootstrap can switch versions safely.

* fix(xray): build core from pinned module

The configured Xray release asset tag is unpublished, and the latest published binary lacks GetUsersStats used by the panel. Build the packaged core from the exact Xray module version already selected in go.mod for Docker and release targets.

* fix(tests): stabilize local network fixtures

Keep the TUIC stream fragmentation check on a working UDP loopback path,
allocate the AmneziaWG fixture ports dynamically, and wait for a real
Xray stats response before starting API mutation assertions. Recreate
failed readiness channels so an early connection cannot stall the probe.

Verified host and offline parallel Go suites, repeated occupied-port and
UDP tests, race/shuffle runs, and golangci-lint. Production behavior and
dependency versions are unchanged.

* fix(i18n): add the kernel-mode AmneziaWG keys to sk-SK

main gained the sk-SK locale after this branch added its kernel-mode keys
to the other 13, so i18n-dead-keys.test.ts failed on the merged tree.

* refactor(amneziawgkern): apply kernel interfaces with direct commands, no awg-quick

awg-quick evaluates PostUp/PostDown through a root shell, so every value the
panel rendered into its config was one validation slip away from root command
execution. The interface is now built with direct ip/awg/nft execs from a
setconf-only file: link add, awg setconf, address, mtu, an atomically replaced
nftables table per inbound, and the IPv4 forwarding sysctl written directly.

- Panel-owned names and paths: xui-awg<id> interfaces, xui_awg<id> NAT tables
  and configs under the panel's data dir, so nothing collides with an
  operator's own awg0/awg1 awg-quick tunnels in /etc/amnezia/amneziawg.
- FORWARD accepts (comment 3x-ui:xui-awg<id>) when iptables is present: Docker
  and ufw set the FORWARD policy to DROP, which silently black-holed the tunnel.
- A full panel stop records the final counters and tears the interfaces down,
  like the embedded datapath; a stopped panel enforces no quota or expiry.
  Crash survivors are still adopted without a rebuild.
- The job reads counters before reconciling, and from every managed interface,
  so a client disabled for quota, or a removed inbound, is charged for its last
  interval instead of losing it with the dropped peer.
- An unchanged reconcile tick forks no process (it probed the live key with
  `awg show` every 10s); readiness no longer loads the module from a GET.

Verified on the real module (WSL2 6.6 kernel, v3.1.20260906): create, peer and
egress changes in place, SIGKILL adoption with the same ifindex, SIGTERM
teardown leaving no link, table, FORWARD rule or config behind.

* fix(amneziawg): gate kernel host readiness only when a save starts the datapath

Every save of a kernel-mode inbound re-ran the host preflight, so a host that
lost its module (a kernel upgrade without a DKMS rebuild) refused even
disabling the inbound or switching it back to embedded mode. Host blockers now
apply only when the save starts the kernel datapath: a new inbound, a switch
from embedded, or enabling a disabled one. Feature blockers (IPv6, forwarded
ports, pinned listen) still apply to every save.

* feat(nodes): refuse kernel-mode AmneziaWG pushes to nodes that predate it

A node built before kernel mode drops the unknown kernelMode field and serves
the inbound on the embedded datapath while the master's form shows kernel
mode on. Nodes now advertise awg-kernel in X-3x-Node-Caps, and Remote refuses
to push a kernel-mode inbound to a node that has not advertised it, learning
the caps with an inbound list first when this Remote has not heard from it.

* fix(amneziawg): pin the kernel module installer to verified commits

The installer ran as root and built whatever upstream tag sorted newest at
install time. It now clones fixed tags (module v3.1.20260906, tools
v3.1.20260812) and refuses unless each resolves to its recorded commit. The
udp_tunnel perl patch is gone: those releases carry the kernel 7.1.5 compat
upstream, so the patch never matched.

* fix(i18n): translate the kernel-mode AmneziaWG strings

All thirteen non-English locales carried the English text verbatim.

* fix(amneziawg): skip the kernel traffic flush when no kernel interface runs

Shutdown listed the desired AmneziaWG inbounds from the database even on a
panel with no kernel interface, an unneeded query on every stop. A server
stopped before its database was opened panicked on it, which turned
TestServerStopWhenXrayNotRunningSucceeds red whenever it ran first.

---------

Co-authored-by: MHSanaei <ho3ein.sanaei@gmail.com>
This commit is contained in:
YoungReckless4
2026-10-25 15:24:50 +03:00
committed by GitHub
parent 4d6db99be8
commit 56ac8d0fc7
66 changed files with 5006 additions and 144 deletions
+6
View File
@@ -274,6 +274,9 @@ func (s *ClientService) Create(inboundSvc *InboundService, payload *ClientCreate
// comment on the Attach path.
clientForInbound.AllowedIPs = nil
}
if err := validateAmneziaWGClientWrite(inbound, []model.Client{clientForInbound}); err != nil {
return false, fmt.Errorf("inbound %d: %w", ibId, err)
}
settingsPayload, mErr := json.Marshal(map[string][]model.Client{"clients": {clientWithInboundFlow(clientForInbound, inbound)}})
if mErr != nil {
return false, fmt.Errorf("inbound %d: %w", ibId, mErr)
@@ -824,6 +827,9 @@ func (s *ClientService) Update(inboundSvc *InboundService, id int, updated model
// instead.
clientForInbound.AllowedIPs = nil
}
if err := validateAmneziaWGClientWrite(inbound, []model.Client{clientForInbound}); err != nil {
return false, fmt.Errorf("inbound %d: %w", ibId, err)
}
settingsPayload, mErr := json.Marshal(map[string][]model.Client{"clients": {clientWithInboundFlow(clientForInbound, inbound)}})
if mErr != nil {
return false, mErr
+48 -5
View File
@@ -8,6 +8,8 @@ import (
"strings"
"time"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawg"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawgkern"
"github.com/mhsanaei/3x-ui/v3/internal/database"
"github.com/mhsanaei/3x-ui/v3/internal/database/model"
"github.com/mhsanaei/3x-ui/v3/internal/logger"
@@ -29,6 +31,45 @@ func droppedClientNeedsRestart() bool {
return on
}
func validateAmneziaWGClientWrite(inbound *model.Inbound, clients []model.Client) error {
if inbound.Protocol != model.AmneziaWG {
return nil
}
var settings amneziawg.InboundSettings
if err := json.Unmarshal([]byte(inbound.Settings), &settings); err != nil {
return fmt.Errorf("amneziawg: invalid settings: %w", err)
}
for _, client := range clients {
for _, value := range []struct{ field, value string }{
{"email", client.Email},
{"publicKey", client.PublicKey},
{"preSharedKey", client.PreSharedKey},
} {
if err := amneziawg.ValidateConfigValue(value.field, value.value); err != nil {
return fmt.Errorf("amneziawg: client %q: %w", client.Email, err)
}
}
for _, allowedIP := range client.AllowedIPs {
if err := amneziawg.ValidateConfigValue("allowedIPs", allowedIP); err != nil {
return fmt.Errorf("amneziawg: client %q: %w", client.Email, err)
}
}
}
if settings.Server == nil || !settings.Server.KernelMode {
return nil
}
inst := amneziawg.Instance{Listen: inbound.Listen, IPv6Enabled: settings.Server.IPv6Enabled}
for _, client := range clients {
inst.Peers = append(inst.Peers, amneziawg.Peer{
AllowedIPs: client.AllowedIPs, ForwardedPorts: client.ForwardedPorts,
})
}
if err := amneziawgkern.ValidateInstanceFeatures(inst, settings.Server.RouteThroughXray); err != nil {
return err
}
return nil
}
func sameClientConfigExceptUpdatedAt(a, b map[string]any) bool {
aa := maps.Clone(a)
bb := maps.Clone(b)
@@ -441,6 +482,9 @@ func (s *ClientService) AddInboundClient(inboundSvc *InboundService, data *model
}
}
}
if err := validateAmneziaWGClientWrite(oldInbound, clients); err != nil {
return false, err
}
var portCtx portConflictContext
if oldInbound.Protocol == model.AmneziaWG {
@@ -801,15 +845,14 @@ func (s *ClientService) UpdateInboundClient(inboundSvc *InboundService, data *mo
if clients[0].KeepAlive == nil {
clients[0].KeepAlive = old.KeepAlive
}
// ForwardedPorts is AmneziaWG-only (WireGuard's own inbound never
// reads it), same carry-forward reasoning as the fields above: a
// partial edit (e.g. a Telegram-bot enable/expiry toggle, or an API
// call that omits the field) must not silently drop a client's
// existing port-forwarding spec.
// AWG partial edits preserve forwarding specs; kernel mode rejects legacy specs.
if oldInbound.Protocol == model.AmneziaWG && clients[0].ForwardedPorts == "" {
clients[0].ForwardedPorts = old.ForwardedPorts
}
}
if err := validateAmneziaWGClientWrite(oldInbound, clients); err != nil {
return false, err
}
if oldInbound.Protocol == model.AmneziaWG {
portCtx, err := inboundSvc.loadPortConflictContext(database.GetDB(), oldInbound.NodeID)
if err != nil {
+4 -5
View File
@@ -1312,7 +1312,7 @@ func (s *InboundService) AddInbound(inbound *model.Inbound) (*model.Inbound, boo
if err := s.normalizeMtprotoXrayPort(inbound, ""); err != nil {
return inbound, false, err
}
if err := s.normalizeAmneziaWGSettings(inbound, ""); err != nil {
if err := s.normalizeAmneziaWGSettings(inbound, nil); err != nil {
return inbound, false, err
}
if inbound.NodeID != nil {
@@ -1458,9 +1458,8 @@ func (s *InboundService) AddInbound(inbound *model.Inbound) (*model.Inbound, boo
if err := tx.Omit("ClientStats").Save(inbound).Error; err != nil {
return err
}
// The relay port is derived from the id, only known after Save, and only a
// local row owns one: checkPortConflictTx ran no relay check with ignoreId==0.
if inbound.NodeID == nil && inbound.Protocol == model.AmneziaWG {
// Embedded AmneziaWG owns a relay port derived from its local row id.
if inbound.NodeID == nil && inbound.Protocol == model.AmneziaWG && !amneziawgKernelMode(inbound) {
if self := amneziawgnetSocksSelfConflict(inbound, inbound.Id); self != "" {
return common.NewError(self)
}
@@ -2003,7 +2002,7 @@ func (s *InboundService) UpdateInbound(inbound *model.Inbound) (*model.Inbound,
if err := normalizeTuicSettings(inbound); err != nil {
return inbound, false, err
}
if err := s.normalizeAmneziaWGSettings(inbound, oldInbound.Settings); err != nil {
if err := s.normalizeAmneziaWGSettings(inbound, oldInbound); err != nil {
return inbound, false, err
}
inbound.SubSortIndex = normalizeSubSortIndex(inbound.SubSortIndex)
+38 -1
View File
@@ -9,6 +9,7 @@ import (
"gorm.io/gorm"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawg"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawgkern"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawgnet"
"github.com/mhsanaei/3x-ui/v3/internal/database"
"github.com/mhsanaei/3x-ui/v3/internal/database/model"
@@ -188,10 +189,14 @@ func storedAmneziaWGServerKeys(oldSettings string) (priv, pub string) {
// valid server block, generating one (fresh obfuscation params + keypair) on
// first save and validating a manually-edited one so a bad entry can't bring
// the interface down on the next apply. A no-op for every other protocol.
func (s *InboundService) normalizeAmneziaWGSettings(inbound *model.Inbound, oldSettings string) error {
func (s *InboundService) normalizeAmneziaWGSettings(inbound *model.Inbound, old *model.Inbound) error {
if inbound.Protocol != model.AmneziaWG {
return nil
}
oldSettings := ""
if old != nil {
oldSettings = old.Settings
}
trimmed := strings.TrimSpace(inbound.Settings)
if trimmed == "" || trimmed == "null" || trimmed == "{}" {
@@ -301,6 +306,35 @@ func (s *InboundService) normalizeAmneziaWGSettings(inbound *model.Inbound, oldS
return err
}
inbound.Settings = string(bs)
// Host readiness gates only a save that starts the kernel datapath: a host that
// later loses its module must still let the admin edit, disable or switch it.
if parsed.Server.KernelMode && inbound.NodeID == nil {
inst, ok := amneziawg.InstanceFromInbound(inbound)
if !ok {
inst = amneziawg.Instance{
Listen: inbound.Listen,
IPv6Enabled: parsed.Server.IPv6Enabled,
RouteThroughXray: parsed.Server.RouteThroughXray,
}
for _, client := range parsed.Clients {
if client.Enable {
inst.Peers = append(inst.Peers, amneziawg.Peer{
AllowedIPs: client.AllowedIPs,
ForwardedPorts: client.ForwardedPorts,
})
}
}
}
check := amneziawgkern.ValidateInstanceFeatures
wasRunningKernel := old != nil && old.Enable && amneziawgKernelModeSettings(old.Settings)
if inbound.Enable && !wasRunningKernel {
check = amneziawgkern.Preflight
}
if err := check(inst, inst.RouteThroughXray); err != nil {
return err
}
}
return nil
}
@@ -387,6 +421,9 @@ func (s *InboundService) checkForwardedPortsConflict(ctx portConflictContext, fo
var socksPort int
switch ib.Protocol {
case model.AmneziaWG:
if amneziawgKernelMode(ib) {
continue
}
socksPort = amneziawgnet.SOCKSPortForInbound(ib.Id)
case model.TUIC:
socksPort = tuic.SOCKSPortForInbound(ib.Id)
+107 -9
View File
@@ -10,6 +10,7 @@ import (
"github.com/op/go-logging"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawg"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawgkern"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawgnet"
"github.com/mhsanaei/3x-ui/v3/internal/database"
"github.com/mhsanaei/3x-ui/v3/internal/database/model"
@@ -144,7 +145,7 @@ func TestNormalizeAmneziaWGSettings_GeneratesFull31Set(t *testing.T) {
setupConflictDB(t)
svc := &InboundService{}
inbound := &model.Inbound{Protocol: model.AmneziaWG, Port: 51820, Settings: ""}
if err := svc.normalizeAmneziaWGSettings(inbound, ""); err != nil {
if err := svc.normalizeAmneziaWGSettings(inbound, nil); err != nil {
t.Fatalf("normalize empty settings: %v", err)
}
@@ -199,7 +200,7 @@ func TestNormalizeAmneziaWGSettings_RejectsBad31Values(t *testing.T) {
Port: 51820,
Settings: `{"server":{"privateKey":"x","publicKey":"y","subnetIp":"10.8.1.0","subnetCidr":24,` + c.snippet + `},"clients":[]}`,
}
if err := svc.normalizeAmneziaWGSettings(inbound, ""); err == nil {
if err := svc.normalizeAmneziaWGSettings(inbound, nil); err == nil {
t.Errorf("%s must be rejected", c.name)
}
}
@@ -214,7 +215,7 @@ func TestNormalizeAmneziaWGSettings_CanonicalizesRangeValues(t *testing.T) {
Settings: `{"server":{"privateKey":"x","publicKey":"y","subnetIp":"10.8.1.0","subnetCidr":24,` +
`"rekeyAfterTime":"110 - 140","rejectAfterTime":"190-250","keepaliveTimeout":" "},"clients":[]}`,
}
if err := svc.normalizeAmneziaWGSettings(inbound, ""); err != nil {
if err := svc.normalizeAmneziaWGSettings(inbound, nil); err != nil {
t.Fatalf("normalize: %v", err)
}
var parsed amneziawg.InboundSettings
@@ -256,7 +257,7 @@ func TestNormalizeAmneziaWGSettings_RejectsInjectedClientAllowedIPs(t *testing.T
`"clients":[{"email":"a@x","enable":true,"publicKey":"pk",` +
`"allowedIPs":["10.8.1.2/32\n[Interface]\nPostUp = touch /tmp/pwned"]}]}`,
}
err := svc.normalizeAmneziaWGSettings(inbound, "")
err := svc.normalizeAmneziaWGSettings(inbound, nil)
if err == nil {
t.Fatalf("an allowedIPs entry carrying a config-injection payload must be rejected; settings became:\n%s", inbound.Settings)
}
@@ -298,7 +299,7 @@ func TestNormalizeAmneziaWGSettings_CanonicalizesClientAllowedIPs(t *testing.T)
Settings: `{"server":{"privateKey":"x","publicKey":"y","subnetIp":"10.8.1.0","subnetCidr":24},` +
`"clients":[{"email":"a@x","enable":true,"publicKey":"pk","allowedIPs":[" 10.8.1.2 "]}]}`,
}
if err := svc.normalizeAmneziaWGSettings(inbound, ""); err != nil {
if err := svc.normalizeAmneziaWGSettings(inbound, nil); err != nil {
t.Fatalf("normalize: %v", err)
}
var parsed amneziawg.InboundSettings
@@ -398,6 +399,26 @@ func TestCheckForwardedPortsConflict_CollidesWithAmneziawgnetSocksPort(t *testin
}
}
func TestCheckForwardedPortsConflict_KernelAmneziaWGHasNoEmbeddedRelay(t *testing.T) {
setupConflictDB(t)
seedInboundConflict(t, "awg-kernel", "0.0.0.0", 51820, model.AmneziaWG, ``,
`{"server":{"privateKey":"priv","publicKey":"pub","subnetIp":"10.8.1.0","subnetCidr":24,"kernelMode":true},"clients":[]}`)
var awgInbound model.Inbound
if err := database.GetDB().Where("tag = ?", "awg-kernel").First(&awgInbound).Error; err != nil {
t.Fatalf("read seeded row: %v", err)
}
svc := &InboundService{}
ctx, err := svc.loadPortConflictContext(database.GetDB(), nil)
if err != nil {
t.Fatalf("loadPortConflictContext: %v", err)
}
relayPort := amneziawgnet.SOCKSPortForInbound(awgInbound.Id)
if hit := svc.checkForwardedPortsConflict(ctx, fmt.Sprintf("%d", relayPort)); hit != "" {
t.Fatalf("kernel-mode AmneziaWG must not reserve embedded relay port %d, got collision %q", relayPort, hit)
}
}
// A cleared DNS field is meaningful (no DNS line in client configs) and must
// survive the save round-trip instead of resurrecting the frontend defaults.
func TestNormalizeAmneziaWGSettingsKeepsClearedDNS(t *testing.T) {
@@ -413,7 +434,7 @@ func TestNormalizeAmneziaWGSettingsKeepsClearedDNS(t *testing.T) {
t.Fatalf("marshal settings: %v", err)
}
inbound := &model.Inbound{Protocol: model.AmneziaWG, Settings: string(bs)}
if err := (&InboundService{}).normalizeAmneziaWGSettings(inbound, ""); err != nil {
if err := (&InboundService{}).normalizeAmneziaWGSettings(inbound, nil); err != nil {
t.Fatalf("normalizeAmneziaWGSettings: %v", err)
}
for _, key := range []string{`"primaryDns"`, `"secondaryDns"`} {
@@ -432,7 +453,7 @@ func TestNormalizeAmneziaWGSettings_RejectsEmptyClientAllowedIPs(t *testing.T) {
"server": {"privateKey":"` + awgTestPrivateKey + `","publicKey":"` + awgTestPublicKey + `","subnetIp":"10.8.1.0","subnetCidr":24},
"clients": [{"email":"ghost","enable":true,"publicKey":"` + awgTestPublicKey + `","allowedIPs":[]}]
}`}
err := svc.normalizeAmneziaWGSettings(inbound, "")
err := svc.normalizeAmneziaWGSettings(inbound, nil)
if err == nil || !strings.Contains(err.Error(), "allowedIPs is required") {
t.Fatalf("err = %v, want an allowedIPs refusal naming the client", err)
}
@@ -449,7 +470,7 @@ func TestNormalizeAmneziaWGSettings_KeepsStoredServerKeysWhenOmitted(t *testing.
stored := `{"server":{"privateKey":"` + awgTestPrivateKey + `","publicKey":"` + awgTestPublicKey + `","subnetIp":"10.8.1.0","subnetCidr":24}}`
inbound := &model.Inbound{Protocol: model.AmneziaWG, Port: 51824, Settings: `{"server":{"subnetIp":"10.8.1.0","subnetCidr":24,"randomTrailers":true}}`}
if err := svc.normalizeAmneziaWGSettings(inbound, stored); err != nil {
if err := svc.normalizeAmneziaWGSettings(inbound, &model.Inbound{Settings: stored}); err != nil {
t.Fatalf("normalize: %v", err)
}
var parsed amneziawg.InboundSettings
@@ -467,7 +488,7 @@ func TestNormalizeAmneziaWGSettings_DerivesServerPublicKeyFromPrivate(t *testing
setupConflictDB(t)
svc := &InboundService{}
inbound := &model.Inbound{Protocol: model.AmneziaWG, Port: 51825, Settings: `{"server":{"privateKey":"` + awgTestPrivateKey + `","subnetIp":"10.8.1.0","subnetCidr":24}}`}
if err := svc.normalizeAmneziaWGSettings(inbound, ""); err != nil {
if err := svc.normalizeAmneziaWGSettings(inbound, nil); err != nil {
t.Fatalf("normalize: %v", err)
}
var parsed amneziawg.InboundSettings
@@ -482,3 +503,80 @@ func TestNormalizeAmneziaWGSettings_DerivesServerPublicKeyFromPrivate(t *testing
t.Fatalf("server publicKey = %q, want %q derived from the supplied private key", parsed.Server.PublicKey, want)
}
}
func TestNormalizeAmneziaWGSettingsDefersNodeKernelPreflightToRemoteHost(t *testing.T) {
setupConflictDB(t)
server, err := defaultAmneziaWGServer()
if err != nil {
t.Fatalf("defaultAmneziaWGServer: %v", err)
}
server.KernelMode = true
settings, err := json.Marshal(amneziawg.InboundSettings{Server: server, Clients: []model.Client{}})
if err != nil {
t.Fatalf("marshal settings: %v", err)
}
inbound := &model.Inbound{
Protocol: model.AmneziaWG,
NodeID: new(7),
Port: 51820,
Settings: string(settings),
}
if err := (&InboundService{}).normalizeAmneziaWGSettings(inbound, nil); err != nil {
t.Fatalf("master must defer kernel readiness to the target node: %v", err)
}
var normalized amneziawg.InboundSettings
if err := json.Unmarshal([]byte(inbound.Settings), &normalized); err != nil {
t.Fatalf("decode normalized settings: %v", err)
}
if normalized.Server == nil || !normalized.Server.KernelMode {
t.Fatalf("normalized node settings lost kernel mode: %+v", normalized.Server)
}
}
// TestNormalizeAmneziaWGSettingsGatesHostReadinessOnlyWhenEnablingKernelMode:
// a host that loses its module after enabling must not lock the inbound's edits.
func TestNormalizeAmneziaWGSettingsGatesHostReadinessOnlyWhenEnablingKernelMode(t *testing.T) {
setupConflictDB(t)
if amneziawgkern.HostReadiness().Ready {
t.Skip("this host can run kernel mode, so host readiness cannot refuse anything here")
}
server, err := defaultAmneziaWGServer()
if err != nil {
t.Fatalf("defaultAmneziaWGServer: %v", err)
}
embeddedServer := *server
server.KernelMode = true
kernelSettings, err := json.Marshal(amneziawg.InboundSettings{Server: server, Clients: []model.Client{}})
if err != nil {
t.Fatal(err)
}
embeddedSettings, err := json.Marshal(amneziawg.InboundSettings{Server: &embeddedServer, Clients: []model.Client{}})
if err != nil {
t.Fatal(err)
}
kernelRow := func(enable bool) *model.Inbound {
return &model.Inbound{Protocol: model.AmneziaWG, Enable: enable, Settings: string(kernelSettings)}
}
cases := []struct {
name string
enable bool
old *model.Inbound
wantRefused bool
}{
{"new inbound turns kernel mode on", true, nil, true},
{"embedded inbound switches to kernel mode", true, &model.Inbound{Protocol: model.AmneziaWG, Enable: true, Settings: string(embeddedSettings)}, true},
{"disabled kernel inbound is enabled", true, kernelRow(false), true},
{"running kernel inbound is edited", true, kernelRow(true), false},
{"running kernel inbound is disabled", false, kernelRow(true), false},
{"disabled inbound saved with kernel mode", false, nil, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
inbound := &model.Inbound{Protocol: model.AmneziaWG, Port: 51820, Enable: tc.enable, Settings: string(kernelSettings)}
err := (&InboundService{}).normalizeAmneziaWGSettings(inbound, tc.old)
if refused := err != nil && strings.Contains(err.Error(), "kernel mode unavailable"); refused != tc.wantRefused {
t.Fatalf("refused = %v (err %v), want %v", refused, err, tc.wantRefused)
}
})
}
}
+21 -2
View File
@@ -9,6 +9,7 @@ import (
"sync"
"time"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawgkern"
"github.com/mhsanaei/3x-ui/v3/internal/database"
"github.com/mhsanaei/3x-ui/v3/internal/database/model"
"github.com/mhsanaei/3x-ui/v3/internal/logger"
@@ -1537,11 +1538,29 @@ func (s *InboundService) restartRemoteNodesOnDisable(nodeIDs []int) {
}
func (s *InboundService) GetOnlineClients() []string {
// Kernel-mode AmneziaWG peers are invisible to Xray's online-users set;
// their handshakes are the equivalent signal, unioned in once here.
kernelOnline := amneziawgkern.GetCollector().OnlineEmails(time.Now())
process := currentXrayProcess()
if process == nil {
return []string{}
return kernelOnline
}
return process.GetOnlineClients()
xrayOnline := process.GetOnlineClients()
if len(kernelOnline) == 0 {
return xrayOnline
}
seen := make(map[string]struct{}, len(xrayOnline)+len(kernelOnline))
out := make([]string, 0, len(xrayOnline)+len(kernelOnline))
for _, email := range append(append([]string{}, xrayOnline...), kernelOnline...) {
if _, dup := seen[email]; dup {
continue
}
seen[email] = struct{}{}
out = append(out, email)
}
return out
}
// GetOnlineClientsByGuid returns online emails keyed by the panelGuid of the
@@ -0,0 +1,199 @@
package service
import (
"encoding/json"
"strings"
"testing"
"github.com/mhsanaei/3x-ui/v3/internal/database"
"github.com/mhsanaei/3x-ui/v3/internal/database/model"
)
func kernelAWGSettings(clients []model.Client) string {
b, _ := json.Marshal(map[string]any{
"server": map[string]any{"subnetIp": "10.8.1.0", "subnetCidr": 24, "kernelMode": true},
"clients": clients,
})
return string(b)
}
func TestKernelAWGClientWritesRejectUnsafeValuesAndUnsupportedFeatures(t *testing.T) {
cases := []struct {
name, want string
client model.Client
}{
{"public key hook injection", "control characters", model.Client{Email: "bad-key@awg", Enable: true, PublicKey: "peer-key\n[Interface]\nPostDown = touch /tmp/pwned", AllowedIPs: []string{"10.8.1.2/32"}}},
{"forwarded ports", "port forwarding is not supported", model.Client{Email: "bad-forward@awg", Enable: true, PublicKey: "peer-key", AllowedIPs: []string{"10.8.1.2/32"}, ForwardedPorts: "54321"}},
{"IPv6 allowed IP", "per-peer IPv6 egress is not supported", model.Client{Email: "bad-v6@awg", Enable: true, PublicKey: "peer-key", AllowedIPs: []string{"2001:db8::2/128"}}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
setupBulkDB(t)
inboundSvc := &InboundService{}
svc := &ClientService{}
ib := mkInbound(t, 52000, model.AmneziaWG, kernelAWGSettings(nil))
tc.client.AllowedIPsByInbound = map[int][]string{ib.Id: tc.client.AllowedIPs}
_, err := svc.CreateOne(inboundSvc, ib.Id, tc.client)
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatalf("CreateOne error = %v, want an error containing %q", err, tc.want)
}
var records int64
if dbErr := database.GetDB().Model(&model.ClientRecord{}).Where("email = ?", tc.client.Email).Count(&records).Error; dbErr != nil {
t.Fatal(dbErr)
}
if records != 0 {
t.Fatalf("rejected create persisted %d client records", records)
}
after, getErr := inboundSvc.GetInbound(ib.Id)
if getErr != nil {
t.Fatal(getErr)
}
if len(mustAWGClients(t, after)) != 0 {
t.Fatal("rejected create changed inbound settings")
}
})
}
}
func mustAWGClients(t *testing.T, ib *model.Inbound) []model.Client {
t.Helper()
clients, err := (&InboundService{}).GetClients(ib)
if err != nil {
t.Fatal(err)
}
return clients
}
func TestKernelAWGClientUpdateRejectsUnsafeValuesAndUnsupportedFeatures(t *testing.T) {
cases := []struct {
name, want string
update model.Client
}{
{"preshared key hook injection", "control characters", model.Client{Email: "kept@awg", PreSharedKey: "psk\n[Interface]\nPreDown = touch /tmp/pwned"}},
{"forwarded ports", "port forwarding is not supported", model.Client{Email: "kept@awg", ForwardedPorts: "54321"}},
{"IPv6 allowed IP", "per-peer IPv6 egress is not supported", model.Client{Email: "kept@awg", AllowedIPs: []string{"2001:db8::2/128"}}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
setupBulkDB(t)
inboundSvc := &InboundService{}
svc := &ClientService{}
seed := model.Client{Email: "kept@awg", Enable: true, PublicKey: "valid-key", PrivateKey: "valid-private", AllowedIPs: []string{"10.8.1.2/32"}}
ib := mkInbound(t, 52001, model.AmneziaWG, kernelAWGSettings(nil))
if _, err := svc.CreateOne(inboundSvc, ib.Id, seed); err != nil {
t.Fatalf("seed client: %v", err)
}
rec := lookupClientRecord(t, seed.Email)
before, getErr := inboundSvc.GetInbound(ib.Id)
if getErr != nil {
t.Fatal(getErr)
}
update := tc.update
if len(update.AllowedIPs) > 0 {
update.AllowedIPsByInbound = map[int][]string{ib.Id: update.AllowedIPs}
}
_, err := svc.Update(inboundSvc, rec.Id, update, 0)
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatalf("ClientService.Update error = %v, want an error containing %q", err, tc.want)
}
after, err := inboundSvc.GetInbound(ib.Id)
if err != nil {
t.Fatal(err)
}
if after.Settings != before.Settings {
t.Fatalf("rejected update changed settings")
}
var unchanged model.ClientRecord
if err := database.GetDB().First(&unchanged, rec.Id).Error; err != nil {
t.Fatal(err)
}
if unchanged.PreSharedKey != rec.PreSharedKey || unchanged.AllowedIPs != rec.AllowedIPs {
t.Fatalf("rejected update changed client record: before %+v after %+v", rec, unchanged)
}
})
}
}
func TestKernelAWGClientFeatureChecksPreserveSupportedPaths(t *testing.T) {
setupBulkDB(t)
inboundSvc := &InboundService{}
svc := &ClientService{}
kernel := mkInbound(t, 52002, model.AmneziaWG, kernelAWGSettings(nil))
if _, err := svc.CreateOne(inboundSvc, kernel.Id, model.Client{Email: "valid@kernel", Enable: true, PublicKey: "valid-key", AllowedIPs: []string{"10.8.1.2/32"}}); err != nil {
t.Fatalf("valid IPv4 kernel peer was rejected: %v", err)
}
embedded := mkInbound(t, 52003, model.AmneziaWG, amneziawgClientTestSettings)
if _, err := svc.CreateOne(inboundSvc, embedded.Id, model.Client{Email: "forward@embedded", Enable: true, PublicKey: "valid-key", AllowedIPs: []string{"10.8.1.3/32"}, ForwardedPorts: "54321"}); err != nil {
t.Fatalf("embedded AWG port forwarding was rejected: %v", err)
}
}
func TestKernelAWGSharedCreateAndUpdateValidateAllTargetsBeforeWrites(t *testing.T) {
t.Run("create", func(t *testing.T) {
setupBulkDB(t)
inboundSvc := &InboundService{}
svc := &ClientService{}
kernel := mkInbound(t, 52005, model.AmneziaWG, kernelAWGSettings(nil))
embedded := mkInbound(t, 52006, model.AmneziaWG, amneziawgClientTestSettings)
_, err := svc.Create(inboundSvc, &ClientCreatePayload{
Client: model.Client{Email: "shared-create@awg", Enable: true, PublicKey: "valid-key", AllowedIPs: []string{"10.8.1.2/32"}, ForwardedPorts: "54321"},
InboundIds: []int{embedded.Id, kernel.Id},
})
if err == nil || !strings.Contains(err.Error(), "port forwarding is not supported") {
t.Fatalf("Create error = %v, want kernel feature rejection", err)
}
for _, ib := range []*model.Inbound{embedded, kernel} {
after, getErr := inboundSvc.GetInbound(ib.Id)
if getErr != nil {
t.Fatal(getErr)
}
if len(mustAWGClients(t, after)) != 0 {
t.Fatalf("failed shared create partially changed inbound %d", ib.Id)
}
}
var records int64
if err := database.GetDB().Model(&model.ClientRecord{}).Where("email = ?", "shared-create@awg").Count(&records).Error; err != nil {
t.Fatal(err)
}
if records != 0 {
t.Fatalf("failed shared create persisted %d client records", records)
}
})
t.Run("update", func(t *testing.T) {
setupBulkDB(t)
inboundSvc := &InboundService{}
svc := &ClientService{}
kernel := mkInbound(t, 52007, model.AmneziaWG, kernelAWGSettings(nil))
embedded := mkInbound(t, 52008, model.AmneziaWG, amneziawgClientTestSettings)
seed := model.Client{Email: "shared-update@awg", Enable: true, PublicKey: "valid-key", AllowedIPsByInbound: map[int][]string{kernel.Id: {"10.8.1.2/32"}, embedded.Id: {"10.8.1.3/32"}}}
if _, err := svc.Create(inboundSvc, &ClientCreatePayload{Client: seed, InboundIds: []int{kernel.Id, embedded.Id}}); err != nil {
t.Fatalf("seed shared client: %v", err)
}
beforeKernel, _ := inboundSvc.GetInbound(kernel.Id)
beforeEmbedded, _ := inboundSvc.GetInbound(embedded.Id)
rec := lookupClientRecord(t, seed.Email)
_, err := svc.Update(inboundSvc, rec.Id, model.Client{Email: seed.Email, ForwardedPorts: "54321"}, 0)
if err == nil || !strings.Contains(err.Error(), "port forwarding is not supported") {
t.Fatalf("Update error = %v, want kernel feature rejection", err)
}
afterKernel, getErr := inboundSvc.GetInbound(kernel.Id)
if getErr != nil {
t.Fatal(getErr)
}
afterEmbedded, getErr := inboundSvc.GetInbound(embedded.Id)
if getErr != nil {
t.Fatal(getErr)
}
if afterKernel.Settings != beforeKernel.Settings || afterEmbedded.Settings != beforeEmbedded.Settings {
t.Fatal("failed shared update partially changed inbound settings")
}
var unchanged model.ClientRecord
if err := database.GetDB().First(&unchanged, rec.Id).Error; err != nil {
t.Fatal(err)
}
if unchanged.ForwardedPorts != rec.ForwardedPorts {
t.Fatal("failed shared update changed the client record")
}
})
}
+34 -16
View File
@@ -129,6 +129,21 @@ func inboundBindAddr(ib *model.Inbound) bindAddr {
return bindAddr{listen: ib.Listen, v6only: streamV6Only(ib.StreamSettings)}
}
func amneziawgKernelMode(ib *model.Inbound) bool {
if ib == nil || ib.Protocol != model.AmneziaWG {
return false
}
return amneziawgKernelModeSettings(ib.Settings)
}
func amneziawgKernelModeSettings(raw string) bool {
var settings amneziawg.InboundSettings
if err := json.Unmarshal([]byte(raw), &settings); err != nil || settings.Server == nil {
return false
}
return settings.Server.KernelMode
}
func streamV6Only(streamSettings string) bool {
if !strings.Contains(streamSettings, "v6only") {
return false
@@ -292,14 +307,7 @@ func checkPortConflictTx(db *gorm.DB, inbound *model.Inbound, ignoreId int) (*po
}, nil
}
// Every enabled local AmneziaWG inbound gets its own automatic Xray
// SOCKS5 relay inbound (see injectAmneziawgnetSocks) on 127.0.0.1 at a
// port derived purely from its id (amneziawgnet.SOCKSPortForInbound) --
// like the internal Xray API inbound above, that relay inbound is not
// itself a database row, so the ordinary DB-backed query below can never
// see it. Without this check, an unrelated inbound saved onto that exact
// port silently fails at the next Xray start, taking every other
// protocol down with it, not just AmneziaWG.
// Local embedded inbounds reserve an automatically assigned Xray SOCKS relay port.
if inbound.NodeID == nil && listenOverlaps(loopbackBind, inboundBindAddr(inbound)) {
conflict, err := checkAmneziawgnetSocksConflict(db, inbound, ignoreId, newBits)
if err != nil {
@@ -319,7 +327,7 @@ func checkPortConflictTx(db *gorm.DB, inbound *model.Inbound, ignoreId int) (*po
// The reverse direction, only meaningful once the id is known -- AddInbound
// runs it after Save. Only a local row owns a relay slot (#6537 review).
if inbound.NodeID == nil && inbound.Protocol == model.AmneziaWG && ignoreId > 0 {
if inbound.NodeID == nil && inbound.Protocol == model.AmneziaWG && ignoreId > 0 && !amneziawgKernelMode(inbound) {
if self := amneziawgnetSocksSelfConflict(inbound, ignoreId); self != "" {
return nil, common.NewError(self)
}
@@ -417,6 +425,9 @@ func amneziawgForwardedPortOwner(db *gorm.DB, inbound *model.Inbound, ignoreId i
if !sameNode(row.NodeID, inbound.NodeID) {
continue
}
if amneziawgKernelMode(row) {
continue
}
instance, ok := amneziawg.InstanceFromInbound(row)
if !ok {
continue
@@ -439,8 +450,7 @@ func amneziawgForwardedPortOwner(db *gorm.DB, inbound *model.Inbound, ignoreId i
return nil, nil
}
// checkAmneziawgnetSocksConflict: inbound's port vs the relay port every matching
// local row reserves, emitted or not; db keeps it in the caller's transaction (#6225).
// checkAmneziawgnetSocksConflict checks the embedded relay port each local row reserves.
func checkAmneziawgnetSocksConflict(db *gorm.DB, inbound *model.Inbound, ignoreId int, newBits transportBits) (*portConflictDetail, error) {
// A disabled row still owns the slot its id derives: SetInboundEnable flips
// the column with no port check, so enabling it later must not collide.
@@ -452,9 +462,11 @@ func checkAmneziawgnetSocksConflict(db *gorm.DB, inbound *model.Inbound, ignoreI
if err := q.Find(&candidates).Error; err != nil {
return nil, err
}
// Ownership does not depend on the peers: the relay appears when the first
// client is added, and the client paths run no port check at all.
// The row reserves its slot before its first peer is added.
for _, c := range candidates {
if amneziawgKernelMode(c) {
continue
}
if amneziawgnet.SOCKSPortForInbound(c.Id) != inbound.Port {
continue
}
@@ -470,8 +482,8 @@ func checkAmneziawgnetSocksConflict(db *gorm.DB, inbound *model.Inbound, ignoreI
return nil, nil
}
// checkAmneziawgnetSocksRelayCollision reports whether id's derived relay port
// is already claimed by another local AmneziaWG inbound, disabled rows included.
// checkAmneziawgnetSocksRelayCollision reports whether id's embedded relay port
// is already claimed by another local AmneziaWG inbound, including disabled rows.
func checkAmneziawgnetSocksRelayCollision(db *gorm.DB, id int) (*portConflictDetail, error) {
relayPort := amneziawgnet.SOCKSPortForInbound(id)
var candidates []*model.Inbound
@@ -481,6 +493,9 @@ func checkAmneziawgnetSocksRelayCollision(db *gorm.DB, id int) (*portConflictDet
return nil, err
}
for _, c := range candidates {
if amneziawgKernelMode(c) {
continue
}
if amneziawgnet.SOCKSPortForInbound(c.Id) != relayPort {
continue
}
@@ -500,7 +515,7 @@ func checkAmneziawgnetSocksRelayCollision(db *gorm.DB, id int) (*portConflictDet
// amneziawgnetSocksSelfConflict: a row's own WireGuard port vs the relay port its
// own id derives -- all three checks below exclude that id, so nothing else does.
func amneziawgnetSocksSelfConflict(inbound *model.Inbound, id int) string {
if id <= 0 || inbound.NodeID != nil || !listenOverlaps(loopbackBind, inboundBindAddr(inbound)) {
if id <= 0 || inbound.NodeID != nil || amneziawgKernelMode(inbound) || !listenOverlaps(loopbackBind, inboundBindAddr(inbound)) {
return ""
}
relayPort := amneziawgnet.SOCKSPortForInbound(id)
@@ -522,6 +537,9 @@ func checkAmneziawgnetSocksReverseConflict(db *gorm.DB, id int) (*portConflictDe
return nil, err
}
for _, c := range candidates {
if amneziawgKernelMode(c) {
continue
}
if !listenOverlaps(loopbackBind, inboundBindAddr(c)) {
continue
}
@@ -816,6 +816,41 @@ func TestCheckPortConflict_AmneziawgnetSocksRelayBlockedLocal(t *testing.T) {
}
}
func TestCheckPortConflict_KernelAmneziaWGDoesNotReserveEmbeddedRelay(t *testing.T) {
setupConflictDB(t)
seedInboundConflict(t, "awg-kernel", "0.0.0.0", 51820, model.AmneziaWG, ``,
`{"server":{"privateKey":"priv","publicKey":"pub","subnetIp":"10.8.1.0","subnetCidr":24,"kernelMode":true},"clients":[]}`)
var awgInbound model.Inbound
if err := database.GetDB().Where("tag = ?", "awg-kernel").First(&awgInbound).Error; err != nil {
t.Fatalf("read seeded row: %v", err)
}
candidate := &model.Inbound{
Tag: "vless-kernel-relay-port",
Listen: "0.0.0.0",
Port: amneziawgnet.SOCKSPortForInbound(awgInbound.Id),
Protocol: model.VLESS,
}
if got, err := (&InboundService{}).checkPortConflict(candidate, 0); err != nil || got != nil {
t.Fatalf("kernel-mode AmneziaWG must not reserve an embedded relay port, including with no peers; got=%v err=%v", got, err)
}
}
func TestCheckPortConflict_KernelAmneziaWGAllowsItsUDPPortToMatchRelayRange(t *testing.T) {
setupConflictDB(t)
seedInboundConflict(t, "awg-kernel", "0.0.0.0", 51820, model.AmneziaWG, ``,
`{"server":{"privateKey":"priv","publicKey":"pub","subnetIp":"10.8.1.0","subnetCidr":24,"kernelMode":true},"clients":[]}`)
var awgInbound model.Inbound
if err := database.GetDB().Where("tag = ?", "awg-kernel").First(&awgInbound).Error; err != nil {
t.Fatalf("read seeded row: %v", err)
}
awgInbound.Port = amneziawgnet.SOCKSPortForInbound(awgInbound.Id)
if got, err := (&InboundService{}).checkPortConflict(&awgInbound, awgInbound.Id); err != nil || got != nil {
t.Fatalf("kernel-mode AmneziaWG UDP port may use a port in the embedded TCP relay range; got=%v err=%v", got, err)
}
}
// Nodes run their own Xray, so a node inbound landing on the central panel's
// AmneziaWG relay port must be allowed -- the relay inbound only ever binds
// 127.0.0.1 on the local panel's own Xray.
+2 -1
View File
@@ -33,6 +33,7 @@ import (
"time"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawg"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawgkern"
"github.com/mhsanaei/3x-ui/v3/internal/amneziawgnet"
"github.com/mhsanaei/3x-ui/v3/internal/config"
"github.com/mhsanaei/3x-ui/v3/internal/database"
@@ -723,7 +724,7 @@ func (s *ServerService) GetStatus(lastStatus *Status) *Status {
logger.Warning("count amneziawg inbounds failed:", err)
}
status.AmneziaWG.Configured = amneziawgCount > 0
status.AmneziaWG.Running = amneziawgnet.GetManager().HasRunning()
status.AmneziaWG.Running = amneziawgnet.GetManager().HasRunning() || amneziawgkern.GetManager().HasRunning()
status.PanelVersion = config.GetPanelVersion()
if guid, err := s.settingService.GetPanelGuid(); err == nil {
+5
View File
@@ -790,6 +790,11 @@ func injectAmneziawgnetSocks(cfg *xray.Config, inbounds []*model.Inbound) {
if !ok {
continue
}
// Kernel-mode inbounds forward in the kernel and never dial Xray, so a
// relay inbound for them would listen forever with nothing behind it.
if inst.KernelMode {
continue
}
if _, taken := existingTags[inbound.Tag]; taken {
logger.Warning("amneziawgnet socks: inbound tag [", inbound.Tag, "] already present in generated config, skipping its relay inbound")
continue