mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
fix(guard): operator-scoped XFF hop peers + tailnet allowlist (live-incident fix) (#650)
Two coupled hardenings from the chain-peers adversarial rounds plus the
root-cause fix for the live post-deploy incident where the CEO was
blocked from the panel ('IP not allowed: 100.x.x.x').
Hop peel-set: the whole docker bridge pool leaves the XFF hop set — hops
are now loopback plus operator-named single addresses only
(ROBOCO_GUARD_TRUSTED_CHAIN_PEERS, plain IPs; CIDR entries rejected with
a warning because a range readmits sibling containers). Default-empty
closes the CGNAT-forge residual outright; a gateway-fronted Tailscale
Serve deploy sets its real gateway IP, and a rate-limited detection log
names exactly that IP when an unconfigured host-proxied tailnet chain is
seen, so the silent-regression shape is observable. The connecting-peer
gate (may nginx present XFF at all) deliberately keeps the broad bridge
pool — different check, unchanged.
Incident root cause: guard-core's whitelist is an EXCLUSIVE allowlist
(any non-member is refused), so honestly resolving the tailnet client IP
made ip_security reject the CEO. The tailnet CGNAT range joins
_guard_whitelist() deliberately: Tailscale authenticates device
membership before a packet arrives, real-IP stamping still buys correct
attribution, and any future non-tailnet exposure keeps full scrutiny.
Both compose files now pass ROBOCO_GUARD_EMERGENCY_WHITELIST through to
the orchestrator (the operator escape hatch previously did nothing in a
compose deploy).
NAS is running ROBOCO_GUARD_PASSIVE_MODE=true as interim mitigation —
flip back to false when this deploys. 66 tests.
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
@@ -249,6 +249,11 @@ ROBOCO_DB_NETWORK_ISOLATED=true
|
|||||||
# ROBOCO_GUARD_PASSIVE_MODE=true
|
# ROBOCO_GUARD_PASSIVE_MODE=true
|
||||||
# Fail CLOSED on an internal guard error (true) vs fail open (false).
|
# Fail CLOSED on an internal guard error (true) vs fail open (false).
|
||||||
# ROBOCO_GUARD_FAIL_SECURE=true
|
# ROBOCO_GUARD_FAIL_SECURE=true
|
||||||
|
# Exact IP address(es), never a range, trusted as an XFF proxy hop beyond loopback
|
||||||
|
# (a CIDR entry is rejected — it would readmit every sibling container). Empty =
|
||||||
|
# loopback-only hop-peel. If Tailscale Serve sits behind this host's docker gateway,
|
||||||
|
# set it to that gateway's exact address to keep the chain resolving (e.g. 172.18.0.1).
|
||||||
|
# ROBOCO_GUARD_TRUSTED_CHAIN_PEERS=172.18.0.1
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Architectural conventions standard + agent-runtime toolchain matching
|
# Architectural conventions standard + agent-runtime toolchain matching
|
||||||
|
|||||||
@@ -420,6 +420,12 @@ services:
|
|||||||
# Spawn preflight — inert in practice (every real delivery role is
|
# Spawn preflight — inert in practice (every real delivery role is
|
||||||
# gateway-enabled); carried at OFF for parity with the build compose.
|
# gateway-enabled); carried at OFF for parity with the build compose.
|
||||||
ROBOCO_SPAWN_PREFLIGHT_ENABLED: ${ROBOCO_SPAWN_PREFLIGHT_ENABLED:-false}
|
ROBOCO_SPAWN_PREFLIGHT_ENABLED: ${ROBOCO_SPAWN_PREFLIGHT_ENABLED:-false}
|
||||||
|
# fastapi-guard's emergency-lockdown whitelist escape hatch. Carried
|
||||||
|
# here even though the guard trio itself (ENABLED/PASSIVE_MODE/
|
||||||
|
# FAIL_SECURE, see the declared-contract note above) is intentionally
|
||||||
|
# omitted — inert while guard stays off by default, but reaches the
|
||||||
|
# container the moment an operator arms guard by hand-editing this file.
|
||||||
|
ROBOCO_GUARD_EMERGENCY_WHITELIST: ${ROBOCO_GUARD_EMERGENCY_WHITELIST:-}
|
||||||
# Cloud auth (FastAPI Users): login-gates the panel/API when exposed
|
# Cloud auth (FastAPI Users): login-gates the panel/API when exposed
|
||||||
# beyond localhost. OFF by default (matches config default, unlike the
|
# beyond localhost. OFF by default (matches config default, unlike the
|
||||||
# build compose which arms it for the personal deploy). Set
|
# build compose which arms it for the personal deploy). Set
|
||||||
|
|||||||
@@ -652,6 +652,10 @@ services:
|
|||||||
ROBOCO_GUARD_ENABLED: ${ROBOCO_GUARD_ENABLED:-true}
|
ROBOCO_GUARD_ENABLED: ${ROBOCO_GUARD_ENABLED:-true}
|
||||||
ROBOCO_GUARD_PASSIVE_MODE: ${ROBOCO_GUARD_PASSIVE_MODE:-false}
|
ROBOCO_GUARD_PASSIVE_MODE: ${ROBOCO_GUARD_PASSIVE_MODE:-false}
|
||||||
ROBOCO_GUARD_FAIL_SECURE: ${ROBOCO_GUARD_FAIL_SECURE:-false}
|
ROBOCO_GUARD_FAIL_SECURE: ${ROBOCO_GUARD_FAIL_SECURE:-false}
|
||||||
|
# Emergency-lockdown whitelist escape hatch. Without this line here,
|
||||||
|
# setting it in .env silently does nothing — only vars listed in this
|
||||||
|
# stanza reach the container.
|
||||||
|
ROBOCO_GUARD_EMERGENCY_WHITELIST: ${ROBOCO_GUARD_EMERGENCY_WHITELIST:-}
|
||||||
volumes:
|
volumes:
|
||||||
# Docker socket - allows spawning agent containers
|
# Docker socket - allows spawning agent containers
|
||||||
- /var/run/docker.sock:/var/run/docker.sock
|
- /var/run/docker.sock:/var/run/docker.sock
|
||||||
|
|||||||
@@ -652,6 +652,10 @@ services:
|
|||||||
ROBOCO_GUARD_ENABLED: ${ROBOCO_GUARD_ENABLED:-true}
|
ROBOCO_GUARD_ENABLED: ${ROBOCO_GUARD_ENABLED:-true}
|
||||||
ROBOCO_GUARD_PASSIVE_MODE: ${ROBOCO_GUARD_PASSIVE_MODE:-false}
|
ROBOCO_GUARD_PASSIVE_MODE: ${ROBOCO_GUARD_PASSIVE_MODE:-false}
|
||||||
ROBOCO_GUARD_FAIL_SECURE: ${ROBOCO_GUARD_FAIL_SECURE:-false}
|
ROBOCO_GUARD_FAIL_SECURE: ${ROBOCO_GUARD_FAIL_SECURE:-false}
|
||||||
|
# Emergency-lockdown whitelist escape hatch. Without this line here,
|
||||||
|
# setting it in .env silently does nothing — only vars listed in this
|
||||||
|
# stanza reach the container.
|
||||||
|
ROBOCO_GUARD_EMERGENCY_WHITELIST: ${ROBOCO_GUARD_EMERGENCY_WHITELIST:-}
|
||||||
volumes:
|
volumes:
|
||||||
# Docker socket - allows spawning agent containers
|
# Docker socket - allows spawning agent containers
|
||||||
- /var/run/docker.sock:/var/run/docker.sock
|
- /var/run/docker.sock:/var/run/docker.sock
|
||||||
|
|||||||
@@ -758,6 +758,28 @@ class Settings(BaseSettings):
|
|||||||
"lockdown, in addition to loopback. Empty = loopback only."
|
"lockdown, in addition to loopback. Empty = loopback only."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
guard_trusted_chain_peers: str = Field(
|
||||||
|
default="",
|
||||||
|
description=(
|
||||||
|
"Comma-separated exact IP address(es), never a range, beyond "
|
||||||
|
"loopback, trusted to appear as a recorded PROXY HOP inside "
|
||||||
|
"X-Forwarded-For when resolving the real client behind a "
|
||||||
|
"host-proxied chain (e.g. Tailscale Serve terminating on the "
|
||||||
|
"docker host in front of nginx). A CIDR/subnet entry is "
|
||||||
|
"rejected (logged, config load still succeeds) rather than "
|
||||||
|
"accepted, since a range would readmit every sibling "
|
||||||
|
"container's real address into the hop set. Empty by default: "
|
||||||
|
"only a loopback rightmost hop peels, so a same-bridge "
|
||||||
|
"container can no longer get its own 172.x address treated as a "
|
||||||
|
"trusted hop just by being on the docker bridge. If Tailscale "
|
||||||
|
"Serve sits behind this host's docker gateway, set this to that "
|
||||||
|
"gateway's exact address (e.g. 172.18.0.1) to keep the "
|
||||||
|
"Serve-behind-gateway chain resolving. Distinct from the docker "
|
||||||
|
"bridge pool nginx itself connects FROM (still trusted "
|
||||||
|
"unconditionally so nginx can keep presenting XFF at all) — this "
|
||||||
|
"only scopes which XFF entries are treated as hops."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
# ==========================================================================
|
# ==========================================================================
|
||||||
# Production self-healing ("engine 4") — DORMANT by default
|
# Production self-healing ("engine 4") — DORMANT by default
|
||||||
|
|||||||
+152
-33
@@ -339,38 +339,100 @@ def _redis_url() -> str:
|
|||||||
# docker allocates them.
|
# docker allocates them.
|
||||||
#
|
#
|
||||||
# The variable-depth proxy chain (guard sees a fixed depth) is handled by
|
# The variable-depth proxy chain (guard sees a fixed depth) is handled by
|
||||||
# ClientIpResolutionMiddleware below: it recursively skips known local hops
|
# ClientIpResolutionMiddleware below: it resolves the real tailnet/LAN client
|
||||||
# in X-Forwarded-For and stamps the guard's request.state.client_ip cache,
|
# behind a NAMED set of local proxy hops in X-Forwarded-For (see
|
||||||
# so Tailscale-Serve/host-proxied traffic resolves to the real tailnet/LAN
|
# `_build_trusted_hop_networks`) and stamps the guard's request.state.client_ip
|
||||||
# client instead of loopback and no longer rides this exemption.
|
# cache, so Tailscale-Serve/host-proxied traffic resolves to the real
|
||||||
|
# tailnet/LAN client instead of loopback and no longer rides this exemption.
|
||||||
_INTERNAL_NETWORKS = [
|
_INTERNAL_NETWORKS = [
|
||||||
"127.0.0.1",
|
"127.0.0.1",
|
||||||
"::1",
|
"::1",
|
||||||
"172.16.0.0/12",
|
"172.16.0.0/12",
|
||||||
]
|
]
|
||||||
|
|
||||||
# XFF entries that can legitimately be a HOP nginx recorded in front of the
|
# Docker's default bridge address-pool range — the WHOLE pool, used only by
|
||||||
# real client: loopback (a host-terminated proxy like Tailscale Serve) and
|
# the broader connecting-peer gate below (never the operator-scoped hop set).
|
||||||
# the docker bridge pool (docker DNAT presents host-originated connections
|
_DOCKER_BRIDGE_POOL = "172.16.0.0/12"
|
||||||
# as the bridge gateway). Deliberately NOT the LAN/tailnet ranges — a
|
|
||||||
# 192.168.x / 100.64.x XFF entry IS the client, never a hop.
|
# Connecting-peer gate for ClientIpResolutionMiddleware: the DIRECT TCP peer
|
||||||
_TRUSTED_HOP_NETWORKS = ("127.0.0.1/32", "::1/128", "172.16.0.0/12")
|
# must be one of these before the middleware consults X-Forwarded-For at
|
||||||
|
# all. Deliberately the WHOLE docker bridge pool, unlike the hop-peel set
|
||||||
|
# below — nginx itself connects from an arbitrary bridge-allocated address
|
||||||
|
# (neither compose file pins a `subnet:`), so it must keep presenting XFF
|
||||||
|
# regardless of which address it lands on. A bare connecting-peer match
|
||||||
|
# alone stamps nothing: the XFF entries themselves are still checked against
|
||||||
|
# the narrower, operator-scoped `_TRUSTED_HOP_NETWORKS`.
|
||||||
|
_CONNECTING_PEER_NETWORKS = ("127.0.0.1/32", "::1/128", _DOCKER_BRIDGE_POOL)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_trusted_hop_networks() -> tuple[str, ...]:
|
||||||
|
"""XFF hop-peel set: loopback ALWAYS, plus operator-named chain peers.
|
||||||
|
|
||||||
|
A peeled entry is a RECORDED PROXY HOP inside X-Forwarded-For (e.g. the
|
||||||
|
docker bridge gateway nginx sees when Tailscale Serve terminates on the
|
||||||
|
host) — not the connecting socket peer (`_CONNECTING_PEER_NETWORKS`
|
||||||
|
above is that separate, deliberately broader check). Default empty
|
||||||
|
(``guard_trusted_chain_peers``): only a loopback rightmost hop ever
|
||||||
|
peels, so a same-bridge container can no longer get its own arbitrary
|
||||||
|
172.x address treated as a trusted hop just by being on the docker
|
||||||
|
bridge — closing the residual where a forged tailnet-CGNAT XFF prefix
|
||||||
|
resolved behind an unnamed 172.x rightmost entry. An operator running
|
||||||
|
Tailscale Serve behind a docker gateway sets
|
||||||
|
``ROBOCO_GUARD_TRUSTED_CHAIN_PEERS`` to that gateway's exact address
|
||||||
|
(e.g. 172.18.0.1) to keep the chain resolving.
|
||||||
|
|
||||||
|
Peers are parsed with ``ip_address`` — SINGLE addresses only, never a
|
||||||
|
CIDR range — and stored as their own /32 (or /128). A subnet-sized entry
|
||||||
|
(docker's typical bridge allocation is an entirely plausible copy-paste)
|
||||||
|
would readmit every sibling container's real address into the hop set,
|
||||||
|
fully reopening the pre-fix forge hole; ``ip_network(strict=True)``
|
||||||
|
would also silently accept a host-bits typo like "172.18.0.5/24" the
|
||||||
|
operator meant as a plain address. Any entry that isn't a plain IP —
|
||||||
|
CIDRs included — is skipped with a warning rather than crashing config
|
||||||
|
load.
|
||||||
|
"""
|
||||||
|
networks = ["127.0.0.1/32", "::1/128"]
|
||||||
|
for raw in settings.guard_trusted_chain_peers.split(","):
|
||||||
|
peer = raw.strip()
|
||||||
|
if not peer:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
addr = ip_address(peer)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
logger.warning(
|
||||||
|
"skipping invalid guard_trusted_chain_peers entry: chain "
|
||||||
|
"peers are single IP addresses; CIDR ranges are rejected "
|
||||||
|
"because a range readmits sibling containers",
|
||||||
|
peer=peer,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
networks.append(f"{addr}/{addr.max_prefixlen}")
|
||||||
|
return tuple(networks)
|
||||||
|
|
||||||
|
|
||||||
|
# Built once at config load, mirroring security_config below — pure, no I/O.
|
||||||
|
_TRUSTED_HOP_NETWORKS = _build_trusted_hop_networks()
|
||||||
|
|
||||||
# Tailscale assigns every tailnet node an address in the CGNAT range. The
|
# Tailscale assigns every tailnet node an address in the CGNAT range. The
|
||||||
# resolver stamps ONLY a candidate in this range: it makes the fix exactly as
|
# resolver stamps ONLY a candidate in this range: it makes the fix exactly as
|
||||||
# wide as the broken case (host-proxied tailnet traffic resolving to a
|
# wide as the broken case (host-proxied tailnet traffic resolving to a
|
||||||
# whitelisted hop IP) and no wider — for every other chain shape the stamp
|
# whitelisted hop IP) and no wider — for every other chain shape the stamp
|
||||||
# abstains and the guard's own depth-1 logic decides, so a same-bridge
|
# abstains and the guard's own depth-1 logic decides. Residual (accepted):
|
||||||
# container relaying a forged public-IP prefix through nginx still resolves
|
# a compromise of a CONFIGURED chain peer itself (e.g. the docker bridge
|
||||||
# to its real bridge IP exactly as before this fix. Residual (accepted): such
|
# gateway) could still relay a forged prefix behind it — an unconfigured
|
||||||
# a container can forge a 100.64/10 prefix — that only DE-privileges it
|
# peer, or any other bridge-allocated address a real container actually
|
||||||
# (loses its whitelist exemption; the fake tailnet IP eats the WAF/bans).
|
# has, cannot: it is never in `_TRUSTED_HOP_NETWORKS` by default.
|
||||||
_TAILNET_NETWORK = "100.64.0.0/10"
|
_TAILNET_NETWORK = "100.64.0.0/10"
|
||||||
|
|
||||||
# The fixable shape needs at least [client, hop] — one real entry behind one
|
# The fixable shape needs at least [client, hop] — one real entry behind one
|
||||||
# recorded proxy hop.
|
# recorded proxy hop.
|
||||||
_MIN_CHAIN_ENTRIES = 2
|
_MIN_CHAIN_ENTRIES = 2
|
||||||
|
|
||||||
|
# Rightmost gateway IPs already warned about (see
|
||||||
|
# `_warn_unconfigured_tailnet_gateway_once`) — bounded by the tiny number of
|
||||||
|
# real docker bridge gateways an operator ever runs behind.
|
||||||
|
_WARNED_UNCONFIGURED_GATEWAYS: set[str] = set()
|
||||||
|
|
||||||
|
|
||||||
def _in_networks(ip: str, networks: tuple[str, ...]) -> bool:
|
def _in_networks(ip: str, networks: tuple[str, ...]) -> bool:
|
||||||
try:
|
try:
|
||||||
@@ -384,23 +446,60 @@ def _is_trusted_hop(ip: str) -> bool:
|
|||||||
return _in_networks(ip, _TRUSTED_HOP_NETWORKS)
|
return _in_networks(ip, _TRUSTED_HOP_NETWORKS)
|
||||||
|
|
||||||
|
|
||||||
|
def _is_trusted_connecting_peer(ip: str) -> bool:
|
||||||
|
return _in_networks(ip, _CONNECTING_PEER_NETWORKS)
|
||||||
|
|
||||||
|
|
||||||
|
def _warn_unconfigured_tailnet_gateway_once(rightmost: str, candidate: str) -> None:
|
||||||
|
"""Surface the ONE regression this fix leaves genuinely silent: with no
|
||||||
|
chain peers configured, a host-proxied tailnet chain behind a real
|
||||||
|
docker bridge gateway silently reverts /tg to the pre-fix inert-WAF
|
||||||
|
state — zero signal otherwise. Only fires while NOTHING is configured
|
||||||
|
(once any peer is set the operator has already addressed this); logs
|
||||||
|
once per distinct gateway IP per process. Never logs the full XFF —
|
||||||
|
only the rightmost IP, the one an operator needs to act.
|
||||||
|
"""
|
||||||
|
if settings.guard_trusted_chain_peers.strip():
|
||||||
|
return
|
||||||
|
if not _in_networks(rightmost, (_DOCKER_BRIDGE_POOL,)):
|
||||||
|
return
|
||||||
|
if not _in_networks(candidate, (_TAILNET_NETWORK,)):
|
||||||
|
return
|
||||||
|
if rightmost in _WARNED_UNCONFIGURED_GATEWAYS:
|
||||||
|
return
|
||||||
|
_WARNED_UNCONFIGURED_GATEWAYS.add(rightmost)
|
||||||
|
logger.warning(
|
||||||
|
"host-proxied tailnet chain detected but no trusted chain peer "
|
||||||
|
"configured — this traffic resolves to a whitelisted bridge IP; "
|
||||||
|
f"set ROBOCO_GUARD_TRUSTED_CHAIN_PEERS to your docker bridge "
|
||||||
|
f"gateway ({rightmost})"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def resolve_forwarded_client_ip(forwarded_for: str) -> str | None:
|
def resolve_forwarded_client_ip(forwarded_for: str) -> str | None:
|
||||||
"""Resolve the tailnet client behind host-proxy hops; None = abstain.
|
"""Resolve the tailnet client behind NAMED host-proxy hops; None = abstain.
|
||||||
|
|
||||||
fastapi-guard peels a FIXED number of XFF hops (trusted_proxy_depth=1:
|
fastapi-guard peels a FIXED number of XFF hops (trusted_proxy_depth=1:
|
||||||
the rightmost entry, which nginx itself recorded). That is correct for
|
the rightmost entry, which nginx itself recorded). That is correct for
|
||||||
every chain except one: host-proxied tailnet traffic (Tailscale Serve →
|
every chain except one: host-proxied tailnet traffic (Tailscale Serve →
|
||||||
nginx) arrives as ``[tailnet-client, <loopback-or-bridge-gateway>]``, so
|
nginx) arrives as ``[tailnet-client, <loopback-or-configured-gateway>]``,
|
||||||
depth-1 resolves it to a whitelisted hop IP and WAF/ban/rate-limit go
|
so depth-1 resolves it to a whitelisted hop IP and WAF/ban/rate-limit go
|
||||||
inert for the whole /tg surface (the documented ceiling).
|
inert for the whole /tg surface (the documented ceiling).
|
||||||
|
|
||||||
This resolver fixes exactly that shape and nothing else: peel trusted
|
This resolver fixes exactly that shape and nothing else: peel hops from
|
||||||
hops from the right; the remaining candidate is returned ONLY if at
|
the right that are loopback or an operator-named chain peer
|
||||||
least one hop was peeled and the candidate is in the tailnet CGNAT
|
(``guard_trusted_chain_peers``, see `_build_trusted_hop_networks`); the
|
||||||
range. Every other shape — direct LAN client, agent container via nginx
|
remaining candidate is returned ONLY if at least one hop was peeled and
|
||||||
(even with a forged public-IP prefix), all-hops operator traffic,
|
the candidate is in the tailnet CGNAT range. Every other shape — direct
|
||||||
malformed entries — returns None, leaving the guard's own depth-1
|
LAN client, an agent container relaying via nginx with an UNNAMED 172.x
|
||||||
resolution in charge, byte-for-byte identical to before this fix.
|
address (even carrying a forged public-IP or tailnet-CGNAT prefix),
|
||||||
|
all-hops operator traffic, malformed entries — returns None, leaving the
|
||||||
|
guard's own depth-1 resolution in charge. With no chain peers configured
|
||||||
|
(the default), only a loopback rightmost hop ever peels — when that
|
||||||
|
abstain is otherwise shaped exactly like a host-proxied tailnet chain
|
||||||
|
behind a real bridge gateway, it is logged once per gateway IP (see
|
||||||
|
`_warn_unconfigured_tailnet_gateway_once`) so the regression isn't
|
||||||
|
silent.
|
||||||
"""
|
"""
|
||||||
entries = [e.strip() for e in forwarded_for.split(",") if e.strip()]
|
entries = [e.strip() for e in forwarded_for.split(",") if e.strip()]
|
||||||
if len(entries) < _MIN_CHAIN_ENTRIES:
|
if len(entries) < _MIN_CHAIN_ENTRIES:
|
||||||
@@ -408,8 +507,11 @@ def resolve_forwarded_client_ip(forwarded_for: str) -> str | None:
|
|||||||
idx = len(entries) - 1
|
idx = len(entries) - 1
|
||||||
while idx >= 0 and _is_trusted_hop(entries[idx]):
|
while idx >= 0 and _is_trusted_hop(entries[idx]):
|
||||||
idx -= 1
|
idx -= 1
|
||||||
if idx == len(entries) - 1 or idx < 0:
|
if idx == len(entries) - 1:
|
||||||
return None # no hop peeled, or all hops: baseline handles both
|
_warn_unconfigured_tailnet_gateway_once(entries[-1], entries[-2])
|
||||||
|
return None # no hop peeled: baseline handles it
|
||||||
|
if idx < 0:
|
||||||
|
return None # all hops: baseline handles it
|
||||||
candidate = entries[idx]
|
candidate = entries[idx]
|
||||||
if not _in_networks(candidate, (_TAILNET_NETWORK,)):
|
if not _in_networks(candidate, (_TAILNET_NETWORK,)):
|
||||||
return None
|
return None
|
||||||
@@ -423,10 +525,12 @@ class ClientIpResolutionMiddleware:
|
|||||||
Pure ASGI, mounted OUTSIDE SecurityMiddleware (added after it, so it runs
|
Pure ASGI, mounted OUTSIDE SecurityMiddleware (added after it, so it runs
|
||||||
first): guard_core's ``extract_client_ip`` returns a pre-cached
|
first): guard_core's ``extract_client_ip`` returns a pre-cached
|
||||||
``state.client_ip`` verbatim, which is the supported seam for custom
|
``state.client_ip`` verbatim, which is the supported seam for custom
|
||||||
resolution. Only honors XFF when the CONNECTING peer is itself a known
|
resolution. Only honors XFF when the CONNECTING peer is itself trusted to
|
||||||
local hop (nginx's bridge IP / loopback) — a directly-connected client's
|
present it (nginx's docker-bridge address / loopback —
|
||||||
forged XFF is never consulted here (the guard's own depth-1 logic keeps
|
`_CONNECTING_PEER_NETWORKS`, deliberately broader than and independent of
|
||||||
handling that class unchanged).
|
the operator-scoped hop-peel set the XFF entries are matched against) —
|
||||||
|
a directly-connected client's forged XFF is never consulted here (the
|
||||||
|
guard's own depth-1 logic keeps handling that class unchanged).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, app: Any) -> None:
|
def __init__(self, app: Any) -> None:
|
||||||
@@ -436,7 +540,7 @@ class ClientIpResolutionMiddleware:
|
|||||||
if scope["type"] == "http":
|
if scope["type"] == "http":
|
||||||
client = scope.get("client")
|
client = scope.get("client")
|
||||||
connecting_ip = client[0] if client else None
|
connecting_ip = client[0] if client else None
|
||||||
if connecting_ip and _is_trusted_hop(connecting_ip):
|
if connecting_ip and _is_trusted_connecting_peer(connecting_ip):
|
||||||
# First occurrence on a repeated header, matching Starlette's
|
# First occurrence on a repeated header, matching Starlette's
|
||||||
# Headers.get — so this layer and the guard's own fallback
|
# Headers.get — so this layer and the guard's own fallback
|
||||||
# read the SAME header value.
|
# read the SAME header value.
|
||||||
@@ -459,7 +563,22 @@ def _guard_whitelist() -> list[str]:
|
|||||||
extra = [
|
extra = [
|
||||||
x.strip() for x in settings.guard_emergency_whitelist.split(",") if x.strip()
|
x.strip() for x in settings.guard_emergency_whitelist.split(",") if x.strip()
|
||||||
]
|
]
|
||||||
return [*_INTERNAL_NETWORKS, *extra]
|
# guard-core's whitelist is EXCLUSIVE once non-empty (guard_core.utils.
|
||||||
|
# is_ip_allowed: a set whitelist replaces, not supplements, the
|
||||||
|
# blacklist check — any non-member IP is refused outright, not merely
|
||||||
|
# unexempted). The resolver above now honestly resolves a host-proxied
|
||||||
|
# tailnet client's real 100.64.0.0/10 address instead of a loopback/
|
||||||
|
# bridge hop, so that address must be a whitelist member or ip_security
|
||||||
|
# rejects it — this blocked the CEO's own tailnet IP live (2026-07-22).
|
||||||
|
# Coupling allowlist membership with scrutiny-exemption here is
|
||||||
|
# deliberate for the tailnet specifically: Tailscale is an authenticated
|
||||||
|
# overlay that gates device membership before a packet ever reaches this
|
||||||
|
# host, so an already-authenticated tailnet peer skipping WAF/ban/rate-
|
||||||
|
# limit is the correct posture, not a gap. The resolver's real-IP
|
||||||
|
# stamping still buys correct attribution in logs/telemetry, and any
|
||||||
|
# FUTURE non-tailnet public exposure still gets full scrutiny — only
|
||||||
|
# 100.64.0.0/10 is exempted here, nothing wider.
|
||||||
|
return [*_INTERNAL_NETWORKS, _TAILNET_NETWORK, *extra]
|
||||||
|
|
||||||
|
|
||||||
def _emergency_whitelist() -> list[str]:
|
def _emergency_whitelist() -> list[str]:
|
||||||
|
|||||||
@@ -209,10 +209,20 @@ def test_internal_agent_mesh_is_whitelisted() -> None:
|
|||||||
"""Agents reach the orchestrator directly on the docker bridge, HMAC-
|
"""Agents reach the orchestrator directly on the docker bridge, HMAC-
|
||||||
authenticated; the guard's threat-ban is for the external surface. Without
|
authenticated; the guard's threat-ban is for the external surface. Without
|
||||||
this the guard IP-banned agent containers the moment it went active
|
this the guard IP-banned agent containers the moment it went active
|
||||||
(2026-07-20 incident) and wedged every subsequent gateway verb."""
|
(2026-07-20 incident) and wedged every subsequent gateway verb.
|
||||||
|
|
||||||
|
The tailnet CGNAT range (100.64.0.0/10) rides the same whitelist:
|
||||||
|
guard-core's whitelist is EXCLUSIVE once non-empty (any non-member IP is
|
||||||
|
refused, not merely unexempted), and the resolver now honestly resolves
|
||||||
|
a host-proxied tailnet client to its real 100.64.0.0/10 address instead
|
||||||
|
of a loopback/bridge hop — omitting it here blocked the CEO's own
|
||||||
|
tailnet IP live (2026-07-22 incident). Tailscale is an authenticated
|
||||||
|
overlay gating device membership before a packet arrives, so coupling
|
||||||
|
allowlisting with scrutiny-exemption is the deliberate posture for this
|
||||||
|
one range."""
|
||||||
cfg = security.build_security_config()
|
cfg = security.build_security_config()
|
||||||
assert cfg.whitelist is not None
|
assert cfg.whitelist is not None
|
||||||
for net in ("127.0.0.1", "::1", "172.16.0.0/12"):
|
for net in ("127.0.0.1", "::1", "172.16.0.0/12", "100.64.0.0/10"):
|
||||||
assert net in cfg.whitelist
|
assert net in cfg.whitelist
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -3,12 +3,16 @@
|
|||||||
fastapi-guard peels a fixed trusted_proxy_depth=1 from X-Forwarded-For (the
|
fastapi-guard peels a fixed trusted_proxy_depth=1 from X-Forwarded-For (the
|
||||||
rightmost entry, which nginx itself recorded). That is correct for every
|
rightmost entry, which nginx itself recorded). That is correct for every
|
||||||
chain except host-proxied tailnet traffic (Tailscale Serve → nginx), which
|
chain except host-proxied tailnet traffic (Tailscale Serve → nginx), which
|
||||||
arrives as ``[tailnet-client, <loopback-or-bridge-gateway>]`` — depth-1
|
arrives as ``[tailnet-client, <loopback-or-configured-gateway>]`` — depth-1
|
||||||
resolves it to a whitelisted hop IP and the WAF goes inert for /tg.
|
resolves it to a whitelisted hop IP and the WAF goes inert for /tg.
|
||||||
``ClientIpResolutionMiddleware`` stamps guard_core's ``state.client_ip``
|
``ClientIpResolutionMiddleware`` stamps guard_core's ``state.client_ip``
|
||||||
cache (the supported pre-resolution seam) for EXACTLY that shape and
|
cache (the supported pre-resolution seam) for EXACTLY that shape and
|
||||||
abstains on every other, so no path resolves differently from the depth-1
|
abstains on every other, so no path resolves differently from the depth-1
|
||||||
baseline unless the candidate is a tailnet CGNAT address behind real hops.
|
baseline unless the candidate is a tailnet CGNAT address behind real hops.
|
||||||
|
|
||||||
|
The XFF hop-peel set is loopback ALWAYS plus operator-named
|
||||||
|
``guard_trusted_chain_peers`` (default empty) — NOT the whole docker bridge
|
||||||
|
pool, so an unnamed 172.x address can never be treated as a hop.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -27,6 +31,19 @@ from roboco.security import (
|
|||||||
)
|
)
|
||||||
from starlette.requests import Request
|
from starlette.requests import Request
|
||||||
|
|
||||||
|
|
||||||
|
def _configure_chain_peers(monkeypatch: pytest.MonkeyPatch, csv: str) -> None:
|
||||||
|
"""Set guard_trusted_chain_peers and rebuild the effective hop set.
|
||||||
|
|
||||||
|
Mirrors what config load does once at import time; tests need to redo
|
||||||
|
it per-case since ``_TRUSTED_HOP_NETWORKS`` is otherwise built once.
|
||||||
|
"""
|
||||||
|
monkeypatch.setattr(settings, "guard_trusted_chain_peers", csv)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
security, "_TRUSTED_HOP_NETWORKS", security._build_trusted_hop_networks()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# resolve_forwarded_client_ip — stamps ONLY the tailnet-behind-hops shape
|
# resolve_forwarded_client_ip — stamps ONLY the tailnet-behind-hops shape
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -38,14 +55,25 @@ def test_tailscale_serve_behind_loopback_resolves_tailnet_peer() -> None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_tailscale_serve_behind_bridge_gateway_resolves_tailnet_peer() -> None:
|
def test_tailscale_serve_behind_bridge_gateway_resolves_tailnet_peer(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
# Docker DNAT presents host-originated connections as the bridge gateway,
|
# Docker DNAT presents host-originated connections as the bridge gateway,
|
||||||
# so nginx may record 172.x instead of loopback for Tailscale Serve.
|
# so nginx may record 172.x instead of loopback for Tailscale Serve. That
|
||||||
|
# 172.x address only peels once the operator names it explicitly.
|
||||||
|
_configure_chain_peers(monkeypatch, "172.18.0.1")
|
||||||
assert (
|
assert (
|
||||||
resolve_forwarded_client_ip("100.101.102.103, 172.18.0.1") == "100.101.102.103"
|
resolve_forwarded_client_ip("100.101.102.103, 172.18.0.1") == "100.101.102.103"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_bridge_gateway_unconfigured_by_default_never_peels() -> None:
|
||||||
|
# Same chain as above, but with NO configured chain peers (the default):
|
||||||
|
# the 172.x gateway is no longer a recognized hop, so the resolver
|
||||||
|
# abstains and depth-1 keeps resolving to the gateway IP itself.
|
||||||
|
assert resolve_forwarded_client_ip("100.101.102.103, 172.18.0.1") is None
|
||||||
|
|
||||||
|
|
||||||
def test_forged_prefix_behind_tailscale_chain_ignored() -> None:
|
def test_forged_prefix_behind_tailscale_chain_ignored() -> None:
|
||||||
assert (
|
assert (
|
||||||
resolve_forwarded_client_ip("6.6.6.6, 100.101.102.103, 127.0.0.1")
|
resolve_forwarded_client_ip("6.6.6.6, 100.101.102.103, 127.0.0.1")
|
||||||
@@ -66,11 +94,25 @@ def test_bridge_peer_with_forged_public_prefix_abstains() -> None:
|
|||||||
assert resolve_forwarded_client_ip("9.9.9.9, 172.20.0.7") is None
|
assert resolve_forwarded_client_ip("9.9.9.9, 172.20.0.7") is None
|
||||||
|
|
||||||
|
|
||||||
def test_bridge_peer_forging_tailnet_prefix_only_deprivileges() -> None:
|
def test_default_empty_chain_peers_closes_the_forge_residual() -> None:
|
||||||
# Documented residual: forging a CGNAT prefix IS stamped — the forger
|
# THE fixed residual: a same-bridge container relaying a forged
|
||||||
# loses its whitelist exemption (fake tailnet IPs eat the WAF); it can
|
# tailnet-CGNAT XFF prefix used to have its unnamed 172.x rightmost
|
||||||
# never gain privilege this way.
|
# entry peeled as a "trusted hop" (the whole /12 was the hop set),
|
||||||
assert resolve_forwarded_client_ip("100.99.1.1, 172.20.0.7") == "100.99.1.1"
|
# stamping the forged CGNAT address. With no chain peers configured
|
||||||
|
# (the default), 172.20.0.7 is never a recognized hop, so no hop is
|
||||||
|
# peeled and the resolver abstains — the forge no longer lands.
|
||||||
|
assert resolve_forwarded_client_ip("100.99.1.1, 172.20.0.7") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_configured_peer_does_not_extend_to_other_bridge_addresses(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
# Even with a real chain peer configured (the docker gateway), a forger
|
||||||
|
# relaying through its OWN bridge IP — never the reserved gateway
|
||||||
|
# address — still abstains: only the exact configured peer(s) peel, not
|
||||||
|
# the whole bridge pool.
|
||||||
|
_configure_chain_peers(monkeypatch, "172.18.0.1")
|
||||||
|
assert resolve_forwarded_client_ip("100.99.1.1, 172.20.0.7") is None
|
||||||
|
|
||||||
|
|
||||||
def test_non_tailnet_client_behind_hop_abstains() -> None:
|
def test_non_tailnet_client_behind_hop_abstains() -> None:
|
||||||
@@ -93,6 +135,123 @@ def test_malformed_entries_abstain() -> None:
|
|||||||
assert resolve_forwarded_client_ip("100.99.1.1:443, 127.0.0.1") is None
|
assert resolve_forwarded_client_ip("100.99.1.1:443, 127.0.0.1") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_loopback_hop_chains_unaffected_by_peer_config(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
# Loopback is unconditionally part of the hop set regardless of what
|
||||||
|
# (if anything) is configured — a chain peer config only ADDS to it.
|
||||||
|
_configure_chain_peers(monkeypatch, "172.18.0.1")
|
||||||
|
assert (
|
||||||
|
resolve_forwarded_client_ip("100.101.102.103, 127.0.0.1") == "100.101.102.103"
|
||||||
|
)
|
||||||
|
assert resolve_forwarded_client_ip("127.0.0.1") is None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# _build_trusted_hop_networks — csv parsing at config-load time
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalid_chain_peer_entry_skipped_without_crash(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
monkeypatch.setattr(
|
||||||
|
settings, "guard_trusted_chain_peers", "not-a-cidr, 172.18.0.1, "
|
||||||
|
)
|
||||||
|
networks = security._build_trusted_hop_networks()
|
||||||
|
assert networks == ("127.0.0.1/32", "::1/128", "172.18.0.1/32")
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_chain_peers_yields_loopback_only() -> None:
|
||||||
|
assert security._build_trusted_hop_networks() == ("127.0.0.1/32", "::1/128")
|
||||||
|
|
||||||
|
|
||||||
|
def test_plain_ip_chain_peer_stored_as_its_own_slash_32(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
monkeypatch.setattr(settings, "guard_trusted_chain_peers", "172.18.0.1")
|
||||||
|
assert security._build_trusted_hop_networks() == (
|
||||||
|
"127.0.0.1/32",
|
||||||
|
"::1/128",
|
||||||
|
"172.18.0.1/32",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_subnet_chain_peer_entry_rejected_and_not_in_hop_set(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
# Docker's typical bridge allocation — an entirely plausible operator
|
||||||
|
# copy-paste — must be rejected, not admitted: a range would readmit
|
||||||
|
# every sibling container's real address into the hop set.
|
||||||
|
monkeypatch.setattr(settings, "guard_trusted_chain_peers", "172.16.0.0/12")
|
||||||
|
assert security._build_trusted_hop_networks() == ("127.0.0.1/32", "::1/128")
|
||||||
|
# And the forge chain a wrongly-admitted /12 would have reopened still
|
||||||
|
# abstains end-to-end.
|
||||||
|
_configure_chain_peers(monkeypatch, "172.16.0.0/12")
|
||||||
|
assert resolve_forwarded_client_ip("100.99.1.1, 172.20.0.7") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_host_bits_typo_cidr_rejected_same_as_a_subnet(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
# A typo'd CIDR the operator meant as a plain address — ip_network's
|
||||||
|
# default strict=True would silently accept this; ip_address rejects it.
|
||||||
|
monkeypatch.setattr(settings, "guard_trusted_chain_peers", "172.18.0.5/24")
|
||||||
|
assert security._build_trusted_hop_networks() == ("127.0.0.1/32", "::1/128")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# _warn_unconfigured_tailnet_gateway_once — the silent-default-gap signal
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_unconfigured_gateway_chain_warns_once_per_gateway(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
monkeypatch.setattr(security, "_WARNED_UNCONFIGURED_GATEWAYS", set())
|
||||||
|
calls: list[tuple[Any, ...]] = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
security.logger, "warning", lambda *a, **kw: calls.append((a, kw))
|
||||||
|
)
|
||||||
|
for _ in range(3):
|
||||||
|
assert resolve_forwarded_client_ip("100.101.102.103, 172.19.0.9") is None
|
||||||
|
assert len(calls) == 1
|
||||||
|
assert "172.19.0.9" in calls[0][0][0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_configured_peer_suppresses_the_gap_warning(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
# A DIFFERENT gateway than the one configured would otherwise trip the
|
||||||
|
# detection, but ANY configured peer means the operator has already
|
||||||
|
# addressed the awareness gap — the warning never fires at all.
|
||||||
|
monkeypatch.setattr(security, "_WARNED_UNCONFIGURED_GATEWAYS", set())
|
||||||
|
_configure_chain_peers(monkeypatch, "172.18.0.1")
|
||||||
|
calls: list[tuple[Any, ...]] = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
security.logger, "warning", lambda *a, **kw: calls.append((a, kw))
|
||||||
|
)
|
||||||
|
assert resolve_forwarded_client_ip("100.99.1.1, 172.20.0.7") is None
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_host_proxied_shapes_never_trigger_the_gap_warning(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
monkeypatch.setattr(security, "_WARNED_UNCONFIGURED_GATEWAYS", set())
|
||||||
|
calls: list[tuple[Any, ...]] = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
security.logger, "warning", lambda *a, **kw: calls.append((a, kw))
|
||||||
|
)
|
||||||
|
# A forged public-IP prefix behind a bridge address isn't the tailnet
|
||||||
|
# (CGNAT) shape.
|
||||||
|
resolve_forwarded_client_ip("9.9.9.9, 172.20.0.7")
|
||||||
|
# A LAN client behind a genuine loopback hop resolves — a hop WAS
|
||||||
|
# peeled, so this never reaches the unconfigured-gateway branch.
|
||||||
|
resolve_forwarded_client_ip("203.0.113.9, 127.0.0.1")
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# ClientIpResolutionMiddleware stamping
|
# ClientIpResolutionMiddleware stamping
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
Reference in New Issue
Block a user