Files
4034380285 fix(security): reject loopback for BYOK asset download URLs (#5478) (#6072)
* fix(security): reject loopback for BYOK asset download URLs (#5478)

The assertExternalAssetUrl guard previously routed through
validateBaseUrlResolved without forbidLoopback, so asset URLs pointing
at 127.0.0.1, localhost, or ::1 passed validation and were fetched by
the daemon. This allowed a compromised or misconfigured upstream gateway
to exfiltrate data from internal services via SSRF.

Changes:
- Add forbidLoopback option to ValidateBaseUrlOptions (packages/contracts)
- validateBaseUrl rejects loopback hosts when forbidLoopback is set
- validateBaseUrlResolved skips the loopback early-return when forbidLoopback
- validateBaseUrlResolved: when forbidLoopback is true, DNS-resolved
  loopback addresses are also rejected (not just literal loopback hosts)
- assertExternalAssetUrl now passes forbidLoopback: true
- User-configured provider endpoints (validateUserProviderBaseUrl) are
  unaffected — they still allow loopback for local gateways
- Adds regression tests covering literal loopback rejection, DNS-resolved
  loopback rejection (IPv4, IPv6, mixed results), and provider endpoint
  exemption

Note: PR #5529 fixed the library-ingest SSRF path. This PR closes the
separate BYOK asset-download path (assertExternalAssetUrl) that retains
the loopback carve-out.

Closes #5478

* fix(security): pin DNS resolution to prevent rebinding in asset fetch (#5478)

The validation step (validateBaseUrlResolved) and the fetch step
(assertAndFetchExternalAsset) previously performed independent DNS
lookups. A DNS-rebinding domain could return a public IP for validation
and then 127.0.0.1 / ::1 at fetch time, bypassing the loopback guard.

Fix:
- validateBaseUrlResolved now attaches the DNS-resolved addresses that
  passed validation to its return value (BaseUrlValidationResult.resolvedAddresses)
- assertExternalAssetUrl passes these addresses through on the ok branch
- assertAndFetchExternalAsset creates an Undici Agent with a custom
  connect.lookup that returns only the validated addresses, pinning the
  TCP connection regardless of any DNS rebind between validation and fetch
- 3 call sites in byok-tools.ts that used validate-then-fetch separately
  now route through assertAndFetchExternalAsset for a single unified path
- 4 regression tests covering: single validated address, DNS-rebind mock
  (public→loopback), round-robin DNS, and IP-literal short-circuit

* fix(security): fail-closed on DNS error + connection-time validating lookup (#5478)

Addresses mrcfps third review: the fallback to unpinned fetch when
validation did not attach resolved addresses left a DNS-rebinding hole
where an attacker could make the validation lookup throw (ENOTFOUND /
SERVFAIL) and then answer loopback for the fetch lookup.

Changes:

1. validateBaseUrlResolved: when forbidLoopback is true, DNS lookup
   failures now return a forbidden result instead of the sync success.
   This prevents the fail-then-rebind vector.

2. assertAndFetchExternalAsset: removed the unpinned-fetch fallback.
   Non-IP-literal hostnames without resolvedAddresses now throw — the
   fetch never happens. IP literals are safe because they were validated
   synchronously and have no hostname to rebind.

3. Replaced the per-request pinned Agent with a long-lived Undici Agent
   (assetDispatcher) whose connection-time lookup (createAssetValidatingLookup)
   rejects any non-public address. This is defense-in-depth on top of the
   pre-validation, following the same pattern as brands/safe-fetch.ts and
   plugins/plugin-asset-cache.ts. A shared dispatcher also avoids the
   keep-alive socket leak of per-request Agents.

4. createAssetValidatingLookup is exported for unit testing.

5. Expanded test coverage:
   - DNS failure → fail-closed (forbidLoopback true and false)
   - createAssetValidatingLookup: rejects loopback, RFC1918, metadata IP;
     allows public
   - assertAndFetchExternalAsset: throws on blocked URLs and internal IPs
   - Updated existing public-CDN tests to use IP literals (fail-closed
     behavior now correctly rejects hostnames when DNS is unavailable)

* fix(security): injectable lookup + testable fetch for asset SSRF guard (#5478)

Addresses mrcfps 4th review: daemon CI was red because fail-closed
behavior caused hostname-based test fixtures to NXDOMAIN, and
undiciFetch bypassed vi.stubGlobal('fetch') stubs.

Changes:

1. assertExternalAssetUrl and assertAndFetchExternalAsset now accept
   an optional injectable `lookup` (DnsLookupFn) and `fetchImpl`
   parameter. Production callers pass neither, so default DNS and
   globalThis.fetch are used. Tests can inject mock lookups/fetch.

2. assertAndFetchExternalAsset uses globalThis.fetch (not undiciFetch)
   so vi.stubGlobal('fetch', ...) stubs in existing test suites
   (byok-tools, aihubmix, proxy-routes, senseaudio) still intercept.

3. Existing test fixtures: replaced cdn.example.test with 93.184.216.34
   (public IP literal) in 6 test files (41 substitutions) so IP
   literals skip DNS resolution and don't trigger fail-closed on
   NXDOMAIN .test hostnames.

4. Added 3 TOCTOU regression tests driving assertAndFetchExternalAsset
   with injected lookup + fetch:
   - Validation lookup public → fetch called with redirect:error
   - Validation lookup throws → assertAndFetchExternalAsset rejects
     without invoking fetch (fail-then-rebind vector closed)
   - IP literal → skips DNS, calls fetch directly

5. assetDispatcher + createAssetValidatingLookup kept for production
   connection-time defense-in-depth (exported, available for future
   wiring to a production fetch dispatcher).

Total: 31 tests in asset-ssrf-loopback.test.ts, all passing.

* fix(security): attach validating dispatcher to asset fetch (#5478)

mrcfps 5th review: assetDispatcher was built but never attached to the
fetch call. The validating lookup only runs at connect time through the
dispatcher, so without it the fetch re-resolves the attacker-controlled
hostname and a public-then-loopback rebind still reaches the daemon.

Fix: attach assetDispatcher to the RequestInit via the dispatcher key
(same pattern as plugins/plugin-asset-cache.ts safeExternalFetch).
Production globalThis.fetch (Node/undici) uses it to refuse connect-time
non-public addresses. Test stubs still see redirect:'error' and can
ignore dispatcher.

Also fixed the misnamed TOCTOU test to assert dispatcher is present on
the captured RequestInit, and removed unused undiciFetch import.

* fix(tests): update dispatcher assertions for assertAndFetchExternalAsset

assertAndFetchExternalAsset now attaches a validating assetDispatcher to
the fetch RequestInit (issue #5478). Tests that blanket-checked
init?.dispatcher === testMock fail because asset-download calls correctly
carry the validating dispatcher instead.

Fix: conditional assertion — when init.redirect === 'error' (asset download
path), check dispatcher exists rather than matching the test mock. API
calls still verify the caller's dispatcher is forwarded unchanged.

* fix(tests): URL-keyed dispatcher assertions + document asset dispatcher override

Address mrcfps's non-blocking review items:

1. Replace the redirect==='error' heuristic with URL-keyed assertions.
   The heuristic silently dropped the submit dispatcher identity check
   on the AIHubMix video path (its submit hop also sets redirect:'error').
   Submit/poll hops now assert toBe(callerDispatcher); the asset hop
   asserts toBe(getAssetValidatingDispatcher()) — exported from
   connectionTest.ts so the exact validating instance is compared.

2. Document the intentional dispatcher override on the
   BYOKToolContext.requestInit JSDoc: asset downloads through
   assertAndFetchExternalAsset intentionally never ride the turn proxy
   dispatcher; submit/poll hops keep it.

---------

Co-authored-by: CVE-Hunter-Leo <cve-hunter-leo@users.noreply.github.com>
2026-08-17 06:59:31 +00:00

377 lines
11 KiB
TypeScript

import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { generateMedia } from '../../src/media/index.js';
const TEST_SENSEAUDIO_BASE_URL = 'https://senseaudio-gateway.example.test';
const WIRE_MODEL = 'senseaudio-tts-1.5-260319';
function buildOkResponse(audioBytes: Buffer, opts?: { audioLength?: number }) {
return new Response(
JSON.stringify({
data: { audio: audioBytes.toString('hex'), status: 2 },
extra_info: {
audio_length: opts?.audioLength ?? 12340,
audio_sample_rate: 32000,
audio_size: audioBytes.length,
bitrate: 128000,
audio_format: 'mp3',
audio_channel: 2,
word_count: 8,
usage_characters: 8,
},
trace_id: 'trace-test',
base_resp: { status_code: 0, status_msg: 'success' },
}),
{ status: 200, headers: { 'content-type': 'application/json' } },
);
}
describe('senseaudio media generation', () => {
let root: string;
let projectRoot: string;
let projectsRoot: string;
const realFetch = globalThis.fetch;
const originalMediaConfigDir = process.env.OD_MEDIA_CONFIG_DIR;
const originalDataDir = process.env.OD_DATA_DIR;
beforeEach(async () => {
root = await mkdtemp(path.join(tmpdir(), 'od-senseaudio-'));
projectRoot = path.join(root, 'project-root');
projectsRoot = path.join(projectRoot, '.od', 'projects');
await mkdir(projectsRoot, { recursive: true });
delete process.env.OD_MEDIA_CONFIG_DIR;
delete process.env.OD_DATA_DIR;
delete process.env.OD_SENSEAUDIO_API_KEY;
delete process.env.SENSEAUDIO_API_KEY;
});
afterEach(async () => {
globalThis.fetch = realFetch;
if (originalMediaConfigDir == null) {
delete process.env.OD_MEDIA_CONFIG_DIR;
} else {
process.env.OD_MEDIA_CONFIG_DIR = originalMediaConfigDir;
}
if (originalDataDir == null) {
delete process.env.OD_DATA_DIR;
} else {
process.env.OD_DATA_DIR = originalDataDir;
}
delete process.env.OD_SENSEAUDIO_API_KEY;
delete process.env.SENSEAUDIO_API_KEY;
await rm(root, { recursive: true, force: true });
});
async function writeConfig(data: unknown) {
const file = path.join(projectRoot, '.od', 'media-config.json');
await mkdir(path.dirname(file), { recursive: true });
await writeFile(file, JSON.stringify(data), 'utf8');
}
it('renders SenseAudio speech with the documented defaults', async () => {
await writeConfig({
providers: {
senseaudio: {
apiKey: 'sense-test-key',
baseUrl: TEST_SENSEAUDIO_BASE_URL,
},
},
});
const mp3Bytes = Buffer.from([0x49, 0x44, 0x33, 0x04, 0x00, 0x00, 0x73, 0x65, 0x6e]);
const fetchMock = vi.fn(async (input: unknown, init?: RequestInit) => {
expect(String(input)).toBe(`${TEST_SENSEAUDIO_BASE_URL}/v1/t2a_v2`);
expect(init?.method).toBe('POST');
expect(init?.headers).toMatchObject({
authorization: 'Bearer sense-test-key',
'content-type': 'application/json',
});
expect(JSON.parse(String(init?.body))).toEqual({
model: WIRE_MODEL,
text: '你好,欢迎使用 SenseAudio 语音合成服务。',
stream: false,
voice_setting: {
voice_id: 'female_0033_b',
speed: 1,
vol: 1,
pitch: 0,
},
audio_setting: {
format: 'mp3',
sample_rate: 32000,
bitrate: 128000,
channel: 2,
},
});
return buildOkResponse(mp3Bytes);
});
vi.stubGlobal('fetch', fetchMock);
const result = await generateMedia({
projectRoot,
projectsRoot,
projectId: 'project-1',
surface: 'audio',
model: 'senseaudio-tts',
audioKind: 'speech',
prompt: '你好,欢迎使用 SenseAudio 语音合成服务。',
output: 'senseaudio-speech.mp3',
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(result.providerId).toBe('senseaudio');
expect(result.providerNote).toContain(`senseaudio/${WIRE_MODEL}`);
expect(result.providerNote).toContain('female_0033_b');
const bytes = await readFile(path.join(projectsRoot, 'project-1', 'senseaudio-speech.mp3'));
expect(bytes.equals(mp3Bytes)).toBe(true);
});
it('passes custom voice id through to the request body', async () => {
await writeConfig({
providers: {
senseaudio: {
apiKey: 'sense-test-key',
baseUrl: TEST_SENSEAUDIO_BASE_URL,
},
},
});
const mp3Bytes = Buffer.from([0xff, 0xfb, 0x90, 0x00]);
const fetchMock = vi.fn(async (_input: unknown, init?: RequestInit) => {
const body = JSON.parse(String(init?.body));
expect(body.voice_setting.voice_id).toBe('male_0001_a');
return buildOkResponse(mp3Bytes);
});
vi.stubGlobal('fetch', fetchMock);
const result = await generateMedia({
projectRoot,
projectsRoot,
projectId: 'project-1',
surface: 'audio',
model: 'senseaudio-tts',
audioKind: 'speech',
voice: 'male_0001_a',
prompt: 'Custom voice line.',
output: 'senseaudio-custom.mp3',
});
expect(result.providerNote).toContain('male_0001_a');
});
it('falls back to the canonical base URL when none is configured', async () => {
await writeConfig({
providers: {
senseaudio: { apiKey: 'sense-test-key' },
},
});
const fetchMock = vi.fn(async (input: unknown) => {
expect(String(input)).toBe('https://api.senseaudio.cn/v1/t2a_v2');
return buildOkResponse(Buffer.from([0x01, 0x02, 0x03]));
});
vi.stubGlobal('fetch', fetchMock);
await generateMedia({
projectRoot,
projectsRoot,
projectId: 'project-1',
surface: 'audio',
model: 'senseaudio-tts',
audioKind: 'speech',
prompt: 'Default base url.',
output: 'senseaudio-default-base.mp3',
});
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it('reads the API key from OD_SENSEAUDIO_API_KEY when storage is empty', async () => {
process.env.OD_SENSEAUDIO_API_KEY = 'env-sense-key';
const fetchMock = vi.fn(async (_input: unknown, init?: RequestInit) => {
expect(init?.headers).toMatchObject({
authorization: 'Bearer env-sense-key',
});
return buildOkResponse(Buffer.from([0x10, 0x20]));
});
vi.stubGlobal('fetch', fetchMock);
await generateMedia({
projectRoot,
projectsRoot,
projectId: 'project-1',
surface: 'audio',
model: 'senseaudio-tts',
audioKind: 'speech',
prompt: 'Env-only key.',
output: 'senseaudio-env.mp3',
});
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it('forwards requestInit.dispatcher through SenseAudio image submit and download fetches', async () => {
await writeConfig({
providers: {
senseaudio: {
apiKey: 'sense-test-key',
baseUrl: TEST_SENSEAUDIO_BASE_URL,
},
},
});
const dispatcher = {} as NonNullable<RequestInit['dispatcher']>;
const fetchMock = vi.fn(async (input: unknown, init?: RequestInit) => {
if (String(input) === `${TEST_SENSEAUDIO_BASE_URL}/v1/image/sync`) {
expect(init?.dispatcher).toBe(dispatcher);
return new Response(JSON.stringify({
url: 'https://93.184.216.34/senseaudio-image.png',
base_resp: { status_code: 0, status_msg: 'success' },
}), {
status: 200,
headers: { 'content-type': 'application/json' },
});
}
expect(String(input)).toBe('https://93.184.216.34/senseaudio-image.png');
// Asset download goes through assertAndFetchExternalAsset which replaces
// the caller's dispatcher with a validating one (issue #5478).
expect(init?.dispatcher).toBeDefined();
expect(init?.redirect).toBe('error');
return new Response(Buffer.from([0x89, 0x50, 0x4e, 0x47]));
});
vi.stubGlobal('fetch', fetchMock);
const result = await generateMedia({
projectRoot,
projectsRoot,
projectId: 'project-1',
surface: 'image',
model: 'senseaudio-image-2.0-260319',
prompt: 'A reference render.',
output: 'senseaudio-image.png',
requestInit: { dispatcher },
});
expect(result.providerId).toBe('senseaudio');
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it('errors when no API key is configured', async () => {
const fetchMock = vi.fn();
vi.stubGlobal('fetch', fetchMock);
await expect(
generateMedia({
projectRoot,
projectsRoot,
projectId: 'project-1',
surface: 'audio',
model: 'senseaudio-tts',
audioKind: 'speech',
prompt: 'Should fail.',
output: 'senseaudio-no-key.mp3',
}),
).rejects.toThrow(/no SenseAudio API key/);
expect(fetchMock).not.toHaveBeenCalled();
});
it('surfaces upstream base_resp failures verbatim', async () => {
await writeConfig({
providers: {
senseaudio: { apiKey: 'sense-test-key', baseUrl: TEST_SENSEAUDIO_BASE_URL },
},
});
const fetchMock = vi.fn(async () =>
new Response(
JSON.stringify({
data: null,
base_resp: { status_code: 1004, status_msg: 'voice_id not found' },
}),
{ status: 200, headers: { 'content-type': 'application/json' } },
),
);
vi.stubGlobal('fetch', fetchMock);
await expect(
generateMedia({
projectRoot,
projectsRoot,
projectId: 'project-1',
surface: 'audio',
model: 'senseaudio-tts',
audioKind: 'speech',
voice: 'does-not-exist',
prompt: 'Bad voice.',
output: 'senseaudio-bad-voice.mp3',
}),
).rejects.toThrow('senseaudio tts api error 1004: voice_id not found');
});
it('errors when the audio payload is missing', async () => {
await writeConfig({
providers: {
senseaudio: { apiKey: 'sense-test-key', baseUrl: TEST_SENSEAUDIO_BASE_URL },
},
});
const fetchMock = vi.fn(async () =>
new Response(
JSON.stringify({
data: {},
base_resp: { status_code: 0, status_msg: 'success' },
}),
{ status: 200, headers: { 'content-type': 'application/json' } },
),
);
vi.stubGlobal('fetch', fetchMock);
await expect(
generateMedia({
projectRoot,
projectsRoot,
projectId: 'project-1',
surface: 'audio',
model: 'senseaudio-tts',
audioKind: 'speech',
prompt: 'Missing audio.',
output: 'senseaudio-missing-audio.mp3',
}),
).rejects.toThrow('senseaudio tts response missing data.audio');
});
it('surfaces HTTP-level failures with the status code and truncated body', async () => {
await writeConfig({
providers: {
senseaudio: { apiKey: 'sense-test-key', baseUrl: TEST_SENSEAUDIO_BASE_URL },
},
});
const fetchMock = vi.fn(async () =>
new Response('unauthorized', {
status: 401,
headers: { 'content-type': 'text/plain' },
}),
);
vi.stubGlobal('fetch', fetchMock);
await expect(
generateMedia({
projectRoot,
projectsRoot,
projectId: 'project-1',
surface: 'audio',
model: 'senseaudio-tts',
audioKind: 'speech',
prompt: 'Bad auth.',
output: 'senseaudio-401.mp3',
}),
).rejects.toThrow('senseaudio tts 401: unauthorized');
});
});