wp-media-rewind/README.md

# wp-media-rewind

A small CLI for the situation nobody wants to be in: you have a WordPress SQL
dump, the original site is gone, and the `wp-content/uploads` folder vanished
with it. `wp-media-rewind` reads the dump, pulls every media URL it can find, and
tries to fetch each file back from the Wayback Machine — keeping the original
folder structure on disk.

It's deliberately not clever. No database, no headless browser, no scraping
the live site. Just: parse the dump, ask `archive.org/wayback/available` for
the closest snapshot, download the raw bytes.

## What it does

- Streams a `.sql` dump line by line, so 2 GB exports don't blow up memory.
- Pulls media URLs from two places:
  - any URL with a media-looking extension found anywhere in the file
    (covers stuff embedded in `post_content`, options, postmeta, etc.),
  - `guid` values from `wp_posts` rows where `post_type='attachment'`.
- For each URL, asks the Wayback availability API for the closest snapshot
  (optionally near a timestamp you pick), then downloads the raw asset using
  the `id_` flag so you get original bytes, not a rewritten archive page.
- Writes files under `<output>/<host>/<original-path>` by default, so a URL
  like `https://example.com/wp-content/uploads/2020/01/photo.jpg` lands in
  `recovered/example.com/wp-content/uploads/2020/01/photo.jpg`.
- Skips files that already exist on disk, so re-runs are cheap.
- Shows a progress bar while it works and prints a summary at the end with
  counts, total bytes, elapsed time, and the first failures.

## Install

```bash
npm install
npm run build
```

## Use

```bash
# Positional argument
node dist/index.js dump.sql --site example.com --output ./recovered

# Or with the explicit flag
node dist/index.js --sql ./dump.sql --site example.com --output ./recovered

# See what would be fetched without downloading anything
node dist/index.js dump.sql --site example.com --dry-run
```

### Options

| Flag | Default | What it does |
| --- | --- | --- |
| `<sql-dump>` / `-f, --sql <file>` | — | Path to the WordPress SQL dump. |
| `-o, --output <dir>` | `./recovered` | Where to write recovered files. |
| `-s, --site <host>` | — | Only keep URLs on this host (recommended; dumps often quote URLs from other sites in `post_content`). |
| `-t, --timestamp <YYYYMMDD>` | — | Prefer snapshots near this date. Useful if you remember roughly when the site was last alive. |
| `-c, --concurrency <n>` | `4` | Parallel downloads. Be polite to archive.org. |
| `--dry-run` | off | Print URLs and exit. |
| `--manifest <file>` | — | Append a JSONL row per URL with status, snapshot, bytes, etc. |
| `--no-host-prefix` | off | Drop the `<host>/` directory and write paths starting at the site root. |

### Final report

When the run finishes you get something like:

```
─── Recovery summary ─────────────────────────
  URLs found in dump : 312 (118 attachment GUIDs)
  Processed          : 312
  ✓ Downloaded       : 287
  ↷ Skipped (exists) : 0
  ✗ Failed           : 25
  Success rate       : 91%
  Total downloaded   : 184.21 MB
  Elapsed            : 4m12s
  Output directory   : ./recovered
  Manifest (JSONL)   : ./run.jsonl

  Failures:
    - https://example.com/wp-content/uploads/2014/03/old.jpg (no_snapshot)
    …
──────────────────────────────────────────────
```

The exit code is non-zero if any URL failed, so you can wire it into a
script.

## Notes & caveats

- The Wayback Machine doesn't have everything. Older or low-traffic files are
  the most common misses. Try a different `--timestamp` if a whole period
  looks empty.
- The tool requests one availability lookup per URL. If you're recovering
  thousands of files, run with low concurrency and expect it to take a while.
- It only restores binaries (images, PDFs, video, etc.). Posts, comments and
  other DB content are already in the dump — you don't need this for those.
- Filenames with characters that are illegal on your filesystem get replaced
  with `_`. Path traversal segments are stripped.

## Development

```bash
npm test          # run unit tests (node:test, no extra deps)
npm run typecheck # strict tsc --noEmit on src + tests
npm run dev       # tsx src/index.ts <args>
```

The pure helpers live in `src/utils/` and have unit tests in `tests/`:

- `utils/url.ts` — host normalization and URL cleanup.
- `utils/sql.ts` — splitting `INSERT VALUES (...)` tuples and pulling
  attachment GUIDs.
- `utils/media.ts` — building the media-extension regex.
- `utils/wayback-url.ts` — converting snapshot URLs to the raw `id_` form
  and mapping URLs to output paths.
- `utils/progress.ts` — terminal progress bar.
- `utils/summary.ts` — final report formatting.

The HTTP code (`wayback.ts`) and the SQL streaming reader (`parser.ts`) are
intentionally thin wrappers around those utilities.

## License

MIT.