mirror of
https://github.com/debba/wp-media-rewind.git
synced 2026-08-03 07:29:00 +02:00
feat: initial project setup with CLI, utils, and tests
This commit is contained in:
Andrea Debernardi
@@ -0,0 +1,125 @@
|
||||
# wp-media-rewind
|
||||
|
||||
A small CLI for the situation nobody wants to be in: you have a WordPress SQL
|
||||
dump, the original site is gone, and the `wp-content/uploads` folder vanished
|
||||
with it. `wp-media-rewind` reads the dump, pulls every media URL it can find, and
|
||||
tries to fetch each file back from the Wayback Machine — keeping the original
|
||||
folder structure on disk.
|
||||
|
||||
It's deliberately not clever. No database, no headless browser, no scraping
|
||||
the live site. Just: parse the dump, ask `archive.org/wayback/available` for
|
||||
the closest snapshot, download the raw bytes.
|
||||
|
||||
## What it does
|
||||
|
||||
- Streams a `.sql` dump line by line, so 2 GB exports don't blow up memory.
|
||||
- Pulls media URLs from two places:
|
||||
- any URL with a media-looking extension found anywhere in the file
|
||||
(covers stuff embedded in `post_content`, options, postmeta, etc.),
|
||||
- `guid` values from `wp_posts` rows where `post_type='attachment'`.
|
||||
- For each URL, asks the Wayback availability API for the closest snapshot
|
||||
(optionally near a timestamp you pick), then downloads the raw asset using
|
||||
the `id_` flag so you get original bytes, not a rewritten archive page.
|
||||
- Writes files under `<output>/<host>/<original-path>` by default, so a URL
|
||||
like `https://example.com/wp-content/uploads/2020/01/photo.jpg` lands in
|
||||
`recovered/example.com/wp-content/uploads/2020/01/photo.jpg`.
|
||||
- Skips files that already exist on disk, so re-runs are cheap.
|
||||
- Shows a progress bar while it works and prints a summary at the end with
|
||||
counts, total bytes, elapsed time, and the first failures.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
## Use
|
||||
|
||||
```bash
|
||||
# Positional argument
|
||||
node dist/index.js dump.sql --site example.com --output ./recovered
|
||||
|
||||
# Or with the explicit flag
|
||||
node dist/index.js --sql ./dump.sql --site example.com --output ./recovered
|
||||
|
||||
# See what would be fetched without downloading anything
|
||||
node dist/index.js dump.sql --site example.com --dry-run
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
| Flag | Default | What it does |
|
||||
| --- | --- | --- |
|
||||
| `<sql-dump>` / `-f, --sql <file>` | — | Path to the WordPress SQL dump. |
|
||||
| `-o, --output <dir>` | `./recovered` | Where to write recovered files. |
|
||||
| `-s, --site <host>` | — | Only keep URLs on this host (recommended; dumps often quote URLs from other sites in `post_content`). |
|
||||
| `-t, --timestamp <YYYYMMDD>` | — | Prefer snapshots near this date. Useful if you remember roughly when the site was last alive. |
|
||||
| `-c, --concurrency <n>` | `4` | Parallel downloads. Be polite to archive.org. |
|
||||
| `--dry-run` | off | Print URLs and exit. |
|
||||
| `--manifest <file>` | — | Append a JSONL row per URL with status, snapshot, bytes, etc. |
|
||||
| `--no-host-prefix` | off | Drop the `<host>/` directory and write paths starting at the site root. |
|
||||
|
||||
### Final report
|
||||
|
||||
When the run finishes you get something like:
|
||||
|
||||
```
|
||||
─── Recovery summary ─────────────────────────
|
||||
URLs found in dump : 312 (118 attachment GUIDs)
|
||||
Processed : 312
|
||||
✓ Downloaded : 287
|
||||
↷ Skipped (exists) : 0
|
||||
✗ Failed : 25
|
||||
Success rate : 91%
|
||||
Total downloaded : 184.21 MB
|
||||
Elapsed : 4m12s
|
||||
Output directory : ./recovered
|
||||
Manifest (JSONL) : ./run.jsonl
|
||||
|
||||
Failures:
|
||||
- https://example.com/wp-content/uploads/2014/03/old.jpg (no_snapshot)
|
||||
…
|
||||
──────────────────────────────────────────────
|
||||
```
|
||||
|
||||
The exit code is non-zero if any URL failed, so you can wire it into a
|
||||
script.
|
||||
|
||||
## Notes & caveats
|
||||
|
||||
- The Wayback Machine doesn't have everything. Older or low-traffic files are
|
||||
the most common misses. Try a different `--timestamp` if a whole period
|
||||
looks empty.
|
||||
- The tool requests one availability lookup per URL. If you're recovering
|
||||
thousands of files, run with low concurrency and expect it to take a while.
|
||||
- It only restores binaries (images, PDFs, video, etc.). Posts, comments and
|
||||
other DB content are already in the dump — you don't need this for those.
|
||||
- Filenames with characters that are illegal on your filesystem get replaced
|
||||
with `_`. Path traversal segments are stripped.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
npm test # run unit tests (node:test, no extra deps)
|
||||
npm run typecheck # strict tsc --noEmit on src + tests
|
||||
npm run dev # tsx src/index.ts <args>
|
||||
```
|
||||
|
||||
The pure helpers live in `src/utils/` and have unit tests in `tests/`:
|
||||
|
||||
- `utils/url.ts` — host normalization and URL cleanup.
|
||||
- `utils/sql.ts` — splitting `INSERT VALUES (...)` tuples and pulling
|
||||
attachment GUIDs.
|
||||
- `utils/media.ts` — building the media-extension regex.
|
||||
- `utils/wayback-url.ts` — converting snapshot URLs to the raw `id_` form
|
||||
and mapping URLs to output paths.
|
||||
- `utils/progress.ts` — terminal progress bar.
|
||||
- `utils/summary.ts` — final report formatting.
|
||||
|
||||
The HTTP code (`wayback.ts`) and the SQL streaming reader (`parser.ts`) are
|
||||
intentionally thin wrappers around those utilities.
|
||||
|
||||
## License
|
||||
|
||||
MIT.
|
||||
Reference in New Issue
Block a user