Contributing to TRAWL

Thanks for your interest in contributing! TRAWL is a self-hosted web scraping engine released under AGPL-3.0. By submitting a contribution, you agree to license your work under the same terms.

Code of Conduct

This project follows the Contributor Covenant. Be respectful, assume good faith, and focus on the technical merits.

Reporting bugs & requesting features

Please use the GitHub issue templates — they ensure we have the context we need:

Bug reports: .github/ISSUE_TEMPLATE/bug_report.md
Feature requests: .github/ISSUE_TEMPLATE/feature_request.md

For security issues, do not open a public issue — see SECURITY.md.

Development setup

Requirements: Bun 1.3.14 and Docker (for the Redis service used in tests).

git clone https://github.com/germondai/trawl.git
cd trawl
bun install
cp .env.example .env

Running the apps

bun run dev:api     # Elysia API on :8191
bun run dev:web     # Nuxt 4 landing page
bun run dev:docs    # VitePress docs site

The API requires Redis. The fastest way is docker compose up -d redis.

Linting & formatting

We use Biome for both:

bun run check       # read-only format, lint, and import-order check
bun run fix         # apply safe Biome fixes and formatting
bun run typecheck   # typecheck all five TypeScript workspaces
bun run build       # production-build the web and docs apps
bun run verify      # full release gate: check, types, tests, and builds

CI runs bun run verify on every PR.

Project layout

This is a Bun monorepo with workspaces:

apps/
  api/      Elysia API (the scraper service)
  web/      Nuxt 4 landing page
  docs/     VitePress documentation
packages/
  browser/  Camoufox Firefox pool
  tiers/    Tier 1–4 execution engine
  types/    Shared TypeScript types

Apps are independently deployable; packages/* are imported via the workspace protocol (e.g. workspace:*).

Commit conventions

We use Conventional Commits. Recent examples:

ci(publish): build images for linux/amd64, linux/arm64, linux/arm/v7
chore: add .gitignore files for api, docs, web, and browser packages
fix(browser): restore pool after worker crash

The type is one of feat, fix, chore, docs, ci, refactor, test, perf. Keep the subject under 72 chars and in the imperative mood.

Pull request process

Open an issue first for non-trivial changes. A two-paragraph problem statement is enough.
Branch from main. Use a descriptive name (feat/captcha-hcaptcha, fix/redis-reconnect).
Run bun run verify before pushing. Lint, types, tests, and production builds must be clean.
Update CHANGELOG.md under ## [Unreleased] for any user-visible change.
Fill out the PR template — the checklist catches the easy-to-miss items.
Keep PRs focused. One feature or fix per PR; large refactors should be split.

Adding a new tier or solver

TRAWL's design centers on a 4-tier escalation ladder (HTTP → cached session → fresh CF solve → residential proxy). If your contribution introduces a new tier or a new solver:

Put tier logic in packages/tiers/.
Put browser/solver adapters in packages/browser/.
Update the tier diagram in README.md.
Add an entry to CHANGELOG.md.

License

TRAWL is licensed under AGPL-3.0. By submitting a pull request, you affirm that your contribution is your own work and you agree to license it under AGPL-3.0. AGPL is more restrictive than MIT/Apache — if your employer might claim ownership of your work, get explicit approval first.

3.8 KiB Raw Permalink Blame History Unescape Escape