mirror of
https://github.com/germondai/trawl.git
synced 2026-08-17 12:11:23 +02:00
103 lines
3.8 KiB
Markdown
103 lines
3.8 KiB
Markdown
# Contributing to TRAWL
|
||
|
||
Thanks for your interest in contributing! TRAWL is a self-hosted web scraping engine released under [AGPL-3.0](LICENSE). By submitting a contribution, you agree to license your work under the same terms.
|
||
|
||
## Code of Conduct
|
||
|
||
This project follows the [Contributor Covenant](CODE_OF_CONDUCT.md). Be respectful, assume good faith, and focus on the technical merits.
|
||
|
||
## Reporting bugs & requesting features
|
||
|
||
Please use the GitHub issue templates — they ensure we have the context we need:
|
||
|
||
- **Bug reports:** [`.github/ISSUE_TEMPLATE/bug_report.md`](.github/ISSUE_TEMPLATE/bug_report.md)
|
||
- **Feature requests:** [`.github/ISSUE_TEMPLATE/feature_request.md`](.github/ISSUE_TEMPLATE/feature_request.md)
|
||
|
||
For security issues, **do not open a public issue** — see [SECURITY.md](SECURITY.md).
|
||
|
||
## Development setup
|
||
|
||
Requirements: **Bun 1.3.14** and **Docker** (for the Redis service used in tests).
|
||
|
||
```bash
|
||
git clone https://github.com/germondai/trawl.git
|
||
cd trawl
|
||
bun install
|
||
cp .env.example .env
|
||
```
|
||
|
||
### Running the apps
|
||
|
||
```bash
|
||
bun run dev:api # Elysia API on :8191
|
||
bun run dev:web # Nuxt 4 landing page
|
||
bun run dev:docs # VitePress docs site
|
||
```
|
||
|
||
The API requires Redis. The fastest way is `docker compose up -d redis`.
|
||
|
||
### Linting & formatting
|
||
|
||
We use [Biome](https://biomejs.dev/) for both:
|
||
|
||
```bash
|
||
bun run check # read-only format, lint, and import-order check
|
||
bun run fix # apply safe Biome fixes and formatting
|
||
bun run typecheck # typecheck all five TypeScript workspaces
|
||
bun run build # production-build the web and docs apps
|
||
bun run verify # full release gate: check, types, tests, and builds
|
||
```
|
||
|
||
CI runs `bun run verify` on every PR.
|
||
|
||
## Project layout
|
||
|
||
This is a Bun monorepo with workspaces:
|
||
|
||
```
|
||
apps/
|
||
api/ Elysia API (the scraper service)
|
||
web/ Nuxt 4 landing page
|
||
docs/ VitePress documentation
|
||
packages/
|
||
browser/ Camoufox Firefox pool
|
||
tiers/ Tier 1–4 execution engine
|
||
types/ Shared TypeScript types
|
||
```
|
||
|
||
Apps are independently deployable; `packages/*` are imported via the workspace protocol (e.g. `workspace:*`).
|
||
|
||
## Commit conventions
|
||
|
||
We use [Conventional Commits](https://www.conventionalcommits.org/). Recent examples:
|
||
|
||
```
|
||
ci(publish): build images for linux/amd64, linux/arm64, linux/arm/v7
|
||
chore: add .gitignore files for api, docs, web, and browser packages
|
||
fix(browser): restore pool after worker crash
|
||
```
|
||
|
||
The `type` is one of `feat`, `fix`, `chore`, `docs`, `ci`, `refactor`, `test`, `perf`. Keep the subject under 72 chars and in the imperative mood.
|
||
|
||
## Pull request process
|
||
|
||
1. **Open an issue first** for non-trivial changes. A two-paragraph problem statement is enough.
|
||
2. **Branch from `main`.** Use a descriptive name (`feat/captcha-hcaptcha`, `fix/redis-reconnect`).
|
||
3. **Run `bun run verify` before pushing.** Lint, types, tests, and production builds must be clean.
|
||
4. **Update `CHANGELOG.md`** under `## [Unreleased]` for any user-visible change.
|
||
5. **Fill out the PR template** — the checklist catches the easy-to-miss items.
|
||
6. **Keep PRs focused.** One feature or fix per PR; large refactors should be split.
|
||
|
||
## Adding a new tier or solver
|
||
|
||
TRAWL's design centers on a 4-tier escalation ladder (HTTP → cached session → fresh CF solve → residential proxy). If your contribution introduces a new tier or a new solver:
|
||
|
||
- Put tier logic in `packages/tiers/`.
|
||
- Put browser/solver adapters in `packages/browser/`.
|
||
- Update the tier diagram in `README.md`.
|
||
- Add an entry to `CHANGELOG.md`.
|
||
|
||
## License
|
||
|
||
TRAWL is licensed under **AGPL-3.0**. By submitting a pull request, you affirm that your contribution is your own work and you agree to license it under AGPL-3.0. AGPL is more restrictive than MIT/Apache — if your employer might claim ownership of your work, get explicit approval first.
|