Compare commits

...

24 Commits

Author SHA1 Message Date
f7d0dfdfb6 feat: UI polish — AI enhancements branding, footer year, admin background fix
- Header: add "+ AI enhancements" italic subtitle below logo
- Footer: update copyright year to 2021–current, add "AI enhancements by Cloud Host" link
- Admin page: wrap all states in pageBg Box so dark-mode background shows correctly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 11:21:34 +01:00
171b40f525 fix: serialise local Cog model calls with a queue to prevent 409 conflicts 2026-03-10 10:39:06 +01:00
41d2aa7295 fix: respect JigsawStack 5000-char limit with proper batching
- buildBatches: groups texts into chunks that fit within 4800 chars
  (200 char safety margin) when joined with the separator
- translateLongText: splits individual cells/paragraphs that exceed
  the limit at paragraph/sentence boundaries, translates each chunk,
  then rejoins — instead of hitting the API with oversized input
- Process batches sequentially to avoid overloading the local model
- Separator fallback still works: if separator gets translated,
  falls back to individual calls per text within that batch

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 10:26:22 +01:00
38c3e3e2cb fix: chown entire /app to nextjs before build so .next dir can be created 2026-03-10 10:03:55 +01:00
b0f7f30f17 fix: replace Progress component with plain Box bar + use npm ci in Docker
- DocumentTranslator: replace Chakra Progress (broken types in 2.2.1
  with fresh installs) with a simple Box-based progress bar — no type
  issues, same visual result
- Dockerfile: switch from npm install to npm ci so Docker uses exact
  locked versions from package-lock.json, preventing type discrepancies
  between local and Docker builds

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 10:00:04 +01:00
c795ef5a54 fix: replace colorScheme on Progress with sx workaround for Chakra 2.2.1 type strictness 2026-03-10 09:52:48 +01:00
a435e8c749 fix: add explicit types to all event handler params in DocumentTranslator 2026-03-10 09:45:51 +01:00
7a30d14f0f fix: skip Cypress binary download in Docker build (CYPRESS_INSTALL_BINARY=0) 2026-03-10 09:42:48 +01:00
2354ddf9be fix: add explicit React.DragEvent type on onDragOver handler 2026-03-10 09:38:06 +01:00
466459373a fix: move next build to image build time, drop yarn, add runner stage
- Build Next.js app during docker build (not on container start) —
  fixes yarn cache permission errors and makes container startup instant
- Remove yarn.lock during build so nothing can accidentally invoke yarn
- Add lean runner stage: copies only .next, node_modules, public —
  reduces final image size
- npm start is now the only thing that runs at container start

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 09:18:23 +01:00
8f98e54b18 fix: explicit subnet for docker-compose to avoid pool exhaustion
Assign a fixed subnet (10.100.50.0/24) to lingvai-net so Docker
doesn't try to auto-allocate from its default address pool, which
fails with 'all predefined address pools have been fully subnetted'
on hosts running many containers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 09:13:32 +01:00
a7ca88cabb fix: docker-compose networking and local translator endpoint
- Replace per-service network_mode:bridge with a shared lingvai-net
  so lingvai-app can reach lingvai-translator by hostname
- Set LOCAL_MODEL_ENDPOINT=http://lingvai-translator:5000/predictions
  and REPLICATE_MODE=local via environment so admin panel defaults
  are pre-configured for the local container
- Add depends_on: translator so the app starts after the model is up
- Host port 5030->5000 kept for direct debugging access

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 09:10:54 +01:00
0be8b0b0f0 fix: resolve 422 from Replicate + add local Cog Docker mode
- replicate-translate: parse owner/model:hash correctly — extract only
  the hash portion for the version field, and use the model endpoint
  (POST /v1/models/{owner}/{model}/predictions) which avoids 422
  'Invalid version' errors when sending the full owner/model:hash string.

- Add local Cog mode: when replicateMode="local", calls the local Docker
  container directly (no Replicate API key needed), default endpoint
  http://localhost:5030/predictions (host port 5030 → container port 5000).

- settings-store: add replicateMode ("cloud"|"local") and localEndpoint
  fields with env var fallbacks REPLICATE_MODE and LOCAL_MODEL_ENDPOINT.

- admin panel: Radio selector for Cloud vs Local mode; shows docker run
  command snippet and local endpoint URL field when local is selected;
  hides Replicate API token field in local mode (not needed).

Local model startup:
  docker run -d -p 5030:5000 \
    r8.im/jigsawstack/text-translate@sha256:454df4c...

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 08:50:40 +01:00
e034771087 fix: handle EACCES on data/settings.json in Docker containers
- settings-store: auto-detect writable path at startup — tries
  <cwd>/data/settings.json first, falls back to /tmp/lingvai-settings.json
  if the directory is not writable. Logs a warning when fallback is used.
  Also supports SETTINGS_PATH env var for explicit override.

- Dockerfile: switch from yarn to npm, explicitly create /app/data with
  chown nextjs:nodejs so the directory is writable at runtime without
  needing a privileged volume mount.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 08:23:29 +01:00
0799101da3 feat: add admin panel, Replicate AI translation, and document translation
- Admin panel (/admin) with JWT auth: configure Replicate API token,
  JigsawStack API key, model version, enable/disable AI translation,
  change admin password. Settings persisted in data/settings.json.

- Replicate AI translation: POST /api/translate/replicate uses
  JigsawStack text-translate model via Replicate API. Main page
  switches to client-side AI translation when enabled.

- Document translation tab: supports PDF, DOCX, XLSX, XLS, CSV.
  Excel/Word formatting fully preserved (SheetJS + JSZip XML manipulation).
  PDF uses pdf-parse extraction + pdf-lib reconstruction.
  Column selector UI for tabular data (per-sheet, All/None toggles).

- Updated README with full implementation documentation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 07:43:54 +01:00
David
0190ea5da9 Added instance lingva.lunar.icu [skip ci] (#147) 2023-01-13 17:53:04 +01:00
igna
bf9471ba56 Instances cleanup [skip ci] (#142) 2022-11-02 16:15:27 +01:00
David
d2d405c3e5 Added editorconfig [skip ci] (#141) 2022-10-25 01:10:16 +02:00
David
81fd11ee62 Fix tests typings (#140) 2022-10-21 15:57:53 +02:00
David
6e254e66a0 Update scraper languages & add and remove instances (#138) 2022-10-09 21:36:33 +02:00
David
af24caf962 Added Docker healthcheck and projectsegfau instance (#131) 2022-07-26 14:30:58 +02:00
Nico
5c3abdf4b6 Add Garudalinux' & dr460nf1r3's instances (#129)
* Add Garudalinux' & dr460nf1r3's instance

* [skip ci]

* Add instances to instances.json  [skip ci]
2022-07-26 13:34:46 +02:00
David
1ad5027412 Added instance lingva.opnxng.com [skip ci] (#125) 2022-06-29 00:16:17 +02:00
David
443e98146b Added instance translate.plausibility.cloud (#122) 2022-06-26 21:03:23 +02:00
34 changed files with 18973 additions and 319 deletions

View File

@@ -53,5 +53,8 @@ cypress/screenshots
Dockerfile Dockerfile
.dockerignore .dockerignore
# editorconfig
.editorconfig
# instance list # instance list
instances.json instances.json

13
.editorconfig Normal file
View File

@@ -0,0 +1,13 @@
root = true
[*]
indent_style = space
charset = utf-8
end_of_line = lf
insert_final_newline = true
[*.{js,ts,tsx}]
indent_size = 4
[*.{json,yml}]
indent_size = 2

3
.gitignore vendored
View File

@@ -45,3 +45,6 @@ cypress/screenshots
**/public/workbox-*.js* **/public/workbox-*.js*
**/public/sw.js* **/public/sw.js*
**/public/worker-*.js* **/public/worker-*.js*
# admin settings (contains API keys)
/data/settings.json

View File

@@ -1,30 +1,63 @@
# https://nextjs.org/docs/deployment#docker-image # ── Stage 1: install dependencies ──────────────────────────────────────────
FROM node:lts-alpine AS deps FROM node:lts-alpine AS deps
RUN apk add --no-cache libc6-compat RUN apk add --no-cache libc6-compat
WORKDIR /app WORKDIR /app
COPY package.json yarn.lock ./ COPY package.json package-lock.json* ./
RUN yarn install --frozen-lockfile # Skip Cypress binary download — not needed in production
ENV CYPRESS_INSTALL_BINARY=0
# Use ci to install exact versions from package-lock.json
RUN npm ci --legacy-peer-deps
# ── Stage 2: build the Next.js app ──────────────────────────────────────────
FROM node:lts-alpine AS builder FROM node:lts-alpine AS builder
WORKDIR /app WORKDIR /app
RUN addgroup -g 1001 -S nodejs # Create non-root user
RUN adduser -S nextjs -u 1001 RUN addgroup -g 1001 -S nodejs && \
adduser -S nextjs -u 1001
# Copy source and deps
COPY --chown=nextjs:nodejs . . COPY --chown=nextjs:nodejs . .
COPY --from=deps /app/node_modules ./node_modules COPY --from=deps --chown=nextjs:nodejs /app/node_modules ./node_modules
RUN chown nextjs:nodejs .
# Remove yarn.lock so nothing accidentally invokes yarn
RUN rm -f yarn.lock
# Give nextjs user ownership of the entire workdir (WORKDIR creates it as root)
RUN chown -R nextjs:nodejs /app
# Build the app at image build time (not at container start)
USER nextjs
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
RUN npm run build
# ── Stage 3: production runner ───────────────────────────────────────────────
FROM node:lts-alpine AS runner
RUN apk add --no-cache curl
WORKDIR /app
RUN addgroup -g 1001 -S nodejs && \
adduser -S nextjs -u 1001
# Copy only what's needed to run
COPY --from=builder --chown=nextjs:nodejs /app/public ./public
COPY --from=builder --chown=nextjs:nodejs /app/.next ./.next
COPY --from=builder --chown=nextjs:nodejs /app/node_modules ./node_modules
COPY --from=builder --chown=nextjs:nodejs /app/package.json ./package.json
COPY --from=builder --chown=nextjs:nodejs /app/next.config.js ./next.config.js
RUN mkdir -p /app/data && chown -R nextjs:nodejs /app/data && chmod 755 /app/data
USER nextjs USER nextjs
EXPOSE 3000 EXPOSE 3000
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
ENV NODE_ENV production HEALTHCHECK --interval=1m --timeout=3s CMD curl -f http://localhost:3000/ || exit 1
ENV NEXT_TELEMETRY_DISABLED 1 CMD NEXT_PUBLIC_SITE_DOMAIN=$site_domain \
CMD NEXT_PUBLIC_SITE_DOMAIN=$site_domain\
NEXT_PUBLIC_FORCE_DEFAULT_THEME=$force_default_theme \ NEXT_PUBLIC_FORCE_DEFAULT_THEME=$force_default_theme \
NEXT_PUBLIC_DEFAULT_SOURCE_LANG=$default_source_lang \ NEXT_PUBLIC_DEFAULT_SOURCE_LANG=$default_source_lang \
NEXT_PUBLIC_DEFAULT_TARGET_LANG=$default_target_lang \ NEXT_PUBLIC_DEFAULT_TARGET_LANG=$default_target_lang \
yarn build && yarn start npm start

392
README.md
View File

@@ -1,236 +1,256 @@
# Lingva Translate # LingvAI
<img src="public/logo.svg" width="128" align="right"> <img src="public/logo.svg" width="128" align="right">
[![Travis Build](https://travis-ci.com/thedaviddelta/lingva-translate.svg?branch=main)](https://travis-ci.com/thedaviddelta/lingva-translate)
[![Vercel Status](https://img.shields.io/github/deployments/thedaviddelta/lingva-translate/Production?label=vercel&logo=vercel&color=f5f5f5)](https://lingva.ml/)
[![Cypress Tests](https://img.shields.io/endpoint?url=https://dashboard.cypress.io/badge/simple/qgjdyd&style=flat&logo=cypress)](https://dashboard.cypress.io/projects/qgjdyd/runs)
[![License](https://img.shields.io/github/license/thedaviddelta/lingva-translate)](./LICENSE) [![License](https://img.shields.io/github/license/thedaviddelta/lingva-translate)](./LICENSE)
[![Awesome Humane Tech](https://raw.githubusercontent.com/humanetech-community/awesome-humane-tech/main/humane-tech-badge.svg?sanitize=true)](https://github.com/humanetech-community/awesome-humane-tech)
[<img src="https://www.datocms-assets.com/31049/1618983297-powered-by-vercel.svg" alt="Powered by Vercel" height="20">](https://vercel.com?utm_source=lingva-team&utm_campaign=oss)
Alternative front-end for Google Translate, serving as a Free and Open Source translator with over a hundred languages available **LingvAI** is an enhanced fork of [Lingva Translate](https://github.com/thedaviddelta/lingva-translate) — a privacy-respecting alternative front-end for Google Translate — extended with **Replicate AI-powered translation**, an **admin panel**, and **document translation** (PDF, Word, Excel, CSV) with full formatting preservation.
---
## How does it work? ## Features
Inspired by projects like [NewPipe](https://github.com/TeamNewPipe/NewPipe), [Nitter](https://github.com/zedeus/nitter), [Invidious](https://github.com/iv-org/invidious) or [Bibliogram](https://git.sr.ht/~cadence/bibliogram), *Lingva* scrapes through Google Translate and retrieves the translation without directly accessing any Google-related service, preventing them from tracking. ### Core (from Lingva Translate)
- 100+ languages via Google Translate scraper (no tracking)
- Audio playback for source and translated text
- Auto-translate mode
- GraphQL and REST API
- PWA support (installable)
- Dark/light mode
For this purpose, *Lingva* is built, among others, with the following Open Source resources: ### New in LingvAI
+ [Lingva Scraper](https://github.com/thedaviddelta/lingva-scraper), a Google Translate scraper built and maintained specifically for this project, which obtains all kind of information from this platform. #### Admin Panel (`/admin`)
+ [TypeScript](https://www.typescriptlang.org/), the JavaScript superset, as the language. - Password-protected settings dashboard (gear icon in header)
+ [React](https://reactjs.org/) as the main front-end framework. - Configure Replicate API token and JigsawStack API key
+ [Next.js](https://nextjs.org/) as the complementary React framework, that provides Server-Side Rendering, Static Site Generation or serverless API endpoints. - Select/change the Replicate model version
+ [ChakraUI](https://chakra-ui.com/) for the in-component styling. - Enable/disable AI translation per-instance
+ [Jest](https://jestjs.io/), [Testing Library](https://testing-library.com/) & [Cypress](https://www.cypress.io/) for unit, integration & E2E testing. - Test translation button
+ [Apollo Server](https://www.apollographql.com/docs/apollo-server/) for handling the GraphQL endpoint. - Change admin password
+ [Inkscape](https://inkscape.org/) for designing both the logo and the banner. - Settings stored server-side in `data/settings.json` (never committed)
#### Replicate AI Translation
- When enabled in admin, uses [Replicate](https://replicate.com) + [JigsawStack](https://jigsawstack.com) text-translate model
- Replaces Google Translate scraper with AI translation when active
- Falls back to original lingva-scraper when Replicate is disabled
- Batch translation with separator trick for efficiency
## Deployment #### Document Translation (new "Document" tab)
Translate whole documents while preserving original formatting:
As *Lingva* is a [Next.js](https://nextjs.org/) project you can deploy your own instance anywhere Next is supported. | Format | Formatting | Notes |
|--------|-----------|-------|
| `.xlsx` / `.xls` | **Fully preserved** | Cell styles, formulas, column widths intact. Select which columns to translate. |
| `.docx` | **Fully preserved** | Fonts, tables, images, paragraph styles preserved via XML manipulation |
| `.csv` | Structure preserved | Column selection supported |
| `.pdf` | Best-effort | Text extracted, translated, new formatted PDF generated |
The only requirement is to set an environment variable called `NEXT_PUBLIC_SITE_DOMAIN` with the domain you're deploying the instance under. This is used for the canonical URL and the meta tags. - Drag-and-drop or click-to-upload (up to 50 MB)
- **Column selector** for Excel/CSV: choose individual columns, use All/None toggles per sheet
- Download translated file (named `original_<lang>.ext`)
Optionally, there are other environment variables available: ---
+ `NEXT_PUBLIC_FORCE_DEFAULT_THEME`: Force a certain theme over the system preference set by the user. The accepted values are `light` and `dark`.
+ `NEXT_PUBLIC_DEFAULT_SOURCE_LANG`: Set an initial *source* language instead of the default `auto`.
+ `NEXT_PUBLIC_DEFAULT_TARGET_LANG`: Set an initial *target* language instead of the default `en`.
### Docker ## Getting Started
An [official Docker image](https://hub.docker.com/r/thedaviddelta/lingva-translate) is available to ease the deployment using Compose, Kubernetes or similar technologies. Remember to also include the environment variables (simplified to `site_domain`, `force_default_theme`, `default_source_lang` and `default_target_lang`) when running the container. ### Prerequisites
- Node.js 16+
- npm
#### Docker Compose: ### Installation
```
version: '3'
services:
lingva:
container_name: lingva
image: thedaviddelta/lingva-translate:latest
restart: unless-stopped
environment:
- site_domain=lingva.ml
- force_default_theme=light
- default_source_lang=auto
- default_target_lang=en
ports:
- 3000:3000
```
#### Docker Run
```bash ```bash
docker run -p 3000:3000 -e site_domain=lingva.ml -e force_default_theme=light -e default_source_lang=auto -e default_target_lang=en thedaviddelta/lingva-translate:latest git clone https://devops.cloudhost.es/CloudHost/LingvAI.git
cd LingvAI
npm install
``` ```
### Vercel ### Environment Variables
Another easy way is to use the Next.js creators' own platform, [Vercel](https://vercel.com/), where you can deploy it for free with the following button. Create a `.env.local` file:
[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/git/external?repository-url=https%3A%2F%2Fgithub.com%2Fthedaviddelta%2Flingva-translate%2Ftree%2Fmain&env=NEXT_PUBLIC_SITE_DOMAIN&envDescription=Your%20domain&utm_source=lingva-team&utm_campaign=oss) ```env
# Admin panel
ADMIN_PASSWORD=your_secure_password # Default: admin
ADMIN_JWT_SECRET=random_secret_string # Used to sign admin session tokens
# Replicate AI (optional - can also be set via admin panel)
REPLICATE_API_TOKEN=r8_...
JIGSAWSTACK_API_KEY=sk_...
## Instances # Optional: override default languages
NEXT_PUBLIC_DEFAULT_SOURCE_LANG=auto
These are the currently known *Lingva* instances. Feel free to make a Pull Request including yours (please remember to add `[skip ci]` to the last commit). NEXT_PUBLIC_DEFAULT_TARGET_LANG=en
| Domain | Hosting | SSL Provider |
|:-------------------------------------------------------------------:|:-----------------------------------------:|:--------------------------------------------------------------------------------------------:|
| [lingva.ml](https://lingva.ml/) (Official) | [Vercel](https://vercel.com/) | [Let's Encrypt](https://www.ssllabs.com/ssltest/analyze.html?d=lingva.ml) |
| [translate.igna.ooo](https://translate.igna.ooo/) | [Vercel](https://vercel.com/) | [Let's Encrypt](https://www.ssllabs.com/ssltest/analyze.html?d=translate.igna.ooo) |
| [lingva.pussthecat.org](https://lingva.pussthecat.org) | [Hetzner](https://hetzner.com/) | [Let's Encrypt](https://www.ssllabs.com/ssltest/analyze.html?d=lingva.pussthecat.org) |
| [lingva.lunar.icu](https://lingva.lunar.icu/) | [Lansol](https://lansol.de/) | [Cloudflare](https://www.ssllabs.com/ssltest/analyze.html?d=lingva.lunar.icu) |
## Public APIs
Nearly all the *Lingva* instances should supply a pair of public developer APIs: a RESTful one and a GraphQL one.
*Note: both APIs return the translation audio as a `Uint8Array` (served as `number[]` in JSON and `[Int]` in GraphQL) with the contents of the audio buffer.*
### REST API v1
+ GET `/api/v1/:source/:target/:query`
```typescript
{
translation: string
info?: TranslationInfo
}
``` ```
+ GET `/api/v1/audio/:lang/:query` ### Running
```typescript
{ ```bash
audio: number[] # Development
} npm run dev
# Production
npm run build
npm start
``` ```
+ GET `/api/v1/languages/?:(source|target)` The app runs on [http://localhost:3000](http://localhost:3000) by default.
```typescript
{ ---
languages: [
{ ## Configuration
code: string,
name: string ### Setting up Replicate AI Translation
}
] 1. Open the app and click the **gear icon** (⚙) in the top-right header
} 2. Log in with your admin password (default: `admin`)
3. Enter your **Replicate API token** — get one at [replicate.com/account/api-tokens](https://replicate.com/account/api-tokens)
4. Enter your **JigsawStack API key** — get one at [jigsawstack.com](https://jigsawstack.com)
5. Optionally change the **model version** (default is the JigsawStack text-translate model)
6. Toggle **Enable Replicate Translation** on
7. Click **Save Settings**
8. Use **Test Translation** to verify it works
Default model:
```
jigsawstack/text-translate:454df4c49941c05dea05175bd37686d0872c73c1f9366d1c2505db32ade52a89
``` ```
In addition, every endpoint can return an error message with the following structure instead. ### Replicate API call format
```typescript
{
error: string
}
```
### GraphQL API ```bash
curl -X POST \
+ `/api/graphql` -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
```graphql -H "Content-Type: application/json" \
query { -H "Prefer: wait" \
translation(source: String target: String query: String!) { -d '{
source: { "version": "jigsawstack/text-translate:454df4c49941c05dea05175bd37686d0872c73c1f9366d1c2505db32ade52a89",
lang: { "input": {
code: String! "text": "Hello, world!",
name: String! "api_key": "<jigsawstack_api_key>",
} "target_language": "es"
text: String!
audio: [Int]!
detected: {
code: String
name: String
}
typo: String
pronunciation: String
definitions: {
type: String
list: {
definition: String
example: String
field: String
synonyms: [String]
}
}
examples: [String]
similar: [String]
}
target: {
lang: {
code: String!
name: String!
}
text: String!
audio: [Int]!
pronunciation: String
extraTranslations: {
type: String
list: {
word: String
article: String
frequency: Int
meanings: [String]
}
}
}
} }
audio(lang: String! query: String!) { }' \
lang: { https://api.replicate.com/v1/predictions
code: String!
name: String!
}
text: String!
audio: [Int]!
}
languages(type: SOURCE|TARGET) {
code: String!
name: String!
}
}
``` ```
---
## Related projects ## API Endpoints
+ [Lingva Scraper](https://github.com/thedaviddelta/lingva-scraper) - Google Translate scraper built and maintained specifically for this project ### Original REST API
+ [SimplyTranslate](https://codeberg.org/SimpleWeb/SimplyTranslate-Web) - Very simple translation front-end with multi-engine support ```
+ [LibreTranslate](https://github.com/LibreTranslate/LibreTranslate) - FOSS translation service that uses the open [Argos](https://github.com/argosopentech/argos-translate) engine GET /api/v1/:source/:target/:query → { translation, info }
+ [Lentil for Android](https://github.com/yaxarat/lingvaandroid) - Unofficial native client for Android that uses Lingva's public API GET /api/v1/audio/:lang/:text → { audio: number[] }
+ [Arna Translate](https://github.com/MahanRahmati/translate) - Unofficial cross-platform native client that uses Lingva's public API GET /api/v1/languages → { languages }
+ [Translate-UT](https://github.com/walking-octopus/translate-ut) - Unofficial native client for Ubuntu Touch that uses Lingva's public API ```
### GraphQL
```
POST /api/graphql
```
## Contributors ### New Endpoints
Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)): ```
POST /api/translate/replicate
Body: { text: string, targetLanguage: string }
Returns: { translation: string }
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --> POST /api/translate/document
<!-- prettier-ignore-start --> Body: multipart/form-data
<!-- markdownlint-disable --> file: <file>
<table> targetLanguage: string
<tr> action: "translate" | "getColumns"
<td align="center"><a href="https://thedaviddelta.com/"><img src="https://avatars.githubusercontent.com/u/6679900?v=4?s=100" width="100px;" alt=""/><br /><sub><b>David</b></sub></a><br /><a href="#a11y-TheDavidDelta" title="Accessibility">️️️️♿️</a> <a href="https://github.com/TheDavidDelta/lingva-translate/commits?author=TheDavidDelta" title="Code">💻</a> <a href="https://github.com/TheDavidDelta/lingva-translate/commits?author=TheDavidDelta" title="Documentation">📖</a> <a href="#design-TheDavidDelta" title="Design">🎨</a> <a href="https://github.com/TheDavidDelta/lingva-translate/commits?author=TheDavidDelta" title="Tests">⚠️</a></td> columnSelections?: JSON string (for Excel/CSV)
<td align="center"><a href="https://github.com/mhmdanas"><img src="https://avatars.githubusercontent.com/u/32234660?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Mohammed Anas</b></sub></a><br /><a href="https://github.com/TheDavidDelta/lingva-translate/commits?author=mhmdanas" title="Code">💻</a></td> Returns: file download (translate) or { columns } (getColumns)
<td align="center"><a href="https://PussTheCat.org/"><img src="https://avatars.githubusercontent.com/u/47571719?v=4?s=100" width="100px;" alt=""/><br /><sub><b>TheFrenchGhosty</b></sub></a><br /><a href="https://github.com/TheDavidDelta/lingva-translate/commits?author=TheFrenchGhosty" title="Documentation">📖</a></td>
</tr>
</table>
<!-- markdownlint-restore --> GET /api/admin/auth → { authenticated: boolean }
<!-- prettier-ignore-end --> POST /api/admin/auth body: { password } → sets session cookie
DELETE /api/admin/auth → clears session cookie
<!-- ALL-CONTRIBUTORS-LIST:END --> GET /api/admin/settings → { replicateApiToken, jigsawApiKey, modelVersion, replicateEnabled }
POST /api/admin/settings body: { ...settings, newPassword? }
```
This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome! ---
## Architecture
```
pages/
[[...slug]].tsx Main translation page (Text + Document tabs)
admin/index.tsx Admin settings panel
api/
admin/
auth.ts JWT-based admin authentication
settings.ts Settings read/write (requires admin auth)
translate/
replicate.ts Replicate AI translation endpoint
document.ts Document upload & translation endpoint
v1/[[...slug]].ts Original REST API
graphql.ts Original GraphQL API
components/
DocumentTranslator.tsx File upload UI, progress, download
ColumnSelector.tsx Per-sheet column selection for Excel/CSV
Header.tsx + admin gear icon link
utils/
settings-store.ts Read/write data/settings.json
admin-auth.ts JWT sign/verify helpers
replicate-translate.ts Replicate API calls + batch helper
document-processors/
excel.ts SheetJS Excel/CSV processor
docx.ts JSZip + XML DOCX processor
pdf.ts pdf-parse + pdf-lib PDF processor
```
---
## Docker
```dockerfile
FROM node:18-alpine
WORKDIR /app
COPY . .
RUN npm install && npm run build
EXPOSE 3000
CMD ["npm", "start"]
```
Or using the included `Dockerfile`:
```bash
docker build -t lingvai .
docker run -p 3000:3000 \
-e ADMIN_PASSWORD=secret \
-e ADMIN_JWT_SECRET=random \
-v ./data:/app/data \
lingvai
```
> Mount `./data` as a volume to persist admin settings across container restarts.
---
## Tech Stack
| Layer | Technology |
|-------|-----------|
| Framework | Next.js 12, React 18, TypeScript |
| UI | Chakra UI 2, Framer Motion |
| Translation (default) | lingva-scraper (Google Translate) |
| Translation (AI) | Replicate + JigsawStack text-translate |
| Document processing | SheetJS (xlsx), JSZip, pdf-lib, pdf-parse |
| Admin auth | jose (JWT), HTTP-only cookie |
| File uploads | formidable v3 |
| API | REST + GraphQL (Apollo Server) |
---
## License ## License
[![](https://www.gnu.org/graphics/agplv3-with-text-162x68.png)](https://www.gnu.org/licenses/agpl-3.0.html) [AGPL-3.0](./LICENSE) — same as the upstream Lingva Translate project.
Copyright © 2021 [thedaviddelta](https://github.com/thedaviddelta) & contributors. Original project by [thedaviddelta](https://github.com/thedaviddelta/lingva-translate).
This project is [GNU AGPLv3](./LICENSE) licensed. LingvAI enhancements: admin panel, Replicate AI integration, document translation.

View File

@@ -0,0 +1,107 @@
import { FC, useState, useEffect } from "react";
import {
Box, Checkbox, CheckboxGroup, VStack, HStack, Text, Button,
Accordion, AccordionItem, AccordionButton, AccordionPanel, AccordionIcon,
Badge
} from "@chakra-ui/react";
export type SheetColumnInfo = {
sheetName: string;
columns: string[];
};
export type ColumnSelection = {
sheetName: string;
columnIndices: number[];
};
type Props = {
sheetColumns: SheetColumnInfo[];
onChange: (selections: ColumnSelection[]) => void;
};
const ColumnSelector: FC<Props> = ({ sheetColumns, onChange }) => {
const [selections, setSelections] = useState<Record<string, Set<number>>>(() => {
const init: Record<string, Set<number>> = {};
sheetColumns.forEach(s => {
init[s.sheetName] = new Set(s.columns.map((_, i) => i));
});
return init;
});
useEffect(() => {
const result: ColumnSelection[] = sheetColumns.map(s => ({
sheetName: s.sheetName,
columnIndices: Array.from(selections[s.sheetName] ?? [])
}));
onChange(result);
}, [selections, sheetColumns, onChange]);
const toggleColumn = (sheetName: string, colIdx: number) => {
setSelections(prev => {
const set = new Set(prev[sheetName] ?? []);
if (set.has(colIdx)) set.delete(colIdx);
else set.add(colIdx);
return { ...prev, [sheetName]: set };
});
};
const selectAll = (sheetName: string, cols: string[]) => {
setSelections(prev => ({
...prev,
[sheetName]: new Set(cols.map((_, i) => i))
}));
};
const selectNone = (sheetName: string) => {
setSelections(prev => ({ ...prev, [sheetName]: new Set() }));
};
return (
<Box w="full" borderWidth={1} borderRadius="md" p={3}>
<Text fontWeight="bold" mb={2} fontSize="sm">Select Columns to Translate</Text>
<Accordion allowMultiple defaultIndex={sheetColumns.map((_, i) => i)}>
{sheetColumns.map(sheet => (
<AccordionItem key={sheet.sheetName}>
<AccordionButton>
<Box flex="1" textAlign="left" fontSize="sm" fontWeight="semibold">
{sheet.sheetName}
<Badge ml={2} colorScheme="lingva">
{selections[sheet.sheetName]?.size ?? 0}/{sheet.columns.length}
</Badge>
</Box>
<AccordionIcon />
</AccordionButton>
<AccordionPanel pb={3}>
<HStack mb={2} spacing={2}>
<Button size="xs" variant="outline" colorScheme="lingva"
onClick={() => selectAll(sheet.sheetName, sheet.columns)}>
All
</Button>
<Button size="xs" variant="outline"
onClick={() => selectNone(sheet.sheetName)}>
None
</Button>
</HStack>
<VStack align="start" spacing={1} maxH="150px" overflowY="auto">
{sheet.columns.map((col, idx) => (
<Checkbox
key={idx}
size="sm"
isChecked={selections[sheet.sheetName]?.has(idx) ?? false}
onChange={() => toggleColumn(sheet.sheetName, idx)}
colorScheme="lingva"
>
<Text fontSize="xs">{col}</Text>
</Checkbox>
))}
</VStack>
</AccordionPanel>
</AccordionItem>
))}
</Accordion>
</Box>
);
};
export default ColumnSelector;

View File

@@ -0,0 +1,263 @@
import { FC, useState, useCallback, useRef } from "react";
import {
Box, Button, VStack, HStack, Text, Input, Alert, AlertIcon,
Select, Badge, Icon, useColorModeValue
} from "@chakra-ui/react";
import { FiUpload, FiDownload, FiFile } from "react-icons/fi";
import { languageList, LangCode } from "lingva-scraper";
import dynamic from "next/dynamic";
import type { ColumnSelection, SheetColumnInfo } from "./ColumnSelector";
const ColumnSelector = dynamic(() => import("./ColumnSelector"), { ssr: false });
const SUPPORTED_TYPES = [".pdf", ".docx", ".xlsx", ".xls", ".csv"];
const TABULAR_TYPES = [".xlsx", ".xls", ".csv"];
type Stage = "idle" | "columns" | "translating" | "done" | "error";
const DocumentTranslator: FC = () => {
const [file, setFile] = useState<File | null>(null);
const [target, setTarget] = useState<string>("en");
const [stage, setStage] = useState<Stage>("idle");
const [progress, setProgress] = useState(0);
const [errorMsg, setErrorMsg] = useState("");
const [sheetColumns, setSheetColumns] = useState<SheetColumnInfo[]>([]);
const [columnSelections, setColumnSelections] = useState<ColumnSelection[]>([]);
const [downloadUrl, setDownloadUrl] = useState<string | null>(null);
const [downloadName, setDownloadName] = useState("");
const fileInputRef = useRef<HTMLInputElement>(null);
const borderColor = useColorModeValue("gray.200", "gray.600");
const dropBg = useColorModeValue("gray.50", "gray.700");
const isTabular = file
? TABULAR_TYPES.some(e => file.name.toLowerCase().endsWith(e))
: false;
const handleFile = useCallback(async (f: File) => {
setFile(f);
setStage("idle");
setErrorMsg("");
setDownloadUrl(null);
setSheetColumns([]);
const ext = "." + f.name.split(".").pop()?.toLowerCase();
if (!SUPPORTED_TYPES.includes(ext)) {
setErrorMsg(`Unsupported file type. Supported: ${SUPPORTED_TYPES.join(", ")}`);
setStage("error");
return;
}
if (TABULAR_TYPES.includes(ext)) {
// Fetch column info
const fd = new FormData();
fd.append("file", f);
fd.append("action", "getColumns");
try {
const res = await fetch("/api/translate/document", { method: "POST", body: fd });
if (res.ok) {
const data = await res.json();
setSheetColumns(data.columns ?? []);
setStage("columns");
} else {
const e = await res.json();
setErrorMsg(e.error ?? "Failed to read columns");
setStage("error");
}
} catch {
setErrorMsg("Network error reading file columns");
setStage("error");
}
}
}, []);
const onDrop = useCallback((e: React.DragEvent) => {
e.preventDefault();
const f = e.dataTransfer.files[0];
if (f) handleFile(f);
}, [handleFile]);
const translate = useCallback(async () => {
if (!file) return;
setStage("translating");
setProgress(10);
setErrorMsg("");
const fd = new FormData();
fd.append("file", file);
fd.append("targetLanguage", target);
fd.append("action", "translate");
if (columnSelections.length > 0) {
fd.append("columnSelections", JSON.stringify(columnSelections));
}
try {
setProgress(30);
const res = await fetch("/api/translate/document", { method: "POST", body: fd });
setProgress(90);
if (!res.ok) {
const e = await res.json().catch(() => ({ error: "Translation failed" }));
setErrorMsg(e.error ?? "Translation failed");
setStage("error");
return;
}
const blob = await res.blob();
const ext = "." + file.name.split(".").pop()!;
const outName = file.name.replace(ext, `_${target}${ext}`);
const url = URL.createObjectURL(blob);
setDownloadUrl(url);
setDownloadName(outName);
setProgress(100);
setStage("done");
} catch (err) {
setErrorMsg("Network error during translation");
setStage("error");
}
}, [file, target, columnSelections]);
const reset = () => {
setFile(null);
setStage("idle");
setErrorMsg("");
setDownloadUrl(null);
setSheetColumns([]);
if (fileInputRef.current) fileInputRef.current.value = "";
};
const targetLangs = Object.entries(languageList.target) as [LangCode<"target">, string][];
return (
<VStack w="full" spacing={4} px={[8, null, 24, 40]}>
{/* Drop Zone */}
<Box
w="full"
border="2px dashed"
borderColor={file ? "lingva.400" : borderColor}
borderRadius="xl"
p={8}
bg={dropBg}
textAlign="center"
cursor="pointer"
onDrop={onDrop}
onDragOver={(e: React.DragEvent) => e.preventDefault()}
onClick={() => fileInputRef.current?.click()}
_hover={{ borderColor: "lingva.400" }}
transition="border-color 0.2s"
>
<Input
ref={fileInputRef}
type="file"
display="none"
accept=".pdf,.docx,.xlsx,.xls,.csv"
onChange={(e: React.ChangeEvent<HTMLInputElement>) => { const f = e.target.files?.[0]; if (f) handleFile(f); }}
/>
{file ? (
<VStack spacing={1}>
<Icon as={FiFile} boxSize={8} color="lingva.400" />
<Text fontWeight="semibold">{file.name}</Text>
<Text fontSize="sm" color="gray.500">
{(file.size / 1024 / 1024).toFixed(2)} MB
</Text>
</VStack>
) : (
<VStack spacing={2}>
<Icon as={FiUpload} boxSize={10} color="gray.400" />
<Text fontWeight="semibold">Drop file here or click to upload</Text>
<Text fontSize="sm" color="gray.500">
Supported: PDF, Word (.docx), Excel (.xlsx, .xls), CSV
</Text>
</VStack>
)}
</Box>
{/* Target language + controls */}
{file && stage !== "error" && (
<HStack w="full" spacing={3}>
<Box flex={1}>
<Select
value={target}
onChange={(e: React.ChangeEvent<HTMLSelectElement>) => setTarget(e.target.value)}
size="md"
aria-label="Target language"
>
{targetLangs.map(([code, name]) => (
<option key={code} value={code}>{name}</option>
))}
</Select>
</Box>
<Button
colorScheme="lingva"
leftIcon={<Icon as={FiFile} />}
onClick={translate}
isLoading={stage === "translating"}
loadingText="Translating…"
isDisabled={stage === "translating" || (isTabular && stage !== "columns" && stage !== "idle")}
>
Translate
</Button>
<Button variant="ghost" size="sm" onClick={reset}>Reset</Button>
</HStack>
)}
{/* Column selector for Excel/CSV */}
{stage === "columns" && sheetColumns.length > 0 && (
<ColumnSelector
sheetColumns={sheetColumns}
onChange={setColumnSelections}
/>
)}
{/* Progress */}
{stage === "translating" && (
<Box w="full">
<Box w="full" h="8px" bg="gray.200" borderRadius="full" overflow="hidden">
<Box h="full" w={`${progress}%`} bg="lingva.400" borderRadius="full" transition="width 0.4s ease" />
</Box>
<Text fontSize="sm" textAlign="center" mt={1} color="gray.500">
Translating this may take a moment for large documents.
</Text>
</Box>
)}
{/* Download */}
{stage === "done" && downloadUrl && (
<Alert status="success" borderRadius="md">
<AlertIcon />
<HStack justify="space-between" w="full">
<Text fontSize="sm">Translation complete!</Text>
<Button
as="a"
href={downloadUrl}
download={downloadName}
size="sm"
colorScheme="lingva"
leftIcon={<Icon as={FiDownload} />}
>
Download
</Button>
</HStack>
</Alert>
)}
{/* Error */}
{stage === "error" && errorMsg && (
<Alert status="error" borderRadius="md">
<AlertIcon />
<VStack align="start" spacing={1}>
<Text fontSize="sm">{errorMsg}</Text>
<Button size="xs" variant="link" colorScheme="red" onClick={reset}>Try again</Button>
</VStack>
</Alert>
)}
<Text fontSize="xs" color="gray.400" textAlign="center">
Document translation requires Replicate AI to be enabled in the admin settings.
<br />
PDF formatting is best-effort; Excel and Word formatting is fully preserved.
</Text>
</VStack>
);
};
export default DocumentTranslator;

View File

@@ -20,12 +20,16 @@ const Footer: FC<Props> = (props) => (
{...props} {...props}
> >
<Link href="https://github.com/thedaviddelta/lingva-translate/blob/main/LICENSE" isExternal={true}> <Link href="https://github.com/thedaviddelta/lingva-translate/blob/main/LICENSE" isExternal={true}>
<Text as="span">&#169; 2021 thedaviddelta & contributors</Text> <Text as="span">&#169; 2021{new Date().getFullYear()} thedaviddelta & contributors</Text>
</Link> </Link>
<Text as="span" display={["none", null, "unset"]}>·</Text> <Text as="span" display={["none", null, "unset"]}>·</Text>
<Link href="https://www.gnu.org/licenses/agpl-3.0.html" isExternal={true}> <Link href="https://www.gnu.org/licenses/agpl-3.0.html" isExternal={true}>
<Text as="span">Licensed under AGPLv3</Text> <Text as="span">Licensed under AGPLv3</Text>
</Link> </Link>
<Text as="span" display={["none", null, "unset"]}>·</Text>
<Link href="https://cloudhost.es" isExternal={true}>
<Text as="span">AI enhancements by Cloud Host</Text>
</Link>
{vercelSponsor && ( {vercelSponsor && (
<> <>
<Text as="span" display={["none", null, "unset"]}>·</Text> <Text as="span" display={["none", null, "unset"]}>·</Text>

View File

@@ -1,8 +1,9 @@
import { FC } from "react"; import { FC } from "react";
import Head from "next/head"; import Head from "next/head";
import NextLink from "next/link"; import NextLink from "next/link";
import { Flex, HStack, IconButton, Link, useColorModeValue } from "@chakra-ui/react"; import { Flex, HStack, IconButton, Link, Text, VStack, useColorModeValue } from "@chakra-ui/react";
import { FaGithub } from "react-icons/fa"; import { FaGithub } from "react-icons/fa";
import { FiSettings } from "react-icons/fi";
import Image from "next/image"; import Image from "next/image";
import { ColorModeToggler } from "."; import { ColorModeToggler } from ".";
@@ -27,19 +28,33 @@ const Header: FC<Props> = (props) => (
{...props} {...props}
> >
<NextLink href="/" passHref={true}> <NextLink href="/" passHref={true}>
<Link display="flex"> <Link display="flex" alignItems="center">
<Image <VStack spacing={0} align="flex-start">
src={useColorModeValue("/banner_light.svg", "/banner_dark.svg")} <Image
alt="Logo" src={useColorModeValue("/banner_light.svg", "/banner_dark.svg")}
width={110} alt="Logo"
height={64} width={110}
/> height={64}
/>
<Text fontSize="xs" fontStyle="italic" opacity={0.7} ml={1}>
+ AI enhancements
</Text>
</VStack>
</Link> </Link>
</NextLink> </NextLink>
<HStack spacing={3}> <HStack spacing={3}>
<ColorModeToggler <ColorModeToggler
variant={useColorModeValue("outline", "solid")} variant={useColorModeValue("outline", "solid")}
/> />
<NextLink href="/admin" passHref={true}>
<IconButton
as={Link}
aria-label="Admin settings"
icon={<FiSettings />}
colorScheme="lingva"
variant={useColorModeValue("outline", "solid")}
/>
</NextLink>
<IconButton <IconButton
as={Link} as={Link}
href="https://github.com/thedaviddelta/lingva-translate" href="https://github.com/thedaviddelta/lingva-translate"

View File

@@ -7,3 +7,5 @@ export { default as ColorModeToggler } from "./ColorModeToggler";
export { default as LangSelect } from "./LangSelect"; export { default as LangSelect } from "./LangSelect";
export { default as TranslationArea } from "./TranslationArea"; export { default as TranslationArea } from "./TranslationArea";
export { default as AutoTranslateButton } from "./AutoTranslateButton"; export { default as AutoTranslateButton } from "./AutoTranslateButton";
export { default as DocumentTranslator } from "./DocumentTranslator";
export { default as ColumnSelector } from "./ColumnSelector";

View File

@@ -1,13 +1,13 @@
import { defineConfig } from 'cypress'; import { defineConfig } from 'cypress';
export default defineConfig({ export default defineConfig({
defaultCommandTimeout: 10000, defaultCommandTimeout: 10000,
waitForAnimations: true, waitForAnimations: true,
retries: 4, retries: 4,
projectId: 'qgjdyd', projectId: 'qgjdyd',
e2e: { e2e: {
setupNodeEvents(on, config) {}, setupNodeEvents(on, config) {},
baseUrl: 'http://localhost:3000', baseUrl: 'http://localhost:3000',
specPattern: 'cypress/e2e/**/*.{js,jsx,ts,tsx}' specPattern: 'cypress/e2e/**/*.{js,jsx,ts,tsx}'
}, }
}); });

View File

@@ -1,30 +1,13 @@
{ {
"extends": "../tsconfig.json",
"compilerOptions": { "compilerOptions": {
"isolatedModules": false, "isolatedModules": false,
"target": "es5",
"lib": [
"dom",
"dom.iterable",
"esnext"
],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"forceConsistentCasingInFileNames": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "node",
"resolveJsonModule": true,
"jsx": "preserve",
"types": [ "types": [
"cypress", "cypress",
"@testing-library/cypress" "@testing-library/cypress"
] ]
}, },
"include": [ "include": [
"../node_modules/cypress", "."
"../node_modules/@testing-library/cypress",
"./*/*.ts"
] ]
} }

0
data/.gitkeep Normal file
View File

36
docker-compose.yml Normal file
View File

@@ -0,0 +1,36 @@
services:
lingvai:
build: .
container_name: lingvai-app
ports:
- "3030:3000"
environment:
- ADMIN_PASSWORD=MuieSteaua09
- ADMIN_JWT_SECRET=the3-29-mu-ar3rwfd-dsfhui7
- LOCAL_MODEL_ENDPOINT=http://lingvai-translator:5000/predictions
- REPLICATE_MODE=local
volumes:
- ./data:/app/data
restart: unless-stopped
depends_on:
- translator
networks:
- lingvai-net
translator:
image: r8.im/jigsawstack/text-translate@sha256:454df4c49941c05dea05175bd37686d0872c73c1f9366d1c2505db32ade52a89
container_name: lingvai-translator
ports:
- "5030:5000"
restart: unless-stopped
networks:
- lingvai-net
networks:
lingvai-net:
driver: bridge
ipam:
driver: default
config:
- subnet: 10.100.50.0/24
gateway: 10.100.50.1

View File

@@ -1,6 +1,10 @@
[ [
"https://lingva.ml", "https://lingva.ml",
"https://translate.igna.ooo", "https://translate.igna.wtf",
"https://lingva.pussthecat.org", "https://translate.plausibility.cloud",
"https://lingva.lunar.icu" "https://lingva.lunar.icu",
"https://translate.projectsegfau.lt",
"https://lingva.garudalinux.org",
"https://translate.dr460nf1r3.org",
"https://translate.jae.fi"
] ]

View File

@@ -5,6 +5,20 @@ module.exports = withPWA({
pwa: { pwa: {
dest: "public" dest: "public"
}, },
// Exclude server-only packages from client bundles
webpack: (config, { isServer }) => {
if (!isServer) {
config.resolve.fallback = {
...config.resolve.fallback,
fs: false,
path: false,
crypto: false,
stream: false,
zlib: false
};
}
return config;
},
async headers() { async headers() {
return [ return [
{ {

16714
package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -18,18 +18,25 @@
"@emotion/react": "^11", "@emotion/react": "^11",
"@emotion/styled": "^11", "@emotion/styled": "^11",
"apollo-server-micro": "^2.25.2", "apollo-server-micro": "^2.25.2",
"formidable": "^3.5.1",
"framer-motion": "^6", "framer-motion": "^6",
"graphql": "^15.8.0", "graphql": "^15.8.0",
"lingva-scraper": "1.0.0", "jose": "^4.14.4",
"jszip": "^3.10.1",
"lingva-scraper": "1.1.0",
"next": "12.1.6", "next": "12.1.6",
"next-pwa": "^5.4.4", "next-pwa": "^5.4.4",
"nextjs-cors": "^2.1.0", "nextjs-cors": "^2.1.0",
"pdf-lib": "^1.17.1",
"pdf-parse": "^1.1.1",
"react": "18.2.0", "react": "18.2.0",
"react-dom": "18.2.0", "react-dom": "18.2.0",
"react-hotkeys-hook": "^3.4.6", "react-hotkeys-hook": "^3.4.6",
"react-icons": "^4.4.0" "react-icons": "^4.4.0",
"xlsx": "^0.18.5"
}, },
"devDependencies": { "devDependencies": {
"@types/formidable": "^3.4.4",
"@testing-library/cypress": "^8.0.3", "@testing-library/cypress": "^8.0.3",
"@testing-library/jest-dom": "^5.16.4", "@testing-library/jest-dom": "^5.16.4",
"@testing-library/react": "^13.3.0", "@testing-library/react": "^13.3.0",

View File

@@ -1,4 +1,4 @@
import { useCallback, useEffect, useReducer } from "react"; import { useCallback, useEffect, useReducer, useState } from "react";
import { GetStaticPaths, GetStaticProps, NextPage } from "next"; import { GetStaticPaths, GetStaticProps, NextPage } from "next";
import { useRouter } from "next/router"; import { useRouter } from "next/router";
import dynamic from "next/dynamic"; import dynamic from "next/dynamic";
@@ -13,9 +13,10 @@ import {
TranslationInfo, TranslationInfo,
LangCode LangCode
} from "lingva-scraper"; } from "lingva-scraper";
import { HStack, IconButton, Stack, VStack } from "@chakra-ui/react"; import { HStack, IconButton, Stack, VStack, Tabs, TabList, Tab, TabPanels, TabPanel } from "@chakra-ui/react";
import { FaExchangeAlt } from "react-icons/fa"; import { FaExchangeAlt } from "react-icons/fa";
import { HiTranslate } from "react-icons/hi"; import { HiTranslate } from "react-icons/hi";
import { FiFileText } from "react-icons/fi";
import { useHotkeys } from "react-hotkeys-hook"; import { useHotkeys } from "react-hotkeys-hook";
import { CustomHead, LangSelect, TranslationArea } from "@components"; import { CustomHead, LangSelect, TranslationArea } from "@components";
import { useToastOnLoad } from "@hooks"; import { useToastOnLoad } from "@hooks";
@@ -24,6 +25,7 @@ import langReducer, { Actions, initialState, State } from "@utils/reducer";
import { localGetItem, localSetItem } from "@utils/storage"; import { localGetItem, localSetItem } from "@utils/storage";
const AutoTranslateButton = dynamic(() => import("@components/AutoTranslateButton"), { ssr: false }); const AutoTranslateButton = dynamic(() => import("@components/AutoTranslateButton"), { ssr: false });
const DocumentTranslator = dynamic(() => import("@components/DocumentTranslator"), { ssr: false });
export enum ResponseType { export enum ResponseType {
SUCCESS, SUCCESS,
@@ -63,6 +65,17 @@ const Page: NextPage<Props> = (props) => {
] = useReducer(langReducer, initialState); ] = useReducer(langReducer, initialState);
const router = useRouter(); const router = useRouter();
const [replicateEnabled, setReplicateEnabled] = useState(false);
const [replicateTranslation, setReplicateTranslation] = useState("");
const [replicateLoading, setReplicateLoading] = useState(false);
// Check if Replicate is enabled
useEffect(() => {
fetch("/api/admin/settings")
.then(r => r.ok ? r.json() : null)
.then(d => { if (d?.replicateEnabled) setReplicateEnabled(true); })
.catch(() => {});
}, []);
const setField = useCallback(<T extends keyof State,>(key: T, value: State[T]) => ( const setField = useCallback(<T extends keyof State,>(key: T, value: State[T]) => (
dispatch({ type: Actions.SET_FIELD, payload: { key, value }}) dispatch({ type: Actions.SET_FIELD, payload: { key, value }})
@@ -101,6 +114,30 @@ const Page: NextPage<Props> = (props) => {
router.push(`/${source}/${target}/${encodeURIComponent(customQuery)}`); router.push(`/${source}/${target}/${encodeURIComponent(customQuery)}`);
}, [isLoading, source, target, props, router, setField]); }, [isLoading, source, target, props, router, setField]);
// Replicate-powered translation
const translateWithReplicate = useCallback(async (text: string) => {
if (!text.trim()) return;
setReplicateLoading(true);
setReplicateTranslation("");
try {
const res = await fetch("/api/translate/replicate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ text, targetLanguage: target })
});
const data = await res.json();
if (res.ok) {
setReplicateTranslation(data.translation ?? "");
} else {
setReplicateTranslation(`[Error: ${data.error}]`);
}
} catch {
setReplicateTranslation("[Error: Network failure]");
} finally {
setReplicateLoading(false);
}
}, [target]);
useEffect(() => { useEffect(() => {
if (router.isFallback) if (router.isFallback)
return; return;
@@ -183,78 +220,107 @@ const Page: NextPage<Props> = (props) => {
: replaceExceptedCode(LanguageType.TARGET, source); : replaceExceptedCode(LanguageType.TARGET, source);
const transLang = replaceExceptedCode(LanguageType.SOURCE, target); const transLang = replaceExceptedCode(LanguageType.SOURCE, target);
const LangControls = (
<HStack px={[1, null, 3, 4]} w="full">
<LangSelect
id="source"
aria-label="Source language"
value={source}
detectedSource={detectedSource}
onChange={e => setLanguage(LanguageType.SOURCE, e.target.value)}
langs={languageList.source}
/>
<IconButton
aria-label="Switch languages"
icon={<FaExchangeAlt />}
colorScheme="lingva"
variant="ghost"
onClick={() => switchLanguages(detectedSource)}
isDisabled={!canSwitch}
/>
<LangSelect
id="target"
aria-label="Target language"
value={target}
onChange={e => setLanguage(LanguageType.TARGET, e.target.value)}
langs={languageList.target}
/>
</HStack>
);
return ( return (
<> <>
<CustomHead home={props.type === ResponseType.HOME} /> <CustomHead home={props.type === ResponseType.HOME} />
<VStack px={[8, null, 24, 40]} w="full"> <VStack px={[8, null, 24, 40]} w="full">
<HStack px={[1, null, 3, 4]} w="full"> <Tabs w="full" colorScheme="lingva" variant="soft-rounded" size="sm">
<LangSelect <TabList mb={4} justifyContent="center">
id="source" <Tab><HiTranslate style={{ marginRight: 6 }} />Text</Tab>
aria-label="Source language" <Tab><FiFileText style={{ marginRight: 6 }} />Document</Tab>
value={source} </TabList>
detectedSource={detectedSource}
onChange={e => setLanguage(LanguageType.SOURCE, e.target.value)} <TabPanels>
langs={languageList.source} {/* Text Translation Tab */}
/> <TabPanel p={0}>
<IconButton <VStack w="full" spacing={4}>
aria-label="Switch languages" {LangControls}
icon={<FaExchangeAlt />} <Stack direction={["column", null, "row"]} w="full">
colorScheme="lingva" <TranslationArea
variant="ghost" id="query"
onClick={() => switchLanguages(detectedSource)} aria-label="Translation query"
isDisabled={!canSwitch} placeholder="Text"
/> value={query}
<LangSelect onChange={e => isLoading || setField("query", e.target.value)}
id="target" onSubmit={() => replicateEnabled ? translateWithReplicate(query) : changeRoute(query)}
aria-label="Target language" lang={queryLang}
value={target} audio={audio.query}
onChange={e => setLanguage(LanguageType.TARGET, e.target.value)} pronunciation={pronunciation.query}
langs={languageList.target} />
/> <Stack direction={["row", null, "column"]} justify="center" spacing={3} px={[2, null, "initial"]}>
</HStack> <IconButton
<Stack direction={["column", null, "row"]} w="full"> aria-label="Translate"
<TranslationArea icon={<HiTranslate />}
id="query" colorScheme="lingva"
aria-label="Translation query" variant="outline"
placeholder="Text" onClick={() => replicateEnabled ? translateWithReplicate(query) : changeRoute(query)}
value={query} isDisabled={isLoading || replicateLoading}
onChange={e => isLoading || setField("query", e.target.value)} isLoading={replicateLoading}
onSubmit={() => changeRoute(query)} w={["full", null, "auto"]}
lang={queryLang} />
audio={audio.query} <AutoTranslateButton
pronunciation={pronunciation.query} isDisabled={isLoading || replicateLoading}
/> onAuto={useCallback(() => {
<Stack direction={["row", null, "column"]} justify="center" spacing={3} px={[2, null, "initial"]}> if (replicateEnabled) {
<IconButton translateWithReplicate(delayedQuery);
aria-label="Translate" } else {
icon={<HiTranslate />} changeRoute(delayedQuery);
colorScheme="lingva" }
variant="outline" }, [delayedQuery, replicateEnabled, translateWithReplicate, changeRoute])}
onClick={() => changeRoute(query)} w={["full", null, "auto"]}
isDisabled={isLoading} />
w={["full", null, "auto"]} </Stack>
/> <TranslationArea
<AutoTranslateButton id="translation"
isDisabled={isLoading} aria-label="Translation result"
// runs on effect update placeholder="Translation"
onAuto={useCallback(() => changeRoute(delayedQuery), [delayedQuery, changeRoute])} value={replicateEnabled ? replicateTranslation : (translation ?? "")}
w={["full", null, "auto"]} readOnly={true}
/> lang={transLang}
</Stack> audio={audio.translation}
<TranslationArea canCopy={true}
id="translation" isLoading={replicateEnabled ? replicateLoading : isLoading}
aria-label="Translation result" pronunciation={replicateEnabled ? undefined : pronunciation.translation}
placeholder="Translation" />
value={translation ?? ""} </Stack>
readOnly={true} </VStack>
lang={transLang} </TabPanel>
audio={audio.translation}
canCopy={true} {/* Document Translation Tab */}
isLoading={isLoading} <TabPanel p={0}>
pronunciation={pronunciation.translation} <DocumentTranslator />
/> </TabPanel>
</Stack> </TabPanels>
</Tabs>
</VStack> </VStack>
</> </>
); );

396
pages/admin/index.tsx Normal file
View File

@@ -0,0 +1,396 @@
import { useState, useEffect, FormEvent } from "react";
import { NextPage } from "next";
import NextLink from "next/link";
import {
Box, VStack, HStack, Heading, Text, Input, Button, Switch,
FormControl, FormLabel, FormHelperText, Alert, AlertIcon,
InputGroup, InputRightElement, IconButton, Badge, Link,
useColorModeValue, Spinner, Textarea, RadioGroup, Radio, Code
} from "@chakra-ui/react";
import { FiEye, FiEyeOff, FiSave, FiLogOut, FiArrowLeft } from "react-icons/fi";
import { CustomHead } from "@components";
type ReplicateMode = "cloud" | "local";
type Settings = {
replicateApiToken: string;
jigsawApiKey: string;
modelVersion: string;
replicateEnabled: boolean;
replicateMode: ReplicateMode;
localEndpoint: string;
};
const DEFAULT_MODEL = "jigsawstack/text-translate:454df4c49941c05dea05175bd37686d0872c73c1f9366d1c2505db32ade52a89";
const DEFAULT_LOCAL_ENDPOINT = "http://localhost:5030/predictions";
const AdminPage: NextPage = () => {
const [authed, setAuthed] = useState<boolean | null>(null);
const [password, setPassword] = useState("");
const [loginError, setLoginError] = useState("");
const [loginLoading, setLoginLoading] = useState(false);
const [settings, setSettings] = useState<Settings>({
replicateApiToken: "",
jigsawApiKey: "",
modelVersion: DEFAULT_MODEL,
replicateEnabled: false,
replicateMode: "cloud",
localEndpoint: DEFAULT_LOCAL_ENDPOINT
});
const [newPassword, setNewPassword] = useState("");
const [saveMsg, setSaveMsg] = useState<{ type: "success" | "error"; text: string } | null>(null);
const [saving, setSaving] = useState(false);
const [showToken, setShowToken] = useState(false);
const [showJigsawKey, setShowJigsawKey] = useState(false);
const [testResult, setTestResult] = useState<string | null>(null);
const [testLoading, setTestLoading] = useState(false);
const pageBg = useColorModeValue("gray.50", "gray.900");
const cardBg = useColorModeValue("white", "gray.800");
const borderCol = useColorModeValue("gray.200", "gray.600");
const codeBg = useColorModeValue("gray.100", "gray.700");
useEffect(() => {
fetch("/api/admin/auth")
.then(r => r.json())
.then(d => setAuthed(d.authenticated))
.catch(() => setAuthed(false));
}, []);
useEffect(() => {
if (!authed) return;
fetch("/api/admin/settings")
.then(r => r.json())
.then(d => setSettings(s => ({ ...s, ...d })))
.catch(() => {});
}, [authed]);
const login = async (e: FormEvent) => {
e.preventDefault();
setLoginLoading(true);
setLoginError("");
try {
const res = await fetch("/api/admin/auth", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ password })
});
if (res.ok) setAuthed(true);
else {
const d = await res.json();
setLoginError(d.error ?? "Login failed");
}
} catch {
setLoginError("Network error");
} finally {
setLoginLoading(false);
}
};
const logout = async () => {
await fetch("/api/admin/auth", { method: "DELETE" });
setAuthed(false);
setPassword("");
};
const save = async (e: FormEvent) => {
e.preventDefault();
setSaving(true);
setSaveMsg(null);
try {
const body: Record<string, unknown> = { ...settings };
if (newPassword.length >= 6) body.newPassword = newPassword;
const res = await fetch("/api/admin/settings", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body)
});
const d = await res.json();
if (res.ok) {
setSettings(s => ({ ...s, ...d }));
setSaveMsg({ type: "success", text: "Settings saved successfully" });
setNewPassword("");
} else {
setSaveMsg({ type: "error", text: d.error ?? "Save failed" });
}
} catch {
setSaveMsg({ type: "error", text: "Network error" });
} finally {
setSaving(false);
}
};
const testTranslation = async () => {
setTestLoading(true);
setTestResult(null);
try {
const res = await fetch("/api/translate/replicate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ text: "Hello, world!", targetLanguage: "es" })
});
const d = await res.json();
setTestResult(res.ok ? `✓ "${d.translation}"` : `${d.error}`);
} catch {
setTestResult("✗ Network error");
} finally {
setTestLoading(false);
}
};
if (authed === null) {
return (
<Box w="full" minH="100%" bg={pageBg} display="flex" justifyContent="center" pt={20}>
<Spinner size="xl" color="lingva.400" />
</Box>
);
}
if (!authed) {
return (
<>
<CustomHead home={false} />
<Box w="full" minH="100%" bg={pageBg}>
<Box
w="full" maxW="400px" mx="auto" mt={10} p={8}
bg={cardBg} borderWidth={1} borderColor={borderCol}
borderRadius="xl" boxShadow="md"
>
<VStack spacing={6} as="form" onSubmit={login}>
<Heading size="md">Admin Login</Heading>
{loginError && (
<Alert status="error" borderRadius="md">
<AlertIcon />{loginError}
</Alert>
)}
<FormControl isRequired>
<FormLabel>Password</FormLabel>
<Input
type="password"
value={password}
onChange={e => setPassword(e.target.value)}
autoFocus
placeholder="Admin password"
/>
<FormHelperText>Default: admin (set ADMIN_PASSWORD env var)</FormHelperText>
</FormControl>
<Button type="submit" colorScheme="lingva" w="full" isLoading={loginLoading}>
Login
</Button>
<NextLink href="/" passHref>
<Link fontSize="sm"> Back to translator</Link>
</NextLink>
</VStack>
</Box>
</Box>
</>
);
}
return (
<>
<CustomHead home={false} />
<Box w="full" minH="100%" bg={pageBg}>
<Box w="full" maxW="700px" mx="auto" px={4} pb={10} pt={2}>
<HStack justify="space-between" mb={6}>
<NextLink href="/" passHref>
<Button as={Link} leftIcon={<FiArrowLeft />} variant="ghost" size="sm">Back</Button>
</NextLink>
<Heading size="md">Admin Settings</Heading>
<Button leftIcon={<FiLogOut />} variant="ghost" size="sm" onClick={logout}>Logout</Button>
</HStack>
<VStack spacing={6} as="form" onSubmit={save}>
{/* ── Replicate Section ── */}
<Box w="full" p={6} bg={cardBg} borderWidth={1} borderColor={borderCol} borderRadius="xl">
<HStack mb={4}>
<Heading size="sm">Replicate AI Translation</Heading>
<Badge colorScheme={settings.replicateEnabled ? "green" : "gray"}>
{settings.replicateEnabled ? "Enabled" : "Disabled"}
</Badge>
</HStack>
<VStack spacing={4} align="stretch">
<FormControl display="flex" alignItems="center">
<FormLabel mb={0}>Enable Replicate Translation</FormLabel>
<Switch
colorScheme="lingva"
isChecked={settings.replicateEnabled}
onChange={e => setSettings(s => ({ ...s, replicateEnabled: e.target.checked }))}
/>
</FormControl>
{/* Mode selector */}
<FormControl>
<FormLabel>Translation Backend</FormLabel>
<RadioGroup
value={settings.replicateMode}
onChange={v => setSettings(s => ({ ...s, replicateMode: v as ReplicateMode }))}
>
<HStack spacing={6}>
<Radio value="cloud" colorScheme="lingva">
<VStack align="start" spacing={0}>
<Text fontSize="sm" fontWeight="semibold">Replicate Cloud</Text>
<Text fontSize="xs" color="gray.500">Uses replicate.com API</Text>
</VStack>
</Radio>
<Radio value="local" colorScheme="lingva">
<VStack align="start" spacing={0}>
<Text fontSize="sm" fontWeight="semibold">Local Docker (Cog)</Text>
<Text fontSize="xs" color="gray.500">Uses local container</Text>
</VStack>
</Radio>
</HStack>
</RadioGroup>
</FormControl>
{/* Cloud fields */}
{settings.replicateMode === "cloud" && (
<>
<FormControl>
<FormLabel>Replicate API Token</FormLabel>
<InputGroup>
<Input
type={showToken ? "text" : "password"}
value={settings.replicateApiToken}
onChange={e => setSettings(s => ({ ...s, replicateApiToken: e.target.value }))}
placeholder="r8_..."
fontFamily="mono"
fontSize="sm"
/>
<InputRightElement>
<IconButton size="xs" variant="ghost"
aria-label="Toggle visibility"
icon={showToken ? <FiEyeOff /> : <FiEye />}
onClick={() => setShowToken(v => !v)}
/>
</InputRightElement>
</InputGroup>
<FormHelperText>
Get your token at{" "}
<Link href="https://replicate.com/account/api-tokens" isExternal color="lingva.400">
replicate.com/account/api-tokens
</Link>
</FormHelperText>
</FormControl>
<FormControl>
<FormLabel>Model Version</FormLabel>
<Textarea
value={settings.modelVersion}
onChange={e => setSettings(s => ({ ...s, modelVersion: e.target.value }))}
fontFamily="mono"
fontSize="xs"
rows={2}
placeholder={DEFAULT_MODEL}
/>
<FormHelperText>
Format: <Code fontSize="xs">owner/model:hash</Code> the hash after the colon is extracted automatically.
</FormHelperText>
</FormControl>
</>
)}
{/* Local fields */}
{settings.replicateMode === "local" && (
<>
<Box bg={codeBg} p={4} borderRadius="md" fontSize="sm">
<Text fontWeight="semibold" mb={2}>Start the local model container:</Text>
<Code display="block" fontSize="xs" whiteSpace="pre" overflowX="auto">
{`docker run -d -p 5030:5000 \\\n r8.im/jigsawstack/text-translate@sha256:454df4c49941c05dea05175bd37686d0872c73c1f9366d1c2505db32ade52a89`}
</Code>
</Box>
<FormControl>
<FormLabel>Local Model Endpoint</FormLabel>
<Input
value={settings.localEndpoint}
onChange={e => setSettings(s => ({ ...s, localEndpoint: e.target.value }))}
placeholder={DEFAULT_LOCAL_ENDPOINT}
fontFamily="mono"
fontSize="sm"
/>
<FormHelperText>
Default: <Code fontSize="xs">{DEFAULT_LOCAL_ENDPOINT}</Code> (container port 5000 mapped to host 5030)
</FormHelperText>
</FormControl>
</>
)}
{/* JigsawStack key — needed in both modes */}
<FormControl>
<FormLabel>JigsawStack API Key</FormLabel>
<InputGroup>
<Input
type={showJigsawKey ? "text" : "password"}
value={settings.jigsawApiKey}
onChange={e => setSettings(s => ({ ...s, jigsawApiKey: e.target.value }))}
placeholder="sk_..."
fontFamily="mono"
fontSize="sm"
/>
<InputRightElement>
<IconButton size="xs" variant="ghost"
aria-label="Toggle visibility"
icon={showJigsawKey ? <FiEyeOff /> : <FiEye />}
onClick={() => setShowJigsawKey(v => !v)}
/>
</InputRightElement>
</InputGroup>
<FormHelperText>
Required by the JigsawStack model in both cloud and local modes.{" "}
<Link href="https://jigsawstack.com" isExternal color="lingva.400">Get yours </Link>
</FormHelperText>
</FormControl>
<HStack>
<Button
size="sm" variant="outline" colorScheme="lingva"
onClick={testTranslation} isLoading={testLoading}
isDisabled={!settings.replicateEnabled || !settings.jigsawApiKey}
>
Test Translation
</Button>
{testResult && (
<Text fontSize="sm" color={testResult.startsWith("✓") ? "green.500" : "red.500"}>
{testResult}
</Text>
)}
</HStack>
</VStack>
</Box>
{/* ── Security ── */}
<Box w="full" p={6} bg={cardBg} borderWidth={1} borderColor={borderCol} borderRadius="xl">
<Heading size="sm" mb={4}>Security</Heading>
<FormControl>
<FormLabel>New Admin Password</FormLabel>
<Input
type="password"
value={newPassword}
onChange={e => setNewPassword(e.target.value)}
placeholder="Leave blank to keep current"
minLength={6}
/>
<FormHelperText>Minimum 6 characters. Leave blank to keep current.</FormHelperText>
</FormControl>
</Box>
{saveMsg && (
<Alert status={saveMsg.type} borderRadius="md">
<AlertIcon />{saveMsg.text}
</Alert>
)}
<Button type="submit" colorScheme="lingva" leftIcon={<FiSave />} isLoading={saving} size="lg" w="full">
Save Settings
</Button>
</VStack>
</Box>
</Box>
</>
);
};
export default AdminPage;

33
pages/api/admin/auth.ts Normal file
View File

@@ -0,0 +1,33 @@
import { NextApiHandler } from "next";
import { signAdminToken, checkPassword, COOKIE_NAME, getTokenFromRequest, verifyAdminToken } from "@utils/admin-auth";
const handler: NextApiHandler = async (req, res) => {
if (req.method === "POST") {
const { password } = req.body ?? {};
if (!password || typeof password !== "string") {
return res.status(400).json({ error: "Password required" });
}
if (!checkPassword(password)) {
return res.status(401).json({ error: "Invalid password" });
}
const token = await signAdminToken();
res.setHeader("Set-Cookie", `${COOKIE_NAME}=${token}; HttpOnly; Path=/; SameSite=Lax; Max-Age=28800`);
return res.status(200).json({ ok: true });
}
if (req.method === "DELETE") {
res.setHeader("Set-Cookie", `${COOKIE_NAME}=; HttpOnly; Path=/; Max-Age=0`);
return res.status(200).json({ ok: true });
}
if (req.method === "GET") {
const token = getTokenFromRequest(req);
const valid = token ? await verifyAdminToken(token) : false;
return res.status(200).json({ authenticated: valid });
}
res.setHeader("Allow", ["GET", "POST", "DELETE"]);
return res.status(405).json({ error: "Method Not Allowed" });
};
export default handler;

View File

@@ -0,0 +1,46 @@
import { NextApiHandler } from "next";
import { requireAdmin } from "@utils/admin-auth";
import { readSettings, writeSettings } from "@utils/settings-store";
const handler: NextApiHandler = async (req, res) => {
if (!(await requireAdmin(req, res))) return;
if (req.method === "GET") {
const settings = readSettings();
// Never expose the adminPasswordHash
const { adminPasswordHash: _omit, ...safe } = settings;
return res.status(200).json(safe);
}
if (req.method === "POST") {
const {
replicateApiToken,
jigsawApiKey,
modelVersion,
replicateEnabled,
replicateMode,
localEndpoint,
newPassword
} = req.body ?? {};
const updates: Parameters<typeof writeSettings>[0] = {};
if (replicateApiToken !== undefined) updates.replicateApiToken = replicateApiToken;
if (jigsawApiKey !== undefined) updates.jigsawApiKey = jigsawApiKey;
if (modelVersion !== undefined) updates.modelVersion = modelVersion;
if (replicateEnabled !== undefined) updates.replicateEnabled = Boolean(replicateEnabled);
if (replicateMode === "cloud" || replicateMode === "local") updates.replicateMode = replicateMode;
if (localEndpoint !== undefined) updates.localEndpoint = localEndpoint;
if (newPassword && typeof newPassword === "string" && newPassword.length >= 6) {
updates.adminPasswordHash = newPassword;
}
const saved = writeSettings(updates);
const { adminPasswordHash: _omit, ...safe } = saved;
return res.status(200).json(safe);
}
res.setHeader("Allow", ["GET", "POST"]);
return res.status(405).json({ error: "Method Not Allowed" });
};
export default handler;

View File

@@ -0,0 +1,136 @@
import { NextApiHandler } from "next";
import formidable, { File, Fields, Files } from "formidable";
import fs from "fs";
import path from "path";
import { translateExcel, getExcelColumns, ColumnSelection } from "@utils/document-processors/excel";
import { translateDocx } from "@utils/document-processors/docx";
import { translatePdf } from "@utils/document-processors/pdf";
import { readSettings } from "@utils/settings-store";
export const config = {
api: {
bodyParser: false,
responseLimit: "50mb"
}
};
type ParsedForm = {
fields: Fields;
files: Files;
};
function parseForm(req: Parameters<NextApiHandler>[0]): Promise<ParsedForm> {
const form = formidable({ maxFileSize: 50 * 1024 * 1024 }); // 50 MB
return new Promise((resolve, reject) => {
form.parse(req, (err, fields, files) => {
if (err) reject(err);
else resolve({ fields, files });
});
});
}
function getFileBuffer(file: File): Buffer {
return fs.readFileSync(file.filepath);
}
function getField(fields: Fields, key: string): string | undefined {
const val = fields[key];
return Array.isArray(val) ? val[0] : (val as string | undefined);
}
const handler: NextApiHandler = async (req, res) => {
if (req.method !== "POST") {
res.setHeader("Allow", ["POST"]);
return res.status(405).json({ error: "Method Not Allowed" });
}
const settings = readSettings();
if (!settings.replicateEnabled) {
return res.status(503).json({ error: "Replicate translation is not enabled. Configure it in the admin panel." });
}
let parsed: ParsedForm;
try {
parsed = await parseForm(req);
} catch (err) {
return res.status(400).json({ error: "Failed to parse upload" });
}
const { fields, files } = parsed;
const fileEntry = files["file"];
const file = Array.isArray(fileEntry) ? fileEntry[0] : fileEntry;
if (!file) {
return res.status(400).json({ error: "No file uploaded" });
}
const targetLanguage = getField(fields, "targetLanguage") ?? "en";
const sourceLanguage = getField(fields, "sourceLanguage");
const action = getField(fields, "action") ?? "translate";
const columnSelectionsRaw = getField(fields, "columnSelections");
const filename = file.originalFilename ?? file.newFilename ?? "file";
const ext = path.extname(filename).toLowerCase();
const buffer = getFileBuffer(file);
try {
// Action: getColumns - return column info for Excel files
if (action === "getColumns") {
if (![".xlsx", ".xls", ".csv"].includes(ext)) {
return res.status(400).json({ error: "Column selection only supported for Excel/CSV files" });
}
const columns = getExcelColumns(buffer, filename);
return res.status(200).json({ columns });
}
// Action: translate
let outBuffer: Buffer;
let outMime: string;
let outFilename: string;
if ([".xlsx", ".xls"].includes(ext)) {
let columnSelections: ColumnSelection[] = [];
if (columnSelectionsRaw) {
try {
columnSelections = JSON.parse(columnSelectionsRaw);
} catch { /* use empty = translate all */ }
}
outBuffer = await translateExcel(buffer, targetLanguage, columnSelections);
outMime = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
outFilename = filename.replace(ext, `_${targetLanguage}${ext}`);
} else if (ext === ".csv") {
// Treat CSV as Excel
let columnSelections: ColumnSelection[] = [];
if (columnSelectionsRaw) {
try {
columnSelections = JSON.parse(columnSelectionsRaw);
} catch { /* use empty = translate all */ }
}
outBuffer = await translateExcel(buffer, targetLanguage, columnSelections);
outMime = "text/csv";
outFilename = filename.replace(ext, `_${targetLanguage}${ext}`);
} else if (ext === ".docx") {
outBuffer = await translateDocx(buffer, targetLanguage);
outMime = "application/vnd.openxmlformats-officedocument.wordprocessingml.document";
outFilename = filename.replace(ext, `_${targetLanguage}${ext}`);
} else if (ext === ".pdf") {
outBuffer = await translatePdf(buffer, targetLanguage, sourceLanguage);
outMime = "application/pdf";
outFilename = filename.replace(ext, `_${targetLanguage}${ext}`);
} else {
return res.status(400).json({
error: `Unsupported file type: ${ext}. Supported: .pdf, .docx, .xlsx, .xls, .csv`
});
}
res.setHeader("Content-Type", outMime);
res.setHeader("Content-Disposition", `attachment; filename="${outFilename}"`);
res.setHeader("Content-Length", outBuffer.length);
return res.status(200).send(outBuffer);
} catch (err) {
const msg = err instanceof Error ? err.message : "Translation failed";
return res.status(500).json({ error: msg });
}
};
export default handler;

View File

@@ -0,0 +1,38 @@
import { NextApiHandler } from "next";
import NextCors from "nextjs-cors";
import { replicateTranslate } from "@utils/replicate-translate";
import { readSettings } from "@utils/settings-store";
type Data = { translation: string } | { error: string };
const handler: NextApiHandler<Data> = async (req, res) => {
await NextCors(req, res, { methods: ["POST"], origin: "*" });
if (req.method !== "POST") {
res.setHeader("Allow", ["POST"]);
return res.status(405).json({ error: "Method Not Allowed" });
}
const settings = readSettings();
if (!settings.replicateEnabled) {
return res.status(503).json({ error: "Replicate translation is not enabled" });
}
const { text, targetLanguage } = req.body ?? {};
if (!text || typeof text !== "string") {
return res.status(400).json({ error: "text is required" });
}
if (!targetLanguage || typeof targetLanguage !== "string") {
return res.status(400).json({ error: "targetLanguage is required" });
}
try {
const translation = await replicateTranslate(text, targetLanguage);
return res.status(200).json({ translation });
} catch (err) {
const msg = err instanceof Error ? err.message : "Translation failed";
return res.status(500).json({ error: msg });
}
};
export default handler;

14
tests/tsconfig.json Normal file
View File

@@ -0,0 +1,14 @@
{
"extends": "../tsconfig.json",
"compilerOptions": {
"types": [
"jest",
"@testing-library/jest-dom",
"@testing-library/react",
"@testing-library/user-event"
]
},
"include": [
"."
]
}

View File

@@ -29,9 +29,6 @@
"**/*.tsx" "**/*.tsx"
], ],
"exclude": [ "exclude": [
"node_modules", "node_modules"
"**/*.test.ts",
"**/*.test.tsx",
"cypress"
] ]
} }

55
utils/admin-auth.ts Normal file
View File

@@ -0,0 +1,55 @@
import { SignJWT, jwtVerify } from "jose";
import { NextApiRequest, NextApiResponse } from "next";
import { readSettings } from "./settings-store";
const JWT_COOKIE = "lingva_admin";
const JWT_EXPIRY = "8h";
function getSecret(): Uint8Array {
const secret = process.env["ADMIN_JWT_SECRET"] ?? "lingva-admin-secret-change-me";
return new TextEncoder().encode(secret);
}
export async function signAdminToken(): Promise<string> {
return new SignJWT({ role: "admin" })
.setProtectedHeader({ alg: "HS256" })
.setIssuedAt()
.setExpirationTime(JWT_EXPIRY)
.sign(getSecret());
}
export async function verifyAdminToken(token: string): Promise<boolean> {
try {
await jwtVerify(token, getSecret());
return true;
} catch {
return false;
}
}
export function getTokenFromRequest(req: NextApiRequest): string | null {
const cookie = req.cookies[JWT_COOKIE];
if (cookie) return cookie;
const auth = req.headers.authorization;
if (auth?.startsWith("Bearer ")) return auth.slice(7);
return null;
}
export async function requireAdmin(
req: NextApiRequest,
res: NextApiResponse
): Promise<boolean> {
const token = getTokenFromRequest(req);
if (!token || !(await verifyAdminToken(token))) {
res.status(401).json({ error: "Unauthorized" });
return false;
}
return true;
}
export function checkPassword(password: string): boolean {
const settings = readSettings();
return password === settings.adminPasswordHash;
}
export const COOKIE_NAME = JWT_COOKIE;

View File

@@ -0,0 +1,106 @@
import JSZip from "jszip";
import { replicateTranslateBatch } from "../replicate-translate";
function escapeXml(text: string): string {
return text
.replace(/&/g, "&amp;")
.replace(/</g, "&lt;")
.replace(/>/g, "&gt;")
.replace(/"/g, "&quot;")
.replace(/'/g, "&apos;");
}
/**
* Extract paragraph texts from document.xml string.
* Returns array of {index, text} where index is the paragraph number.
*/
function extractParagraphs(xml: string): { index: number; text: string; start: number; end: number }[] {
const paragraphs: { index: number; text: string; start: number; end: number }[] = [];
const pRegex = /<w:p[ >]/g;
const pCloseTag = "</w:p>";
let idx = 0;
let match: RegExpExecArray | null;
while ((match = pRegex.exec(xml)) !== null) {
const start = match.index;
const end = xml.indexOf(pCloseTag, start) + pCloseTag.length;
if (end < pCloseTag.length) break;
const paraXml = xml.slice(start, end);
// Extract all text content within this paragraph
const textParts: string[] = [];
const tRegex = /<w:t[^>]*>([\s\S]*?)<\/w:t>/g;
let tMatch: RegExpExecArray | null;
while ((tMatch = tRegex.exec(paraXml)) !== null) {
textParts.push(tMatch[1]);
}
const text = textParts.join("").trim();
if (text) {
paragraphs.push({ index: idx, text, start, end });
}
idx++;
}
return paragraphs;
}
/**
* Replace text within a paragraph XML while preserving formatting of first run.
* Empties all other text runs.
*/
function replaceParagraphText(paraXml: string, translatedText: string): string {
let firstDone = false;
return paraXml.replace(/<w:t([^>]*)>([\s\S]*?)<\/w:t>/g, (_match, attrs, content) => {
if (!firstDone && content.trim()) {
firstDone = true;
return `<w:t xml:space="preserve">${escapeXml(translatedText)}</w:t>`;
}
if (firstDone) {
return `<w:t></w:t>`;
}
return _match; // preserve empty runs before the first text
});
}
/**
* Translate a DOCX file buffer, preserving formatting.
*/
export async function translateDocx(
buffer: Buffer,
targetLanguage: string
): Promise<Buffer> {
const zip = await JSZip.loadAsync(buffer);
const docFile = zip.file("word/document.xml");
if (!docFile) throw new Error("Invalid DOCX: missing word/document.xml");
let xml = await docFile.async("string");
const paragraphs = extractParagraphs(xml);
if (paragraphs.length === 0) {
return Buffer.from(await zip.generateAsync({ type: "nodebuffer" }));
}
// Translate all paragraphs
const translations = await replicateTranslateBatch(
paragraphs.map(p => p.text),
targetLanguage
);
// Replace paragraphs from end to start to preserve offsets
const sorted = [...paragraphs].sort((a, b) => b.start - a.start);
for (const para of sorted) {
const translationIdx = paragraphs.findIndex(p => p.start === para.start);
const translated = translations[translationIdx] ?? para.text;
const originalPara = xml.slice(para.start, para.end);
const translatedPara = replaceParagraphText(originalPara, translated);
xml = xml.slice(0, para.start) + translatedPara + xml.slice(para.end);
}
zip.file("word/document.xml", xml);
const outBuffer = await zip.generateAsync({
type: "nodebuffer",
compression: "DEFLATE"
});
return Buffer.from(outBuffer);
}

View File

@@ -0,0 +1,95 @@
import * as XLSX from "xlsx";
import { replicateTranslateBatch } from "../replicate-translate";
export type SheetColumnInfo = {
sheetName: string;
columns: string[]; // header names or A, B, C...
};
export type ColumnSelection = {
sheetName: string;
columnIndices: number[]; // 0-based column indices to translate
};
/**
* Parse an Excel/CSV buffer and return sheet/column metadata for column selection UI.
*/
export function getExcelColumns(buffer: Buffer, filename: string): SheetColumnInfo[] {
const wb = XLSX.read(buffer, { type: "buffer" });
return wb.SheetNames.map(sheetName => {
const ws = wb.Sheets[sheetName];
const range = XLSX.utils.decode_range(ws["!ref"] ?? "A1");
const columns: string[] = [];
for (let c = range.s.c; c <= range.e.c; c++) {
// Try to get header from first row
const cellAddr = XLSX.utils.encode_cell({ r: 0, c });
const cell = ws[cellAddr];
const header = cell && cell.v != null ? String(cell.v) : XLSX.utils.encode_col(c);
columns.push(header);
}
return { sheetName, columns };
});
}
/**
* Translate selected columns in an Excel buffer. Returns translated buffer.
* columnSelections: array of {sheetName, columnIndices}
* If columnSelections is empty, all text columns are translated.
*/
export async function translateExcel(
buffer: Buffer,
targetLanguage: string,
columnSelections: ColumnSelection[]
): Promise<Buffer> {
const wb = XLSX.read(buffer, { type: "buffer", cellStyles: true, cellNF: true });
for (const sheet of wb.SheetNames) {
const ws = wb.Sheets[sheet];
if (!ws["!ref"]) continue;
const range = XLSX.utils.decode_range(ws["!ref"]);
const selection = columnSelections.find(s => s.sheetName === sheet);
const columnsToTranslate = selection
? selection.columnIndices
: Array.from({ length: range.e.c - range.s.c + 1 }, (_, i) => i + range.s.c);
// Collect all text cells for batch translation
type CellRef = { addr: string; text: string };
const cellRefs: CellRef[] = [];
for (const colIdx of columnsToTranslate) {
// Start from row 1 to skip headers (row 0)
for (let r = range.s.r + 1; r <= range.e.r; r++) {
const addr = XLSX.utils.encode_cell({ r, c: colIdx });
const cell = ws[addr];
if (cell && cell.t === "s" && typeof cell.v === "string" && cell.v.trim()) {
cellRefs.push({ addr, text: cell.v });
}
}
}
if (cellRefs.length === 0) continue;
// Translate in batches of 50
const BATCH_SIZE = 50;
for (let i = 0; i < cellRefs.length; i += BATCH_SIZE) {
const batch = cellRefs.slice(i, i + BATCH_SIZE);
const translations = await replicateTranslateBatch(
batch.map(c => c.text),
targetLanguage
);
batch.forEach((cellRef, idx) => {
const cell = ws[cellRef.addr];
if (cell) {
cell.v = translations[idx];
if (cell.h) cell.h = translations[idx];
if (cell.r) cell.r = undefined;
if (cell.w) cell.w = translations[idx];
}
});
}
}
const out = XLSX.write(wb, { type: "buffer", bookType: "xlsx" });
return Buffer.from(out);
}

View File

@@ -0,0 +1,12 @@
declare module "pdf-parse/lib/pdf-parse.js" {
interface PdfData {
numpages: number;
numrender: number;
info: Record<string, unknown>;
metadata: Record<string, unknown>;
text: string;
version: string;
}
function pdfParse(dataBuffer: Buffer, options?: Record<string, unknown>): Promise<PdfData>;
export = pdfParse;
}

View File

@@ -0,0 +1,115 @@
import { PDFDocument, rgb, StandardFonts } from "pdf-lib";
import { replicateTranslateBatch } from "../replicate-translate";
type PdfParseResult = {
numpages: number;
text: string;
info: Record<string, unknown>;
};
async function parsePdf(buffer: Buffer): Promise<PdfParseResult> {
// Avoid Next.js issues with pdf-parse test file imports
// eslint-disable-next-line @typescript-eslint/no-var-requires
const pdfParse = require("pdf-parse/lib/pdf-parse.js");
return pdfParse(buffer);
}
function wrapText(text: string, maxCharsPerLine: number): string[] {
const words = text.split(/\s+/);
const lines: string[] = [];
let current = "";
for (const word of words) {
if ((current + " " + word).trim().length > maxCharsPerLine) {
if (current) lines.push(current);
current = word;
} else {
current = current ? current + " " + word : word;
}
}
if (current) lines.push(current);
return lines;
}
/**
* Translate a PDF buffer. Since PDFs don't support in-place text editing,
* this extracts text, translates it, and creates a new formatted PDF.
*/
export async function translatePdf(
buffer: Buffer,
targetLanguage: string,
sourceLanguage?: string
): Promise<Buffer> {
const parsed = await parsePdf(buffer);
const rawText = parsed.text;
// Split into paragraphs (separated by double newlines or page breaks)
const paragraphs = rawText
.split(/\n{2,}|\f/)
.map(p => p.replace(/\n/g, " ").trim())
.filter(p => p.length > 0);
if (paragraphs.length === 0) {
throw new Error("No extractable text found in PDF");
}
// Translate all paragraphs
const translations = await replicateTranslateBatch(paragraphs, targetLanguage);
// Build output PDF
const pdfDoc = await PDFDocument.create();
const font = await pdfDoc.embedFont(StandardFonts.Helvetica);
const boldFont = await pdfDoc.embedFont(StandardFonts.HelveticaBold);
const PAGE_WIDTH = 595;
const PAGE_HEIGHT = 842;
const MARGIN = 50;
const FONT_SIZE = 11;
const TITLE_SIZE = 13;
const LINE_HEIGHT = 16;
const MAX_LINE_CHARS = 80;
let page = pdfDoc.addPage([PAGE_WIDTH, PAGE_HEIGHT]);
let y = PAGE_HEIGHT - MARGIN;
function ensureSpace(needed: number) {
if (y - needed < MARGIN) {
page = pdfDoc.addPage([PAGE_WIDTH, PAGE_HEIGHT]);
y = PAGE_HEIGHT - MARGIN;
}
}
// Title
const title = `Translation to: ${targetLanguage}${sourceLanguage ? ` (from: ${sourceLanguage})` : ""}`;
ensureSpace(TITLE_SIZE + LINE_HEIGHT);
page.drawText(title, {
x: MARGIN,
y,
size: TITLE_SIZE,
font: boldFont,
color: rgb(0.2, 0.2, 0.7)
});
y -= TITLE_SIZE + LINE_HEIGHT;
// Draw translated paragraphs
for (const para of translations) {
const lines = wrapText(para, MAX_LINE_CHARS);
ensureSpace(lines.length * LINE_HEIGHT + LINE_HEIGHT);
for (const line of lines) {
ensureSpace(LINE_HEIGHT);
page.drawText(line, {
x: MARGIN,
y,
size: FONT_SIZE,
font,
color: rgb(0, 0, 0)
});
y -= LINE_HEIGHT;
}
y -= LINE_HEIGHT * 0.5; // paragraph gap
}
const pdfBytes = await pdfDoc.save();
return Buffer.from(pdfBytes);
}

View File

@@ -0,0 +1,244 @@
import { readSettings } from "./settings-store";
type CogOutput = {
status?: string;
output?: unknown;
error?: string;
};
type ReplicateOutput = string | string[] | { translation?: string; translated_text?: string; output?: string };
function extractText(output: unknown): string {
if (typeof output === "string") return output;
if (Array.isArray(output)) return output.join("");
if (output && typeof output === "object") {
const o = output as { translation?: string; translated_text?: string; output?: string };
return o.translation ?? o.translated_text ?? o.output ?? JSON.stringify(output);
}
throw new Error("Unexpected output format from model");
}
/**
* Parse modelVersion string into components.
* Accepts:
* "jigsawstack/text-translate:454df4c..." → { owner: "jigsawstack", model: "text-translate", hash: "454df4c..." }
* "jigsawstack/text-translate" → { owner: "jigsawstack", model: "text-translate", hash: undefined }
* "454df4c..." → { owner: undefined, model: undefined, hash: "454df4c..." }
*/
function parseModelVersion(mv: string) {
const [ownerModel, hash] = mv.split(":");
if (ownerModel.includes("/")) {
const [owner, model] = ownerModel.split("/");
return { owner, model, hash };
}
// Bare hash
return { owner: undefined, model: undefined, hash: ownerModel };
}
async function translateViaCloud(
text: string,
targetLanguage: string
): Promise<string> {
const settings = readSettings();
if (!settings.replicateApiToken) throw new Error("Replicate API token not configured");
if (!settings.jigsawApiKey) throw new Error("JigsawStack API key not configured");
const { owner, model, hash } = parseModelVersion(settings.modelVersion);
let url: string;
let body: Record<string, unknown>;
if (owner && model && !hash) {
// No version pinned — use the model's deployment endpoint (latest)
url = `https://api.replicate.com/v1/models/${owner}/${model}/predictions`;
body = { input: { text, api_key: settings.jigsawApiKey, target_language: targetLanguage } };
} else if (owner && model && hash) {
// Pinned version — use model endpoint with version (avoids 422 from bare /predictions)
url = `https://api.replicate.com/v1/models/${owner}/${model}/predictions`;
body = {
version: hash,
input: { text, api_key: settings.jigsawApiKey, target_language: targetLanguage }
};
} else {
// Bare hash only — use the generic predictions endpoint
url = "https://api.replicate.com/v1/predictions";
body = {
version: hash,
input: { text, api_key: settings.jigsawApiKey, target_language: targetLanguage }
};
}
const response = await fetch(url, {
method: "POST",
headers: {
"Authorization": `Bearer ${settings.replicateApiToken}`,
"Content-Type": "application/json",
"Prefer": "wait"
},
body: JSON.stringify(body)
});
if (!response.ok) {
const err = await response.text();
throw new Error(`Replicate API error: ${response.status} ${err}`);
}
const data = await response.json();
if (data.error) throw new Error(`Replicate model error: ${data.error}`);
return extractText(data.output as ReplicateOutput);
}
// The local Cog model handles exactly one prediction at a time.
// This queue ensures calls are strictly serialised regardless of
// how many batches / concurrent requests come in.
let localQueue: Promise<void> = Promise.resolve();
async function callLocalModel(text: string, targetLanguage: string): Promise<string> {
const settings = readSettings();
if (!settings.jigsawApiKey) throw new Error("JigsawStack API key not configured");
const endpoint = (settings.localEndpoint || "http://localhost:5030/predictions").replace(/\/$/, "");
const response = await fetch(endpoint, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
input: {
text,
api_key: settings.jigsawApiKey,
target_language: targetLanguage
}
})
});
if (!response.ok) {
const err = await response.text();
throw new Error(`Local model error: ${response.status} ${err}`);
}
const data: CogOutput = await response.json();
if (data.error) throw new Error(`Local model error: ${data.error}`);
const output = data.output !== undefined ? data.output : data;
return extractText(output);
}
async function translateViaLocal(
text: string,
targetLanguage: string
): Promise<string> {
// Enqueue — wait for any in-flight local call to finish first
let resolve!: () => void;
const slot = new Promise<void>(r => { resolve = r; });
const turn = localQueue.then(() => callLocalModel(text, targetLanguage));
localQueue = turn.then(resolve, resolve); // advance queue whether success or error
return turn;
}
export async function replicateTranslate(
text: string,
targetLanguage: string
): Promise<string> {
const { replicateMode } = readSettings();
return replicateMode === "local"
? translateViaLocal(text, targetLanguage)
: translateViaCloud(text, targetLanguage);
}
const SEPARATOR = "\n{{SEP}}\n";
const MAX_CHARS = 4800; // JigsawStack limit is 5000 — leave headroom
/**
* Split a single text that exceeds MAX_CHARS into translatable chunks,
* preferring paragraph/sentence boundaries.
*/
async function translateLongText(text: string, targetLanguage: string): Promise<string> {
const chunks: string[] = [];
let remaining = text;
while (remaining.length > MAX_CHARS) {
let splitAt = MAX_CHARS;
const para = remaining.lastIndexOf("\n", MAX_CHARS);
const sentence = remaining.lastIndexOf(". ", MAX_CHARS);
if (para > MAX_CHARS * 0.5) splitAt = para + 1;
else if (sentence > MAX_CHARS * 0.5) splitAt = sentence + 2;
chunks.push(remaining.slice(0, splitAt));
remaining = remaining.slice(splitAt);
}
if (remaining.trim()) chunks.push(remaining);
const results = await Promise.all(chunks.map(c => replicateTranslate(c, targetLanguage)));
return results.join(" ");
}
/**
* Build groups of texts that fit within MAX_CHARS when joined with SEPARATOR.
* Texts that individually exceed MAX_CHARS are kept alone for chunked translation.
*/
function buildBatches(texts: string[]): { indices: number[]; long: boolean }[] {
const batches: { indices: number[]; long: boolean }[] = [];
let current: number[] = [];
let currentLen = 0;
for (let i = 0; i < texts.length; i++) {
const t = texts[i];
if (t.length > MAX_CHARS) {
// Flush current group first
if (current.length > 0) { batches.push({ indices: current, long: false }); current = []; currentLen = 0; }
batches.push({ indices: [i], long: true });
continue;
}
const added = currentLen === 0 ? t.length : currentLen + SEPARATOR.length + t.length;
if (added > MAX_CHARS && current.length > 0) {
batches.push({ indices: current, long: false });
current = [i];
currentLen = t.length;
} else {
current.push(i);
currentLen = added;
}
}
if (current.length > 0) batches.push({ indices: current, long: false });
return batches;
}
export async function replicateTranslateBatch(
texts: string[],
targetLanguage: string
): Promise<string[]> {
if (texts.length === 0) return [];
const results: string[] = new Array(texts.length);
const batches = buildBatches(texts);
// Process batches sequentially to avoid hammering the model
for (const batch of batches) {
if (batch.long) {
// Single oversized text — chunk it
results[batch.indices[0]] = await translateLongText(texts[batch.indices[0]], targetLanguage);
} else if (batch.indices.length === 1) {
results[batch.indices[0]] = await replicateTranslate(texts[batch.indices[0]], targetLanguage);
} else {
// Multi-text batch within limit
const joined = batch.indices.map(i => texts[i]).join(SEPARATOR);
const translated = await replicateTranslate(joined, targetLanguage);
const parts = translated.split(SEPARATOR);
if (parts.length === batch.indices.length) {
batch.indices.forEach((idx, i) => { results[idx] = parts[i]; });
} else {
// Separator got translated — fall back to individual calls
const individual = await Promise.all(
batch.indices.map(i => replicateTranslate(texts[i], targetLanguage))
);
batch.indices.forEach((idx, i) => { results[idx] = individual[i]; });
}
}
}
return results;
}

80
utils/settings-store.ts Normal file
View File

@@ -0,0 +1,80 @@
import fs from "fs";
import path from "path";
export type ReplicateMode = "cloud" | "local";
export type Settings = {
replicateApiToken: string;
jigsawApiKey: string;
modelVersion: string;
replicateEnabled: boolean;
replicateMode: ReplicateMode;
localEndpoint: string;
adminPasswordHash: string;
};
const DEFAULT_SETTINGS: Settings = {
replicateApiToken: process.env["REPLICATE_API_TOKEN"] ?? "",
jigsawApiKey: process.env["JIGSAWSTACK_API_KEY"] ?? "",
modelVersion: "jigsawstack/text-translate:454df4c49941c05dea05175bd37686d0872c73c1f9366d1c2505db32ade52a89",
replicateEnabled: false,
replicateMode: (process.env["REPLICATE_MODE"] as ReplicateMode) ?? "cloud",
localEndpoint: process.env["LOCAL_MODEL_ENDPOINT"] ?? "http://localhost:5030/predictions",
adminPasswordHash: process.env["ADMIN_PASSWORD"] ?? "admin"
};
/**
* Resolve a writable path for settings.json.
* Priority:
* 1. SETTINGS_PATH env var (explicit override)
* 2. <cwd>/data/settings.json (default, works when data/ is writable)
* 3. /tmp/lingvai-settings.json (fallback for read-only containers)
*/
function resolveSettingsPath(): string {
if (process.env["SETTINGS_PATH"]) {
return process.env["SETTINGS_PATH"];
}
const primary = path.join(process.cwd(), "data", "settings.json");
try {
const dir = path.dirname(primary);
fs.mkdirSync(dir, { recursive: true });
// Test write access by opening with 'a' (append/create without truncating)
const fd = fs.openSync(primary, "a");
fs.closeSync(fd);
return primary;
} catch {
return path.join("/tmp", "lingvai-settings.json");
}
}
// Resolve once at module load so every call uses the same path
const SETTINGS_PATH = resolveSettingsPath();
if (SETTINGS_PATH.startsWith("/tmp")) {
console.warn(
`[lingvai] data/settings.json is not writable. ` +
`Settings will be stored at ${SETTINGS_PATH}. ` +
`Mount a writable volume at /app/data or set SETTINGS_PATH to persist across restarts.`
);
}
export function readSettings(): Settings {
try {
if (!fs.existsSync(SETTINGS_PATH)) {
return { ...DEFAULT_SETTINGS };
}
const raw = fs.readFileSync(SETTINGS_PATH, "utf-8");
return { ...DEFAULT_SETTINGS, ...JSON.parse(raw) };
} catch {
return { ...DEFAULT_SETTINGS };
}
}
export function writeSettings(updates: Partial<Settings>): Settings {
const current = readSettings();
const next = { ...current, ...updates };
const dir = path.dirname(SETTINGS_PATH);
fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(SETTINGS_PATH, JSON.stringify(next, null, 2), "utf-8");
return next;
}

View File

@@ -6109,10 +6109,10 @@ lines-and-columns@^1.1.6:
resolved "https://registry.yarnpkg.com/lines-and-columns/-/lines-and-columns-1.2.4.tgz#eca284f75d2965079309dc0ad9255abb2ebc1632" resolved "https://registry.yarnpkg.com/lines-and-columns/-/lines-and-columns-1.2.4.tgz#eca284f75d2965079309dc0ad9255abb2ebc1632"
integrity sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg== integrity sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg==
lingva-scraper@1.0.0: lingva-scraper@1.1.0:
version "1.0.0" version "1.1.0"
resolved "https://registry.yarnpkg.com/lingva-scraper/-/lingva-scraper-1.0.0.tgz#92da8b6dc4f922527b03075348bf4e52e1efad50" resolved "https://registry.yarnpkg.com/lingva-scraper/-/lingva-scraper-1.1.0.tgz#d3ad4f58b1fd65a666fdd1794cd9085d69830eac"
integrity sha512-7CRtXExRjKM+fq18EX33R+LB6QV0i5XNNUeAYE8xZgvFCZcua11VH4d+bV0n4+AO4kGu5mnLyHz17MY1Oa8q8g== integrity sha512-1BqktPfrqSYFvkeib7JUzkm0aJjN8YZn+ijzG7fewntB7sqj9pvR5KdPhkmvWmLlHx6PRDwnxKoVwCtH/7MkqQ==
dependencies: dependencies:
axios "^0.27.2" axios "^0.27.2"
cheerio "^1.0.0-rc.11" cheerio "^1.0.0-rc.11"