From e7cf771b96665343ea2a9a404d5d5ca88492dab1 Mon Sep 17 00:00:00 2001 From: patel-lyzr Date: Mon, 11 May 2026 19:58:11 +0530 Subject: [PATCH] feat: add contributing guidelines and enhance README structure --- CONTRIBUTING.md | 103 +++++++++++++ README.md | 373 ++++++++++++++++++++++++++++++++++++------------ aude.md | 112 --------------- claude.md | 2 +- 4 files changed, 383 insertions(+), 207 deletions(-) create mode 100644 CONTRIBUTING.md delete mode 100644 aude.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..349738c --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,103 @@ +# Contributing to Langship + +Thanks for your interest. This guide covers the dev setup, what to run +before opening a PR, and the conventions we follow. + +By contributing, you agree your contributions are licensed under the +[Apache License 2.0](./LICENSE) (the project's license). No CLA or DCO +sign-off is required. + +## Code of conduct + +Participation is governed by [CODE_OF_CONDUCT.md](./CODE_OF_CONDUCT.md). +Report concerns to . + +## Repo layout + +| Path | What | +|---|---| +| `cmd/flow/` | the `flow` server binary (API + Restate worker entry point) | +| `pkg/api/` | REST + SSE handlers | +| `pkg/orchestrator/`, `pkg/engine/` | DAG walk, execution context, events | +| `pkg/executors/` | node implementations (Build, Push, SAST, Approval, Promote, Deploy, …) | +| `pkg/awsdeploy/` | AWS Bedrock AgentCore adapter (STS, ECR, IAM, control-plane SigV4) | +| `pkg/storage/` | Mongo-backed stores (pipelines, runs, agents, credentials, environments) | +| `pkg/secrets/` | AES-GCM seal/open keyed off `FLOW_SECRET_KEY` | +| `web/` | Next.js UI (static export) | +| `langship-cli/` | the `langship` Python CLI | + +## Dev environment + +Backing services (Mongo, Restate, BuildKit, MinIO, registry) come up via +docker-compose; the Go API runs on the host. See the **Quickstart** and +**Dev (hot reload)** sections in the [README](./README.md) for the exact +commands. In short: + +```bash +make services # docker-compose: mongo, restate, buildkitd, registry, minio +make watch # Go API with air (rebuilds on .go change) — or `make serve` for a stable binary +cd web && pnpm dev # Next dev server, /api proxies to :8090 +``` + +Set `FLOW_SECRET_KEY` in your environment before working on anything that +touches credentials/environments — the API refuses credential writes +without it. + +CLI: + +```bash +pip install -e ./langship-cli +langship login --api-url http://localhost:8090 +``` + +## Before you open a PR + +- **Go**: `go build ./...` and `go test ./...` must pass. Run + `gofmt`/`go vet` on changed files. +- **Web**: `cd web && pnpm tsc --noEmit` must pass (and `pnpm build` if you + changed anything that affects the static export). +- **CLI**: `pip install -e ./langship-cli && langship --help` should work; + smoke-test any command you touched against a local server. +- Keep changes focused — one logical change per PR. +- Update the README / CLI README / `aude.md` if you changed behavior they + describe. + +## Conventions + +- **Commits**: imperative subject, ≤ ~72 chars (`feat:`/`fix:`/`refactor:` + prefixes are welcome but not required). Squash noise before pushing. +- **Branches**: `main` is the trunk; branch off it. Don't push directly to + `main` — open a PR. +- **Code style**: match the surrounding code. Comments explain *why*, not + *what*. Executors emit `__` summary objects on output items; + follow that pattern. +- **Errors**: wrap with context (`fmt.Errorf("deploy: %w", err)`); the API + layer maps `storage.ErrNotFound` → 404 and `storage.ErrAlreadyExists` → + 409. +- **No secrets in code or commits.** Anything sensitive goes through + `pkg/secrets` and is never returned by the API. + +## Adding a node executor + +1. Implement `executors.NodeExecutor` in `pkg/executors/.go`. Read + the trigger payload via `firstItem(inputs)` for `agentId` / + `environment` / `fromBranch`. Emit each input item with a `__` + summary attached. +2. Register it in `pkg/executors/registry.go` (pass any stores it needs + via `RegistryDeps`). +3. Add a catalog entry in `web/lib/node-catalog.ts` and a `Form` in + `web/components/canvas/node-form.tsx`. +4. If it pauses (like Approval), special-case it in + `pkg/orchestrator/walk.go` so it isn't wrapped in `restate.Run` (it + calls Restate context methods directly). + +## Reporting bugs + +Open a GitHub issue with: what you did, what you expected, what happened, +and the relevant logs (`flow` server output + the run's node logs from +`langship runs logs ` or `/api/executions/{id}/logs/{node}`). + +## Security + +Vulnerabilities go to privately — see +[SECURITY.md](./SECURITY.md). Do not open a public issue. diff --git a/README.md b/README.md index 7620463..0dc876a 100644 --- a/README.md +++ b/README.md @@ -1,130 +1,315 @@ -# flow +
-A durable agent-pipeline runtime — the engine + control plane behind -[Langship](./aude.md) (Deployment / Governance / Operations for agent apps). +# Langship -- **Pipeline canvas**: drag-and-drop CI/CD nodes (Trigger → Build → Test → - Eval → Policy → Approval → Deploy → Promote → Rollback) -- **n8n-shaped JSON** as the on-disk pipeline format; flow's own DAG + - executor catalog at runtime -- **Restate-backed durability**: every node wrapped in `restate.Run` — - crash-safe journaling, replay, awakeable-based human approvals -- **Real BuildKit OCI builds** with private-repo PAT support; pushes to - GHCR or any registry -- **GitHub webhook receiver** with HMAC verification, agent ↔ pipeline - attachments, branch filtering -- **Live execution view**: SSE-streamed per-node status + per-node log - lines (BuildKit progress, shell stdout, stub events), canvas-overlay - status rings +**Any framework. Any runtime.** -## Status +Open-source, self-hosted **deployment · governance · operations** for agent apps. +One pipeline definition → Kubernetes, AWS Bedrock AgentCore, or Vertex AI Agent +Engine — same governance everywhere. Works with LangGraph, LangChain, LlamaIndex, +CrewAI, AutoGen, or raw-SDK agents. No framework lock-in. -Pre-v0.1. Working but moving fast — APIs and node types may change. +[langship.sh](https://langship.sh) · [CLI](./langship-cli/) · [Positioning](./aude.md) · Apache 2.0 -## Architecture +
-``` -┌────────────────┐ ┌────────────────┐ -│ web (nginx) │ │ flow API (Go) │ ┌──────────────┐ -│ Next static │◄──►│ :8090 │◄────►│ Mongo │ -│ :3000 │ │ /api/* CRUD │ │ pipelines │ -└────────────────┘ │ /api/.../stream │ runs │ - │ SSE (live) │ │ agents │ - └────────┬───────┘ └──────────────┘ - │ - │ ingress.Send ┌──────────────┐ - ├──────────────►│ Restate │ - │ │ :8081 ing │ - │ (executes via │ :9070 admin│ - │ callback) └──────┬───────┘ - │ │ - │ ┌────────────────┘ - ▼ ▼ workflow callback - ┌──────────────────────┐ - │ flow service :9080 │ walkDurable + executors - │ Trigger/Build/Test/ │ → BuildKit (tcp:1234) for OCI - │ Eval/Policy/Approve │ → Registry (tcp:5000) for push - │ /Deploy/Promote/RB │ - └──────────────────────┘ -``` +> **Repo orientation.** This is the engine + control plane. The Go service is +> codenamed `flow`; the product is **Langship**. The CLI lives in +> [`langship-cli/`](./langship-cli/). -## Quickstart +--- + +- [What you get](#what-you-get) +- [5 minutes to a green run](#5-minutes-to-a-green-run) +- [Architecture](#architecture) +- [Repo map](#repo-map) +- [The `langship` CLI](#the-langship-cli) +- [Concepts](#concepts) +- [Nodes](#nodes) +- [Reference — env vars & make targets](#reference) +- [Contributing & community](#contributing--community) +- [License](#license) + +--- + +## What you get + +| | | +|---|---| +| **Pipelines as graphs** | Drag-and-drop CI/CD nodes — Trigger → Build → Scan/SAST → Eval → Policy → Approval → Deploy → Promote → Rollback. n8n-shaped JSON on disk; YAML in git is the source of truth. | +| **Governance is a node** | Approvals, policy checks, eval gates, PII/secret scans are first-class, reorderable steps in the graph — not middleware you can't see. | +| **Any runtime, one pipeline** | Same definition deploys to K8s, Bedrock AgentCore, or Vertex Agent Engine. (Today the Deploy node ships to **Bedrock AgentCore** end-to-end; K8s / Vertex are stubbed.) | +| **Durable by construction** | Restate journals every node (`restate.Run("node:", fn)`) — crash-safe replay, awakeable-based human approvals (timeout → auto-reject). | +| **GitOps promotion** | A Promote node opens/merges a PR `fromBranch → toBranch` on the agent's repo; the merge fires the next environment's pipeline. Promotion is an auditable event. | +| **Real OCI builds** | BuildKit solves your Dockerfile against the cloned repo, pushes to GHCR or any registry (private-repo PAT support). Mirror to N registries with the Push node. | +| **Operate, don't just deploy** | Live SSE log streams + canvas-overlay status rings; per-node logs archived to S3-compatible storage. | +| **Self-hosted, end-to-end** | Your cloud credentials, agent code, and run history never leave your network. Secrets AES-GCM sealed at rest. | +| **CLI-first** | `langship` — agents, envs, pipelines, creds, runs from your terminal. `git push` to ship. | + +--- + +## 5 minutes to a green run + +**0. Start everything.** Base compose bundles every service `flow` needs — +`mongo, restate, buildkitd, registry, minio, flow, web`. ```sh docker compose up -# UI: http://localhost:3000 -# API: http://localhost:8090 -# Restate: :8081 ingress, :9070 admin -# BuildKit: 127.0.0.1:1234 -# Registry: 127.0.0.1:5050 (host port; buildkitd pushes to registry:5000 internally) -# MinIO: 127.0.0.1:9000 (S3 API), :9001 (console; minio / minio12345) ``` -The base compose now bundles every service flow needs: **mongo, restate, -buildkitd, registry, minio, flow, web**. If a sibling stack already owns -one of those host ports (e.g. another langship-* set), stop that -container or override the port mapping in a `compose.override.yml`. +| | URL | +|---|---| +| UI | http://localhost:3000 | +| API | http://localhost:8090 | +| Restate | `:8081` ingress · `:9070` admin | +| BuildKit | `tcp://127.0.0.1:1234` | +| Registry | `127.0.0.1:5050` (host port; buildkitd pushes to `registry:5000` internally) | +| MinIO | `127.0.0.1:9000` (S3 API) · `:9001` console (`minio` / `minio12345`) | -## Dev (hot reload) +> If a sibling stack already owns one of those host ports, stop it or override the +> mapping in a `compose.override.yml`. -Three terminals: +**1. Install the CLI and point it at the API.** ```sh -# 1) backing services -docker compose up -d mongo restate -# (and buildkitd/registry from the standalone overlay or the sibling stack) +pip install -e ./langship-cli # optional: pip install pyyaml (for -o yaml) +langship login --api-url http://localhost:8090 +``` -# 2) Go API with air (rebuilds on .go changes) -make watch +**2. Register an agent (a git repo) and push a pipeline.** + +```sh +langship agents create --repo https://github.com/you/your-agent --pat ghp_... +langship pipelines push examples/hello.json # prints the new pipeline id +``` + +**3. Wire it into an environment, follow it, run it.** + +```sh +langship envs create dev -d "Auto-deploy on push" +langship envs add-pipeline dev +langship agents follow-env dev +langship agents trigger # → prints execution id(s) +``` + +**4. Watch it run.** + +```sh +langship runs logs -f # live SSE stream +# or open the UI: http://localhost:3000/executions/view?id= +``` + +That's the loop: `agent → env → pipeline → trigger → durable run → status`. + +### Hot-reload dev (three terminals) + +```sh +# 1) backing services only +docker compose up -d mongo restate # + buildkitd/registry from the overlay + +# 2) Go API with air — rebuilds on .go change +make watch # or `make serve` for a stable binary # 3) Next dev server with HMR; /api proxies to :8090 make dev ``` -Open `http://localhost:3000`. +`make watch` pre-exports env defaults matching the compose host ports — override +any at the CLI, e.g. `make watch MINIO_ENDPOINT=...`. Set `FLOW_SECRET_KEY` in +your shell before touching anything credential/environment-related (the API +refuses credential writes without it). -## Env vars +--- -The flow process (`./bin/flow serve` or `make watch`): +## Architecture -| Var | Default | Notes | -|---|---|---| -| `FLOW_ADDR` | `:8090` | API listen address | -| `FLOW_CORS_ORIGINS` | `*` | CSV allowlist | -| `FLOW_PUBLIC_URL` | (empty) | Externally-reachable base URL — used to render webhook callback URLs. Set to your `cloudflared` tunnel for GitHub webhooks. | -| `MONGO_URI` | (required) | e.g. `mongodb://localhost:27017` | -| `MONGO_DB` | `flow` | | -| `RESTATE_INGRESS_URL` | `http://localhost:8081` | | -| `RESTATE_ADMIN_URL` | `http://localhost:9070` | | -| `RESTATE_SERVICE_ADDR` | `:9080` | Service-endpoint listen addr | -| `RESTATE_DEPLOYMENT_URI` | `http://host.docker.internal:9080` | How Restate reaches us. In docker-compose this is overridden to `http://flow:9080`. | -| `BUILDKIT_HOST` | `tcp://127.0.0.1:1234` | BuildKit gRPC. In docker-compose: `tcp://buildkitd:1234` (or `host.docker.internal` when buildkitd is external). | +Three layers, all run by you: + +``` + ┌──────────────────────────────────────────────────────────┐ + CLI ──────► API / control plane (Go — pkg/api) │ + UI ──────► REST + SSE · agents/envs/pipelines/creds/runs · webhooks │ + └─────────────┬────────────────────────────────────────────┘ + │ RunAsync + ┌─────────────▼────────────────────────────────────────────┐ + │ Orchestration (Restate cluster + worker) │ + │ DAG walk (pkg/orchestrator) → executors (pkg/executors) │ + │ every node = restate.Run("node:", fn) │ + └─────────────┬────────────────────────────────────────────┘ + │ + ┌─────────────▼────────────────────────────────────────────┐ + │ Data: MongoDB (pipelines · runs · agents · creds · │ + │ environments) │ + │ MinIO / S3 (archived per-node logs, artifacts) │ + │ Postgres — Restate's backing store ONLY │ + │ pkg/secrets — AES-GCM seal/open (FLOW_SECRET_KEY)│ + └──────────────────────────────────────────────────────────┘ + ▲ + GitHub webhook │ /webhooks/github/{id} (HMAC-verified) + │ push → branch filter → dispatch run(s) +``` + +A run's lifecycle: webhook (or `langship agents trigger`) → the dispatcher walks +the agent's followed environments, applies each pipeline's branch filter, stamps +`agentId / environment / fromBranch` into the trigger payload, and calls +`orchestrator.RunAsync` → the DAG walker runs nodes in topological order, each +wrapped in `restate.Run` → terminal status written back to Mongo `runs` → SSE +clients (`/api/executions/{id}/stream`) get `node_started / node_log / +node_completed / node_error / done` events live. + +> **Why these choices** — Restate gives crash-safe journaling + awakeables (human +> approval that survives a restart) for free; Mongo is the app store; Postgres is +> *only* Restate's persistence and is never touched by app code; BuildKit does +> real OCI builds without a Docker daemon. See [aude.md](./aude.md) for the full +> rationale. + +--- + +## Repo map + +``` +cmd/flow/ the `flow` server binary (API + Restate worker entry point) +pkg/ + api/ REST + SSE handlers (agents, envs, pipelines, creds, runs, webhooks) + orchestrator/ DAG walk; Approval is special-cased out of restate.Run (it + calls restate.Set/Clear directly) + engine/ execution context, ExecutionEvent, the executor lookup + executors/ node implementations + the registry: + trigger · build · push · sast · imagescan · approval · + promote · deploy · (test/eval/policy/rollback stubs) + awsdeploy/ AWS Bedrock AgentCore adapter — STS AssumeRole, idempotent + ECR + IAM bootstrap, control-plane SigV4, endpoint wait + github/ REST helpers — webhook install/verify, PRs, merges + storage/ Mongo-backed stores: pipelines, runs, agents, credentials, + environments + secrets/ AES-GCM SealString/OpenString keyed off FLOW_SECRET_KEY +web/ Next.js 15 UI (static export) — canvas, runs, agents, + environments, credentials +langship-cli/ the `langship` Python CLI (Typer / Rich / httpx) +examples/ sample pipeline JSON +``` + +--- + +## The `langship` CLI + +The daily driver for agent devs; the bootstrap surface for platform engineers. + +```sh +pip install -e ./langship-cli # + pip install pyyaml for -o yaml +langship login --api-url http://localhost:8090 # saved to ~/.langship/config.toml + +# the loop +langship agents create --repo https://github.com/you/agent --pat ghp_... +langship pipelines push prod.yaml --id # create-or-update from a file +langship envs create prod -d "Strict gates" +langship envs add-pipeline prod +langship envs reorder prod # promotion order +langship agents follow-env prod +langship agents trigger +langship runs logs -f + +# credentials (server needs FLOW_SECRET_KEY) +langship creds create prod-aws --type aws \ + --aws-region us-east-1 --aws-account 123456789012 \ + --aws-role-arn arn:aws:iam::123456789012:role/FlowDeployRole +``` + +Command groups: `agents`, `envs`, `pipelines`, `creds`, `runs` — each with +`--help`. `-o json` / `-o yaml` on list/get commands. `LANGSHIP_API_URL` / +`LANGSHIP_TOKEN` override the saved config. Full reference: +[`langship-cli/README.md`](./langship-cli/README.md). + +--- ## Concepts - **Agent** — a registered git repo (URL + PAT). One-click GitHub webhook - install; `/webhooks/github/{id}` verifies HMAC and dispatches runs on - push. Agents attach to pipelines. -- **Pipeline** — a DAG of nodes built on the canvas (n8n-shape JSON - underneath). Saved to Mongo; loaded fresh per run. -- **Run** — one execution of a pipeline. Restate journals each node - (`restate.Run("node:", fn)`). Terminal status is written back to - Mongo's `runs` collection. + install; `/webhooks/github/{id}` verifies the HMAC signature and dispatches runs + on push. An agent **follows environments** (`agent.environments[]`); triggering + it runs the pipelines of every followed env. Agents may carry per-agent + credential overrides. +- **Environment** — a named, **ordered list of pipelines** (the promotion + sequence; reorderable). Global. Purely a sequencing container — per-deploy + config lives on the nodes, not the env. `dev` / `staging` / `prod` / custom. +- **Pipeline** — a DAG of nodes built on the canvas (n8n-shape JSON underneath), + stored in Mongo, loaded fresh per run. The Trigger node carries `fromBranch` / + `toBranch`; a per-pipeline branch filter decides which pipelines run for a given + push. +- **Credential** — a named record (`aws` / `gcp` / `kv`) in a global pool, with + optional per-agent overrides. Secret fields are AES-GCM sealed at rest with + `FLOW_SECRET_KEY`. Deploy / Push look one up by name. +- **Run** — one execution of a pipeline. Restate journals each node. Terminal + status is written back to Mongo's `runs` collection. The dispatcher stamps + `agentId`, `environment`, and `fromBranch` into the trigger payload; each node + emits a `__` summary object on its output items. - **Live view** — `/executions/view?id=…` subscribes to - `/api/executions/{id}/stream` (SSE) for `node_started`, - `node_completed`, `node_error`, **`node_log`**, and `done` events. + `/api/executions/{id}/stream` (SSE) for `node_started`, `node_completed`, + `node_error`, **`node_log`**, and `done` events; the canvas overlays status + rings on each node. -## Build node +--- -Two modes: +## Nodes -- `mode: "docker"` — BuildKit solves the Dockerfile against the cloned - repo and pushes to a registry. Auth: GHCR uses the agent's PAT - (`write:packages`); `localhost:*` / `registry:*` are anonymous + - insecure. Streams BuildKit's plain-mode progress as `node_log` events. -- `mode: "shell"` — escape hatch, runs `/bin/sh -c ` in the - cloned repo. Stdout/stderr line-streamed to the log channel. +| Node | What it does | +|---|---| +| **Trigger** | Entry point; carries `fromBranch` / `toBranch` for the branch filter + Promote. | +| **Build** | Clones the agent repo (`fromBranch`), builds an OCI image via BuildKit (`mode: docker`) or runs `/bin/sh -c ` in the clone (`mode: shell`). GHCR auth uses the agent's PAT (`write:packages`); `localhost:*` / `registry:*` are anonymous + insecure. Streams BuildKit's plain-mode progress as `node_log` events. | +| **Push** | Mirrors the built image to one or more registries (go-containerregistry's `crane`). | +| **SAST / ImageScan** | Sibling-container scanners — trivy / semgrep / gitleaks / SonarCloud / grype — over the source / image. Configurable severity threshold and fail-on-finding. | +| **Approval** | Pauses on a Restate awakeable until resumed via `POST /api/executions/{id}/resume` (UI or `langship`). `method: ui \| quorum \| auto`; optional `timeoutSeconds` → auto-reject. Two outputs: approved (0) / rejected (1). | +| **Promote** | Opens or merges a PR `fromBranch → toBranch` on the agent's repo via the GitHub API — idempotent (re-finds an existing PR). Modes: `open-pr` / `merge` / `merge-pr`. Emits `__promote` with the PR number / URL. The merge fires the next env's pipeline. | +| **Deploy** | Deploys the upstream Push image to **AWS Bedrock AgentCore** (`target: agentcore`; k8s / vertex are stubs). Looks up an `aws` credential by name, assumes the cross-account role, idempotently provisions the ECR repo + the shared `agentcore-runtime-role` IAM role, creates/updates the runtime, waits for the endpoint to be `READY`, and emits `__deploy` with the public invoke URL. | +| **Test / Eval / Policy / Rollback** | Stubbed for now — visible on the canvas, no-op executors. | + +Adding a node? See the "Adding a node executor" section in +[CONTRIBUTING.md](./CONTRIBUTING.md). + +--- + +## Reference + +### Env vars (the `flow` process — `./bin/flow serve`, `make watch`, or compose) + +| Var | Default | Notes | +|---|---|---| +| `FLOW_ADDR` | `:8090` | API listen address | +| `FLOW_CORS_ORIGINS` | `*` (compose: `http://localhost:3000`) | CSV allowlist | +| `FLOW_PUBLIC_URL` | (empty) | Externally-reachable base URL for webhook callback URLs. Set to your `cloudflared` tunnel for GitHub webhooks. | +| `FLOW_SECRET_KEY` | (unset → credential writes refused) | Master key for AES-GCM sealing of credentials/secrets. Any string; hashed to 32 bytes. **Losing it makes sealed data unrecoverable.** | +| `MONGO_URI` | (required; compose: `mongodb://localhost:27017`) | | +| `MONGO_DB` | `flow` | | +| `RESTATE_INGRESS_URL` | `http://localhost:8081` | | +| `RESTATE_ADMIN_URL` | `http://localhost:9070` | | +| `RESTATE_SERVICE_ADDR` | `:9080` | Service-endpoint listen addr | +| `RESTATE_DEPLOYMENT_URI` | `http://host.docker.internal:9080` | How Restate reaches us; compose overrides to `http://flow:9080` | +| `BUILDKIT_HOST` | `tcp://127.0.0.1:1234` | BuildKit gRPC; compose: `tcp://buildkitd:1234` | +| `MINIO_ENDPOINT` / `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` / `MINIO_BUCKET` / `MINIO_USE_SSL` | `127.0.0.1:9000` / `minio` / `minio12345` / `flow-logs` / `false` | Archived per-node log storage | + +CLI env: `LANGSHIP_API_URL`, `LANGSHIP_TOKEN` (override `~/.langship/config.toml`). + +### Make targets + +| | | +|---|---| +| `make build` | build the web bundle then the Go binary (`bin/flow`) | +| `make build-go` | Go binary only (expects `web/dist` to exist) | +| `make serve` | `build-go` then `./bin/flow serve` — stable binary | +| `make watch` | Go API with `air` (rebuilds on `.go` change), env defaults pre-exported | +| `make dev` | Next dev server with HMR (`/api` proxies to `:8090`) | +| `make web` | build the Next static export | +| `make test` / `make vet` / `make tidy` | `go test ./...` / `go vet ./...` / `go mod tidy` | + +--- + +## Contributing & community + +- **Issues & discussion** — open a GitHub issue for bugs and feature requests. Search first. +- **Contributing** — [CONTRIBUTING.md](./CONTRIBUTING.md): dev setup, what to run before a PR, conventions, how to add a node executor. Contributions accepted under Apache 2.0. +- **Code of conduct** — [CODE_OF_CONDUCT.md](./CODE_OF_CONDUCT.md) (Contributor Covenant). Report concerns to . +- **Security** — **do not** file public issues for vulnerabilities. See [SECURITY.md](./SECURITY.md) — report privately to . ## License -Apache 2.0 +[Apache 2.0](./LICENSE) diff --git a/aude.md b/aude.md deleted file mode 100644 index 8cb8bf5..0000000 --- a/aude.md +++ /dev/null @@ -1,112 +0,0 @@ -# Langship - -Langship is a framework-agnostic deployment, governance, and operations layer for agent applications. It supports LangChain/LangGraph and other agent frameworks (LlamaIndex, CrewAI, AutoGen, Pydantic AI, raw SDK agents, etc.) — not tied to any single framework. - -## Three pillars - -- **Deployment** — packaging, versioning, rollouts, env/secret management for agent apps -- **Governance** — policies, budgets, approvals, audit logs, tenant isolation, safety filters -- **Operations** — monitoring, replay/debug, incident response, cost & performance management - -## Deployment targets - -Langship is multi-runtime. Supported deployment targets include: - -- **Kubernetes** (self-hosted / any cloud) -- **AWS Bedrock AgentCore Runtime** -- **GCP Vertex AI Agent Engine** - -The deployment layer abstracts over these runtimes so the same agent app definition can ship to any of them. Governance and operations policies must apply uniformly across runtimes. - -## Positioning - -Closer to a "platform for agents" (Kubernetes/Datadog-style) than a single-framework tool like LangSmith or LangGraph Platform. - -## Top-level scope — Project - -Langship's top-level scoping primitive is a **Project**. A Project owns: users + roles, environments, workflows, secrets, deployment configs, audit log. Self-hosted Langship typically serves several teams, so Projects isolate one team/agent from another from day one (rather than retrofitting tenancy later). API paths are scoped: `/projects/:id/...`. Note: "Project" in this doc always refers to the Langship Project; cloud-provider projects (e.g., GCP project) are always qualified with the provider name. - -## CI/CD — configurable workflow builder - -CI/CD is **not** a fixed pipeline. It is a **configurable, drag-and-drop workflow builder** where pipelines are graphs of nodes (triggers, build, test, eval, policy, approval, deploy, promote, rollback). The visual canvas is the UI; YAML in git is the source of truth (GitOps). - -- **Two modes** — replace CI entirely (Langship runs build/test/eval/deploy) or CD + release-gates only (external CI hands off an artifact, Langship picks up at eval/governance/deploy). -- **Role-based views** — agent devs, platform/DevOps, governance owners see different node palettes on the same underlying graph. -- **Environments are first-class** — dev, staging, prod each have their own pipeline. Per-env config (secrets, runtime, scaling) is separate from the graph. Different runtimes per env are expected (e.g., dev on K8s, prod on Vertex Agent Engine). -- **Promotion is gated and branching-strategy-driven** — evals must pass → approval gate (human, automated policy, or quorum) → `Promote` node executes per project's branching strategy (trunk-based, env-branches, release branches, or custom). Promotion and rollback are auditable events. -- **Governance is a node, not a wrapper** — policies/approvals/budget gates are first-class, visible, reorderable steps in the graph. - - -## Deployment model — self-hosted - -Langship is **self-hosted by the customer**. The customer runs the whole stack (API server, Restate, Postgres, workers, secrets manager) in their own infrastructure. Distribution is via Helm chart / installer / Docker Compose for local dev. - -**Why self-hosted:** -- Customer's cloud credentials, agent code, eval data, and audit logs never leave their network — strong fit for the governance-focused positioning -- Clear regulatory story for finance/healthcare/gov buyers who can't adopt hosted control planes -- Simpler security model: no cross-tenant credential storage, no proxy of LLM traffic -- Customer's compliance team monitors logs in systems they already operate - -**Trade-offs accepted:** -- Higher friction to adopt vs. hosted SaaS — installer/upgrade UX matters more -- Support is harder — no production access by default; need good telemetry-with-consent + clear runbooks -- Distribution: ship a Helm chart as primary path; Docker Compose for local dev / small teams - -**Hosted offering may come later** as a managed deployment of the same stack, but the product is designed self-hosted-first. CLI/UI/API contracts assume the server is something the customer operates. - -## Server architecture — three layers - -The "server" is actually three layers, all running in the customer's infrastructure: - -1. **API / control plane** — REST or gRPC. CLI and UI call this. Handles auth, RBAC, workflow CRUD, run triggers, approvals, audit queries. Stateless app servers. -2. **Orchestration layer** — Restate (primary) cluster + worker pool. Workers execute node logic (build, eval, deploy, etc.). Long-lived and durable. -3. **Data layer** — MongoDB (runs, approvals, audit, users, projects, workflows index); S3-compatible object store (artifacts, large eval outputs, trace blobs); secrets manager (Vault or cloud-native KMS) for cloud credentials. Postgres runs alongside, but **only** as Restate's required persistence backend — never accessed by Langship app code. - -Plus a **GitOps sync** component (initially inside the API, possibly its own service later) that watches git refs and triggers workflows on push/tag/merge events. - -### Minimal v0 server - -For the CLI-first vertical slice: -- One API server process, Postgres-backed -- One Restate (Restate Cloud is fine for prototyping; final product self-hosts Restate too) -- One worker process executing a few node types -- CLI talks to the API -- No UI yet - -That's the smallest thing that closes the loop: `langship deploy` → API receives → Restate workflow runs → status updates → CLI shows result. - -## User flow — two roles, two experiences - -Langship has two distinct user journeys. The product must serve both well. - -### Platform engineer — one-time project setup - -Heavy, infrequent. Done once per project, occasionally revisited. Mixes CLI + UI. - -1. **Environments** — define dev / staging / prod (and any custom envs like `eu-prod`, `preview`) -2. **Branching strategy** — trunk-based / env-branches / release-branches / custom -3. **CI/CD pipeline with stages** — the workflow graph (build → eval → approval → deploy → promote, etc.) per env -4. **Deployment configs** — per-env cloud credentials and runtime targets (e.g., K8s cluster X for dev, Bedrock AgentCore account Y for staging, Vertex Agent Engine project Z for prod) - -Output: a configured project that agent developers consume. - -### Agent developer — repeatable deploy loop - -Light, frequent. Daily driver. CLI-first. - -5. **Drop in agent repo** — link the repo to a Langship project (GitOps: Langship watches refs per env, not a one-time blob upload) -6. **Select** — pick project / pipeline / env target -7. **Deploy** — trigger the run; pipeline executes; agent ships - -### Implications for product surface - -- **CLI is the daily-use surface** for agent devs (steps 5–7) and the bootstrap surface for platform engineers (steps 1–4 initially). -- **UI is the visualization + governance surface** — workflow canvas, run history, approval inbox, audit log, release-flow view across envs. Comes after CLI proves the model. -- **Build CLI first.** A working CLI gives end-to-end usefulness sooner; UI built on top of unproven flows risks designing for the wrong thing. -- **Agent repo is git-based, not uploaded.** Langship references the repo by URL + ref; pipelines trigger on push/tag/merge events per env's branching strategy. Preserves traceability, reproducibility, and the GitOps story. - -## Design principle - -Core APIs and data models must stay framework-agnostic. The key abstraction is a common interface across frameworks (runs, traces via OpenTelemetry/OpenLLMetry) so governance and operations policies apply uniformly regardless of the underlying agent framework. Avoid LangChain-only assumptions in core abstractions. - -Engine Restate) must stay wrapped behind Langship's own DSL — users never see the engine directly. Swapping later is possible but disruptive; pick deliberately. diff --git a/claude.md b/claude.md index 8cb8bf5..b8cc1e6 100644 --- a/claude.md +++ b/claude.md @@ -109,4 +109,4 @@ Light, frequent. Daily driver. CLI-first. Core APIs and data models must stay framework-agnostic. The key abstraction is a common interface across frameworks (runs, traces via OpenTelemetry/OpenLLMetry) so governance and operations policies apply uniformly regardless of the underlying agent framework. Avoid LangChain-only assumptions in core abstractions. -Engine Restate) must stay wrapped behind Langship's own DSL — users never see the engine directly. Swapping later is possible but disruptive; pick deliberately. +Engine Restate must stay wrapped behind Langship's own DSL — users never see the engine directly. Swapping later is possible but disruptive; pick deliberately.