mirror of
https://github.com/runbear-io/beardrive.git
synced 2026-08-25 08:08:08 +02:00
feat(webapp): refuse to share a file that looks like it holds credentials (BEA-111) (#137)
Minting a share link ran zero content checks: a file holding an AWS-shaped key became a public URL on one click, and the CLI printed nothing but the link. handleShareCreate now reads the first 1 MiB and runs six anchored rules between the synced-path check and Shares.Create, answering 409 with rule ids and line numbers unless the request carries confirm: true. The matched text never leaves scanSecrets — not into the body, not into a log line. TestShareSecretNeverEchoed greps both for the planted string, because a 409 body is the easiest place in this codebase to leak it. Both callers carry the override, since the gate alone would turn any false positive into a hard block with no way out: `bdrive share --force`, and the browser's Share-anyway dialog on modalConfirm (no new component). A path that already has a live link skips the scan — its content is public already, so withholding the URL protects nobody — but alreadyPublic drops links whose creator left the org, since those 404 at /s/ and would otherwise wave a secrets file straight through. A failed blob read is 503, not a silent pass: the repo's "degrade rather than fail" posture is for sync cycles, and a check that skips itself on a storage hiccup is the false confidence this exists to remove. Every user-facing string says the file was checked at the moment you shared it. A link serves the file's LATEST content forever, so a key written into an already-shared file is never caught — that open loop stays open, and the copy is the only thing stopping v1 from claiming otherwise. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
594a027c15
commit
d3d92bf904
@@ -486,8 +486,27 @@ carry a strict CSP, never see auth cookies, and sit behind a generous
|
||||
per-IP rate limit (`share_rpm`), so a malicious shared file's scripts
|
||||
can't touch hub sessions and a scraper can't turn the hub into a CDN.
|
||||
Any org member can mint links, and a link is public to whoever has the
|
||||
URL — don't share folders that hold secrets, and note a LAN-bound hub
|
||||
means LAN-only links.
|
||||
URL — note that a LAN-bound hub means LAN-only links.
|
||||
|
||||
Before it mints, the hub reads the **first 1 MiB** of the file and refuses
|
||||
if it finds credential-shaped strings — an AWS access key, a private key
|
||||
block, a GitHub, Slack, GitLab or OpenAI token. It names the rule and the
|
||||
line, never the matched text, and nothing is shared:
|
||||
|
||||
```
|
||||
$ bdrive share deploy.md
|
||||
Error: deploy.md looks like it contains credentials (checked at the moment you shared it):
|
||||
line 12 aws_access_key_id
|
||||
line 40 private_key
|
||||
Nothing was shared. Re-run with --force if that is intentional.
|
||||
```
|
||||
|
||||
`--force` shares it anyway (the web UI's Share button asks the same
|
||||
question and offers **Share anyway**). Know the two limits: the check runs
|
||||
**at the moment you share**, and a link serves the file's latest content
|
||||
forever — so a key committed into an already-shared file is never caught —
|
||||
and it only reads the first 1 MiB. It shortens the odds; it is not a
|
||||
promise that a file is clean.
|
||||
|
||||
### Agent integration
|
||||
|
||||
|
||||
@@ -258,6 +258,18 @@ classDiagram
|
||||
}
|
||||
note for ShareDB "A share is now re-checked at READ time, not only at mint time: shareCreatorStillBelongs refuses /s/<token> once its creator has left the project's org, so a link cannot outlive the access that justified it"
|
||||
|
||||
class secretScan {
|
||||
<<secrets.go>>
|
||||
secretScanLimit = 1 MiB
|
||||
secretRules six anchored regexes
|
||||
+scanSecrets(buf) []secretFinding
|
||||
}
|
||||
class secretFinding {
|
||||
+Rule string
|
||||
+Line int
|
||||
}
|
||||
note for secretScan "Mint-time gate on handleShareCreate: the one place a member turns private bytes into a public URL is the one place the bytes are read first. It returns rule ids and LINE NUMBERS only — the matched text never reaches a response body, a log line, or a metric label, the same rule ReadLedger keeps for actor identity. Bypassed by confirm:true (bdrive share --force, the UI's Share anyway) and by Server.alreadyPublic, since a path that already has a live link is public already. Fails CLOSED: an unreadable blob is 503, not a silent pass"
|
||||
|
||||
class sandboxInline {
|
||||
<<Server, every bytes-out route>>
|
||||
inlineMarkup(ct) / inlineType(ct)
|
||||
@@ -371,6 +383,8 @@ classDiagram
|
||||
reservations *-- grant
|
||||
reservations ..> QuotaProvider : CheckWrite(size + outstanding), RecordUsage on landing
|
||||
ShareDB ..> QuotaProvider : CheckRead before the stream, RecordEgress after
|
||||
Server ..> secretScan : handleShareCreate scans the first 1 MiB unless confirmed or alreadyPublic
|
||||
secretScan ..> secretFinding
|
||||
Server ..> countingWriter : every bytes-out route that bills
|
||||
reservations ..> Backend : reconcile — did the blob land
|
||||
Server *-- journalDoor : /store/* is the only way a device writes
|
||||
|
||||
+44
-2
@@ -23,6 +23,7 @@ func shareCmd() *cobra.Command {
|
||||
var expires time.Duration
|
||||
var list bool
|
||||
var revoke string
|
||||
var force bool
|
||||
c := &cobra.Command{
|
||||
Use: "share [file]",
|
||||
Short: "Share a synced file publicly by URL",
|
||||
@@ -32,9 +33,16 @@ PDFs open inline — with no account. The link always serves the file's
|
||||
latest synced content and lives until revoked (use --expires to limit it).
|
||||
|
||||
The file must be inside an initialized project and already synced (the
|
||||
daemon usually gets it there within seconds of saving).`,
|
||||
daemon usually gets it there within seconds of saving).
|
||||
|
||||
Before minting, the hub reads the first 1 MiB of the file and refuses if it
|
||||
finds credential-shaped strings — an AWS key, a private key block, a GitHub
|
||||
or Slack or GitLab token. That check happens at the moment you share: a link
|
||||
serves the file's latest content, so later changes are never re-checked.
|
||||
Use --force to share anyway.`,
|
||||
Example: ` bdrive share wiki/report.html
|
||||
bdrive share deck.pdf --expires 168h # link dies after a week
|
||||
bdrive share deploy.md --force # share despite a credentials warning
|
||||
bdrive share --list # this project's links
|
||||
bdrive share --revoke <token-or-url>`,
|
||||
Args: cobra.MaximumNArgs(1),
|
||||
@@ -68,10 +76,13 @@ daemon usually gets it there within seconds of saving).`,
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
body := map[string]string{"path": filepath.ToSlash(rel)}
|
||||
body := map[string]any{"path": filepath.ToSlash(rel)}
|
||||
if expires > 0 {
|
||||
body["expires_in"] = expires.String()
|
||||
}
|
||||
if force {
|
||||
body["confirm"] = true
|
||||
}
|
||||
data, _ := json.Marshal(body)
|
||||
resp, err := serverDo(http.MethodPost, server+"/api/p/"+projectID+"/shares", settings.Token, data)
|
||||
if err != nil {
|
||||
@@ -81,6 +92,11 @@ daemon usually gets it there within seconds of saving).`,
|
||||
if resp.StatusCode == http.StatusNotFound {
|
||||
return fmt.Errorf("%s (if you just saved it, wait a few seconds for the daemon or run `bdrive sync`)", strings.TrimSpace(readBody(resp)))
|
||||
}
|
||||
// Before the generic fallthrough: httpBodyError would print the raw
|
||||
// JSON, and this is the one status the user can act on.
|
||||
if resp.StatusCode == http.StatusConflict {
|
||||
return secretsFound(filepath.ToSlash(rel), resp)
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpBodyError(resp)
|
||||
}
|
||||
@@ -101,12 +117,38 @@ daemon usually gets it there within seconds of saving).`,
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().BoolVar(&force, "force", false, "share it even if the file looks like it contains credentials")
|
||||
c.Flags().DurationVar(&expires, "expires", 0, "make the link expire (e.g. 24h, 168h); default: lives until revoked")
|
||||
c.Flags().BoolVar(&list, "list", false, "list this project's share links")
|
||||
c.Flags().StringVar(&revoke, "revoke", "", "revoke a share link (token or full URL)")
|
||||
return c
|
||||
}
|
||||
|
||||
// secretsFound turns the hub's 409 into the message that stops the share.
|
||||
// The wording is load-bearing: a link serves the file's LATEST content
|
||||
// forever, so this says the file was checked *at the moment you shared it* —
|
||||
// never that the file is clean.
|
||||
func secretsFound(rel string, resp *http.Response) error {
|
||||
var out struct {
|
||||
Findings []struct {
|
||||
Rule string `json:"rule"`
|
||||
Line int `json:"line"`
|
||||
} `json:"findings"`
|
||||
}
|
||||
// Not readBody: that does a single 256-byte Read and would truncate a
|
||||
// long findings list mid-JSON.
|
||||
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil || len(out.Findings) == 0 {
|
||||
return fmt.Errorf("%s looks like it contains credentials; nothing was shared (re-run with --force if that is intentional)", rel)
|
||||
}
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "%s looks like it contains credentials (checked at the moment you shared it):\n", rel)
|
||||
for _, f := range out.Findings {
|
||||
fmt.Fprintf(&b, " line %-4d %s\n", f.Line, f.Rule)
|
||||
}
|
||||
b.WriteString("Nothing was shared. Re-run with --force if that is intentional.")
|
||||
return fmt.Errorf("%s", b.String())
|
||||
}
|
||||
|
||||
func listShares(settings config.Settings) error {
|
||||
root, proj, err := findProject(".")
|
||||
if err != nil {
|
||||
|
||||
@@ -496,6 +496,10 @@ func startTestHub(t *testing.T) *httptest.Server {
|
||||
}
|
||||
srv := &Server{Root: be, Projects: db, Device: webDevice, Upload: UploadConfig{Enabled: true}}
|
||||
srv.Devices, _ = OpenDeviceRegistry(filepath.Join(state, "devices.json"))
|
||||
srv.Shares, err = OpenShareDB(filepath.Join(state, "shares.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
auth, err := OpenBuiltinAuth(filepath.Join(state, "auth.json"), false, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -713,3 +717,68 @@ func TestCLITemplateRefusals(t *testing.T) {
|
||||
t.Fatal("a refused init still initialized the folder")
|
||||
}
|
||||
}
|
||||
|
||||
// `bdrive share` refuses a file that looks like it holds credentials, and
|
||||
// --force is the way past it. This is the flow BEA-111 exists for: the CLI
|
||||
// used to print the URL and nothing else.
|
||||
func TestCLIShareSecretGate(t *testing.T) {
|
||||
e := newCLIEnv(t)
|
||||
run := e.run
|
||||
|
||||
work := t.TempDir()
|
||||
// Fabricated, AWS-shaped. Not a credential.
|
||||
const plantedKey = "AKIAIOSFODNN7EXAMPLE"
|
||||
if err := os.WriteFile(filepath.Join(work, "deploy.md"), []byte(
|
||||
"# Deploy\n\nexport AWS_ACCESS_KEY_ID="+plantedKey+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(work, "clean.md"), []byte("# Clean\n\nnothing here\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if out, err := run(work, "init", "--name", "share-gate", "--yes"); err != nil {
|
||||
t.Fatalf("init: %v\n%s", err, out)
|
||||
}
|
||||
defer run(work, "stop", work)
|
||||
if out, err := run(work, "sync"); err != nil {
|
||||
t.Fatalf("sync: %v\n%s", err, out)
|
||||
}
|
||||
|
||||
// A clean file shares as it always did.
|
||||
out, err := run(work, "share", "clean.md")
|
||||
if err != nil || !strings.Contains(out, "/s/") {
|
||||
t.Fatalf("clean file did not share: %v\n%s", err, out)
|
||||
}
|
||||
|
||||
// The planted file is refused, by rule and line, and names the way out.
|
||||
out, err = run(work, "share", "deploy.md")
|
||||
if err == nil {
|
||||
t.Fatalf("share of a file holding a key succeeded:\n%s", out)
|
||||
}
|
||||
for _, want := range []string{"aws_access_key_id", "line 3", "--force", "at the moment you shared it"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Fatalf("share refusal missing %q:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
if strings.Contains(out, plantedKey) {
|
||||
t.Fatalf("the CLI echoed the secret back:\n%s", out)
|
||||
}
|
||||
if strings.Contains(out, "/s/") {
|
||||
t.Fatalf("a refused share still printed a URL:\n%s", out)
|
||||
}
|
||||
|
||||
// --force is the override, and the link it mints works.
|
||||
out, err = run(work, "share", "deploy.md", "--force")
|
||||
if err != nil || !strings.Contains(out, "/s/") {
|
||||
t.Fatalf("share --force: %v\n%s", err, out)
|
||||
}
|
||||
link := strings.TrimSpace(strings.SplitN(out, "\n", 2)[0])
|
||||
resp, err := http.Get(link)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != 200 || !strings.Contains(string(body), "Deploy") {
|
||||
t.Fatalf("forced link does not serve: %d %s", resp.StatusCode, body)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -239,6 +239,9 @@ func seedE2E(t *testing.T, state, prefix, projectID string) {
|
||||
put("guide.md", "# Guide\n\nSecond version of the guide, with more detail.\n", 2*time.Hour)
|
||||
put("notes/readme.md", "# Notes\n\nNested folder content.\n", 24*time.Hour)
|
||||
put("notes/deep/topic.md", "# Topic\n\nDeeply nested file.\n", 24*time.Hour)
|
||||
// The share gate needs something to fire on. Fabricated, AWS-shaped —
|
||||
// not a credential, and the only seeded file that holds one.
|
||||
put("deploy.md", "# Deploy\n\nexport AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\n", 24*time.Hour)
|
||||
// Tiny valid PNG (1x1), enough to exercise the binary/download path.
|
||||
png := "\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4\x89" +
|
||||
"\x00\x00\x00\nIDATx\x9cc\x00\x01\x00\x00\x05\x00\x01\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82"
|
||||
|
||||
@@ -220,6 +220,49 @@ test("share mints a public link that serves the file, revoke kills it", async ({
|
||||
expect(gone.status()).toBe(404);
|
||||
});
|
||||
|
||||
// BEA-111: sharing a file that looks like it holds credentials asks first.
|
||||
// Cancel mints nothing; Share anyway mints the link it would have.
|
||||
test("share on a file holding a key asks before it mints, and Cancel mints nothing", async ({
|
||||
page,
|
||||
}) => {
|
||||
await login(page);
|
||||
const pid = await wikiId(page);
|
||||
await page.goto(`/${pid}/deploy.md`);
|
||||
|
||||
// Cancel: the dialog names the finding, and no link exists afterwards.
|
||||
await page.click("#share-btn");
|
||||
const dialog = page.locator(".modal", { hasText: "This file may contain credentials" });
|
||||
await expect(dialog).toBeVisible();
|
||||
await expect(dialog).toContainText("an AWS access key (line 3)");
|
||||
// The copy may only ever claim what was true at mint time.
|
||||
await expect(dialog).toContainText("at the moment you share it");
|
||||
await expect(dialog).toContainText("later changes are never checked");
|
||||
// …and it must never echo the thing it found.
|
||||
await expect(dialog).not.toContainText("AKIA");
|
||||
await dialog.locator("button:has-text('Cancel')").click();
|
||||
await expect(page.locator(".modal-url")).toHaveCount(0);
|
||||
const before = await (await page.request.get(`/api/p/${pid}/shares`)).json();
|
||||
expect(before.shares.filter((s: { path: string }) => s.path === "deploy.md")).toHaveLength(0);
|
||||
|
||||
// Share anyway: the same click, carried through.
|
||||
await page.click("#share-btn");
|
||||
await page.locator(".modal button:has-text('Share anyway')").click();
|
||||
const url = (await page.locator(".modal-url").textContent())!;
|
||||
expect(url).toContain("/s/");
|
||||
const publicRes = await page.request.get(url);
|
||||
expect(publicRes.status()).toBe(200);
|
||||
await page.click(".modal button:has-text('Done')");
|
||||
|
||||
// Already public: a second Share hands back the same link without asking.
|
||||
await page.reload();
|
||||
await page.click("#share-btn");
|
||||
await expect(page.locator(".modal", { hasText: "This file may contain credentials" })).toHaveCount(0);
|
||||
expect(await page.locator(".modal-url").textContent()).toBe(url);
|
||||
await page.click(".modal button:has-text('Done')");
|
||||
|
||||
await page.request.delete(`/api/shares/${url.split("/s/")[1]}`);
|
||||
});
|
||||
|
||||
// BEA-29: the CLI has had --expires all along; the dialog now offers it on
|
||||
// the link you just minted, without changing that link's URL.
|
||||
test("share dialog sets an expiry on the link it just minted", async ({ page }) => {
|
||||
|
||||
@@ -32,6 +32,33 @@ import { Insights, useInsightsDevices } from "../components/Insights";
|
||||
import { HistoryView, historyTitle } from "../components/HistoryView";
|
||||
import { VersionBanner } from "../components/VersionBanner";
|
||||
|
||||
// The hub's six share-time credential rules, in words. Only one caller
|
||||
// (shareNow), so it lives here rather than in its own file.
|
||||
const SECRET_LABELS: Record<string, string> = {
|
||||
aws_access_key_id: "an AWS access key",
|
||||
openai_api_key: "an OpenAI API key",
|
||||
github_pat: "a GitHub token",
|
||||
slack_token: "a Slack token",
|
||||
private_key: "a private key",
|
||||
gitlab_pat: "a GitLab token",
|
||||
};
|
||||
|
||||
// secretsMessage phrases the 409 for the confirm dialog. The second sentence
|
||||
// is not decoration: a link always serves the file's LATEST content, so the
|
||||
// copy may only ever claim what was true at the moment of sharing — never
|
||||
// that the file is clean.
|
||||
function secretsMessage(findings: { rule: string; line: number }[] = []): string {
|
||||
const parts = findings.map((f) => `${SECRET_LABELS[f.rule] ?? f.rule} (line ${f.line})`);
|
||||
const what =
|
||||
parts.length > 1
|
||||
? parts.slice(0, -1).join(", ") + " and " + parts[parts.length - 1]
|
||||
: parts[0] || "something credential-shaped";
|
||||
return (
|
||||
`BearDrive found ${what} in this file. The check covers the file at the moment you share it — ` +
|
||||
`a link always serves the file's latest content, so later changes are never checked. Share anyway?`
|
||||
);
|
||||
}
|
||||
|
||||
// The browsing surface shared by hub projects and single-volume mode: the
|
||||
// file tree, folder listings, file views, and every topbar action. Sidebar
|
||||
// chrome (vault header, project nav, org bar) is injected by the caller;
|
||||
@@ -208,12 +235,23 @@ export default function Browser(props: {
|
||||
|
||||
const shareNow = useCallback(async () => {
|
||||
// Shares are per-file; a selected folder has nothing to mint.
|
||||
try {
|
||||
const r = await fetch(apiBase + "shares", {
|
||||
const post = (confirm: boolean) =>
|
||||
fetch(apiBase + "shares", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ path }),
|
||||
body: JSON.stringify(confirm ? { path, confirm: true } : { path }),
|
||||
});
|
||||
try {
|
||||
let r = await post(false);
|
||||
// 409: the hub found credential-shaped strings and minted nothing.
|
||||
// Read the structured body — this is a raw fetch, so it never passes
|
||||
// through errorFor() in api/http.ts, which would flatten it to a toast.
|
||||
if (r.status === 409) {
|
||||
const { findings } = (await r.json()) as { findings?: { rule: string; line: number }[] };
|
||||
if (!(await modalConfirm("This file may contain credentials", secretsMessage(findings), "Share anyway", true)))
|
||||
return; // Cancel mints nothing, and fires no share_created
|
||||
r = await post(true);
|
||||
}
|
||||
if (!r.ok) throw new Error(await r.text());
|
||||
const s = await r.json();
|
||||
// Fired here rather than by the table in api/http.ts, because this is
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
package webapp
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"regexp"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// Minting a share link is the one place on the hub where a member turns private
|
||||
// bytes into a public URL, so it is the one place worth reading the bytes
|
||||
// first. The check is deliberately narrow: six anchored rules over the first
|
||||
// 1 MiB, at mint time only.
|
||||
//
|
||||
// It says nothing about the file tomorrow. A link serves the file's LATEST
|
||||
// content forever (see the package comment in shares.go), so every string a
|
||||
// user sees says the file was checked *at the moment you shared it* — never
|
||||
// that the file is clean.
|
||||
|
||||
// secretScanLimit is how much of a file the share gate reads. The boundary is
|
||||
// a decision, not an accident: a key past the first MiB mints silently, which
|
||||
// is asserted in shares_test.go so nobody "fixes" it by accident.
|
||||
const secretScanLimit = 1 << 20
|
||||
|
||||
// secretFinding is one credential-shaped string: which rule fired, and where.
|
||||
// Never the matched text — see scanSecrets.
|
||||
type secretFinding struct {
|
||||
Rule string `json:"rule"`
|
||||
Line int `json:"line"`
|
||||
}
|
||||
|
||||
var secretRules = []struct {
|
||||
id string
|
||||
re *regexp.Regexp
|
||||
}{
|
||||
{"aws_access_key_id", regexp.MustCompile(`AKIA[0-9A-Z]{16}`)},
|
||||
// The bodies below are what keep the prefixes off prose: a bare `sk-` in a
|
||||
// sentence is not a key. If one still fires on real docs, tighten the body
|
||||
// rather than dropping the rule — `--force` and Share anyway are the
|
||||
// escape hatch, which is why they ship in the same change.
|
||||
{"openai_api_key", regexp.MustCompile(`sk-[A-Za-z0-9_-]{20,}`)},
|
||||
{"github_pat", regexp.MustCompile(`ghp_[A-Za-z0-9]{36}`)},
|
||||
{"slack_token", regexp.MustCompile(`xox[baprs]-[A-Za-z0-9-]{10,}`)},
|
||||
{"private_key", regexp.MustCompile(`-----BEGIN [A-Z ]*PRIVATE KEY-----`)},
|
||||
{"gitlab_pat", regexp.MustCompile(`glpat-[A-Za-z0-9_-]{20,}`)},
|
||||
}
|
||||
|
||||
// scanSecrets reports credential-shaped strings in buf, as rule ids and line
|
||||
// numbers ONLY. The matched text must never reach a response body, a log line,
|
||||
// or a metric label — the same argument reads.go:28-40 makes for actor
|
||||
// identity, and a 409 body is the easiest place in the codebase to leak it.
|
||||
//
|
||||
// Byte-oriented on purpose: a bufio.Scanner over a 1 MiB minified file with no
|
||||
// newline blows its 64 KiB token limit and returns nothing at all, which is a
|
||||
// check that silently passes everything.
|
||||
func scanSecrets(buf []byte) []secretFinding {
|
||||
seen := map[secretFinding]bool{}
|
||||
var out []secretFinding
|
||||
for _, rule := range secretRules {
|
||||
for _, m := range rule.re.FindAllIndex(buf, -1) {
|
||||
f := secretFinding{Rule: rule.id, Line: bytes.Count(buf[:m[0]], []byte("\n")) + 1}
|
||||
if !seen[f] {
|
||||
seen[f] = true
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].Line != out[j].Line {
|
||||
return out[i].Line < out[j].Line
|
||||
}
|
||||
return out[i].Rule < out[j].Rule
|
||||
})
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package webapp
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Fabricated, structurally-valid-looking strings. None is a real credential.
|
||||
const (
|
||||
fakeAWSKey = "AKIAIOSFODNN7EXAMPLE"
|
||||
fakeOpenAIKey = "sk-abcdefghijklmnopqrstuvwxyz0123456789"
|
||||
fakeGitHubPAT = "ghp_" + "abcdefghijklmnopqrstuvwxyz0123456789ab"
|
||||
fakeSlackTok = "xoxb-1234567890-abcdefghij"
|
||||
fakeGitLabPAT = "glpat-abcdefghij0123456789XY"
|
||||
fakePrivKey = "-----BEGIN RSA PRIVATE KEY-----"
|
||||
)
|
||||
|
||||
func TestScanSecrets(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
buf string
|
||||
want []secretFinding
|
||||
}{
|
||||
{"clean", "# Notes\n\nnothing to see, sk- is just a prefix here\n", nil},
|
||||
{"aws", "line1\nline2\nkey = " + fakeAWSKey + "\n", []secretFinding{{"aws_access_key_id", 3}}},
|
||||
{"openai", "OPENAI=" + fakeOpenAIKey, []secretFinding{{"openai_api_key", 1}}},
|
||||
{"github", "\n\n" + fakeGitHubPAT, []secretFinding{{"github_pat", 3}}},
|
||||
{"slack", "token: " + fakeSlackTok, []secretFinding{{"slack_token", 1}}},
|
||||
{"gitlab", "x\n" + fakeGitLabPAT, []secretFinding{{"gitlab_pat", 2}}},
|
||||
{"private key", "a\nb\nc\n" + fakePrivKey + "\nMIIE...\n", []secretFinding{{"private_key", 4}}},
|
||||
{
|
||||
// One line, one rule, three keys: one finding, not three.
|
||||
"multi key line deduped",
|
||||
"a=" + fakeAWSKey + " b=AKIAZZZZZZZZZZZZZZZZ c=AKIAYYYYYYYYYYYYYYYY",
|
||||
[]secretFinding{{"aws_access_key_id", 1}},
|
||||
},
|
||||
{
|
||||
"two rules same line",
|
||||
"env: " + fakeAWSKey + " " + fakeSlackTok,
|
||||
[]secretFinding{{"aws_access_key_id", 1}, {"slack_token", 1}},
|
||||
},
|
||||
{
|
||||
// A bufio.Scanner would blow its 64 KiB token limit here and report
|
||||
// nothing at all — which is why scanSecrets is byte-oriented.
|
||||
"no newline in a big buffer",
|
||||
strings.Repeat("x", 300_000) + fakeAWSKey,
|
||||
[]secretFinding{{"aws_access_key_id", 1}},
|
||||
},
|
||||
{
|
||||
"sorted by line then rule",
|
||||
fakeSlackTok + "\n" + fakeAWSKey,
|
||||
[]secretFinding{{"slack_token", 1}, {"aws_access_key_id", 2}},
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := scanSecrets([]byte(tc.buf))
|
||||
if !reflect.DeepEqual(got, tc.want) {
|
||||
t.Fatalf("scanSecrets = %v, want %v", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1392,6 +1392,13 @@ func storageErr(w http.ResponseWriter, code int, msg string, err error) {
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, v any) {
|
||||
writeJSONStatus(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
// writeJSONStatus is writeJSON for the answers a client has to read the body
|
||||
// of — a 409 whose findings the CLI and the browser both decode.
|
||||
func writeJSONStatus(w http.ResponseWriter, code int, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(code)
|
||||
json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
@@ -249,6 +249,7 @@ func (s *Server) handleShareCreate(v *volume, w http.ResponseWriter, r *http.Req
|
||||
var req struct {
|
||||
Path string `json:"path"`
|
||||
ExpiresIn string `json:"expires_in,omitempty"` // Go duration, e.g. "168h"
|
||||
Confirm bool `json:"confirm,omitempty"` // share it anyway, secrets and all
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
|
||||
http.Error(w, "bad request: "+err.Error(), http.StatusBadRequest)
|
||||
@@ -275,6 +276,33 @@ func (s *Server) handleShareCreate(v *volume, w http.ResponseWriter, r *http.Req
|
||||
return
|
||||
}
|
||||
}
|
||||
if !req.Confirm && !s.alreadyPublic(r.PathValue("project"), p) {
|
||||
rc, err := v.source.Open(r.Context(), p, snap.files[p])
|
||||
if err != nil {
|
||||
// Fails CLOSED. The repo's "degrade rather than fail" posture is for
|
||||
// sync cycles; minting is a rare interactive action, and a check
|
||||
// that skips itself on a storage hiccup is exactly the false
|
||||
// confidence this gate exists to remove.
|
||||
storageErr(w, http.StatusServiceUnavailable, "could not read the file to check it for credentials", err)
|
||||
return
|
||||
}
|
||||
// 1 MiB and close: source.Open streams from the object store, so this
|
||||
// aborts the rest of the transfer rather than pulling a 500 MB file
|
||||
// down to look at its first megabyte. Don't "fix" it into a ReadAll.
|
||||
buf, err := io.ReadAll(io.LimitReader(rc, secretScanLimit))
|
||||
rc.Close()
|
||||
if err != nil {
|
||||
storageErr(w, http.StatusServiceUnavailable, "could not read the file to check it for credentials", err)
|
||||
return
|
||||
}
|
||||
if findings := scanSecrets(buf); len(findings) > 0 {
|
||||
writeJSONStatus(w, http.StatusConflict, map[string]any{
|
||||
"error": "this file looks like it contains credentials",
|
||||
"findings": findings,
|
||||
})
|
||||
return
|
||||
}
|
||||
}
|
||||
sh, err := s.Shares.Create(r.PathValue("project"), p, s.requestUser(r).Email, ttl)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
@@ -284,6 +312,28 @@ func (s *Server) handleShareCreate(v *volume, w http.ResponseWriter, r *http.Req
|
||||
writeJSON(w, shareJSON(r, sh, nil))
|
||||
}
|
||||
|
||||
// alreadyPublic reports whether this path is already served to anyone with a
|
||||
// URL. If it is, minting skips the credential scan: the content is public
|
||||
// already, so withholding the link protects nothing and would break the
|
||||
// "clicking Share again gives me the same link" behaviour the dialog is built
|
||||
// on.
|
||||
//
|
||||
// It cannot key off ShareDB.Create's reuse branch, which is narrower (that one
|
||||
// also requires no expiry on either side), and it has to drop links whose
|
||||
// creator left the org — those 404 at /s/ (shareCreatorStillBelongs), so a
|
||||
// secrets file whose only link is already dead must not wave through.
|
||||
func (s *Server) alreadyPublic(project, p string) bool {
|
||||
if s.Shares == nil {
|
||||
return false
|
||||
}
|
||||
for _, sh := range s.Shares.List(project) { // List already drops expired
|
||||
if sh.Path == p && s.shareCreatorStillBelongs(sh) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (s *Server) handleShareList(v *volume, w http.ResponseWriter, r *http.Request) {
|
||||
if s.Shares == nil {
|
||||
http.Error(w, "sharing is not enabled on this server", http.StatusNotFound)
|
||||
|
||||
@@ -2,10 +2,14 @@ package webapp
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
@@ -677,6 +681,136 @@ func authAs(t *testing.T, srv *Server, req *http.Request) {
|
||||
req.Header.Set("Authorization", "Bearer "+tok)
|
||||
}
|
||||
|
||||
// planted is a fabricated AWS-shaped string. No real credential is involved,
|
||||
// here or in the fixtures below.
|
||||
const planted = "AKIAIOSFODNN7EXAMPLE"
|
||||
|
||||
// postShare mints as a signed-in member and hands back the raw recorder, so a
|
||||
// test can look at a non-200.
|
||||
func postShare(t *testing.T, srv *Server, h http.Handler, project string, body any) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
req := jsonReq(t, "POST", "/api/p/"+project+"/shares", body)
|
||||
authAs(t, srv, req)
|
||||
return doHTTP(h, req)
|
||||
}
|
||||
|
||||
func decodeFindings(t *testing.T, rec *httptest.ResponseRecorder) []secretFinding {
|
||||
t.Helper()
|
||||
var out struct {
|
||||
Error string `json:"error"`
|
||||
Findings []secretFinding `json:"findings"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
|
||||
t.Fatalf("decode 409 body %q: %v", rec.Body, err)
|
||||
}
|
||||
if out.Error == "" {
|
||||
t.Fatalf("409 body carries no error message: %s", rec.Body)
|
||||
}
|
||||
return out.Findings
|
||||
}
|
||||
|
||||
// TestShareSecretScan is the gate: a credential-shaped file does not become a
|
||||
// public URL by accident, and does become one on purpose.
|
||||
func TestShareSecretScan(t *testing.T) {
|
||||
srv, p, _, f, h := shareHub(t)
|
||||
f.put("dev1", "deploy.md", "# Deploy\n\nrun it\n\nAWS_ACCESS_KEY_ID="+planted+"\n")
|
||||
|
||||
// 1. blocked, and nothing minted
|
||||
rec := postShare(t, srv, h, p.ID, map[string]string{"path": "deploy.md"})
|
||||
if rec.Code != http.StatusConflict {
|
||||
t.Fatalf("share of a file holding a key: %d %s, want 409", rec.Code, rec.Body)
|
||||
}
|
||||
if got := decodeFindings(t, rec); !reflect.DeepEqual(got, []secretFinding{{"aws_access_key_id", 5}}) {
|
||||
t.Fatalf("findings = %v, want aws_access_key_id on line 5", got)
|
||||
}
|
||||
if n := len(srv.Shares.List(p.ID)); n != 0 {
|
||||
t.Fatalf("409 minted %d shares, want 0", n)
|
||||
}
|
||||
|
||||
// 2. the same request with confirm mints a working link
|
||||
rec = postShare(t, srv, h, p.ID, map[string]any{"path": "deploy.md", "confirm": true})
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("confirmed share: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
var out struct{ Token string }
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pub := do(t, h, "GET", "/s/"+out.Token, nil); pub.Code != 200 || !strings.Contains(pub.Body.String(), "Deploy") {
|
||||
t.Fatalf("confirmed link does not serve: %d %s", pub.Code, pub.Body)
|
||||
}
|
||||
|
||||
// 3. a path that is already public skips the scan — the content is out
|
||||
// there, so a second Share click still hands back the same URL.
|
||||
rec = postShare(t, srv, h, p.ID, map[string]string{"path": "deploy.md"})
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("re-share of an already-public file: %d %s, want 200", rec.Code, rec.Body)
|
||||
}
|
||||
|
||||
// 4. a clean file is unaffected
|
||||
if rec := postShare(t, srv, h, p.ID, map[string]string{"path": "wiki/notes.md"}); rec.Code != 200 {
|
||||
t.Fatalf("clean file: %d %s, want 200", rec.Code, rec.Body)
|
||||
}
|
||||
|
||||
// 5. the boundary is a decision: a key past the first MiB mints silently.
|
||||
f.put("dev1", "big.md", strings.Repeat("filler line\n", 100_000)+planted+"\n")
|
||||
if rec := postShare(t, srv, h, p.ID, map[string]string{"path": "big.md"}); rec.Code != 200 {
|
||||
t.Fatalf("key past the 1 MiB limit: %d %s, want 200 (documented boundary)", rec.Code, rec.Body)
|
||||
}
|
||||
}
|
||||
|
||||
// TestShareSecretNeverEchoed is the one rule that cannot bend: the matched
|
||||
// bytes never leave scanSecrets — not in the body, not in the log.
|
||||
func TestShareSecretNeverEchoed(t *testing.T) {
|
||||
srv, p, _, f, h := shareHub(t)
|
||||
f.put("dev1", "creds.md", "key = "+planted+"\n")
|
||||
|
||||
var logs bytes.Buffer
|
||||
prev := log.Writer()
|
||||
log.SetOutput(&logs)
|
||||
t.Cleanup(func() { log.SetOutput(prev) })
|
||||
|
||||
rec := postShare(t, srv, h, p.ID, map[string]string{"path": "creds.md"})
|
||||
if rec.Code != http.StatusConflict {
|
||||
t.Fatalf("share: %d %s, want 409", rec.Code, rec.Body)
|
||||
}
|
||||
if strings.Contains(rec.Body.String(), planted) {
|
||||
t.Errorf("the 409 body echoed the secret: %s", rec.Body)
|
||||
}
|
||||
if strings.Contains(logs.String(), planted) {
|
||||
t.Errorf("the secret reached the log: %s", logs.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestShareSecretScanFailsClosed: a blob the hub cannot read mints nothing.
|
||||
// A check that skips itself on a storage hiccup is the false confidence this
|
||||
// gate exists to remove.
|
||||
func TestShareSecretScanFailsClosed(t *testing.T) {
|
||||
srv, p, _, f, h := shareHub(t)
|
||||
f.put("dev1", "gone.md", "harmless")
|
||||
// Drop the blob but keep the journal op: the file is "synced" and unreadable.
|
||||
blobs, err := os.ReadDir(filepath.Join(f.dir, "blobs"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sum := sha256.Sum256([]byte("harmless"))
|
||||
want := hex.EncodeToString(sum[:])
|
||||
for _, b := range blobs {
|
||||
if b.Name() == want {
|
||||
if err := os.Remove(filepath.Join(f.dir, "blobs", b.Name())); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
rec := postShare(t, srv, h, p.ID, map[string]string{"path": "gone.md"})
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("unreadable blob: %d %s, want 503", rec.Code, rec.Body)
|
||||
}
|
||||
if n := len(srv.Shares.List(p.ID)); n != 0 {
|
||||
t.Fatalf("failed scan minted %d shares, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
// A share token is a promise about ONE file. These three pin the direction
|
||||
// it goes when a path changes hands — the opposite of the viewer's, which is
|
||||
// an address and always serves whatever lives there now.
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -5,7 +5,7 @@
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>BearDrive</title>
|
||||
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32' fill='%23f5a623'><rect x='4' y='4' width='5.6' height='24'/><rect x='11.2' y='4' width='14.4' height='11.2'/><rect x='11.2' y='16.8' width='16.8' height='11.2'/></svg>">
|
||||
<script type="module" crossorigin src="/assets/index-DRd_YZgy.js"></script>
|
||||
<script type="module" crossorigin src="/assets/index-Q_8ZOeQ7.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-Do25j1to.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
@@ -91,8 +91,11 @@ scripts can't touch hub sessions and a scraper can't turn your hub into a CDN.
|
||||
|
||||
:::caution
|
||||
Any org member can mint links, and a link is public to whoever holds the URL.
|
||||
Don't put secrets in a synced folder. Note also that a LAN-bound hub means
|
||||
LAN-only links.
|
||||
Before minting, the hub scans the first 1 MiB of the file for
|
||||
credential-shaped strings and refuses (`--force`, or **Share anyway** in the
|
||||
UI, overrides it) — but that check runs *at the moment you share*, and a link
|
||||
serves the file's latest content forever, so don't put secrets in a synced
|
||||
folder. Note also that a LAN-bound hub means LAN-only links.
|
||||
:::
|
||||
|
||||
## Who wrote what
|
||||
|
||||
@@ -231,3 +231,12 @@ Regardless of configuration:
|
||||
the seeded default covers the common cases. But treat it as hygiene, not a
|
||||
security control: any org member can mint a public link for any synced file.
|
||||
Secrets belong in a secret manager, not in a folder you hand to agents.
|
||||
|
||||
Minting a share link is the one place the hub reads the bytes first: it scans
|
||||
the **first 1 MiB** for credential-shaped strings (AWS access key, private key
|
||||
block, GitHub/Slack/GitLab/OpenAI token) and refuses, naming the rule and the
|
||||
line but never the matched text. `bdrive share --force` — or **Share anyway**
|
||||
in the web UI — overrides it. Know what that check does *not* cover: it runs
|
||||
**at the moment you share**, and a link serves the file's latest content
|
||||
forever, so a key written into an already-shared file is never caught. It is a
|
||||
last-line backstop, not a reason to sync a folder you wouldn't otherwise.
|
||||
|
||||
@@ -20,7 +20,7 @@ One binary, `bdrive` — the CLI, the sync daemon, and the web server.
|
||||
| `bdrive grep <pattern> [folder]` | Search the text **inside** the files a project syncs. `pattern` is a Go RE2 regexp, or a literal string with `-F`. `-i` ignores case, `-l` prints matching paths only, `-n` caps the lines printed (default 200, `0` = all). Pure read: no daemon, no lock, no network |
|
||||
| `bdrive forget <path>...` | Stop syncing a path and remove it from the hub. Adds the rule to `.bdriveignore` (which syncs) and prunes in one step. Local files are never touched, here or on teammates' devices |
|
||||
| `bdrive url [path]` | Internal hub link for a file or folder — sign-in and membership required. `--sync` pushes first; no argument gives the project home. Computed locally |
|
||||
| `bdrive share <file>` | Public URL for a synced file. `--list`, `--revoke`, `--expires` (the hub's Share dialog can also set an expiry on an existing link) |
|
||||
| `bdrive share <file>` | Public URL for a synced file. `--list`, `--revoke`, `--expires` (the hub's Share dialog can also set an expiry on an existing link). Refuses a file whose first 1 MiB holds credential-shaped strings — `--force` shares it anyway |
|
||||
| `bdrive sync [folder]` | Run one sync cycle now. Refuses folders this device never `init`ed and folders paused by `bdrive stop`. `--note <text>` stamps session context onto changes; `--note-ttl` (default 30m) bounds it. `--prune` also removes from the hub what `.bdriveignore` now excludes (files stay on disk everywhere). `--hook <label>` is agent-hook plumbing |
|
||||
| `bdrive hooks [install\|uninstall]` | Register turn-boundary sync hooks in each detected agent platform's user config — once per machine, covering every folder. Run automatically by `bdrive init`; idempotent; `--agent` overrides detection. `uninstall` removes only BearDrive's own hook entries |
|
||||
| `bdrive read-log [folder]` | Hook plumbing: queue agent file reads for the hub's read heatmap. Registered by `bdrive hooks install` |
|
||||
|
||||
Reference in New Issue
Block a user