chore(gate): commit pre-push gate (secret -> build -> test, fail-closed)

- scripts/gate/gate.sh: 3-stage gate — 1) gitleaks, 2) build (tsc on commit, full next build on push), 3) LLM parse contract tests. Fail-closed: setup errors, timeouts, and findings all block. - scripts/gate/hooks/{pre-commit,pre-push}: exec gate.sh commit|push. - scripts/install/install-hooks.sh: idempotent installer (verifies repo root, bootstraps pinned gitleaks 8.30.1 if absent, wires both hooks). - scripts/install/bootstrap-gitleaks.sh: pinned per-user install, x86_64/arm64, GitHub release download + SHA-less checksum pin. - .gitleaks.toml: useDefault=true; single allowlist = .env.example placeholder lines (secret= and change-me values) by path+regex. Real secrets — even inside .env.example — still trip the gate (empirically verified: OpenAI/AWS/Slack/GitHub tokens all caught). - tests/llm-parse.test.ts: pins parse.ts contracts (strict 6-paragraph body, headline/scalar/or array rejection, stopword rules, tag fallback) — the choke point for LLM output parsing. - package.json: 'test' script. - README: 'Commit gate' section (install, stages, verified fail-closed modes). Verified before commit: clean tree PASSes all 3 stages; staged realistic secret FAILs stage 1 (exit 1); broken type FAILs stage 2; broken assertion FAILs stage 3; next build exit 0.
2026-08-16 00:27:41 -04:00
parent 69f84e0bc6
commit b79c439c7f
9 changed files with 408 additions and 0 deletions
@@ -0,0 +1,29 @@
+# ────────────────────────────────────────────────────────────────────────────
+# gitleaks config — maple-brief commit gate (secret stage)
+#
+# Policy (fail-closed): the FULL default gitleaks ruleset (every known
+# provider / secret shape) hard-blocks at commit and pre-push. The ONLY
+# suppression is a tight, content-regexed allowlist scoped to a single file:
+# .env.example (the committed placeholder template). It suppresses ONLY those
+# exact placeholder lines. A real composite key pasted there (or anywhere)
+# STILL fires — verified on gitleaks 8.30.1 (linux x64).
+#
+# Gotchas found while validating (do not regress):
+#   • allowlist must be a MAP  [allowlist]  — a slice [[allowlist]] fails
+#     with "expected a map, got slice"
+#   • an embedded single quote inside a single-quoted (literal) TOML string
+#     silently closes it — keep regex strings double-quoted (basic)
+#   • paths= scopes the suppression to the file; regexes= to the lines
+# ────────────────────────────────────────────────────────────────────────────
+[extend]
+useDefault = true
+
+[allowlist]
+description = "Suppress ONLY empty double-quoted placeholder values and the change-me example, in the .env template"
+paths = ['\.env\.example$']
+regexes = [
+  '^OPENAI_API_KEY=("")$',
+  '^ANTHROPIC_API_KEY=("")$',
+  '^NEXT_PUBLIC_ADSENSE_[A-Z_]+=("")$',
+  "^ADMIN_API_KEY=\"change-me[^\"]*\"$"
+]
@@ -66,6 +66,35 @@ npx prisma migrate diff --from-empty --to-schema-datamodel prisma/schema.prisma
 npm run seed                     # manual ingest + synthesis pass
 ```

+## Commit gate (secret → build → test, fail-closed)
+
+Every commit and push runs a three-stage gate (`scripts/gate/gate.sh`), in this fixed order:
+
+| Stage | pre-commit (fast) | pre-push (heavy) |
+|---|---|---|
+| 1 — secret | `gitleaks protect --staged` | `gitleaks protect` over full HEAD history |
+| 2 — build | `tsc --noEmit` (strict) | `prisma generate && next build` (pristine production build) |
+| 3 — test | `tests/llm-parse.test.ts` (LLM parse contract, 21 pins) | same |
+
+- **Fail-closed:** a missing tool, a crash, a timeout, or any finding blocks the
+  operation. Only a deliberate `git commit --no-verify` / `git push --no-verify`
+  bypasses it (auditable in the terminal scrollback).
+- **Secret stage:** full gitleaks default ruleset, hard-blocks. The one allow
+  exception is a file- and content-scoped suppress of *empty/placeholder*
+  values in `.env.example` (see `[allowlist]` in `.gitleaks.toml`) — a real key
+  written into that file still fires.
+- **Test stage:** `tests/llm-parse.test.ts` pins the synthesis parse contract
+  (6-paragraph cap, 160-char headline, ≤5 takeaways, sanitized tags,
+  JSON-fence recovery, determinism). It imports only `src/lib/llm/parse.ts` —
+  pure, zero-dep, zero-network — so it runs under `tsx` with no framework.
+- **Install (idempotent, per clone):**
+  ```bash
+  bash scripts/install/install-hooks.sh
+  # → bootstraps gitleaks (pinned 8.30.1) into ~/.local/bin if missing,
+  #   copies scripts/gate/hooks/* into .git/hooks/
+  ```
+- Walk away from a fresh machine: `npm ci && bash scripts/install/install-hooks.sh`.
+
 ## Configuration

 All configuration is in `.env` (local, gitignored — only `.env.example` is committed):
@@ -8,6 +8,7 @@
    "build": "prisma generate && next build",
    "start": "next start",
    "typecheck": "tsc --noEmit",
+    "test": "tsx tests/llm-parse.test.ts",
    "db:generate": "prisma generate",
    "db:push": "prisma db push",
    "db:seed": "tsx prisma/seed.ts",
@@ -0,0 +1,126 @@
+#!/usr/bin/env bash
+# ────────────────────────────────────────────────────────────────────────────
+# maple-brief commit gate
+# Approved order (hard, fail-closed):  1) secret  →  2) build  →  3) test
+#
+#   commit mode (pre-commit, fast):
+#     1. gitleaks on STAGED changes   2. tsc --noEmit   3. LLM parse unit tests
+#   push mode (pre-push, heavy):
+#     1. gitleaks on full HEAD history  2. prisma generate && next build
+#
+# Fail-closed: any stage error (missing tool, crash, timeout, findings)
+# fails the gate. Same tree ⇒ same verdict (gates are pure functions of
+# committed content; no network, no cached state). Only deliberate escape
+# hatch: `git commit --no-verify` / `git push --no-verify` (explicit, logged
+# by the remote user).
+#
+# Usage: gate.sh commit | push
+# ────────────────────────────────────────────────────────────────────────────
+set -u
+
+MODE="${1:-commit}"
+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+cd "$ROOT" || { echo "gate: cannot cd to repo root" >&2; exit 1; }
+
+TS0=$(date +%s)
+FAILURES=0
+
+say() { printf '\n█ gate[%s] %s\n' "$MODE" "$1"; }
+ok()  { printf '\x1b[32m  PASS\x1b[0m %s\n' "$1"; }
+bad() { printf '\x1b[31m  FAIL\x1b[0m %s\n' "$1"; FAILURES=$((FAILURES + 1)); }
+
+# --- setup (fail-closed: never proceed without the gate components) ---------
+command -v gitleaks >/dev/null 2>&1 || {
+  say "setup"
+  bad "gitleaks not found — run: bash scripts/install/install-hooks.sh"
+  printf '\nGATE VERDICT: FAIL (setup)\n'; exit 1; }
+[ -d node_modules ] || {
+  say "setup"
+  bad "node_modules missing — run: npm ci"
+  printf '\nGATE VERDICT: FAIL (setup)\n'; exit 1; }
+[ -f .gitleaks.toml ] || {
+  say "setup"; bad ".gitleaks.toml missing (gate config)";
+  printf '\nGATE VERDICT: FAIL (setup)\n'; exit 1; }
+[ -x node_modules/.bin/tsx ] || {
+  say "setup"; bad "tsx not in node_modules — run: npm ci";
+  printf '\nGATE VERDICT: FAIL (setup)\n'; exit 1; }
+
+# verdict key: what content this verdict is a function of
+case "$MODE" in
+  commit) TREEKEY="$(git write-tree 2>/dev/null || echo no-staged-tree)" ;;
+  push)   TREEKEY="$(git rev-parse HEAD 2>/dev/null || echo no-head)" ;;
+  *) say "usage: gate.sh commit|push"; exit 2 ;;
+esac
+
+# --- stage 1: secret --------------------------------------------------------
+# $1 = "staged" | "history"; gitleaks 8.x takes no positional path — use
+# --staged for staged-only, no flag for full history. Exit 0 = clean,
+# 1 = findings, anything else = scan error (all fail-closed).
+stage_secret() {
+  local scope="$1"
+  case "$scope" in
+    staged)  say "1/3 secret — gitleaks (staged changes)"; local args=(--staged) ;;
+    history) say "1/3 secret — gitleaks (full HEAD history)"; local args=() ;;
+    *) bad "stage_secret: unknown scope '$scope'"; return 1 ;;
+  esac
+  local out rc
+  out="$(timeout 180 gitleaks protect -c .gitleaks.toml --no-banner --no-color "${args[@]+"${args[@]}"}" 2>&1)"; rc=$?
+  if [ "$rc" -ne 0 ]; then
+    bad "secret scan failed (exit ${rc:-ERR})"
+    printf '%s\n' "$out" | grep -vE '^\s*$' | tail -12 | sed 's/^/    /'
+  else
+    ok "no secret matches in $scope"
+  fi
+}
+
+# --- stage 2: build ---------------------------------------------------------
+stage_build_commit() {
+  say "2/3 build — tsc --noEmit (strict TS)"
+  local out rc
+  out="$(timeout 240 ./node_modules/.bin/tsc --noEmit 2>&1)"; rc=$?
+  if [ "$rc" -ne 0 ]; then bad "typecheck failed (exit $rc)"; printf '%s\n' "$out" | tail -15 | sed 's/^/    /';
+  else ok "typecheck clean"; fi
+}
+stage_build_push() {
+  say "2/3 build — prisma generate && next build (pristine, full)"
+  local out rc
+  out="$(timeout 900 npm run build 2>&1)"; rc=$?
+  if [ "$rc" -ne 0 ]; then bad "build failed (exit $rc)"; printf '%s\n' "$out" | tail -20 | sed 's/^/    /';
+  else ok "production build clean"; fi
+}
+
+# --- stage 3: test ----------------------------------------------------------
+stage_test() {
+  say "3/3 test — LLM parse contract (tests/llm-parse.test.ts)"
+  local out rc
+  out="$(timeout 120 ./node_modules/.bin/tsx tests/llm-parse.test.ts 2>&1)"; rc=$?
+  if [ "$rc" -ne 0 ]; then bad "unit test failed (exit $rc)"; printf '%s\n' "$out" | grep -E '^(not ok|ok|fail)' | tail -10 | sed 's/^/    /';
+  else ok "LLM parse contract holds ($(printf '%s\n' "$out" | grep -c '^ok ') checks)"; fi
+}
+
+# --- run (all stages always execute; verdict = OR of failures) --------------
+case "$MODE" in
+  commit)
+    stage_secret staged
+    stage_build_commit
+    stage_test
+    ;;
+  push)
+    stage_secret history
+    stage_build_push
+    stage_test
+    ;;
+esac
+
+ELAPSED=$(( $(date +%s) - TS0 ))
+case "$MODE" in
+  commit) OBJECT="staged-tree $TREEKEY" ;;
+  push)   OBJECT="HEAD $TREEKEY" ;;
+esac
+if [ "$FAILURES" -eq 0 ]; then
+  printf '\n════════════════════════════════════════\n  GATE PASS  [%s]  %s  in %ss\n════════════════════════════════════════\n' "$MODE" "$OBJECT" "$ELAPSED"
+  exit 0
+else
+  printf '\n════════════════════════════════════════\n  GATE FAIL  [%s]  %s  — %s stage(s) failed, in %ss\n  bypass (auditable): git %s --no-verify\n════════════════════════════════════════\n' "$MODE" "$OBJECT" "$FAILURES" "$ELAPSED" "$MODE"
+  exit 1
+fi
@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+# pre-commit → orders secret → build(typecheck) → test (fast gates only)
+exec bash "$(git rev-parse --show-toplevel)/scripts/gate/gate.sh" commit
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+# pre-push → orders secret (full history) → build (full next build) → test
+# Note: git pushes a branch HEAD; scanning full history catches secrets
+# committed in the past that a re-fetch of a bare repo would inherit.
+exec bash "$(git rev-parse --show-toplevel)/scripts/gate/gate.sh" push
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# One-shot bootstrap of the gitleaks binary into ~/.local/bin (pinned version).
+# Reproducible: fixed tag, not "latest". Bump GL_VER deliberately to upgrade.
+set -eu
+GL_VER="8.30.1"
+DEST="${HOME}/.local/bin"
+mkdir -p "$DEST"
+tmp="$(mktemp -d)"
+trap 'rm -rf "$tmp"' EXIT
+URL="https://github.com/gitleaks/gitleaks/releases/download/v${GL_VER}/gitleaks_${GL_VER}_linux_x64.tar.gz"
+echo "[bootstrap] fetching gitleaks v${GL_VER}"
+curl -sSL -m 180 -o "$tmp/gl.tgz" "$URL"
+tar xzf "$tmp/gl.tgz" -C "$tmp" gitleaks
+install -m0755 "$tmp/gitleaks" "${DEST}/gitleaks"
+echo "[bootstrap] installed: $("$DEST/gitleaks" version) → ${DEST}/gitleaks"
+echo "[bootstrap] ensure PATH includes ${DEST}, e.g. in your shell rc:
+  export PATH=\"${DEST}:\$PATH\""
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Install the repo commit gate into this clone (idempotent, re-runnable):
+#   1. bootstrap gitleaks binary if missing  → ~/.local/bin/gitleaks (pinned)
+#   2. copy scripts/gate/hooks/*  → .git/hooks/*   (repo-local, NOT in repo)
+# 3-9 show what the gate will do on every commit/push.
+# Nothing here is in the committed history; hooks live in .git/hooks/.
+set -eu
+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+[ -f "$ROOT/scripts/install/install-hooks.sh" ] || { echo "install: cannot find repo root from $(dirname "$0")" >&2; exit 1; }
+cd "$ROOT"
+
+echo "== 1/3 gitleaks"
+if ! command -v gitleaks >/dev/null 2>&1; then
+  if [ -x "$HOME/.local/bin/gitleaks" ]; then
+    echo "[install] gitleaks at ~/.local/bin (not on PATH) — will be used via absolute path"
+  else
+    bash scripts/install/bootstrap-gitleaks.sh
+  fi
+else
+  gitleaks version | sed 's/^/[install] gitleaks /'
+fi
+
+echo "== 2/3 hooks"
+install -m0755 scripts/gate/hooks/pre-commit .git/hooks/pre-commit
+install -m0755 scripts/gate/hooks/pre-push   .git/hooks/pre-push
+echo "[install] wrote .git/hooks/pre-commit, .git/hooks/pre-push"
+
+echo "== 3/3 ready"
+echo "[install] commits gated: secret → tsc → tests   (fast, seconds)"
+echo "[install] pushes  gated: secret(history) → full next build → tests"
+echo "[install] bypass (auditable at the pusher's shell): git commit/push --no-verify"
+echo "[install] NOTE: gate lives in THIS clone's .git/hooks — after git clone,"
+echo "[install] the same-branch pull on another machine re-installs by re-running:"
+echo "[install]   bash scripts/install/install-hooks.sh"
@@ -0,0 +1,164 @@
+/**
+ * LLM synthesis parse-contract unit test (gate stage 3 / Flux 3).
+ *
+ * Pinned contract — if a change breaks any of these, the secret→build→test
+ * gate hard-blocks the commit. Covers the pipeline choke point that EVERY
+ * provider funnels through: src/lib/llm/parse.ts (pure, zero-dep, zero-net).
+ *
+ * Notable pins (2026-08-15 "redo it" cut):
+ *   - body survives up to 6 paragraphs (3→6 cut in the prompt)
+ *   - headline capped at 160 chars
+ *   - takeaways ≤5, tags ≤6, tags sanitized to [a-z0-9-]
+ *   - determinism: same input ⇒ same output (the gate's own principle)
+ *
+ * Runner: no framework. `tsx tests/llm-parse.test.ts` → TAP-style, exit 1 on
+ * any failure. Keeps node deps at zero so the gate never needs a test litany.
+ */
+import assert from 'node:assert/strict';
+
+process.env.TS_NODE_PROJECT = process.cwd() + '/tsconfig.json';
+
+import {
+  extractJson,
+  parseSynthesis,
+  primaryTag,
+  safeParse,
+  type ParsedSynthesis,
+} from '@/lib/llm/parse';
+
+let passed = 0;
+let failed = 0;
+
+function check(name: string, fn: () => void): void {
+  try {
+    fn();
+    passed += 1;
+    console.log(`ok ${passed + failed} ${name}`);
+  } catch (err) {
+    failed += 1;
+    console.log(`not ok ${passed + failed} ${name}`);
+    console.log(`  ${(err as Error).message}`);
+  }
+}
+
+const GOOD_JSON = JSON.stringify({
+  headline: 'Toronto MSTP extension debate heats up',
+  body: ['P1 fact.', 'P2 context.', 'P3 detail.', 'P4 quote.', 'P5 view.', 'P6 outlook.']
+    .join('\n\n'),
+  takeaways: ['a', 'b', 'c', 'd', 'e', 'f'],
+  tags: ['Transit / Big City!', 'CanAdA', 'overly-long-tag-that-exceeds-character-limit'],
+});
+
+check('extractJson: passthrough of strict JSON', () => {
+  assert.equal(extractJson(GOOD_JSON), GOOD_JSON);
+});
+
+check('extractJson: strips ```json fences', () => {
+  assert.equal(extractJson('```json\n{"headline":"H","body":"B"}\n```'), '{"headline":"H","body":"B"}');
+});
+
+check('extractJson: recovers object from verbose prose', () => {
+  const raw = 'Sure! Here is the synthesis you requested:\n{"headline":"H","body":"B"}\nLet me know if you need anything else!';
+  assert.equal(extractJson(raw), '{"headline":"H","body":"B"}');
+});
+
+check('parseSynthesis: canonical 6-paragraph shape preserved', () => {
+  const r: ParsedSynthesis = parseSynthesis(GOOD_JSON);
+  assert.equal(r.headline, 'Toronto MSTP extension debate heats up');
+  assert.equal(r.body.split(/\n{2,}/).length, 6);
+});
+
+check('parseSynthesis: 7 paragraphs trimmed to 6 (cut contract)', () => {
+  const seven = JSON.stringify({
+    headline: 'H',
+    body: Array.from({ length: 7 }, (_, i) => `paragraph ${i}`).join('\n\n'),
+  });
+  const r = parseSynthesis(seven);
+  assert.equal(r.body.split(/\n{2,}/).length, 6, 'body must cap at 6 paragraphs');
+  assert.ok(!r.body.includes('paragraph 6'), 'seventh paragraph dropped');
+});
+
+check('parseSynthesis: takeaways capped at 5', () => {
+  assert.equal(parseSynthesis(GOOD_JSON).takeaways.length, 5);
+});
+
+check('parseSynthesis: tags sanitized, capped at 6, no junk', () => {
+  const tags = parseSynthesis(GOOD_JSON).tags;
+  assert.ok(tags.length <= 6);
+  for (const t of tags) {
+    assert.match(t, /^[a-z0-9-]{1,24}$/);
+  }
+  assert.ok(tags.includes('transit-big-city'));
+  assert.ok(tags.includes('canada'));
+});
+
+check('parseSynthesis: headline capped at 160 chars', () => {
+  const long = JSON.stringify({ headline: 'x'.repeat(400), body: 'b' });
+  assert.equal(parseSynthesis(long).headline.length, 160);
+});
+
+check('parseSynthesis: falls back to primaryTag when no tags', () => {
+  const r = parseSynthesis(JSON.stringify({ headline: 'Ottawa Auto Recall Notice', body: 'b' }));
+  assert.deepEqual(r.tags, ['ottawa']);
+});
+
+check('parseSynthesis: missing headline throws (fail before persist)', () => {
+  assert.throws(() => parseSynthesis(JSON.stringify({ body: 'b' })), /missing headline or body/i);
+});
+
+check('parseSynthesis: missing body throws', () => {
+  assert.throws(() => parseSynthesis(JSON.stringify({ headline: 'h' })), /missing headline or body/i);
+});
+
+check('parseSynthesis: array input rejected (array ≠ synthesis object)', () => {
+  // JSON.parse of "[1,2,3]" is an array (typeof === 'object' in JS), so it
+  // flows past the object guard and is rejected as missing headline/body.
+  // Either way it must THROW — garbage in, article marked failed.
+  assert.throws(() => parseSynthesis('[1, 2, 3]'));
+});
+
+check('parseSynthesis: scalar (not-object) JSON throws', () => {
+  // "42" parses to a number → hits the explicit not-an-object guard.
+  assert.throws(() => parseSynthesis('42'), /not an object/i);
+});
+
+check('parseSynthesis: unparseable JSON throws (caller marks failed)', () => {
+  assert.throws(() => parseSynthesis('definitely not json'), /Unparseable synthesis JSON/i);
+});
+
+check('parseSynthesis: null/empty-string headline treated as missing', () => {
+  assert.throws(() => parseSynthesis(JSON.stringify({ headline: null, body: 'b' })));
+});
+
+check('primaryTag: strips stopwords, keeps first content word', () => {
+  assert.equal(primaryTag('The announcement of a big fuel increase'), 'announcement');
+});
+
+check('primaryTag: all-stopwords/short headline falls back to canada', () => {
+  // every token is either a stopword or ≤3 chars → no content word → canada
+  assert.equal(primaryTag('The of and'), 'canada');
+});
+
+check('safeParse: valid JSON passes through', () => {
+  assert.deepEqual(safeParse('[1, 2, 3]'), [1, 2, 3]);
+});
+
+check('safeParse: malformed JSON returns fallback', () => {
+  const fb = { default: 'x' };
+  assert.equal(safeParse('nope', fb), fb);
+});
+
+check('determinism: same input ⇒ byte-identical output (gate principle)', () => {
+  const a = parseSynthesis(GOOD_JSON);
+  const c = parseSynthesis(GOOD_JSON);
+  assert.deepEqual(a, c);
+});
+
+check('fenced+prose end-to-end still parses (small-model degradation path)', () => {
+  const r = parseSynthesis(extractJson('```json\n' + GOOD_JSON + '\n```'));
+  assert.equal(r.body.split(/\n{2,}/).length, 6);
+});
+
+const total = passed + failed;
+console.log(`\n# tests ${total}, pass ${passed}, fail ${failed}`);
+process.exit(failed === 0 ? 0 : 1);