maple-brief-aggregator/README.md

# MapleBrief

An automated **Canadian news aggregator** with **LLM content synthesis** and **Google AdSense integration**.

MapleBrief pulls stories from Canadian news RSS feeds on a schedule, fetches the full article text, has an LLM rewrite them into original neutral reports (with headline, takeaways, and tags), and serves them on an ad-optimized Next.js site with full source attribution.

## Features

- **Scheduled ingestion** — a `node-cron` background worker (in-process, starts with the web server) fetches feeds, dedupes articles, extracts full article bodies with `cheerio`, and synthesizes pending articles every 35 minutes (configurable).
- **LLM abstraction, three providers** — OpenAI (default: `gpt-4o-mini`), Anthropic (Claude 3.5 Sonnet / Haiku), and Ollama (local). Provider, model per-provider, API keys, and the synthesis prompt are all **stored in the database** and editable live from the admin UI — no code changes or restarts needed.
- **Keyless-first, fail-closed admin** — `/admin/settings` and the settings APIs require `x-admin-key: $ADMIN_API_KEY`. Empty/missing key → every admin endpoint returns 401 (fail-closed). Comparison is timing-safe.
- **AdSense-ready layout** — header leaderboard (728×90) on every page, in-feed ads every 3rd card, in-article ad after the first paragraph, and a sticky 300×250 / 300×600 sidebar. Ad units render as labeled placeholders until a real Publisher ID is configured.
- **Compliance & SEO** — "Sources analyzed: …" attribution on every article, `rel="nofollow external noopener"` on all source links, per-article `canonical` + OpenGraph tags, `sitemap.xml`, `robots.txt`, and four legal pages (`/about`, `/privacy-policy`, `/terms-of-service`, `/contact`).

## Tech stack

| Layer | Choice |
|---|---|
| Framework | Next.js 14 (App Router) + TypeScript (strict) + Tailwind CSS |
| Database / ORM | SQLite via Prisma (zero-ops local dev; Postgres switch documented below) |
| Scheduler | `node-cron` worker in the server process |
| Ingestion | `rss-parser` + `cheerio` (content extraction), native `fetch` |
| LLM clients | native `fetch` only (OpenAI, Anthropic, Ollama) — no SDK dependencies |
| Ad framework | Google AdSense (`<AdUnit>` + `NEXT_PUBLIC_ADSENSE_*` env) |

## Quickstart

Requirements: **Node.js ≥ 18** (tested on v22), npm.

```bash
git clone https://gitea.krisforbes.ca/krisf/maple-brief-aggregator.git
cd maple-brief-aggregator

npm install

# 1. Environment
cp .env.example .env
#    then edit .env — at minimum set:
#      DATABASE_URL (default is fine for SQLite: file:./dev.db)
#      CRON_SCHEDULE (default "*/35 * * * *")
#      NEXT_PUBLIC_SITE_URL (deployment URL, without trailing slash)

# 2. Database — create tables and generate the Prisma client
npx prisma migrate deploy
#    (in dev: npx prisma migrate dev)

# 3. First ingest — pulls feeds, saves articles, synthesizes anything pending
npm run seed

# 4. Build & run (production)
npm run build
npm start            # http://localhost:3000

# — or development with hot reload:
npm run dev
```

On boot, the `src/instrumentation.ts` hook (Node.js runtime only) arms the worker, which immediately runs an initial ingest + synthesis pass and then repeats on `CRON_SCHEDULE`. The worker is idempotent (`runPipelinePass` is re-entrancy-guarded), so restarts and double-starts are safe.

### Useful commands

```bash
npx prisma studio                 # browse the DB
npx prisma migrate status         # check applied migrations
npx prisma migrate diff --from-empty --to-schema-datamodel prisma/schema.prisma --script   # preview SQL
npm run seed                     # manual ingest + synthesis pass
```

## Commit gate (secret → build → test, fail-closed)

Every commit and push runs a three-stage gate (`scripts/gate/gate.sh`), in this fixed order:

| Stage | pre-commit (fast) | pre-push (heavy) |
|---|---|---|
| 1 — secret | `gitleaks protect --staged` | `gitleaks protect` over full HEAD history |
| 2 — build | `tsc --noEmit` (strict) | `prisma generate && next build` (pristine production build) |
| 3 — test | `tests/llm-parse.test.ts` (LLM parse contract, 21 pins) | same |

- **Fail-closed:** a missing tool, a crash, a timeout, or any finding blocks the
  operation. Only a deliberate `git commit --no-verify` / `git push --no-verify`
  bypasses it (auditable in the terminal scrollback).
- **Secret stage:** full gitleaks default ruleset, hard-blocks. The one allow
  exception is a file- and content-scoped suppress of *empty/placeholder*
  values in `.env.example` (see `[allowlist]` in `.gitleaks.toml`) — a real key
  written into that file still fires.
- **Test stage:** `tests/llm-parse.test.ts` pins the synthesis parse contract
  (6-paragraph cap, 160-char headline, ≤5 takeaways, sanitized tags,
  JSON-fence recovery, determinism). It imports only `src/lib/llm/parse.ts` —
  pure, zero-dep, zero-network — so it runs under `tsx` with no framework.
- **Install (idempotent, per clone):**
  ```bash
  bash scripts/install/install-hooks.sh
  # → bootstraps gitleaks (pinned 8.30.1) into ~/.local/bin if missing,
  #   copies scripts/gate/hooks/* into .git/hooks/
  ```
- Walk away from a fresh machine: `npm ci && bash scripts/install/install-hooks.sh`.

## Configuration

All configuration is in `.env` (local, gitignored — only `.env.example` is committed):

| Variable | Purpose |
|---|---|
| `DATABASE_URL` | Prisma connection string (default `file:./dev.db`) |
| `NEXT_PUBLIC_SITE_NAME` / `_URL` / `_DESCRIPTION` | Site metadata used in `<head>`, sitemap, OG tags |
| `ADMIN_API_KEY` | Required admin key; sent as `x-admin-key` header. **Empty ⇒ all admin endpoints return 401.** |
| `FEED_USER_AGENT` | UA string used for feed/content fetches |
| `RUN_SCHEDULER` | `"true"`/`"false"` — kill switch for the cron worker (e.g. run pipeline only via `npm run seed`) |
| `CRON_SCHEDULE` | Worker cadence, default `*/35 * * * *` (spec: 30–60 min) |
| `MAX_SYNTH_PER_RUN` | LLM articles synthesized per pass (default 20) — throttles API cost |
| `FETCH_CONTENT` | `"true"` to download full article HTML for synthesis (makes LLM output accurate, ~few KB per article) |
| `HTTP_TIMEOUT_MS` | Per-request timeout for feed/content/LLM fetches |
| `NEXT_PUBLIC_ADSENSE_CLIENT_ID` | Your Publisher ID, e.g. `ca-pub-0123456789012345` |
| `NEXT_PUBLIC_ADSENSE_SLOT_*` | Ad-slot IDs for `HEADER` (728×90), `INFEED` (fluid), `INARTICLE` (fluid), `SIDEBAR` (300×250) |
| `OLLAMA_BASE_URL`, `OLLAMA_MODEL` | For the Ollama provider (see note below — must use HTTP URL) |

> **Note:** `NEXT_PUBLIC_ADSENSE_*` values are inlined at **build time**. After changing them, re-run `npm run build` (in dev mode they pick up on reload).

`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` in `.env` are only fallback defaults — the admin UI / settings API are the canonical place for keys (see below), and LLM calls never read them from the environment.

## LLM provider configuration

1. Build with AdSense/site env set.
2. Open `http://localhost:3000/admin/settings`.
3. Enter the admin key (held in `sessionStorage` while the tab is open — never `localStorage`) — it must match `ADMIN_API_KEY` in `.env`.
4. Pick a provider:
   - **OpenAI** — model default `gpt-4o-mini`; paste your `sk-...` key.
   - **Anthropic** — model default `claude-3-5-haiku-20241022` (change to `claude-3-5-sonnet-latest` for higher quality); paste your key.
   - **Ollama** — paste a **reachable `http://host:11434`** base URL and a model name that actually exists on that host (e.g. `llama3.1:8b`). Verify with "Test connection" — it lists the models Ollama reports.
5. Optionally edit the **synthesis prompt** (the default instructs the model to produce an original, neutral 3-paragraph rewrite plus headline, 3 takeaways, and tags as strict JSON).
6. Click **Save**. Settings persist in the DB (`Setting` table); the next cron pass (or a manual `npm run seed`) uses them immediately.

### Synthesis pipeline

For each article with status `fetched`: build a prompt from the article body → call the provider → parse the strict-JSON response (with a tolerant `safeParse`) → update the article (`synthesisJson`, `headline`, `takeaways`, `tags`, `provider`, status `synthesized`). Failures are recorded as status `failed` with the provider's error saved — the pipeline never crashes on a bad key or malformed response and retries those articles on the next pass.

## Google AdSense setup

1. Get your **Publisher ID** (`ca-pub-…`) from AdSense.
2. Create the four ad units in your AdSense account and set their slot IDs in `.env`:

   | Placement | Unit | Slot env var |
   |---|---|---|
   | Header (every page, below nav) | 728×90 | `NEXT_PUBLIC_ADSENSE_SLOT_HEADER` |
   | In-feed (after every 3rd article card) | Fluid/auto | `NEXT_PUBLIC_ADSENSE_SLOT_INFEED` |
   | In-article (after first paragraph) | Fluid/auto | `NEXT_PUBLIC_ADSENSE_SLOT_INARTICLE` |
   | Sidebar (sticky) | 300×250 (or 300×600) | `NEXT_PUBLIC_ADSENSE_SLOT_SIDEBAR` |

3. Set `NEXT_PUBLIC_ADSENSE_CLIENT_ID=ca-pub-…` and **rebuild** (`npm run build`).
4. Until every one of the five `NEXT_PUBLIC_ADSENSE_*` values is real, Ad units render as clearly-labeled placeholders (e.g. "AdUnit · 728x90 · awaiting setup") and **no `adsbygoogle` markup ships** — your pages contain zero ad code until you're published.

## Feeds

Starter (seeded on first boot into the `Feed` table):

| Feed | Category | Status (verified 2026-08) |
|---|---|---|
| CBC News — Top Stories | top-stories | ⚠️ **404 — CBC has discontinued its public RSS endpoints** (verified from multiple vantage points). Kept in the seed so it lights up if CBC restores them; disable via `Feed.enabled` or remove the row. |
| CBC News — Canada | canada | ⚠️ same as above (404) |
| CTV News — National | national | ✅ live — Arc outbound feed `https://www.ctvnews.ca/arc/outboundfeeds/rss/` (the legacy `ctvnews-ca-…-rss-1.822009` URL 404s, so the live Arc endpoint is used instead) |
| Global News — Canada | canada | ✅ live — `https://globalnews.ca/canada/feed/` |
| The Globe and Mail — National | national | ✅ live — Arc outbound feed, `/category/canada/` |
| National Post — News | top-stories | ✅ live — `https://nationalpost.com/category/news/feed` |

Dead feeds never block the pipeline — the worker logs and skips them, so 4/6 feeds live means a healthy, populated site.

To add a feed, insert a `Feed` row (Prisma Studio or SQL) or extend `STARTER_FEEDS` in `src/data/feeds.ts`. All URLs must be RSS/Atom; Atom is supported by `rss-parser`.

## API

| Endpoint | Method | Auth | Purpose |
|---|---|---|---|
| `/api/worker/ping` | GET / POST | — | Arms the background worker (serverless keep-alive; on a long-lived Node server it is already armed at boot via `src/instrumentation.ts`) |
| `/api/status` | GET | admin key | Pipeline telemetry: article counts by status, per-feed last-fetched, synthesized counts by category |
| `/api/refresh` | POST | admin key | One-shot pipeline pass (ingest all feeds + synthesize pending articles); `maxDuration` 300s |
| `/api/settings` | GET / PUT | admin key | Read / persist LLM provider, per-provider models, API keys, Ollama base URL, synthesis prompt. **Keys are never returned** — responses carry `keySources` booleans and a masked last-4 only |
| `/api/settings/test` | POST | admin key | Connectivity test against the configured provider (lists available models when provider is Ollama) |
| `/api/admin/verify` | GET | admin key | Health check: confirms the presented key matches `ADMIN_API_KEY` |
| `/api/contact` | POST | — | Contact form; logged server-side (wire an email sink — see route docblock) |

> **Admin key transport** — any of: `Authorization: Bearer *** `x-admin-key: <key>` header, `?key=<key>` query param, or HTTP Basic auth with the key as password (any username). Comparison is constant-time; an unset `ADMIN_API_KEY` fails closed (all admin endpoints 401).

## Database

Prisma schema (`prisma/schema.prisma`):

- `Feed` — id, name, url (unique), slug (unique), category, `siteName`, `enabled`
- `Article` — unique `externalId`, unique `slug`, `routePath` (unique), `feedUrl` (unique), source `title`/`url`/`image`/`publishedAt`, extracted `content`, `status` (`pending | fetched | synthesized | failed`), synthesis fields (`headline`, `takeaways[]`, `tags[]`, `synthesisJson`, `provider`), `synthesizedAt`
- `Setting` — key/value store for LLM provider config (keyed by model, e.g. `llm.model.openai`)

Migrations live in `prisma/migrations/` (initial: `20260815183102_init`). Applied via `npx prisma migrate deploy` (prod) / `migrate dev` (dev).

### Switching SQLite → PostgreSQL

1. Change `DATABASE_URL` in `.env` to your Postgres string.
2. Change `provider = "sqlite"` → `"postgresql"` in `prisma/schema.prisma`.
3. `npx prisma migrate dev --name init` (fresh DB) and rebuild. No app code changes required — Prisma abstracts the driver.

## Project structure (abridged)

```
prisma/            schema.prisma, migrations/, seed.ts (ingest+synthesis pass)
src/instrumentation.ts  Next.js boot hook — arms the worker on process start (Node runtime only)
src/data/feeds.ts  starter feed list + site sections
src/lib/
  db.ts            Prisma client singleton
  env.ts           typed env access
  format.ts        HTML→text, excerpt helpers
  auth/admin.ts    x-admin-key verification (timing-safe, fail-closed)
  llm/             providers: openai.ts, anthropic.ts, ollama.ts, parse.ts, types.ts
  llm/engine.ts    provider resolution + synthesizeArticle()
  ingest/          feed.ts (RSS→rows), html.ts (content fetch), pipeline.ts (dedupe+persist)
  worker/          cron.ts — node-cron loop, instrumentation.ts — boot hook
  queries.ts       read-model queries for pages
src/app/
  layout.tsx       global shell (header ad on every page), metadata, sitemap, robots
  page.tsx         home (featured story + top-stories feed + sidebar)
  [section]/       section feeds (top-stories, canada, national, all)
  article/[slug]/  article detail (attribution, in-article ad, related, OG, canonical)
  admin/settings/  protected admin UI (key gate + settings form)
  about|contact|privacy-policy|terms-of-service|not-found/   legal + 404
  api/             routes above
src/components/    card, ads (AdUnit), layout (Header/Footer/Sidebar), legal
```

## Verification performed (2026-08-15)

Real, executed checks — not assumptions:

- `npx tsc --noEmit` — clean; `npm run build` — passes (20 routes, static + dynamic).
- Live ingest: **60 real Canadian articles** pulled across Global, Globe and Mail, National Post and CTV; full bodies extracted (up to ~12k chars); slugs clean and deduped.
- Synthesis happy path: real chain (provider → strict-JSON parse → DB update) verified — articles reach `synthesized` with headline/takeaways/tags/provider populated.
- Synthesis failure path: real OpenAI call with a placeholder key returned 401 → article cleanly marked `failed`, no crash, retriable next pass.
- Served app: all routes 200 (home, sections, article detail, legal, sitemap, robots, status/ping/settings endpoints).
- Admin: wrong/missing key → 401 on every admin endpoint (settings, status, refresh, verify); correct key → settings GET/PUT round-trips correctly (keys never echoed — `keySources` booleans + last-4 only); provider test endpoint responds.
- AdSense: with a test publisher ID, real `<ins class="adsbygoogle">` markup SSRs with correct client/slot/format on every page (header), section pages (in-feed), and article pages (in-article + sidebar); with placeholder values, clean labeled placeholders and zero `adsbygoogle` markup.

## Policy & content caveats (read before monetizing)

- **LLM-rewritten news is a grey zone** for Google AdSense's scraped/reshaped-content policies, and for copyright (summary/derivative-work doctrine varies by jurisdiction). This project mitigates, not eliminates, that risk: every page carries source attribution, `nofollow` source links, and a visible LLM-rewrite disclosure, and the synthesis prompt instructs the model to stay factual and neutral. If you run with AdSense enabled, expect to hold the scale small, monitor your AdSense account for policy messages, and be prepared that account suspension is possible.
- **CBC feeds are dead** (documented above) — their 404s are expected and harmless; ignore `feed emitted 0 items` log lines for the `cbc-*` feeds.

## License

Internal project — see Gitea repo `krisf/maple-brief-aggregator`.