maple-brief-aggregator/README.md

MapleBrief

An automated Canadian news aggregator with LLM content synthesis and Google AdSense integration.

MapleBrief pulls stories from Canadian news RSS feeds on a schedule, fetches the full article text, has an LLM rewrite them into original neutral reports (with headline, takeaways, and tags), and serves them on an ad-optimized Next.js site with full source attribution.

Features

• Scheduled ingestion — a node-cron background worker (in-process, starts with the web server) fetches feeds, dedupes articles, extracts full article bodies with cheerio, and synthesizes pending articles every 35 minutes (configurable).
• LLM abstraction, three providers — OpenAI (default: gpt-4o-mini), Anthropic (Claude 3.5 Sonnet / Haiku), and Ollama (local). Provider, model per-provider, API keys, and the synthesis prompt are all stored in the database and editable live from the admin UI — no code changes or restarts needed.
• Keyless-first, fail-closed admin — /admin/settings and the settings APIs require x-admin-key: $ADMIN_API_KEY. Empty/missing key → every admin endpoint returns 401 (fail-closed). Comparison is timing-safe.
• AdSense-ready layout — header leaderboard (728×90) on every page, in-feed ads every 3rd card, in-article ad after the first paragraph, and a sticky 300×250 / 300×600 sidebar. Ad units render as labeled placeholders until a real Publisher ID is configured.
• Compliance & SEO — "Sources analyzed: …" attribution on every article, rel="nofollow external noopener" on all source links, per-article canonical + OpenGraph tags, sitemap.xml, robots.txt, and four legal pages (/about, /privacy-policy, /terms-of-service, /contact).

Tech stack

Layer	Choice
Framework	Next.js 14 (App Router) + TypeScript (strict) + Tailwind CSS
Database / ORM	SQLite via Prisma (zero-ops local dev; Postgres switch documented below)
Scheduler	node-cron worker in the server process
Ingestion	rss-parser + cheerio (content extraction), native fetch
LLM clients	native fetch only (OpenAI, Anthropic, Ollama) — no SDK dependencies
Ad framework	Google AdSense (<AdUnit> + NEXT_PUBLIC_ADSENSE_* env)

Quickstart

Requirements: Node.js ≥ 18 (tested on v22), npm.

git clone https://gitea.krisforbes.ca/krisf/maple-brief-aggregator.git
cd maple-brief-aggregator

npm install

# 1. Environment
cp .env.example .env
#    then edit .env — at minimum set:
#      DATABASE_URL (default is fine for SQLite: file:./dev.db)
#      CRON_SCHEDULE (default "*/35 * * * *")
#      NEXT_PUBLIC_SITE_URL (deployment URL, without trailing slash)

# 2. Database — create tables and generate the Prisma client
npx prisma migrate deploy
#    (in dev: npx prisma migrate dev)

# 3. First ingest — pulls feeds, saves articles, synthesizes anything pending
npm run seed

# 4. Build & run (production)
npm run build
npm start            # http://localhost:3000

# — or development with hot reload:
npm run dev

On boot, the src/instrumentation.ts hook (Node.js runtime only) arms the worker, which immediately runs an initial ingest + synthesis pass and then repeats on CRON_SCHEDULE. The worker is idempotent (runPipelinePass is re-entrancy-guarded), so restarts and double-starts are safe.

Useful commands

npx prisma studio                 # browse the DB
npx prisma migrate status         # check applied migrations
npx prisma migrate diff --from-empty --to-schema-datamodel prisma/schema.prisma --script   # preview SQL
npm run seed                     # manual ingest + synthesis pass

Commit gate (secret → build → test, fail-closed)

Every commit and push runs a three-stage gate (scripts/gate/gate.sh), in this fixed order:

Stage	pre-commit (fast)	pre-push (heavy)
1 — secret	gitleaks protect --staged	gitleaks protect over full HEAD history
2 — build	tsc --noEmit (strict)	prisma generate && next build (pristine production build)
3 — test	tests/llm-parse.test.ts (LLM parse contract, 21 pins)	same

• Fail-closed: a missing tool, a crash, a timeout, or any finding blocks the operation. Only a deliberate git commit --no-verify / git push --no-verify bypasses it (auditable in the terminal scrollback).
• Secret stage: full gitleaks default ruleset, hard-blocks. The one allow exception is a file- and content-scoped suppress of empty/placeholder values in .env.example (see [allowlist] in .gitleaks.toml) — a real key written into that file still fires.
• Test stage: tests/llm-parse.test.ts pins the synthesis parse contract (6-paragraph cap, 160-char headline, ≤5 takeaways, sanitized tags, JSON-fence recovery, determinism). It imports only src/lib/llm/parse.ts — pure, zero-dep, zero-network — so it runs under tsx with no framework.
• Install (idempotent, per clone):

  bash scripts/install/install-hooks.sh
  # → bootstraps gitleaks (pinned 8.30.1) into ~/.local/bin if missing,
  #   copies scripts/gate/hooks/* into .git/hooks/
  

• Walk away from a fresh machine: npm ci && bash scripts/install/install-hooks.sh.

Configuration

All configuration is in .env (local, gitignored — only .env.example is committed):

Variable	Purpose
DATABASE_URL	Prisma connection string (default file:./dev.db)
NEXT_PUBLIC_SITE_NAME / _URL / _DESCRIPTION	Site metadata used in <head>, sitemap, OG tags
ADMIN_API_KEY	Required admin key; sent as x-admin-key header. Empty ⇒ all admin endpoints return 401.
FEED_USER_AGENT	UA string used for feed/content fetches
RUN_SCHEDULER	"true"/"false" — kill switch for the cron worker (e.g. run pipeline only via npm run seed)
CRON_SCHEDULE	Worker cadence, default */35 * * * * (spec: 30–60 min)
MAX_SYNTH_PER_RUN	LLM articles synthesized per pass (default 20) — throttles API cost
FETCH_CONTENT	"true" to download full article HTML for synthesis (makes LLM output accurate, ~few KB per article)
HTTP_TIMEOUT_MS	Per-request timeout for feed/content/LLM fetches
NEXT_PUBLIC_ADSENSE_CLIENT_ID	Your Publisher ID, e.g. ca-pub-0123456789012345
NEXT_PUBLIC_ADSENSE_SLOT_*	Ad-slot IDs for HEADER (728×90), INFEED (fluid), INARTICLE (fluid), SIDEBAR (300×250)
OLLAMA_BASE_URL, OLLAMA_MODEL	For the Ollama provider (see note below — must use HTTP URL)

Note: NEXT_PUBLIC_ADSENSE_* values are inlined at build time. After changing them, re-run npm run build (in dev mode they pick up on reload).

OPENAI_API_KEY / ANTHROPIC_API_KEY in .env are only fallback defaults — the admin UI / settings API are the canonical place for keys (see below), and LLM calls never read them from the environment.

LLM provider configuration

1. Build with AdSense/site env set.
2. Open http://localhost:3000/admin/settings.
3. Enter the admin key (held in sessionStorage while the tab is open — never localStorage) — it must match ADMIN_API_KEY in .env.
4. Pick a provider:
   • OpenAI — model default gpt-4o-mini; paste your sk-... key.
   • Anthropic — model default claude-3-5-haiku-20241022 (change to claude-3-5-sonnet-latest for higher quality); paste your key.
   • Ollama — paste a reachable http://host:11434 base URL and a model name that actually exists on that host (e.g. llama3.1:8b). Verify with "Test connection" — it lists the models Ollama reports.
5. Optionally edit the synthesis prompt (the default instructs the model to produce an original, neutral 3-paragraph rewrite plus headline, 3 takeaways, and tags as strict JSON).
6. Click Save. Settings persist in the DB (Setting table); the next cron pass (or a manual npm run seed) uses them immediately.

Synthesis pipeline

For each article with status fetched: build a prompt from the article body → call the provider → parse the strict-JSON response (with a tolerant safeParse) → update the article (synthesisJson, headline, takeaways, tags, provider, status synthesized). Failures are recorded as status failed with the provider's error saved — the pipeline never crashes on a bad key or malformed response and retries those articles on the next pass.

Google AdSense setup

1. Get your Publisher ID (ca-pub-…) from AdSense.

2. Create the four ad units in your AdSense account and set their slot IDs in .env:

   Placement	Unit	Slot env var
   Header (every page, below nav)	728×90	NEXT_PUBLIC_ADSENSE_SLOT_HEADER
   In-feed (after every 3rd article card)	Fluid/auto	NEXT_PUBLIC_ADSENSE_SLOT_INFEED
   In-article (after first paragraph)	Fluid/auto	NEXT_PUBLIC_ADSENSE_SLOT_INARTICLE
   Sidebar (sticky)	300×250 (or 300×600)	NEXT_PUBLIC_ADSENSE_SLOT_SIDEBAR

3. Set NEXT_PUBLIC_ADSENSE_CLIENT_ID=ca-pub-… and rebuild (npm run build).

4. Until every one of the five NEXT_PUBLIC_ADSENSE_* values is real, Ad units render as clearly-labeled placeholders (e.g. "AdUnit · 728x90 · awaiting setup") and no adsbygoogle markup ships — your pages contain zero ad code until you're published.

Feeds

Starter (seeded on first boot into the Feed table):

Feed	Category	Status (verified 2026-08)
CBC News — Top Stories	top-stories	⚠️ 404 — CBC has discontinued its public RSS endpoints (verified from multiple vantage points). Kept in the seed so it lights up if CBC restores them; disable via Feed.enabled or remove the row.
CBC News — Canada	canada	⚠️ same as above (404)
CTV News — National	national	✅ live — Arc outbound feed https://www.ctvnews.ca/arc/outboundfeeds/rss/ (the legacy ctvnews-ca-…-rss-1.822009 URL 404s, so the live Arc endpoint is used instead)
Global News — Canada	canada	✅ live — https://globalnews.ca/canada/feed/
The Globe and Mail — National	national	✅ live — Arc outbound feed, /category/canada/
National Post — News	top-stories	✅ live — https://nationalpost.com/category/news/feed

Dead feeds never block the pipeline — the worker logs and skips them, so 4/6 feeds live means a healthy, populated site.

To add a feed, insert a Feed row (Prisma Studio or SQL) or extend STARTER_FEEDS in src/data/feeds.ts. All URLs must be RSS/Atom; Atom is supported by rss-parser.

API

Endpoint	Method	Auth	Purpose
/api/worker/ping	GET / POST	—	Arms the background worker (serverless keep-alive; on a long-lived Node server it is already armed at boot via src/instrumentation.ts)
/api/status	GET	admin key	Pipeline telemetry: article counts by status, per-feed last-fetched, synthesized counts by category
/api/refresh	POST	admin key	One-shot pipeline pass (ingest all feeds + synthesize pending articles); maxDuration 300s
/api/settings	GET / PUT	admin key	Read / persist LLM provider, per-provider models, API keys, Ollama base URL, synthesis prompt. Keys are never returned — responses carry keySources booleans and a masked last-4 only
/api/settings/test	POST	admin key	Connectivity test against the configured provider (lists available models when provider is Ollama)
/api/admin/verify	GET	admin key	Health check: confirms the presented key matches ADMIN_API_KEY
/api/contact	POST	—	Contact form; logged server-side (wire an email sink — see route docblock)

Admin key transport — any of: Authorization: Bearer *** x-admin-key: header,?key=query param, or HTTP Basic auth with the key as password (any username). Comparison is constant-time; an unsetADMIN_API_KEY` fails closed (all admin endpoints 401).

Database

Prisma schema (prisma/schema.prisma):

• Feed — id, name, url (unique), slug (unique), category, siteName, enabled
• Article — unique externalId, unique slug, routePath (unique), feedUrl (unique), source title/url/image/publishedAt, extracted content, status (pending | fetched | synthesized | failed), synthesis fields (headline, takeaways[], tags[], synthesisJson, provider), synthesizedAt
• Setting — key/value store for LLM provider config (keyed by model, e.g. llm.model.openai)

Migrations live in prisma/migrations/ (initial: 20260815183102_init). Applied via npx prisma migrate deploy (prod) / migrate dev (dev).

Switching SQLite → PostgreSQL

1. Change DATABASE_URL in .env to your Postgres string.
2. Change provider = "sqlite" → "postgresql" in prisma/schema.prisma.
3. npx prisma migrate dev --name init (fresh DB) and rebuild. No app code changes required — Prisma abstracts the driver.

Project structure (abridged)

prisma/            schema.prisma, migrations/, seed.ts (ingest+synthesis pass)
src/instrumentation.ts  Next.js boot hook — arms the worker on process start (Node runtime only)
src/data/feeds.ts  starter feed list + site sections
src/lib/
  db.ts            Prisma client singleton
  env.ts           typed env access
  format.ts        HTML→text, excerpt helpers
  auth/admin.ts    x-admin-key verification (timing-safe, fail-closed)
  llm/             providers: openai.ts, anthropic.ts, ollama.ts, parse.ts, types.ts
  llm/engine.ts    provider resolution + synthesizeArticle()
  ingest/          feed.ts (RSS→rows), html.ts (content fetch), pipeline.ts (dedupe+persist)
  worker/          cron.ts — node-cron loop, instrumentation.ts — boot hook
  queries.ts       read-model queries for pages
src/app/
  layout.tsx       global shell (header ad on every page), metadata, sitemap, robots
  page.tsx         home (featured story + top-stories feed + sidebar)
  [section]/       section feeds (top-stories, canada, national, all)
  article/[slug]/  article detail (attribution, in-article ad, related, OG, canonical)
  admin/settings/  protected admin UI (key gate + settings form)
  about|contact|privacy-policy|terms-of-service|not-found/   legal + 404
  api/             routes above
src/components/    card, ads (AdUnit), layout (Header/Footer/Sidebar), legal

Verification performed (2026-08-15)

Real, executed checks — not assumptions:

• npx tsc --noEmit — clean; npm run build — passes (20 routes, static + dynamic).
• Live ingest: 60 real Canadian articles pulled across Global, Globe and Mail, National Post and CTV; full bodies extracted (up to ~12k chars); slugs clean and deduped.
• Synthesis happy path: real chain (provider → strict-JSON parse → DB update) verified — articles reach synthesized with headline/takeaways/tags/provider populated.
• Synthesis failure path: real OpenAI call with a placeholder key returned 401 → article cleanly marked failed, no crash, retriable next pass.
• Served app: all routes 200 (home, sections, article detail, legal, sitemap, robots, status/ping/settings endpoints).
• Admin: wrong/missing key → 401 on every admin endpoint (settings, status, refresh, verify); correct key → settings GET/PUT round-trips correctly (keys never echoed — keySources booleans + last-4 only); provider test endpoint responds.
• AdSense: with a test publisher ID, real <ins class="adsbygoogle"> markup SSRs with correct client/slot/format on every page (header), section pages (in-feed), and article pages (in-article + sidebar); with placeholder values, clean labeled placeholders and zero adsbygoogle markup.

Policy & content caveats (read before monetizing)

• LLM-rewritten news is a grey zone for Google AdSense's scraped/reshaped-content policies, and for copyright (summary/derivative-work doctrine varies by jurisdiction). This project mitigates, not eliminates, that risk: every page carries source attribution, nofollow source links, and a visible LLM-rewrite disclosure, and the synthesis prompt instructs the model to stay factual and neutral. If you run with AdSense enabled, expect to hold the scale small, monitor your AdSense account for policy messages, and be prepared that account suspension is possible.
• CBC feeds are dead (documented above) — their 404s are expected and harmless; ignore feed emitted 0 items log lines for the cbc-* feeds.

License

Internal project — see Gitea repo krisf/maple-brief-aggregator.