krisf/maple-brief-aggregator
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
This repository has been archived on 2026-08-19. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
9ebc3bffab4f6fe708724d4f1e767f12eea9b722
maple-brief-aggregator/README.md
T
krisf 1010f27f44 MapleBrief: production Canadian news aggregator 
Next.js 14 + TypeScript + Tailwind + Prisma/SQLite.
- LLM synthesis abstraction (OpenAI/Anthropic/Ollama) + admin settings UI
- RSS ingestion pipeline (parser + cheerio content) + node-cron worker
- AdSense AdUnit placements (header/in-feed/in-article/sidebar)
- Sources analysis attribution, nofollow links, canonical/OG, sitemap/robots
- Admin auth (ADMIN_API_KEY, timingSafeEqual, fail-closed 401)
- Multi-cell lady-ga-ga marker, sitemap, robots, legal pages
- Comprehensive README (setup, LLM config, AdSense, migrations, deploy)
2026-08-15 16:07:07 -04:00

15 KiB
Raw Blame History

MapleBrief

An automated Canadian news aggregator with LLM content synthesis and Google AdSense integration.

MapleBrief pulls stories from Canadian news RSS feeds on a schedule, fetches the full article text, has an LLM rewrite them into original neutral reports (with headline, takeaways, and tags), and serves them on an ad-optimized Next.js site with full source attribution.

Features

  • Scheduled ingestion — a node-cron background worker (in-process, starts with the web server) fetches feeds, dedupes articles, extracts full article bodies with cheerio, and synthesizes pending articles every 35 minutes (configurable).
  • LLM abstraction, three providers — OpenAI (default: gpt-4o-mini), Anthropic (Claude 3.5 Sonnet / Haiku), and Ollama (local). Provider, model per-provider, API keys, and the synthesis prompt are all stored in the database and editable live from the admin UI — no code changes or restarts needed.
  • Keyless-first, fail-closed admin — /admin/settings and the settings APIs require x-admin-key: $ADMIN_API_KEY. Empty/missing key → every admin endpoint returns 401 (fail-closed). Comparison is timing-safe.
  • AdSense-ready layout — header leaderboard (728×90) on every page, in-feed ads every 3rd card, in-article ad after the first paragraph, and a sticky 300×250 / 300×600 sidebar. Ad units render as labeled placeholders until a real Publisher ID is configured.
  • Compliance & SEO — "Sources analyzed: …" attribution on every article, rel="nofollow external noopener" on all source links, per-article canonical + OpenGraph tags, sitemap.xml, robots.txt, and four legal pages (/about, /privacy-policy, /terms-of-service, /contact).

Tech stack

Layer Choice
Framework Next.js 14 (App Router) + TypeScript (strict) + Tailwind CSS
Database / ORM SQLite via Prisma (zero-ops local dev; Postgres switch documented below)
Scheduler node-cron worker in the server process
Ingestion rss-parser + cheerio (content extraction), native fetch
LLM clients native fetch only (OpenAI, Anthropic, Ollama) — no SDK dependencies
Ad framework Google AdSense (<AdUnit> + NEXT_PUBLIC_ADSENSE_* env)

Quickstart

Requirements: Node.js ≥ 18 (tested on v22), npm.

git clone https://gitea.krisforbes.ca/krisf/maple-brief-aggregator.git
cd maple-brief-aggregator

npm install

# 1. Environment
cp .env.example .env
#    then edit .env — at minimum set:
#      DATABASE_URL (default is fine for SQLite: file:./dev.db)
#      CRON_SCHEDULE (default "*/35 * * * *")
#      NEXT_PUBLIC_SITE_URL (deployment URL, without trailing slash)

# 2. Database — create tables and generate the Prisma client
npx prisma migrate deploy
#    (in dev: npx prisma migrate dev)

# 3. First ingest — pulls feeds, saves articles, synthesizes anything pending
npm run seed

# 4. Build & run (production)
npm run build
npm start            # http://localhost:3000

# — or development with hot reload:
npm run dev

On boot, the src/instrumentation.ts hook (Node.js runtime only) arms the worker, which immediately runs an initial ingest + synthesis pass and then repeats on CRON_SCHEDULE. The worker is idempotent (runPipelinePass is re-entrancy-guarded), so restarts and double-starts are safe.

Useful commands

npx prisma studio                 # browse the DB
npx prisma migrate status         # check applied migrations
npx prisma migrate diff --from-empty --to-schema-datamodel prisma/schema.prisma --script   # preview SQL
npm run seed                     # manual ingest + synthesis pass

Configuration

All configuration is in .env (local, gitignored — only .env.example is committed):

Variable Purpose
DATABASE_URL Prisma connection string (default file:./dev.db)
NEXT_PUBLIC_SITE_NAME / _URL / _DESCRIPTION Site metadata used in <head>, sitemap, OG tags
ADMIN_API_KEY Required admin key; sent as x-admin-key header. Empty ⇒ all admin endpoints return 401.
FEED_USER_AGENT UA string used for feed/content fetches
RUN_SCHEDULER "true"/"false" — kill switch for the cron worker (e.g. run pipeline only via npm run seed)
CRON_SCHEDULE Worker cadence, default */35 * * * * (spec: 30–60 min)
MAX_SYNTH_PER_RUN LLM articles synthesized per pass (default 20) — throttles API cost
FETCH_CONTENT "true" to download full article HTML for synthesis (makes LLM output accurate, ~few KB per article)
HTTP_TIMEOUT_MS Per-request timeout for feed/content/LLM fetches
NEXT_PUBLIC_ADSENSE_CLIENT_ID Your Publisher ID, e.g. ca-pub-0123456789012345
NEXT_PUBLIC_ADSENSE_SLOT_* Ad-slot IDs for HEADER (728×90), INFEED (fluid), INARTICLE (fluid), SIDEBAR (300×250)
OLLAMA_BASE_URL, OLLAMA_MODEL For the Ollama provider (see note below — must use HTTP URL)

Note: NEXT_PUBLIC_ADSENSE_* values are inlined at build time. After changing them, re-run npm run build (in dev mode they pick up on reload).

OPENAI_API_KEY / ANTHROPIC_API_KEY in .env are only fallback defaults — the admin UI / settings API are the canonical place for keys (see below), and LLM calls never read them from the environment.

LLM provider configuration

  1. Build with AdSense/site env set.
  2. Open http://localhost:3000/admin/settings.
  3. Enter the admin key (held in sessionStorage while the tab is open — never localStorage) — it must match ADMIN_API_KEY in .env.
  4. Pick a provider:
    • OpenAI — model default gpt-4o-mini; paste your sk-... key.
    • Anthropic — model default claude-3-5-haiku-20241022 (change to claude-3-5-sonnet-latest for higher quality); paste your key.
    • Ollama — paste a reachable http://host:11434 base URL and a model name that actually exists on that host (e.g. llama3.1:8b). Verify with "Test connection" — it lists the models Ollama reports.
  5. Optionally edit the synthesis prompt (the default instructs the model to produce an original, neutral 3-paragraph rewrite plus headline, 3 takeaways, and tags as strict JSON).
  6. Click Save. Settings persist in the DB (Setting table); the next cron pass (or a manual npm run seed) uses them immediately.

Synthesis pipeline

For each article with status fetched: build a prompt from the article body → call the provider → parse the strict-JSON response (with a tolerant safeParse) → update the article (synthesisJson, headline, takeaways, tags, provider, status synthesized). Failures are recorded as status failed with the provider's error saved — the pipeline never crashes on a bad key or malformed response and retries those articles on the next pass.

Google AdSense setup

  1. Get your Publisher ID (ca-pub-…) from AdSense.

  2. Create the four ad units in your AdSense account and set their slot IDs in .env:

    Placement Unit Slot env var
    Header (every page, below nav) 728×90 NEXT_PUBLIC_ADSENSE_SLOT_HEADER
    In-feed (after every 3rd article card) Fluid/auto NEXT_PUBLIC_ADSENSE_SLOT_INFEED
    In-article (after first paragraph) Fluid/auto NEXT_PUBLIC_ADSENSE_SLOT_INARTICLE
    Sidebar (sticky) 300×250 (or 300×600) NEXT_PUBLIC_ADSENSE_SLOT_SIDEBAR

  3. Set NEXT_PUBLIC_ADSENSE_CLIENT_ID=ca-pub-… and rebuild (npm run build).

  4. Until every one of the five NEXT_PUBLIC_ADSENSE_* values is real, Ad units render as clearly-labeled placeholders (e.g. "AdUnit · 728x90 · awaiting setup") and no adsbygoogle markup ships — your pages contain zero ad code until you're published.

Feeds

Starter (seeded on first boot into the Feed table):

Feed Category Status (verified 2026-08)
CBC News — Top Stories top-stories ⚠️ 404 — CBC has discontinued its public RSS endpoints (verified from multiple vantage points). Kept in the seed so it lights up if CBC restores them; disable via Feed.enabled or remove the row.
CBC News — Canada canada ⚠️ same as above (404)
CTV News — National national ✅ live — Arc outbound feed https://www.ctvnews.ca/arc/outboundfeeds/rss/ (the legacy ctvnews-ca-…-rss-1.822009 URL 404s, so the live Arc endpoint is used instead)
Global News — Canada canada ✅ live — https://globalnews.ca/canada/feed/
The Globe and Mail — National national ✅ live — Arc outbound feed, /category/canada/
National Post — News top-stories ✅ live — https://nationalpost.com/category/news/feed

Dead feeds never block the pipeline — the worker logs and skips them, so 4/6 feeds live means a healthy, populated site.

To add a feed, insert a Feed row (Prisma Studio or SQL) or extend STARTER_FEEDS in src/data/feeds.ts. All URLs must be RSS/Atom; Atom is supported by rss-parser.

API

Endpoint Method Auth Purpose
/api/worker/ping GET / POST — Arms the background worker (serverless keep-alive; on a long-lived Node server it is already armed at boot via src/instrumentation.ts)
/api/status GET admin key Pipeline telemetry: article counts by status, per-feed last-fetched, synthesized counts by category
/api/refresh POST admin key One-shot pipeline pass (ingest all feeds + synthesize pending articles); maxDuration 300s
/api/settings GET / PUT admin key Read / persist LLM provider, per-provider models, API keys, Ollama base URL, synthesis prompt. Keys are never returned — responses carry keySources booleans and a masked last-4 only
/api/settings/test POST admin key Connectivity test against the configured provider (lists available models when provider is Ollama)
/api/admin/verify GET admin key Health check: confirms the presented key matches ADMIN_API_KEY
/api/contact POST — Contact form; logged server-side (wire an email sink — see route docblock)

Admin key transport — any of: Authorization: Bearer *** x-admin-key: header,?key=query param, or HTTP Basic auth with the key as password (any username). Comparison is constant-time; an unsetADMIN_API_KEY` fails closed (all admin endpoints 401).

Database

Prisma schema (prisma/schema.prisma):

  • Feed — id, name, url (unique), slug (unique), category, siteName, enabled
  • Article — unique externalId, unique slug, routePath (unique), feedUrl (unique), source title/url/image/publishedAt, extracted content, status (pending | fetched | synthesized | failed), synthesis fields (headline, takeaways[], tags[], synthesisJson, provider), synthesizedAt
  • Setting — key/value store for LLM provider config (keyed by model, e.g. llm.model.openai)

Migrations live in prisma/migrations/ (initial: 20260815183102_init). Applied via npx prisma migrate deploy (prod) / migrate dev (dev).

Switching SQLite → PostgreSQL

  1. Change DATABASE_URL in .env to your Postgres string.
  2. Change provider = "sqlite" → "postgresql" in prisma/schema.prisma.
  3. npx prisma migrate dev --name init (fresh DB) and rebuild. No app code changes required — Prisma abstracts the driver.

Project structure (abridged)

prisma/            schema.prisma, migrations/, seed.ts (ingest+synthesis pass)
src/instrumentation.ts  Next.js boot hook — arms the worker on process start (Node runtime only)
src/data/feeds.ts  starter feed list + site sections
src/lib/
  db.ts            Prisma client singleton
  env.ts           typed env access
  format.ts        HTML→text, excerpt helpers
  auth/admin.ts    x-admin-key verification (timing-safe, fail-closed)
  llm/             providers: openai.ts, anthropic.ts, ollama.ts, parse.ts, types.ts
  llm/engine.ts    provider resolution + synthesizeArticle()
  ingest/          feed.ts (RSS→rows), html.ts (content fetch), pipeline.ts (dedupe+persist)
  worker/          cron.ts — node-cron loop, instrumentation.ts — boot hook
  queries.ts       read-model queries for pages
src/app/
  layout.tsx       global shell (header ad on every page), metadata, sitemap, robots
  page.tsx         home (featured story + top-stories feed + sidebar)
  [section]/       section feeds (top-stories, canada, national, all)
  article/[slug]/  article detail (attribution, in-article ad, related, OG, canonical)
  admin/settings/  protected admin UI (key gate + settings form)
  about|contact|privacy-policy|terms-of-service|not-found/   legal + 404
  api/             routes above
src/components/    card, ads (AdUnit), layout (Header/Footer/Sidebar), legal

Verification performed (2026-08-15)

Real, executed checks — not assumptions:

  • npx tsc --noEmit — clean; npm run build — passes (20 routes, static + dynamic).
  • Live ingest: 60 real Canadian articles pulled across Global, Globe and Mail, National Post and CTV; full bodies extracted (up to ~12k chars); slugs clean and deduped.
  • Synthesis happy path: real chain (provider → strict-JSON parse → DB update) verified — articles reach synthesized with headline/takeaways/tags/provider populated.
  • Synthesis failure path: real OpenAI call with a placeholder key returned 401 → article cleanly marked failed, no crash, retriable next pass.
  • Served app: all routes 200 (home, sections, article detail, legal, sitemap, robots, status/ping/settings endpoints).
  • Admin: wrong/missing key → 401 on every admin endpoint (settings, status, refresh, verify); correct key → settings GET/PUT round-trips correctly (keys never echoed — keySources booleans + last-4 only); provider test endpoint responds.
  • AdSense: with a test publisher ID, real <ins class="adsbygoogle"> markup SSRs with correct client/slot/format on every page (header), section pages (in-feed), and article pages (in-article + sidebar); with placeholder values, clean labeled placeholders and zero adsbygoogle markup.

Policy & content caveats (read before monetizing)

  • LLM-rewritten news is a grey zone for Google AdSense's scraped/reshaped-content policies, and for copyright (summary/derivative-work doctrine varies by jurisdiction). This project mitigates, not eliminates, that risk: every page carries source attribution, nofollow source links, and a visible LLM-rewrite disclosure, and the synthesis prompt instructs the model to stay factual and neutral. If you run with AdSense enabled, expect to hold the scale small, monitor your AdSense account for policy messages, and be prepared that account suspension is possible.
  • CBC feeds are dead (documented above) — their 404s are expected and harmless; ignore feed emitted 0 items log lines for the cbc-* feeds.

License

Internal project — see Gitea repo krisf/maple-brief-aggregator.

Reference in New Issue View Git Blame Copy Permalink