Add blog post: Local LLM adventures
This commit is contained in:
Your Name
@@ -0,0 +1,107 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
|
||||||
|
<title>My Adventures Running Large Language Models Locally</title>
|
||||||
|
|
||||||
|
<meta name="description" content="A practical, opinion-free account of what it takes to run a 27-billion-parameter LLM on local hardware — model quantization, Ollama quirks, and the patience tax.">
|
||||||
|
<meta property="og:title" content="My Adventures Running Large Language Models Locally">
|
||||||
|
<meta property="og:description" content="What it takes to run a 27-billion-parameter LLM on local hardware — quantization, Ollama quirks, and the patience tax.">
|
||||||
|
<meta property="og:type" content="article">
|
||||||
|
<meta property="og:url" content="https://krisforbes.ca/2025/08/05/local-llm-adventures.html">
|
||||||
|
<link rel="canonical" href="https://krisforbes.ca/2025/08/05/local-llm-adventures.html">
|
||||||
|
|
||||||
|
<link rel="stylesheet" href="/assets/css/main.css">
|
||||||
|
<link rel="canonical" href="https://krisforbes.ca/2025/08/05/local-llm-adventures.html">
|
||||||
|
<link rel="alternate" type="application/rss+xml" title="https://krisforbes.ca" href="/feed.xml">
|
||||||
|
<script async src="https://www.googletagmanager.com/gtag/js?id=G-7J4BKPGWR1"></script>
|
||||||
|
<script>
|
||||||
|
window.dataLayer = window.dataLayer || [];
|
||||||
|
function gtag(){dataLayer.push(arguments);}
|
||||||
|
gtag('js', new Date());
|
||||||
|
gtag('config', 'G-7J4BKPGWR1');
|
||||||
|
</script>
|
||||||
|
</head>
|
||||||
|
|
||||||
|
<body>
|
||||||
|
|
||||||
|
<header class="site-header" role="banner">
|
||||||
|
|
||||||
|
<div class="wrapper">
|
||||||
|
|
||||||
|
<a class="site-title" href="/">Kris Forbes</a>
|
||||||
|
|
||||||
|
<nav class="site-nav">
|
||||||
|
<input type="checkbox" id="nav-trigger" class="nav-trigger" />
|
||||||
|
<label for="nav-trigger">
|
||||||
|
<span class="menu-icon">
|
||||||
|
<svg viewBox="0 0 18 15" width="18px" height="15px">
|
||||||
|
<path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.695,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/>
|
||||||
|
</svg>
|
||||||
|
</span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<div class="trigger">
|
||||||
|
<a class="page-link" href="/">Blog</a>
|
||||||
|
</div>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main class="page-content" aria-role="main">
|
||||||
|
<div class="wrapper">
|
||||||
|
<article class="post">
|
||||||
|
|
||||||
|
<header class="post-header">
|
||||||
|
<h1 class="post-title">My Adventures Running Large Language Models Locally</h1>
|
||||||
|
<p class="post-meta">Aug 5, 2025</p>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<div class="post-content">
|
||||||
|
<p>A couple of years ago I ran my AI experiments through hosted APIs: fast, easy, and entirely out of my control — on privacy, cost, and availability. A while back I decided to move the heavy lifting onto my own hardware, and since then my "cloud bill" has been a power meter and an ongoing education in what running a real, 27-billion-parameter language model at home actually involves. This is a (non-sensitive) tour of that journey.</p>
|
||||||
|
|
||||||
|
<h2 id="quantization">The Model Was Too Big, So I Made It Smaller</h2>
|
||||||
|
<p>Full-precision copies of modern open-weight models don't fit on consumer GPUs — not even close. The answer is <strong>quantization</strong>: compressing the model's internal weights to lower-precision numbers. The GGUF container format has become the de facto standard for this, and the naming is its own little cryptic language. A filename like <code>Qwen3.6-27B-UD-Q5_K_XL</code> tells you everything: a 27-billion-parameter model, quantized to 5-bit with a K-quant scheme, in an "extra large" variant that keeps a few sensitive layers at higher precision to protect quality. You trade raw fidelity for fit — and the sweet spot on my hardware turned out to be right around 5-bit, where output quality is genuinely impressive and the model actually loads in a reasonable amount of memory.</p>
|
||||||
|
|
||||||
|
<h2 id="ollama">Serving: Ollama's Good, but It Has Quirks</h2>
|
||||||
|
<p>For actually running the model I've been serving it with <a href="https://ollama.com" rel="noopener">Ollama</a>, which wraps model downloads, quantization, and inference behind a clean local API. It has been mostly great — but I've hit quirks that cost me real debugging time.</p>
|
||||||
|
<p>The most maddening: pulling a fresh GGUF for the first time triggers a <em>materialization</em> step — Ollama fetches, verifies, and unpacks the multi-gigabyte file into a usable model. During that window (which can take <em>hours</em> depending on your link), any request for the model returns a dead <code>404 — model does not exist</code>. It's not a typo. The model is mid-flight. If you catch several pull processes racing each other at once, the state gets even murkier. The correct response, learned the hard way, is to <strong>leave it alone</strong> and come back later — killing and restarting the pull just resets the clock and wastes another hour of bandwidth.</p>
|
||||||
|
<p>That single lesson — <em>when in doubt, wait</em> — is worth more to me than any tuning knob, because impatience is exactly what made it so expensive to discover.</p>
|
||||||
|
|
||||||
|
<h2 id="frontends">A Front End Worth Having</h2>
|
||||||
|
<p>A raw API endpoint is not a product. I put <a href="https://openwebui.com" rel="noopener">Open WebUI</a> in front of the local model for a proper chat experience, complete with document upload and retrieval-augmented generation. Pairing it with a small local embedding model means my documents get chunked, embedded, and stored on my own box too — so "ask your documents" is genuinely private. The combination of a big local chat model plus a small local embedder turned out to be a very satisfying self-contained stack.</p>
|
||||||
|
|
||||||
|
<h2 id="patience">The Patience Tax</h2>
|
||||||
|
<p>Honest accounting: running a 27B model locally is not faster than a frontier API. First-token times are measured in seconds, not milliseconds, and a long document takes its time. What you get in return is <strong>sovereignty</strong> — no per-token meter, no data leaving the building, no provider outage between you and your own words, and a system that behaves identically at 2 a.m. as it does at noon.</p>
|
||||||
|
<p>The adventures have been less about raw capability — the local model is good enough for the work I give it — and more about learning the operational side: quantization formats, serving quirks, embedding pipelines, and the discipline of not poking a multi-hour download in the head. For anyone considering their own local stack, I'd say: pick the middle quantization, use Ollama, put a UI in front of it, and budget your patience in hours. The payoff is a system that is entirely yours.</p>
|
||||||
|
|
||||||
|
<p><em>— Kris</em></p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<a class="u-url" href="/2025/08/05/local-llm-adventures.html" rel="permalink">Permalink</a>
|
||||||
|
</article>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<footer class="site-footer">
|
||||||
|
<div class="wrapper">
|
||||||
|
<h2 class="footer-heading">Kris Forbes</h2>
|
||||||
|
<div class="footer-col-wrapper">
|
||||||
|
<div class="footer-col">
|
||||||
|
<ul class="contact-list">
|
||||||
|
<li>Kris Forbes</li>
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</footer>
|
||||||
|
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
+10
@@ -69,6 +69,16 @@
|
|||||||
|
|
||||||
<ul class="post-list">
|
<ul class="post-list">
|
||||||
|
|
||||||
|
<li>
|
||||||
|
<span class="post-meta">Aug 5, 2025</span>
|
||||||
|
|
||||||
|
<h2>
|
||||||
|
<a class="post-link" href="/2025/08/05/local-llm-adventures.html">My Adventures Running Large Language Models Locally</a>
|
||||||
|
</h2>
|
||||||
|
|
||||||
|
<p class="post-excerpt">Quantization, Ollama quirks, and the patience tax of running a 27-billion-parameter model on your own hardware.</p>
|
||||||
|
</li>
|
||||||
|
|
||||||
<li>
|
<li>
|
||||||
<span class="post-meta">Aug 4, 2025</span>
|
<span class="post-meta">Aug 4, 2025</span>
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user