Local models got good. What that means for your laptop
Open-weight models you can run on your own machine have crossed the line from novelty to useful. Here is where they win, where they lose, and what to try first.
A few years ago, running a language model on your own laptop was a party trick. The output was rough, the machine got hot, and you went back to the API. That is no longer the story. Open-weight models now handle a real share of everyday programming work on hardware you already own.
Two things got us here, and neither is a single dramatic breakthrough.
Quantization stopped hurting
A model's weights are normally stored at 16 bits each. Quantization squeezes them down, often to 4 bits, which cuts memory use roughly fourfold. Early on this cost real quality. The techniques have improved to the point where a well-quantized model is close enough to the original that most users cannot tell in ordinary use.
The practical effect is arithmetic. A model with billions of parameters that needed a data-center card now fits in the memory of a laptop with unified memory or a mid-range consumer GPU.
The runtimes got boring, in the good way
Running a model locally used to mean compiling something. Tools in the mould of llama.cpp and Ollama turned it into one command. Boring tooling is what a technology looks like just before a lot of people start using it.
Where local wins
Privacy. The prompt never leaves your machine. For anyone working with client code, medical data, or anything under a confidentiality agreement, this is not a preference, it is the requirement. For a Berlin company handling personal data, it can be the difference between a lawful workflow and an unlawful one.
Cost at volume. The marginal cost of a local token is electricity. If you are running a model over ten thousand files in a loop, that matters. If you are asking it three questions a day, it does not.
Latency and offline. No round trip, no rate limit, no outage. On a train, on a plane, in a badly connected office, the local model is the only model.
Learning. This is the underrated one. Running a model yourself forces you to meet concepts that the API hides: context length as a memory budget, quantization as a quality trade, sampling parameters as knobs with consequences. You understand the technology differently once you have watched it fill your RAM.
Where frontier models still win
Hard reasoning over long context. A local model with a short context window cannot hold your codebase in its head, and no amount of clever prompting fixes that. Multi-step agent work, where a single mistake early compounds through twenty tool calls, still favours the strongest available model. And the frontier labs ship capability improvements faster than the open-weight ecosystem absorbs them.
The realistic setup for most working developers is both. A local model for the high-volume, low-stakes, privacy-sensitive work. A frontier model for the problems that are actually hard.
What to try first
Install a runtime, pull a small quantized model, and give it a job you would otherwise do by hand: summarize a diff, draft a commit message, explain an unfamiliar function. Notice where it holds up and where it falls over. That boundary moves every few months, and knowing where it currently sits is a genuinely useful thing to carry around.
Then go one level deeper and ask why. Why does context length cost quadratic attention? What actually happens when you quantize a weight? We cover that ground in the AI and Machine Learning course, where you train and deploy models rather than only calling them.