A field guide to running open-weight LLMs locally

How deep can your machine go?

Open-weight language models released in the past year, arranged as floors in an elevator shaft. Small models live near the surface; the giants are far below. Every machine’s elevator only goes so deep. The stops show where a few reference Macs run out of memory, and you can add your own even if it’s not a Mac!

Where does your elevator stop?

  1. What kind of memory?

    The CPU and GPU share one pool: Apple Silicon Macs, AMD Strix Halo, NVIDIA DGX Spark.

  2. How much unified memory?

    Pick a 🍎 Mac

    GB

Enter your memory to add your own stop to the shaft.

Reading the shaft

4-bit (Q4) memory 8-bit (Q8) memory Popularity Opens on Hugging Face Solid box: last stop. Dashed box: recommended stop for multitasking. Why a filing cabinet? Unicode has no Mac Studio emoji. This was the closest match: a squat metal box that lives by your desk and holds a frankly alarming amount of weights. Why gems? It’s a mine shaft, so digging deeper ought to turn up something shinier. They run roughly from cheapest to priciest, quartz to red diamond, with one exhibit at the bottom that no mine has produced. They’re decoration only: a gem says nothing about a model, and real gem prices swing too much with quality for the order to be more than loose. That’s… a Chaos Emerald! Sorry, we’re not playing Sonic right now. It’s a plain emerald on B10, and collecting all seven won’t make your GPU go Super.

Rule of thumb

4-bit takes about 0.6GB per billion parameters; 8-bit about 1.1GB. Those figures cover weights only, so leave room for context and the OS.

Mixture-of-experts

An MoE model needs memory for all its parameters but runs at the speed of the few active per token. That’s why most big models today are MoE.

Data center GPUs

The 🏢 line counts 80GB H100s (A100s are the same size) and ~180GB B200s needed to hold 8-bit (FP8) weights, using 90% of each card. Real deployments round up to 2, 4, or 8 GPUs per server and want spare memory for long contexts and many users.

Comparable Claude model

Names the Claude model you’d reach for to do the same kind of task. It isn’t a claim of equal performance; open models still trail on harder versions of these tasks.

Shared-memory caps

Mac: the GPU gets roughly two-thirds to three-quarters by default. Raise it with sudo sysctl iogpu.wired_limit_mb=….

Strix Halo: set the GPU’s share in the BIOS or AMD’s Variable Graphics Memory (up to 96GB of 128GB on Windows; Linux can go a little higher via the amdgpu GTT limit).

DGX Spark: no fixed split; the GPU can use whatever the OS isn’t.

About popularity

A rough three-step signal based on OpenRouter token usage and Hugging Face downloads. Big models are judged mostly by API traffic; small ones mostly by downloads.