Running LLMs locally on a Mac

1.1 Compute backends

The compute backend is the runtime that runs the matmuls — where on the chip the work actually happens. Three of them cover local text inference on the M-series, and which one a tool picks sets both its speed and how soon it handles a new model.

• PyTorch + MPS (Metal Performance Shaders) is the baseline. Most ML code reaches Apple Silicon through PyTorch, so coverage of new architectures arrives first — it is the lingua franca. Our embedding code runs here; speed is decent, not remarkable.
• llama.cpp carries its own hand-written Metal kernels rather than going through MPS. It is the engine under Ollama and llama-server, fast and wide-coverage, and the one that eats GGUF.
• MLX is Apple’s own array framework — faster still on the work it covers, less mainstream, and lagging by months on new architectures. Osaurus, mlx-lm, and the JANG stack all sit on it.

The image twin adds two backends that scarcely figure for text: CoreML on the Neural Engine for the lowest-footprint path, and Draw Things’ custom Swift + Metal stack — see front-end clients for AI image models.

1.2 Storage backends

The storage backend is the on-disk format the weights ship in — and, past full precision, the quantization scheme baked into it. At the sizes worth running on a 128 GB Mac this is the load-bearing choice: a 70B model at full fp16 is ~140 GB and simply will not fit, so quantization — trading a little quality for a much smaller footprint — is what gets it resident at all. A 4-bit build of that same 70B lands near 40 GB. The calibration data deciding which weights keep their bits is the other, quality half of the story. Every format below except plain safetensors is doing this fitting job, and each pairs with a compute backend that can load it.

• safetensors — the Hugging Face baseline, full precision; what PyTorch + MPS loads when the model fits without help.
• GGUF — the llama.cpp format. Not Apple-specific (it runs on CUDA and CPU too), it has the widest coverage and the finest quant ladder — down to the very-low-bit IQ2/IQ3 imatrix quants for the tightest squeezes.
• MLX builds (mlx-community) — the MLX-native format, Apple-only. Quantized to fit exactly like GGUF — the 4-bit builds are why a 70B loads at all — with added speed on the M-series; the cost is that coverage lags and the quant range is coarser.
• JANG — mixed-precision MLX: per-tensor bit-widths instead of one width for the whole model. MLX’s answer to imatrix, closing GGUF’s quality-per-bit edge at the low end.

So the choice between GGUF and an MLX build is about speed, coverage, and how tight a squeeze we need. Rule of thumb: prefer an MLX build for the speed when one is published; reach for GGUF when no MLX port exists yet, when we want the cross-platform llama.cpp ecosystem, or when only a low-bit imatrix quant will fit the model at all.

The fixed MLX width is indeed sus though — worth verifying that MLX models work OK. I suspect that JANG solves this better, but it is even more fringe.

2.1 Osaurus

Osaurus (MIT, brew install --cask osaurus, osaurus-ai/osaurus) is Swift-native, no Electron, no Python, and behaves like a proper Mac app.

I’m YOLOing all in on this because it seems efficient and easy. It locks us in to the Mac ecosystem so might not be for everyone. Also, it’s run by one person, so the bus factor is 1, which is risky. But — it’s so good!

The window has a model picker, a chat pane, a status indicator; the inference engine underneath is MLX (Apple’s own array framework for the M-series), so it gets fast tokens per second, fully leveraging the hardware.

Osaurus is also a server — it exposes OpenAI-, Anthropic-, and Ollama-compatible HTTP endpoints on localhost:1337, seemingly all three at once, which is some deep magic. Once we have it installed, anything else we want to point at a local model (code editor, embedding pipeline, Claude-Code-style tool) can talk to this local endpoint. It is also a full agent harness — the project now pitches itself as “the AI harness for macOS”, and the README leads with agents, not the chat window. Each agent carries its own prompts and persistent memory; pointing one at a working folder grants file, search, and git tools, and on macOS 26+ a sandbox toggle adds shell access in an isolated Linux VM via Apple’s Containerization framework — though anything running in that VM is Linux-CPU-only, beyond the reach of MLX and MPS. It imports agentskills.io-format skills (and whole Claude plugins) from GitHub or local files, selecting them by RAG at runtime, and speaks MCP in both directions, server and client. The harness layer is model-agnostic, fronting cloud providers as happily as the local MLX runtime.

One CLI gotcha: the osaurus command is embedded in the app bundle, and only the Homebrew install links it onto PATH automatically. If it is missing, symlink it: ln -sf “/Applications/osaurus.app/Contents/Helpers/osaurus” “$(brew --prefix)/bin/osaurus”. The documentation claims it is ln -sf “/Applications/Osaurus.app/Contents/MacOS/osaurus” “$(brew --prefix)/bin/osaurus”; but I think this is a typo — that launches the app, not the CLI helper. The CLI is worth having: osaurus serve / stop / status / list / run <model> / mcp, plus a plugin manager.

Osaurus is also the intended runtime for a custom mixed-precision MLX quantization format called JANG.

2.2 Jan

Jan (brew install --cask jan) is a FOSS cross-platform option. It looks nice. The UI is built on the mildly cursed Electron, but the plus side is that it runs on Linux, Windows, and macOS. It supports both llama.cpp (via Cortex) and MLX backends. For cross-platform use it looks pretty good. For brute-force running non-Apple-optimised models, it seems to go.

Jan is closer to a frontend + harness bundle than a pure chat client. The Projects / Assistants / Agents / MCP Connectors quartet gives it a tool-calling agent loop — MCP servers connect under Settings → MCP Servers (Exa search and a Jan Browser MCP are pre-configured), models call those tools, and the Agents mode runs multi-step autonomous workflows (Jan v2 VL is pitched as a 49-step multimodal agent). Jan Server is the self-hosted multi-step-orchestration variant. So in stack-vocabulary terms Jan covers frontend + harness + server + runtime, much like Osaurus does — just with a more chat-shaped UI and the Electron tax.

2.3 LM Studio

LM Studio (brew install --cask lm-studio) is closed-source, relatively slick and turnkey, and not free for commercial use. It runs both llama.cpp and — since 0.3.4, October 2024 — its own MIT-licensed mlx-engine (mlx-lm + Outlines + mlx-vlm), so MLX-native inference is not unique to Osaurus. I’m mildly sceptical of it because so many Cool LLM Technologies ship special bug fixes or alternate install paths for LM Studio, which hints at a slightly non-standard stack — though that might just be sampling bias, since more people file bug reports when more people run the thing.

3.1 Ollama

Ollama (brew install ollama) is a llama.cpp wrapper with its own model registry — fast enough, wide model coverage, and notably good for embedding models :

brew services start ollama
ollama pull qwen3.5  # LLM/chats etc
ollama pull mxbai-embed-large    # also handles embeddings

Anything OpenAI-API-compatible can now point at http://localhost:11434/v1. It’s smart to make it evict idle models, or it will load up many and tank the machine. OLLAMA_KEEP_ALIVE=15m does that.

Gotchas:

• The .gguf files come from Ollama’s registry, not Hugging Face.
• Some weird reimplementation headaches — e.g. the tokenizer baked into the GGUF can differ from the original for unclear reasons.
• The “llama.cpp wrapper” framing is loosening: the registry now ships -mlx tags for some models (e.g. qwen3.5:35b-mlx).

3.2 Unsloth Studio

Unsloth is a different beast from Ollama — it’s a fine-tuning toolkit first, with a serving endpoint as a side effect. The headline claim is “2× faster training with 70% less VRAM” against vanilla transformers, achieved via custom kernels. Unsloth Studio wraps the toolkit in a browser-based no-code UI: ingest PDFs / CSVs / DOCX into a dataset, run training, export to GGUF or safetensors, evaluate side by side against the base model, and serve the result through a local OpenAI-compatible /v1 endpoint.

uv pip install unsloth
unsloth studio -p 8888    # browser UI on 127.0.0.1:8888

(The docs’ own example launches with -H 0.0.0.0, which binds every interface — leave that off unless we want the studio on the wifi.)

The reason to install Unsloth instead of (or alongside) Ollama is the fine-tuning side: train a LoRA on our own data, deploy it, query it, all from one place. The serving endpoint is the means of using what we just fine-tuned, not a competitor to Ollama as a generic backend.

Two Mac-specific caveats. First, Unsloth is CUDA-first; its signature kernels target NVIDIA GPUs. Apple Silicon support exists per the docs, but some paths fall back to slower implementations, and the VRAM-saving benchmarks are calibrated against a CUDA transformers baseline, not against MLX. I have not measured the Mac numbers myself. Second, if the priority is fine-tuning specifically on Apple Silicon, mlx-lm ships its own LoRA training path (mlx_lm.lora — it moved out of mlx-examples into its own repo) — fewer features, less polish, but native MLX kernels and no fallback paths.

3.3 (even-more-)Power-user options

llama.cpp itself ships a server: llama-server -m model.gguf exposes the same OpenAI-compatible endpoint with no daemon, no registry, no opinions. brew install llama.cpp. I’m not really sure when this would seem like a good idea? Ultrahobbyists? Trying to get a job at Anthropic? If I were going this deep I’d probably run it via transformers in my own Python process.

Apple Silicon optimised: mlx-lm ships its own server too: uv tool install mlx-lm then mlx_lm.server --model mlx-community/<repo>. One model per process; restart to switch.

5.1 Getting models into Osaurus

A mechanical note before the tables, because it confused me: Osaurus has no CLI download command, and its in-app Model Manager (⌘⇧M → Models) browses a curated catalogue — the Gemma-4 family, Qwen3.5/3.6, and the JANG zoo — not the whole of Hugging Face. A model being absent from that menu doesn’t mean it won’t run; it means we sideload it. Osaurus discovers anything dropped into its models directory:

hf download mlx-community/DeepSeek-R1-0528-Qwen3-8B-4bit \
  --local-dir ~/MLXModels/deepseek-r1-0528-qwen3-8b-4bit
osaurus list    # confirms discovery, and gives the exact API name for `osaurus run`

So in the tables below, “MLX via Osaurus” means this sideload unless the row says the model is in the curated menu. One more switch worth knowing: to keep two models resident at once (say the agentic daily-driver plus the maths model), set Settings → Local Inference → Model Management to Flexible — under the default Strict policy, loading one evicts the other.

5.2 For mathematical reasoning

(I’ve dropped the older DeepSeek-R1-Distill-Qwen-32B here — it’s a Qwen2.5-era distill that the 0528-Qwen3-8B now matches on math at a quarter the size.)

Two Nemotrons, don’t confuse them. Cascade-2 above and the Nemotron-3-Nano-Omni-30B-A3B that Osaurus ships by default are both 30B-A3B post-trains off the same Nemotron-3-Nano base — same hybrid Mamba-MoE skeleton, different specialisation. Cascade-2 is the text reasoning/math build (Ollama; JANG quants via the Python stack only); Nano-Omni is the multimodal one (image + audio + video in), which is exactly why JANG packaged it and Osaurus defaults to it. So the Osaurus default is a local multimodal generalist, not the math champion — use it for “look at this screenshot / transcribe this” chat, and switch to Cascade-2 on Ollama (or the -Reasoning Omni variant) when we need it to actually think.

5.3 For agentic flows

These are the weights a coding harness wants behind it — Running a coding agent offline covers the harness/UI choice (OpenCode, pi, Cline, …) that points at them.

5.4 What I would skip (for now)

NousCoder-14B is not what its name suggests in the agentic sense. It’s a Qwen3-14B post-trained via RL on 24k one-shot competitive-programming problems. The model card is explicit: LiveCodeBench v6 Pass@1 67.87%, up 7.08% from baseline. That’s a model for “give me a hard algorithm problem and I will solve it by myself”, not “run a tool-using coding agent over my repo”. Use Hermes-4 for the latter; pick NousCoder if Codeforces-style problems are the actual use case.

5.5 Suggested audition order

1. DeepSeek-R1-0528-Qwen3-8B for math; Qwen3.6-35B-A3B (MXFP4 + MTP, one click in Osaurus) for agentic. Two pulls, two control samples. If 3.6’s young MTP path misbehaves, drop back to Qwen3.5-35B-A3B or Hermes 4.3-Seed-36B.
2. Phi-4-Reasoning-Plus-14B alongside DeepSeek for triangulation on hard math.
3. Hermes-4-70B to test whether the 70B tier earns its higher throughput cost over the 36B at our workloads.
4. Nemotron-Cascade-2 (Ollama — its JANG quants don’t load in Osaurus, see the table), specifically against long tool-call traces — the one Mamba experiment.
5. DwarfStar (V4 Flash) is its own thing, covered below.

For the equivalent question on a serious GPU (single H100 to a full DGX), see the small-GPU section of the sovereign-compute companion.

6.1 Solving problems

The 14B OpenMath-Nemotron is the efficiency cut: most of the 32B’s numbers at half the footprint, with MLX and GGUF both present.

These models run in both CoT and TIR modes; the headline AIME scores are maj@k figures with test-time scaling switched on, so a single local greedy decode at \(T = 0\) will look notably worse than the marketing number.

6.2 Running TIR on a Mac

For the TIR loop itself and the four ways to wire it up (roll-our-own / Qwen-Agent / NeMo-Skills / generic), see Reasoning harnesses. A handful of Mac-specific runner caveats:

• Qwen-Agent. Current main runs the sandbox in Docker, so we need Docker Desktop or Colima. Older releases ran the kernel on the host directly.
• NeMo-Skills. Two Mac wrinkles here. Host the model on mlx_lm.server, not Ollama — the tool path uses the raw /v1/completions text endpoint that Ollama’s shim handles poorly and mlx_lm.server handles properly. Use the documented no-Docker Python sandbox, since the Docker one wants --network=host, which is degraded on Docker Desktop for Mac.
• Open WebUI’s Pyodide interpreter (SymPy and NumPy preloaded, no network) is the one point-and-click option that fires reliably — but only with a capable general model (Qwen3 30B-class and up), not the maths specialists. Open Interpreter is unmaintained as of early 2026.

For sandboxing model-written Python on a Mac: Linux seccomp doesn’t apply at all, so use the Docker --network none pattern documented in Sandboxing model-generated code rather than reaching for the Linux primitives.

6.3 Proving theorems

Whole-proof Lean provers emit Lean 4 and little else useful — hard-specialisation taken to its limit; a Lean prover can’t hold a conversation. For why the compile-feedback loop avoids false positives (and therefore why provers are sampled at Pass@32+ without anyone worrying), see Proving theorems with a compiler in the loop; for the lean-repl / Pantograph / LeanDojo harness tier and Mathlib version pinning, see Proof self-correction loops.

Autoformalization (turning a natural-language statement into Lean) is the awkward step locally — the clearest specialist (Herald) ships safetensors only, no GGUF or MLX, so on the Mac that conversion currently has to come from a hand-written statement or a hosted endpoint.

7.1 Tech stack

JANG (“Jang Adaptive N-bit Grading”) is mixed-precision quantisation for MLX. Standard MLX quantisation compresses every tensor to the same bit width. JANG classifies tensors by sensitivity — attention and MoE router layers (small share of params, large share of model behaviour) get 6–8 bits; expert MLPs get 2–4. The hybrid network is a mildly extended version of the standard MLX safetensors format with a per-tensor bit-width manifest. At the same total size, accuracy improves, notionally. The pitch is “GGUF for MLX”, which … sounds fine? I’m not really competent to judge. Apparently llama.cpp’s K-quants do something similar. The other half of the GGUF quality story is the imatrix calibration data fed in at quantisation time — which is why a bartowski/…-GGUF repo (like the Nemotron one above) is a slightly different, usually better thing than a bare K-quant of the same weights.

jangq.ai claims impressive performance: at time of writing, JANG_2L at 82.5 GB scoring 74% MMLU against MLX 4-bit at 119.8 GB scoring 26.5% on MiniMax-M2.5; 397B-parameter models fitting on 128 GB Macs. The page also notes it has been “filtered to decisive smaller wins only” — close comparisons and unfavourable cases are not shown — so calibrate accordingly. At least one third-party benchmark is pretty impressed.

Osaurus is JANG native. Pull a model from JANGQ-AI and it loads — mostly. The Swift engine’s coverage tracks the JANGTQ path; a plain JANG_* quant of an exotic architecture can fail at weight-load (the Cascade-2 sideload failure above — the Python jang-tools stack handles those, the Swift engine does not yet). Elsewhere, support is partial: MLX Studio and vMLX natively, oMLX via a merged integration (per the JANG README), LM Studio / Ollama / Jan not yet. From Python: uv pip install “jang[mlx]”, then jang_tools.loader.load_jang_model(...).

7.2 MLX Studio

MLX Studio is the same author’s other Mac desktop app — Electron + Python rather than Swift, broader feature surface (image generation via Flux and Z-Image, ~26 built-in agentic tools, in-app GGUF→MLX and MLX→JANG conversion, an Anthropic-compatible API). Install via the signed DMG on the releases page, or engine-only with uv tool install vmlx and vmlx serve mlx-community/<repo> (OpenAI-compatible on localhost:8000). It doesn’t quite land for me, at least not in comparison to Osaurus.

8.1 Running DwarfStar via the pi stack

The default harness for ds4 seems to be: pi, an MIT-licensed agent harness by Mario Zechner (badlogic, of libGDX fame) — itself a strong offline coding agent once a model is behind it. There is an easy install via the pi extension by Armin Ronacher (mitsuhiko, of Flask): mitsuhiko/pi-ds4. It handles process management for ds4-server — per-PID leases, watchdog shutdown, OpenAI-compatible local endpoint on 127.0.0.1:8000:

pi install https://github.com/mitsuhiko/pi-ds4

First-time install clones antirez/ds4, builds it, downloads the GGUF (~87 GB), and registers a ds4/deepseek-v4-flash model with pi. Subsequent runs spawn the server on demand and shut it down when no client process holds a lease. OpenClaw embeds pi, so the same extension can in principle load there.

pi from the terminal opens a TUI (“textual user interface” — I think that’s what it means, i.e. it lives in the terminal).

Audrey Tang maintains audreyt/pi-ds4, a fork that swaps in cyberneurova’s abliterated IQ2XXS quants and turns on uncertainty-mode directional steering by default — an activation-space edit that puts the model into “this is a contested question” mode on CCP-sensitive topics (Taiwan, Crimea, Kashmir, Western Sahara).

8.2 Manual setup for non-pi harnesses

Outside the pi ecosystem, the manual setup is four commands plus a config edit.

# antirez/ds4 for upstream; also audreyt/ds4 looks cool
# optimisations + steering-vector work — pick one
git clone https://github.com/audreyt/ds4
cd ds4
make
tmutil addexclusion -p (realpath ./gguf)
./download_model.sh                     # ~87 GB into ./gguf/
./ds4-server                            # listens on 127.0.0.1:8000

For lifecycle, we could wrap ./ds4-server in a launchd plist with KeepAlive: true; this is probably not what we want on a typical laptop where we do other things than inference, like, you know, use it as a laptop. I think pi is more automatic in that regard.

ds4-server’s context window is set at launch via --ctx <tokens> (max accepted per conversation) and --tokens <n> (default max output); --kv-cache <path> spills the KV store to disk for long-context runs, and --think turns on DeepSeek’s reasoning mode. DeepSeek V4 Flash nominally supports 1M tokens, but ds4 is RAM-bound: the 2-bit IQ2XXS weights are ~81 GB, and a full 1M-token KV/index sits around 26 GB on top. Rough budget on unified memory:

• 64 GB: 50k–150k --ctx with headroom.
• 96 GB: 150k–250k works but is tight; quit Slack.
• 128 GB: 200k–300k is comfortable; >300k starts risking OOM.
• 1M: only with very generous memory and nothing else running.

If a client (Hermes, OpenClaw, OpenCode, anything OpenAI-compatible) advertises a context larger than --ctx, requests will get cut off — match the client’s contextWindow / limit.context to the server’s --ctx. DeepSeek’s sparse attention means raising --ctx doesn’t blow up compute the way dense attention would, but RAM is still the binding constraint. For most interactive coding, 32k–100k plus a retrieval layer beats brute-forcing the whole history into the prompt. See antirez/ds4’s README and the OpenClaw ds4 provider docs for the full flag list and client-side config.

Reasonable defaults:

./ds4-server --ctx 200000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

To bake in audreyt’s directional-steering defaults:

DS4_DIR_STEERING_FFN=-0.75 \
DS4_DIR_STEERING_ATTN=0 \
DS4_REPRODUCIBLE=1 \
./ds4-server

To use with Hermes, add an OpenAI-compatible provider entry to the Hermes config (sketch — confirm the exact schema with hermes config):

# ~/.hermes/config.yaml
custom_providers:
- name: ds4
  base_url: http://127.0.0.1:8000/v1
  model: deepseek-v4-flash
  models:
    deepseek-v4-flash:
      context_length: 200000

From inside Hermes, /model ds4/deepseek-v4-flash — matching the provider name in the YAML above. Done.

Anyway, this gets us a generic token endpoint, so we’re free to plug in whatever on the front end.

The protagonists of this play have a lot of clout — antirez (Redis), mitsuhiko (Flask), badlogic (libGDX), and audreyt (Taiwan’s former Digital Minister, Pugs / Perl 6). Some kind of critical mass seems feasible for a certain type of nerd.