Running LLMs locally on a Mac

1.1 Runtimes on Apple Silicon

The compute backend is the runtime that runs the matmuls — where on the chip the work actually happens. Three of them cover local text inference on the M-series, and which one a tool picks determines both its speed and how quickly it supports new models.

• PyTorch + MPS (Metal Performance Shaders) is the baseline. Most ML code reaches Apple Silicon through PyTorch, so coverage of new architectures arrives first — it is the lingua franca. My embedding code runs here; speed is acceptable, if not amazing.
• llama.cpp brings its own hand-written Metal kernels rather than going through MPS. It is the engine under Ollama and llama-server, fast and wide-coverage, and the one that consumes GGUF.
• MLX is Apple’s own array framework — faster still on the models it supports, less mainstream, and lagging by months on new architectures. Osaurus, mlx-lm, and the JANG stack all use it.

Not treated here because it is only relevant to image models: CoreML on the Neural Engine for the lowest-footprint path, and Draw Things’ custom Swift + Metal stack.

1.2 Weight formats and quantization

The storage format is what the weights ship in, subject to whatever quantization (if any) has been applied etc. RAM is usually the main constraint on consumer hardware, which is why quantization is helpful. A 70B model at full fp16 is ~140 GB and will not fit on even a 128 GB Mac. A 4-bit build of that same 70B lands near 40 GB, which is tolerable. There are several formats in play.

• safetensors — the Hugging Face baseline, full precision; what PyTorch + MPS loads when the model fits without help.
• GGUF — the llama.cpp format. It is not Apple-specific (it also runs on CUDA and CPU), and it has the widest coverage and the finest quant ladder — down to the very-low-bit IQ2/IQ3 imatrix quants.
• MLX (mlx-community) — the MLX-native format, Apple-only. Also quantized to fit but with added speed on the M-series; coverage tends to lag.
• JANG — mixed-precision extension to MLX: per-tensor bit-widths instead of one width for the whole model.

Generally we prefer an MLX build for the speed when one is published; GGUF when no MLX port exists yet, and classic safetensors when neither is available.

1.3 Mixed-precision MLX

I used to write that the fixed bit width in MLX was sus, and guess that JANG solved it better at the cost of being fringe. Half right: mixed precision did win, but JANG’s specific format seems to have lost. MLX’s own quantization config allows per-module overrides — a quantization block in config.json carrying a {bits, group_size} entry per tensor alongside the global default — and as of mid-2026 two toolchains publish calibrated mixed-precision builds into mlx-community using nothing but that.

Both document their calibration-data hygiene, which I cannot say for JANG. Neither declares a custom format, so anything that loads mlx-lm loads them: mlx-lm itself, mlx-vlm, vllm-mlx, LM Studio, oMLX. JANG’s pitch was that mixed precision needs a custom format, a converter, a model zoo and a runtime that understands all three; here are two counter-examples that needed none of it.

3.1 Osaurus

Osaurus (MIT, brew install --cask osaurus, osaurus-ai/osaurus) is Swift-native, no Electron, no Python, and behaves like a proper Mac app.

It seems efficient and easy. It also locks me into the Mac ecosystem, so it might not be for everyone. Also, it’s run by one person, so the bus factor is 1, which is a very small number. But — it’s so good!

The window has a model picker, a chat pane, and a status indicator; the inference engine underneath is Apple’s fast MLX, so it gets many tokens per second. It is the intended runtime for a custom mixed-precision MLX quantization format called JANG.

Osaurus is not just a chat client but a full native macOS agent harness. It supports various hip features like persistent memory and sandboxed working folders in an isolated Linux VM via Apple’s Containerization framework. It understands agentskills.io-format skills (and whole Claude plugins) from GitHub or local files, selecting them by RAG at runtime, and speaks MCP in both directions, as server and client. The harness layer is model-agnostic, fronting cloud providers as happily as the local MLX-ish runtime.

There is no CLI download command; the in-app Model Manager (⌘⇧M → Models) browses a curated catalogue of models, especially JANG ones, and will sideload others too — though not all of them work equally well. Nemotron 3 Nano Omni 30B A3B JANGTQ4 seems like a reliable workaday default. Osaurus also discovers anything dropped into its models directory:

hf download mlx-community/DeepSeek-R1-0528-Qwen3-8B-4bit \
  --local-dir ~/MLXModels/mlx-community/DeepSeek-R1-0528-Qwen3-8B-4bit   # mirror the org/repo path
osaurus list    # confirms discovery, and gives the exact API name for `osaurus run`

Osaurus is also a first-class server, covered below.

3.2 Jan

Jan (brew install --cask jan) is a FOSS cross-platform option — a full frontend + harness + server + runtime bundle. The UI is built on the mildly cursed Electron, but the plus side is that it runs on Linux, Windows, and macOS. It supports both llama.cpp (via Cortex) and MLX backends, so it seems well-suited to brute-force running non-Apple-optimized models.

It looks nice, and the Projects / Assistants / Agents / MCP Connectors quartet gives it a tool-calling agent loop — connect MCP servers under Settings, and Agents mode runs multi-step autonomous workflows (Jan v2 VL is pitched as a 49-step multimodal agent). Jan Server is the self-hosted orchestration variant.

3.3 LM Studio

LM Studio (brew install --cask lm-studio) is closed-source, relatively slick and turnkey, and not free for commercial use. It runs both llama.cpp and lately its own MIT-licensed mlx-engine (mlx-lm + Outlines + mlx-vlm). Like the others it is a bundle — frontend + runtime + server — with an OpenAI endpoint it can expose headlessly (lms server). I’m mildly sceptical of it because so many Cool LLM Technologies ship special bug fixes or alternate install paths for LM Studio, which hints at a slightly non-standard stack — though that might just be sampling bias, since more people file bug reports when more people run the thing. Also the licence sucks.

These three are general-purpose chat apps. To drive a local model as a coding agent instead — terminal harnesses like OpenCode or Aider, VS Code sidebars like Cline — see Code agents and assistants, which points back here for the local backend they run against.

4.1 Osaurus as a server

If we already have Osaurus running, we are mostly done: it is already that daemon, exposing OpenAI-, Anthropic-, and Ollama-compatible endpoints on localhost:1337 all at once. Anything we want to point at a local model can talk to it. But also, to avoid installing an idiosyncratic stack, or for superior customization, we might want to install the standard Linux server stack.

To keep two models resident at once (say the agentic daily-driver plus the maths model), set Settings → Local Inference → Model Management to Flexible — under the default Strict policy, loading one evicts the other.

Context length is automatic — Osaurus picks a sane per-model default and does not expose it as a plain setting, so the actual ceiling is hard to read off (the cheat-sheet has the detail).

To drive all this from the terminal, there is one gotcha: the osaurus command is embedded in the app bundle, and only the Homebrew install links it onto PATH automatically. If it is missing, symlink it: ln -sf "/Applications/osaurus.app/Contents/Helpers/osaurus" "$(brew --prefix)/bin/osaurus" or use the special button from the settings menu¹ The CLI supports stuff like osaurus serve / stop / status / list / run <model> / mcp, plus a plugin manager.

4.2 Ollama

Ollama (brew install ollama) is a llama.cpp wrapper with its own model registry — fast enough, wide model coverage, and notably good for embedding models:

brew services start ollama
ollama pull qwen3.5  # LLM/chats etc
ollama pull mxbai-embed-large    # also handles embeddings

Anything OpenAI-API-compatible can now point at http://localhost:11434/v1. Ollama is the hands-off one on lifecycle: it loads a model on first request, keeps several resident at once (up to OLLAMA_MAX_LOADED_MODELS, default 3), and unloads each after OLLAMA_KEEP_ALIVE of idleness (default 5m). Left unbounded, the pool tanks the machine — OLLAMA_MAX_LOADED_MODELS=1 forces evict-on-switch, OLLAMA_KEEP_ALIVE=0 drops a model the moment it idles (or 15m to keep it warm longer). Context window is num_ctx: set it per request (options.num_ctx), bake it into a Modelfile (PARAMETER num_ctx), or lean on the OLLAMA_CONTEXT_LENGTH default — which on recent Ollama auto-scales to VRAM (4k / 32k / 256k) rather than the old fixed 2048.

Gotchas:

• The .gguf files come from Ollama’s registry, not Hugging Face.
• Some weird reimplementation headaches — e.g. the tokenizer baked into the GGUF can differ from the original for unclear reasons.
• The “llama.cpp wrapper” framing is loosening: the registry now ships -mlx tags for some models (e.g. qwen3.5:35b-mlx).

4.3 mlx-lm and mlx-vlm

mlx-lm is Apple’s reference language-model runtime on MLX. mlx-vlm is its sibling package: same MLX backend, same mlx-community/<repo> weights and HF cache, but for VLMs (“vision-language models”) and omni models — image/video/audio in, text out — instead of pure text LLMs. Where this page says “VLM” it means a model in that family; mlx-vlm is what runs one locally, e.g. DeepSeek-OCR. Osaurus and JANG are inspired by MLX-type Apple-Silicon-friendly execution, but mlx-lm is the OG.

uv tool install mlx-lm drops a family of commands onto PATH, all reading the same weights:

• mlx_lm.generate --model mlx-community/<repo> — one-shot completion from the CLI.
• mlx_lm.chat — an interactive REPL in the terminal.
• mlx_lm.server --model mlx-community/<repo> — an OpenAI-compatible daemon; holds one model, swapping on demand per request (evict + reload, not a restart). Two live at once means two processes on two ports.
• mlx_lm.lora — its LoRA fine-tuning path.

mlx-vlm mirrors this shape (mlx_vlm.generate, mlx_vlm.chat, mlx_vlm.server) but takes an image/video/audio argument alongside the text prompt.

It uses the standard Hugging Face links: mlx-community/<repo> resolves straight to Hugging Face, and the weights land in the shared HF cache (~/.cache/huggingface/hub).

mlx_lm.server has no --ctx flag, so it grows the KV cache to fit whatever we send — up to, presumably, the model’s declared max, capped only by RAM, at which point it presumably kernel-panics the machine. Cap the context in the harness (limit.context / contextWindow) and mind the memory budget.

A reason to keep this around even with Osaurus installed is that it’s great when it runs, but its Swift engine’s coverage lags the Python MLX options. A plain mlx-lm loads interesting MLX conversions that Osaurus can’t (e.g. Cascade-2).

4.4 vllm-mlx

vllm-mlx (uv tool install vllm-mlx, Apache 2.0) is a vLLM-style inference server for Apple Silicon. Active and popular by this page’s standards — 1,300+ stars. Core pitch: continuous batching, paged KV cache with prefix sharing, an SSD-tiered cache for spilling prefixes to disk, and both OpenAI (/v1/*) and Anthropic (/v1/messages) endpoints from one process. It also has many fancy bonus features:

• native TTS (Kokoro, Chatterbox, VibeVoice, VoxCPM) and STT alongside text/image/video/audio chat
• multi-token prediction (--enable-mtp)
• an embedding and reranker endpoint in the same process (--embedding-model, --rerank-model)
• MCP tool integration (--mcp-config)
• Prometheus metrics (--enable-metrics) and a built-in benchmarker (vllm-mlx bench-serve)

MTP is not classic draft-model speculation here — it drives the model’s own built-in MTP head via cache snapshot and restore, so the head has to live inside the model directory, and the verifier is bypassed for any non-greedy or multi-request batch, which is to say for exactly the fan-out workload I run. It supports multi-model residency via a --models-config models.yaml registry: named models behind one process, lazy load on first use, LRU eviction under a memory_budget_gb, and a contention_policy (fail / wait / preempt / wait_then_fail / wait_then_preempt) for what happens when a request needs a model that does not currently fit; clients pick one via the normal OpenAI model field.

uv tool install vllm-mlx
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching

For a chat window over this endpoint — one that renders the equations the maths models emit — Open WebUI points at it unmodified: add http://localhost:8000/v1 as an OpenAI connection and skip the bundled-Ollama path the tutorials assume.

vllm-mlx serve ~/MLXModels/mlx-community/Qwen3.6-35B-A3B-4bit --port 8000 --continuous-batching

manager:
  # WEIGHTS only — the KV cache is a separate pool, sized on the serve command.
  # 68 + 30 (--cache-memory-mb) + ~14 (prefill/activations) ≈ the 112 GB envelope below.
  memory_budget_gb: 68
  contention_policy:
    strategy: wait_then_preempt
    wait_timeout_s: 45
    preempt_after_s: 15

models:
  - name: driver
    path: /Users/dan/MLXModels/mlx-community/Qwen3.6-35B-A3B-4bit
    continuous_batching: true
    estimated_memory_gb: 22

  - name: solver
    path: /Users/dan/MLXModels/mlx-community/VibeThinker-3B-8bit
    preload: true
    estimated_memory_gb: 3

vllm-mlx serve --models-config ~/.config/vllm-mlx/models.yaml \
  --port 8000 \
  --gpu-memory-utilization 0.88 \      # hard process ceiling ≈112 GB (0.88 × 128)
  --continuous-batching \
  --use-paged-cache \
  --cache-memory-mb 30720 \
  --max-cache-blocks 16384 \
  --max-num-seqs 16 \
  --max-tokens 131072 \
  --max-request-tokens 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser auto \
  --reasoning-parser qwen3 \
  --enable-metrics \
  --kv-cache-quantization \
  --kv-cache-quantization-bits 8 \
  --ssd-cache-dir ~/.cache/vllm-mlx/kv \
  --ssd-cache-max-gb 40 \
  --timeout 1200

uv tool install --force vllm-mlx --with 'mlx-vlm==0.6.3'

4.5 oMLX

oMLX (jundot/omlx, Apache 2.0, brew tap jundot/omlx https://github.com/jundot/omlx && brew install omlx) is a fork of vllm-mlx with a different frontend grown on top, so it inherits that feature set — continuous batching, multi-model residency, both OpenAI and Anthropic endpoints. Three things set it apart:

• Its SSD prefix cache is automatic and block-addressed — hot blocks in RAM, cold blocks spilled to disk, longest-prefix matched and surviving a restart — where llama-server’s --slot-save-path and ds4’s --kv-disk-dir are manual slot-save knobs. Aimed at agentic coding, where the pitch is TTFT dropping from 30–90s to under 5s on the second turn of a long context.
• An explicit Claude Code accommodation: it rescales reported token counts so auto-compact fires at the right time, and holds the connection open with SSE keep-alives through a long prefill. The frontend is a signed SwiftUI menu-bar app (not Electron) with a web admin panel, and it reuses an existing LM Studio model directory.
• It is the one non-Osaurus server with merged JANG support, so it can load the mixed-precision JANG quants that make sub-4-bit MoE models behave — otherwise Osaurus-only.

omlx serve --model-dir ~/MLXModels \
  --paged-ssd-cache-dir ~/.omlx/cache \
  --hot-cache-max-size 16GB \
  --max-concurrent-requests 8

Same caveats as the rest of the page (bus factor 1, MLX-only, benchmarks from an M3 Ultra 512GB), but the clean lineage and the oQ converter make it worth a run as the headless daily-driver if the Osaurus/mlx_lm.server pair leaves us wanting persistent prefix reuse. The SSD cache is less of a differentiator than I first thought: vllm-mlx has the same cold tier behind --ssd-cache-dir, just switched off by default rather than on.

hf download gabfssilva/VibeThinker-3B-MLX-BF16  --local-dir ~/MLXModels/gabfssilva/VibeThinker-3B-MLX-BF16  # solver, hi-fi bf16
hf download mlx-community/VibeThinker-3B-8bit   --local-dir ~/MLXModels/mlx-community/VibeThinker-3B-8bit   # solver, near-lossless and faster
hf download mlx-community/Qwen3.6-35B-A3B-4bit  --local-dir ~/MLXModels/mlx-community/Qwen3.6-35B-A3B-4bit  # the driver

omlx serve --model-dir ~/MLXModels \
  --memory-guard-gb 110 \          # 128 GB Mac: driver (~22 GB) + solver (~3 GB) + KV headroom
  --max-concurrent-requests 16     # the fan-out width; default is 8

4.6 llama.cpp / llama-server

llama.cpp ships its own server (brew install llama.cpp): llama-server -m model.gguf exposes an OpenAI-compatible endpoint with no daemon, no registry, no opinions.

Ollama wraps this same engine. However, llama-server is more configurable, exposing llama.cpp flags that Ollama does not, notably: YaRN context extension on a GGUF (which Ollama cannot do at all), the finer KV-cache quant ladder, speculative decoding with a draft model, and per-slot KV save/restore to disk.

It loads a .gguf from disk — no ollama pull into a separate store, no background service — which suits scripted or reproducible runs.

It loads the one model named at launch, though a newer router mode (start it with no -m) does dynamic multi-model load and unload. Context is the -c / --ctx-size flag, defaulting to 0 (the model’s full trained window). Other useful config options can be found in the cheat-sheet.

4.7 Sampling defaults

Sampling — temperature, top-p, top-k, and the output-token budget — is a decode-time choice set on each request. In particular, a server typically not use the model’s recommended sampling settings automatically.

mlx_lm.server, for example, defaults to temperature=0.0 (greedy), top_p=1.0, top_k=0, and max_tokens=512, and it does not read the model’s generation_config.json. So a reasoning model’s advised settings — VibeThinker advises temperature 1.0 / top-p 0.95 and a 64K-plus output budget — need the client to specify them; each model table below carries a Recommended sampling column with the per-model picks. Notably temperature=0.0 breaks maj@k voting, since every sample comes back identical.

4.8 Which Sampling parameters work

vllm-mlx 0.4.0 accepts parameters that it ignores. The sampler is built as make_sampler(temp, top_p, min_p), and logits processors are constructed only from repetition_penalty. So:

The two dead ones are logged on the way in and dropped on the way through, so any top-k 20 in a model card’s recipe — and most of the recipes in the tables below say exactly that — is doing nothing here. Two more traps in the same family:

• mlx-lm’s repetition_context_size defaults to 20 tokens and vllm-mlx never overrides it, so the penalty sees a 20-token window. It will break a short stutter but will not touch a repeated paragraph.
• vllm-mlx reads a model’s generation_config.json for stop tokens only, so a card’s recommended temperature never reaches the server on its own.

When the registry has no per-model sampling — vllm-mlx’s does not — a server-wide default is the only lever, and that makes it a compromise rather than a setting. Mine splits: 0.6 for the agentic models, 1.0 for Cascade-2 and VibeThinker, so any single temperature is wrong for half of them, and the 0.7 fallback is a defensible midpoint. min_p and repetition_penalty are the two that are safe to set globally, because they truncate the degenerate tail without pinning a preferred temperature:

--default-min-p 0.02 --default-repetition-penalty 1.05

Where to pin the values depends on which layer issues the prompt.

• mlx_lm.generate — flags per call: --temp 1.0 --top-p 0.95 --max-tokens 40000 (--top-k already defaults to 0). Wrap it in an alias.
• mlx_lm.server — launch-time defaults via --temp / --top-p / --top-k / --min-p, overridden per request; Osaurus keeps the same defaults in its app settings, also overridden per request.
• transformers — a GenerationConfig passed at generate() time.
• Ollama — an exception, baking in a per-model default through a Modelfile: PARAMETER temperature 1.0, PARAMETER top_p 0.95, PARAMETER num_predict 40000 for the output budget, PARAMETER num_ctx 65536 for the context window.

Output budget and context window are separate limits: the first caps how much the model may emit, the second how much prompt-plus-output the KV cache holds. A long-reasoning model can need both raised, or it truncates mid-derivation — and Ollama in particular drops the overflow silently once num_ctx is exceeded.

4.9 What this costs us against vllm-mlx

The server-side story above assumes we can send a value with a prompt, and which we can is varies with the client — Goose sends temperature and nothing else, OpenCode sends arbitrary keys past one badly-named flag, Open WebUI pins a full set per model preset, Jan does too but only on a custom provider. That table lives with the frontends, since it is the same table whatever the server.

What makes it bite here is that vllm-mlx’s registry has no per-model sampling of its own, so there is no server-side fallback: whatever the client cannot send is a setting we simply do not have. That is why Open WebUI’s presets matter disproportionately on this stack — a preset per model is the only place the per-model temperatures in the tables below can actually be written down.

Gotcha: Open WebUI’s Advanced Params panel names the penalty repeat_penalty — Ollama’s spelling — while vllm-mlx reads repetition_penalty, so it is dropped in transit and the model keeps looping. The panel suffixes eight Ollama-only fields with (Ollama) — num_ctx, keep_alive, think and friends — but repeat_penalty is not among them, so it looks portable and is not. The fix is the Add Custom Parameter button at the bottom of the panel: add repetition_penalty by hand and skip the built-in row.

4.10 Prefix caching

The KV cache is the only state these servers keep between requests — there is no session object, we resend the whole history each call. What they reuse is the prefix: the shared start of the conversation (system prompt, tool definitions, history so far) keeps its KV, so an agentic loop prefills only the new suffix. Matching is content-addressed — on the tokens, not a session ID — so it happens by itself. This matching is likely not optimal in general.

When planning around this we need to be aware that trimming the front of the history will break this caching and require everything to be re-computed.

Different servers clear this cache at different times:

• mlx_lm.server — an in-memory LRU, longest-prefix matched, reporting cached_tokens in the usage block; bounded by --prompt-cache-size / --prompt-cache-bytes, held for the process.
• llama-server — --cache-prompt is on by default (one KV per slot; --cache-reuse even salvages chunks after a mid-prompt edit), and slots can be saved to disk.
• Ollama — the same llama.cpp reuse, alive as long as the model stays loaded (OLLAMA_KEEP_ALIVE).
• Osaurus — automatic; headless under osaurus serve the cache lives for the server process (governed by the Strict/Flexible policy), and in the GUI it is per chat window, warmed the moment one opens.

Prefill is chunked and continuously batched besides, so concurrent requests interleave rather than queue — but the prefix skip is the bigger win.

4.11 Stretching the context window

Deep lore for the optimizers.

A model’s positional encoding is trained out to some fixed length, and that trained length is baked into its knowledge of the context window. Some of the huge context numbers advertised are that same window stretched at load time, rather than a property of the weights. Qwen3.6, for example, trains its rotary positions to 256K (262,144 tokens); the 1M figure quoted for it is that same window extended ~4×, and getting the extension costs us something in both quality and RAM.

The mechanism is RoPE interpolation. RoPE encodes each token’s position as a rotation; interpolation rescales those rotations so a position past the trained window maps back into the range the model saw during training, instead of falling off the end into rotations it has never seen. The common variant is YaRN (“yet another RoPE extension”), which rescales per frequency rather than uniformly. A factor of 4.0 takes Qwen3.6’s 256K to ~1M, for example.

Every mainstream implementation AFAICT applies the rescaling statically, fixing it at load and applying it to every prompt regardless of length. A model loaded with factor 4.0 rescales a 2K-token prompt exactly as hard as a 900K one, and short prompts lose some accuracy for a long window they aren’t using. So we switch YaRN on only when we want the long window, and set factor to the longest context we actually expect rather than the largest the model will accept.

How to turn it on depends on the runtime.

transformers and mlx-lm read a rope_scaling block straight from the config.json that ships in the model’s own directory:

"rope_scaling": {
  "rope_type": "yarn",
  "factor": 4.0,
  "original_max_position_embeddings": 262144
}

• llama.cpp / llama-server take flags: --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 262144, alongside the usual -c.
• Ollama exposes no YaRN settings. We inherit whatever the person who converted the GGUF baked into its metadata.
• Osaurus’s Swift engine has the YaRN code, but its Qwen3 and Llama wrappers do not route through it, so YaRN is unavailable for now.

YaRN increases the maximum context window, but does not shrink the KV cache cost of storing that context, so we are still RAM-constrained.

4.12 Configuring each server

All the settings in one place.

Flash attention doesn’t get its own column, because it isn’t really a per-server decision anymore: llama.cpp defaults -fa to auto and turns it on wherever Metal supports it, Ollama switches it on per architecture for the families here (Qwen3.x, Nemotron, Gemma, gpt-oss), and the MLX servers plus ds4 always run a fused attention kernel. The one place it still needs a hand is Ollama’s KV-cache quant, which only takes effect with OLLAMA_FLASH_ATTENTION=1 set alongside it.

Sensible headless starting points:

# 64K context (below): a useful bound well under the trained max — raise or lower for your RAM
# llama.cpp — flash attention is automatic; quantize the cache, choose a context
llama-server -m model.gguf -c 65536 --cache-type-k q8_0 --cache-type-v q8_0

# Ollama — env vars; cache quant needs flash attention enabled
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_CONTEXT_LENGTH=65536 ollama serve

# mlx-lm — no cache-quant/context flag; set the model’s sampling values, cap context in the harness
mlx_lm.server --model mlx-community/<repo> --temp 0.6 --top-p 0.95 --top-k 20

# oMLX — point it at the shared MLX tree; turn on the SSD cold tier and a memory ceiling
omlx serve --model-dir ~/MLXModels --paged-ssd-cache-dir ~/.omlx/cache --memory-guard-gb 96

The --paged-ssd-cache-dir on that last line persists across restarts and gets rewritten every session, so it is the one path here that we want to keep out of backups and Spotlight.

Osaurus is tuned in its Settings pane rather than via a command line, and ds4’s full flag set is in its own section.

6.1 For mathematical reasoning

The Role and Target columns are the reasoning notes’ taxonomy — the role is which slot in a harness the model fills, the target is what it emits, and that determines which checker we owe it. A generalist does maths alongside chat and tool-use, takes whichever target we ask it for, and can be driven by an ordinary agent. A solver emits a boxed final answer, trading chat fluency for reasoning, and wants a specialized loop — tool-integrated reasoning where the model runs code, maj@k voting where it does not. An informal prover emits a proof in natural language, which only a judge can grade. A prover emits Lean, which a compiler accepts or rejects.

The solver rows above are pure CoT, innocent of tool use. They can fit into a larger workflow still, as a solve() oracle a general agent dispatches to. In this case we would serve it on a batching endpoint that keeps it resident beside the agentic driver, and let the driver call it when it needs some help on a sub-problem.

So the Target column is a procurement decision, not a label. maj@k voting pays off on a solver and buys nothing on either proof target, because a set of arguments has no mode to take — download Goedel and we are also committing to a Lean toolchain; download nomos-1 or QED-Nano and we are committing to a judge instead. Which is why nomos-1 ships its own harness and why the two are not interchangeable on this page, however similar their file sizes look.

VibeThinker is a case where the harness matters a lot. Its Claim-Level Reliability Assessment lifts AIME26 from 94.3 to 97.1 off the same 3 GB checkpoint (Xu et al. 2026).

6.2 For agentic flows

These are the weights a coding harness needs. Sizes are the on-disk figures for the MLX build named in the Run via column, since that is the number that has to clear the weights budget.

The largest open weights of mid-2026 do not fit: GLM-5.2 is 753B and 235 GB even at mxfp4, MiniMax-M3 is 427B, Kimi-K2.7 is in the same territory. Hy3 at 299B has a 99 GB oQ2 build that technically loads on a 128 GB machine, in the sense that nothing else then does.

6.3 Diffusion models

Everything above is autoregressive. Diffusion LLMs are a thing though. Do any run locally? I know of one: DiffusionGemma — Google’s experimental Gemma 4 variant, which denoises a whole 256-token “canvas” in parallel instead of emitting tokens left to right. OsaurusAI/diffusiongemma-26B-A4B-it-MXFP8 runs natively in Osaurus through its vmlx-swift block-diffusion engine, ~26 GB on disk and ~24 GB resident.

Manage expectations on speed. Docs report 28–42 tok/s at 48 denoising steps: Osaurus defaults to 16, roughly twice as fast as the bundle default and still coherent, and the output falls apart below 12. Quality trails plain Gemma 4 as well. Vision, tool-calling, and a reasoning channel all work in this checkpoint; audio and video do not.

Beyond speed, there is a family-level reason I have not chased Gemma further, and it is why no Gemma appears in the tables above. In my brief tests Gemma models come across as brittle and anxious — quick to hedge, and prone to a sort of performed distress when pushed. Soligo et al. report the same at scale, and locate it in post-training rather than the base weights: instruct-tuned Gemma expresses substantially more distress than its own base model does, whereas instruct-tuned Qwen and OLMo express less (Soligo, Mikulik, and Saunders 2026). They also take high-frustration responses from 35% to 0.3% with DPO on 280 preference pairs at no capability cost, which reads as a shallow trait rather than a baked-in one — just not one fixed in any checkpoint I can download.

So it is not the agentic daily-driver. I am interested in the interaction model though: bi-directional attention over the canvas makes it potentially useful for infilling and structure-preserving rewrite, which is a different way to drive a coding tool than streaming tokens into a chat box.

7.1 Tech stack

JANG (“Jang Adaptive N-bit Grading”) is mixed-precision quantization for MLX. Standard MLX quantization compresses every tensor to the same bit width. JANG classifies tensors by sensitivity — attention and MoE router layers (small share of params, large share of model behaviour) get 6–8 bits; expert MLPs get 2–4. The hybrid network is a mildly extended version of the standard MLX safetensors format with a per-tensor bit-width manifest. At the same total size, accuracy improves, notionally. The pitch is “GGUF for MLX”, which … sounds good? I’m not really competent to judge. Apparently llama.cpp’s K-quants do something similar.

jangq.ai claims impressive performance, regularly beating models with a larger footprint. At least one third-party benchmarker is impressed.

The other part of the GGUF quality story is the calibration data fed in at quantization time — which is why a bartowski/…-GGUF repo (like the Nemotron one) is a slightly different, usually better thing than a bare K-quant of the same weights. I am unsure if JANG does this. OptiQ and oQ both do, and document it, which is why I currently pull an OptiQ or oQ build over a JANG one: they get the same per-tensor bit allocation out of stock MLX config.

Osaurus is JANG native. Pull a model from JANGQ-AI and it loads — usually. The Swift engine’s coverage tracks the JANGTQ path; a plain JANG_* quant of an exotic architecture can fail at weight-load (notably Cascade-2 doesn’t work — the Python jang-tools stack handles those, the Swift engine does not yet). Elsewhere, support is partial: MLX Studio, vMLX, and oMLX all load JANG natively; LM Studio / Ollama / Jan not yet. From Python: uv pip install "jang[mlx]", then jang_tools.loader.load_jang_model(...).

7.2 MLX Studio

MLX Studio is the JANG/Osaurus author’s other Mac desktop app — Electron + Python rather than Swift, broader feature surface (image generation via Flux and Z-Image, ~26 built-in agentic tools, in-app GGUF→MLX and MLX→JANG conversion, an Anthropic-compatible API). Install via the signed DMG on the releases page, or engine-only with uv tool install vmlx and vmlx serve mlx-community/<repo> (OpenAI-compatible on localhost:8000).

8.1 Running DwarfStar via the pi stack

The default harness for ds4 seems to be: pi, an MIT-licensed agent harness by Mario Zechner (badlogic, of libGDX fame) — itself a strong offline coding agent once a model is behind it. There is an easy install via the pi extension by Armin Ronacher (mitsuhiko, of Flask): mitsuhiko/pi-ds4. It handles process management for ds4-server — per-PID leases, watchdog shutdown, OpenAI-compatible local endpoint on 127.0.0.1:8000:

pi install https://github.com/mitsuhiko/pi-ds4

First-time install clones antirez/ds4, builds it, downloads the GGUF (~87 GB), and registers a ds4/deepseek-v4-flash model with pi. Subsequent runs spawn the server on demand and shut it down when no client process holds a lease. OpenClaw embeds pi, so the same extension can in principle load there.

Running pi from the terminal opens a TUI (“textual user interface” — I think that’s what it means, i.e., it lives in the terminal).

Audrey Tang maintains audreyt/pi-ds4, a fork that swaps in cyberneurova’s abliterated IQ2XXS quants and turns on uncertainty-mode directional steering by default — an activation-space edit that puts the model into “this is a contested question” mode on CCP-sensitive topics (Taiwan, Crimea, Kashmir, Western Sahara).

8.2 Manual setup for non-pi harnesses

Outside the pi ecosystem, the manual setup is four commands plus a config edit.

# antirez/ds4 for upstream; also audreyt/ds4 looks cool
# optimisations + steering-vector work — pick one
git clone https://github.com/audreyt/ds4
cd ds4
make
tmutil addexclusion -p (realpath ./gguf)
./download_model.sh                     # ~87 GB into ./gguf/
./ds4-server                            # listens on 127.0.0.1:8000

For lifecycle, we could wrap ./ds4-server in a launchd plist with KeepAlive: true; this probably isn’t what we want on a typical laptop, where we do other things besides inference — like, you know, use it as a laptop. I think pi is more automatic in that regard.

ds4-server’s context window is set at launch via --ctx <tokens> (max accepted per conversation); output length is a per-request API field, not a launch flag. --kv-disk-dir <path> (with --kv-disk-space-mb <n>) persists the KV cache to disk, so a prefix survives restarts and session switches rather than being reprocessed — durable prefix storage, not a long-context spill. Thinking mode is on by default, toggled per request, running DeepSeek’s reasoning mode. DeepSeek V4 Flash nominally supports 1M tokens, but ds4 is RAM-bound: the 2-bit IQ2XXS weights are ~81 GB, and a full 1M-token KV/index sits around 26 GB on top. Rough budget on unified memory:

• 64 GB: 50k–150k --ctx with headroom.
• 96 GB: 150k–250k works but is tight; quit Slack.
• 128 GB: 200k–300k is comfortable; >300k starts risking OOM.
• 1M: only with very generous memory and nothing else running.

If a client (Hermes, OpenClaw, OpenCode, anything OpenAI-compatible) advertises a context larger than --ctx, requests will get cut off — match the client’s contextWindow / limit.context to the server’s --ctx. DeepSeek’s sparse attention means raising --ctx doesn’t blow up compute the way dense attention would, but RAM is still a constraint. For most interactive coding, 32k–100k plus a retrieval layer beats brute-forcing the whole history into the prompt. See antirez/ds4’s README and the OpenClaw ds4 provider docs for the full flag list and client-side config.

Reasonable defaults:

./ds4-server --ctx 200000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

I need clarification on which text to copy-edit — you’ve provided guidelines but no actual blog post content to edit.

Could you please share the text you’d like me to copy-edit?

DS4_DIR_STEERING_FFN=-0.75 \
DS4_DIR_STEERING_ATTN=0 \
DS4_REPRODUCIBLE=1 \
./ds4-server

To use with Hermes, add an OpenAI-compatible provider entry to the Hermes config (sketch — confirm the exact schema with hermes config):

# ~/.hermes/config.yaml
custom_providers:
- name: ds4
  base_url: http://127.0.0.1:8000/v1
  model: deepseek-v4-flash
  models:
    deepseek-v4-flash:
      context_length: 200000

From inside Hermes, /model ds4/deepseek-v4-flash — matching the provider name in the YAML. Done.

Anyway, this gets us a generic token endpoint, so we’re free to plug in whatever on the front end.

The protagonists of this play have a lot of clout — antirez (Redis), mitsuhiko (Flask), badlogic (libGDX), and audreyt (Taiwan’s former Digital Minister, Pugs / Perl 6). Some kind of critical mass seems feasible for a certain type of nerd.