Running LLMs locally on a Mac

1.1 Compute backends

The compute backend is the runtime that runs the matmuls — where on the chip the work actually happens. Three of them cover local text inference on the M-series, and which one a tool picks sets both its speed and how soon it handles a new model.

• PyTorch + MPS (Metal Performance Shaders) is the baseline. Most ML code reaches Apple Silicon through PyTorch, so coverage of new architectures arrives first — it is the lingua franca. Our embedding code runs here; speed is decent, not amazing.
• llama.cpp carries its own hand-written Metal kernels rather than going through MPS. It is the engine under Ollama and llama-server, fast and wide-coverage, and the one that eats GGUF.
• MLX is Apple’s own array framework — faster still on the work it covers, less mainstream, and lagging by months on new architectures. Osaurus, mlx-lm, and the JANG stack all use it.

Not treated here because they are only relevant to image models: CoreML on the Neural Engine for the lowest-footprint path, and Draw Things’ custom Swift + Metal stack.

1.2 Storage backends

The storage backend is the on-disk format the weights ship in — subject to whatever the quantization (if any) of the weights. RAM is usually the main constraint on consumer hardware, which is why quantization is helpful. A 70B model at full fp16 is ~140 GB and simply will not fit on even a 128 GB Mac. A 4-bit build of that same 70B lands near 40 GB, which is tolerable. There are several formats in play.

• safetensors — the Hugging Face baseline, full precision; what PyTorch + MPS loads when the model fits without help.
• GGUF — the llama.cpp format. Not Apple-specific (it runs on CUDA and CPU too), it has the widest coverage and the finest quant ladder — down to the very-low-bit IQ2/IQ3 imatrix quants.
• MLX (mlx-community) — the MLX-native format, Apple-only. Also quantized to fit but with added speed on the M-series; the quant range is coarser, and the coverage tends to lag.
• JANG — mixed-precision extension to MLX: per-tensor bit-widths instead of one width for the whole model.

Generally we prefer an MLX build for the speed when one is published; GGUF when no MLX port exists yet, and classic safetensors when neither is available.

The fixed width in MLX is indeed sus though — worth verifying that MLX models work OK. I suspect that JANG solves this better, but it is even more fringe.

2.1 Osaurus

Osaurus (MIT, brew install --cask osaurus, osaurus-ai/osaurus) is Swift-native, no Electron, no Python, and behaves like a proper Mac app.

I’m YOLOing all in on this because it seems efficient and easy. It locks me in to the Mac ecosystem so might not be for everyone. Also, it’s run by one person, so the bus factor is 1, which is a very small number. But — it’s so good!

The window has a model picker, a chat pane, a status indicator; the inference engine underneath is Apple’s fast MLX, so it gets many tokens per second. It is also the intended runtime for a custom mixed-precision MLX quantization format called JANG.

Osaurus is not just a chat client but a full native macOS agent harness. It supports various hip features like persistent memory, sandboxed working folders in an isolated Linux VM via Apple’s Containerization framework… It understands agentskills.io-format skills (and whole Claude plugins) from GitHub or local files, selecting them by RAG at runtime, and speaks MCP in both directions, server and client. The harness layer is model-agnostic, fronting cloud providers as happily as the local MLX-ish runtime.

There is no CLI download command; the in-app Model Manager (⌘⇧M → Models) browses a curated catalogue of models, especially JANG ones, and will sideload others too — though not all of them work equally well. Nemotron 3 Nano Omni 30B A3B JANGTQ4 seems like a reliable workaday default. Osaurus also discovers anything dropped into its models directory:

hf download mlx-community/DeepSeek-R1-0528-Qwen3-8B-4bit \
  --local-dir ~/MLXModels/mlx-community/DeepSeek-R1-0528-Qwen3-8B-4bit   # mirror the org/repo path
osaurus list    # confirms discovery, and gives the exact API name for `osaurus run`

Osaurus is also a first-class server, covered below.

2.2 Jan

Jan (brew install --cask jan) is a FOSS cross-platform option — a full frontend + harness + server + runtime bundle. The UI is built on the mildly cursed Electron, but the plus side is that it runs on Linux, Windows, and macOS. It supports both llama.cpp (via Cortex) and MLX backends, so for brute-force running non-Apple-optimised models it seems to go.

It looks nice, and the Projects / Assistants / Agents / MCP Connectors quartet gives it a tool-calling agent loop — connect MCP servers under Settings, and Agents mode runs multi-step autonomous workflows (Jan v2 VL is pitched as a 49-step multimodal agent). Jan Server is the self-hosted orchestration variant.

2.3 LM Studio

LM Studio (brew install --cask lm-studio) is closed-source, relatively slick and turnkey, and not free for commercial use. It runs both llama.cpp and lately its own MIT-licensed mlx-engine (mlx-lm + Outlines + mlx-vlm). Like the others it is a bundle — frontend + runtime + server — with an OpenAI endpoint it can expose headlessly (lms server). I’m mildly sceptical of it because so many Cool LLM Technologies ship special bug fixes or alternate install paths for LM Studio, which hints at a slightly non-standard stack — though that might just be sampling bias, since more people file bug reports when more people run the thing. Also the licence sucks.

These three are general-purpose chat apps. To drive a local model as a coding agent instead — terminal harnesses like OpenCode or Aider, VS Code sidebars like Cline — see Code agents and assistants, which points back here for the local backend they run against.

3.1 Osaurus as a server

If we already have Osaurus running we are mostly done: it is already that daemon, exposing OpenAI-, Anthropic-, and Ollama-compatible endpoints on localhost:1337 all at once (some deep magic). Anything we want to point at a local model can talk to it. But also, for reasons of not installing an idiosyncratic stack, or superior customisation, we might want to install the standard Linux server stack.

To keep two models resident at once (say the agentic daily-driver plus the maths model), set Settings → Local Inference → Model Management to Flexible — under the default Strict policy, loading one evicts the other.

Context length is automatic — Osaurus picks a sane per-model default and does not expose it as a plain setting, so the actual ceiling is hard to read off (the cheat-sheet has the detail).

To drive all this from the terminal there is one gotcha: the osaurus command is embedded in the app bundle, and only the Homebrew install links it onto PATH automatically. If it is missing, symlink it: ln -sf “/Applications/osaurus.app/Contents/Helpers/osaurus” “$(brew --prefix)/bin/osaurus” or use the special button from the settings menu¹ The CLI supports stuff like osaurus serve / stop / status / list / run <model> / mcp, plus a plugin manager.

3.2 Ollama

Ollama (brew install ollama) is a llama.cpp wrapper with its own model registry — fast enough, wide model coverage, and notably good for embedding models:

brew services start ollama
ollama pull qwen3.5  # LLM/chats etc
ollama pull mxbai-embed-large    # also handles embeddings

Anything OpenAI-API-compatible can now point at http://localhost:11434/v1. Ollama is the hands-off one on lifecycle: it loads a model on first request, keeps several resident at once (up to OLLAMA_MAX_LOADED_MODELS, default 3), and unloads each after OLLAMA_KEEP_ALIVE of idleness (default 5m). Left unbounded the pool tanks the machine — OLLAMA_MAX_LOADED_MODELS=1 forces evict-on-switch, OLLAMA_KEEP_ALIVE=0 drops a model the moment it idles (or 15m to keep it warm longer). Context window is num_ctx: set it per request (options.num_ctx), bake it into a Modelfile (PARAMETER num_ctx), or lean on the OLLAMA_CONTEXT_LENGTH default — which on recent Ollama auto-scales to VRAM (4k / 32k / 256k) rather than the old fixed 2048.

Gotchas:

• The .gguf files come from Ollama’s registry, not Hugging Face.
• Some weird reimplementation headaches — e.g. the tokenizer baked into the GGUF can differ from the original for unclear reasons.
• The “llama.cpp wrapper” framing is loosening: the registry now ships -mlx tags for some models (e.g. qwen3.5:35b-mlx).

3.3 mlx-lm and mlx-vlm

mlx-lm is Apple’s reference language-model runtime on MLX. mlx-vlm is its sibling package: same MLX backend, same mlx-community/<repo> weights and HF cache, but for VLMs (“vision-language models”) and omni models — image/video/audio in, text out — instead of pure text LLMs. Where this page says “VLM” it means a model in that family; mlx-vlm is what runs one locally, e.g. DeepSeek-OCR. Osaurus and JANG are inspired by MLX-type Apple-Silicon-friendly execution, but mlx-lm is the OG.

uv tool install mlx-lm drops a family of commands onto PATH, all reading the same weights:

• mlx_lm.generate --model mlx-community/<repo> — one-shot completion from the CLI.
• mlx_lm.chat — an interactive REPL in the terminal.
• mlx_lm.server --model mlx-community/<repo> — an OpenAI-compatible daemon; holds one model, swapping on demand per request (evict + reload, not a restart). Two live at once means two processes on two ports.
• mlx_lm.lora — its LoRA fine-tuning path.

mlx-vlm mirrors this shape (mlx_vlm.generate, mlx_vlm.chat, mlx_vlm.server) but takes an image/video/audio argument alongside the text prompt.

It uses the standard HuggingFace links: mlx-community/<repo> resolves straight to Hugging Face, and the weights land in the shared HF cache (~/.cache/huggingface/hub).

mlx_lm.server has no --ctx flag, so it grows the KV cache to fit whatever we send — up to, presumably, the model’s declared max, capped only by RAM, at which point it presumably kernel-panics the machine. Cap the context in the harness (limit.context / contextWindow) and mind the memory budget.

A reason to keep this around even with Osaurus installed is that Osaurus is great when it runs, but its Swift engine’s coverage lags the Python MLX options. A plain mlx-lm loads interesting MLX conversions that Osaurus can’t (e.g. Cascade-2).

3.4 vllm-mlx

vllm-mlx (uv tool install vllm-mlx, Apache 2.0) is a vLLM-style inference server for Apple Silicon. Active and popular by this page’s standards — 1,300+ stars. Core pitch: ontinuous batching, paged KV cache with prefix sharing, an SSD-tiered cache for spilling prefixes to disk, and both OpenAI (/v1/*) and Anthropic (/v1/messages) endpoints from one process. Many fancy bous features: Native TTS (Kokoro, Chatterbox, VibeVoice, VoxCPM) and STT alongside text/image/video/audio chat, speculative decoding (--mtp) for Qwen3-Next, MoE expert-count reduction (--moe-top-k, a claimed 7–16% speedup on Qwen3-30B-A3B), Prometheus metrics (--metrics), and a built-in benchmarker (vllm-mlx bench-serve). Support multi-model residency, via a --models-config models.yaml registry: named models behind one process, lazy load on first use, LRU eviction under a memory_budget_gb, and a contention_policy (fail / wait / preempt / wait_then_fail / wait_then_preempt) for what happens when a request needs a model that does not currently fit; clients pick one via the normal OpenAI model field.

uv tool install vllm-mlx
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching

For a chat window over this endpoint — one that renders the equations the maths models emit — Open WebUI points at it unmodified: add http://localhost:8000/v1 as an OpenAI connection and skip the bundled-Ollama path the tutorials assume.

vllm-mlx serve ~/MLXModels/mlx-community/Qwen3.6-35B-A3B-4bit --port 8000 --continuous-batching

manager:
  memory_budget_gb: 100
  contention_policy:
    strategy: wait_then_preempt
    wait_timeout_s: 45
    preempt_after_s: 15

models:
  - name: driver
    path: /Users/dan/MLXModels/mlx-community/Qwen3.6-35B-A3B-4bit
    continuous_batching: true
    estimated_memory_gb: 22

  - name: solver
    path: /Users/dan/MLXModels/mlx-community/VibeThinker-3B-8bit
    preload: true
    estimated_memory_gb: 3

vllm-mlx serve --models-config ~/.config/vllm-mlx/models.yaml \
  --port 8000 \
  --gpu-memory-utilization 0.88 \      # hard process ceiling ≈112 GB (0.88 × 128)
  --continuous-batching \
  --use-paged-cache \
  --cache-memory-mb 30720 \
  --max-cache-blocks 16384 \
  --max-num-seqs 16 \
  --max-tokens 131072 \
  --max-request-tokens 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser auto \
  --reasoning-parser qwen3 \
  --enable-metrics \
  --kv-cache-quantization \
  --kv-cache-quantization-bits 8 \
  --timeout 1200

3.5 oMLX

oMLX (jundot/omlx, Apache 2.0, brew tap jundot/omlx https://github.com/jundot/omlx && brew install omlx) is vllm-mlx forked and grown a different frontend, so it inherits that feature set — continuous batching, multi-model residency, both OpenAI and Anthropic endpoints. Three things set it apart:

• Its SSD prefix cache is automatic and block-addressed — hot blocks in RAM, cold blocks spilled to disk, longest-prefix matched and surviving a restart — where llama-server’s --slot-save-path and ds4’s --kv-disk-dir are manual slot-save knobs. Aimed at agentic coding, where the pitch is TTFT dropping from 30–90s to under 5s on the second turn of a long context.
• An explicit Claude Code accommodation: it rescales reported token counts so auto-compact fires at the right time, and holds the connection open with SSE keep-alives through a long prefill. The frontend is a signed SwiftUI menu-bar app (not Electron) with a web admin panel, and it reuses an existing LM Studio model directory.
• It is the one non-Osaurus server with merged JANG support, so it can load the mixed-precision JANG quants that make sub-4-bit MoE models behave — otherwise Osaurus-only.

omlx serve --model-dir ~/MLXModels \
  --paged-ssd-cache-dir ~/.omlx/cache \
  --hot-cache-max-size 16GB \
  --max-concurrent-requests 8

Same caveats as the rest of the page (bus factor 1, MLX-only, benchmarks from an M3 Ultra 512GB), but the automatic SSD KV cache and clean lineage make it worth a run as the headless daily-driver if the Osaurus/mlx_lm.server pair leaves us wanting persistent prefix reuse.

hf download gabfssilva/VibeThinker-3B-MLX-BF16  --local-dir ~/MLXModels/gabfssilva/VibeThinker-3B-MLX-BF16  # solver, hi-fi bf16
hf download mlx-community/VibeThinker-3B-8bit   --local-dir ~/MLXModels/mlx-community/VibeThinker-3B-8bit   # solver, near-lossless and faster
hf download mlx-community/Qwen3.6-35B-A3B-4bit  --local-dir ~/MLXModels/mlx-community/Qwen3.6-35B-A3B-4bit  # the driver

omlx serve --model-dir ~/MLXModels \
  --memory-guard-gb 110 \          # 128 GB Mac: driver (~22 GB) + solver (~3 GB) + KV headroom
  --max-concurrent-requests 16     # the fan-out width; default is 8

3.6 llama.cpp / llama-server

llama.cpp ships its own server (brew install llama.cpp): llama-server -m model.gguf exposes an OpenAI-compatible endpoint with no daemon, no registry, no opinions.

Ollama wraps this same engine. However, it is more configurable, exposing llama.cpp flags that Ollama does not, notably: YaRN context extension on a GGUF (which Ollama cannot do at all), the finer KV-cache quant ladder, speculative decoding with a draft model, and per-slot KV save/restore to disk.

It loads a .gguf from disk — no ollama pull into a separate store, no background service — which suits scripted or reproducible runs.

It loads the one model named at launch, though a newer router mode (start it with no -m) does dynamic multi-model load and unload. Context is the -c / --ctx-size flag, defaulting to 0 (the model’s full trained window). Other useful config options can be found in the cheat-sheet.

3.7 Sampling defaults

Sampling — temperature, top-p, top-k, and the output-token budget — is a decode-time choice set on each request. In particular the server does not use the model’s recommended sampling settings automatically.

mlx_lm.server, for example, defaults to temperature=0.0 (greedy), top_p=1.0, top_k=0, and max_tokens=512, and it does not read the model’s generation_config.json. So a reasoning model’s advised settings — VibeThinker wants temperature 1.0 / top-p 0.95 and a 64K-plus output budget — need the client to specify them; each model table below carries a Recommended sampling column with the per-model picks. Notably temperature=0.0 breaks maj@k voting, since every sample comes back identical.

Where to pin the values depends on which layer issues the prompt.

• mlx_lm.generate — flags per call: --temp 1.0 --top-p 0.95 --max-tokens 40000 (--top-k already defaults to 0). Wrap it in an alias.
• mlx_lm.server — launch-time defaults via --temp / --top-p / --top-k / --min-p, overridden per request; Osaurus keeps the same defaults in its app settings, also overridden per request.
• transformers — a GenerationConfig passed at generate() time.
• Ollama — is an exception, baking in a per-model default through a Modelfile: PARAMETER temperature 1.0, PARAMETER top_p 0.95, PARAMETER num_predict 40000 for the output budget, PARAMETER num_ctx 65536 for the context window.

Output budget and context window are separate limits: the first caps how much the model may emit, the second how much prompt-plus-output the KV cache holds. A long-reasoning model needs both raised, or it truncates mid-derivation — and Ollama in particular drops the overflow silently once num_ctx is exceeded.

3.8 Prefix caching

The KV cache is the only state these servers keep between requests — there is no session object, we resend the whole history each call. What they reuse is the prefix: the shared start of the conversation (system prompt, tool definitions, history so far) keeps its KV, so an agentic loop prefills only the new suffix. Matching is content-addressed — on the tokens, not a session ID — so it happens by itself. It is likely not optimal in general.

When planning around this we need to be aware that trimming the front of the history will break this caching and require everything to be pre-computed.

Different servers clear this cache at different times:

• mlx_lm.server — an in-memory LRU, longest-prefix matched, reporting cached_tokens in the usage block; bounded by --prompt-cache-size / --prompt-cache-bytes, held for the process.
• llama-server — --cache-prompt is on by default (one KV per slot; --cache-reuse even salvages chunks after a mid-prompt edit), and slots can be saved to disk.
• Ollama — the same llama.cpp reuse, alive as long as the model stays loaded (OLLAMA_KEEP_ALIVE).
• Osaurus — automatic; headless under osaurus serve the cache lives for the server process (governed by the Strict/Flexible policy), and in the GUI it is per chat window, warmed the moment one opens.

Prefill is chunked and continuously batched besides, so concurrent requests interleave rather than queue — but the prefix skip is the bigger win.

3.9 Stretching the context window

A model’s positional encoding is trained out to some fixed length, and that trained length is the context window the weights actually know. Some fancy huge numbers that get advertised are that window stretched at load time, not a property of the weights. Qwen3.6, for example, trains its rotary positions to 256K (262,144 tokens); the 1M figure quoted for it is that same window extended ~4×, and getting the extension costs us something in both quality and RAM.

The mechanism is RoPE interpolation. RoPE encodes each token’s position as a rotation; interpolation rescales those rotations so a position past the trained window maps back into the range the model saw during training, instead of falling off the end into rotations it has never seen. The common variant is YaRN (“yet another RoPE extension”), which rescales per frequency rather than uniformly. A factor of 2.0 takes Qwen3.6’s 256K to ~512K, 4.0 to ~1M.

Every mainstream implementation AFAICT applies the rescaling statically, fixing it at load and applying it to every prompt regardless of length. A model loaded with factor 4.0 rescales a 2K-token prompt exactly as hard as a 900K one, and short prompts lose some accuracy for a long window they aren’t using. So we switch YaRN on only when we want the long window, and set factor to the longest context we actually expect rather than the largest the model will accept.

How to turn it on depends on the runtime.

transformers and mlx-lm read a rope_scaling block straight from the config.json that ships in the model’s own directory:

"rope_scaling": {
  "rope_type": "yarn",
  "factor": 4.0,
  "original_max_position_embeddings": 262144
}

• llama.cpp / llama-server take flags: --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 262144, alongside the usual -c.
• Ollama exposes no YaRN settings. We inherit whatever the person who converted the GGUF baked into its metadata.
• Osaurus’s Swift engine has the YaRN code, but its Qwen3 and Llama wrappers do not route through it, so it is unavailable for now.

YaRN increases the maximum context window, but does not shrink the KV cache cost of storing that context, so we are still RAM-constrained.

3.10 Configuring each server

All the settings in one place.

Flash attention has no column because it has stopped being a per-server decision: llama.cpp defaults -fa to auto and enables it wherever Metal supports it, Ollama switches it on per-architecture for the families here (Qwen3.x, Nemotron, Gemma, gpt-oss), and the MLX servers and ds4 always run a fused attention kernel. The one place it still needs a hand is Ollama’s KV-cache quant, which only takes effect with OLLAMA_FLASH_ATTENTION=1 set alongside it.

Sensible headless starting points:

# 64K context (below): a useful bound well under the trained max — raise or lower for your RAM
# llama.cpp — flash attention is automatic; quantize the cache, choose a context
llama-server -m model.gguf -c 65536 --cache-type-k q8_0 --cache-type-v q8_0

# Ollama — env vars; cache quant needs flash attention enabled
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_CONTEXT_LENGTH=65536 ollama serve

# mlx-lm — no cache-quant/context flag; set the model's sampling values, cap context in the harness
mlx_lm.server --model mlx-community/<repo> --temp 0.6 --top-p 0.95 --top-k 20

# oMLX — point it at the shared MLX tree; turn on the SSD cold tier and a memory ceiling
omlx serve --model-dir ~/MLXModels --paged-ssd-cache-dir ~/.omlx/cache --memory-guard-gb 96

The --paged-ssd-cache-dir on that last line persists across restarts and is rewritten every session, so it is the one path here that wants keeping out of backups and Spotlight.

Osaurus is tuned in its Settings pane rather than on a command line, and ds4’s full flag set is in its own section.

5.1 For mathematical reasoning

Two kinds here, split by type from the theory page. Generalists reason about maths alongside chat and tool-use; solvers trade that away for raw maths and lean on specialised harnesses — tool-integrated reasoning, maj@k voting, cloud fan-out.

These pure-CoT solvers carry no tools, so the tidy way to fold one into a larger workflow is as a solve() oracle a general agent dispatches to, not a bespoke chat window: serve it on a batching endpoint like oMLX that keeps it resident beside the agentic driver, and let the driver call it when it meets a sub-problem it should not attempt itself.

5.2 For agentic flows

These are the weights a coding harness needs:

5.3 Diffusion models

Everything above is autoregressive. Diffusion LLMs are a thing though. Do any run locally? I know of one: DiffusionGemma — Google’s experimental Gemma 4 variant, which denoises a whole 256-token canvas in parallel instead of emitting tokens left to right. OsaurusAI/diffusiongemma-26B-A4B-it-MXFP8 runs natively in Osaurus through its vmlx-swift block-diffusion engine, ~26 GB on disk and ~24 GB resident.

Manage expectations on speed. Docs report 28–42 tok/s at 48 denoising steps: Osaurus defaults to 16, roughly twice as fast as the bundle default and still coherent, and the output falls apart below 12. Quality trails plain Gemma 4 as well. Vision, tool-calling, and a reasoning channel all work in this checkpoint; audio and video do not.

So it is not the agentic daily-driver. I am interested in the interaction model though: bi-directional attention over the canvas makes it potentially useful for infilling and structure-preserving rewrite, which is a different way to drive a coding tool than streaming tokens into a chat box.

6.1 Tech stack

JANG (“Jang Adaptive N-bit Grading”) is mixed-precision quantisation for MLX. Standard MLX quantisation compresses every tensor to the same bit width. JANG classifies tensors by sensitivity — attention and MoE router layers (small share of params, large share of model behaviour) get 6–8 bits; expert MLPs get 2–4. The hybrid network is a mildly extended version of the standard MLX safetensors format with a per-tensor bit-width manifest. At the same total size, accuracy improves, notionally. The pitch is “GGUF for MLX”, which … sounds good? I’m not really competent to judge. Apparently llama.cpp’s K-quants do something similar.

The other part of the GGUF quality story is the calibration data fed in at quantisation time — which is why a bartowski/…-GGUF repo (like the Nemotron one) is a slightly different, usually better thing than a bare K-quant of the same weights. I am unsure if JANG does this.

jangq.ai claims impressive performance, regularly beating models with a larger footprint. At least one third-party benchmarker is impressed.

Osaurus is JANG native. Pull a model from JANGQ-AI and it loads — usually. The Swift engine’s coverage tracks the JANGTQ path; a plain JANG_* quant of an exotic architecture can fail at weight-load (notably Cascade-2 doesn’t work — the Python jang-tools stack handles those, the Swift engine does not yet). Elsewhere, support is partial: MLX Studio, vMLX, and oMLX all load JANG natively; LM Studio / Ollama / Jan not yet. From Python: uv pip install “jang[mlx]”, then jang_tools.loader.load_jang_model(...).

6.2 MLX Studio

MLX Studio is the JANG/Osaurus author’s other Mac desktop app — Electron + Python rather than Swift, broader feature surface (image generation via Flux and Z-Image, ~26 built-in agentic tools, in-app GGUF→MLX and MLX→JANG conversion, an Anthropic-compatible API). Install via the signed DMG on the releases page, or engine-only with uv tool install vmlx and vmlx serve mlx-community/<repo> (OpenAI-compatible on localhost:8000).

7.1 Running DwarfStar via the pi stack

The default harness for ds4 seems to be: pi, an MIT-licensed agent harness by Mario Zechner (badlogic, of libGDX fame) — itself a strong offline coding agent once a model is behind it. There is an easy install via the pi extension by Armin Ronacher (mitsuhiko, of Flask): mitsuhiko/pi-ds4. It handles process management for ds4-server — per-PID leases, watchdog shutdown, OpenAI-compatible local endpoint on 127.0.0.1:8000:

pi install https://github.com/mitsuhiko/pi-ds4

First-time install clones antirez/ds4, builds it, downloads the GGUF (~87 GB), and registers a ds4/deepseek-v4-flash model with pi. Subsequent runs spawn the server on demand and shut it down when no client process holds a lease. OpenClaw embeds pi, so the same extension can in principle load there.

pi from the terminal opens a TUI (“textual user interface” — I think that’s what it means, i.e. it lives in the terminal).

Audrey Tang maintains audreyt/pi-ds4, a fork that swaps in cyberneurova’s abliterated IQ2XXS quants and turns on uncertainty-mode directional steering by default — an activation-space edit that puts the model into “this is a contested question” mode on CCP-sensitive topics (Taiwan, Crimea, Kashmir, Western Sahara).

7.2 Manual setup for non-pi harnesses

Outside the pi ecosystem, the manual setup is four commands plus a config edit.

# antirez/ds4 for upstream; also audreyt/ds4 looks cool
# optimisations + steering-vector work — pick one
git clone https://github.com/audreyt/ds4
cd ds4
make
tmutil addexclusion -p (realpath ./gguf)
./download_model.sh                     # ~87 GB into ./gguf/
./ds4-server                            # listens on 127.0.0.1:8000

For lifecycle, we could wrap ./ds4-server in a launchd plist with KeepAlive: true; this is probably not what we want on a typical laptop where we do other things than inference, like, you know, use it as a laptop. I think pi is more automatic in that regard.

ds4-server’s context window is set at launch via --ctx <tokens> (max accepted per conversation); output length is a per-request API field, not a launch flag. --kv-disk-dir <path> (with --kv-disk-space-mb <n>) persists the KV cache to disk, so a prefix survives restarts and session switches rather than being reprocessed — durable prefix storage, not a long-context spill. Thinking mode is on by default, toggled per request, running DeepSeek’s reasoning mode. DeepSeek V4 Flash nominally supports 1M tokens, but ds4 is RAM-bound: the 2-bit IQ2XXS weights are ~81 GB, and a full 1M-token KV/index sits around 26 GB on top. Rough budget on unified memory:

• 64 GB: 50k–150k --ctx with headroom.
• 96 GB: 150k–250k works but is tight; quit Slack.
• 128 GB: 200k–300k is comfortable; >300k starts risking OOM.
• 1M: only with very generous memory and nothing else running.

If a client (Hermes, OpenClaw, OpenCode, anything OpenAI-compatible) advertises a context larger than --ctx, requests will get cut off — match the client’s contextWindow / limit.context to the server’s --ctx. DeepSeek’s sparse attention means raising --ctx doesn’t blow up compute the way dense attention would, but RAM is still a constraint. For most interactive coding, 32k–100k plus a retrieval layer beats brute-forcing the whole history into the prompt. See antirez/ds4’s README and the OpenClaw ds4 provider docs for the full flag list and client-side config.

Reasonable defaults:

./ds4-server --ctx 200000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

To bake in audreyt’s directional-steering defaults:

DS4_DIR_STEERING_FFN=-0.75 \
DS4_DIR_STEERING_ATTN=0 \
DS4_REPRODUCIBLE=1 \
./ds4-server

To use with Hermes, add an OpenAI-compatible provider entry to the Hermes config (sketch — confirm the exact schema with hermes config):

# ~/.hermes/config.yaml
custom_providers:
- name: ds4
  base_url: http://127.0.0.1:8000/v1
  model: deepseek-v4-flash
  models:
    deepseek-v4-flash:
      context_length: 200000

From inside Hermes, /model ds4/deepseek-v4-flash — matching the provider name in the YAML. Done.

Anyway, this gets us a generic token endpoint, so we’re free to plug in whatever on the front end.

The protagonists of this play have a lot of clout — antirez (Redis), mitsuhiko (Flask), badlogic (libGDX), and audreyt (Taiwan’s former Digital Minister, Pugs / Perl 6). Some kind of critical mass seems feasible for a certain type of nerd.