2  Installing a local backend

NoteLearning objective

You can stand up a local model server on an Apple-Silicon Mac in two ways (the MLX-LM server and LM Studio), explain why unified memory makes Apple Silicon a good fit, and you know which backend can actually drive pi’s tool-using agent loop and which is better kept for plain text generation.

pi does not care what serves the model, only that it speaks the OpenAI Chat Completions protocol over the Hypertext Transfer Protocol (HTTP) (Chapter 4). On Apple Silicon two MLX-based local backends expose exactly that endpoint, and this book uses both:

Backend Endpoint Models from Best job in this book
mlx_lm.server http://localhost:8080/v1 Hugging Face (MLX) generation-only workloads
LM Studio http://localhost:1234/v1 Hugging Face (MLX/GGUF) pi’s agent loop (tool-calling)

The split in the last column is the single most important fact in this chapter, and Section 2.2 explains why: the bare MLX server serves text well but does not return structured tool calls, so pi’s agent loop needs LM Studio.

2.1 Why Apple Silicon

Apple-Silicon Macs are unusually good at local inference because the central processing unit (CPU) and the graphics processing unit (GPU) share one pool of unified memory. There is no separate “VRAM” (video RAM) to copy weights into: the backend offloads the model to the GPU (via Metal) and the model’s memory simply comes out of system random-access memory (RAM). The practical rule of thumb is therefore blunt:

A model needs roughly its on-disk size in free RAM to run comfortably.

On the 128 GB machine used for this book that means even 40+ GB models load entirely on the GPU. On an 8–16 GB Mac you should stay with 3B–8B models.

Table 2.1 lists the models worth running on a 128 GB machine such as the Apple M5 Max as of July 2026.1 Each entry is a ready-converted MLX repository under mlx-community/… on Hugging Face, and the disk column is the actual download size, which by the rule of thumb above is also the memory the loaded model occupies.

Table 2.1: Recommended local models for a 128 GB Apple-Silicon Mac (Apple M5 Max), July 2026.
Model (mlx-community/…) Parameters Quantization Disk Best for
Qwen3.5-122B-A10B-4bit 122 B MoE, 10 B active 4-bit 70 GB strongest general model that fits; multimodal
gpt-oss-120b-MXFP4-Q8 117 B MoE, 5.1 B active MXFP4 63 GB deep reasoning with tool use
Qwen3-Coder-Next-4bit 80 B MoE, 3 B active 4-bit 45 GB agentic coding; pi’s tool loop; 262 K context
Qwen3-Coder-30B-A3B-Instruct-4bit 30 B MoE, 3 B active 4-bit 17 GB this book’s primary example model
Devstral-Small-2-24B-Instruct-2512-4bit 24 B dense 4-bit 15 GB lightweight agentic coding
gpt-oss-20b-MXFP4-Q8 21 B MoE, 3.6 B active MXFP4 12 GB this book’s reasoning alternative

The frontier open-weight releases of 2026 remain out of reach even here: Kimi-K2.5 needs 658 GB and GLM-5.2 roughly 400 GB in 4-bit, an order of magnitude beyond a laptop. The largest three models in Table 2.1 are the practical ceiling, and together with the two coder models they cover every task in this book: the coder models drive pi’s agent loop, the general models handle plain generation.

Two advances make this practical at all. Quantization compresses model weights to 4–8 bits with little quality loss, which is the reason a multi-billion-parameter model fits on a laptop (Zhu et al. 2024; Lin et al. 2025). And research on inference under tight memory budgets (including Apple’s own “LLM in a Flash” (Alizadeh et al. 2024)) pushes the envelope further, while reproducible benchmarks now quantify the performance–energy trade-offs of Apple Silicon for machine learning (Sierra-Herrera et al. 2026). Apple’s MLX framework (Hannun et al. 2023) is a native array framework optimized for exactly this unified-memory hardware, and it is what both backends in this chapter build on.

2.2 MLX-LM — running Hugging Face models

MLX-LM (Apple Machine Learning Research 2026) is the language-model toolkit built on Apple’s MLX framework (Hannun et al. 2023). It pulls models directly from the Hugging Face hub (typically the mlx-community/* repositories, which are already converted and quantized for MLX) and is often faster than a GGUF (GPT-Generated Unified Format) runtime on M-series hardware. Install it into the book’s uv environment (see the README) or any virtualenv:

uv add mlx-lm      # or: pip install mlx-lm

2.2.1 Generate and chat directly

The quickest way to use a model needs no server at all. mlx_lm.generate runs a single prompt and exits; mlx_lm.chat opens an interactive read–eval–print loop (REPL). Both download the model from Hugging Face on first use:

# one-shot generation
mlx_lm.generate --model mlx-community/Mistral-Small-24B-Instruct-2501-4bit \
  --prompt "Erkläre MoE-Modelle"

# interactive chat
mlx_lm.chat --model mlx-community/Qwen3-30B-A3B-4bit

On the test machine, a one-shot run against a smaller coder model produced real output and reported its own throughput:

$ uv run mlx_lm.generate --model mlx-community/Qwen2.5-Coder-7B-Instruct-4bit \
    --prompt "Erkläre MoE-Modelle in einem Satz." --max-tokens 70
==========
MoE-Modelle sind KI-Modelle, die mehrere Untermodelle enthalten und bei der
Vorhersage jeweils das Ergebnis eines geeigneten Untermodells verwenden.
==========
Prompt: 42 tokens, 213.866 tokens-per-sec
Generation: 47 tokens, 108.851 tokens-per-sec
Peak memory: 4.440 GB
TipDownload economics — Hugging Face

Public mlx-community/* repositories download anonymously over the fast Hugging Face content delivery network (CDN), with no token needed. A Hugging Face token is required only for gated repos (the official meta-llama/*, google/gemma-*, and some mistralai/*).

2.2.2 Serve an OpenAI-compatible endpoint

To let pi (or any OpenAI client) reach an MLX model, start the bundled server:

mlx_lm.server --model mlx-community/Qwen2.5-Coder-7B-Instruct-4bit --port 8080

It serves the OpenAI API (application programming interface) at http://localhost:8080/v1 (default host and port are 127.0.0.1:8080). Confirm it:

curl -s http://localhost:8080/v1/models
# {"object": "list", "data": [{"id": "mlx-community/Qwen2.5-Coder-7B-Instruct-4bit", ...}]}
Importantmlx_lm.server does not emit structured tool calls

Plain chat and JSON (JavaScript Object Notation) generation through mlx_lm.server work well, which makes it a fine backend for generation-only workloads (spotregen’s local driver, for instance, talks to exactly such an endpoint; see Regeneration drivers). But as of mlx-lm 0.31.3 the server does not return OpenAI-style tool_calls: when the model decides to call a tool, the server hands back the attempt as ordinary assistant text (a fenced JSON block) with tool_calls: null. A real pi run against it shows this clearly: asked to create a file, the model emitted

{
  "name": "bash",
  "arguments": { "command": "echo hi > marker.txt" }
}

as text, and marker.txt was never created. Because pi (like any tool-calling client) needs structured tool calls to dispatch a tool, pi’s agent loop cannot run through mlx_lm.server. Use it for generation, and use LM Studio (next) to drive pi.

2.3 LM Studio — MLX with tool-calling for pi

LM Studio (LM Studio 2026) runs the same mlx-community models but, unlike the bare MLX server, its OpenAI endpoint parses the model’s tool attempt into structured tool_calls, so pi’s agent loop works. Install the app and its command-line interface (CLI), lms:

brew install --cask lm-studio
lms server start                 # OpenAI API on http://localhost:1234/v1

Fetch a tool-capable MLX model from Hugging Face and load it:

lms get "https://huggingface.co/mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit"
lms load qwen3-coder-30b-a3b-instruct
lms ps                           # shows the loaded model, its context and device
WarningUse the full Hugging Face URL with lms get

The bare author/repo form fails for arbitrary community repos: LM Studio lowercases it and searches its own catalogue, reporting artifact … does not exist. Pass the full https://huggingface.co/… URL (uniform resource locator) instead. If lms rejects you with Invalid passkey for lms CLI client, a stale copy in ~/.lmstudio/bin is shadowing the app; re-bootstrap from the installed bundle (Chapter 8).

ImportantThe 8192 context trap

lms ps also reports the loaded model’s CONTEXT. LM Studio loads models with an 8192-token context window by default, regardless of what the model technically supports. For short agent tasks this is fine, but long sessions will silently truncate. Reload the model with a larger context window for real agent work. We return to this in Chapter 8.

With the model loaded, pi drives it through the lmstudio provider (Chapter 4). The agent loop fires correctly: asked to create a file, pi executed the bash tool and the file appeared on disk:

$ pi -p --offline --provider lmstudio --model qwen3-coder-30b-a3b-instruct \
    "Use the bash tool to run exactly: echo hi > marker.txt — then say done."
Done. The command `echo hi > marker.txt` has been executed successfully, creating a
file named `marker.txt` with the content "hi".
$ cat marker.txt
hi

This is the same uncheatable, on-disk proof used in Chapter 5, and it is the contrast that matters: the identical prompt left no file behind through mlx_lm.server.

TipExercises
  1. Run mlx_lm.generate with a short prompt against a small mlx-community model and read off the reported prompt and generation throughput (tokens per second) and peak memory. How does the peak memory compare with the model’s download size?
  2. Start mlx_lm.server and send it the curl …/v1/models request, then a /v1/chat/completions request with a tools payload. Confirm the model’s tool attempt comes back as text with tool_calls: null.
  3. Load a model in LM Studio and check its CONTEXT with lms ps. LM Studio defaults to 8192; raise it (reload with a larger context) and re-check.
  1. mlx_lm.generate prints the generated text between ========== markers, followed by three self-reported metrics: prompt tokens-per-sec, generation tokens-per-sec, and peak memory. Peak memory lands close to (slightly above) the model’s on-disk size, because the quantized weights are loaded into unified memory in full, plus a small overhead for the key–value (KV) cache and activations. This is the rule of thumb from this chapter made visible: a model needs roughly its on-disk size in free RAM.

  2. Expected observation: curl …/v1/models returns a JSON list with the loaded model’s id under data[].id. The /v1/chat/completions request with a tools payload comes back with the model’s tool attempt as ordinary assistant text (a fenced JSON block naming the tool and arguments) while tool_calls is null. As of mlx-lm 0.31.3 the bare server does not emit structured tool calls, so no tool is dispatched and any file the attempt describes is not created. This is why pi’s agent loop needs LM Studio rather than mlx_lm.server.

    curl -s http://localhost:8080/v1/models
    curl -s http://localhost:8080/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
        "model": "mlx-community/Qwen2.5-Coder-7B-Instruct-4bit",
        "messages": [{"role": "user", "content": "create marker.txt with hi"}],
        "tools": [{"type": "function", "function": {"name": "bash", "description": "Run a shell command"}}]
      }'
  3. Expected outcome: lms ps first reports CONTEXT 8192 (LM Studio’s default). After reloading the model with a larger context window and running lms ps again, the reported context reflects the new, higher value. The lesson: the default window is smaller than the model can support, so long sessions silently truncate unless you raise it deliberately. The larger window costs more memory.


  1. Recommendations cross-checked against community guides for M5-Max-class machines (https://apxml.com/posts/best-local-llms-apple-silicon-mac, https://insiderllm.com/guides/best-local-llms-mac-2026/); parameter counts and download sizes verified against the Hugging Face API on 2026-07-16.↩︎