Towards Data Science

How to Use Apple's fm CLI to Run On-Device LLMs From Python

7.1内容质量

TL;DR · AI 摘要

How to Use the Apple fm CLI With Python and fm serve Large Language Models How to Use Apple's fm CLI to Run On-Device LL...

核心要点

  • 主题聚焦:How to Use Apple's fm CLI to Run On-Device LLMs
  • 来源:Towards Data Science,建议结合原文判断细节。
  • AI 分析暂不可用,本条为保底评分与摘要。
#AI#编程#后端#云计算#安全
打开原文

How to Use the Apple fm CLI With Python and fm serve

Large Language Models

How to Use Apple's fm CLI to Run On-Device LLMs From Python

A Sate Padang vision test and a Malaysia capital question show why the free, private model needs grounding and validation before it leaves prototyping.

Irfan Alghani Khalid

November 11, 2026

10 min read

Prefer us on Google

Share

Photo by Solen Feyissa on Unsplash

Introduction

AI engineers and LLMs are increasingly inseparable . When engineers want to build an AI feature, they're more and more likely to use a cloud-hosted LLM to deliver AI solutions to their systems.

While cloud LLMs offer state-of-the-art reasoning, relying entirely on these remote endpoints during early-stage prototyping introduces noticeable operational expenses.

Imagine a development workflow where an engineering team tests automated image parsing pipelines during R&D. While lightweight cloud vision models (such as GPT-4o-mini or Gemini 1.5 Flash) are inexpensive per request, running continuous tests and prompt-engineering iterations can accumulate API costs and slow down local feedback loops due to network round-trips.

What if developers could execute these visual extraction test loops locally on Apple Silicon with zero network latency, complete data privacy, and zero API token cost?

Included in recent macOS releases, Apple provides a native terminal utility accessible straight from the command line: the fm (Foundation Model) CLI.

In this article, we conduct a practical engineering evaluation of Apple's built-in fm tool. We will test its text reasoning, visual extraction accuracy, integration patterns in Python, and also discuss its architectural limitations.

Apple Foundation Model

Apple Foundation Model has been part of Apple's development tools since last year. It is designed to run Apple's on-device models directly, leveraging the Apple Neural Engine and Unified Memory. With this integration, developers need no complex setup and get optimized performance out of the box.

Since the release of macOS 27 Golden Gate, Apple has made this model more accessible by adding it to the Mac's Terminal. With this addition, developers can integrate the LLM inside their applications, even if the app is web-based!

Before getting into the Terminal, make sure that you update your device to the newest macOS. Okay, without further ado, let's get started!

To use the Apple Foundation Model, type the fm command inside the Terminal.

If you use the fm command for the first time, it will ask you to agree to its license first. Type sudo fm license in your terminal. Then, read the terms and type yes to agree. From there, you can try playing with the fm command.

The screenshot is captured by the author.

After you agree to the license and re-run the fm command, it will show an output like this:

As you can see above, there are several subcommands to use. This time, we will focus on the fm respond first. This command is the gateway for interacting with the LLM. If I want to ask What is the capital of Malaysia? , we can write a command like this:

bash

code
fm respond 'What is the capital of Malaysia'

During the testing, the model responded with "Ipoh" instead of "Kuala Lumpur".

The screenshot is taken by the author.

While this might seem like a minor hallucination, it highlights a crucial architectural trade-off of compact on-device models. To maintain low memory usage and high inference speed on consumer hardware, on-device models sacrifice vast parametric knowledge.

From here, we should not use Apple's fm model as an ungrounded factual knowledge base . Instead, we have to pair it with Retrieval-Augmented Generation or structured external context.

The great thing about this model is that it runs directly inside your hardware, and no data is sent outside of your device. So you not only get a free model but also a private, hardware-optimized model.

Multimodal Vision

And another thing that I'm truly excited about with this Apple LLM is the capability of parsing an image! How do I find that out? The fm respond --help output confirms this.

By typing the fm respond --help command, it will show you that you can use the --text and --image flags with the fm command so it can process the image while adding the context on what you are going to do inside of this image.

Let me give you an example. Suppose that I want to analyze this tasty yet sinful Sate Padang.

Sate Padang: An Indonesian Delicacy. The image is provided by the author

I want the model to extract the dish name, the cooking method, and whether the dish contains coconut milk (santan). Inside the terminal, we can write this command:

code
fm respond \
--image [YOUR_IMAGE_PATH] \
--text "Analyze the food image and output ONLY raw JSON: { 'menu_name': string, 'is_fried': boolean, 'contains_santan': boolean }"

From here, the model will show an output that looks like this:

json

code
{
"menu_name": "Satay",
"is_fried": true,
"contains_santan": false
}

The model successfully generated valid JSON structure and identified the core dish as "Satay." However, it misclassified the preparation method as fried ( is_fried: true ), whereas satay is traditionally grilled.

This misclassification demonstrates the limits of lightweight vision encoders when processing fine-grained visual features (e.g., distinguishing grilled texture from fried oil sheen). If an application relies on this data for strict nutritional or medical tracking, unvalidated outputs could introduce severe data quality issues.

Even a well-formed JSON response only guarantees the shape of the output, not its truth, so any field that feeds a real decision should be checked against external context rather than trusted from the model alone.

Where fm Fits (and Where It Doesn't)

The two errors above are not reasons to dismiss fm . They define its boundary. Based on the tests, I place the tool in three roles rather than treating it as a drop-in replacement for a cloud API.

  1. Local prototyping for web applications

During development, fm lets us iterate on prompts, JSON parsing, and the Streamlit UI without per-token charges or network round-trips. This is also the sense in which a web-based app can use fm : the model runs on the developer's Mac while the interface and parsing logic are being built. It is not the service that answers requests from end users.

fm serve is useful in this stage. Because it exposes an OpenAI-compatible endpoint, moving to a cloud provider later is mostly a change of base_url and model name. Two caveats apply. Prompts tuned on a compact model can behave differently on a larger one, so the switch needs re-testing. And this workflow assumes every developer on the team works on Apple Silicon.

  1. On-device inference for native Apple apps

For iOS and macOS applications, Apple's Foundation Models framework runs inference on the user's own device. This is the case where per-request inference cost genuinely drops to zero for the developer, and where user data never leaves the device. For shipping to end users, the Swift framework is the intended path; the CLI remains a scripting and development interface.

  1. Standalone production backends still need cloud LLMs

When an application serves users across platforms from a shared server, handles long documents, or requires strong factual accuracy, a hosted model remains the right choice.

The Ipoh response and the grilled/fried mix-up are the concrete reasons: a compact on-device model without grounding should not be the only source of truth.

Other Tools Worth Considering

fm is not the only option for local inference, and for some tasks it is not even the most direct one.

Tool

When it is more relevant than fm

Ollama / LM Studio / llama.cpp

Cross-platform local inference. Lets you swap open-weight vision models instead of staying with Apple's fixed model.

MLX / mlx-vlm

Still Apple Silicon, but with open weights and more control over quantization and fine-tuning.

Apple Vision framework / OCR

Plain text extraction from receipts or documents. The fm respond --help output itself lists an ocr tool; for OCR alone, a dedicated vision API is more predictable than an LLM.

GPT-4o-mini / Gemini 1.5 Flash

Cheap cloud baselines for the same extraction task when the team is not on Apple Silicon, or when the prototype moves to a shared backend.

None of these makes fm redundant. Its advantage is still zero setup, no weights to download, and data that stays on the machine. The point is to choose the layer deliberately instead of keeping one model as the default for every stage.

Integrating FM into applications

There are two ways to call the model from an application. They differ in lifecycle, not in model quality.

The fm serve method

First, you can run fm serve inside the terminal. It will run an API on your Mac, and you can call it to generate output from the AI model. This is the screenshot after you run the fm serve command:

If you want to interact with the API, you can run a curl command to send a POST request to the LLM model in your terminal like this:

code
curl -X POST http://127.0.0.1:1976/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "system",
"messages": [
{"role": "user", "content": "Hello! What model are you?"}
]
}'

And the output looks like this:

From the API, the LLM will generate a stream of JSON-formatted outputs consisting of chunks.

For example, if the LLM generates an answer like " Hello! I'm a foundation model. ", it will generate separate entities as an output like the screenshot above instead of a unified sentence. If you want to return a single response body, you can add "stream": false inside the body request like this:

text

code
curl -X POST http://127.0.0.1:1976/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "system",
"messages": [
{"role": "user", "content": "Hello! What model are you?"}
],
"stream": false
}'

And here is the output:

code
{"usage":{"total_tokens":77,"completion_tokens_details":{"reasoning_tokens":0},"prompt_tokens":62,"completion_tokens":15,"prompt_tokens_details":{"cached_tokens":0}},"id":"chatcmpl-99BCE1F6-A9C9-4100-AD61-1AE33F20B6A9","object":"chat.completion","choices":[{"index":0,"message":{"content":"I’m an LLM.","role":"assistant","refusal":null},"finish_reason":"stop"}],"created":1794108065,"model":"system"}

The practical advantage here is portability. Application code written against the OpenAI SDK can point at http://127.0.0.1:1976/v1 during development and later switch base_url and model to a hosted provider. As noted in the previous section, that switch still requires re-testing the prompts.

Call the fm with subprocess

For a single-machine Streamlit prototype, a long-running server is more infrastructure than needed. The alternative is to invoke fm respond directly and let Python own the output contract.

And for analyzing the image, I call the fm command using the subprocess library. The code snippet of the system looks like this:

python

code
#### [OTHER CODE]
st.image(temp_path, caption="Uploaded Image", use_column_width=True)
if st.button("Analyze with On-Device AI"):
with st.spinner("Processing on Apple Neural Engine..."):
# Construct the CLI command
prompt = "Analyze this food image and output ONLY raw JSON: { 'menu_name': string, 'is_fried': boolean, 'contains_santan': boolean }"
cmd = [
"fm", "respond",
"--image", temp_path,
"--text", prompt
]
try:
# Execute native Apple binary
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
raw_output = result.stdout.strip()
#### [OTHER CODE]

And you can see from above that the application will extract the image and send the path along with the prompt to the fm command. From here, it will be processed, and a JSON-formatted output will be generated.

The table below compares fm serve and subprocess .

fm serve

subprocess

Process model

One long-running local server

One process per request

Integration

OpenAI SDK / HTTP

Direct CLI call from Python

Overhead

HTTP + stream parsing

Process spawn

Migration to cloud

Change base_url and model

Replace the call layer

Best fit

Code already written against OpenAI-compatible APIs

Local scripts and Streamlit prototypes on one Mac

Neither path is inherently better. I used subprocess for this demo because the interface and the CLI run on the same machine. For a codebase that already targets the OpenAI SDK, fm serve is the lower-friction path.

And now, here is the preview of my application that only uses Python and Apple's Foundation Model:

For more details on how I built this application, you can head to my ShipAI page called AI Food Police, which you can access through the link below.

AI Food Police

Tracking fried food from meal photos with the language model built into macOS

Irfan Alghani Khalid · 1 min read

Inside, you can see the walkthrough and the repository for interacting with the FM model. But make sure to have a Mac and already updated the device to the newest macOS 27!

Conclusion

For the past two years, the default pattern for AI features has been to wrap a cloud API, write a prompt, and pay per token. That pattern is still the right one for shared backends, long documents, and high-stakes factual accuracy.

Apple's fm CLI does not remove that pattern. It adds a cheaper, private step before it. During prototyping, prompts, JSON parsing, and UI flows can be iterated on locally with no token cost and no network round-trips. For native Apple applications, inference can stay on the user's device entirely.

The limits matter as much as the convenience. The model returned "Ipoh" for a factual question and classified grilled satay as fried. Both are expected from a compact on-device model, and both are reasons to keep grounding, validation, and a cloud fallback in the architecture.

For bootstrapped teams, the value is straightforward: more iterations before the first API bill. For larger teams, it is a local, private test bed that fits alongside the existing stack. In both cases, fm works best when its boundary is explicit.

If you have questions or want to reproduce the tests, feel free to reach out on LinkedIn.

References

[1] Apple, Build AI-powered scripts with the fm CLI and Python SDK (2026), https://developer.apple.com/videos/play/wwdc2026/334/