Small Language Models (SLMs)
Small language models (SLMs) are language models compact enough for phones, laptops, and cheap serving, usually with under 10 billion parameters.
A small language model (SLM) is a language model compact enough to run on a phone, laptop, or single modest GPU, or to serve at high volume for little cost. There is no fixed cutoff: NVIDIA researchers define an SLM as one that fits on a common consumer device and responds fast enough to serve one user's requests, and as of 2025 they count most models under 10 billion parameters. Microsoft's Phi-3-mini, a 3.8-billion-parameter model released in April 2024, scored 69% on the MMLU benchmark, comparable to GPT-3.5, while small enough to run on a phone. Other widely used SLMs include Google's Gemma 3 models from 270 million to 4 billion parameters, Meta's Llama 3.2 1B and 3B, Alibaba's smaller Qwen3 models, and the roughly 3-billion-parameter on-device model behind Apple Intelligence.
How SLMs Are Built#
SLMs use the same transformer architecture as large language models (LLMs) and make up for size with training choices:
- Data quality: Microsoft trained Phi-3-mini on 3.3 trillion tokens of heavily filtered public web text and synthetic data, and credits that dataset for its results.
- Overtraining: Scaling laws call for about 20 tokens per parameter to minimize training compute, while SLMs train on hundreds or thousands, trading extra training compute for a smaller model that costs less per query. Meta trained Llama 3 8B on 15 trillion tokens.
- Knowledge distillation: Google trained Gemma 2's 2B and 9B models to match the next-token distributions of a larger teacher in place of plain next-token prediction.
- Model quantization: A 3.8-billion-parameter model needs about 7.6 GB for its weights at 16 bits and about 1.9 GB at 4 bits. Apple compresses its on-device model to 2 bits per weight with quantization-aware training (QAT), and GGUF files package 4-bit models for laptops.
Where SLMs Run#
On-device assistants are the most visible use. Apple's Foundation Models framework, announced in 2025, gives app developers direct access to the on-device model that powers Apple Intelligence, and phone chips run such models on their neural processing units (NPUs) without sending text to a server, which helps with data privacy. Companies fine-tune SLMs with LoRA for narrow jobs such as classification, extraction, and support replies, often adding retrieval-augmented generation (RAG) for company knowledge, and Google built Gemma 3 270M for this kind of task-specific tuning, with 170 million of its 270 million parameters in the embeddings of a 256K-token vocabulary.
AI agents are the newest use. The NVIDIA paper argues that most agent subtasks are narrow and repetitive enough for an SLM, with a larger model reserved for the rest, a form of model routing:
The same split applies to vision pipelines on edge AI hardware. The Ultralytics LLM interface accepts a base_url for any OpenAI-compatible endpoint, including local servers such as Ollama, so a YOLO26 pipeline on an edge device can send a frame or its detections to an SLM on the same machine and get a text report without a cloud call.
SLMs vs. Related Terms#
- Large language model: The same kind of model at tens to hundreds of billions of parameters or more, with broader knowledge and stronger multi-step reasoning, and usually served from data-center GPUs.
- Vision-language model (VLM): Adds an image encoder to a language model. Small VLMs, such as Apple's FastVLM, apply SLM techniques to image understanding. The VLM Leaderboard includes small VLMs such as Gemma 4 E2B and Qwen3.5 2B, measuring how accurately they detect objects next to much larger models.
- TinyML: Machine learning on microcontrollers with kilobytes to a few megabytes of memory, where even a 270-million-parameter SLM, at about 270 MB in 8-bit weights, does not fit.
