llama.cpp: The Engine Underneath Almost Everything

llama.cpp is an open-source C/C++ implementation of LLM inference, originally written to run Meta's LLaMA models on a MacBook with no special hardware. It has since become the de facto inference engine for the local-model world: efficient on CPUs, well-tuned on Apple Silicon, with optional GPU offload via CUDA, ROCm, Metal, and Vulkan. If you are running a GGUF file anywhere on the planet, llama.cpp is probably involved.

Why this is worth your attention

• It is the layer where performance is actually decided — wrappers inherit its tuning
• Knowing its flags lets you wring 2-5x more throughput out of the same hardware
• It compiles cleanly on almost every platform — including embedded devices
• Its tools (llama-bench, llama-perplexity) are how you objectively compare quantizations

# Build and run llama.cpp directly
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && make

# Run a chat with a downloaded GGUF
./llama-cli -m models/llama-3.1-8b-instruct.Q5_K_M.gguf \
    -ngl 99 \
    -c 8192 \
    -p "Hello."

# Server mode — same OpenAI-compatible API
./llama-server -m models/llama-3.1-8b-instruct.Q5_K_M.gguf \
    -ngl 99 -c 8192 --port 8080

The flags that actually matter

1. 1-ngl N: number of layers to offload to GPU. More is faster until you run out of VRAM
2. 2-c N: context size in tokens — must be set high enough for your prompts but not wastefully
3. 3-b / -ub: batch and micro-batch sizes — affects throughput on long prompts
4. 4--threads N: CPU thread count — usually no benefit beyond physical cores
5. 5-fa: flash attention, when supported, often a free speedup

Apply this

• Build llama.cpp from source and run a GGUF you already have via Ollama
• Run llama-bench on the same model with two different -ngl values and compare tokens/second
• Read the README's Build section once — the optional features list is full of small wins

The big idea: every local-model tool you love is mostly llama.cpp underneath. Knowing the engine pays off the moment a wrapper's defaults stop being enough.

llama.cpp and GGUF explain why one model file can run across many consumer machines and local AI apps. In local AI, the model family is only one part of the system. The runtime, file format, serving path, hardware budget, evaluation set, and safety policy decide whether the model becomes useful.

Current source signal

Build the small version

Have students inspect a GGUF filename and decode family, size, quantization, and intended runtime before running it.

1. 1Define the user task in one sentence.
2. 2Choose the smallest model and runtime that might pass that task.
3. 3Run one happy-path prompt and one failure-path prompt.
4. 4Record speed, memory pressure, output quality, and the exact reason for any failure.
5. 5Write the operating rule you would give a non-expert user.

filename_decoder:
  Qwen3-8B-Instruct-Q4_K_M.gguf

  family: Qwen3
  size: 8B
  type: instruct
  quantization: Q4_K_M
  format: GGUF

question: who made this file and what template does it need?

The big idea: GGUF decoder. A local model app is not done when the model answers once; it is done when the whole workflow can be installed, measured, trusted, and recovered.