Model Extraction and Distillation Attacks

Imagine a closed API: you send prompts, you get outputs. The model's weights are secret. Can you reconstruct enough of it to build a useful clone? This is model extraction, and the answer for many real scenarios is yes, partially, and for a price that is often surprisingly low.

The basic attack

1. 1Collect a diverse set of input prompts (use a large language corpus).
2. 2Query the target model; log (prompt, output) pairs.
3. 3Train a student model on those pairs via standard fine-tuning or knowledge distillation.
4. 4The student approximates the teacher's behavior on the queried distribution.
5. 5If the student matches well enough on downstream tasks, the attack succeeded.

Real results

• Tramèr et al. 2016: stole decision-tree-equivalents of ML-as-a-service models for under $10
• Orekondy et al. 2019: Knockoff Nets extracted image classifiers from commercial APIs
• Krishna et al. 2020: extracted BERT-class models for natural language
• Carlini, Jagielski et al. 2024: Stealing Part of a Production Language Model recovered the last-layer embedding matrix of several OpenAI and Google models via API queries
• Various 2023-2024 papers: fine-tuning smaller open models on outputs of GPT-4 produced surprisingly capable students

What extraction costs

Why this matters legally and strategically

• Distillation may violate API terms of service (most now explicitly prohibit training competing models on outputs)
• Open-model landscape includes many models whose quality traces to GPT-4-class teachers
• Export control on a closed model is limited by extractability
• Defensive watermarking of outputs can help prove a downstream model was distilled
• Lab competition dynamics shift when the leader can be approximated cheaply

Defenses

• Rate limiting: caps on queries per account
• Monitoring for distributional probing patterns
• Output perturbation: add small noise to logits at the cost of quality
• Watermarking: embed a signal in outputs that a distilled model would inherit
• Legal: terms of service, lawsuits (OpenAI has pursued several alleged distillation cases)

“When a model is exposed via API, you should assume that a lower-fidelity copy of it exists somewhere you do not control.”

The big idea: APIs leak. Extraction is not always cheap, but for most intermediate-quality models it is cheaper than people think. The field is adjusting, and so are the contracts, but the underlying game is adversarial and ongoing.