The Full Machine Learning Pipeline

Nearly every production ML system, from spam filters to LLMs, flows through the same skeleton. Knowing the skeleton lets you place any new paper or product into context quickly.

The stages

1. 1Data collection
2. 2Data cleaning and labeling
3. 3Feature engineering or tokenization
4. 4Train/validation/test split
5. 5Model architecture selection
6. 6Training with a loss function and optimizer
7. 7Evaluation on held-out data
8. 8Fine-tuning or post-training alignment
9. 9Deployment to inference infrastructure
10. 10Monitoring, feedback collection, and retraining

Where time actually goes

Newcomers assume training is the main event. In practice, data prep and evaluation consume 70 to 90 percent of engineering time. Training is usually a well-defined, scriptable step. Cleaning messy real-world data is not.

Concrete example: fine-tuning for a legal use case

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments

dataset = load_dataset("legal_memos", split="train")
tok = AutoTokenizer.from_pretrained("base-model")
model = AutoModelForCausalLM.from_pretrained("base-model")

def preprocess(ex):
    return tok(ex["text"], truncation=True, max_length=2048)

ds = dataset.map(preprocess, batched=True)

args = TrainingArguments(
    output_dir="./legal-tuned",
    per_device_train_batch_size=4,
    num_train_epochs=3,
    learning_rate=2e-5,
    evaluation_strategy="epoch",
)

trainer = Trainer(model=model, args=args, train_dataset=ds)
trainer.train()

Train vs. inference infrastructure

• Training: massive parallel GPUs, weeks of runtime, measured in FLOPs
• Inference: latency-sensitive, often needs quantization or distillation
• Fine-tuning: smaller GPUs, hours to days, often with LoRA adapters
• Monitoring: logging, drift detection, A/B tests in production

Common pipeline failures

• Train/serve skew: features built differently in training vs. production
• Data leakage: test data accidentally appears in training
• Distribution drift: real inputs change over time, model decays
• Unmonitored bias: groups see worse outcomes and nobody notices

“Production ML is 5 percent machine learning and 95 percent engineering.”

The big idea: the ML pipeline is the real substrate of AI products. Papers describe stages 5 and 6. Careers are built on stages 1, 2, 7, 9, and 10.