Emergence, Capability Forecasting, and Safety

The Forecasting Problem

Investors, regulators, and safety teams all want to know: what will the next model be able to do? If abilities emerge in jumps, the question is harder than it sounds. You cannot measure a capability that does not exist yet.

Two views of emergence

The Schaeffer et al. (2023) paper argued that many reported emergent abilities disappear when you use continuous metrics. But follow-up work by other labs found residual sudden jumps even after metric smoothing. The honest answer is: it depends on the task.

Capability evaluations

• Static benchmarks: MMLU, GPQA, HumanEval — broad but easy to saturate
• Task-specific suites: SWE-Bench for coding, AIME for math, long-context retrieval
• Agentic evals: METR tasks, CyBench, SWE-Lancer for multi-step work
• Capability evaluations: probing specific dangerous skills like weapon synthesis reasoning
• Scaling evals: same benchmark re-run at multiple model sizes to interpolate

Responsible scaling policies

Frontier labs like Anthropic, OpenAI, and Google DeepMind publish policies committing to specific evaluations at capability thresholds. Anthropic's Responsible Scaling Policy defines AI Safety Levels (ASL) and requires mitigations proportional to the risk tier.

Red teaming and dangerous capabilities

1. Pre-deployment evals target CBRN uplift, cyber offense, autonomous replication
2. External red teams get early model access under NDA
3. Pre-registered evals with hold-out data prevent training-time contamination
4. Findings are documented and, increasingly, published

Capability forecasting techniques

• Fit power laws at smaller scales and extrapolate (often overconfident)
• Use proxy tasks correlated with the capability of interest
• Run prediction markets internally to aggregate researcher beliefs
• Simulate agents in structured environments and measure horizon
• Study mechanistic signals — does a circuit for the capability exist before behavior is robust?

The scariest part of a capability evaluation is the tasks nobody remembered to include.

— A safety researcher

The big idea: emergence makes forecasting genuinely hard. Responsible labs publish policies, run structured evals, and accept that surprise is a load-bearing assumption of the field.