Long-Context Code Understanding — The 1M-Token Era

Claude, Gemini, and GPT now offer 1M+ token context windows for coding workloads. That's roughly 750,000 words, or most mid-sized repos. When the whole codebase fits in one prompt, architectural questions, cross-file refactors, and full-repo audits become viable in a single shot.

What you can now do in one pass

• Read an entire small-to-medium repo into context
• Ask architectural questions that cross many files
• Refactor a shared type across 50 call sites at once
• Summarize every change between two git tags
• Audit for duplicated logic repo-wide

Long context is not free context

Just because you can paste a million tokens doesn't mean you should. Cost scales linearly with input tokens, and attention quality degrades unevenly across the window. The middle of a long context tends to be the worst-attended region — a phenomenon called the lost-in-the-middle effect.

Prompt caching is the game-changer

Both Anthropic and OpenAI support prompt caching. Once a long context is cached, subsequent calls reuse it at a fraction of the input cost, typically around 10 percent. This makes long-context workflows economically sane. Structure prompts so the stable parts (codebase, docs, instructions) come first — they cache. The variable part (the question) comes last.

# Anthropic prompt caching example
response = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=2048,
    system=[
        {
            "type": "text",
            "text": "You are a senior engineer reviewing a codebase."
        },
        {
            "type": "text",
            "text": open("whole_repo.txt").read(),  # 800k tokens
            "cache_control": {"type": "ephemeral"}  # CACHE THIS
        }
    ],
    messages=[{
        "role": "user",
        "content": "What's the purpose of utils/parser.ts?"
    }]
)
# First call: full price for the 800k tokens.
# Every subsequent call in the next 5 min: ~10% of that cost.

Design patterns that matter

1. 1Put stable content first (repo, docs, instructions), variable content last
2. 2Use explicit section markers (file path tags) — models attend to structure
3. 3Ask the model to cite which file/line its answer came from — keeps it grounded
4. 4Chunk very large repos by concern, not by file order
5. 5Always measure: run the same query with and without long context and compare

Needle-in-haystack: the eval to know

The needle-in-haystack eval plants a specific fact in a long context and asks the model to retrieve it. Frontier models score near-perfectly on simple versions, but real-world performance on complex questions across long context is noticeably worse. Test your actual workflow before trusting reported benchmarks.

“Long context is a better memory, not a better brain.”

The big idea: 1M-token context opens whole-repo reasoning, but only if you pair it with caching, structure, and citations. Long context is a superpower priced by the token — spend it deliberately.