Qualitative Coding With AI: Inter-Rater Reliability Still Matters

Qualitative coding is the slow heart of interview research: read 40 transcripts, tag every meaningful passage, let themes emerge. LLMs can do a first pass in minutes. But 'a first pass' is not a finished analysis. Treating the first pass as the final product is how AI-assisted research gets rejected at peer review.

The defensible workflow

1. 1Have the LLM propose an initial codebook from 3-5 transcripts
2. 2Human researchers refine the codebook, collapsing redundant codes and naming them carefully
3. 3LLM codes all transcripts using the refined codebook
4. 4Humans independently re-code a random 15-20% sample
5. 5Calculate inter-rater reliability (Cohen's kappa) between human and LLM
6. 6If kappa < 0.7, revise the codebook and re-run

What to disclose in the methods section

• Which model did the coding (including version and date)
• The exact prompt template used for coding
• The codebook (as a supplementary appendix)
• The human-AI agreement statistics
• Any cases where humans overrode the LLM's codes

The big idea: AI accelerates qualitative coding — it does not replace the validation work. Kappa statistics and disclosures are non-negotiable.