Qualitative Coding With AI: Inter-Rater Reliability Still Matters

Qualitative coding is the slow heart of interview research: read 40 transcripts, tag every meaningful passage, let themes emerge. LLMs can do a first pass in minutes. But 'a first pass' is not a finished analysis. Treating the first pass as the final product is how AI-assisted research gets rejected at peer review.

The defensible workflow

1. 1Have the LLM propose an initial codebook from 3-5 transcripts
2. 2Human researchers refine the codebook, collapsing redundant codes and naming them carefully
3. 3LLM codes all transcripts using the refined codebook
4. 4Humans independently re-code a random 15-20% sample
5. 5Calculate inter-rater reliability (Cohen's kappa) between human and LLM
6. 6If kappa < 0.7, revise the codebook and re-run

What to disclose in the methods section

• Which model did the coding (including version and date)
• The exact prompt template used for coding
• The codebook (as a supplementary appendix)
• The human-AI agreement statistics
• Any cases where humans overrode the LLM's codes

The big idea: AI accelerates qualitative coding — it does not replace the validation work. Kappa statistics and disclosures are non-negotiable.

AI-assisted qualitative coding can multiply researcher capacity; validity requires explicit validation methodology.

What AI does well here

• Use AI for initial open coding at scale
• Maintain researcher review on a sample for validation
• Calculate intercoder reliability between AI and human coders
• Document the AI methodology in publications for transparency

What AI cannot do

• Substitute AI for the interpretive insight researchers bring
• Replace validation with pure trust in AI output
• Generate genuine novel theme discovery

AI can suggest code-merge candidates and sub-theme groupings from coded transcripts to support iterative codebook refinement.

What AI does well here

• Cluster similar codes by definition and exemplar overlap.
• Generate intercoder-disagreement summaries to inform calibration.

What AI cannot do

• Decide the final code structure or theoretical framing.
• Replace researcher reflexivity in interpretive work.

Two human coders cost time you don't have; AI is a cheap second coder that catches different things than you do.

What AI does well here

• Apply your existing codebook to new transcripts
• Surface candidate themes you didn't include
• Flag where the same passage could fit two codes
• Suggest where a theme might need splitting

What AI cannot do

• Replace a human collaborator's interpretive depth
• Notice cultural cues without explicit framing