Using AI to Run Cognitive Pretests on Survey Items

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-research-ai-survey-cognitive-pretest-creators

1. Which of the following is a task that AI is specifically noted to do well when pretesting survey items?

   1. Identify the exact statistical margin of error
   2. Determine if respondents will enjoy taking the survey
   3. Generate complete survey instruments from scratch
   4. Surface ambiguity in wording

2. A researcher wants to use AI to improve a survey before distributing it. What is the primary benefit of using AI in this pretesting phase?

   1. AI can simulate how target population members might interpret items
   2. AI can interview actual respondents face-to-face
   3. AI can guarantee the survey will receive a 100% response rate
   4. AI can translate the survey into any language instantly

3. What limitation of AI in survey pretesting is most relevant when working with a culturally diverse population?

   1. AI cannot detect statistical outliers in responses
   2. AI cannot generate enough question variants to test
   3. AI cannot identify biased question ordering
   4. AI cannot detect cultural unfamiliarity with terms or concepts

4. A student claims that using AI to pretest survey items means they no longer need to conduct any human interviews. How does the lesson characterize this claim?

   1. It is partially correct only if the survey has fewer than 10 items
   2. It is incorrect because AI pretest supplements but does not replace cognitive interviews
   3. It is correct because AI can fully simulate human thought
   4. It is irrelevant to the pretesting process

5. Which survey item issue can AI help identify during the pretesting phase?

   1. Questions that are double-barreled (asking two things at once)
   2. Questions that will make respondents laugh
   3. Questions that match the exact length of previous successful surveys
   4. Questions that are guaranteed to receive honest answers

6. A researcher types: 'Cognitively probe these survey items as a urban teenagers: [items]. Flag confusion, ambiguity, and double-barreled wording.' This represents what aspect of the lesson?

   1. A method to replace all human survey administration
   2. An example of how to instruct AI for survey pretesting
   3. A technique to automatically field the survey to respondents
   4. A prompt that will likely produce invalid results

7. A survey asks: 'Do you agree that the new park is both beautiful and provides good recreation facilities?' This item exemplifies which problem AI should flag?

   1. Statistical bias in sampling
   2. Insufficient response options
   3. Double-barreled wording
   4. Cultural unfamiliarity

8. When using AI to pretest survey items, what does it mean to 'surface ambiguity'?

   1. To remove all technical vocabulary from questions
   2. To make vague questions more detailed
   3. To identify questions that could be interpreted in multiple ways
   4. To calculate the exact number of words in each question

9. A research team plans to use AI pretesting results as the sole basis for finalizing their survey. Based on the lesson, what is the concern with this approach?

   1. AI cannot replace human cognitive interviews with actual population members
   2. AI results are always statistically invalid
   3. The survey will automatically become too long
   4. The team might violate copyright laws

10. Which term describes the process of asking respondents to think aloud about how they interpret survey questions?

    1. Random sampling
    2. Statistical analysis
    3. Field testing
    4. Cognitive interview

11. An AI system flags a survey item as potentially confusing. What should a researcher do next, according to the workflow suggested by the lesson?

    1. Immediately remove the question without further review
    2. Ignore the flag since AI is not always accurate
    3. Report the AI system as faulty
    4. Use the flag as a guide to revise and then test with real humans

12. What is the primary purpose of pretesting a survey before full distribution?

    1. To identify and fix problems with questions
    2. To collect data for the actual study
    3. To select which respondents should receive incentives
    4. To determine how many respondents will participate

13. A researcher tells AI: 'Pretest this survey for middle school students.' Why might this instruction be insufficient based on the lesson?

    1. Middle school students do not need pretesting
    2. The instruction lacks specificity about what problems to look for
    3. AI cannot work with surveys about education
    4. AI will automatically translate the survey into Spanish

14. Which of the following best describes the relationship between AI pretesting and human cognitive interviews?

    1. Human interviews are now obsolete due to AI
    2. They are competing methods where one must be chosen
    3. They measure completely different things
    4. They are complementary methods that can be used together

15. A survey item reads: 'How often do you exercise and eat healthy food?' An AI pretest system should flag this as what type of problem?

    1. Double-barreled wording
    2. Too personal of a question
    3. Culturally inappropriate
    4. Missing response options