AI and Survey Instrument Drafts: Question-Stem Generation

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-research-AI-survey-instrument-draft-r12a3-creators

1. An AI system is asked to generate a 30-item questionnaire about employee engagement. What can the AI reliably produce on its own?

   1. Validated constructs that accurately measure the intended psychological traits
   2. Consistent question stems and answer choices using standard formats like Likert scales
   3. Finalized survey data ready for statistical analysis without further review
   4. Complete cognitive interview results showing how respondents interpret each question

2. A researcher uses AI to draft a survey, then immediately distributes it to 5,000 participants without any review. Which fundamental limitation of AI-generated surveys is most likely to cause problems?

   1. AI cannot determine whether respondents will understand questions as intended
   2. AI will refuse to create surveys with more than 20 items
   3. AI cannot generate enough questions to reach statistical significance
   4. AI automatically excludes biased question wordings from drafts

3. What is the primary purpose of conducting cognitive interviews after an AI drafts a survey instrument?

   1. To increase the sample size for preliminary data collection
   2. To check whether participants understand questions the way the researcher intended
   3. To translate the survey into additional languages
   4. To train the AI model to write better future surveys

4. Which of the following is NOT a capability of AI when first drafting a survey instrument?

   1. Testing whether questions have acceptable reliability across different groups
   2. Generating consistent question stems and answer choices
   3. Suggesting standard Likert scale response options
   4. Drafting demographic and consent sections

5. A student asks an AI to 'draft a survey about exercise habits' but doesn't specify anything else. What is the most significant risk in using this AI-generated survey without modification?

   1. The AI will refuse to generate questions without more detailed prompts
   2. The survey will automatically be biased toward certain exercise types
   3. The questions may measure constructs different from what the researcher actually wants to study
   4. The AI will produce questions that are too simple for adult respondents

6. What does construct validity refer to in survey research?

   1. Whether a survey question actually measures the theoretical concept it claims to measure
   2. The number of response options provided for each question
   3. The consistency of responses when the same survey is taken multiple times
   4. The way questions are organized into logical sections

7. A researcher notices that different respondents interpret an AI-generated question about 'work-life balance' in very different ways—some think about hours worked, others think about emotional stress, and others think about time with family. What should the researcher have done before fielding this survey?

   1. Submitted the survey to an ethics board for additional review
   2. Asked the AI to add more response options to that question
   3. Deleted that question and replaced it with a true/false question
   4. Conducted cognitive interviews with a small sample to test interpretation

8. What is reliability in the context of survey measurement?

   1. The consistency of scores when the same people take the survey multiple times
   2. The degree to which a survey measures what it claims to measure
   3. The use of multiple choice questions instead of open-ended ones
   4. The process of selecting a representative sample

9. A methodologist is reviewing an AI-generated survey about job satisfaction. They find that one question uses a 7-point scale while all others use a 5-point scale. What issue does this represent?

   1. Inconsistent formatting that could confuse respondents and reduce data quality
   2. A deliberate choice to increase sensitivity for that particular question
   3. An opportunity to test which scale length produces better responses
   4. A fundamental measurement error that invalidates the entire survey

10. Why are standard scale types like Likert, frequency, and agreement scales particularly suited for AI to generate?

    1. They require deep understanding of human psychology to design properly
    2. They follow predictable templates with clear structural rules that AI can apply consistently
    3. They must be custom-created for each research study to maintain originality
    4. They have been proven to produce unreliable data when generated by any automated system

11. What are the essential components that a complete survey instrument should include, beyond the main topical questions?

    1. Informed consent information and demographic questions
    2. Statistical formulas for calculating results
    3. Funding source disclosures and conflict of interest statements
    4. Interviewer instructions for in-person administration

12. A researcher wants to measure 'math anxiety' using a survey. They use an AI to generate 20 questions, but the questions actually measure 'dislike of math classes.' What type of problem does this illustrate?

    1. Formatting error—the questions used the wrong scale type
    2. Reliability problem—the questions produce inconsistent results
    3. Sampling bias—too many students were included in the pilot
    4. Construct validity issue—the questions measure a related but different concept

13. What is the primary reason that cognitive interviewing is considered irreplaceable in survey development, despite AI's ability to draft questions quickly?

    1. It automatically corrects any biased language in the drafted questions
    2. It reduces the cost of survey development by eliminating the need for researchers
    3. It reveals how actual respondents mentally process and interpret each question
    4. It generates statistical reports that can be submitted directly to journals

14. Which of the following best explains why AI-generated surveys must still go through a piloting phase?

    1. Piloting is required by law for all academic research surveys
    2. AI-generated surveys are typically too long to be reviewed manually
    3. AI always includes biased questions that must be removed before use
    4. AI cannot verify that the survey will work as intended when administered to real respondents

15. A researcher gives an AI this prompt: 'Draft a 25-item survey on remote-work satisfaction with a 5-point Likert scale, plus consent and demographics blocks.' The AI produces the survey. What should happen next?

    1. The researcher should review, validate constructs, and conduct cognitive interviews before fielding
    2. The survey should be sent immediately to participants to gather data quickly
    3. The researcher should ask the AI to add 10 more questions for statistical power
    4. The survey should be posted online for anyone to complete without screening