AI third-party model evaluation rubric for procurement teams

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-ethics-ai-third-party-model-evaluation-rubric-creators

1. What is the primary purpose of an AI evaluation rubric designed for procurement teams comparing third-party models?

   1. To let AI systems automatically select vendors based on scores
   2. To replace the need for security reviews entirely
   3. To compare third-party models consistently using defined criteria
   4. To eliminate vendor interviews from the procurement process

2. Which of the following tasks can AI appropriately perform when helping build an evaluation rubric?

   1. Translate organizational principles into observable criteria
   2. Score vendors based on submitted evidence
   3. Approve a vendor for organizational use
   4. Conduct interviews with vendor representatives

3. When procurement teams use the evaluation rubric to assess vendors, what must they provide for each criterion?

   1. Automatic approval decisions
   2. Predetermined vendor rankings
   3. Actual evidence from model cards, audits, or contracts
   4. AI-generated recommendations

4. In which category would a criterion about 'documented bias testing results' most appropriately belong?

   1. Safety
   2. Fairness
   3. Transparency
   4. Operational

5. Why cannot AI fully replace vendor interviews even when using a comprehensive evaluation rubric?

   1. Vendor interviews are legally prohibited
   2. AI systems cannot read any documents
   3. AI has already conducted all necessary interviews
   4. Interviews reveal qualitative insights that documents alone cannot provide

6. What role does a model card play in the evaluation rubric process?

   1. It automatically generates scores for all criteria
   2. It serves as the final approval document for vendor selection
   3. It provides evidence for evaluating specific criteria
   4. It replaces the need for human review

7. Which group or groups are responsible for actually scoring vendors against the rubric criteria?

   1. Only senior leadership
   2. Procurement teams, security teams, and the responsible AI team
   3. Only the AI system that created the rubric
   4. Only external third-party auditors

8. What does a 1-5 scoring scale represent on the evaluation rubric?

   1. The age of the AI model being evaluated
   2. The price tier of the vendor's services
   3. The degree of compliance or achievement for each criterion
   4. The number of employees at the vendor company

9. Which of the following would best represent an 'operational' category criterion?

   1. Whether the model can explain its reasoning process
   2. Whether the model produces toxic or harmful content
   3. Whether the model demonstrates demographic parity across outputs
   4. Whether the vendor provides uptime guarantees and support SLAs

10. What is a fundamental limitation of using AI when building and applying the evaluation rubric?

    1. AI cannot understand organizational principles
    2. AI cannot suggest evidence sources for criteria
    3. AI cannot independently score vendors or approve vendors for use
    4. AI cannot format rubrics into spreadsheet layouts

11. When translating organizational responsible AI principles into rubric criteria, what characteristics should each criterion have?

    1. They should focus exclusively on cost factors
    2. They should be vague to allow flexibility
    3. They should be observable and tied to specific evidence sources
    4. They should be based solely on vendor marketing claims

12. Why is it important for each rubric criterion to have identified evidence sources?

    1. To make the rubric document longer and more impressive
    2. To eliminate the need for any human judgment
    3. To ensure scoring is based on verifiable information rather than assumptions
    4. To automatically generate final vendor rankings

13. What is the key advantage of using a structured rubric approach over ad-hoc vendor evaluation?

    1. It uses AI to make all final decisions
    2. It removes security teams from the evaluation process
    3. It eliminates the need for any documentation
    4. It enables consistent criteria for comparing different vendors

14. Which of the following would NOT be an appropriate task for AI in the rubric process?

    1. Translating principles into observable criteria
    2. Formatting the rubric for spreadsheet-style scoring
    3. Suggesting evidence sources for each criterion
    4. Approving a vendor based on their rubric scores

15. Which of the following best describes an appropriate 'transparency' category criterion?

    1. Whether the vendor guarantees 99.9% uptime
    2. Whether the model has been tested for demographic bias
    3. Whether the model filters harmful content
    4. Whether the model provides explanations for its outputs