AI Tool Modal for Distributed Evaluation: Drafting a Fan-Out Job

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-tools-modal-distributed-eval-r9a4-creators

1. In a Modal distributed evaluation setup, what is the primary purpose of a fan-out function?

   1. To validate that the AI generated correct code
   2. To retry failed test cases automatically
   3. To split test cases across multiple containers for parallel processing
   4. To combine all test results into a single summary value

2. Why must a human operator set a cost ceiling for an AI-generated Modal evaluation job?

   1. Modal requires payment upfront before any computation
   2. AI doesn't know the operator's financial constraints or budget priorities
   3. The cost ceiling prevents the job from running at all
   4. Modal charges a flat fee regardless of usage

3. Which of the following can AI automatically generate when scaffolding a Modal evaluation job?

   1. The operator's maximum budget
   2. The final evaluation scores for each test case
   3. A fan-out function with batching and concurrency controls
   4. Which transient failures should count as case failures

4. What does it mean for result aggregation to 'preserve per-case detail'?

   1. Results are sorted by cost before being displayed
   2. Individual test case results and metadata are retained in the final output
   3. All results are combined into a single average score
   4. The aggregation function fails if any single case fails

5. What specific risk arises from not setting a hard concurrency cap in a Modal fan-out job?

   1. Too many containers could launch simultaneously, causing unexpected costs
   2. Results will be aggregated incorrectly
   3. The job will fail due to syntax errors
   4. The AI will refuse to generate the code

6. Which decision about transient failures must be made by the human operator rather than the AI?

   1. How to implement exponential backoff
   2. How many retries to attempt
   3. Which types of failures should mark a test case as failed
   4. Whether to log failure details

7. Why do AI-generated Modal evaluation jobs have the potential to scale costs unexpectedly fast?

   1. AI optimizes for throughput and parallelism without inherent cost constraints
   2. The AI generates inefficient code that wastes compute
   3. Modal automatically charges premium rates for AI-generated code
   4. Modal charges per line of code generated

8. What is the purpose of setting a maximum runtime limit on a Modal evaluation job?

   1. To ensure all test cases pass
   2. To prevent runaway costs from infinitely running containers
   3. To make the job run faster
   4. To generate the result aggregation code

9. In the context of Modal distributed evaluation, what is 'batching' in a fan-out function?

   1. Grouping test cases into chunks to send to each container
   2. Validating that each test case has correct syntax
   3. Retry logic for failed test cases
   4. Combining results after all tests complete

10. When scaffolding a Modal eval job, which of these components would you expect an AI to produce?

    1. A complete budget spreadsheet for the project
    2. An app definition, eval function, and fan-out call
    3. A decision about whether the job is worth running
    4. The final grades for each student submission

11. What distinguishes distributed evaluation from running tests sequentially on a single machine?

    1. Distributed evaluation always produces correct results
    2. The results are automatically formatted for presentation
    3. Distributed evaluation requires no code to be written
    4. Test cases are spread across multiple compute resources running in parallel

12. Why is it important to retain individual test case details when aggregating results from a distributed evaluation?

    1. To make the job run faster
    2. To allow investigation of specific failures and debugging
    3. Because Modal requires it by default
    4. So the summary can be longer

13. What would happen if retry logic is not specified for a Modal job handling transient failures?

    1. A transient failure would immediately mark the test case as failed
    2. The AI would automatically add retries
    3. Results would be more accurate
    4. The job would run infinitely

14. What does the term 'distributed evaluation' refer to in the context of Modal?

    1. Evaluating AI systems located in different countries
    2. A debugging technique for finding bugs
    3. A method for grading student assignments
    4. Running test cases across multiple containers simultaneously

15. Which of these is NOT something an AI can determine when scaffolding a Modal evaluation job?

    1. What batching strategy to use
    2. Whether $50 or $500 is an appropriate cost ceiling
    3. How to implement the fan-out logic
    4. How to handle container timeouts