Shannon and the Birth of Information

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-history-shannon-information-creators

1. According to Shannon's definition, what does information measure?

   1. The amount of data transmitted
   2. The reduction of uncertainty
   3. The cost of transmitting data
   4. The physical length of a message

2. In Shannon's framework, what is a 'bit'?

   1. A type of error-correcting code
   2. A binary digit representing either 0 or 1
   3. The fundamental unit of information, representing a choice between two equally likely options
   4. A measurement of data storage capacity

3. In information theory, what does 'entropy' represent?

   1. The average uncertainty or unpredictability of information source output
   2. The total amount of data in a message
   3. The speed at which data can be transmitted
   4. The physical entropy of the communication device

4. What does 'channel capacity' represent in Shannon's theory?

   1. The cost of building a communication channel
   2. The maximum rate at which information can be transmitted without error
   3. The physical length of the communication channel
   4. The number of devices connected to a network

5. What happens when you attempt to transmit information faster than a channel's capacity?

   1. The message is automatically compressed
   2. The transmission takes longer to arrive
   3. The channel expands to accommodate the data
   4. Errors become unavoidable no matter what coding is used

6. How does redundancy in a message help with communication?

   1. It allows the receiver to reconstruct the original message despite errors
   2. It makes the message longer and harder to intercept
   3. It increases the information content of the message
   4. It reduces the need for error-checking

7. What is the theoretical limit on lossless compression?

   1. The bandwidth available
   2. The entropy of the source
   3. 50% of the original file size
   4. The channel capacity

8. What was 'Theseus', created by Claude Shannon?

   1. A communication protocol for telephone networks
   2. A mechanical mouse that could learn to solve mazes
   3. A mathematical formula for calculating entropy
   4. A type of error-correcting code

9. What did Shannon prove about the relationship between compression and entropy?

   1. Compression is possible up to the entropy limit, but not beyond it
   2. Compression can eliminate entropy entirely
   3. Entropy increases when messages are compressed
   4. Compression has no theoretical limits

10. Why are modern language models described as 'entropy engines'?

    1. They generate random text without structure
    2. They predict the next token by estimating probability distributions over words
    3. They compress text files to reduce storage needs
    4. They measure the entropy of input data

11. What does 'cross-entropy' measure in machine learning?

    1. The total amount of data processed by a neural network
    2. The physical entropy of computer hardware
    3. The uncertainty of the training data
    4. The difference between predicted probability distributions and true distributions

12. What was historically significant about Shannon's 1948 paper?

    1. It described the first neural network architecture
    2. It proved that information could be measured, encoded, and transmitted with mathematical limits
    3. It introduced the first computer programming language
    4. It invented the transistor

13. What is the relationship between entropy and uncertainty in information theory?

    1. Entropy eliminates uncertainty
    2. They are unrelated concepts
    3. Higher entropy means higher uncertainty
    4. They are inversely proportional

14. What did Shannon's work provide to artificial intelligence researchers?

    1. The hardware for building AI systems
    2. The vocabulary and mathematics to pose problems about probability and prediction
    3. The first neural network architecture
    4. The first AI application

15. How does a noisy channel affect information transmission according to Shannon?

    1. It introduces uncertainty that must be accounted for in the channel capacity
    2. It reduces the entropy of the message
    3. It automatically compresses the data
    4. It makes transmission impossible