Measure surprise, uncertainty, and prediction mistakes.
Information theory studies how much we learn when something happens. AI uses it to measure uncertainty, choose useful questions, compress data, and improve predictions.
Free preview — no account needed. This is the Basic level of a Mathematical Foundations lesson from the AI Professional Series. The Intermediate and Advanced levels, with 20 more practice exercises and a Concept Quiz, are part of AI Professional. Optional practice only: it does not affect course totals, exams or certificates.
See AI Professional plansThis is the Basic level. There are 10 practice exercises, each with a hint and a full worked solution.
Start here if probabilities and logarithms are new. Learn to measure surprise using bits.
An event is something that can happen. Its probability is its chance of happening, written from 0 to 1. Information measures how surprising an observed event is under those chances. A less likely event gives more information when it happens.
In everyday words: A delivery arriving on its usual day tells you less than a delivery arriving a week early.
Worked example: A bag has 8 equally likely tickets. Seven are blue and one is gold. Which color is more surprising?
Answer: Gold is more surprising and gives more information.
Why it matters in AI: AI can flag an unusual event when its predicted probability is small. A surprising event is not automatically an error.
A bit is a unit of information. One fair yes/no choice has two equally likely answers and gives one bit. A sequence of balanced questions can identify one item from a group. Balanced means that each question divides the remaining possibilities into equal groups.
In everyday words: Finding a name by repeatedly choosing between two equal halves of a list is like following a trail of bits.
Number of equally likely possibilities = 2ⁿRead it as: With n balanced yes/no answers, we can distinguish two multiplied by itself n times possibilities.
Worked example: How many balanced yes/no questions identify one item among 8 equally likely items?
Answer: 3 balanced yes/no questions identify the item, giving 3 bits.
Why it matters in AI: Computers store data using two possible values per bit. Information theory explains the connection between choices and storage.
An exponent tells us how many times to multiply a number by itself. For example, 2³ means 2 × 2 × 2. A logarithm asks for the exponent instead of the result. The base in log2 is 2, so log2 asks how many doublings produce a number.
In everyday words: If doubling is climbing a staircase, log2 tells you how many steps you climbed.
log2 8 = 3 because 2³ = 2 × 2 × 2 = 8Read it as: The base-2 logarithm of eight is three because two multiplied by itself three times equals eight.
Worked example: Find log2 16.
Answer: log2 16 = 4.
from math import log2
print(log2(8)) # 3.0
print(log2(16)) # 4.0Why it matters in AI: Logarithms turn counts of possibilities into information measured in bits. They also turn multiplication into addition.
Let p mean the probability assigned to an event. Its information is the negative base-2 logarithm of p. Negative means reversing the sign, so a negative logarithm becomes positive. A certain event gives zero information; an event assigned probability zero has no finite information value if it occurs.
In everyday words: A rare winning ticket teaches you more than an ordinary losing ticket.
I(p) = −log2 p = log2(1/p)Read it as: The information of an event is minus the base-2 logarithm of its probability, measured in bits.
Worked example: How much information comes from an event with probability 1/8?
Answer: The event gives 3 bits of information.
Why it matters in AI: AI uses the same idea to measure how surprising an observed answer was. Giving the correct answer a very small probability creates a large penalty.
Try each one on paper first. Use the hint if you get stuck, then check the full solution.
Exercise 1. Which event gives more information when observed: one with probability 1/2 or one with probability 1/8?
Exercise 2. How many equally likely possibilities can one balanced yes/no answer distinguish?
Exercise 3. How many equally likely possibilities can three balanced yes/no answers distinguish?
Exercise 4. Calculate log2 4.
Exercise 5. Calculate log2 32.
Exercise 6. How many bits does an event with probability 1 give?
Exercise 7. How many bits does an event with probability 1/2 give?
Exercise 8. How many bits does an event with probability 1/8 give?
Exercise 9. You identify one card among 16 equally likely cards using balanced yes/no questions. How many questions are needed?
Exercise 10. Two events are independent, meaning one does not change the other's probability. Their probabilities are 1/2 and 1/4. Find the probability and information of both happening.