Bayesian Inference: Alice’s Coin, Likelihood, MLE, and MAP

One hidden coin connects conditional independence, Bayesian updating, prediction, Naive Bayes, and maximum-likelihood estimation.

The central distinction is between how probable an observation is under a hypothesis and how probable the hypothesis is after an observation. Alice’s coin example makes that distinction concrete. It also separates updating a belief, predicting an outcome, fitting a parameter, and choosing a class.

Alice picks one coin, once

Alice has a fair coin and a loaded coin whose probability of heads is $𝑝$, where $\frac{1}{2}<𝑝\le 1$. She chooses each coin with probability $\frac{1}{2}$, then flips that same coin twice. Define the events:

• $𝐹$: Alice picked the fair coin.
• $𝐿$: Alice picked the loaded coin.
• $𝐴$: the first flip is heads.
• $𝐵$: the second flip is heads.

Once we know which coin she picked, the probability of the second head is straightforward:

The first and second flips are independent given the coin’s identity. For example, $𝑃\left(𝐵\mid 𝐴\cap 𝐿\right)=𝑝$: learning that the first flip was heads does not change the loaded coin’s bias.

Why the first and second flips have the same marginal probability

If we do not know which coin Alice picked and have not observed either flip, we average over the two possible coin choices:

Exactly the same reasoning applies to the second flip:

Thus $𝑃\left(𝐴\right)=𝑃\left(𝐵\right)$. This equality is what “by symmetry” means here: before observing results, neither flip position has an advantage.

We are averaging over the single choice made at the start. This does not mean Alice chooses another coin before the second flip. If we know the selected coin, we use the corresponding conditional probability instead.

Equal probabilities do not mean independent outcomes

For two heads, each possible coin produces a different probability:

Average over the original coin choice again:

For $𝑝=0.9$,

but $𝑃\left(𝐴\right)𝑃\left(𝐵\right)=0.49$. Because $0.53\ne 0.49$, the two events are not independent when the coin is unknown.

There is one shared hidden choice behind both flips. Seeing a head gives us information about that choice, which helps us predict the other flip. The coin does not change; our information changes.

Prior, likelihood, evidence, and posterior

Suppose the first flip is heads. We now want to know which coin Alice picked. Bayes’ theorem reverses the direction of conditioning:

The four quantities have different jobs:

• Prior: $𝑃\left(𝐿\right)$ is the probability assigned to the loaded-coin hypothesis before using the first flip’s result.
• Likelihood: $𝑃\left(𝐴\mid 𝐿\right)$ is the probability of the observed head if the loaded-coin hypothesis is true.
• Evidence probability: $𝑃\left(𝐴\right)$ is the overall probability of a head, averaging over the possible coins. It normalizes the posterior.
• Posterior: $𝑃\left(𝐿\mid 𝐴\right)$ is the probability assigned to the loaded-coin hypothesis after using the observed head.

A prior is not necessarily an uninformed guess. Here it comes from knowing that Alice chooses each coin with equal probability. “Prior” means before considering this evidence, not before knowing anything at all.

For $𝑝=0.9$,

A head raises our probability for the loaded coin from $50\%$ to about $64.3\%$, because a head is more likely with that coin. This revision is Bayesian updating. If more observations arrive, the current posterior can serve as the prior for the next update, using the likelihood of the new observations given the hypothesis and any relevant previous data.

A likelihood assumes a hypothesis, not a prior

A useful phrasing is:

The likelihood tells us how probable the observed evidence would be if a particular hypothesis were true.

We do not need a prior to calculate $𝑃\left(𝐴\mid 𝐿\right)=0.9$. We need the prior when turning that likelihood into the posterior $𝑃\left(𝐿\mid 𝐴\right)$.

For general observed data $𝐷$ and a candidate hypothesis $ℎ$, the likelihood function is

Hold the observations $𝐷$ fixed and vary $ℎ$. The likelihood compares how well different hypotheses account for those same observations. It is not a probability distribution over hypotheses: its values need not sum to one across the candidates.

In particular, $𝑃\left(𝐴\mid 𝐿\right)=0.9$ does not mean that the probability Alice picked the loaded coin is $0.9$. That question asks for $𝑃\left(𝐿\mid 𝐴\right)$.

Predicting the second flip after a head

Once we have updated the probabilities of the two coins, use those new weights to predict the next outcome:

Given the coin, its flips are independent. Therefore,

Read this as two cases: the loaded coin gives a head with probability $𝑝$; the fair coin gives a head with probability $\frac{1}{2}$. Weight each case by how likely that coin is after the first head.

For $𝑝=0.9$,

This agrees with the conditional-probability calculation:

Using the posterior to predict a new observation is posterior prediction. It is a different step from updating the coin probabilities themselves.

If the question instead asks for tails on the second flip, use $1-𝑝$:

Here ${𝐵}^{𝑐}$ means “not $𝐵$.” The fair-coin term remains $\frac{1}{2}$ because either side of a fair coin has that probability.

Maximum likelihood: what are we maximizing?

Maximum likelihood holds the observed data fixed and chooses the hypothesis or parameter value that makes those observations most probable:

The expression $arg\; max$ returns the candidate that wins, not the value of the winning likelihood.

If the data is one head and the only candidates are the fair coin and the loaded coin with $𝑝=0.9$, maximum likelihood chooses the loaded coin:

This choice does not use the prior and does not give a posterior probability. It also does not claim that the chosen coin certainly produced the head.

Why a sample proportion is an MLE

Now consider a parameter-estimation problem. Let $𝜃$ be the unknown probability that a spam email contains “prize.” We observe 100 independently sampled spam emails: 80 contain the word and 20 do not.

Writing $\widehat{𝜃}=\frac{80}{100}$ only states an estimate. To explain why it is maximum likelihood, we must show what is being maximized.

Model each email’s word-presence indicator as an independent Bernoulli observation with probability $𝜃$. Let $𝐷$ be the observed sequence of 80 presences and 20 absences. All 100 emails are already known to be spam. The likelihood function is

This is still the probability of evidence under an assumed hypothesis. Here, the hypothesis is a proposed value of $𝜃$. For example:

• Hypothesis: $𝜃=0.8$. A spam email contains “prize” with probability $0.8$. Under this hypothesis, the observed sequence has probability $\mathrm{ℒ︀}\left(0.8\right)=𝑃\left(𝐷\mid 𝜃=0.8\right)={0.8}^{80}{\left(0.2\right)}^{20}$.
• Hypothesis: $𝜃=0.5$. A spam email contains “prize” with probability $0.5$. Under this hypothesis, the same observed sequence has probability $\mathrm{ℒ︀}\left(0.5\right)=𝑃\left(𝐷\mid 𝜃=0.5\right)={0.5}^{80}{\left(0.5\right)}^{20}$.

The connection to Alice’s coin is direct. In $𝑃\left(𝐴\mid 𝐿\right)$, the evidence is a head and the hypothesis is “Alice picked the loaded coin.” In $𝑃\left(𝐷\mid 𝜃\right)$, the evidence is the training sequence and the hypothesis specifies the word-presence probability. Both ask: How probable is this evidence under this hypothesis?

The calligraphic $\mathrm{ℒ︀}$ simply names the likelihood function. Writing $\mathrm{ℒ︀}\left(𝜃\right)$ emphasizes that we keep the evidence fixed and vary the candidate $𝜃$. It is not $𝑃\left(𝜃\mid 𝐷\right)$, which would describe a posterior over the unknown parameter after specifying a prior.

Nor are we trying to make “prize” as probable as possible. Setting $𝜃=1$ would make the 20 observed absences impossible and give $\mathrm{ℒ︀}\left(1\right)=0$. We must make the whole observed sequence probable, including both presences and absences.

The 80 and 20 are fixed observations. We vary $𝜃$, balancing the probability of the observed presences against the observed absences. Maximizing the logarithm gives the same result because the logarithm is strictly increasing:

On $0<𝜃<1$, set the derivative to zero:

This is the unique maximum: the second derivative is negative throughout the interior, and the likelihood is zero at both endpoints for these data.

If we record only the count 80 rather than the ordered sequence, the likelihood also contains the binomial coefficient $\left(\genfrac{}{}{0ex}{}{100}{80}\right)$. That factor does not depend on $𝜃$, so it does not change the maximizer.

Keep these three objects separate:

• Likelihood being maximized: ${𝜃}^{80}{\left(1-𝜃\right)}^{20}$, a function of the candidate parameter and the entire training dataset.
• Resulting estimate: ${\widehat{𝜃}}_{\text{MLE}}=0.8$.
• Fitted probability for a new email: using that estimate, a new spam email has probability $0.8$ of containing “prize” under the model.

The sample proportion is a shortcut formula derived from maximization. The number $0.8$ is the estimated parameter, not the likelihood of the 100-email dataset.

MAP: choose the highest posterior

Maximum a posteriori (MAP) chooses the hypothesis with the greatest posterior probability:

The denominator $𝑃\left(𝐷\right)$ in Bayes’ theorem is the same for every candidate and is positive for the evidence under consideration. It therefore does not affect which candidate wins.

The distinction is:

• Maximum likelihood: Which hypothesis makes these observations most probable?
• MAP: Which hypothesis is most probable after considering the observations and the prior?

For a contrasting example, change Alice’s selection rule: she now picks the loaded coin only $10\%$ of the time. Keep $𝑝=0.9$ and observe one head. The unnormalized posterior scores are

Maximum likelihood still chooses loaded, since $0.9>0.5$. MAP chooses fair, since $0.45>0.09$. Normalizing gives posterior probabilities $\frac{1}{6}$ for loaded and $\frac{5}{6}$ for fair.

A head favours the loaded coin relative to its prior probability, but not enough to make it the more probable coin overall. With equal priors across the candidates, MAP and maximum likelihood make the same choice.

MAP selects one hypothesis. It is not the same as keeping the full posterior. In Alice’s prediction problem, we retained both possible coins and averaged their predictions rather than simply using the MAP coin.

Naive Bayes is a classifier, not an estimation method

For classification, a hypothesis is a candidate class $𝑐$, such as spam or not spam. The observed evidence is a vector of features $𝑥=\left({𝑥}_1,\dots ,{𝑥}_{𝑛}\right)$, such as whether particular words are present. Bayes’ theorem gives

Naive Bayes assumes that the features are mutually independent given the class. That assumption supplies the factorization

It does not require the features to be independent when the class is unknown. This is the same kind of distinction as Alice’s flips: independent once we know the hidden category, but potentially dependent when we do not.

The usual Naive Bayes decision rule chooses the class with the highest posterior:

This is a MAP class decision, because it includes the class prior. There is no need to compute $𝑃\left(𝑥\right)$ just to choose a class; it is common to all candidates. To report normalized class probabilities, we do need to normalize the scores.

Where MLE fits into Naive Bayes

Before classifying new examples, we must estimate the model’s probabilities from labelled training data. MLE is one way to do that. In a Bernoulli Naive Bayes model, the 80-out-of-100 calculation above estimates a word-presence probability within the spam class. Class proportions can also estimate the class priors.

These are two different stages:

1. Training: estimate parameters from the training data, perhaps using MLE.
2. Prediction: use those parameters and the class priors to make a MAP decision for a new example.

A MAP class decision does not imply that the parameters themselves were estimated by MAP. Likewise, a class prior can be estimated from training data; it is “prior” relative to the new example’s features.

In practice, word probabilities are often smoothed to avoid assigning zero probability to an entire example because one feature was unseen in a class. Smoothed estimates generally differ from the raw MLE proportions.

Naive Bayes specifies a model and classification rule. MLE specifies how to fit parameters. MAP specifies how to select a posterior maximizer. They are related ideas, not interchangeable names.

The formal foundation: a probability space

A probability space is a three-tuple

Briefly, its components are:

• Sample space $\mathrm{\Omega }$: all possible complete outcomes.
• Event collection $\mathrm{ℱ︀}$: the sets of outcomes to which we assign probabilities. Formally, this is a sigma-algebra: it contains $\mathrm{\Omega }$ and is closed under complements and countable unions.
• Probability measure $𝑃$: a function $𝑃:\mathrm{ℱ︀}\to \left[0,1\right]$ with $𝑃\left(\mathrm{\Omega }\right)=1$ and countable additivity. For pairwise disjoint events ${𝐸}_1,{𝐸}_2,\dots$, this means $𝑃\left({\cup }_{𝑖=1}^{\infty }{𝐸}_{𝑖}\right)={\sum }_{𝑖=1}^{\infty }𝑃\left({𝐸}_{𝑖}\right)$.

For Alice’s experiment, a complete outcome records both the one coin choice and the two flip results:

There are eight possible outcomes, such as $\left(𝐿,𝐻,𝑇\right)$. Here $𝐹$ and $𝐿$ inside the tuples label the coin; the corresponding events are the sets of outcomes with that coin label. Because this space is finite, we can use all subsets as events: $\mathrm{ℱ︀}=2^{\mathrm{\Omega }}$.

Each fair-coin outcome has probability $\frac{1}{8}$. The loaded-coin outcomes have probabilities

The factor $\frac{1}{2}$ is the probability of selecting that coin. Event probabilities are sums of the probabilities of their member outcomes. For example, $𝐴$ consists of all outcomes whose first flip is $𝐻$.

Finally, conditional probability is defined, for $𝑃\left(𝐴\right)>0$, by

Conditioning restricts attention to outcomes consistent with $𝐴$ and rescales their probabilities to total one. Bayesian updating uses this same definition to revise probabilities of hypotheses. It changes the probabilities we assign after learning evidence, not the coin Alice already selected.

A compact map of the ideas

• Prior $𝑃\left(ℎ\right)$: probability of a hypothesis before using the current evidence.
• Likelihood $𝑃\left(𝐷\mid ℎ\right)$: probability of the fixed observed data under a candidate hypothesis; viewed as a function of that candidate.
• Posterior $𝑃\left(ℎ\mid 𝐷\right)$: probability of the hypothesis after using the evidence.
• Bayesian updating: turn a prior into a posterior using the evidence.
• Posterior prediction: average predictions using posterior weights.
• MLE: choose parameters that maximize the training-data likelihood.
• MAP: choose a hypothesis or parameter value with maximum posterior probability or, for continuous parameters, posterior density.
• Naive Bayes: a conditional-independence model used to classify by MAP, often with probabilities fitted by MLE or smoothed estimates.
• Probability space: the outcomes, events, and probability measure that make these calculations well-defined.