Maximum Likelihood and Cross-Entropy

A mathematical textbook connecting likelihood, entropy, KL divergence, statistical estimation, and probabilistic machine learning, with worked examples and solved exercises.

The central idea. Maximum-likelihood estimation chooses a model that makes the observed data as plausible as possible. Cross-entropy measures its average logarithmic prediction loss. For independent observations, these are the same optimization problem. At the population level, the excess cross-entropy above an irreducible baseline is a Kullback–Leibler divergence.

This textbook assumes undergraduate probability, derivatives, gradients, matrices, and basic integration. Expectations, convergence, and asymptotic normality are explained where they enter, but this is not a first course in probability. Work through the finite examples before the asymptotic theory. All logarithms are natural unless a base is explicitly given; losses are in nats. Divide by $ln 2$ to convert nats to bits.

Learning route. Read Sections 1–4 for likelihood and estimation, 5–7 for the information-theoretic connection, 8–10 for classification and implementation, and 11–12 for statistical guarantees and limitations. Section 13 is a reference sheet. Sections 14–15 contain exercises and complete solutions.

1 Models, data, and the direction of inference

1.1 A statistical model is a family of distributions

Let $𝑍$ be an observation with unknown distribution $𝑃_{0}$ . A parametric model is a family $𝒫︀ = {𝑃_{𝜃} : 𝜃 \in Θ}$ , where $Θ \subseteq ℝ^{𝑑}$ is the parameter space. Write $𝑝_{𝜃} (𝑧)$ for a probability mass function or a density relative to a fixed dominating measure $𝜇$ .

For discrete observations, $𝜇$ can be counting measure and $𝑝_{𝜃} (𝑧)$ is a probability. For continuous observations, $𝜇$ can be Lebesgue measure and $𝑝_{𝜃} (𝑧)$ is a density: probabilities are integrals of it. A density may exceed one. The probability of a single specified real-valued outcome is usually zero; this does not make continuous-data inference impossible.

We first observe independent, identically distributed data $𝐷 = (𝑧_{1}, \dots, 𝑧_{𝑛})$ from $𝑃_{0}$ , with $𝑛 \geq 1$ . If $𝑃_{0} = 𝑃_{𝜃_{0}}$ for some parameter, the model is correctly specified. Otherwise it is misspecified: every model distribution is an approximation.

Symbol	Meaning
$𝑧_{𝑖}$	An observed value; fixed during fitting.
$𝜃$	A candidate parameter; varied during fitting.
$𝑝_{𝜃} (𝑧)$	The model’s mass or density at $𝑧$ .
$𝑃_{0}$	The unknown data-generating distribution.
${\hat{𝜃}}_{𝑛}$	An estimator: a function of the sample.
$𝑞$	A candidate predictive distribution on a finite alphabet.

An estimator is random before sampling because its input data are random. After a sample is observed, its computed value is an estimate. Keep those two perspectives separate when interpreting uncertainty statements.

1.2 Likelihood is not a posterior

The joint model density of the observed sample, viewed as a function of the parameter, is the likelihood:

𝐿_{𝑛} (𝜃; 𝐷) = \prod_{𝑖 = 1}^{𝑛} 𝑝_{𝜃} (𝑧_{𝑖}) .

Hold $𝐷$ fixed and vary $𝜃$ . A likelihood need not integrate to one over $Θ$ ; it is not a probability distribution over parameters. Bayesian inference instead uses a prior density $𝜋 (𝜃)$ and forms

𝜋 (𝜃 ∣ 𝐷) \propto 𝐿_{𝑛} (𝜃; 𝐷) 𝜋 (𝜃) .

MLE needs no prior. A maximum a posteriori estimate maximizes the posterior, not just the likelihood. Those optimizers coincide under a constant prior on the relevant parameter space, when such a prior and maxima are meaningful.

Why independence matters. The product above follows from the i.i.d. assumption. For dependent observations the general factorization is

𝑝_{𝜃} (𝑧_{1}, \dots, 𝑧_{𝑛}) = 𝑝_{𝜃} (𝑧_{1}) \prod_{𝑖 = 2}^{𝑛} 𝑝_{𝜃} (𝑧_{𝑖} ∣ 𝑧_{1}, \dots, 𝑧_{𝑖 - 1}) .

One may still maximize this joint likelihood, but replacing its conditional factors with marginals is a different model, not a harmless algebraic step.

2 From a likelihood to an optimization problem

2.1 Products become sums

The maximum-likelihood estimator is any maximizer

{\hat{𝜃}}_{𝑛} \in \underset{𝜃 \in Θ}{arg max} 𝐿_{𝑛} (𝜃; 𝐷) .

Because the logarithm is strictly increasing, maximizing a positive likelihood is equivalent to maximizing the log-likelihood

ℓ_{𝑛} (𝜃) = ln 𝐿_{𝑛} (𝜃; 𝐷) = \sum_{𝑖 = 1}^{𝑛} ln 𝑝_{𝜃} (𝑧_{𝑖}) .

Use $ln 0 = - \infty$ as an extended-real convention. A candidate assigning zero mass or density to an observed value has zero likelihood and infinite negative log-likelihood.

Define total negative log-likelihood (NLL) and average NLL by

{NLL}_{𝑛} (𝜃) = - ℓ_{𝑛} (𝜃), 𝑅_{𝑛} (𝜃) = - \frac{1}{𝑛} ℓ_{𝑛} (𝜃) .

Multiplication by the positive constant $\frac{1}{𝑛}$ does not change the optimizer:

\underset{𝜃}{arg max} 𝐿_{𝑛} (𝜃) = \underset{𝜃}{arg min} {NLL}_{𝑛} (𝜃) = \underset{𝜃}{arg min} 𝑅_{𝑛} (𝜃) .

The values are different, however. An average loss is comparable across sample sizes; a total loss scales with $𝑛$ . If a penalty is added, its scaling must be adjusted too.

2.2 Which constants may be dropped?

If $ℓ_{𝑛} (𝜃) = 𝑎 (𝜃) + 𝑏 (𝐷)$ , a term $𝑏 (𝐷)$ independent of every fitted parameter can be omitted when finding the maximizer. It cannot necessarily be omitted when reporting a likelihood value or comparing likelihoods from different observation models.

For example, the Gaussian normalization includes $ln 𝜎^{2}$ . It is a constant if the variance is known, but not if the variance is being fitted. Parameter-dependent support indicators must also be retained.

2.3 An optimizer need not exist or be unique

The notation ${\hat{𝜃}}_{𝑛}$ does not prove existence. A continuous function on a compact parameter space attains a finite maximum if it is finite there; open or unbounded spaces need not have this property. Distinct parameters may also induce the same distribution. Identifiability means $𝑃_{𝜃} = 𝑃_{𝜂}$ implies $𝜃 = 𝜂$ ; without it, a unique distribution need not correspond to a unique parameter.

A zero gradient is only a candidate for an interior optimum. Check the parameter space, boundaries, curvature, and whether the likelihood can grow without bound. These checks are part of MLE, not optional finishing touches.

3 Worked example: Bernoulli and categorical MLE

3.1 A coin with unknown probability

Let $𝑍_{𝑖} \in {0, 1}$ , with $𝑃_{𝜃} (𝑍_{𝑖} = 1) = 𝜃$ and $Θ = [0, 1]$ . If $𝑘 = \sum_{𝑖} 𝑧_{𝑖}$ successes are observed, then

𝐿_{𝑛} (𝜃) = 𝜃^{𝑘} {(1 - 𝜃)}^{𝑛 - 𝑘}, ℓ_{𝑛} (𝜃) = 𝑘 ln 𝜃 + (𝑛 - 𝑘) ln (1 - 𝜃) .

These formulas describe the particular ordered sequence. If the observation were only the count $𝑘$ , its binomial mass would include $(\binom{𝑛}{𝑘})$ . That factor is independent of $𝜃$ , so both experiments give the same MLE.

For $0 < 𝑘 < 𝑛$ , differentiation on $(0, 1)$ gives

ℓ_{𝑛}^{'} (𝜃) = \frac{𝑘}{𝜃} - \frac{𝑛 - 𝑘}{1 - 𝜃} = \frac{𝑘 - 𝑛 𝜃}{𝜃 (1 - 𝜃)},

ℓ_{𝑛}^{″} (𝜃) = - \frac{𝑘}{𝜃^{2}} - \frac{𝑛 - 𝑘}{{(1 - 𝜃)}^{2}} < 0 .

Strict concavity makes the stationary point the unique global maximizer:

{\hat{𝜃}}_{𝑛} = \frac{𝑘}{𝑛} = \bar{𝑧} .

If $𝑘 = 0$ , the maximum on $[0, 1]$ is at zero. If $𝑘 = 𝑛$ , it is at one. On the open space $(0, 1)$ , these two samples have a supremum but no MLE. A derivative calculation alone would miss this distinction.

3.2 A complete numerical fit

Suppose ten flips give seven heads and three tails. The fitted success probability is $0.7$ .

$𝜃$	Sequence likelihood	Average NLL
$0.5$	${0.5}^{10} \approx 0.000977$	$0.693147$
$0.7$	${0.7}^{7} {0.3}^{3} \approx 0.002224$	$0.610864$
$0.9$	${0.9}^{7} {0.1}^{3} \approx 0.000478$	$0.764528$

All sequence probabilities are small. Their absolute smallness is not a failure: any particular long sequence can be unlikely. The MLE compares candidates for the same observed sample. Predicting 0.9 after observing mostly heads overstates confidence and is penalized by the three tails.

Figure 1: Cross-entropy keeps the data frequency 0.7 fixed while varying the model probability

𝑝

. Entropy varies the distribution in both positions. These are different functions, even though they coincide at

𝑝 = 0.7

3.3 More than two outcomes

For $𝐾$ classes, let $𝜃 = (𝜃_{1}, \dots, 𝜃_{𝐾})$ belong to the probability simplex: $𝜃_{𝑗} \geq 0$ and $\sum_{𝑗} 𝜃_{𝑗} = 1$ . If class $𝑗$ occurs $𝑛_{𝑗}$ times, the ordered-data log-likelihood is

ℓ_{𝑛} (𝜃) = \sum_{𝑗 = 1}^{𝐾} 𝑛_{𝑗} ln 𝜃_{𝑗} .

When all counts are positive, introduce a Lagrange multiplier:

𝒥︀ (𝜃, 𝜆) = \sum_{𝑗} 𝑛_{𝑗} ln 𝜃_{𝑗} + 𝜆 (1 - \sum_{𝑗} 𝜃_{𝑗}) .

Stationarity gives $\frac{𝑛_{𝑗}}{𝜃_{𝑗}} = 𝜆$ . Summing the implied $𝜃_{𝑗} = \frac{𝑛_{𝑗}}{𝜆}$ yields $𝜆 = 𝑛$ , hence

{\hat{𝜃}}_{𝑗} = \frac{𝑛_{𝑗}}{𝑛} .

If a class is unobserved, the maximum over the closed simplex assigns it zero probability. Assigning mass to an unobserved class would leave less mass for the observed classes and reduce their likelihood. A strictly positive softmax parameterization can approach this boundary but cannot attain it with finite logits.

MLE does not automatically smooth. A zero empirical count can produce a zero fitted probability. Priors or other regularization can change this, but doing so changes the estimation rule.

4 Worked example: Gaussian MLE and squared error

4.1 Estimating mean and variance

Assume $𝑍_{𝑖}$ are i.i.d. normal with mean $𝑚 \in ℝ$ and variance $𝑣 > 0$ :

𝑝_{𝑚, 𝑣} (𝑧) = \frac{1}{\sqrt{2 𝜋 𝑣}} exp (- \frac{{(𝑧 - 𝑚)}^{2}}{2 𝑣}) .

The log-likelihood is

ℓ_{𝑛} (𝑚, 𝑣) = - \frac{𝑛}{2} ln (2 𝜋) - \frac{𝑛}{2} ln 𝑣 - \frac{1}{2 𝑣} \sum_{𝑖} {(𝑧_{𝑖} - 𝑚)}^{2} .

For fixed $𝑣$ , maximizing over $𝑚$ minimizes the sum of squares:

\frac{𝜕 ℓ_{𝑛}}{𝜕 𝑚} = \frac{1}{𝑣} \sum_{𝑖} (𝑧_{𝑖} - 𝑚) = 0 \Rightarrow \hat{𝑚} = \bar{𝑧} .

Let $𝑆 = \sum_{𝑖} {(𝑧_{𝑖} - \bar{𝑧})}^{2}$ . Substituting the fitted mean gives

\frac{𝜕 ℓ_{𝑛} (\bar{𝑧}, 𝑣)}{𝜕 𝑣} = - \frac{𝑛}{2 𝑣} + \frac{𝑆}{2 𝑣^{2}} .

If $𝑆 > 0$ , the maximum is

\hat{𝑣} = \frac{𝑆}{𝑛} .

This is not the usual unbiased estimator $\frac{𝑆}{𝑛 - 1}$ . Indeed, $𝔼 [𝑆] = (𝑛 - 1) 𝑣_{0}$ , so the MLE has expectation $(𝑛 - 1) \frac{𝑣_{0}}{𝑛}$ . MLE optimizes likelihood; unbiasedness is a different criterion.

If all data values coincide, $𝑆 = 0$ . Taking $𝑚$ equal to their common value and $𝑣$ down to zero sends the likelihood to infinity. There is no MLE with $𝑣 > 0$ . This includes any sample of size one when both parameters are free.

Numerical example. For $(1, 2, 3)$ , $\hat{𝑚} = 2$ and $𝑆 = 2$ , so $\hat{𝑣} = \frac{2}{3}$ . The unbiased variance estimate is $1$ .

4.2 Regression is a likelihood model too

Suppose the responses satisfy

𝑌_{𝑖} ∣ 𝑋_{𝑖} = 𝑥_{𝑖} \sim 𝒩︀ (𝑓_{𝜃} (𝑥_{𝑖}), 𝑣),

conditionally independently, with known common variance $𝑣$ . Conditional NLL is

- \sum_{𝑖} ln 𝑝_{𝜃} (𝑦_{𝑖} ∣ 𝑥_{𝑖}) = \frac{𝑛}{2} ln (2 𝜋 𝑣) + \frac{1}{2 𝑣} \sum_{𝑖} {(𝑦_{𝑖} - 𝑓_{𝜃} (𝑥_{𝑖}))}^{2} .

Thus least squares is Gaussian conditional MLE under these assumptions. With known unequal variances $𝑣_{𝑖}$ , the squared residuals instead carry weights $\frac{1}{𝑣_{𝑖}}$ . With a fitted variance, the normalization must remain. Absolute-error loss corresponds to a Laplace error model with fixed scale, not to the same Gaussian likelihood.

5 Entropy, cross-entropy, and KL divergence

5.1 Surprise and entropy

For a finite alphabet, an event predicted with probability $𝑞_{𝑗}$ incurs logarithmic loss, or surprise, $- ln 𝑞_{𝑗}$ . An outcome assigned probability one incurs zero loss; a possible outcome assigned zero probability incurs infinite loss. Independent events have additive surprise because $- ln (𝑎 𝑏) = - ln 𝑎 - ln 𝑏$ .

If the true probabilities are $𝑝 = (𝑝_{1}, \dots, 𝑝_{𝐾})$ , the expected surprise under a model $𝑞$ is the cross-entropy

𝐻 (𝑝, 𝑞) = - \sum_{𝑗 = 1}^{𝐾} 𝑝_{𝑗} ln 𝑞_{𝑗} .

The entropy is the self-prediction baseline

𝐻 (𝑝) = 𝐻 (𝑝, 𝑝) = - \sum_{𝑗} 𝑝_{𝑗} ln 𝑝_{𝑗} .

Use $0 ln 0 = 0$ and interpret a term with $𝑝_{𝑗} = 0$ as zero, even if $𝑞_{𝑗} = 0$ . If $𝑝_{𝑗} > 0$ but $𝑞_{𝑗} = 0$ , cross-entropy is infinite.

Entropy measures uncertainty in the source distribution. Cross-entropy also depends on the model. Calling them interchangeable hides exactly the quantity that fitting can improve.

5.2 KL divergence is the excess cross-entropy

The Kullback–Leibler divergence from $𝑝$ to $𝑞$ is

𝐷_{KL} (𝑝 ∥ 𝑞) = \sum_{𝑗 : 𝑝_{𝑗} > 0} 𝑝_{𝑗} ln (\frac{𝑝_{𝑗}}{𝑞_{𝑗}}) .

If $𝑞$ is positive on the support of $𝑝$ , simple algebra gives

𝐻 (𝑝, 𝑞) = 𝐻 (𝑝) + 𝐷_{KL} (𝑝 ∥ 𝑞) .

It remains true with infinite cross-entropy under the conventions above. The first term does not depend on $𝑞$ , so minimizing cross-entropy over a model family minimizes the forward divergence $𝐷_{KL} (𝑝 ∥ 𝑞)$ .

Gibbs’ inequality, proved. The inequality $ln 𝑡 \leq 𝑡 - 1$ for $𝑡 > 0$ follows by minimizing $𝑡 - 1 - ln 𝑡$ at $𝑡 = 1$ . Therefore

- 𝐷_{KL} (𝑝 ∥ 𝑞) = \sum_{𝑗 : 𝑝_{𝑗} > 0} 𝑝_{𝑗} ln (\frac{𝑞_{𝑗}}{𝑝_{𝑗}}) \leq \sum_{𝑗 : 𝑝_{𝑗} > 0} (𝑞_{𝑗} - 𝑝_{𝑗}) \leq 0 .

Hence $𝐷_{KL} (𝑝 ∥ 𝑞) \geq 0$ . Equality requires $𝑞_{𝑗} = 𝑝_{𝑗}$ on the support of $𝑝$ and no extra mass outside it, so $𝑞 = 𝑝$ . If $𝑞$ vanishes at a positive-probability outcome, the divergence is infinite and the inequality is immediate.

KL divergence is not a distance: it is generally asymmetric and does not satisfy the triangle inequality. The order matters. Likelihood fitting corresponds to $𝐷_{KL} (𝑝_{0} ∥ 𝑝_{𝜃})$ , not its reversal.

5.3 A worked three-class calculation

Take $𝑝 = (0.6, 0.3, 0.1)$ and a model $𝑞 = (0.8, 0.1, 0.1)$ . Then

𝐻 (𝑝) = - 0.6 ln 0.6 - 0.3 ln 0.3 - 0.1 ln 0.1 \approx 0.897946,

𝐻 (𝑝, 𝑞) = - 0.6 ln 0.8 - 0.3 ln 0.1 - 0.1 ln 0.1 \approx 1.054920,

𝐷_{KL} (𝑝 ∥ 𝑞) = 𝐻 (𝑝, 𝑞) - 𝐻 (𝑝) \approx 0.156974 .

The extra loss comes from probability mismatch, not additional randomness in the source. Both distributions select class 1 by their largest probability, so a class-decision metric alone does not reveal this mismatch.

5.4 Coding intuition and its limits

With base-two logarithms, ideal code lengths have the form $- {log}_{2} 𝑞_{𝑗}$ . A code designed for $𝑞$ uses expected ideal length $𝐻_{2} (𝑝, 𝑞)$ when the source is $𝑝$ , with excess $𝐷_{KL, 2} (𝑝 ∥ 𝑞)$ . Individual prefix-code lengths must be integers and obey coding constraints, so cross-entropy need not equal the exact expected length of a particular one-symbol code. Block or arithmetic coding makes the ideal-rate interpretation useful.

6 The exact bridge from MLE to cross-entropy

6.1 Empirical distributions turn sample sums into expectations

For categorical data, define the empirical distribution ${\hat{𝑝}}_{𝑗} = \frac{𝑛_{𝑗}}{𝑛}$ . Then

𝑅_{𝑛} (𝜃) = - \frac{1}{𝑛} \sum_{𝑗} 𝑛_{𝑗} ln 𝜃_{𝑗} = - \sum_{𝑗} {\hat{𝑝}}_{𝑗} ln 𝜃_{𝑗} = 𝐻 (\hat{𝑝}, 𝜃) .

Consequently

𝑅_{𝑛} (𝜃) = 𝐻 (\hat{𝑝}) + 𝐷_{KL} (\hat{𝑝} ∥ 𝜃) .

The data-dependent entropy is fixed during optimization. The unrestricted categorical MLE sets $𝜃 = \hat{𝑝}$ and makes the divergence zero.

This is an exact finite-sample identity, not a large-sample approximation. The approximation enters when we replace the unknown population distribution by its empirical counterpart.

6.2 Population risk and the best model in a restricted family

Define the population logarithmic risk

𝑅 (𝜃) = 𝔼_{𝑍 \sim 𝑃_{0}} [- ln 𝑝_{𝜃} (𝑍)] .

For a finite alphabet, write the true mass function as $𝑝_{0}$ . Then

𝑅 (𝜃) = 𝐻 (𝑝_{0}) + 𝐷_{KL} (𝑝_{0} ∥ 𝑝_{𝜃}) .

If the true distribution belongs to an identifiable model, the unique risk minimizer is $𝜃_{0}$ . If it does not, a minimizer $𝜃^{⋆}$ is a pseudo-true parameter: the model’s best forward-KL approximation, assuming a minimizer exists. It need not reproduce every property of $𝑃_{0}$ .

Restricted-model example. Suppose the three-class model is $𝑝_{𝜃} = (𝜃, \frac{1 - 𝜃}{2}, \frac{1 - 𝜃}{2})$ and the true distribution is $(0.6, 0.3, 0.1)$ . Its population cross-entropy is

𝑅 (𝜃) = - 0.6 ln 𝜃 - 0.4 ln (\frac{1 - 𝜃}{2}) .

Differentiation gives $𝜃^{⋆} = 0.6$ , hence the best model is $(0.6, 0.2, 0.2)$ . It matches the first class and the combined mass of the other two, but cannot distinguish their frequencies. Its risk is about $0.950271$ , leaving KL divergence about $0.052325$ .

The sample objective $𝑅_{𝑛}$ estimates $𝑅$ ; it is not identical to it. A flexible model can reduce training NLL without reducing risk on fresh data.

7 Continuous observations require care

7.1 Differential entropy is not discrete entropy

If $𝑃_{0}$ has density $𝑝_{0}$ relative to the same measure as $𝑝_{𝜃}$ , define, when the integrals are finite,

ℎ (𝑝_{0}) = - \int 𝑝_{0} (𝑧) ln 𝑝_{0} (𝑧) d 𝑧,

ℎ (𝑝_{0}, 𝑝_{𝜃}) = - \int 𝑝_{0} (𝑧) ln 𝑝_{𝜃} (𝑧) d 𝑧 .

Then

ℎ (𝑝_{0}, 𝑝_{𝜃}) = ℎ (𝑝_{0}) + 𝐷_{KL} (𝑃_{0} ∥ 𝑃_{𝜃}) .

In settings where an entropy is infinite, do not subtract infinities to justify the identity. Work directly with well-defined expectations or likelihood ratios and verify integrability.

Differential entropy can be negative. For a uniform density on $(0, 𝑎)$ , $ℎ = ln 𝑎$ , which is negative for $𝑎 < 1$ . Density values and differential entropy depend on coordinates and units. Under a smooth bijection $𝑈 = 𝑔 (𝑍)$ with nonzero Jacobian determinant, the change-of-variables formula gives

- ln 𝑝_{𝜃}^{𝑈} (𝑔 (𝑧)) = - ln 𝑝_{𝜃}^{𝑍} (𝑧) + ln | det 𝐷 𝑔 (𝑧) | .

If $𝑔$ is independent of $𝜃$ , the additional term is the same for all candidates, so the MLE is unchanged. KL divergence is invariant when both distributions undergo the same bijective transformation, subject to the usual measure-theoretic definitions.

7.2 Why empirical continuous-data NLL is not an empirical KL

The empirical probability measure puts mass $\frac{1}{𝑛}$ at each observation (counting repeated values with their multiplicities). A continuous model assigns zero probability to this finite set. Therefore the empirical measure is not absolutely continuous with respect to that model, and $𝐷_{KL} ({\hat{𝑃}}_{𝑛} ∥ 𝑃_{𝜃}) = \infty$ .

Nevertheless

𝑅_{𝑛} (𝜃) = - \frac{1}{𝑛} \sum_{𝑖} ln 𝑝_{𝜃} (𝑧_{𝑖})

is a perfectly useful empirical density log-loss. It estimates the population cross-entropy under integrability assumptions. Do not claim that it equals a finite empirical KL plus an empirical differential entropy. That finite-discrete derivation does not transfer to atomic empirical data and continuous model measures.

8 Conditional likelihood and binary cross-entropy

8.1 Predicting a label given features

For pairs $(𝑋_{𝑖}, 𝑌_{𝑖})$ , a discriminative model specifies $𝑝_{𝜃} (𝑦 ∣ 𝑥)$ . Conditionally independent labels give

𝐿_{𝑛} (𝜃 ∣ 𝑥_{1}, \dots, 𝑥_{𝑛}) = \prod_{𝑖} 𝑝_{𝜃} (𝑦_{𝑖} ∣ 𝑥_{𝑖}) .

When the feature distribution has no fitted $𝜃$ dependence, its factor can be omitted from a joint likelihood for fitting $𝜃$ . We are modeling responses conditional on features, not asserting that features are random noise that can always be ignored.

For binary labels $𝑦_{𝑖} \in {0, 1}$ , let $𝑝_{𝑖} = 𝑃_{𝜃} (𝑌_{𝑖} = 1 ∣ 𝑥_{𝑖})$ . The Bernoulli conditional mass is

𝑝_{𝜃} (𝑦_{𝑖} ∣ 𝑥_{𝑖}) = 𝑝_{𝑖}^{𝑦_{𝑖}} {(1 - 𝑝_{𝑖})}^{1 - 𝑦_{𝑖}} .

Its negative log is the binary cross-entropy (BCE):

ℒ︀ (𝑦_{𝑖}, 𝑝_{𝑖}) = - 𝑦_{𝑖} ln 𝑝_{𝑖} - (1 - 𝑦_{𝑖}) ln (1 - 𝑝_{𝑖}) .

For a hard label 1 it is $- ln 𝑝_{𝑖}$ ; for a hard label 0 it is $- ln (1 - 𝑝_{𝑖})$ . Both terms are needed for the distribution, even though one vanishes for each hard-labeled observation.

8.2 Soft targets and the optimal probability

Let $𝑟 (𝑥) = 𝑃_{0} (𝑌 = 1 ∣ 𝑋 = 𝑥)$ . Conditional expected loss at a fixed $𝑥$ is

𝐶 (𝑟, 𝑝) = - 𝑟 ln 𝑝 - (1 - 𝑟) ln (1 - 𝑝) .

For $0 < 𝑟 < 1$ ,

\frac{𝜕 𝐶}{𝜕 𝑝} = \frac{𝑝 - 𝑟}{𝑝 (1 - 𝑝)}, \frac{𝜕^{2} 𝐶}{𝜕 𝑝^{2}} = \frac{𝑟}{𝑝^{2}} + \frac{1 - 𝑟}{{(1 - 𝑝)}^{2}} > 0 .

The unique minimum is $𝑝 = 𝑟$ . The analogous minima for $𝑟 = 0$ and $𝑟 = 1$ are at the respective boundaries. Log loss is thus a strictly proper scoring rule: in expectation, reporting the true distribution is optimal.

A soft label $𝑟$ can represent an actual conditional class distribution or a chosen training target. Applying BCE to it computes an expected Bernoulli log-loss. A fractional target is not a single observed Bernoulli outcome; its interpretation matters.

For general classes, conditional population cross-entropy decomposes as

𝑅 (𝜃) = 𝐻 (𝑌 ∣ 𝑋) + 𝔼_{𝑋 [𝐷_{KL} (𝑃_{0} (𝑌 ∣ 𝑋) ∥ 𝑃_{𝜃} (𝑌 ∣ 𝑋))]} .

Here $𝑌$ is finite-valued and expectations use the true feature distribution. The baseline is label uncertainty remaining after observing $𝑋$ .

8.3 Logistic regression: gradient and curvature

Let a logit be $𝑎_{𝑖} = 𝒘^{𝑇} 𝒙_{𝑖} + 𝑏$ , with

𝑝_{𝑖} = 𝜎 (𝑎_{𝑖}) = \frac{1}{1 + 𝑒^{- 𝑎_{𝑖}}} .

Using $𝜎^{'} (𝑎) = 𝜎 (𝑎) (1 - 𝜎 (𝑎))$ and the chain rule,

\frac{𝜕 ℒ︀}{𝜕 𝑎_{𝑖}} = 𝑝_{𝑖} - 𝑦_{𝑖},

\nabla_{𝒘} 𝑅_{𝑛} = \frac{1}{𝑛} \sum_{𝑖} (𝑝_{𝑖} - 𝑦_{𝑖}) 𝒙_{𝑖}, \frac{𝜕 𝑅_{𝑛}}{𝜕 𝑏} = \frac{1}{𝑛} \sum_{𝑖} (𝑝_{𝑖} - 𝑦_{𝑖}) .

Absorb the intercept into an augmented feature vector ${\tilde{𝒙}}_{𝑖}$ . The Hessian for all coefficients is

\nabla^{2} 𝑅_{𝑛} = \frac{1}{𝑛} \sum_{𝑖} 𝑝_{𝑖} (1 - 𝑝_{𝑖}) {\tilde{𝒙}}_{𝑖} {\tilde{𝒙}}_{𝑖}^{𝑇} .

For any vector $𝑣$ , its quadratic form is a sum of nonnegative squares, so logistic NLL is convex in these coefficients. It is strictly convex at finite coefficients if the augmented design has full column rank. Convexity does not imply a finite minimizer exists; separation is a counterexample. A neural network’s logits are nonlinear functions of its weights, so the same convexity conclusion does not generally hold for all network weights.

9 Multiclass cross-entropy and softmax

9.1 One categorical outcome per example

For mutually exclusive classes, let $𝑦_{𝑖 𝑗} = 1$ if observation $𝑖$ belongs to class $𝑗$ , and zero otherwise. A probability vector $𝑞_{𝑖} = (𝑞_{𝑖 1}, \dots, 𝑞_{𝑖 𝐾})$ yields loss

{ℒ︀}_{𝑖} = - \sum_{𝑗 = 1}^{𝐾} 𝑦_{𝑖 𝑗} ln 𝑞_{𝑖 𝑗} = - ln 𝑞_{𝑖, 𝑐_{𝑖}},

where $𝑐_{𝑖}$ is the observed class index. This is categorical NLL and cross-entropy between a one-hot target and the predicted distribution. Its target entropy is zero. The entropy of an aggregate class-frequency distribution need not be zero: these are different distributions.

With soft targets $𝑟_{𝑖 𝑗} \geq 0$ summing to one, the loss becomes $- \sum_{𝑗} 𝑟_{𝑖 𝑗} ln 𝑞_{𝑖 𝑗}$ . A uniform prediction has loss $ln 𝐾$ for any such target, providing a useful numerical reference, not a universal performance threshold for every dataset.

9.2 Softmax converts logits into a probability vector

For logits $𝑎_{1}, \dots, 𝑎_{𝐾} \in ℝ$ , define

𝑞_{𝑗} = \frac{exp (𝑎_{𝑗})}{\sum_{𝑘 = 1}^{𝐾} exp (𝑎_{𝑘})} .

The loss for one observation with target $𝑟$ is

ℒ︀ (𝑟, 𝑎) = ln (\sum_{𝑘} 𝑒^{𝑎_{𝑘}}) - \sum_{𝑗} 𝑟_{𝑗} 𝑎_{𝑗} .

Differentiating log-sum-exp gives

\frac{𝜕 ℒ︀}{𝜕 𝑎_{𝑗}} = 𝑞_{𝑗} - 𝑟_{𝑗}, \nabla_{𝑎}^{2} ℒ︀ = diag (𝑞) - 𝑞 𝑞^{𝑇} .

For any $𝑣$ , the quadratic form equals $\sum_{𝑗} 𝑞_{𝑗} 𝑣_{𝑗}^{2} - {(\sum_{𝑗} 𝑞_{𝑗} 𝑣_{𝑗})}^{2}$ , a variance and hence nonnegative. The loss is convex in logits. Adding the same constant to every logit leaves $𝑞$ unchanged, so logits are not identifiable without a constraint, such as setting one reference logit to zero.

Numerical example. Logits $(ln 2, 0, - ln 2)$ give probabilities $(\frac{4}{7}, \frac{2}{7}, \frac{1}{7})$ . If class 2 is observed, the loss is $ln (\frac{7}{2}) \approx 1.252763$ and the logit gradient is $(\frac{4}{7}, - \frac{5}{7}, \frac{1}{7})$ . Its components sum to zero, consistent with invariance to a common logit shift.

9.3 Binary, multiclass, and multilabel are distinct

Binary: one Bernoulli outcome; use one logit and BCE.
Multiclass: exactly one of $𝐾$ classes; use categorical NLL, often with softmax probabilities summing to one.
Multilabel: several labels may simultaneously be present; a common model uses one Bernoulli probability per label and sums BCE terms. This models labels as conditionally independent given the features. Their probabilities need not sum to one.

A two-class softmax is equivalent to a single sigmoid with logit equal to the difference of the two logits. Multiple independent sigmoids are not a replacement for a single categorical distribution over exclusive classes.

10 Computing and interpreting log loss correctly

10.1 Stable formulas use logits directly

A naive implementation can overflow by computing $exp (1000)$ , or round a probability to zero or one before taking a logarithm. Use algebraically equivalent logit formulas instead. For finite binary logit $𝑎$ and target $𝑦 \in [0, 1]$ ,

ℒ︀ (𝑦, 𝑎) = ln (1 + 𝑒^{𝑎}) - 𝑦 𝑎 = max (𝑎, 0) - 𝑦 𝑎 + ln (1 + 𝑒^{- | 𝑎 |}) .

The last expression avoids exponentiating a large positive number. Implement the final logarithm with a dedicated log1p operation. For multiclass logits, let $𝑚 = {max}_{𝑗} 𝑎_{𝑗}$ :

ln (\sum_{𝑗} 𝑒^{𝑎_{𝑗}}) = 𝑚 + ln (\sum_{𝑗} 𝑒^{𝑎_{𝑗} - 𝑚}) .

At least one shifted exponent is one and none exceeds one. Fused cross-entropy-with-logits routines implement these stable transformations; check whether an API expects logits, probabilities, hard class indices, or soft targets. Applying softmax twice is not the same objective.

Extreme-logit check. A binary positive label with $𝑎 = - 1000$ has loss approximately $1000$ and derivative approximately $- 1$ , not an undefined logarithm. With $𝑎 = 1000$ , its loss is effectively zero at ordinary floating precision. Reversing the label reverses which prediction is badly wrong.

Clipping probabilities changes the objective and can flatten or distort its gradients. It may be a deliberate policy when evaluating externally supplied probabilities, but it is not equivalent to stable evaluation of the original logit loss.

10.2 Accuracy and log loss answer different questions

Suppose the true binary frequency is $𝑟 = 0.7$ . Predictors always reporting $0.7$ and $0.99$ make the same class decision at threshold $0.5$ , so both have expected accuracy $0.7$ . Their expected log losses differ:

𝐶 (0.7, 0.7) \approx 0.610864, 𝐶 (0.7, 0.99) \approx 1.388586 .

The second is overconfident. Accuracy ignores the reported confidence until it crosses the decision threshold; log loss evaluates the full probability. Good average log loss also does not, by itself, prove calibration in every subgroup or good performance under distribution shift.

10.3 Class weights and label smoothing change the target

For binary class weights $𝛼, 𝛽 > 0$ , conditional expected weighted loss is

𝐶_{𝑤} (𝑟, 𝑝) = - 𝛼 𝑟 ln 𝑝 - 𝛽 (1 - 𝑟) ln (1 - 𝑝) .

Its optimum is

𝑝_{𝑤}^{⋆} = \frac{𝛼 𝑟}{𝛼 𝑟 + 𝛽 (1 - 𝑟)} .

Unless the weights are equal or a boundary case applies, this is not the original class probability $𝑟$ . Class weighting can serve a cost-sensitive or resampling objective, but the resulting raw probabilities need not be calibrated to the original population.

For label smoothing, replacing target $𝑦$ by $𝑟 = (1 - 𝜀) 𝑦 + 𝜀 𝑢$ , with a chosen reference distribution $𝑢$ , optimizes cross-entropy to that modified target. It is not ordinary MLE of the unchanged hard labels. Be explicit about the goal being optimized.

10.4 Penalties, MAP, and normalization

If a prior has density $𝜋 (𝜃) \propto exp (- 𝜆 𝐽 (𝜃))$ , a MAP estimate minimizes

- ℓ_{𝑛} (𝜃) + 𝜆 𝐽 (𝜃) .

In average-loss form the same objective is $𝑅_{𝑛} (𝜃) + (\frac{𝜆}{𝑛}) 𝐽 (𝜃)$ . Using $𝑅_{𝑛} + 𝜆 𝐽$ instead corresponds to a total penalty $𝑛 𝜆 𝐽$ . A Gaussian prior produces a quadratic penalty, with coefficients determined by its variance. Regularization can make an otherwise nonexistent unpenalized estimate finite, but it changes the estimator.

MLE is equivariant under a one-to-one reparameterization: transform a maximizer to obtain a maximizer in the new coordinate. A MAP density mode is generally not invariant, because transforming a density introduces a Jacobian. Neither statement licenses dropping a parameter-dependent factor.

11 Why MLE works: consistency and asymptotic inference

11.1 Consistency is an argmin argument

For each fixed $𝜃$ , the law of large numbers suggests $𝑅_{𝑛} (𝜃) \to 𝑅 (𝜃)$ when the log loss is integrable. Pointwise convergence alone is not enough to move a minimizer through a limit: a sample-dependent optimizer can chase narrow random dips.

Here is one useful, deliberately sufficient formulation. Suppose:

The observations are i.i.d.; $Θ$ is compact.
For almost every $𝑧$ , $𝜃 \mapsto - ln 𝑝_{𝜃} (𝑧)$ is finite and continuous, and its absolute value is bounded uniformly over $Θ$ by an integrable envelope. Assume the measurability needed for the suprema.
The continuous risk $𝑅$ has a unique minimizer $𝜃^{⋆}$ .
${\hat{𝜃}}_{𝑛}$ is a measurable exact minimizer, or an approximate minimizer satisfying $𝑅_{𝑛} ({\hat{𝜃}}_{𝑛}) \leq {inf}_{𝜃} 𝑅_{𝑛} (𝜃) + 𝛿_{𝑛}$ , with $𝛿_{𝑛} \to 0$ in probability.

Compactness, continuity, and the envelope give a uniform law of large numbers:

Δ_{𝑛} = sup_{𝜃 \in Θ} | 𝑅_{𝑛} (𝜃) - 𝑅 (𝜃) | \to 0 in probability .

For any fixed $𝜀 > 0$ , compactness and uniqueness give a positive gap

𝑐_{𝜀} = inf_{‖ 𝜃 - 𝜃^{⋆} ‖ \geq 𝜀} (𝑅 (𝜃) - 𝑅 (𝜃^{⋆})) > 0,

when the set is nonempty. On the other hand, approximate optimality implies

𝑅 ({\hat{𝜃}}_{𝑛}) - 𝑅 (𝜃^{⋆}) \leq 2 Δ_{𝑛} + 𝛿_{𝑛} .

With probability tending to one, this is smaller than $𝑐_{𝜀}$ , so $‖ {\hat{𝜃}}_{𝑛} - 𝜃^{⋆} ‖ < 𝜀$ . Thus ${\hat{𝜃}}_{𝑛} \to 𝜃^{⋆}$ in probability: consistency.

For a correctly specified, identifiable model with the required finite risks, the KL identity identifies $𝜃^{⋆}$ with $𝜃_{0}$ . Under misspecification, the same argument targets the pseudo-true minimizer. These assumptions are sufficient, not necessary: normal mean estimation on an unbounded space is consistent too, but requires a different compactness or tightness argument. Boundary probabilities and parameter-dependent supports can violate the displayed assumptions.

11.2 Score and Fisher information

For one observation, define the score and observed log-likelihood curvature at parameter $𝜃$ :

𝑠_{𝜃} (𝑍) = \nabla_{𝜃} ln 𝑝_{𝜃} (𝑍), \nabla_{𝜃}^{2} ln 𝑝_{𝜃} (𝑍) .

Assume a common support and enough domination to differentiate integrals. Under the model $𝑃_{𝜃}$ ,

𝔼_{𝜃 [𝑠_{𝜃} (𝑍)]} = \int \nabla_{𝜃} 𝑝_{𝜃} (𝑧) d 𝜇 (𝑧) = \nabla_{𝜃} 1 = 0 .

Differentiating normalization once more yields the information identity

𝐼 (𝜃) = 𝔼_{𝜃 [𝑠_{𝜃} 𝑠_{𝜃}^{𝑇}]} = - 𝔼_{𝜃 [\nabla_{𝜃}^{2} ln 𝑝_{𝜃} (𝑍)]} .

This is per-observation Fisher information; $𝑛$ independent observations have information $𝑛 𝐼 (𝜃)$ . The identity can fail when support depends on the parameter or differentiation under the integral is invalid.

For Bernoulli $0 < 𝜃 < 1$ ,

𝑠_{𝜃} (𝑍) = \frac{𝑍 - 𝜃}{𝜃 (1 - 𝜃)}, 𝐼 (𝜃) = \frac{1}{𝜃 (1 - 𝜃)} .

Therefore the asymptotic variance of the sample proportion is $𝜃_{0} \frac{1 - 𝜃_{0}}{𝑛}$ , consistent with its exact variance.

11.3 The local normal approximation

In a correctly specified, identifiable, fixed-dimensional model, suppose $𝜃_{0}$ is interior, the MLE is consistent, the likelihood is sufficiently smooth with valid differentiation and local Hessian convergence, and $𝐼 (𝜃_{0})$ is finite and nonsingular. A Taylor expansion of the score at an interior MLE gives, to first order,

0 = \nabla ℓ_{𝑛} ({\hat{𝜃}}_{𝑛}) \approx \nabla ℓ_{𝑛} (𝜃_{0}) + \nabla^{2} ℓ_{𝑛} (𝜃_{0}) ({\hat{𝜃}}_{𝑛} - 𝜃_{0}) .

Consequently

\sqrt{𝑛} ({\hat{𝜃}}_{𝑛} - 𝜃_{0}) \approx {𝐼 (𝜃_{0})}^{- 1} \frac{1}{\sqrt{𝑛}} \sum_{𝑖} 𝑠_{𝜃_{0}} (𝑍_{𝑖}) .

The multivariate central limit theorem gives

\sqrt{𝑛} ({\hat{𝜃}}_{𝑛} - 𝜃_{0}) \overset{𝑑}{\to} 𝒩︀ (0, {𝐼 (𝜃_{0})}^{- 1}) .

The arrow denotes convergence in distribution. Approximate standard errors are square roots of diagonal entries of $\frac{{𝐼 ({\hat{𝜃}}_{𝑛})}^{- 1}}{𝑛}$ , or of a suitable observed-information inverse. This is a local, large-sample result, not a guarantee for small samples or boundary estimates.

For seven successes in ten trials, the plug-in Bernoulli standard error is $\sqrt{0.7 \cdot \frac{0.3}{10}} \approx 0.1449$ . A naive 95% Wald interval is about $[0.416, 0.984]$ . Ten observations are a small sample; the approximation can be poor, and at all-success or all-failure samples it degenerates. Score intervals, such as Wilson’s interval, or exact binomial procedures avoid that particular plug-in pathology.

11.4 Misspecification needs a sandwich covariance

At an interior pseudo-true parameter, put

𝐴 = - 𝔼_{0} [\nabla^{2} ln 𝑝_{𝜃^{⋆}} (𝑍)], 𝐵 = 𝔼_{0} [𝑠_{𝜃^{⋆}} (𝑍) 𝑠_{𝜃^{⋆}} {(𝑍)}^{𝑇}] .

If the score has mean zero there and the corresponding consistency, smoothness, moment, central-limit, and nonsingularity conditions hold,

\sqrt{𝑛} ({\hat{𝜃}}_{𝑛} - 𝜃^{⋆}) \overset{𝑑}{\to} 𝒩︀ (0, 𝐴^{- 1} 𝐵 𝐴^{- 1}) .

The information identity generally does not equate $𝐴$ and $𝐵$ under the true misspecified distribution. The middle matrix is not optional. Empirical averages of fitted scores and Hessians estimate these matrices in the i.i.d. setting; dependent data require an appropriate covariance of the score sum, not automatically this independent-observation formula.

12 Failure modes that test the assumptions

12.1 Separation: perfect classification without a finite MLE

Absorb the binary intercept into $𝒘$ and the features. Suppose there is a vector $𝑣$ with

(2 𝑦_{𝑖} - 1) 𝑣^{𝑇} 𝒙_{𝑖} > 0 for every 𝑖 .

This is complete linear separation. Set $𝒘 = 𝑡 𝑣$ and increase $𝑡$ . Every observed-class probability tends to one, so every NLL term tends to zero. At finite coefficients all sigmoid probabilities lie strictly between zero and one, so the loss is positive. Its infimum zero is not attained.

The classifier may already have perfect training accuracy while its weight norm keeps growing. That is not proof of a finite unregularized MLE. An $𝐿_{2}$ penalty on all coefficients makes the objective coercive and strictly convex, producing a unique finite minimizer; leaving an intercept unpenalized can preserve exceptional nonexistence cases.

12.2 Parameter-dependent support: uniform estimation

Let $𝑍_{𝑖}$ be i.i.d. uniform on $[0, 𝜃]$ , with $𝜃 > 0$ . If $𝑀 = {max}_{𝑖} 𝑧_{𝑖} > 0$ , the full likelihood is

𝐿_{𝑛} (𝜃) = 𝜃^{- 𝑛} 𝟏 {𝜃 \geq 𝑀} .

It decreases for $𝜃 \geq 𝑀$ and is zero below $𝑀$ , so the MLE is ${\hat{𝜃}}_{𝑛} = 𝑀$ . Differentiating only $- 𝑛 ln 𝜃$ and searching for a zero score would fail. The maximum lies at the support constraint.

The usual regular normal limit fails too. Under a true endpoint $𝜃_{0}$ , for fixed $𝑡 \geq 0$ and sufficiently large $𝑛$ ,

𝑃_{𝜃_{0}} (\frac{𝑛 (𝜃_{0} - 𝑀)}{𝜃_{0}} > 𝑡) = {(1 - \frac{𝑡}{𝑛})}^{𝑛} \to 𝑒^{- 𝑡} .

Thus the scaled endpoint error has an exponential limit at rate $𝑛$ , not a normal limit at rate $\sqrt{𝑛}$ . Regularity assumptions carry mathematical content; they are not ceremonial qualifications.

12.3 Training fit does not establish predictive quality

Empirical NLL measures fit to the data used in its average. Evaluating probabilities on independent validation or test observations estimates predictive log loss without reusing those observations for fitting. Repeated model selection on a test set makes it part of the fitting process.

Model misspecification, small samples, dependent data, changing populations, and optimization error are different problems. More training iterations address only some optimization error; they do not automatically fix any of the statistical problems. Likewise, a convex objective does not make an incorrect data model true.

13 Reference sheet: the equivalences and their limits

Object	Formula and interpretation
Likelihood	$𝐿_{𝑛} (𝜃) = \prod_{𝑖} 𝑝_{𝜃} (𝑧_{𝑖})$ for i.i.d. data.
Log-likelihood	$ℓ_{𝑛} (𝜃) = \sum_{𝑖} ln 𝑝_{𝜃} (𝑧_{𝑖})$ . Maximize it.
Average NLL	$𝑅_{𝑛} (𝜃) = - ℓ_{𝑛} \frac{𝜃}{𝑛}$ . Minimize it.
Discrete entropy	$𝐻 (𝑝) = - \sum_{𝑗} 𝑝_{𝑗} ln 𝑝_{𝑗}$ . A fixed-source baseline.
Cross-entropy	$𝐻 (𝑝, 𝑞) = - \sum_{𝑗} 𝑝_{𝑗} ln 𝑞_{𝑗}$ . Expected model log loss.
Forward KL	$𝐷_{KL} (𝑝 ∥ 𝑞) = 𝐻 (𝑝, 𝑞) - 𝐻 (𝑝) \geq 0$ .
Binary loss	$- 𝑦 ln 𝑝 - (1 - 𝑦) ln (1 - 𝑝)$ . Bernoulli NLL.
Multiclass loss	$- ln 𝑞_{𝑐}$ for observed class $𝑐$ . Categorical NLL.
Gaussian regression	Fixed-variance NLL equals a positive multiple of squared error plus a parameter-independent constant.

Before using an equivalence, ask:

What is random, what is conditioned on, and what parameter is fitted?
Are these probabilities or densities relative to a fixed measure?
Does the factorization follow from the stated dependence assumptions?
Which quantities are constant with respect to all fitted parameters?
Is the target empirical or population, hard-labeled or deliberately modified?
Does an optimizer exist in the actual parameter space?
Do the regularity assumptions justify any uncertainty approximation?

The shortest reliable chain is

i.i.d. likelihood maximization \to average log-loss minimization .

For finite discrete outcomes this is empirical cross-entropy minimization. At the population level, under finite-entropy conditions, it is forward-KL minimization plus a constant. The qualifiers distinguish a theorem from a slogan.

14 Exercises

Try these before reading the solutions. Give the parameter space as part of every MLE answer.

Bernoulli boundaries. Derive the MLE after four successes and one failure. What changes after five successes and no failures on $[0, 1]$ versus $(0, 1)$ ? Compute the minimum average NLL for the first sample.
A categorical zero. Counts are $(6, 3, 1, 0)$ . Find the unrestricted categorical MLE and its training cross-entropy. Can finite softmax logits represent that MLE exactly?
Poisson estimation. For $𝑧_{𝑖} \in {0, 1, 2, \dots}$ , assume $𝑝_{𝜆} (𝑧) = 𝑒^{- 𝜆} \frac{𝜆^{𝑧}}{𝑧!}$ with $𝜆 \geq 0$ (including the degenerate distribution at zero). Derive the MLE and evaluate it for $(0, 1, 1, 2, 2, 2, 3, 5)$ . Handle an all-zero sample.
Known and unknown Gaussian means. For $(1, 2, 3)$ , compute the variance MLE when the mean is known to be zero. Compare with the result when the mean is fitted. Why can the normalization not be discarded?
KL asymmetry. For $𝑝 = (0.9, 0.1)$ and $𝑞 = (0.5, 0.5)$ , calculate $𝐷_{KL} (𝑝 ∥ 𝑞)$ and $𝐷_{KL} (𝑞 ∥ 𝑝)$ . Explain which appears when the data come from $𝑝$ and the predictor is $𝑞$ .
One logistic update. Let $𝒙 = (1, 2)$ , $𝑦 = 1$ , no separate intercept, and initial $𝒘 = (0, 0)$ . Compute the single-example loss, gradient, and one gradient-descent step with learning rate $0.1$ .
Class weighting. A feature group has true positive rate $𝑟 = 0.2$ . Positive examples have weight four and negative examples weight one. Find the probability minimizing weighted BCE. Is it the original positive rate? Derive the formula that recovers $𝑟$ from the weighted optimum.
Logits and shifting. For class-1 target and logits $(1000, 1001, 999)$ , compute a stable expression for the loss and its approximate value. Explain why subtracting 1000 from all logits changes nothing.
Misspecified Gaussian uncertainty. Fit a normal mean with fixed model variance $𝑣$ to i.i.d. observations with true mean $𝑚_{0}$ and finite true variance $𝜏^{2}$ , not necessarily Gaussian. Find the pseudo-true mean, $𝐴$ , $𝐵$ , and the asymptotic variance of the fitted mean.
Nonregular endpoint. For i.i.d. uniform $[0, 𝜃_{0}]$ observations, derive $𝑃 (𝑀 \leq 𝑚)$ , $𝔼 [𝑀]$ , and a bias-corrected estimator from the sample maximum. Is that correction still the MLE?

15 Solutions

15.1 Bernoulli boundaries

The sample has $𝑛 = 5$ , $𝑘 = 4$ , so $\hat{𝜃} = \frac{4}{5}$ . Strict concavity of the log-likelihood on $(0, 1)$ proves uniqueness. The minimum average NLL is

- \frac{4}{5} ln (\frac{4}{5}) - \frac{1}{5} ln (\frac{1}{5}) \approx 0.500402 .

For five successes, $𝐿 (𝜃) = 𝜃^{5}$ . On $[0, 1]$ the MLE is one. On $(0, 1)$ there is no maximizer: the supremum is approached as $𝜃$ tends to one from below. The limiting average NLL is zero.

15.2 A categorical zero

The MLE is $(0.6, 0.3, 0.1, 0)$ and the training cross-entropy is $- 0.6 ln 0.6 - 0.3 ln 0.3 - 0.1 ln 0.1 \approx 0.897946$ . The fourth term contributes zero, not an undefined product. Finite softmax logits give strictly positive probabilities in every class. One can approach this distribution by sending the fourth relative logit to negative infinity, but cannot represent it exactly with finite logits.

15.3 Poisson estimation

Writing $𝑇 = \sum_{𝑖} 𝑧_{𝑖}$ , for $𝜆 > 0$ ,

ℓ_{𝑛} (𝜆) = - 𝑛 𝜆 + 𝑇 ln 𝜆 - \sum_{𝑖} ln (𝑧_{𝑖!}) .

If $𝑇 > 0$ , the derivative $- 𝑛 + \frac{𝑇}{𝜆}$ vanishes at $\frac{𝑇}{𝑛}$ and the second derivative $- \frac{𝑇}{𝜆^{2}}$ is negative. The sample sum is $16$ and the sample size is $8$ , giving $\hat{𝜆} = 2$ . The factorial terms may be dropped for optimization because they do not involve $𝜆$ .

If $𝑇 = 0$ , every observation is zero. The likelihood is $𝑒^{- 𝑛 𝜆}$ , maximized at $𝜆 = 0$ in the stated closed parameter space. If the model required $𝜆 > 0$ , that sample would have no MLE.

15.4 Known and unknown Gaussian means

With known mean zero, maximize over $𝑣 > 0$ to get $\hat{𝑣} = \frac{1^{2} + 2^{2} + 3^{2}}{3} = \frac{14}{3}$ . With a fitted mean, $\hat{𝑚} = 2$ and $\hat{𝑣} = \frac{{(1 - 2)}^{2} + {(2 - 2)}^{2} + {(3 - 2)}^{2}}{3} = \frac{2}{3}$ . The known-mean estimator measures squared deviations from zero, not from the sample mean. The term $- 𝑛 ln \frac{𝑣}{2}$ penalizes spreading the density too widely. Without it, the retained residual term would favor $𝑣$ tending to infinity rather than the correct finite optimum.

15.5 KL asymmetry

Direct substitution gives

𝐷_{KL} (𝑝 ∥ 𝑞) = 0.9 ln 1.8 + 0.1 ln 0.2 \approx 0.368064,

𝐷_{KL} (𝑞 ∥ 𝑝) = 0.5 ln (\frac{5}{9}) + 0.5 ln 5 \approx 0.510826 .

Data drawn from $𝑝$ weight the loss contributions by $𝑝$ , so the excess expected log loss is $𝐷_{KL} (𝑝 ∥ 𝑞)$ . Reversing the arguments changes the averaging distribution and answers a different question.

15.6 One logistic update

The initial logit is zero and $𝑝 = \frac{1}{2}$ . The loss is $ln 2 \approx 0.693147$ . The gradient is

(𝑝 - 𝑦) 𝒙 = - \frac{1}{2} (1, 2) = (- 0.5, - 1) .

A descent step gives $𝒘_{new} = (0.05, 0.1)$ . The new logit for this example is $0.25$ , so its positive-class probability is $𝜎 (0.25) \approx 0.562177$ , and its loss falls to about $0.575939$ . This demonstrates the update direction; it is not a proof that an arbitrary learning rate decreases every dataset’s loss.

15.7 Class weighting

With $𝛼 = 4$ , $𝛽 = 1$ , and $𝑟 = 0.2$ ,

𝑝_{𝑤}^{⋆} = \frac{4 \cdot 0.2}{4 \cdot 0.2 + 0.8} = 0.5 .

That is not the original rate $0.2$ . Solving the general optimum formula for $𝑟$ gives

𝑟 = \frac{𝛽 𝑝_{𝑤}^{⋆}}{𝛼 (1 - 𝑝_{𝑤}^{⋆}) + 𝛽 𝑝_{𝑤}^{⋆}} .

This inversion applies when the predicted value really is the corresponding weighted conditional optimum and the weighting scheme is known; it is not a universal calibration guarantee for an imperfect fitted model.

15.8 Logits and shifting

The maximum logit is $1001$ . Stable log-sum-exp yields

ℒ︀ = 1001 + ln (𝑒^{- 1} + 1 + 𝑒^{- 2}) - 1000 = 1 + ln (1 + 𝑒^{- 1} + 𝑒^{- 2}) \approx 1.407606 .

Subtracting a common constant multiplies numerator and denominator of softmax by the same factor. It cancels in probabilities and in the log-sum-exp-minus-target-logit expression. No large positive exponential needs to be evaluated.

15.9 Misspecified Gaussian uncertainty

The population NLL, apart from a constant, is $𝔼_{0} \frac{[{(𝑍 - 𝑚)}^{2}]}{2 𝑣}$ . Since $𝔼_{0} [{(𝑍 - 𝑚)}^{2}] = 𝜏^{2} + {(𝑚 - 𝑚_{0})}^{2}$ , its unique minimizer is $𝑚_{0}$ . The score is $\frac{𝑍 - 𝑚}{𝑣}$ and its negative derivative is $\frac{1}{𝑣}$ , so

𝐴 = \frac{1}{𝑣}, 𝐵 = \frac{𝜏^{2}}{𝑣^{2}}, 𝐴^{- 1} 𝐵 𝐴^{- 1} = 𝜏^{2} .

Thus $\sqrt{𝑛} (\hat{𝑚} - 𝑚_{0})$ converges in distribution to $𝒩︀ (0, 𝜏^{2})$ by the central limit theorem. The fitted mean is the sample mean, with asymptotic variance $\frac{𝜏^{2}}{𝑛}$ , not necessarily $\frac{𝑣}{𝑛}$ . The assumed model variance cannot replace the true variance without a justification such as correct specification of that variance.

15.10 Nonregular endpoint

For $0 \leq 𝑚 \leq 𝜃_{0}$ , independence gives

𝑃 (𝑀 \leq 𝑚) = \prod_{𝑖} 𝑃 (𝑍_{𝑖} \leq 𝑚) = {(\frac{𝑚}{𝜃_{0}})}^{𝑛} .

Its density on that interval is $𝑛 \frac{𝑚^{𝑛 - 1}}{𝜃_{0}^{𝑛}}$ . Therefore

𝔼 [𝑀] = \int_{0}^{𝜃_{0}} 𝑚 \frac{𝑛 𝑚^{𝑛 - 1}}{𝜃_{0}^{𝑛}} d 𝑚 = \frac{𝑛}{𝑛 + 1} 𝜃_{0} .

The estimator $(\frac{𝑛 + 1}{𝑛}) 𝑀$ is unbiased. For $𝑀 > 0$ it is not the MLE, because it exceeds the smallest admissible endpoint $𝑀$ , where the likelihood is largest. An unbiasedness correction changes the optimization criterion’s answer; unbiasedness and maximum likelihood are not synonyms.

16 Further reading

George Casella and Roger L. Berger, Statistical Inference, second edition (2002): likelihood, point estimation, and large-sample methods.
A. W. van der Vaart, Asymptotic Statistics (1998), Chapter 5: M-estimators and rigorous consistency and asymptotic-normality arguments.
Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, second edition (2006), Chapter 2: entropy, relative entropy, and information inequalities.
Christopher M. Bishop, Pattern Recognition and Machine Learning (2006), Chapters 3–4: regression, probabilistic classification, and cross-entropy objectives.

Final takeaway. Choose a probabilistic observation model first. Its likelihood determines the log-loss; the data distribution determines the expectation; and the parameter space determines which optima exist. MLE, cross-entropy, and KL divergence then connect precisely, without erasing the distinctions that make each concept useful.