brand banner

MAE vs MSE: The Nudge Argument

Why the median minimises absolute error and the mean minimises squared error, by nudging the guess.

Updated Β· 3 min read Β· #statistics Β· #machine-learning

MAE vs MSE: The Nudge Argument

Why the median minimises absolute error and the mean minimises squared error, by nudging the guess.

Summarise data π‘₯1,…,π‘₯𝑛 with a single guess 𝑐. Which 𝑐 is best depends on the loss:

𝐿1(𝑐)=βˆ‘π‘–|π‘₯π‘–βˆ’π‘|,𝐿2(𝑐)=βˆ‘π‘–(π‘₯π‘–βˆ’π‘)2.

Nudge test: move 𝑐 by a small 𝑑π‘₯>0 and ask whether the loss goes up or down. At the optimum, no nudge in either direction helps.

Absolute error: only the side matters

Each term |π‘₯π‘–βˆ’π‘| changes by exactly ±𝑑π‘₯. It does not matter how far away π‘₯𝑖 is, only which side it is on:

Δ𝐿1β‰ˆπ‘‘π‘₯(𝑛leftβˆ’π‘›right).

Points you move away from cost 𝑑π‘₯ each; points you move toward save 𝑑π‘₯ each.

At the median (odd 𝑛), the other points split π‘›βˆ’12 on each side, so their changes cancel exactly. The median’s own term goes from 0 to 𝑑π‘₯, so the net change is +𝑑π‘₯ in either direction. Any nudge hurts. For even 𝑛 there is no median term, and every 𝑐 between the two middle values is equally optimal.

Squared error: the size of the residual matters

The derivative of π‘Ÿ2 is 2π‘Ÿ, so each point pulls with a force proportional to its residual:

Δ𝐿2β‰ˆβˆ’2𝑑π‘₯βˆ‘π‘–(π‘₯π‘–βˆ’π‘).

Two consequences:

  1. Leaving the zero-error point is almost free. The median’s term goes from 0 to 𝑑π‘₯2, which is second order. Under 𝐿1 leaving it cost 𝑑π‘₯, which is first order and enough to block the move.
  2. What drives the move is imbalance, not big residuals in general. If one side’s residuals add up to more than the other’s, that side pulls harder. The optimum is where βˆ‘π‘–(π‘₯π‘–βˆ’π‘)=0, i.e. 𝑐=1π‘›βˆ‘π‘–π‘₯𝑖, the mean.

If the data is symmetric, the residuals balance, mean = median, and there is no incentive to move. Skew and outliers create the imbalance, and squaring makes large residuals count for more. That is why the mean follows outliers and the median does not.

Example

Take 1,2,3,4,10. The median is 3 and the mean is 4.

𝑐𝐿1(𝑐)𝐿2(𝑐)
3 (median)2+1+0+1+7=114+1+0+1+49=55
3.52.5+1.5+0.5+0.5+6.5=11.56.25+2.25+0.25+0.25+42.25=51.25
4 (mean)3+2+1+0+6=129+4+1+0+36=50

Moving from 3 to 3.5 under 𝐿1, the two points on the left cost 0.5 each, the two on the right save 0.5 each, and the median term costs 0.5: net +0.5. Under 𝐿2 the single far point at 10 pulls with residual 7, outweighing the three small residuals on the left combined (2+1+0), so the loss falls until the residuals balance at the mean.

Pull per pointCost of leaving the zero-error pointOptimum
𝐿1Β±1 (sign only)𝑑π‘₯ (first order)median: equal counts each side
𝐿2∝ residual𝑑π‘₯2 (negligible)mean: residuals sum to zero


Β© 2026 James Yap

Personal Website and Knowledge Base