brand banner

The Kernel Trick (SVM)

Understanding the kernel trick in SVMs, focusing on the roles of alpha and the inference formula.

Updated ยท 3 min read ยท #comp3105 ยท #machine-learning

The Kernel Trick (SVM)

Understanding the kernel trick in SVMs, focusing on the roles of alpha and the inference formula.

Lecture slide comparing primal and dual SVM inference

This slide shows the switch from a modelโ€™s โ€œprimal formโ€ (using w) to its โ€œdual formโ€ (using ฮฑ), which is the key to using the kernel trick. Letโ€™s break down your questions intuitively.


๐Ÿค” What is Alpha (๐›ผ) Intuitively?

Intuitively, ๐›ผ is a vector of weights, where each weight ๐›ผ๐‘– tells you the importance of the i-th training example in defining the decision boundary.

This is a fundamental shift in perspective:

  • w (Primal weights): A weight vector where each element ๐‘ค๐‘— corresponds to a feature. It tells you how important the j-th feature (e.g., โ€œSizeโ€ or โ€œWeightโ€ from your last example) is to the modelโ€™s prediction.
  • ฮฑ (Dual weights): A weight vector where each element ๐›ผ๐‘– corresponds to a training data point. It tells you how important the i-th data point (e.g., a specific apple) is to the modelโ€™s prediction.

In an SVM, most of the ๐›ผ๐‘– values will be zero. The only non-zero ๐›ผ๐‘– values belong to the support vectors. This means the model is defined only by the most critical points on the margin.

So, to answer your question, ฮฑ does not replace w directly, but it provides an alternative way to construct it. The mathematical relationship is:

๐’˜=๐‘‹๐‘‡๐›ผ=โˆ‘๐‘–=1๐‘›๐›ผ๐‘–๐’™๐‘–

This formula says that the feature-weight vector w is simply a linear combination of the training data points, where the coefficients of that combination are the importance weights in ฮฑ.


๐Ÿ”Ž Why is Inference ๐‘ฆฬ‚=๐’Œ0๐‘‡๐›ผโˆ—?

This formula looks complex, but itโ€™s just the original prediction formula, ๐‘ฆฬ‚=๐’™0๐‘‡๐’˜, rewritten using the relationship we just defined. Letโ€™s walk through it.

1. Start with the original prediction formula: To predict a new point ๐’™0, we use:

๐‘ฆฬ‚=๐’™0๐‘‡๐’˜โˆ—

2. Substitute the dual form of w: We know that ๐’˜โˆ—=๐‘‹๐‘‡๐›ผโˆ—. Letโ€™s plug that in:

๐‘ฆฬ‚=๐’™0๐‘‡(๐‘‹๐‘‡๐›ผโˆ—)

3. Rearrange using linear algebra: Because of the properties of transpose, we can re-group the terms:

๐‘ฆฬ‚=(๐’™0๐‘‡๐‘‹๐‘‡)๐›ผโˆ—โ‡’๐‘ฆฬ‚=(๐‘‹๐’™0)๐‘‡๐›ผโˆ—

4. Define ๐’Œ0: The slide defines the term ๐‘˜0=๐‘‹๐’™0. Letโ€™s see what that actually is.

  • ๐‘‹ is the matrix where each row is a training example (๐‘ฅ1๐‘‡,๐‘ฅ2๐‘‡,โ€ฆ).
  • ๐‘ฅ0 is the new point we want to classify.
  • The product ๐‘‹๐’™0 results in a vector where the i-th element is the dot product of the i-th training point and the new point (๐‘ฅ๐‘–๐‘‡๐‘ฅ0).

So, ๐’Œ0 is a vector containing the dot products of your new point (๐’™0) with every single point in the training set. Itโ€™s a measure of similarity between the new point and all the old points.

5. The Final Formula: By substituting ๐‘˜0 back into the equation from step 3, we get:

๐‘ฆฬ‚=๐’Œ0๐‘‡๐›ผโˆ—

The reason itโ€™s ๐’Œ0๐‘‡ (transpose) is a notational convention for the dot product between two column vectors. The formula simply calculates a weighted sum, where you are summing the โ€œsimilarity scoresโ€ in ๐’Œ0, weighted by the โ€œimportance scoresโ€ in ๐›ผโˆ—.

This is powerful because the entire processโ€”both training and predictionโ€”now depends only on inner products (dot products), which allows us to swap them out for kernels.



ยฉ 2026 James Yap

Personal Website and Knowledge Base