The Kernel Trick (SVM)
Understanding the kernel trick in SVMs, focusing on the roles of alpha and the inference formula.

This slide shows the switch from a modelโs โprimal formโ (using w) to its โdual formโ (using ฮฑ), which is the key to using the kernel trick. Letโs break down your questions intuitively.
๐ค What is Alpha () Intuitively?
Intuitively, is a vector of weights, where each weight tells you the importance of the i-th training example in defining the decision boundary.
This is a fundamental shift in perspective:
- w (Primal weights): A weight vector where each element corresponds to a feature. It tells you how important the j-th feature (e.g., โSizeโ or โWeightโ from your last example) is to the modelโs prediction.
- ฮฑ (Dual weights): A weight vector where each element corresponds to a training data point. It tells you how important the i-th data point (e.g., a specific apple) is to the modelโs prediction.
In an SVM, most of the values will be zero. The only non-zero values belong to the support vectors. This means the model is defined only by the most critical points on the margin.
So, to answer your question, ฮฑ does not replace w directly, but it provides an alternative way to construct it. The mathematical relationship is:
This formula says that the feature-weight vector w is simply a linear combination of the training data points, where the coefficients of that combination are the importance weights in ฮฑ.
๐ Why is Inference ?
This formula looks complex, but itโs just the original prediction formula, , rewritten using the relationship we just defined. Letโs walk through it.
1. Start with the original prediction formula: To predict a new point , we use:
2. Substitute the dual form of w: We know that . Letโs plug that in:
3. Rearrange using linear algebra: Because of the properties of transpose, we can re-group the terms:
4. Define : The slide defines the term . Letโs see what that actually is.
- is the matrix where each row is a training example ().
- is the new point we want to classify.
- The product results in a vector where the i-th element is the dot product of the i-th training point and the new point ().
So, is a vector containing the dot products of your new point () with every single point in the training set. Itโs a measure of similarity between the new point and all the old points.
5. The Final Formula: By substituting back into the equation from step 3, we get:
The reason itโs (transpose) is a notational convention for the dot product between two column vectors. The formula simply calculates a weighted sum, where you are summing the โsimilarity scoresโ in , weighted by the โimportance scoresโ in .
This is powerful because the entire processโboth training and predictionโnow depends only on inner products (dot products), which allows us to swap them out for kernels.