- Kernel methods allow linear learning algorithms to be applied to non-linear problems by mapping data into high-dimensional feature spaces. This overcomes limitations of insufficient capacity in linear models.
- Both frequentist and Bayesian approaches to kernel methods have been developed. Frequentist approaches analyze generalization error, while Bayesian approaches use probabilistic modeling and inference.
- Support vector machines optimize the margin between classes in feature space, improving generalization. The dual formulation allows computation using kernel functions without explicitly working in feature space.
1. Kernel Methods: the Emergence of a Well-founded Machine Learning John Shawe-Taylor Centre for Computational Statistics and Machine Learning University College London
54. Data size 342 Surprising fact: kernel methods are invariant to rotations of the coordinate system – so any special information encoded in the choice of coordinates will be lost! Surprisingly one of the most successful applications of SVMs has been for text classification in which one would expect the encoding to be very informative!
55.
56.
57.
58.
59.
60.
61.
62.
63.
64. Examples of bound evaluation 0.212 ±.002 0.198 ±.002 0.254 ±.003 0.334 ±.005 PAC-Bayes 0.109 ±.024 0.151 ±.005 0.184 ±.010 0.306 ±.018 PAC-Bayes prior 0.026 ±.005 Ringnorm 0.089 ±.005 Waveform 0.074 ±.014 (0.056 ±.01) Image 0.073 ±.021 Wdbc Test Error Problem Surprising fact: optimising the bound does not typically improve the test error – despite significant improvements in the bound itself! Training an SVM on half the data to learn a prior and then using the rest to learn relative to this prior further improves the bound with almost no effect on the test error!
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76. aligned text E 1 E 2 E N E i . . . . F 1 F 2 F N F i . . . .
77. Canadian parliament corpus LAND MINES Ms. Beth Phinney (Hamilton Mountain, Lib.): Mr. Speaker, we are pleased that the Nobel peace prize has been given to those working to ban land mines worldwide. We hope this award will encourage the United States to join the over 100 countries planning to come to … LES MINES ANTIPERSONNEL Mme Beth Phinney (Hamilton Mountain, Lib.): Monsieur le Président, nous nous réjouissons du fait que le prix Nobel ait été attribué à ceux qui oeuvrent en faveur de l'interdiction des mines antipersonnel dans le monde entier. Nous espérons que cela incitera les Américains à se joindre aux représentants de plus de 100 pays qui ont l'intention de venir à … E 12 F 12
78. cross-lingual lsi via svd M. L. Littman, S. T. Dumais, and T. K. Landauer. Automatic cross-language information retrieval using latent semantic indexing. In G. Grefenstette, editor, Cross-language information retrieval . Kluwer, 1998.
79. cross-lingual kernel canonical correlation analysis input “English” space input “French” space f F 1 f F 2 Φ(x) feature “English” space feature “French” space f E 1 f E 2
82. pseudo query test E i q e i F 1 F 2 F N F i . . . . Queries were generated from each test document by extracting 5 words with the highest TFIDF weights and using them as a query.
92. Idealised view of progress Study problem to develop theoretical model Derive analysis that indicates factors that affect solution quality Translate into optimisation maximising factors – relaxing to ensure convexity Develop efficient solutions using specifics of the task
93.
94.
Notas do Editor
The combined image and text database is obtained from the Internet by searching for images and downloading adjacent text. Images less then 72x72 were discarded 192 image textures features, 768 image colour features and 3591 text features (terms). [was retrieved from www.yahoo.com and www.warpig.com]
![Overview ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Surprising fact: there are now 13,444 citations of Support Vectors reported by Google scholar – the vast majority from papers not about machine learning! Ratio NNs/SVs in Google scholar 1987-1996: 220 1997-2006: 11](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)