Report: Deep Learning Workshop 2023
The 2-day Deep Learning Workshop with a focus on "Recent Advances in Kernel Methods for Neural Networks" took place in October 5&6, 2023 at the Triangel Space (Kronenplatz). The workshop was jointly presented by KCDS, the Institute of Economics (ECON) at KIT, MathSEE, and Helmholtz Information and Data Science Academy (HIDA).
The workshop was open to MathSEE and KCDS members as well as other interested (doctoral) researchers at KIT, other Helmholtz centers or other research institutions and universities. We were happy to welcome 25 participants from various institutes (including ECON) and graduate schools (including KCDS) at KIT and from other organizations such as Heidelberg Institute for Theoretical Studies (HITS).
The workshop was taught by visiting researcher Adit Radhakrishnan, who is currently a George F. Carrier Postdoctoral Fellow at Harvard. Adit completed his Ph.D. in electrical engineering and computer science (EECS) at MIT advised by Caroline Uhler and was a Ph.D. fellow at the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard. His research focuses on advancing theoretical foundations of machine learning in order to develop new methods for tackling biomedical problems.
The workshop program included theory and hands-on coding sessions, with an introduction and overview in the beginning, followed by different topics on Gradient descent analysis for linear regression, Kernel regression, Neural networks, Neural Network Gaussian Processes + Neural Tangent Kernels, Recursive Feature Machines, and finally some discussions on future directions. More detailed information about the contents can be found in the description below.
The Deep Learning Workshop was made possible with the friendly financial support of MathSEE workshop funding and HIDA course funding.
We'd like to thank everyone who contributed and participated in the workshop and look forward to future opportunities for such workshops! The lecturer Adit said it was a very interesting experience and he enjoyed the workshop a lot.
Report: Jieyu Chen
| Name | Role at KCDS |
|---|---|
| Hühnerfuß, Angela | Coordination KIT Graduate School Computational and Data Science |
| Koster, Nils | Doctoral Researcher at Institute of Economics (ECON) |
| Krüger, Fabian | SEE Supervisor |
| Lerch, Sebastian | MATH Supervisor |
| 1 additional person visible within KIT only. | |
Deep Learning - Recent Advances in Kernel Methods for Neural Networks
- Type: 2-day theory and hands-on workshop
-
Place:
Triangel.space at Kronenplatz, KIT Campus South
-
Time:
Oct 5 & 6, 2023
- Start: 10:00-18:00h
-
Lecturer:
Adit Radhakrishnan (MIT and Broad Institute)
- ECTS: 1
The Institute of Economics at KIT, MathSEE, KCDS and Helmholtz Information and Data Science Academy (HIDA) proudly present the Deep Learning workshop. The workshop is made possible with financial support by MathSEE workshop funding and HIDA course funding.
How to register
The workshop is open to MathSEE and KCDS members as well as other interested (doctoral) researchers at KIT, other Helmholtz centers or other research institutions and universities. As places are limited, available spots will be given to those with a corresponding research interest and strong motivation to join the workshop on a first-come-first-serve basis. Please fill out the application form below.
Motivation
Sparked by the success of deep neural networks in many practical applications, there has been an increasing interest in analyzing these methods from a statistical perspective. In recent years, advances in the theory of neural networks and kernels have brought both fields closer together. This not only sparked new interest and with that fresh ideas in the field of kernels, it also enabled research to explain phenomena occurring in neural networks. Double descent [Belkin et al., 2018], the usefulness of overparametrization (for example in Autoencoder architectures) [Radhakrishnan et al., 2018] or the Neural Tangent Kernel [Jacot et al., 2018] are among those. Notably, Radhakrishnan and co-authors [Radhakrishnan et al., 2022] showed that neural networks merely learn something called the ‘Expected Gradient Outer Product’ (EGOP). They showed that using the EGOP within a simple kernel framework outperforms not only neural networks but also methods such as Gradient Boosting and Random Forests in most cases.
Workshop Content
Adit will teach the mathematical reasoning behind the above-mentioned methods, but also how they are applied in practical applications (mainly from biology). In interactive Python coding sessions, participants will have the chance to develop these methods themselves. The overarching goal of the workshop therefore is to learn about a mathematically founded way of applying deep learning in practice and to disentangle myth and actual capabilities of deep learning and kernels. The power of these methods is that they make use of mechanisms in neural networks without the overhead of training them. These methods are hence applicable to many practical problems, with or without a plethora of available data.
References
- Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the bias-variance trade-off. Proceedings of the National Academy of Sciences of the United States of America, 116: 15849–15854, 2018.
- Adityanarayanan Radhakrishnan, Karren Yang, Mikhail Belkin, and Carolin Uhler. Memorization in overparameterized autoencoders. 2018.
- Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, pages 8571–8580, 2018.
- Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Feature learning in neural networks and kernel machines that recursively learn features, 2022.
