Technology

How does KAN (Kolmogorov-Arnold Networks) act as a better replacement for Multi-Layer Perceptrons (MLPs)?

Multilayer perceptrons (MLPs), also known as fully connected feedforward neural networks, have played a significant role in modern deep learning. Since the universal approximation theorem guarantees expressiveness, they are often used to approximate nonlinear functions. MLPs are common; However, they have disadvantages such as high parameter consumption and poor interpretability in complex models such as transformers.

Kolmogorov-Arnold networks (KANs), inspired by the Kolmogorov-Arnold representation theorem, offer a possible replacement to address these drawbacks. Similar to MLPs, KANs have a fully connected topology, but use a different approach by placing learnable activation functions on edges (weights) instead of learning fixed activation functions on nodes (neurons). A learnable 1D function, parameterized as a spline, takes on the role of each weight parameter in a KAN. As a result, KANs forego traditional linear weight matrices and their nodes aggregate incoming signals without going through nonlinear transformations.

Compared to MLPs, KANs are more efficient at creating smaller computational graphs, which helps offset their potential computational costs. For example, empirical data shows that a two-layer KAN with a width of 10 can achieve better accuracy (lower mean squared error) and parameter efficiency (fewer parameters) than a four-layer MLP with a width of 100.

When it comes to accuracy and interpretability, using splines as activation functions in KANs has several advantages over MLPs. When it comes to accuracy, smaller KANs can perform as well or even better than larger MLPs on tasks such as solving partial differential equations (PDE) and data fitting. This advantage is demonstrated both theoretically and experimentally, as KANs have faster neural network scaling laws compared to MLPs.

KANs also perform exceptionally well in interpretability, which is essential for understanding and using neural network models. Because KANs use structured splines to express functions more transparently and understandably than MLPs, they can be visualized intuitively. Because of its interpretability, the model and human users can collaborate more easily, resulting in better insights.

The team shared two examples showing how KANs can be useful tools for scientists to rediscover and understand complicated mathematical and physical laws: one from physics, which is Anderson localization, and one from the Mathematics, which is knot theory. Deep learning models can contribute more effectively to scientific research if KANs improve understanding of the underlying data representations and model behavior.

In summary, KANs represent a viable replacement for MLPs and leverage the Kolmogorov-Arnold representation theorem to overcome important limitations in neural network architecture. Compared to traditional MLPs, KANs exhibit better accuracy, faster scaling qualities, and better interpretability because they use learnable spline-based activation functions on edges. This development expands the possibilities for deep learning innovations and improves the capabilities of current neural network architectures.


Visit the Paper. All credit for this research goes to the researchers of this project. Also don’t forget to follow us Twitter. Join our… Telegram channel, Discord channelAnd LinkedIn GrOup.

If you like our work, you will love ours Newsletter..

Don’t forget to join our 41k+ ML SubReddit


Tanya Malhotra is a final year student at the University of Petroleum & Energy Studies, Dehradun, studying BTech in Computer Science with a specialization in Artificial Intelligence and Machine Learning.
She is a data science enthusiast with good analytical and critical thinking and a keen interest in learning new skills, leading groups and managing work in an organized manner.




Source link