Representation Learning

Background Modern deep neural networks, especially those in the overparameterized regime with a very large number of parameters, perform impressively well. Traditional learning theories contradict these empirical results and fail to explain this phenomenon, leading to new approaches that aim to understand why deep learning generalizes. A common belief is that flat minima [1] in the parameter space lead to models with good generalization properties. For instance, such models may learn to extract high-quality features from the data, known as representations. However, it has also been shown that models with equivalent performance can exist at sharp minima [2, 3]. These contradictory findings motivate us to study optimization, learned representations, and their impact on generalization from a geometric perspective. ...

Background Relative representations are a powerful tool in machine learning and data analysis, where data points are represented based on their distances or similarities to a set of reference points called anchors. Traditional methods often rely on similarity measures, such as cosine similarity, which are invariant under rotation and scaling but may not satisfy the properties of a metric space, particularly the triangle inequality. This limitation can hinder the effectiveness of certain algorithms that require a proper distance metric. ...