Skip to yearly menu bar Skip to main content


Poster

A Universal Class of Sharpness-Aware Minimization Algorithms

Behrooz Tahmasebi · Ashkan Soleymani · Dara Bahri · Stefanie Jegelka · Patrick Jaillet


Abstract:

Recently, there has been a surge in interest in developing optimization algorithms for overparameterized models as achieving generalization is believed to require algorithms with suitable biases. This interest centers on minimizing sharpness of the original loss function; the Sharpness-Aware Minimization (SAM) algorithm has proven effective. However, existing literature focuses on only a few sharpness measures (such as the maximum eigenvalue/trace of the training loss Hessian).Moreover, many sharpness measures show sensitivity to parameter invariances in neural networks, e.g., they magnify significantly under rescaling parameters.Hence, here we introduce a new class of sharpness measures leading to sharpness-aware objective functions. We prove that these measures are \textit{universally expressive}, allowing any function of the training loss Hessian matrix to be represented by choosing appropriate hyperparameters. Furthermore, we show that the proposed objective functions explicitly bias towards minimizing their corresponding sharpness measures. Finally, we demonstrate how the structure of this new class allows meaningful applications to models with parameter invariances, including scale-invariant neural networks.

Live content is unavailable. Log in and register to view live content