EvoLved Sign Momentum (Lion) is a Deep Learning stochastic optimization algorithm that builds upon the concept of sign momentum optimization. Lion, proposed in a research paper in 2023, aims to address the limitations of traditional optimization algorithms such as Stochastic Gradient Descent (SGD) and AdamW.
Lion is similar to AdamW with slight adjustments. Like sign momentum optimization, Lion only considers the sign of the gradient rather than the magnitude.
Lion also incorporates population-based training, where models with different hyperparameters are maintained and evolved over time. This helps to find better hyperparameters and improve the model's overall performance.
In experiments conducted by the developers, Lion was shown to outperform traditional optimization algorithms such as SGD and Adam on several benchmark datasets. Lion was also found to be more robust to changes in hyperparameters and more resistant to overfitting.
Noteworthy, developers highlight that Lion likely performs no better than AdanW if the batch size is less than 64. So, they suggest increasing the batch size as the advantage of Lion over AdamW enlarges as the batch size increases.
Overall, Lion is a promising optimization algorithm that has shown competitive performance compared to other state-of-the-art optimization algorithms on various deep learning tasks, including image classification, object detection, and natural language processing.
To get a deeper understanding of the optimizer, please refer to the original paper
Currently, Lion is not a built-in option for any Deep Learning framework. However, you can quickly test it on your data through other frameworks-compatible libraries
from torch import nn
model = nn.Linear(10, 1)
from lion_pytorch import Lion
opt = Lion(model.parameters(), lr=1e-4, weight_decay=1e-2)
loss = model(torch.randn(10))