Pdf Barzilai Borwein Based Adaptive Learning Rate For Deep Learning

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.© Copyright 2025 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions. The Barzilai-Borwein(BB) method is essential in solving unconstrained optimization problems. The momentum method accelerates optimization algorithms with exponentially weighted moving average. In order to design reliable deep learning optimization algorithms, this paper proposes applying the BB method in four variants to the optimization algorithm of deep learning. The momentum method generates the BB step size under different step range limits.

We also apply the momentum method and its variants to the stochastic gradient descent with the BB step size. The algorithm's robustness has been demonstrated through experiments on the initial learning rate and random seeds. The algorithm's sensitivity is tested by choosing different momentum factors until a suitable momentum factor is found. Moreover, we compare our algorithms with popular algorithms in various neural networks. The results show that the new algorithms improve the efficiency of the BB step size in deep learning and provide a variety of optimization algorithm choices. arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

People Also Search

• PDF Barzilai-Borwein-based Adaptive Learning Rate for Deep Learning
• Barzilai-Borwein-based adaptive learning rate for deep learning
• Incorporating the Barzilai-Borwein Adaptive Step Size into Sugradient ...
• The Improved Adaptive Algorithm of Deep Learning with Barzilai-Borwein ...
• Adaptive learning rate algorithms based on the improved Barzilai ...
• [2205.13711] Incorporating the Barzilai-Borwein Adaptive Step Size into ...