Adaptive Learning Rate Algorithms Based On The Improved Barzilai

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.© Copyright 2025 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions. The Barzilai-Borwein(BB) method is essential in solving unconstrained optimization problems. The momentum method accelerates optimization algorithms with exponentially weighted moving average. In order to design reliable deep learning optimization algorithms, this paper proposes applying the BB method in four variants to the optimization algorithm of deep learning. The momentum method generates the BB step size under different step range limits.

We also apply the momentum method and its variants to the stochastic gradient descent with the BB step size. The algorithm's robustness has been demonstrated through experiments on the initial learning rate and random seeds. The algorithm's sensitivity is tested by choosing different momentum factors until a suitable momentum factor is found. Moreover, we compare our algorithms with popular algorithms in various neural networks. The results show that the new algorithms improve the efficiency of the BB step size in deep learning and provide a variety of optimization algorithm choices.

People Also Search

• Adaptive learning rate algorithms based on the improved Barzilai ...
• The Improved Adaptive Algorithm of Deep Learning with Barzilai-Borwein ...
• PDF Barzilai-Borwein-based Adaptive Learning Rate for Deep Learning
• Incorporating the Barzilai-Borwein Adaptive Step Size into Sugradient ...
• Adaptive learning rate optimization algorithms with dynamic bound based ...