Crandi

New Optimization Algorithm Desgd For Improved Performance

Leo Migdal
-
new optimization algorithm desgd for improved performance

Scientific Reports volume 15, Article number: 40389 (2025) Cite this article In modern machine learning, optimization algorithms are crucial; they steer the training process by skillfully navigating through complex, high-dimensional loss landscapes. Among these, stochastic gradient descent with momentum (SGDM) is widely adopted for its ability to accelerate convergence in shallow regions. However, SGDM struggles in challenging optimization landscapes, where narrow, curved valleys can lead to oscillations and slow progress. This paper introduces dual enhanced SGD (DESGD), which addresses these limitations by dynamically adapting both momentum and step size on the same update rules of SGDM. In two optimization test functions, the Rosenbrock and Sum Square functions, the suggested optimizer typically performs better than SGDM and Adam.

For example, it accomplishes comparable errors while achieving up to 81–95% fewer iterations and 66–91% less CPU time than SGDM and 67–78% fewer iterations with 62–70% quicker runtimes than Adam. On the MNIST dataset, the proposed optimizer achieved the highest accuracies and lowest test losses across the majority of batch sizes. Compared to SGDM, they consistently improved accuracy by about 1–2%, while performing on par with or slightly better than Adam in accuracy and error. Although SGDM remained the fastest per-step optimizer, our method’s computational cost is aligned with that of other adaptive optimizers like Adam. This marginal increase in per-iteration overhead is decisively justified by the substantial gains in model accuracy and reduction in training loss, demonstrating a favorable cost-to-performance ratio. The results demonstrate that DESGD is a promising practical optimizer to handle scenarios demanding stability in challenging landscapes.

Machine learning (ML) optimization is critical to the development of models that exhibit efficiency, scalability and superior performance. With the continuous advancement of modern ML approaches, the necessity of optimization in the training of complex models, such as deep neural networks, is becoming more essential. Recent developments, such as gradient-based approaches, adaptive learning rate strategies and stochastic optimization techniques, have greatly enhanced the performance of models in various applications like disease diagnosis1,2,3, photovoltaic power forecasting4,5, large language models training6... In addition, improving model performance plays a crucial role in enhancing computational efficiency, resulting in reduced training time, lower resource utilization and an increase in the accessibility and deployment of ML solutions in real... Thus, the area of ML optimization remains a promising research area, offering significant ideas that enhance the progress of artificial intelligence across diverse sectors. One of the powerful methods used in ML optimization is the gradient descent (GD), which minimizes the loss function \(J(\theta )\) where \(\theta\) are the model’s parameters by updating them in the negative direction...

The step size to reach a local minimum is found by the learning rate \(\alpha\). The GD method has different variants according to the number of data samples that will be fed into the optimization process. Stochastic gradient descent (SGD) performs the parameters update using one data sample at a time instead of using all the data samples and completes one epoch i.e. one iteration, after finishing all the data samples. I’m excited to share our recently published research paper, now appearing in Scientific Reports (Nature Portfolio): 📄 “A Dual Enhanced Stochastic Gradient Descent Method with Dynamic Momentum and Step Size Adaptation for Improved Optimization... Specifically: 1) Adaptive Momentum using a conjugate-gradient–inspired update, with stability ensured through truncation schemes.

2) Adaptive Step Size using gradient alignment (cosine similarity), which is line-search-free and computationally lightweight. Our proposed optimizer shows faster and more stable convergence than traditional methods, requiring fewer training iterations while maintaining a competitive computational cost. It delivers strong, consistent performance across both classical optimization benchmarks and a neural network experiment, highlighting its potential as a practical and efficient alternative in challenging optimization settings. This work opens the door for more robust and efficient training—especially in high-curvature optimization landscapes. This paper represents our proof of concept, and we plan to expand this idea into more advanced models, larger datasets, and real-world applications in future work. I am deeply grateful to my supervisors and co-authors, Dr.

Mohamed Fathy , Dr. Yasser Dahab, and Dr. Emad Abdallah, for their invaluable support and dedicated effort. 🔗 Open-access full paper: Scientific Reports (2025) https://lnkd.in/dDa5CFDy Being my student makes me really proud, and I wish you continued success and personal development. Optimization algorithms are essential tools used in various fields to find the best solution from a set of possible choices.

These algorithms help in making decisions, improving processes, and solving complex problems. In recent years, researchers have focused on how these algorithms work over time, treating them as systems that evolve continuously. This perspective allows for a deeper understanding of how different methods can be developed and improved. When working with optimization algorithms, it's crucial to ensure that they reach an optimal solution efficiently. One way to evaluate how well they do this is through different types of stability: Exponential Stability: This means that if the algorithm starts close to the optimal solution, it will quickly converge to that solution as time goes on.

The closer it starts, the faster it will reach the target. Finite-Time Stability: In this case, the algorithm is designed to reach the optimal solution within a specific timeframe, regardless of where it starts. This approach guarantees that all initial conditions lead to a solution within a predictable time limit. Fixed-time Stability: Similar to finite-time stability, this type requires that the algorithm always settles to the optimal solution within the same time, no matter the starting point. This ensures consistency in performance. You have full access to this open access article

The Archimedes Optimization Algorithm (AOA) is a recent physics-based metaheuristic inspired by Archimedes’ principle. Since its introduction by Hashim et al. in 2021, it has gained significant attention and has been applied to various real-world optimization problems. Its popularity stems from its simple structure, adaptability, ease of implementation, and satisfactory convergence. This paper presents a comprehensive review of the AOA algorithm, including its modified, multi-objective, and hybrid variants, and examines its applications in several domains such as classification, feature selection, parameter tuning, scheduling, photovoltaic systems,... The performance of the AOA algorithm is evaluated against some well-known metaheuristic algorithms, including Genetic Algorithm (GA), Differential Evolution (DE), Tabu Search (TS), Firefly Algorithm (FA), Bat Algorithm (BA), Whale Optimization Algorithm (WOA), Grey...

Finally, future research directions for improving the effectiveness and applicability of the AOA algorithm are outlined. Avoid common mistakes on your manuscript. Optimization is the process of determining the best possible value of a fitness function while adhering to multiple constraints. It has recently emerged as one of the most interesting issues for solving several real-world problems across various domains. Optimization techniques can generally be divided into two main categories: traditional optimization methods and meta-heuristic optimization methods. Traditional optimization methods, also referred to as conventional optimization techniques, are known for their fast convergence and potential to yield highly accurate optimal solutions.

However, they typically require stringent conditions, including fully defined constraints and continuously differentiable objective functions. In practice, many real-world problems are complex and nonlinear, making traditional methods susceptible to getting trapped in local optima. Furthermore, these methods often struggle with high-dimensional problems and complex search spaces, where they may fail to find the global optimum. To address the limitations of traditional optimization methods, researchers have increasingly turned to meta-heuristic algorithms. Meta-heuristics are intelligent, computer-based techniques that tackle complex optimization problems using iterative search strategies inspired by natural behaviors, biological processes, physical phenomena, or social interactions. These algorithms simulate such mechanisms to guide the search toward optimal or near-optimal solutions.

Unlike traditional methods, meta-heuristics do not rely on gradient information or require the objective function to be differentiable. Owing to their adaptability and robustness, they have gained popularity among researchers and have been successfully applied to a wide range of complex optimization problems. Meta-heuristics can be categorized into two main classes: those based on a single solution and those based on a population of solutions, as shown in Fig. 1. This chapter presents a brief overview of the most recent optimization algorithms. The chapter starts by showing the difference between classical methods and modern methods.

And then, some special types of problems solved by optimization algorithms are presented, including some open problems. After that, the initial modern optimization algorithms are listed; the recent optimization algorithms are shown with their main features. This section starts with the algorithms inspired by quantum mechanics and their solutions to intelligent systems. Also, this section shows other modern approaches merging intelligent techniques, neuroscience, data science, and machine learning. *Address all correspondence to: germanoltorres@gmail.com Optimization is a fundamental process in several areas of knowledge, such as engineering, economics, computer science, and mathematics, aiming to find the best solution or the optimal point for a given problem.

This process involves maximizing or minimizing an objective function, representing the metric to be improved, subject to constraints that delimit the space of viable solutions. Optimization can be applied in varied contexts, from improving industrial processes to efficiently allocating resources to complex projects [1]. There are different optimization techniques, which can be classified into classical methods and modern methods. Classical methods include linear programming, nonlinear programming, and differential calculus, which are widely used in continuous and differentiable problems. Using intelligent systems, modern techniques are often employed in complex, nonlinear, or multi-optimal problems. The choice of the appropriate method depends on the characteristics of the problem, such as the nature of the variables, the presence of constraints, and the complexity of the objective function [2, 3].

With the advancement of technology and the increase in computing capacity, optimization has become increasingly relevant, especially with the emergence of large volumes of data and the need to make quick and accurate decisions. Optimization tools are essential in applications such as machine learning, where models are tuned to minimize forecasting errors, or logistics, where routes and resources are planned to reduce costs and time. Optimization is crucial in the search for efficiency and innovation, contributing to developing more competent and sustainable solutions in an increasingly complex world [4]. Scientific Reports , Article number: (2025) Cite this article We are providing an unedited version of this manuscript to give early access to its findings. Before final publication, the manuscript will undergo further editing.

Please note there may be errors present which affect the content, and all legal disclaimers apply. This research proposes an innovative methodology for acurate remore sensing scene classification. Here, a new design of dynamic arithmetic optimization algorithm (DAOA) has been proposed to enhance the performance of a Ridgelet neural network (RNN) in this purpose. The RNN is commonly used in image processing and computer vision, but its effectiveness can be hindered by subpar hyperparameter selection. To tackle this problem, the utilization of DAOA has been proposed as a robust optimization technique to automatically search for optimal hyperparameters in the RNN model. The proposed method has been assessed on UC Merced Land Use publicly available dataset frequently employed in remote sensing scene classification.

The experimental results show that the proposed approach significantly enhances the efficiency of the RNN when compared to other cutting-edge methods. The findings indicate that the combination of optimization algorithms like DAOA with deep learning models such as RNNs has the potential to yield more precise and efficient solutions for remote sensing scene classification tasks. The datasets used and/or analyzed during the current study available from the corresponding author on reasonable request. Based on the university limitations, this code cant be access freely. However, enough information about the code and mathematical formulations can be achieved by contacting with corresponding author. A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.© Copyright 2025 IEEE - All rights reserved.

People Also Search

Scientific Reports Volume 15, Article Number: 40389 (2025) Cite This

Scientific Reports volume 15, Article number: 40389 (2025) Cite this article In modern machine learning, optimization algorithms are crucial; they steer the training process by skillfully navigating through complex, high-dimensional loss landscapes. Among these, stochastic gradient descent with momentum (SGDM) is widely adopted for its ability to accelerate convergence in shallow regions. However,...

For Example, It Accomplishes Comparable Errors While Achieving Up To

For example, it accomplishes comparable errors while achieving up to 81–95% fewer iterations and 66–91% less CPU time than SGDM and 67–78% fewer iterations with 62–70% quicker runtimes than Adam. On the MNIST dataset, the proposed optimizer achieved the highest accuracies and lowest test losses across the majority of batch sizes. Compared to SGDM, they consistently improved accuracy by about 1–2%,...

Machine Learning (ML) Optimization Is Critical To The Development Of

Machine learning (ML) optimization is critical to the development of models that exhibit efficiency, scalability and superior performance. With the continuous advancement of modern ML approaches, the necessity of optimization in the training of complex models, such as deep neural networks, is becoming more essential. Recent developments, such as gradient-based approaches, adaptive learning rate st...

The Step Size To Reach A Local Minimum Is Found

The step size to reach a local minimum is found by the learning rate \(\alpha\). The GD method has different variants according to the number of data samples that will be fed into the optimization process. Stochastic gradient descent (SGD) performs the parameters update using one data sample at a time instead of using all the data samples and completes one epoch i.e. one iteration, after finishing...

2) Adaptive Step Size Using Gradient Alignment (cosine Similarity), Which

2) Adaptive Step Size using gradient alignment (cosine similarity), which is line-search-free and computationally lightweight. Our proposed optimizer shows faster and more stable convergence than traditional methods, requiring fewer training iterations while maintaining a competitive computational cost. It delivers strong, consistent performance across both classical optimization benchmarks and a ...