Using the Quasi-Newton Method to Solve Nonlinear Least Squares Regression Problems
Atallah, E., Hagag, A., Ali, E., Abassy, T. (2025). Using the Quasi-Newton Method to Solve Nonlinear Least Squares Regression Problems. EKB Journal Management System, 1(1), 9-15. doi: 10.21608/ijaici.2025.340085.1004
Esraa S. Atallah; Ahmed Hagag; Eman M. Ali; Tamer A. Abassy. "Using the Quasi-Newton Method to Solve Nonlinear Least Squares Regression Problems". EKB Journal Management System, 1, 1, 2025, 9-15. doi: 10.21608/ijaici.2025.340085.1004
Atallah, E., Hagag, A., Ali, E., Abassy, T. (2025). 'Using the Quasi-Newton Method to Solve Nonlinear Least Squares Regression Problems', EKB Journal Management System, 1(1), pp. 9-15. doi: 10.21608/ijaici.2025.340085.1004
Atallah, E., Hagag, A., Ali, E., Abassy, T. Using the Quasi-Newton Method to Solve Nonlinear Least Squares Regression Problems. EKB Journal Management System, 2025; 1(1): 9-15. doi: 10.21608/ijaici.2025.340085.1004

Using the Quasi-Newton Method to Solve Nonlinear Least Squares Regression Problems

International Journal of Applied Intelligent Computing and Informatics
Volume 1, Issue 1, May 2025, Page 9-15  XML PDF (633.93 K)
Document Type: Original Article
DOI: 10.21608/ijaici.2025.340085.1004
View on SCiNiTO View on SCiNiTO
Authors
Esraa S. Atallah1; Ahmed Hagag email orcid 2; Eman M. Ali1; Tamer A. Abassy1
1Faculty of Computers and Artificial Intelligence, Benha University, Benha, Qalyubia Governorate, Egypt.
2Faculty of Computers and Artificial Intelligence, Benha University, Benha, Qalyubia Governorate, Egypt.
Abstract
In regression modeling, Gradient Descent (GD) is widely used to update parameters by iteratively minimizing a cost function. However, GD often converges slowly and may suffer from instability due to its reliance on first-order derivatives only. To improve convergence speed and stability, Newton’s method utilizes second-order derivative information, aiming to find the point where the gradient vanishes. While Newton’s method offers faster convergence, computing the exact Hessian matrix is often computationally expensive or infeasible.

Quasi-Newton methods overcome this limitation by approximating the Hessian matrix. These methods iteratively update an estimate of the Hessian, typically denoted as H_k≈∇^2 f(x_k), to guide the search direction. A notable quasi-Newton algorithm is the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method.

In this paper, we apply BFGS and its variants—including Memoryless BFGS, Limited-memory BFGS (L-BFGS), and Scaled BFGS—to nonlinear least squares regression problems. Their performance is compared with traditional Gradient Descent and Newton’s method, focusing on convergence behavior and optimization efficiency. The results demonstrate the potential advantages of quasi-Newton approaches in practical regression scenarios.
Keywords
BFGS method; Gradient Descant; Newton method; Nonlinear least square; Quasi-Newton
Statistics
Article View: 91
PDF Download: 54