# Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes

Source: [Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes](<https://www.sota2.com/research/paper/68fa76891e209131cea6d821>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:51:36.949Z

Authors: Jihao Andreas Lin, Shreyas Padhy, Bruno Mlodozeniec, Javier Antor\\'an, Jos\\'e Miguel Hern\\'andez-Lobato

Published: 2024-05-28T16:58:37.000Z

## Abstract

Scaling hyperparameter optimisation to very large datasets remains an open problem in the Gaussian process community. This paper focuses on iterative methods, which use linear system solvers, like conjugate gradients, alternating projections or stochastic gradient descent, to construct an estimate of the marginal likelihood gradient. We discuss three key improvements which are applicable across solvers: (i) a pathwise gradient estimator, which reduces the required number of solver iterations and amortises the computational cost of making predictions, (ii) warm starting linear system solvers with the solution from the previous step, which leads to faster solver convergence at the cost of negligible bias, (iii) early stopping linear system solvers after a limited computational budget, which synergises with warm starting, allowing solver progress to accumulate over multiple marginal likelihood steps. These techniques provide speed-ups of up to $72\\times$ when solving to tolerance, and decrease the average residual norm by up to $7\\times$ when stopping early.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Marginal Likelihood Estimation on POL (mean over 10 splits)](<https://www.sota2.com/research/sota/marginal-likelihood-estimation-on-pol-mean-over-10-splits.md>): Marginal Likelihood Estimation; POL (mean over 10 splits); Test Log-Likelihood: 1.27.

- [Marginal Likelihood Estimation on ELEV mean over 10 splits](<https://www.sota2.com/research/sota/marginal-likelihood-estimation-on-elev-mean-over-10-splits.md>): Marginal Likelihood Estimation; ELEV mean over 10 splits; Test Log-Likelihood: -0.39.

- [Marginal Likelihood Estimation on BIKE (mean over 10 splits)](<https://www.sota2.com/research/sota/marginal-likelihood-estimation-on-bike-mean-over-10-splits.md>): Marginal Likelihood Estimation; BIKE (mean over 10 splits); Test Log-Likelihood: 2.15.

- [Marginal Likelihood Estimation on PROT mean over 10 splits](<https://www.sota2.com/research/sota/marginal-likelihood-estimation-on-prot-mean-over-10-splits.md>): Marginal Likelihood Estimation; PROT mean over 10 splits; Test Log-Likelihood: -0.59.

- [Marginal Likelihood Estimation on KEGG](<https://www.sota2.com/research/sota/marginal-likelihood-estimation-on-kegg.md>): Marginal Likelihood Estimation; KEGG; Test Log-Likelihood: 1.08.

- [Marginal Likelihood Estimation on Aggregate (POL, ELEV, BIKE, PROT, KEGG)](<https://www.sota2.com/research/sota/marginal-likelihood-estimation-on-aggregate-pol-elev-bike-prot-kegg.md>): Marginal Likelihood Estimation; Aggregate (POL, ELEV, BIKE, PROT, KEGG); Average Speed-Up: 7.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2405.18457>)

[PDF](<https://arxiv.org/pdf/2405.18457.pdf>)
