aeons: approximating the end of nested sampling

In our latest work, presented in “aeons: approximating the end of nested sampling,” we introduce a novel technique to forecast the runtime of nested sampling algorithms. This research, led by Zixiao Hu in collaboration with Artem Baryshnikov and Will Handley, addresses a significant practical challenge in computational statistics: predicting whether a complex analysis will complete in hours, days, or weeks.

Nested sampling, a cornerstone algorithm for Bayesian inference first proposed by John Skilling (10.1214/06-ba127), is instrumental in fields from cosmology to particle physics. However, its computational cost is often unknown beforehand, making resource management on supercomputing clusters and project planning difficult. Our new method, aeons, provides a principled solution by analysing the “anatomy” of a run in progress.

Dissecting the Nested Sampling Run

Rather than relying on crude heuristics, aeons builds a predictive model of the remaining posterior landscape using the information gathered mid-run. The core insight is to extrapolate the likelihood profile based on its inferred geometric properties, offering a robust forecast of the total computational effort required. This is achieved by examining several key aspects of the algorithm’s progression:

• Prior Volume Compression: The algorithm works by iteratively shrinking the prior volume X. The total compression required to map the prior to the posterior is determined by the Kullback-Leibler divergence, which quantifies the information gain. Our method models the relationship between X and the likelihood L(X) to predict the final, target prior volume.
• Effective Dimensionality and Temperature: A key innovation is the use of the Bayesian Model Dimensionality (BMD), a measure of the number of constrained parameters detailed in prior work (10.1103/physrevd.100.023512). We track the effective dimensionality as the algorithm progresses and introduce a “Bayesian temperature” to re-weight samples, revealing the posterior’s structure at any given stage. This allows us to approximate the full posterior as a high-dimensional Gaussian, even when we have only explored a fraction of it.
• Anisotropic Compression: Many real-world likelihoods, particularly in cosmology, are highly elongated. aeons accounts for this anisotropic compression, where different parameter directions are constrained at different rates. The method correctly anticipates that the full dimensionality of the problem may only become apparent late in the run and adjusts its forecast accordingly.

From Anatomy to Prediction

The practical implementation of aeons involves a three-step process executed at intervals during a run:

1. Infer the posterior’s current effective dimensionality using our Bayesian temperature framework.
2. Fit a multi-dimensional Gaussian profile to the existing live and dead points, constrained by this inferred dimensionality.
3. Analytically solve for the remaining prior volume (X_f) that must be compressed to meet the standard termination criterion.

This provides a robust estimate of the final iteration count. Our tests show this approach is far more stable than alternatives, such as extrapolating evidence increments or using the integral-based progress metric found in tools like Ultranest, especially for high-dimensional problems.

Validation on Cosmological Data

We tested aeons extensively on both toy models and real-world cosmological inference tasks. The method successfully predicts the endpoint for challenging, high-dimensional likelihoods, such as those encountered in analyses of Planck data. For instance, in tests mirroring the curvature quantification work described in 10.1103/physrevd.103.l041301, our predictions converged to the true endpoint within the standard error by the halfway point of the run, providing a correct order-of-magnitude estimate from the very beginning.

By transforming runtime prediction from guesswork into a principled statistical estimate, aeons provides a much-needed tool for the scientific community, enabling more efficient and predictable large-scale computational research.

Content generated by gemini-2.5-pro using this prompt.

Image generated by imagen-3.0-generate-002 using this prompt.