Skip to content

[WIP] Overhaul statistics computation

Vicentini Filippo requested to merge pv/stats into master

Created by: PhilipVinc

This PR rewrites the logic computing the statistics of our Stats class in order to use single-pass algorithms that need traverse the data only once.

With this PR, once a Stats object is created, it can be Stats.merge(other_stats) with another object in order to combine the indicators.

This is a requirement for properly computing such statistical diagnostics when working with minibatches while keeping memory requirements contained.

My idea is to compute the tau_corr and error of the mean by using the variance among the chains if there is a sufficient number of chains, otherwise switch to the variance among blocks.

Interestingly, this implementation requires 1 less MPI call then before, so it might be marginally faster in MPI contexts when computing many observables because it reduces the number of sync points.

Merge request reports