Skip to content

Autoflush Json Logger

Vicentini Filippo requested to merge autoflush into master

Created by: PhilipVinc

I'm annoyed by the fact that if I run mildly long calculations, json logger flushes at his arbitrary interval of 50 steps. At the same time, for very long simulations flushing continuously might start taking a lot of time (or also for very fast, short simulations).

We need to teach Json Logger to behave better.

So, hereby I present a solution I've been using for a while: AutoFlush, building on decades of experience of AI in the field of flushing everything, the JsonLogger would use a very advanced algorithm to decide whever to flush before the step_interval has passed:

  • a maximum Flushing runtime cost is specified as a fraction of total runtime FRAC (0.005 by default, which is .5 per cent)
  • Every flushing is monitored and timed
  • Every time we log data, if T_FLUSH / (TIME.NOW() - TIME_OF_LAST_FLUSH) > FRAC then we also flush.

Evidently, as I am aware of the adverse health effects of never flushing, we still flush when we would flush normally ( so every 50 steps, or when specified, or at the end of simulations).

We simply will be flushing reasonably more often to provide a better experience to customers while not wasting too many precious resources.

Merge request reports