Created by: PhilipVinc
I think the gradient for real-valued models we are using is wrong
I think we forgot to account for a factor of 2 that arises from taking the real part when parameters are real. When weights are complex, I think the gradient is correct.
Proving that this gradient was wrong, you can see that in the test we were dividing by two the finite-difference gradient for no reason when weights were real.
(Noticed because when doing the dynamics of some model with real parameter my timescale was off by a factor of 2 wrt the exact dynamics, but I'll double check...)
THIS IS BREAKING because some models will now have a gradient doubled (so learning rate must be changed) but I think it's a bug fix