Skip to content
Snippets Groups Projects

Change default initializers for RBM to be scale invariant

Open Vicentini Filippo requested to merge fixes into master

Created by: PhilipVinc

This PR does 2 thins: changes the default initialzier of DenseSymm and RBMSymm to have a variance 1/(features * n_sites), and also change the default initializer of Dense to lecun_normal (so a rescaled normal distrbution with variance 1/n_sites) instead of the fixed value of 0.01 we had previously. This also affect RBM.

With regards to RBM/Dense, I think a more appropriate initializer should be variance 1/(alpha*n_sites), but flax reccomends lecun_normal so i'm not sure what would be best...

before merging this i'd like everyone to run a few tests on this branch, for example for code that uses the default initializers of Dense and DenseSymm.

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading