Looks like this paper gives some background:
https://arxiv.org/abs/1907.06845
I've dabbled in VAEs, but haven't read this paper. Anyone want to try an ELI5thYearGradStudent?
I've dabbled in VAEs, but haven't read this paper. Anyone want to try an ELI5thYearGradStudent?
When you use an VAE, you implicitly assume that your data never ever leaves the area between 0 and 1 and that it'll never ever touch the borders.
But if you have pictures of scanned-in handwritten digits, there's a chance that in reality, the line was blacker than the darkest color that your image file format can store. For example, if you did contrast enhancement after scanning, you likely cut off some extra dark or extra bright pixels.
A regular VAE cannot handle this situation that multiple brightness=0 (black) pixels actually had varying negative brightnesses which were clamped to 0 for technical reasons. A continuous Bernoulli can model 0 really meaning <=0. And 1 meaning >=1.
That's why with a continuous Bernoulli you get more saturated white and blacks than with just a Bernoulli. See the image in 5.1, for example. That's a good thing because it increases contrast, thus making your features better and dependent learning tasks easier.