Continuous Bernoulli distribution of Variational autoencoders | Hacker News Reader

Continuous Bernoulli distribution of Variational autoencoders

github.com

31 pointsmeiyulee11 comments

> License: Proerietary license

This right here means I will never look at this again. Not because of the misspelling, but because of the fact that I do not know what conditions there are on my use.

I made the mistake of buying this book https://www.amazon.com/dp/1119482089 at full price, only to find out that all code examples are some version of GPL. Only, this information is hidden in the small print.

Please tell people upfront what they are allowed to do with the code and documentation you share.

Also, `s/lamda/lambda/`.

Hi, nanis Thank u for pointing out the misspelling. I have corrected it. All souse codes are free.

Looks like this paper gives some background: https://arxiv.org/abs/1907.06845

I've dabbled in VAEs, but haven't read this paper. Anyone want to try an ELI5thYearGradStudent?

My reading is this:

When you use an VAE, you implicitly assume that your data never ever leaves the area between 0 and 1 and that it'll never ever touch the borders.

But if you have pictures of scanned-in handwritten digits, there's a chance that in reality, the line was blacker than the darkest color that your image file format can store. For example, if you did contrast enhancement after scanning, you likely cut off some extra dark or extra bright pixels.

A regular VAE cannot handle this situation that multiple brightness=0 (black) pixels actually had varying negative brightnesses which were clamped to 0 for technical reasons. A continuous Bernoulli can model 0 really meaning <=0. And 1 meaning >=1.

That's why with a continuous Bernoulli you get more saturated white and blacks than with just a Bernoulli. See the image in 5.1, for example. That's a good thing because it increases contrast, thus making your features better and dependent learning tasks easier.

Having a quick read, they're pointing out that the Bernoulli distribution is only supported for values of 0 and 1 (i.e. it's binary), whereas pixel values for a grayscale image are a decimal value in the interval [0, 1]. When you train a VAE, it's pretty standard to use a BCE loss, but this is wrong because the data isn't binary (i.e. it's not a Bernoulli distribution). They define a continuous analogue of a Bernoulli distribution that is supported in [0, 1], and use this as the loss function for training a VAE, which gives them reconstructions that are closer to the input.

Yeah, agree having skimmed the paper now. It seems kinda meh to me: it's obvious that improving the loss function is a good thing to do, and for any real application (ie, not greyscale mnist) you'd never use this binary value target.

What is the Continuous Bernoulli distribution good for?

Edit: I see it’s equivalent to a Beta distribution.

Not even a full-blown beta distribution, just a special case. And it was published in neurips...

This is not correct. See above.

> I see it’s equivalent to a Beta distribution.

It's not.

The beta distribution is p(x) ∝ x^(α-1) (1-x)^(β-1). x is in the bases.

The continuous Bernoulli distribution is p(x) ∝ λ^x (1-λ)^x. x is in the exponents.

As https://arxiv.org/abs/1907.06845 says:

> the continuous Bernoulli is not a beta distribution (the main difference between these two distributions is how they concentrate mass around the extrema, see appendix 1 for details)

The appendix is available here (under "Supplemental"): https://papers.nips.cc/paper/2019/hash/f82798ec8909d23e55679...

unrelated, but - why are some names green?

Reply on news.ycombinator.com