1
$\begingroup$

I'm working on an implementation of a Variational Autoencoder (VAE). There are lots of helpful examples and guides out there, which typically introduce VAE in the context of image data, e.g. MNIST. Since pixels (input features) are scaled between zero and one - $x\in[0,1]$ - these examples use sigmoid activation in the last layer of the decoder $d: z \mapsto x$. That makes sense, but what if I cannot assume any knowledge about the scale of $x$? Do I just not use any activation at all? And if so, does that make learning significantly harder?

Any help would be much appreciated!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

Your output activation function (in combination with your loss function) correspond to (implicit) assumptions you make about the distribution of the data:

  • logistic sigmoid + binary cross-entropy = Bernoulli assumption
  • identity function + mean squared error = Gaussian assumption
  • softmax function + cross-entropy = Categorical assumption

You can easily see the difference (also for plain AEs) by running the following experiment (on a simple dataset like MNIST):

  1. train an AE on images with pixel intensities in [0, 1] using logistic sigmoid + BCE
  2. train an AE on images with normalised pixel intensities (subtract mean and divide by standard deviation) using the identity funtion + MSE.
  3. observe the difference between the reproductions. The different assumptions should be clearly visible.

Obviously this explanation only explains what to do in a few limited cases. If your data follows some exotic distribution, there might not be a fixed/well-known loss function and/or activation function. However, it should provide a basis to make informed decisions about what functions to use to achieve good reconstructions.

Improve this answer
$\endgroup$
1
$\begingroup$

By taking a probabilistic view, the outputs of the decoder are realizations of the random variables $x\sim p_{\theta}(x|z)$.

If you want to model realizations in $[0, 1]$, you can use the continuous Bernoulli for example. In this can your decoder will output the parameter $\lambda$ of the distribution which needs to be in $[0, 1]$, hence the sigmoid function taken over the output for example.

If you want to model realizations in $\mathbb{R}$, you can use the Gaussian distribution for example. Here, the decoder will typically output the mean $m$ of your distribution. There is no restriction for $m$ so this can simply at scalar output at your last decoder layer.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ why would the decoder output the variance? Typically, only the encoder produces mean and variance... $\endgroup$ Commented Jun 2, 2022 at 9:35
  • $\begingroup$ Agree with this comment, in typical use for unconstrained output you would just have decoder output $f:z \rightarrow \mu$ without any activation in the final layer. It is then customary to assume diagonal output noise so that $p(x|z)\sim N(f(z), \sigma^{-2}I)$ $\endgroup$ Commented Jun 2, 2022 at 11:15
  • $\begingroup$ @MrTsjolder You're right, I responded too quickly. I will edit $\endgroup$ Commented Jun 2, 2022 at 11:41

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.