2
$\begingroup$

For a problem I'm dealing with, I'm trying to understand if my approach could make sense.

I'm using a Variational AutoEncoder (VAE) having relatively low-dimensional inputs, say $x \in \mathbb{R}^n$. Since my dimension $n$ is small, I'm trying to use the VAE by substantially inverting the role of encoder and decoder (inverse-bottleneck) to create high dimensional latent representation $z \in \mathbb{R}^m$ where $m > n$ and then reducing its dimension again obtaining the reconstruction $\hat{x}$.

So by the means of neural networks I'm substantially trying to approximate a decoder to the high-dimensional space $\phi : \mathbb{R}^ n \rightarrow \mathbb{R}^m$ and then an encoder back to the original space $\psi : \mathbb{R}^m \rightarrow \mathbb{R}^n$.

This usage of VAE is not common at all (it is used instead as dimensionality reduction approach). But I'm still trying to understand why my approach could eventually be wrong at the bottom from a mathematical perspective.

I mean, in general it's reasonable to project features in higher dimensional space to enhance separation (kernel methods), so here we are just learning the best way to project these inputs to a such space..

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

You did not say, what your actual final goal is. But in any case, what a VAE will learn in this setting, contrary to when a proper bottleneck is used, will simply be (a noisy version of) the identity. And the (noisy) values $z\in\mathbb{R}^m$ might also not be of much help, since it can actually be anything since there is no restriction. Think of any arbitrary mapping from $x\in\mathbb{R}^n$ to (a distribution in) $\mathbb{R}^m$ and this mapping could, in principle, be obtained by this VAE. So I don't see that the latent values would give useful representations of your data.

However, adding some constraints might be worthwhile. E.g. people have, IIRC, tried to get better models via "anti-bottlenecks" by penalizing denseness of the $z$-vectors (or, favoring sparsity). Or you could try to severely restrict the architecture of your network.

Improve this answer
$\endgroup$
5
  • $\begingroup$ My goal would be learning to generate potentially new instances of low-dimensional vectors. I get your point, so in the case of vae, we may only approximate a noisy identity function, but what about the case of projecting features to high-dimensional space instead? There the situation would be different? $\endgroup$ Commented Apr 7, 2022 at 12:54
  • $\begingroup$ @JamesArten What means: "generate potentially new instances of low-dimensional vectors"? Do you want a low dimensional representation of your data, or an embedding as submanifold, or ...? And: What does "projecting features to high-dimensional space" refer to? $\endgroup$ Commented Apr 7, 2022 at 13:23
  • $\begingroup$ I just want to use VAE as generative model to generate new instances of my original vectors: isn’t this a very common goal while using VAEs? Also, with projecting to high dimensional space I simply refer to the context while we start with original features, $x \in \mathbb{R}^d$ and by using a map $\phi : \mathbb{R}^d \rightarrow \mathbb{R}^m$ we project them into a space $\mathbb{R}^m$ with $m >> d$. $\endgroup$ Commented Apr 7, 2022 at 14:02
  • $\begingroup$ The latter situation may occur when we want to obtain a better separation of our instances, eventually to fit an optimal decision hyperplane. $\endgroup$ Commented Apr 7, 2022 at 14:11
  • $\begingroup$ Yes, so, as I said, mapping to higher dimensions should not get you any advantage unless you find a way to otherwise cleverly restrict it. It will get you the noisy identity in $x\in\mathbb{R}^n$ and nothing useful in the latent $z\in\mathbb{R}^m$. $\endgroup$ Commented Apr 7, 2022 at 15:12

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.