{"id":1541,"date":"2016-06-30T17:24:18","date_gmt":"2016-06-30T08:24:18","guid":{"rendered":"http:\/\/blog.themusio.com\/?p=1541"},"modified":"2024-05-01T11:01:18","modified_gmt":"2024-05-01T02:01:18","slug":"variational-autoencoder","status":"publish","type":"post","link":"https:\/\/blog.themusio.com\/?p=1541","title":{"rendered":"Variational Autoencoder"},"content":{"rendered":"

Goal<\/strong>
\nVariational methods for inference of latent variables became popular in the last years.
\nHere, we have a look at variational autoencoders from a slightly more mathematical point of view.<\/p>\n

Motivation<\/strong>
\nSome deep generative models for vision tasks, such as style transfer or simply generating images similar to some given images, rely heavily on variational autoencoder frameworks.
\nOnly very recently latent variables have been introduced into the hierarchical recurrent encoder decoder framework to enhance the expressive power of the model when coming up with responses to utterances.
\nFurther variational autoencoder allow to perform unsupervised learning and are thus in general interesting to solving artificial intelligence.<\/p>\n

Ingredients<\/strong>
\nvariational inference, posterior distribution, latent variable, Bayes model, Kullback-Leibler divergence, objective function, lower bound<\/p>\n

Steps<\/strong>
\nImagine we are in the situation were we have a distribution over some data points and we are not interested in putting a label on them, but we want to generate more data points which are similar to the ones we got.
\nIn particular, we can think of a set of images of birds or 3D models of spaceships and we would like to come up with more of the same kind.
\nSo far, model building for generation required either strong assumptions on the structure of the data, severe approximations which lead to sub-optimal models or were simply computationally expensive.
\nOne approach to tackle these problems is to introduce latent variables into the models.
\nIn principle these are capable of capturing some of the underlying structure of the data we want to model and allow us to maximize the probability of the seen data points such that we can generate probable new ones.<\/p>\n

The variational autoencoder is one model that incorporates latent variables and only shares its name with the sparsity and denoising autoencoder because of its encoder decoder framework.
\nThe most important question we have to answer is how we manage to maximize the probability of the data points for the variational autoencoder.
\nFirst, we have a look at how to choose the latent variables.
\nIn general, these might experience a very complicated dependency structure, since for example for handwritten character generation the angle of the character might be influenced by the writing style and velocity which we all have to capture in the latent variable.
\nLuckily, we have neural networks around, we draw the latent variables from a normal distribution and hope that our network will come up with a useful distribution over the latent variables in some layer.
\nNext, we have to approximate the data distribution which can be done by sampling.
\nSo we sample a bunch of latent variables and calculate the expectation of the conditional data distribution.
\nNow for high dimensional spaces we will run into problems since generated data points will mostly have low probability since they will in general look very different from the data points we already have.
\nIt would be a better idea to sample latent variables that are more likely to have produced the data points we observed and then compute the conditional distribution using these values for the latent variables.
\nThe mathematical tool to use is called Kullback-Leibler divergence and measures the similarity of two probability distributions.
\nThis leads us to an objective function which can be optimized by looking at a lower bound to the logarithm of the data distribution.
\nThe lower bound in particular includes a term that drives an arbitrarily chosen distribution over the latent variables to one that gives high probability to those latent variable values that have generated the observed data points.
\nAnother advantages of this is that we make the inference of the posterior distribution over the latent variables tractable in this way.<\/p>\n

The optimization of this lower bound can be done by the conventional methods of stochastic gradient descent.
\nWe choose an inference model that comes up with parameters for a normal distribution which we try to match to a standard normal distribution over the latent variables.
\nSampling becomes also a minor problem since during gradient descent we already go over the whole data set and it suffices to pick one sample for the latent variables.
\nSince the sampling process is non-continuous a reparametrization trick is needed to push the gradients through to the inference model.
\nInstead of sampling over the the normal distribution, we sample an error value from a standard normal distribution and deterministically calculate the latent variables using the parameter output of the inference model.
\nIn this way none of the expectations in our objective depend on the parameters of our models and we can perform stochastic gradient descent.
\nDuring testing we sample a latent variable from the standard normal distribution and we get a lower bound on the logarithm of the probability for the generated data point.<\/p>\n

There is also a nice interpretation of lower bound in terms of information theory.
\nUsually the logarithm of the data distribution tells us the total number bits needed to reconstruct the data point from an ideal encoding.
\nWhat we are doing with the lower bound is calculating the extra information when the latent variable is sampled from the inference model instead of the standard normal distribution plus the information needed to reconstruct the data point given this latent variable.
\nIn the end there is a small penalty for sub-optimal encoding since the inference model is used instead of the posterior distribution coming directly from the observed data points.<\/p>\n

From a more practical point of view and having applications with respect to natural language processing in mind the shortcoming of recurrent language models in producing averaged responses to given utterances might be circumvented by allowing the models to generate several probable responses.
\nWe will pursue this direction further and follow recent developments with interest.
\n
\nResources<\/strong>
\n\u201cTutorial on Variational Autoencoders<\/a>\u201d (PDF). Tutorial on Variational Autoencoders. Published\u00a0<\/em>June 21, 2016.\u00a0<\/em>Accessed 29 June\u00a02016.
\n\u201cVariational auto-encoders do not train complex generative models<\/a>\u201d (WEB). Variational auto-encoders do not train complex generative models. Published\u00a0<\/em>June 22, 2016.\u00a0<\/em>Accessed 29\u00a0June\u00a02016.
\n\u201cChainer-Variational-Recurrent-Autoencoder<\/a>\u201d (GIT). Chainer-Variational-Recurrent-Autoencoder. Published\u00a0August<\/em>\u00a023, 2015.\u00a0<\/em>Accessed 29\u00a0June\u00a02016.<\/p>\n","protected":false},"excerpt":{"rendered":"

Goal Variational methods for inference of latent variables became popular in the last years. Here, we have a l […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[3642,3640],"tags":[3656,3984,3986,3988,3990,3992,3994,3996],"class_list":["post-1541","post","type-post","status-publish","format-standard","hentry","category-ai-en","category-all-en","tag-baggage-en","tag-bayes-model-en","tag-kullback-leibler-divergence-en","tag-latent-variable-en","tag-lower-bound-en","tag-objective-function-en","tag-posterior-distribution-en","tag-variational-inference-en"],"aioseo_notices":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/posts\/1541","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1541"}],"version-history":[{"count":3,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/posts\/1541\/revisions"}],"predecessor-version":[{"id":10878,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/posts\/1541\/revisions\/10878"}],"wp:attachment":[{"href":"https:\/\/blog.themusio.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1541"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1541"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1541"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}