{"id":1503,"date":"2016-06-13T02:12:03","date_gmt":"2016-06-13T02:12:03","guid":{"rendered":"http:\/\/blog.themusio.com\/?p=1503"},"modified":"2024-05-01T11:02:04","modified_gmt":"2024-05-01T02:02:04","slug":"expectation-maximization-algorithm","status":"publish","type":"post","link":"https:\/\/blog.themusio.com\/?p=1503","title":{"rendered":"Expectation Maximization Algorithm"},"content":{"rendered":"

Goal<\/strong>
\nIn today’s summary we have a look at the expectation maximization algorithm that allows to optimize latent variable models when analytic inference of the posterior probability of latent variables is intractable.<\/p>\n

Motivation<\/strong>
\nLatent variable models are itself interesting, because they are related to variational autoencoders and encoder-decoder frameworks that are popular in unsupervised and semi-supervised learning.
\nThey allow to sample from the data distribution and are believed to enhance the expressiveness of the hierarchical recurrent encoder decoder models.
\nWe can think of them as memorizing higher abstract information, such as emotional states that allow to generate sentimental utterances in the encoder.<\/p>\n

Ingredients<\/strong>
\nvariational autoencoder, observable variables, latent variables, maximum likelihood, posterior probability, complete data log likelihood<\/p>\n

Steps<\/strong>
\nIn general we are concerned with finding good models, which means determining parameters of this model that can explain the data.
\nMore mathematically spoken, we try to find the maximum likelihood of the probability function of the model given the observed data.
\nBut now and then we encounter data sets, where not all data points have labels and we are left with a certain amount of unobserved data.
\nIt is also sometimes helpful to assume that there simply are hidden variables that lead to the observation of the data.
\nThese hidden variables are also called latent variables and might better explain the structure of the data.<\/p>\n

Then in order to determine the likelihood of the observed data we have to marginalize over these latent variables and have to take into account the values the latent variables can take.
\nThe sum over these values of the latent variables becomes particular tricky since it sits inside the logarithm of the likelihood function and an analytic solution is hard to find.<\/p>\n

Therefore it makes sense to look out for alternatives that may allow us to eventually get a handle on the marginalized log likelihood.
\nWe could make use of the complete data log likelihood function, if we would be able to observe the latent variables.
\nFor this we need the posterior probability of the latent variables given the data, which itself is a quantity that is not easily accessible.
\nBut there are a certain number of tools around that allow to approximate it, like Monte Carlo methods and variational inference.
\nNow we are at the point where we can calculate the expectation of the complete data log likelihood function and that is also where the naming for the expectation maximization algorithm comes from.
\nThis first step is also called the E-step and usually followed by the maximization of the expectation called the M-step.
\nThere is now some math left to be done to show that this algorithm of picking initial parameters of the network, estimating the latent variables and determining the optimized parameters will increase the marginalized log likelihood function.
\nWe will skip the proof here and only state that the expectation maximization algorithm converges, but it is not ensured that we can reach a global maximum.
\nHowever in practice the algorithm works well for optimizing latent variable models.<\/p>\n

Outlook<\/strong>
\nAlternative methods for inferring the posterior probability of latent variables have recently been under consideration in situations where the sampling process becomes more difficult.
\nIn particular, variational inference seems to scale well enough to be used in deep neural networks, such as autoencoder or encoder decoder frameworks.<\/p>\n

Resources
\n<\/strong>“Machine Learning<\/a>” (PDF). Machine Learning<\/em>. Published Fall 2012. Accessed 10 June 2016.
\n“The Expectation Maximization Algorithm<\/a>” (PDF). The Expectation Maximization Algorithm<\/em>. Published July 18 2004.\u00a0Accessed 10 June 2016.
\n“Expectation\u2013maximization algorithm<\/a>” (WEB).\u00a0Expectation\u2013maximization algorithm<\/em>.\u00a0Accessed 10 June 2016.<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Goal In today’s summary we have a look at the expectation maximization algorithm that allows to optimize […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[3642,3640],"tags":[3656,4016,3896,4018,4020,4022,3898],"class_list":["post-1503","post","type-post","status-publish","format-standard","hentry","category-ai-en","category-all-en","tag-baggage-en","tag-complete-data-log-likelihood-en","tag-latent-variables-en","tag-maximum-likelihood-en","tag-observable-variables-en","tag-posterior-probability-en","tag-variational-autoencoder-en"],"aioseo_notices":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/posts\/1503","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1503"}],"version-history":[{"count":2,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/posts\/1503\/revisions"}],"predecessor-version":[{"id":10879,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=\/wp\/v2\/posts\/1503\/revisions\/10879"}],"wp:attachment":[{"href":"https:\/\/blog.themusio.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1503"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1503"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.themusio.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1503"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}