Meet Musio

Alternatives to the softmax layer

Table of Contents 1. Alternatives to the softmax layer   softmax 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Alternatives to the softmax layer goal This weeks posts deals with some possible alternatives to the softmax layer when calculating probabilities for words over large vocabularies. motivation Natural language tasks as neural machine translation or dialogue generation rely on word embeddings at the input and output layer. Further for decent performances a very large vocabulary is needed to reduce the number of out of vocabulary words that cannot be properly embedded and therefore not processed. The natural language models used for these task usually come with a final softmax layer to compute the probabilities over the words in the […]

Musio’s emotion classifier

Table of Contents 1. Musio’s emotion classifier   emotion classifier 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook Musio’s emotion classifier goal In today’s summary we take a look at the emotion classifier applied in Musio and layout some details of the data and models we use. motivation Sentiment classification is in general an important task, since as humans our intention is never to only convey plain content. The way we phrase things is as important as the message itself in human interaction. And sometimes misinterpreting the emotions of one’s counterpart will lead to awkward situations. Hence, Musio has to learn to read the emotional status of it’s users to take part in their daily life. ingredients emotion, sentiment analysis, MLP, […]

Syntactic Parse Trees for sentence representation

Table of Contents 1. Syntactic Parse Trees for sentence representation   syntax parse tree 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Syntactic Parse Trees for sentence representation :syntax:parse:tree: goal Today’s summary deals with the question of modeling additional information present in natural language and how models could take advantage of this hierarchical syntactic structure. motivation Besides semantics, being about meaning and logic in language, syntax determines the structure of sentences. Basic linguistic information in the form of parse trees, characterizing relationships among words in a sentence, might in principle lead to better word and sentence representations which should enhance natural language processing. ingredients parse tree, constituency, dependency, syntax, semantics, recursive neural network, recurrent neural network, reduce shift algorithm, shift […]

Musio’s intent classifier

Table of Contents 1. Musio’s intent classifier   Musio keras classifier 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook Musio’s intent classifier goal In today’s post we will explain by means of presenting an example classifier for the user utterance how Musio is capable of determining the intent of the user. motivation Since Musio’s interior consists of several modules, all associated to solving a specific tasks, from question answering to general conversation, we have to forward the user utterance to the correct module that is able to generate a sensible response to the user utterance. ingredients data set, cross validation, modules, classifier, keras, spacy steps We skip the part on speech recognition and assume that we received a properly trancscripted user utterance. […]

Variational Autoencoder

Goal Variational methods for inference of latent variables became popular in the last years. Here, we have a look at variational autoencoders from a slightly more mathematical point of view. Motivation Some deep generative models for vision tasks, such as style transfer or simply generating images similar to some given images, rely heavily on variational autoencoder frameworks. Only very recently latent variables have been introduced into the hierarchical recurrent encoder decoder framework to enhance the expressive power of the model when coming up with responses to utterances. Further variational autoencoder allow to perform unsupervised learning and are thus in general interesting to solving artificial intelligence. Ingredients variational inference, posterior distribution, latent variable, Bayes model, Kullback-Leibler divergence, objective function, lower bound […]


Goal Autoencoder have long been proposed to tackle the problem of unsupervised learning. In this week’s summary we have a look at their capabilities of providing a features that can be successfully used in supervised tasks and sketch their framework architecture. Motivation In supervised learning, back in the days, deeper architectures need some kind of pretraining of layers before the actual supervised tasked could be pursued. Autoencoder came in handy for this and allowed to train one layer after the other and were able to find useful features for the supervised learning. Ingredients unsupervised learning, features, representation, encoder, decoder, denoising Steps Let us start by looking at the general architecture. An autoencoder consists of two basic parts: the encoder and […]

Expectation Maximization Algorithm

Goal In today’s summary we have a look at the expectation maximization algorithm that allows to optimize latent variable models when analytic inference of the posterior probability of latent variables is intractable. Motivation Latent variable models are itself interesting, because they are related to variational autoencoders and encoder-decoder frameworks that are popular in unsupervised and semi-supervised learning. They allow to sample from the data distribution and are believed to enhance the expressiveness of the hierarchical recurrent encoder decoder models. We can think of them as memorizing higher abstract information, such as emotional states that allow to generate sentimental utterances in the encoder. Ingredients variational autoencoder, observable variables, latent variables, maximum likelihood, posterior probability, complete data log likelihood Steps In general […]