Meet Musio

Sentences with style and topic

Table of Contents 1. “Generating sentences from a continuous space” 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Sentences with style and topic goal In this week’s post we will have a closer look at a paper dealing with the modeling of style, topic and high-level syntactic structures in language models by introducing global distributed latent representations. In particular, the variational autoencoder seems to be a promising candidate for pushing generative language models forwards and including global features. motivation Recurrent neural network language models are known to be capable of modeling complex distributions over sequences. However, their architecture limits them to modeling local statistics over sequences and therefore global features have to be captured otherwise. ingredients […]

Alternatives to the softmax layer

Table of Contents 1. Alternatives to the softmax layer   softmax 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Alternatives to the softmax layer goal This weeks posts deals with some possible alternatives to the softmax layer when calculating probabilities for words over large vocabularies. motivation Natural language tasks as neural machine translation or dialogue generation rely on word embeddings at the input and output layer. Further for decent performances a very large vocabulary is needed to reduce the number of out of vocabulary words that cannot be properly embedded and therefore not processed. The natural language models used for these task usually come with a final softmax layer to compute the probabilities over the words in the […]

Musio’s emotion classifier

Table of Contents 1. Musio’s emotion classifier   emotion classifier 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook Musio’s emotion classifier goal In today’s summary we take a look at the emotion classifier applied in Musio and layout some details of the data and models we use. motivation Sentiment classification is in general an important task, since as humans our intention is never to only convey plain content. The way we phrase things is as important as the message itself in human interaction. And sometimes misinterpreting the emotions of one’s counterpart will lead to awkward situations. Hence, Musio has to learn to read the emotional status of it’s users to take part in their daily life. ingredients emotion, sentiment analysis, MLP, […]

Syntactic Parse Trees for sentence representation

Table of Contents 1. Syntactic Parse Trees for sentence representation   syntax parse tree 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Syntactic Parse Trees for sentence representation :syntax:parse:tree: goal Today’s summary deals with the question of modeling additional information present in natural language and how models could take advantage of this hierarchical syntactic structure. motivation Besides semantics, being about meaning and logic in language, syntax determines the structure of sentences. Basic linguistic information in the form of parse trees, characterizing relationships among words in a sentence, might in principle lead to better word and sentence representations which should enhance natural language processing. ingredients parse tree, constituency, dependency, syntax, semantics, recursive neural network, recurrent neural network, reduce shift algorithm, shift […]

Musio’s intent classifier

Table of Contents 1. Musio’s intent classifier   Musio keras classifier 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook Musio’s intent classifier goal In today’s post we will explain by means of presenting an example classifier for the user utterance how Musio is capable of determining the intent of the user. motivation Since Musio’s interior consists of several modules, all associated to solving a specific tasks, from question answering to general conversation, we have to forward the user utterance to the correct module that is able to generate a sensible response to the user utterance. ingredients data set, cross validation, modules, classifier, keras, spacy steps We skip the part on speech recognition and assume that we received a properly trancscripted user utterance. […]

Variational Autoencoder

Goal Variational methods for inference of latent variables became popular in the last years. Here, we have a look at variational autoencoders from a slightly more mathematical point of view. Motivation Some deep generative models for vision tasks, such as style transfer or simply generating images similar to some given images, rely heavily on variational autoencoder frameworks. Only very recently latent variables have been introduced into the hierarchical recurrent encoder decoder framework to enhance the expressive power of the model when coming up with responses to utterances. Further variational autoencoder allow to perform unsupervised learning and are thus in general interesting to solving artificial intelligence. Ingredients variational inference, posterior distribution, latent variable, Bayes model, Kullback-Leibler divergence, objective function, lower bound […]


Goal Autoencoder have long been proposed to tackle the problem of unsupervised learning. In this week’s summary we have a look at their capabilities of providing a features that can be successfully used in supervised tasks and sketch their framework architecture. Motivation In supervised learning, back in the days, deeper architectures need some kind of pretraining of layers before the actual supervised tasked could be pursued. Autoencoder came in handy for this and allowed to train one layer after the other and were able to find useful features for the supervised learning. Ingredients unsupervised learning, features, representation, encoder, decoder, denoising Steps Let us start by looking at the general architecture. An autoencoder consists of two basic parts: the encoder and […]