AKA Story

BRAVE NEW BEGINNING

Musio

Latest Upload

Adversarial techniques for dialogue generation

Musio TeamA.I.

Table of Contents 1. Adversarial techniques for dialogue generation 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Adversarial techniques for dialogue generation goal This week we are going to have a look at the latest developments of generative adversarial networks (GANs) in the field of dialogue generation by summarizing the paper “Adversarial Learning for Neural Dialogue Generation”. motivation General encoder decoder models for response generation are usually not able to produce meaningful utterances and instead come up with short, generic, repetitive and non-informative sequences of words. The idea here is to apply adversarial methods so far only successful in computer vision to NLP problems, in particular dialogues. Adversarial training with respect to modeling conversations can be […]

Compression of neural networks

Musio TeamA.I.

Table of Contents 1. Compression and distillation of models 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Compression of neural networks goal In this blogpost we will have a look at methods for compressing and reducing deep neural network models in size. motivation The simple fact that bigger and deeper is better for training, leads to models that take up quite some space in memory. However, most of the time one is limited when it comes to memory. Either it is the budget on hardware or more recently developing models for mobile devices is becoming more and more popular. Another important point for deploying neural networks in applications is inference time. In general, larger models also […]

Dilated causal convolutions for audio and text generation

Musio TeamA.I.

Table of Contents 1. Dilated causal convolutions for audio and text generation   causal dilation convolution 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Dilated causal convolutions for audio and text generation goal In today’s summary we dive into the architecture of WaveNet and its successor ByteNet which are autoregressive generative models for generating audio and respectively sentences on character-level. motivation The architectures behind both models are based on dilated causal convolutional layers which recently got much attention also in image generation tasks. Especially modeling sequential data with long term dependencies like audio or text seem to benefit from convolutions with dilations to increase the receptive field. ingredients dilation, causal convolution, residual blocks, skip connection, gated activation function, steps Without […]

Covering rare words

Musio TeamA.I., Musio

Table of Contents 1. Covering rare words 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Covering rare words goal This week’s blogpost treats a new network architecture, named pointer models, for taking care of rare words. We will dive into some details of the implementation and give a short analysis of the benefits of these kind of models. motivation Motivation for the introduction of new architectures comes directly from short-comings of RNN language models, as well as encoder decoder frameworks. Rare words, especially named entities do not experience good word embeddings and hence do not lead to appropriate sentence embeddings which might be used to initialize a decoder component for predicting an output sequence. Furthermore, the […]

Sentences with style and topic

Musio TeamA.I.

Table of Contents 1. “Generating sentences from a continuous space” 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Sentences with style and topic goal In this week’s post we will have a closer look at a paper dealing with the modeling of style, topic and high-level syntactic structures in language models by introducing global distributed latent representations. In particular, the variational autoencoder seems to be a promising candidate for pushing generative language models forwards and including global features. motivation Recurrent neural network language models are known to be capable of modeling complex distributions over sequences. However, their architecture limits them to modeling local statistics over sequences and therefore global features have to be captured otherwise. ingredients […]

Alternatives to the softmax layer

Musio TeamA.I.

Table of Contents 1. Alternatives to the softmax layer   softmax 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Alternatives to the softmax layer goal This weeks posts deals with some possible alternatives to the softmax layer when calculating probabilities for words over large vocabularies. motivation Natural language tasks as neural machine translation or dialogue generation rely on word embeddings at the input and output layer. Further for decent performances a very large vocabulary is needed to reduce the number of out of vocabulary words that cannot be properly embedded and therefore not processed. The natural language models used for these task usually come with a final softmax layer to compute the probabilities over the words in the […]

Musio’s emotion classifier

Musio TeamA.I.

Table of Contents 1. Musio’s emotion classifier   emotion classifier 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook Musio’s emotion classifier goal In today’s summary we take a look at the emotion classifier applied in Musio and layout some details of the data and models we use. motivation Sentiment classification is in general an important task, since as humans our intention is never to only convey plain content. The way we phrase things is as important as the message itself in human interaction. And sometimes misinterpreting the emotions of one’s counterpart will lead to awkward situations. Hence, Musio has to learn to read the emotional status of it’s users to take part in their daily life. ingredients emotion, sentiment analysis, MLP, […]