Meet Musio

Compression of neural networks

Table of Contents 1. Compression and distillation of models 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Compression of neural networks goal In this blogpost we will have a look at methods for compressing and reducing deep neural network models in size. motivation The simple fact that bigger and deeper is better for training, leads to models that take up quite some space in memory. However, most of the time one is limited when it comes to memory. Either it is the budget on hardware or more recently developing models for mobile devices is becoming more and more popular. Another important point for deploying neural networks in applications is inference time. In general, larger models also […]

Dilated causal convolutions for audio and text generation

Table of Contents 1. Dilated causal convolutions for audio and text generation   causal dilation convolution 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Dilated causal convolutions for audio and text generation goal In today’s summary we dive into the architecture of WaveNet and its successor ByteNet which are autoregressive generative models for generating audio and respectively sentences on character-level. motivation The architectures behind both models are based on dilated causal convolutional layers which recently got much attention also in image generation tasks. Especially modeling sequential data with long term dependencies like audio or text seem to benefit from convolutions with dilations to increase the receptive field. ingredients dilation, causal convolution, residual blocks, skip connection, gated activation function, steps Without […]

Covering rare words

Table of Contents 1. Covering rare words 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Covering rare words goal This week’s blogpost treats a new network architecture, named pointer models, for taking care of rare words. We will dive into some details of the implementation and give a short analysis of the benefits of these kind of models. motivation Motivation for the introduction of new architectures comes directly from short-comings of RNN language models, as well as encoder decoder frameworks. Rare words, especially named entities do not experience good word embeddings and hence do not lead to appropriate sentence embeddings which might be used to initialize a decoder component for predicting an output sequence. Furthermore, the […]

Sentences with style and topic

Table of Contents 1. “Generating sentences from a continuous space” 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Sentences with style and topic goal In this week’s post we will have a closer look at a paper dealing with the modeling of style, topic and high-level syntactic structures in language models by introducing global distributed latent representations. In particular, the variational autoencoder seems to be a promising candidate for pushing generative language models forwards and including global features. motivation Recurrent neural network language models are known to be capable of modeling complex distributions over sequences. However, their architecture limits them to modeling local statistics over sequences and therefore global features have to be captured otherwise. ingredients […]

Alternatives to the softmax layer

Table of Contents 1. Alternatives to the softmax layer   softmax 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Alternatives to the softmax layer goal This weeks posts deals with some possible alternatives to the softmax layer when calculating probabilities for words over large vocabularies. motivation Natural language tasks as neural machine translation or dialogue generation rely on word embeddings at the input and output layer. Further for decent performances a very large vocabulary is needed to reduce the number of out of vocabulary words that cannot be properly embedded and therefore not processed. The natural language models used for these task usually come with a final softmax layer to compute the probabilities over the words in the […]

Musio’s emotion classifier

Table of Contents 1. Musio’s emotion classifier   emotion classifier 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook Musio’s emotion classifier goal In today’s summary we take a look at the emotion classifier applied in Musio and layout some details of the data and models we use. motivation Sentiment classification is in general an important task, since as humans our intention is never to only convey plain content. The way we phrase things is as important as the message itself in human interaction. And sometimes misinterpreting the emotions of one’s counterpart will lead to awkward situations. Hence, Musio has to learn to read the emotional status of it’s users to take part in their daily life. ingredients emotion, sentiment analysis, MLP, […]

Syntactic Parse Trees for sentence representation

Table of Contents 1. Syntactic Parse Trees for sentence representation   syntax parse tree 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Syntactic Parse Trees for sentence representation :syntax:parse:tree: goal Today’s summary deals with the question of modeling additional information present in natural language and how models could take advantage of this hierarchical syntactic structure. motivation Besides semantics, being about meaning and logic in language, syntax determines the structure of sentences. Basic linguistic information in the form of parse trees, characterizing relationships among words in a sentence, might in principle lead to better word and sentence representations which should enhance natural language processing. ingredients parse tree, constituency, dependency, syntax, semantics, recursive neural network, recurrent neural network, reduce shift algorithm, shift […]