Meet Musio

Conditional Neural Network Architectures

Table of Contents 1. Conditional Neural Network Architectures 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Conditional Neural Network Architectures goal Today we are going to have a look at conditional neural network architectures and present some of the findings in the recent papers “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer” and “PathNet: Evolution Channels Gradient Descent in Super Neural Networks”. motivation The interest in conditional models is mainly based on their capability to incorporate a huge number of parameters compared to standard architectures without increasing the need for more computationally powerful hardware. Furthermore such models seem to be able to reduce the training time and are interesting for multi-, online-task learning and transfer learning. […]

Interview with Musio: About Musio as an AI

Q&A with AKA – About Musio as an ‘AI Robot’

Ever since we’ve introduced Musio to the whole world, we’ve received countless questions from our partners, investors, customers as well as curious enthusiasts. So we thought it would be a good idea to answer the most frequently asked questions to make Musio more engaging and easier to understand. We hope that the following Q&A sessions enlighten our readers on who we are and what we dream with Musio. The first round of questions is about Musio as an AI robot. So let’s start!   Q. I’ve heard lots of AIs in the market, starting from likes of Apple’s Siri to  robots like Pepper. What kind of AI is Musio?  Musio is a robot run on AKA’s special AI system (software engine) that […]

Adversarial techniques for dialogue generation

Table of Contents 1. Adversarial techniques for dialogue generation 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Adversarial techniques for dialogue generation goal This week we are going to have a look at the latest developments of generative adversarial networks (GANs) in the field of dialogue generation by summarizing the paper “Adversarial Learning for Neural Dialogue Generation”. motivation General encoder decoder models for response generation are usually not able to produce meaningful utterances and instead come up with short, generic, repetitive and non-informative sequences of words. The idea here is to apply adversarial methods so far only successful in computer vision to NLP problems, in particular dialogues. Adversarial training with respect to modeling conversations can be […]

Compression of neural networks

Table of Contents 1. Compression and distillation of models 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Compression of neural networks goal In this blogpost we will have a look at methods for compressing and reducing deep neural network models in size. motivation The simple fact that bigger and deeper is better for training, leads to models that take up quite some space in memory. However, most of the time one is limited when it comes to memory. Either it is the budget on hardware or more recently developing models for mobile devices is becoming more and more popular. Another important point for deploying neural networks in applications is inference time. In general, larger models also […]

Dilated causal convolutions for audio and text generation

Table of Contents 1. Dilated causal convolutions for audio and text generation   causal dilation convolution 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Dilated causal convolutions for audio and text generation goal In today’s summary we dive into the architecture of WaveNet and its successor ByteNet which are autoregressive generative models for generating audio and respectively sentences on character-level. motivation The architectures behind both models are based on dilated causal convolutional layers which recently got much attention also in image generation tasks. Especially modeling sequential data with long term dependencies like audio or text seem to benefit from convolutions with dilations to increase the receptive field. ingredients dilation, causal convolution, residual blocks, skip connection, gated activation function, steps Without […]

Covering rare words

Table of Contents 1. Covering rare words 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Covering rare words goal This week’s blogpost treats a new network architecture, named pointer models, for taking care of rare words. We will dive into some details of the implementation and give a short analysis of the benefits of these kind of models. motivation Motivation for the introduction of new architectures comes directly from short-comings of RNN language models, as well as encoder decoder frameworks. Rare words, especially named entities do not experience good word embeddings and hence do not lead to appropriate sentence embeddings which might be used to initialize a decoder component for predicting an output sequence. Furthermore, the […]

Sentences with style and topic

Table of Contents 1. “Generating sentences from a continuous space” 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Sentences with style and topic goal In this week’s post we will have a closer look at a paper dealing with the modeling of style, topic and high-level syntactic structures in language models by introducing global distributed latent representations. In particular, the variational autoencoder seems to be a promising candidate for pushing generative language models forwards and including global features. motivation Recurrent neural network language models are known to be capable of modeling complex distributions over sequences. However, their architecture limits them to modeling local statistics over sequences and therefore global features have to be captured otherwise. ingredients […]