AKA Story

Tag: Attention

Covering rare words

Musio TeamA.I., Musio

Table of Contents 1. Covering rare words 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Covering rare words goal This week’s blogpost treats a new network architecture, named pointer models, for taking care of rare words. We will dive into some details of the implementation and give a short analysis of the benefits of these kind of models. motivation Motivation for the introduction of new architectures comes directly from short-comings of RNN language models, as well as encoder decoder frameworks. Rare words, especially named entities do not experience good word embeddings and hence do not lead to appropriate sentence embeddings which might be used to initialize a decoder component for predicting an output sequence. Furthermore, the […]

Attention/Memory in Deep Learning

Musio TeamA.I.

Goal Attention mechanisms in neural networks are a quite new phenomena and we are going to provide some background on them here. Motivation Generally speaking attention mechanisms allow the network to focus only on a certain subset of the data provided for a given task. Being able to distinguish between the necessary information at a specific step of a task further reduces the amount of information that has to be processed. Ingredients recurrent neural networks, convolutional neural networks, encoder, decoder, embedding, weights, memory, reinforcement learning Steps The idea behind attention mechanisms is certainly motivated by observing the visual attention of humans. Despite processing the visual input all at the same time, humans rather pay attention to small regions one after […]