AKA Story

Covering rare words

Table of Contents 1. Covering rare words 1.1. goal 1.2. motivation 1.3. ingredients 1.4. steps 1.5. outlook 1.6. resources Covering rare words goal This week’s blogpost treats a new network architecture, named pointer models, for taking care of rare words. We will dive into some details of the implementation and give a short analysis of the benefits of these kind of models. motivation Motivation for the introduction of new architectures comes directly from short-comings of RNN language models, as well as encoder decoder frameworks. Rare words, especially named entities do not experience good word embeddings and hence do not lead to appropriate sentence embeddings which might be used to initialize a decoder component for predicting an output sequence. Furthermore, the […]

Word Embedding

Thoughts about character-based word embedding and vocabularies in NLP :character:word:embedding:vocabulary: Goal In this summary we compare the two standard methods of single character embedding and full word embedding. Motivation In order to teach a computer to understand words in order to perform natural language tasks, we have to map characters or words to a vector space the computer naturally acts on. Ingredients vocabulary, convolutional layers, highway layers, vector space, out of vocabulary words, semantics, syntax Steps The mapping of character, words or even complete sentences into a vector space is usually called embedding. Given some text, there are two distinct methods to compute word embeddings manageable by a computer. Children learning to read to start by recognizing individual characters before […]