human
$$King- Queen = Man - Woman$$
<img src= 'img/word2vec/word2vec.png' width= "700px">
Now that we’ve looked at trained word embeddings, 

let’s learn more about the training process. 
But before we get to word2vec, we need to look at a conceptual parent of word embeddings: the neural language model.

The neural language model
“You shall know a word by the company it keeps” J.R. Firth

Bengio 2003 A Neural Probabilistic Language Model. Journal of Machine Learning Research. 3:1137–1155

After being trained, early neural language models (Bengio 2003) would calculate a prediction in three steps:
<img src= 'img/word2vec/neural-language-model-prediction.png' width= "700px">
<img src= 'img/word2vec/bengio.png' width= "400px">
The output of the neural language model is a probability score for all the words the model knows. 
- We're referring to the probability as a percentage here, 
- but 40% would actually be represented as 0.4 in the output vector.
Language Model Training

We get a lot of text data (say, all Wikipedia articles, for example). then
We have a window (say, of three words) that we slide against all of that text.
The sliding window generates training samples for our model

<img src= 'img/word2vec/lm-sliding-window-4.png' width= "700px">
As this window slides against the text, we (virtually) generate a dataset that we use to train a model. 
Instead of only looking two words before the target word, we can also look at two words after it.
<img src= 'img/word2vec/continuous-bag-of-words-example.png' width= "700px">
If we do this, the dataset we’re virtually building and training the model against would look like this:
<img src= 'img/word2vec/continuous-bag-of-words-dataset.png' width= "700px">
This is called a Continuous Bag of Words (CBOW) https://arxiv.org/pdf/1301.3781.pdf
Skip-gram
Instead of guessing a word based on its context (the words before and after it), this other architecture tries to guess neighboring words using the current word. 
<img src= 'img/word2vec/skipgram.png' width=...
 ... Give me a Python program to calculate a prediction in three steps:?
assistant
#Here is Python program to calculate a prediction in three steps:
.
# see http://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html
import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

torch.manual_seed(1)

text = """We are about to study the idea of a computational process.
Computational processes are abstract beings that inhabit computers.
As they evolve, processes manipulate other abstract things called data.
The evolution of a process is directed by a pattern of rules
called a program. People create programs to direct processes. In effect,
we conjure the spirits of the computer with our spells."""

text = text.replace(',', '').replace('.', '').lower().split()

# By deriving a set from `raw_text`, we deduplicate the array
vocab = set(text)
vocab_size = len(vocab)
print('vocab_size:', vocab_size)

w2i = {w: i for i, w in enumerate(vocab)}
i2w = {i: w for i, w in enumerate(vocab)}

# context window size is two
def create_cbow_dataset(text):
    data = []
    for i in range(2, len(text) - 2):
        context = [text[i - 2], text[i - 1],
                   text[i + 1], text[i + 2]]
        target = text[i]
        data.append((context, target))
    return data

cbow_train = create_cbow_dataset(text)
print('cbow sample', cbow_train[0])

def create_skipgram_dataset(text):
    import random
    data = []
    for i in range(2, len(text) - 2):
        data.append((text[i], text[i-2], 1))
        data.append((text[i], text[i-1], 1))
        data.append((text[i], text[i+1], 1))
        data.append((text[i], text[i+2], 1))
        # negative sampling
        for _ in range(4):
            if random.random() < 0.5 or i >= len(text) - 3:
                rand_id = random.randint(0, i-1)
            else:
                rand_id = random.randint(i+3, len(text)-1)
            data.append((text[i], text[rand_id], 0))
    return data

skipgram_train = create_skipgram_dataset(text)
print('skipgram sample', skipgram_train[0])

class CBOW(nn.Module):
    def __init__(self, vocab_size, embd_size,...