human
2.2 Training Clssifier
Before achieving this intersting idea, we need to train a classifier firstly. Here we train a simply fully connected network with best accuracy=93.1% on testing sets.
2.3  Generate Classes Prototype
Regarding to the best accuray=93.1% on testing sets, actually this is a poor performance on MNIST because we only build a fully connected network and do nothing with optimization even no activation function.
Next we will generate  corresponding classes prototype for all categories from digit 0 to digit 9. To do this, we need a few tricks as following:

Initializing input with random noise. Here, we initialize input from standard normal distribution.
Freezing except updating the initial input the parameters of model in backpropagation stage.

To make the distribution of input closer to the distribution of real data, we also calculate the mean squared error between pixel-wise mean for each categories and input as regularization term, which is defined by $z$. Therefore, the loss function is as following:
$$
\begin{equation}
\mathcal{L}=cross_entropy(f(x),y)+\lambda mse_loss(x, z)
\end{equation}
$$
where we use $f(.)$ denote the pretrained model, $x$ denote the input, $y$ denote corresponding label, $cross_entropy$ and $mse_loss$ denote cross entropy error and mean squared error, repectively. Note that we need to set the mode of this model to eval.
 ... What is a Python program to calculate the mean squared error between pixel-wise mean for each categories and input as regularization term, which is defined by $z$?
assistant
#Here is Python program to calculate the mean squared error between pixel-wise mean for each categories and input as regularization term, which is defined by $z$.
import torch
from torch.autograd import Variable
from torch import nn, optim

import matplotlib.pyplot as plt
import numpy as np

from class_prototype.model import MNIST_DNN
from utils.utils import set_parameters_static
from utils.utils import get_img_means

lmda = 0.1

class Prototype(nn.Module):
    def __init__(self):
        super(Prototype, self).__init__()

criterion = nn.CrossEntropyLoss()
regular = nn.MSELoss()

model = MNIST_DNN()

model.load_state_dict(torch.load('./class_prototype/MNIST_CNN.pkl'))
# don't update parameters of original model
model = set_parameters_static(model)

# init images and labels:0-9
x_prototype = torch.zeros(10, 784)
y_prototype = torch.linspace(0, 9, 10)
y_prototype = y_prototype.type(torch.LongTensor)

x_prototype = Variable(x_prototype, requires_grad=True)
y_prototype = Variable(y_prototype)

imgs_means = get_img_means('./dataset/mnist')
imgs_means = Variable(imgs_means)

optimizer = optim.Adam([x_prototype], lr=0.01)
print('begin training...')
for i in range(10000):
    optimizer.zero_grad()
    logits_prototype = model(x_prototype)
    cost_protype = criterion(logits_prototype, y_prototype) + lmda * regular(x_prototype, imgs_means)
    cost_protype.backward()
    optimizer.step()
    if i % 500 == 0:
        print('cost_protype={:.6f}'.format(cost_protype.item()))