human
I am working on the file "10158006_10185453_11896493_lab2.ipynb".
The first task is:
Part 2. Multilayer perceptron
You discover that the predictions by the logistic regression classifier are not good enough for your application: the model is too simple. You want to increase the accuracy of your predictions by using a better model. For this purpose, you're going to use a multilayer perceptron (MLP), a simple kind of neural network. The perceptron wil have a single hidden layer $\bh$ with $L$ elements. The parameters of the model are $\bV$ (connections between input $\bx$ and hidden layer $\bh$), $\ba$ (the biases/intercepts of $\bh$), $\bW$ (connections between $\bh$ and $\log q$) and $\bb$ (the biases/intercepts of $\log q$.
The conditional probability of the class label $j$ is given by:
$\log p(t = j \;|\.
\bx, \bb, \bW) = \log q_j - \log Z$
where $q_j$ are again the unnormalized probabilities per class, and $Z = \sum_j q_j$ is again the probability normalizing factor. Each $q_j$ is computed using:
$\log q_j = \bw_j^T \bh + b_j$
where $\bh$ is a $L \times 1$ vector with the hidden layer activations (of a hidden layer with size $L$), and $\bw_j$ is the $j$-th column of $\bW$ (a $L \times 10$ matrix). Each element of the hidden layer is computed from the input vector $\bx$ using:
$h_j = \sigma(\bv_j^T \bx + a_j)$
where $\bv_j$ is the $j$-th column of $\bV$ (a $784 \times L$ matrix), $a_j$ is the $j$-th element of $\ba$, and $\sigma(.)$ is the so-called sigmoid activation function, defined by:
$\sigma(x) = \frac{1}{1 + \exp(-x)}$
Note that this model is almost equal to the multiclass logistic regression model, but with an extra 'hidden layer' $\bh$. The activations of this hidden layer can be viewed as features computed from the input, where the feature transformation ($\bV$ and $\ba$) is learned.
2.1 Derive gradient equations (20 points)
State (shortly) why $\nabla_{\bb} \mathcal{L}^{(n)}$ is equal to the earlier (multiclass logistic regression) case, and why $\nabla_{\bw_j} \mathcal{L}^{(n)}$ is almost equal to the earlier case.
Like in multiclass logistic regression, you should use...
Can you write Python program for it?
assistant
#Here is the Python program you requested.

import math

def sigmoid(z):
    return 1.0/(1.0+np.exp(np.negative(z)))

def sigmoid_prime(z):
    return sigmoid(z)*(1-sigmoid(z))

class Network(object):
    def  __init__(self,sizes):
        self.num_layers = len(sizes)-1
        self.sizes = sizes
        self.biases = [np.random.randn(1,j) for j in self.sizes[1:]]
        self.b_grads = [np.zeros((1,j)) for j in self.sizes[1:]]
        
        self.weights = [(np.random.randn(y, x)/np.sqrt(x)).transpose()                
                        for x, y in zip(self.sizes[:-1], self.sizes[1:])]
        self.w_grads = [np.zeros((y, x)).transpose()
                        for x, y in zip(self.sizes[:-1], self.sizes[1:])]
        
        self.activations = [np.zeros((1,j)) for j in self.sizes]
#         print('V',self.weights[0].shape,'a',self.biases[0].shape,'activ:',self.activations[0].shape)
#         print('W',self.weights[1].shape,'b',self.biases[1].shape,'activ2:',self.activations[1].shape)
    
    def feedForward(self,x):
        self.activations[0] =  x.reshape((1,len(x)))
        for i,b,w in zip(range(1,self.num_layers),self.biases[:-1],self.weights[:-1]):
            self.activations[i]= sigmoid(self.activations[i-1].dot(w) + b)
        # last layer activation no activation function
        self.activations[-1] = self.activations[-2].dot(self.weights[-1]) + self.biases[-1]
        return self.activations[-1]
    
    def SGD(self,x,t,learn):
        idx = list(range(len(x_train)))
        random.shuffle(idx)
        log_train = 0
        for i in idx:
            self.feedForward(x[i,:])
            err = self.backprop(t[i])
            for i in range(self.num_layers):
                self.biases[i] += learn * self.b_grads[i]
                self.weights[i] += learn * self.w_grads[i]
            log_train +=err
        return log_train
        
    
    def backprop(self,t):
        max_activ = np.max(self.activations[-1])
        logZ = max_activ + np.log(np.sum(np.exp(self.activations[-1] - max_activ)))
        logp = self.activations[-1]...