human
4. Batch normalization
One of the most significant recent ideas that impacted NNs a lot is Batch normalization. The idea is simple, yet effective: the features should be whitened ($mean = 0$, $std = 1$) all the way through NN. This improves the convergence for deep models letting it train them for days but not weeks. You are to implement the first part of the layer: features normalization. The second part (ChannelwiseScaling layer) is implemented below.

input:   batch_size x n_feats
output: batch_size x n_feats

The layer should work as follows. While training (self.training == True) it transforms input as $$y = \frac{x - \mu}  {\sqrt{\sigma + \epsilon}}$$
where $\mu$ and $\sigma$ - mean and variance of feature values in batch and $\epsilon$ is just a small number for numericall stability. Also during training, layer should maintain exponential moving average values for mean and variance: 
self.moving_mean = self.moving_mean * alpha + batch_mean * (1 - alpha)
    self.moving_variance = self.moving_variance * alpha + batch_variance * (1 - alpha)
During testing (self.training == False) the layer normalizes input using moving_mean and moving_variance. 
Note that decomposition of batch normalization on normalization itself and channelwise scaling here is just a common implementation choice. In general "batch normalization" always assumes normalization + scaling.

Can you provide a the Python code you requested?
assistant
#Here is the Python code you requested.
class BatchNormalization(Module):
    EPS = 1e-3
    def __init__(self, alpha = 0.):
        super(BatchNormalization, self).__init__()
        self.alpha = alpha
        self.moving_mean = 0 
        self.moving_variance = 0
        
    def updateOutput(self, input):
        # Your code goes here. ################################################
        # use self.EPS please
        if (self.training == True):
            self.mean = np.mean(input, axis=0, keepdims=True)
            self.var = np.var(input, axis=0, keepdims=True)
            self.moving_mean = self.moving_mean * self.alpha + self.mean * (1 - self.alpha)
            self.moving_variance = self.moving_variance * self.alpha + self.var * (1 - self.alpha)
            self.output = (input - self.mean)/np.sqrt(self.var + self.EPS)
        else:
            self.output = (input - self.moving_mean) / np.sqrt(self.moving_variance + self.EPS)
            
        return self.output
    
    def updateGradInput(self, input, gradOutput):
        # Your code goes here. ################################################
        self.gradInput = input.shape[0] * gradOutput
        self.gradInput -= np.sum(gradOutput, axis=0, keepdims=True)
        self.gradInput -= self.output * np.sum(gradOutput * self.output, axis=0, keepdims=True)
        self.gradInput = 1/(input.shape[0] * np.sqrt(self.var + self.EPS)) * self.gradInput
        
        return self.gradInput
    
    def __repr__(self):
        return "BatchNormalization"

class ChannelwiseScaling(Module):
    """
       Implements linear transform of input y = \gamma * x + \beta
       where \gamma, \beta - learnable vectors of length x.shape[-1]
    """
    def __init__(self, n_out):
        super(ChannelwiseScaling, self).__init__()

        stdv = 1./np.sqrt(n_out)
        self.gamma = 1 #np.random.uniform(-stdv, stdv, size=n_out) # ACHTUNG!! 1
        self.beta = 0 #0 #np.random.uniform(-stdv, stdv, size=n_out) # 0
        
        self.gradGamma = np.zeros_like(self.gamma)
        self.gradBeta =...