human
I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
5. Batch Normalization
Batch Normalization is used to allow creation of deeper networks. You'll find if you try to make the previous networks much deeper, they'll start encountering NaN and 0 losses and little in between. BatchNorm increases the resiliency of a neural network.
Can you write Python program for it?
assistant
#Here is Python program to find if you try to make the previous networks much deeper, they'll start encountering NaN and 0 losses and little in between.

class BnLayer(nn.Module):
    def __init__(self, ni, nf, stride=2, kernel_size=3):
        super().__init__()
        self.conv = nn.Conv2d(ni, nf, kernel_size=kernel_size, stride=stride, 
                              bias=False, padding=1)
        self.a = nn.Parameter(torch.zeros(nf, 1, 1)) # adder
        self.m = nn.Parameter(torch.ones(nf, 1, 1))  # multiplier
        
    def forward(self, x):
        x = F.relu(self.conv(x))
        x_chan = x.transpose(0,1).contiguous().view(x.size(1), -1)
        if self.training: # true during training; false during evaluation
            self.means = x_chan.mean(1)[:, None, None] # calc mean of each conv filter (channel)
            self.stds  = x_chan.std(1)[:, None, None]  # calc stdev ofeach conv filter (channel)
        return (x - self.means) / self.stds*self.m + self.a # subtract means, divide by stdevs