human
I am working on the file "FADL1/L7CA_lesson7-cifar10.ipynb".
The first task is:
5. BatchNorm
An issue up above, is that we're having trouble training the ConvNet as we add more layers. If we use larger learningRates, we get NaNs (Infinities), and smaller lrs take forever and doesn't have a chance to explore properly. So it isn't resilient.
To make the model more resilient, we'll use Batched Normalization. BatchNorm is a couple years old now (2018), and makes it much easier to train deep networks.
The network we're going to create will have more layers: 5 Conv layers and 1 Full layer. Back in the day that'd be considered a pretty deep network and would be hard to train. It's very simple now thanks to Batch Norm.
Batch Norm can be used by calling nn.BatchNorm.., we'll write it from scratch to learn about it.
Can you write Python program for it?

What is a the Python program you requested?
assistant
#Here is the Python program you requested.

class BnLayer(nn.Module):
    def __init__(self, ni, nf, stride=2, kernel_size=3):
        super().__init__()
        self.conv = nn.Conv2d(ni, nf, kernel_size=kernel_size, stride=stride,
                              bias=False, padding=1)
        self.a = nn.Parameter(torch.zeros(nf,1,1))
        self.m = nn.Parameter(torch.ones(nf, 1,1))
        
    def forward(self, x):
        x = F.relu(self.conv(x))
        x_chan = x.transpose(0,1).contiguous().view(x.size(1), -1)
        if self.training: # true for training set. false for val set.
            self.means = x_chan.mean(1)[:, None, None]
            self.stds  = x_chan.std(1)[:, None, None]
        return (x-self.means) / self.stds * self.m + self.a