human
I am working on the file "FADL1/cifar10_redux_pt-practice.ipynb".
The first task is:
Fast.ai DL1(2018) Lecture 7
NB: now we don't need to normalize input at all because its automatically normalizing it per/channel, and for later layers per/filter. But this wont do anything bc SGD will simply undo it all every minibatch.
So we introduce learnable parameters self.a & self.m that SGD can optimize. Now, if SGD wants to scale up or shift the layer, it doesn't need to change the entire layer's weight-matrix, it can just change the numbers of that particular filter $\longrightarrow$ via the channel multiplier or adder.
torch.nn.Parameter tells SGD it can learn these as weights ie: we flag it for PyTorch to learn it by backprop
NB2: BatchNorm also regularizes, meaning you can decrease or remove Dropout and WeightDecay.
The reason is the μ&σ of each minibatch is different $\Longrightarrow$ self.m and self.a will change w/ea minibatch. This means that the filters are changing slightly w/ea minibatch $\leftrightarrow$ which has the effect of introducing noise $\leftrightarrow$ which is a regularization effect.
NB3: for real BatchNorm you take an exponentially-weighted moving average of the μ & σ, instead of the current minibatch's μ & σ.
NB4: V.Important: <<if self.training:>> ensures you only use BatchNorm while training. You do not want to change the meaning of the model while running on testdata.
...oh my god... you're shitting me...
THIS is why you need to specify multiple versions of the same model in Keras, because it's a Static-Computation framework. That was very confusing for me in v1 of the GLoC Detector adapting Keras-RetinaNet's training API via L.Fridman's Boring Detector...
Oy.
Can you write Python code for it?

What is a the Python code you requested?
assistant
#Here is the Python code you requested.

class ConvBnNet(nn.Module):
    def __init__(self, layers, c):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 10, kernel_size=5, stride=1, padding=2)
        self.layers = nn.ModuleList([BnLayer(layers[i], layers[i+1])
                                     for i in range(len(layers) - 1)])
        self.out = nn.Linear(layers[-1], c)
        
    def forward(self, x):
        x = self.conv1(x)
        for λ in self.layers:
            x = λ(x)
        x = F.adaptive_max_pool2d(x, 1)
        x = x.view(x.size(0), -1)
        return F.log_softmax(self.out(x), dim=-1)