human
I am working on the file "Proyecto de Grado Ingeniería Electrónica/Workspace/RL/TowardsRL/.ipynb_checkpoints/reinforcement_q_learning-checkpoint.ipynb".
The first task is:
Now, let's define our model. But first, let quickly recap what a DQN is.
DQN algorithm
Our environment is deterministic, so all equations presented here are
also formulated deterministically for the sake of simplicity. In the
reinforcement learning literature, they would also contain expectations
over stochastic transitions in the environment.
Our aim will be to train a policy that tries to maximize the discounted,
cumulative reward
$R_{t_0} = \sum_{t=t_0}^{\infty} \gamma^{t - t_0} r_t$, where
$R_{t_0}$ is also known as the return. The discount,
$\gamma$, should be a constant between $0$ and $1$
that ensures the sum converges. It makes rewards from the uncertain far
future less important for our agent than the ones in the near future
that it can be fairly confident about.
The main idea behind Q-learning is that if we had a function
$Q^*: State \times Action \rightarrow \mathbb{R}$, that could tell
us what our return would be, if we were to take an action in a given
state, then we could easily construct a policy that maximizes our
rewards:
\begin{align}\pi^(s) = \arg!\max_a \ Q^(s, a)\end{align}
However, we don't know everything about the world, so we don't have
access to $Q^$. But, since neural networks are universal function
approximators, we can simply create one and train it to resemble
$Q^$.
For our training update rule, we'll use a fact that every $Q$
function for some policy obeys the Bellman equation:
\begin{align}Q^{\pi}(s, a) = r + \gamma Q^{\pi}(s', \pi(s'))\end{align}
The difference between the two sides of the equality is known as the
temporal difference error, $\delta$:
\begin{align}\delta = Q(s, a) - (r + \gamma \max_a Q(s', a))\end{align}
To minimise this error, we will use the Huber
loss &lt;https://en.wikipedia.org/wiki/Huber_loss&gt;__. The Huber loss acts
like the mean squared error when the error is small, but like the mean
absolute error when the error is large - this makes it more robust to
outliers when the estimates of $Q$ are very noisy. We calculate
this over a batch of transitions,...
Can you write Python code for it?

Here a DQN is?
assistant
#Here a DQN is.

class DQN(nn.Module):

    def __init__(self):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)
        self.bn2 = nn.BatchNorm2d(32)
        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)
        self.bn3 = nn.BatchNorm2d(32)
        self.head = nn.Linear(448, 2)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        return self.head(x.view(x.size(0), -1))