human
I am working on the file "Reinforcement_Learing_An_Introduction/Multi-armed_Bandits/note.ipynb".
The first task is:
Chapter 2: Multi-armed Bandits
2.1 A k-armed Bandit Problem
nonassociative, evaluative feedback problem:
reward = sample(P(action))
+ P(action) is a stationary probability distribution.
+ reward: $R$
+ $k$ different actions $a$.
+ goal: maximize the expected total reward over some time period.
The value $q$ of $a$ is the expected reward:
\begin{align}
    q(a) &= \mathbb{E} \left [ R_t \mid A_t = a \right ] \
         &\approx \frac1{N} \sum_t^N R_t \mid A_t = a \quad \text{law of large numbers} \
         &= Q_t(a)
\end{align}
2.2 Action-value Methods
one natural way to estimate the value of $a$ is by averaging the rewards actually received: (sample-average method)
\begin{equation}
    Q_t(a) = \frac{\sum_{i=1}^{t-1} R_i \cdot \mathbb{1}(A_i = a)}{\sum_{i=1}^{t-1} \cdot \mathbb{1}(A_i = a)}
\end{equation}
strategies:
+ greedy action: $A_t = \operatorname{argmax}_a Q_t(a)$
+ $\epsilon$-greedy actions:
\begin{equation}
    A_t =
    \begin{cases}
        \operatorname{argmax}_a Q_t(a) & \quad \text{ with probability 1 - $\epsilon$} \
        \text{a random action} & \quad \text{ with probability $\epsilon$}
    \end{cases}
\end{equation}
Exercise
2.1:
$0.5 * 1 + 0.5 * 1/2 = 0.75$
2.2: 

definitely occur: steps = 2, 5
possiblly occur: steps= 1, 3, 4

2.3 The 10-armed Testbed
Can you write Python code for it?

Give me a the Python code you requested?
assistant
#Here is the Python code you requested.

reward_funcs = [
    lambda size: np.random.normal(0.2, 1, size),
    lambda size: np.random.normal(-0.8, 1, size),
    lambda size: np.random.normal(1.5, 1, size),
    lambda size: np.random.normal(0.4, 1, size),
    lambda size: np.random.normal(1.2, 1, size),
    lambda size: np.random.normal(-1.3, 1, size),
    lambda size: np.random.normal(-0.2, 1, size),
    lambda size: np.random.normal(-1.0, 1, size),
    lambda size: np.random.normal(1.0, 1, size),
    lambda size: np.random.normal(-0.4, 1, size),
]

examples = pd.DataFrame([func(2000) for func in reward_funcs]).T
examples.index.name = "Reward distribution"
plt.figure(figsize=(15, 7))
sns.violinplot(data=examples)
del examples

class GreedyAction(object):
    def __init__(self, k, varepsilon=0):
        self.varepsilon = varepsilon
        self.k = k
        # sum of rewards, number of action
        self._records = np.zeros((k, 2))
    
    def _Q_t(self):
        r = self._records[:, 0]
        n = self._records[:, 1]
        res = np.divide(r, n, out=np.zeros_like(r), where=n!=0)
        logger.info("Q_t: {}".format(res))
        return res
    
    def action(self):
        if np.random.random() < self.varepsilon:
            a = np.random.randint(low=0, high=self.k)
            logger.info("random action: {}".format(a))
            return a
        else:
            a = np.argmax(self._Q_t())
            logger.info("greedy action: {}".format(a))
            return a
    
    def update_reward(self, action, reward):
        self._records[action, 0] += reward
        self._records[action, 1] += 1
        
        
class Environment(object):
    def __init__(self, reward_funcs):
        self.reward_funcs = reward_funcs
        self.k = len(reward_funcs)
       
    def feedback(self, action):
        assert 0 <= action < self.k, "action: {} must be in [0, {}]".format(action, self.k)
        return self.reward_funcs[action](1)[0]

def run_10_armed(varepsilon):
    opt_action = 2
    env = Environment(reward_funcs)
    actioner = GreedyAction(env.k,...