human
I am working on the file "Joe #3 TD Learning in Easy21/Joe #3 TD Learning in Easy21.ipynb".
The first task is:
Implement Sarsa(λ) in 21s.

[x] Initialise the value function to zero. 
[x] Use the same step-size and exploration schedules as in the previous section. 
[x] Run the algorithm with parameter values λ ∈ {0, 0.1, 0.2, ..., 1}. 
[x] Stop each run after 1000 episodes 
    and report the mean-squared error over all states and actions, 
    comparing the true values Q∗(s,a) computed in the previous section with the estimated values Q(s, a) computed by Sarsa. 
[x] Plot the mean- squared error against λ. 
[x] For λ = 0 and λ = 1 only, plot the learning curve of mean-squared error against episode number.
Can you write Python program for it?

=====
What is a the Python program you requested?
assistant
#Here is the Python program you requested.

class Sarsa_Agent:
    def __init__(self, environment, n0, mlambda):
        self.n0 = float(n0)
        self.env = environment
        self.mlambda = mlambda
        
        # N(s) is the number of times that state s has been visited
        # N(s,a) is the number of times that action a has been selected from state s.
        self.N = np.zeros((self.env.dealer_values_count,
                           self.env.player_values_count, 
                           self.env.actions_count))
        
        self.Q = np.zeros((self.env.dealer_values_count,
                           self.env.player_values_count, 
                           self.env.actions_count))
        self.E = np.zeros((self.env.dealer_values_count, self.env.player_values_count, self.env.actions_count))

        # Initialise the value function to zero. 
        self.V = np.zeros((self.env.dealer_values_count, self.env.player_values_count))
        
        self.count_wins = 0
        self.iterations = 0

        # get optimal action, with epsilon exploration (epsilon dependent on number of visits to the state)
    # ε-greedy exploration strategy with εt = N0/(N0 + N(st)), 
    def train_get_action(self, state):
        dealer_idx = state.dealer-1
        player_idx = state.player-1
        try:
            n_visits = sum(self.N[dealer_idx, player_idx, :])
        except:
            n_visits = 0        

        # epsilon = N0/(N0 + N(st)
        curr_epsilon = self.n0 / (self.n0 + n_visits)

        # epsilon greedy policy
        if random.random() < curr_epsilon:
            r_action = Actions.hit if random.random()<0.5 else Actions.stick
#             if (dealer_idx == 0 and player_idx == 0):
#                 print ("epsilon:%s, random:%s " % (curr_epsilon, r_action))
            return r_action
        else:
            action = Actions.to_action(np.argmax(self.Q[dealer_idx, player_idx, :]))
#             if (dealer_idx == 0 and player_idx == 0):
#                 print ("epsilon:%s Qvals:%s Q:%s" % (curr_epsilon, self.Q[dealer_idx,...