human
Write me a Python program to compute the direction of fastest decrease: $\mathbf{d}_0=-\nabla E(\mathbf{x}_0)$ given the following:
I am working on the file "SC1v2/Bonus - 07-08 - Gradient Descent and Nonlinear Least-Square.ipynb".
The first task is:
In the previous implementation we used the following notation:
- $n$: Number of iteration of Gradient Descent
- Black dot: Solution $[a_n,b_n]$ at $n$-th step of the Gradient Descent.
- Red-Yellow streamplot: Stream plot of the vector field generated by minus the Gradient of the error function $E(a,b)$
- Blue-Green contour: Contour plot of the error function $E(a,b)$.
Questions:
- Try: $\alpha=0.02$, $n=20$, and $m=10$. What do you observe? (keep initialization values of $a_0$ and $a_1$)
- Try: $\alpha=0.04$, $n=20$, and $m=10$. What do you observe? (keep initialization values of $a_0$ and $a_1$)
- Try: $\alpha=0.08$, $n=20$, and $m=10$. What do you observe? (keep initialization values of $a_0$ and $a_1$)
- Can we use a large value of $\alpha$?
- How is related $\alpha$ and the iteration number $n$?
<div id='GD_2D_NonLinearLeastSquare' />

Gradient Descent for a 2D nonlinear least-square problem
Back to TOC
In this case, we will explore the use of the the Gradient Descent algorithm applied to a nonlinear least-square problem with an exponential fit.
Let the function to be fit be,
$$
\begin{equation}
    y(t) = a\,\exp(b\,t),
\end{equation}
$$
where the error function is defined as follows,
$$
\begin{equation}
    E(a,b)=\sum_{i=1}^m (y_i-a\,\exp(b\,t_i))^2.
\end{equation}
$$
Now, to apply the Gradient Descent algorithm we need to compute the Gradient of $E(a,b)$ with respect to $a$ and $b$, which is the following,
$$
\begin{align}
    \frac{\partial E}{\partial a} &= \sum_{i=1}^m 2\,\exp(b\,t_i)(a\,\exp(b\,t_i)-y_i),\
    \frac{\partial E}{\partial b} &= \sum_{i=1}^m 2\,a\,\exp(b\,t_i)\,t_i\,(a\,\exp(b\,t_i)-y_i).
\end{align}
$$
As you may expect, this approach may create very large values for the gradient, which will be very challenging to handle them numerically.
So, an alternative approach is the following,  which we will call it "The Variant":
- Select an initial guess, say $\mathbf{x}_0$
- Compute the direction of fastest decrease: $\mathbf{d}_0=-\nabla E(\mathbf{x}_0)$
- Update the approximation...
Can you write Python program for it?
assistant
#Here is Python program to compute the direction of fastest decrease: $\mathbf{d}_0=-\nabla E(\mathbf{x}_0)$
.

def GD_2D_nonlinear(a0=0.75, b0=0.75, alpha=0, n=0, m=10, TheVariantFlag=False):
    
    # Building data.
    np.random.seed(0)
    a = 1.1
    b = 0.23
    y = lambda t: a*np.exp(b*t)
    T = 10
    ti = T*(np.random.rand(m)*2-1)
    yi = y(ti)+np.random.normal(loc=0, scale=0.1, size=m)
    
    # Defining the Gradient
    E = lambda a, b: np.sum(np.power(yi-a*np.exp(b*ti),2))
    G = lambda a, b: np.array([np.sum(2*np.exp(b*ti)*(a*np.exp(b*ti)-yi)), np.sum(2*a*np.exp(b*ti)*ti*(a*np.exp(b*ti)-yi))],dtype=float)
    # This fucntion will help us to evaluate the Gradient on the points (X[i,j],Y[i,j])
    def E_mG_XY(AA,BB):
        Z = np.zeros_like(AA)
        U = np.zeros_like(AA)
        V = np.zeros_like(AA)
        for i in range(m):
            for j in range(m):
                Z[i,j]=E(AA[i,j],BB[i,j])
                uv = -G(AA[i,j],BB[i,j])
                U[i,j] = uv[0]
                V[i,j] = uv[1]
        return Z, U, V
    
    # Plotting the function and its gradient.
    # Credits: 
    #  https://matplotlib.org/stable/gallery/images_contours_and_fields/plot_streamplot.html
    #  https://scipython.com/blog/visualizing-a-vector-field-with-matplotlib/
    x = np.linspace(-3,3,m)
    AA, BB = np.meshgrid(x,x)
    fig = plt.figure(figsize=(14,10))
    Z, U, V = E_mG_XY(AA,BB)
    cont = plt.contour(AA,BB,Z, 10)
    stream = plt.streamplot(AA, BB, U, V, color=Z, linewidth=2, cmap='autumn', arrowstyle='->', arrowsize=2)
    fig.colorbar(stream.lines)
    fig.colorbar(cont)
    plt.scatter(a0, b0, s=300, marker='.', c='k')
    my_grad = G(a0,b0)
    my_title = r'$\alpha=$ %.4f, $E(a,b)=$ %.4f, $\nabla E(a,b)=$ [%.4f, %.4f]' %  (alpha, E(a0,b0), my_grad[0], my_grad[1])
    plt.title(my_title)
    
    # Computing steps of Gradient Descent
    if n>0:
        ab_output=np.zeros((n+1,2))
        z0 = np.array([a0,b0],dtype=float)
        z0[0] = a0
        z0[1] = b0
        ab_output[0,:]=z0
        # The Gradient Descent Algorithm
        for k in range(n):
            G_E_0=G(z0[0],z0[1])
          ...