human
I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
We get the same values in only $O(n)$ work!
The astute reader may recall that we already worked out this case,
$$ \frac{\partial \mathbf c^T \mathbf x}{\partial \mathbf x} = \mathbf c^T .$$

Shape of the gradient (Jacobian)
Suppose we have a vector-valued function of vector-valued input, $\mathbf f(\mathbf x)$ where $\mathbf f$ has length $m$ and $\mathbf x$ has length $n$.
* The gradient (Jacobian) matrix $J = \nabla_{\mathbf x} \mathbf f$ has shape $m\times n$.
* Usually in optimization, $m=1$ because we only have one objective
* If $m\ll n$ then finite differencing and forward-mode differentiation will be much more expensive than reverse-mode differentiation
  * Find a way to use reverse-mode!
* If $m \approx n$ then either is about as efficient, but forward-mode is simpler.
* If $m \gg n$ then forward-mode is the ticket.
* In real computations, there may be expensive stages that have lower dimension inputs or outputs, in which case those can be captured. An example is
$$ \mathbf f(\mathbf x) = \mathbf q \sigma(\mathbf q^T \mathbf x) $$
where $\sigma$ is an expensive nonlinear function.
The Jacobian $J = \nabla_{\mathbf x} \mathbf f$ is a square matrix, but naive forward- and reverse-mode would both require $n$ evaluations of $\sigma$.
Since $\sigma$ is a scalar-valued function of a scalar argument, $\sigma'(\mathbf q^T \mathbf x)$ is just one number, and thus $J = (\sigma') \mathbf q \mathbf q^T$ is readily available (and you know it's rank-1 so don't need to store all $n^2$ entries). Models of this sort show up frequently in physical modeling.
Algorithmic (automatic) differentiation
Next, we'll consider ways to have libraries/compilers generate by-hand code such as we see above.
We'll use the JAX library, which offers differentiation of NumPy computations (and offload to GPUs, which we won't use now).
Uncomment the line below if you need to install jax and jaxlib.
Can you write Python program for it?

What is a Python program to find a way to use reverse-mode!?
assistant
#Here is Python program to find a way to use reverse-mode!
.

# ! pip install jax jaxlib

import jax
import jax.numpy as jnp

# def g_jax(x):
#     """Same function as before, but using jnp in place of np."""
#     y = x
#     for i in range(2):
#         y = jnp.cos(y**jnp.pi) * jnp.log(y)
#     return y

g_jax = lambda x : g(x, m=jnp)

gprime_jax = jax.grad(g_jax)
print(gprime_jax(1.9))
print(gprime(1.9)[1])