human
I am working on the file "README.ipynb".
The first task is:
Bias-Variance Tradeoff
The model trained with a maximum depth of 1 seems to suffer from high bias. This can be identified by the fact that the training score is significantly lower than any other model configuration shown. A depth of one is equivalent to a single threshold, and in this case, the true relationship seems to be a bit more complex.
On the other hand, the model with a maximum depth of 10 seems to suffer from high complexity in the DecisionTree. This can be seen by the divergence between the training score and validation score, with the training score being much higher. As long as your model and fitting procedure is well defined, adding complexity should almost always allow a better training score. We see here though that the added complexity seems to produce worse performance on unseen data once you go beyond a maximum depth of 5. This is likely caused by the overly complex model fitting beyond the underlying relationship to fitting sample selection noise.
Best-Guess Optimal Model
In this case, a maximum depth of 5 seems to produce the best validation score. This indicates it generalizes the best for the most important data-- data we haven't seen.

Evaluating Model Performance
Grid Search
Grid search is a technique used to determine hyper-parameters-- in this case, maximum depth. It works by iterating through paramater candidates in a brute-force fashion, keeping the hyper-parameters with the best score. As we saw above, we selected the maximum depth based on the validation score, where the grid was a range between 1-10. The catch here is that now our validation score will likely exaggerate our out of sample score, since it was used to train our model (via hyper-parameters). We almost surely fit <i>some</i> noise related to the selection in our validation data that will not generalize. Still, this configuration is a good guess.
Cross-Validation
K-fold cross-validation is very similiar to the cross-validation we did above. The difference is we use K unique subsets of the data for K train and...
Can you write Python code for it?

What is a Python code to determine hyper-parameters-- in this case, maximum depth?
assistant
#Here is Python code to determine hyper-parameters-- in this case, maximum depth.

from sklearn.grid_search import RandomizedSearchCV
from sklearn.metrics import make_scorer

# Allows us to try random parameters in a log space
class LogspaceSampler:
    def __init__(self, min_val, max_val, seed):
        self.log_min_val = np.log(min_val)
        self.log_max_val = np.log(max_val)
        assert self.log_max_val > self.log_min_val
        self.log_spread = self.log_max_val - self.log_min_val
        self.random_state = np.random.RandomState(seed)

    def rvs(self):
        log_value = self.random_state.rand() * self.log_spread + self.log_min_val
        return np.exp(log_value)

def fit_model(X, y):
    """ Performs grid search over the 'max_depth' parameter for a 
        decision tree regressor trained on the input data [X, y]. """
    
    # Create cross-validation sets from the training data
    cv_sets = ShuffleSplit(X.shape[0], n_iter=10, test_size=0.20, random_state=28)

    # Create a Gradient Boosted regressor object
    regressor = GBRegressor()

    # Create a dictionary of parameter distributions to try
    params = {'max_depth': range(3, 7),
              'learning_rate': LogspaceSampler(0.01, 1.0, seed=29385),
              'gamma': LogspaceSampler(0.000001, 1.0, seed=46435),
              'reg_lambda': LogspaceSampler(0.000001, 1.0, seed=92385)}

    # Transform 'performance_metric' into a scoring function using 'make_scorer' 
    scoring_fnc = make_scorer(performance_metric)

    # Create the grid search object
    grid = RandomizedSearchCV(regressor, param_distributions=params, 
                              scoring=scoring_fnc, cv=cv_sets, n_iter=20)

    # Fit the grid search object to the data to compute the optimal model
    grid = grid.fit(X, y)

    # Return the optimal model after fitting the data
    return grid.best_estimator_