Introduction
In this article, we will explore different techniques for finding the optimal hyperparameter values from a given set of parameters in a grid. Particularly we will look at RandomizedSearchCV, GridSearchCV, and BayesSearchCV.
In this blog you will learn:
- How to initialize the parameter grid.
- How to find the optimal hyperparameters based on a given technique.
- How to build a model (XGBClassifier) to use the hyperparameters.
- How to score the performance of the model.
RandomizedSearchCV
param_grid = { "gamma": [0, 0.1, 0.2, 0.5, 1, 1.5, 2, 3, 6, 12, 20], "learning_rate": [0.01, 0.02, 0.03, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8], "max_depth": [1, 2, 3, 4, 5, 6, 8, 12], "n_estimators": [25, 50, 65, 80, 100, 115, 200] } grid_search = RandomizedSearchCV(estimator=classifier_0, param_distributions=param_grid, scoring=scoring) GridSearchCV
param_grid = { "gamma": [0, 0.1, 0.2, 0.5, 1, 1.5, 2, 3, 6, 12, 20], "learning_rate": [0.01, 0.02, 0.03, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8], "max_depth": [2, 3, 4, 5, 6, 8, 12], "n_estimators": [25, 50, 65, 80, 100, 115, 200] } grid_search = GridSearchCV(estimator=classifier_0, param_grid=param_grid, scoring=scoring) BayesSearchCV
param_bayes = { 'gamma': Categorical(param_grid['gamma']), 'learning_rate': Categorical(param_grid['learning_rate']), 'max_depth': Categorical(param_grid['max_depth']), 'n_estimators': Categorical(param_grid['n_estimators']) } grid_search = BayesSearchCV(estimator=classifier_0, search_spaces=param_bayes, scoring=scoring, n_jobs=-1, cv=10) Finding the Best HyperParameters
best_model = grid_search.fit(X_train, y_train) hyperparams = best_model.best_params_ Building and Scoring the Classifier using the HyperParameters
# Fitting the Model ne = hyperparams['n_estimators'] lr = hyperparams['learning_rate'] md = hyperparams['max_depth'] gm = hyperparams['gamma'] print("Recommended Params >>", f"ne: {ne},", f"lr: {lr}", f"md: {md}", f"gm: {gm}") # Build Classification Model classifier_1 = XGBClassifier( base_score=0.5, colsample_bylevel=1, colsample_bynode=1, objective=objective, booster="gbtree", eval_metric=eval_metric_list, n_estimators=ne, learning_rate=lr, max_depth=md, gamma=gm, subsample=0.8, colsample_bytree=1, random_state=1 ) # Fit Model eval_set = [(X_train, y_train)] classifier_1.fit( X_train, y_train, eval_set=eval_set, verbose=False ) # Get predictions for training data train_yhat = classifier_1.predict(X_train) print("Training Preds: \n", train_yhat[:5]) # Set K-Fold Cross Validation Levels cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=1) # Training Results train_results = cross_val_score(classifier_1, X_train, y_train, scoring=scoring, cv=cv, n_jobs=1) # Brief Review of Training Results print("Average Accuracy K-Fold: ", round(train_results.mean(), 2)) print("Std Deviation K-Fold: ", round(train_results.std(), 2)) print("Precision Score 0: ", round(precision_score(y_train, train_yhat, average=None)[0], 3)) print("Precision Score 1: ", round(precision_score(y_train, train_yhat, average=None)[1], 3)) Performance
Machine: Laptop
Processor: AMD Ryzen 7
OS: Windows
DataFrame Shape: (7282, 17)
Top comments (0)