One important skill is to know how to build models that would produce good results, and not take a whole day to run at the same time. That said, I have time, and I want to try lots of hyperparameter values for my xgboost model. I want to show you my code, and knowing I don't mind it to take some time, can you please help me with the tuning process? what do you think about my code? I filled lots of values for each hyperparameter hoping to get good results. My model so far is not performing well, it is overfitting, as it produces AUC = 0.85 in the train set and AUC = 0.60 in the test set.
I know it could be a problem with the data itself, but I have several solutions that I want to try and one of them is an extreme tuning of the model.
param_Grid = {
'objective':['binary:logistic','binary:hinge'],
'n_estimators': [30,50,100,120,150,200,250,300,400,500,1000],
'subsample': [0.3,0.4,0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
'max_depth': [2,3,4,6,7,8,9,10],
'min_child_weight': [0.5, 1.0, 3.0, 5.0, 7.0, 10.0],
'colsample_bytree': [0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
'colsample_bylevel': [0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
'gamma': [0, 0.25, 0.5, 1.0],
'learning_rate': [0.001, 0.01, 0.1, 0.2, 0,3],
'reg_lambda': [0.1, 1.0, 5.0, 10.0, 50.0, 100.0],
'colsample_bytree': [0.5,0.6,0.8,1]
}
optimal_params = GridSearchCV(estimator=xgb.XGBClassifier(seed = 42),
param_grid=param_Grid,
scoring = 'roc_auc',
verbose = 2,
cv = 10)
optimal_params.fit(X_train,
Y_train,
early_stopping_rounds = 10,
eval_metric = 'auc',
eval_set = [(X_test, Y_test)],
verbose = 1)
As you see I want to use the ROC metric to evaluate the model. I also use cross validation of k = 10. The data at hand has 1550 observations, with about 26 predictive features (after feature selection) and one binary target feature.
