The purpose of this experiment is to show how heuristics such as Simulated Annealing can be used to find efficiently good combinations of hyper-parameters in machine learning algorithms. This approach is better than blind random generation of parameters. It is also preferable to fine-tuning each hyper-parameter separately because typically there are interactions between them.

The XGBoost algorithm is a good show case because it has many hyper-parameters. Exhaustive grid search can be computationally prohibitive.

For a very good discussion of the theoretical details of XGBoost, see this Slideshare presentation of the algorithm with title "Kaggle Winning Solution Xgboost algorithm -- Let us learn from its author" by Tianqi Chen.

This is a Kaggle dataset taken from here which contains credit card transactions data and a fraud flag. It appeared originally in Dal Pozzolo, Andrea, et al. "Calibrating Probability with Undersampling for Unbalanced Classification." Computational Intelligence, 2015 IEEE Symposium Series on. IEEE, 2015. There is a Time variable (seconds from the first transaction in the dataset), an Amount variable, the Class variable (1=fraud, 0= no fraud) and the rest (V1-V28) are factor variables obtained through Principal Components Analysis from the original variables.

This is not a very difficult case for XGBoost as it will be seen. The main objective in this experiment is to show that the heuristic search finds a suitable set of hyper-parameters out of a quite large set of potential combinations.

We can verify below that this is a highly imbalanced dataset, typical of fraud detection data. We will take this into account when setting the weights of observations in XGBoost parameters.

Since we have plenty of data we are going to calibrate the hyper-parameters on a validation dataset and evaluate performance on an unseen testing dataset.

import numpy as np import pandas as pd import matplotlib.pyplot as plt %matplotlib inline import random random.seed(1234) plt.style.use('ggplot') dat = pd.read_csv('creditcard.csv') print(dat.head()) print('\nThe distribution of the target variable:\n') dat['Class'].value_counts()

 Time V1 V2 V3 V4 V5 V6 V7 \ 0 0.0 -1.359807 -0.072781 2.536347 1.378155 -0.338321 0.462388 0.239599 1 0.0 1.191857 0.266151 0.166480 0.448154 0.060018 -0.082361 -0.078803 2 1.0 -1.358354 -1.340163 1.773209 0.379780 -0.503198 1.800499 0.791461 3 1.0 -0.966272 -0.185226 1.792993 -0.863291 -0.010309 1.247203 0.237609 4 2.0 -1.158233 0.877737 1.548718 0.403034 -0.407193 0.095921 0.592941 V8 V9 ... V21 V22 V23 V24 \ 0 0.098698 0.363787 ... -0.018307 0.277838 -0.110474 0.066928 1 0.085102 -0.255425 ... -0.225775 -0.638672 0.101288 -0.339846 2 0.247676 -1.514654 ... 0.247998 0.771679 0.909412 -0.689281 3 0.377436 -1.387024 ... -0.108300 0.005274 -0.190321 -1.175575 4 -0.270533 0.817739 ... -0.009431 0.798278 -0.137458 0.141267 V25 V26 V27 V28 Amount Class 0 0.128539 -0.189115 0.133558 -0.021053 149.62 0 1 0.167170 0.125895 -0.008983 0.014724 2.69 0 2 -0.327642 -0.139097 -0.055353 -0.059752 378.66 0 3 0.647376 -0.221929 0.062723 0.061458 123.50 0 4 -0.206010 0.502292 0.219422 0.215153 69.99 0 [5 rows x 31 columns] The distribution of the target variable: 0 284315 1 492 Name: Class, dtype: int64 

Although the context of most of the variables is not known (recall that V1-V28 are factors summarizing the transactional data), we know that V1-V28 are by construction standardized, with a mean of 0 and and a standard deviation of 1. We standardize the Time and Amount variables too.

The data exploration and in particular Welch’s t-tests reveal that almost all the factors are significantly associated with the Class variable. The mean of these variables is almost zero in Class 0 and clearly non-zero in Class 1. The Time and Amount variables are also significant. There does not seem to by any reason for variable selection yet. We could drop the Time variable which is probably useless.

Some of the factors (and the Amount variable) are quite skewed and have very thin distributions. If we were to apply some other method (say, logistic regression) we could apply some transformations (and probably binning) but XGBoost is insensitive to such deviations from normality.

from scipy import stats from sklearn import preprocessing dat['Time'] = preprocessing.scale(dat['Time']) dat['Amount'] = preprocessing.scale(dat['Amount']) print('\nMeans of variables in the two Class categories:\n') pt = pd.pivot_table(dat, values=dat.columns, columns = 'Class', aggfunc='mean') print(pt.loc[dat.columns]) print('\nP-values of Welch’s t-tests and shape statistics:\n') for i in range(30): col_name = dat.columns[i] t, p_val = stats.ttest_ind(dat.loc[ dat['Class']==0, col_name], dat.loc[ dat['Class']==1, col_name],equal_var=False) skewness = dat.loc[:,col_name].skew() kurtosis = stats.kurtosis(dat.loc[:,col_name]) print('Variable: {:7s}'.format(col_name),end='') print('p-value: {:6.3f} skewness: {:6.3f} kurtosis: {:6.3f}'.format(p_val, skewness, kurtosis)) fig, axes = plt.subplots(nrows=6, ncols=5,figsize=(10,10)) axes = axes.flatten() columns = dat.columns for i in range(30): axes[i].hist(dat[columns[i]], bins=50,facecolor='b',alpha=0.5) axes[i].set_title(columns[i]) axes[i].set_xlim([-4., +4.]) plt.setp(axes[i].get_xticklabels(), visible=False) plt.setp(axes[i].get_yticklabels(), visible=False) 

Means of variables in the two Class categories: Class 0 1 Time 0.000513 -0.296223 V1 0.008258 -4.771948 V2 -0.006271 3.623778 V3 0.012171 -7.033281 V4 -0.007860 4.542029 V5 0.005453 -3.151225 V6 0.002419 -1.397737 V7 0.009637 -5.568731 V8 -0.000987 0.570636 V9 0.004467 -2.581123 V10 0.009824 -5.676883 V11 -0.006576 3.800173 V12 0.010832 -6.259393 V13 0.000189 -0.109334 V14 0.012064 -6.971723 V15 0.000161 -0.092929 V16 0.007164 -4.139946 V17 0.011535 -6.665836 V18 0.003887 -2.246308 V19 -0.001178 0.680659 V20 -0.000644 0.372319 V21 -0.001235 0.713588 V22 -0.000024 0.014049 V23 0.000070 -0.040308 V24 0.000182 -0.105130 V25 -0.000072 0.041449 V26 -0.000089 0.051648 V27 -0.000295 0.170575 V28 -0.000131 0.075667 Amount -0.000234 0.135382 Class NaN NaN P-values of Welch’s t-tests and shape statistics: Variable: Time p-value: 0.000 skewness: -0.036 kurtosis: -1.294 Variable: V1 p-value: 0.000 skewness: -3.281 kurtosis: 32.486 Variable: V2 p-value: 0.000 skewness: -4.625 kurtosis: 95.771 Variable: V3 p-value: 0.000 skewness: -2.240 kurtosis: 26.619 Variable: V4 p-value: 0.000 skewness: 0.676 kurtosis: 2.635 Variable: V5 p-value: 0.000 skewness: -2.426 kurtosis: 206.901 Variable: V6 p-value: 0.000 skewness: 1.827 kurtosis: 42.642 Variable: V7 p-value: 0.000 skewness: 2.554 kurtosis: 405.600 Variable: V8 p-value: 0.063 skewness: -8.522 kurtosis: 220.583 Variable: V9 p-value: 0.000 skewness: 0.555 kurtosis: 3.731 Variable: V10 p-value: 0.000 skewness: 1.187 kurtosis: 31.988 Variable: V11 p-value: 0.000 skewness: 0.357 kurtosis: 1.634 Variable: V12 p-value: 0.000 skewness: -2.278 kurtosis: 20.241 Variable: V13 p-value: 0.028 skewness: 0.065 kurtosis: 0.195 Variable: V14 p-value: 0.000 skewness: -1.995 kurtosis: 23.879 Variable: V15 p-value: 0.050 skewness: -0.308 kurtosis: 0.285 Variable: V16 p-value: 0.000 skewness: -1.101 kurtosis: 10.419 Variable: V17 p-value: 0.000 skewness: -3.845 kurtosis: 94.798 Variable: V18 p-value: 0.000 skewness: -0.260 kurtosis: 2.578 Variable: V19 p-value: 0.000 skewness: 0.109 kurtosis: 1.725 Variable: V20 p-value: 0.000 skewness: -2.037 kurtosis: 271.011 Variable: V21 p-value: 0.000 skewness: 3.593 kurtosis: 207.283 Variable: V22 p-value: 0.835 skewness: -0.213 kurtosis: 2.833 Variable: V23 p-value: 0.571 skewness: -5.875 kurtosis: 440.081 Variable: V24 p-value: 0.000 skewness: -0.552 kurtosis: 0.619 Variable: V25 p-value: 0.249 skewness: -0.416 kurtosis: 4.290 Variable: V26 p-value: 0.015 skewness: 0.577 kurtosis: 0.919 Variable: V27 p-value: 0.006 skewness: -1.170 kurtosis: 244.985 Variable: V28 p-value: 0.002 skewness: 11.192 kurtosis: 933.381 Variable: Amount p-value: 0.004 skewness: 16.978 kurtosis: 845.078 

In this step, we partition the dataset into 40% training, 30% validation and 30% testing. Note the use of the random.shuffle() function from numpy. We also make the corresponding matrices train, valid and test containing predictors only with labels trainY, validY and testY, respectively.

Class = dat['Class'].values allIndices = np.arange(len(Class)) np.random.shuffle(allIndices) ## shuffle the indices of the observations numTrain = int(round(0.40*len(Class))) numValid = int(round(0.30*len(Class))) numTest = len(Class)-numTrain-numValid inTrain = allIndices[:numTrain] inValid = allIndices[numTrain:(numTrain+numValid)] inTest = allIndices[(numTrain+numValid):] train = dat.iloc[inTrain,:30] valid= dat.iloc[inValid,:30] test = dat.iloc[inTest,:30] trainY = Class[inTrain] validY = Class[inValid] testY = Class[inTest]

First we create the matrices in the format required by XGBoost with the xgb.DMatrix() function, passing for each dataset the predictors data and the labels. Then we set some fixed parameters. The number of boosting iterations (num_rounds) is set to 20. Normally we would use a larger number, but we want to keep the processing time low for the purposes of this experiment.

We initialize the param dictionary with silent=1 (no messages). Parameter min_child_weight is set at the default value of 1 because the data is highly unbalanced. This is the minimum weighted number of observations in a child node for further partitioning. The objective is binary classification and the evaluation metric is the Area Under Curve (AUC), the default for binary classification. In a more advanced implementation we could make a customized evaluation function, as described in XGBoost API. The internal random numbers seed is set to a constant for reproducible results (this is not guaranteed though, among other reasons because XGBoost runs in threads).

We are going to expand the param dictionary with the parameters in the heuristic search.

import xgboost as xgb dtrain = xgb.DMatrix(train, label=trainY) dvalid = xgb.DMatrix(valid, label=validY) dtest = xgb.DMatrix(test, label=testY) ## fixed parameters num_rounds=20 # number of boosting iterations param = {'silent':1, 'min_child_weight':1, 'objective':'binary:logistic', 'eval_metric':'auc', 'seed' : 1234} 

In what follows we combine the suggestions from several sources, notably:

1. The official XGBoost documentation and in particular the Notes on Parameter Tuning

2. The article "Complete Guide to Parameter Tuning in XGBoost" from Analytics Vidhya

3. Another Slideshare presentation with title "Open Source Tools & Data Science Competitions"

We select several important parameters for the heuristic search:

• max_depth: the maximum depth of a tree, in [1,∞], with default value 6. This is highly data-dependent. [2] quotes as typical values: 3-10 and [3] advises to start from 6. We choose to explore also larger values and select 5-25 levels in steps of 5.
• subsample, in (0,1] with default value 1. This is the proportion of the training instances used in trees and smaller values can prevent over-fitting. In [2] values in 0.5-1 are suggested. [3] suggests to leave this at 1. We decide to test values in 0.5-1.0 in steps of 0.1.
• colsample_bytree, in (0,1] with default value 1. This is the subsample ratio of columns (features) used to construct a tree. In [2] values in 0.5-1 are suggested. The advice in [3] is 0.3-0.5. We will try similar values as with subsample.
• eta (or learning_rate), in [0,1], with default value 0.3. This is the shrinking rate of the feature weights and larger values (but not too high!) can be used to prevent overfitting. A suggestion in [2] is to use values in 0.01-0.2. We can select some values in [0.01,0.4].
• gamma, in [0, ∞], with default value 0. This is the minimum loss function reduction required for a split. [3] suggests to leave this at 0. We can experiment with values in 0-2 in steps of 0.05.
• scale_pos_weight which controls the balance of positive and negative weights with default value 1. The advice in [1] is to use the ratio of negative to positive cases which is 595 here, i.e. to put a weight that large to the positive cases. [2] similarly suggests a large value in case of high class imbalance as is the case here. We can try some small values and some larger ones.

The total number of possible combinations is 43200 and we are only going to test a small fraction of 100 of them, i.e. as many as the number of the heuristic search iterations.

We also initialize a dataframe which will hold the results, for later examination.

from collections import OrderedDict ratio_neg_to_pos = sum(trainY==0)/sum(trainY==1) ## = 608 print('Ratio of negative to positive instances: {:6.1f}'.format(ratio_neg_to_pos)) ## parameters to be tuned tune_dic = OrderedDict() tune_dic['max_depth']= [5,10,15,20,25] ## maximum tree depth tune_dic['subsample']=[0.5,0.6,0.7,0.8,0.9,1.0] ## proportion of training instances used in trees tune_dic['colsample_bytree']= [0.5,0.6,0.7,0.8,0.9,1.0] ## subsample ratio of columns tune_dic['eta']= [0.01,0.05,0.10,0.20,0.30,0.40] ## learning rate tune_dic['gamma']= [0.00,0.05,0.10,0.15,0.20] ## minimum loss function reduction required for a split tune_dic['scale_pos_weight']=[30,40,50,300,400,500,600,700] ## relative weight of positive/negative instances lengths = [len(lst) for lst in tune_dic.values()] combs=1 for i in range(len(lengths)): combs *= lengths[i] print('Total number of combinations: {:16d}'.format(combs)) maxiter=100 columns=[*tune_dic.keys()]+['F-Score','Best F-Score'] results = pd.DataFrame(index=range(maxiter), columns=columns) ## dataframe to hold training results

Ratio of negative to positive instances: 595.5 Total number of combinations: 43200 

Next we define two functions:

Function perf_measures() accepts some predictions and labels, optionally prints the confusion matrix, and returns the F-Score This is a measure of performance combining precision and recall and will guide the heuristic search.

Function do_train() accepts as parameters:

• the current choice of variable parameters in a dictionary (cur_choice),
• the full dictionary of parameters to be passed to the main XGBoost training routine (param),
• a train dataset in XGBoost format (train),
• a string identifier (train_s),
• its labels (trainY),
• the corresponding arguments for a validation dataset (valid, valid_s, validY),
• and the option to print the confusion matrix (print_conf_matrix).

It then trains the model and returns the F-score of the predictions on the validation dataset together with the model. The call to the main train routine xgb.train() has the following arguments:

• the full dictionary of the parameters (param),
• the train dataset in XGBoost format (train),
• the number of boosting iterations (num_boost),
• [ a watchlist with datasets information to show progress (evals) ] --> this is commented-out below.

def perf_measures(preds, labels, print_conf_matrix=False): act_pos=sum(labels==1) ## actual positive act_neg=len(labels) - act_pos ## actual negative pred_pos=sum(1 for i in range(len(preds)) if (preds[i]>=0.5)) ## predicted positive true_pos=sum(1 for i in range(len(preds)) if (preds[i]>=0.5) & (labels[i]==1)) ## predicted negative false_pos=pred_pos - true_pos ## false positive false_neg=act_pos-true_pos ## false negative true_neg=act_neg-false_pos ## true negative precision = true_pos/pred_pos ## tp/(tp+fp) percentage of correctly classified predicted positives recall = true_pos/act_pos ## tp/(tp+fn) percentage of positives correctly classified f_score = 2*precision*recall/(precision+recall) if print_conf_matrix: print('\nconfusion matrix') print('----------------') print( 'tn:{:6d} fp:{:6d}'.format(true_neg,false_pos)) print( 'fn:{:6d} tp:{:6d}'.format(false_neg,true_pos)) return(f_score) def do_train(cur_choice, param, train,train_s,trainY,valid,valid_s,validY,print_conf_matrix=False): ## train with given fixed and variable parameters ## and report the F-score on the validation dataset print('Parameters:') for (key,value) in cur_choice.items(): print(key,': ',value,' ',end='') param[key]=value print('\n') ## the commented-out segment below uses a watchlist to monitor the progress of the boosting iterations  ## evallist = [(train,train_s), (valid,valid_s)] ## model = xgb.train( param, train, num_boost_round=num_rounds, ## evals=evallist,verbose_eval=False)  model = xgb.train( param, train, num_boost_round=num_rounds) preds = model.predict(valid) labels = valid.get_label() f_score = perf_measures(preds, labels,print_conf_matrix) return(f_score, model) 

Next we define a function next_choice() which either produces a random combination of the variable parameters (if no current parameters are passed with cur_params) or generates a neighboring combination of the parameters passed in cur_params.

In the second case we first select at random a parameter to be changed. Then:

• If this parameter currently has the smallest value, we select the next (larger) one.
• If this parameter currently has the largest value, we select the previous (smaller) one.
• Otherwise, we select the left (smaller) or right (larger) value randomly.

Repetitions are avoided in the function which carries out the heuristic search.

def next_choice(cur_params=None): ## returns a random combination of the variable parameters (if cur_params=None) ## or a random neighboring combination from cur_params if cur_params: ## chose parameter to change ## parameter name and current value choose_param_name, cur_value = random.choice(list(cur_choice.items())) ## parameter name  all_values = list(tune_dic[choose_param_name]) ## all values of selected parameter cur_index = all_values.index(cur_value) ## current index of selected parameter if cur_index==0: ## if it is the first in the range select the second one next_index=1 elif cur_index==len(all_values)-1: ## if it is the last in the range select the previous one next_index=len(all_values)-2 else: ## otherwise select the left or right value randomly direction=np.random.choice([-1,1]) next_index=cur_index + direction next_params = dict((k,v) for k,v in cur_params.items()) next_params[choose_param_name] = all_values[next_index] ## change the value of the selected parameter print('selected move: {:10s}: from {:6.2f} to {:6.2f}'. format(choose_param_name, cur_value, all_values[next_index] )) else: ## generate a random combination of parameters next_params=dict() for i in range(len(tune_dic)): key = [*tune_dic.keys()][i] values = [*tune_dic.values()][i] next_params[key] = np.random.choice(values) return(next_params) 

At each iteration of the Simulated Annealing algorith, one combination of hyper-parameters is selected. The XGBoost algorithm is trained with these parameters and the F-score on the validation set is produced.

• If this F-score is better (larger) than the one at the previous iteration, i.e. there is a "local" improvement, the combination is accepted as the current combination and a neighbouring combination is selected for the next iteration through a call to the next_choice() function.
• Otherwise, i.e. if this F-score is worse (smaller) than the one at the previous iteration and the decline is Δf < 0, the combination is accepted as the current one with probability exp(-beta Δf/T) where beta is a constant and T is the current "temperature". The idea is that we start with a high temperature and "bad" solutions are easily accepted at first, in the hope of exploring wide areas of the search space. But as the temperature drops, bad solutions are less likely to be accepted and the search becomes more focused.

The temperature starts at a fixed value T0 and is reduced by a factor of alpha < 1 every n number of iterations. Here T0 = 0.40, n=5 and alpha = 0.85. The beta constant is 1.3.

The selection of the parameters of this "cooling schedule" can be done easily in MS Excel. In this example we select the average acceptance probabilities for F-Score deterioration of 0.150, 0.075, 0.038, 0.019, 0.009, i.e. starting from 0.150 and dividing by 2. We set these average probabilities to be around 50% during the first, second,...,fifth 20 iterations respectively and we use Excel Solver to find suitable parameters. The Excel file can be found here.

Repetitions are avoided with a simple hashing scheme.

A warning: if the number of iterations is not suficiently smaller than the total number of combinations, there may be too many repetitions and delays. The simple approach for producing combinations implemented here does not address such cases.

import time t0 = time.clock() T=0.40 best_params = dict() ## initialize dictionary to hold the best parameters best_f_score = -1. ## initialize best f-score prev_f_score = -1. ## initialize previous f-score prev_choice = None ## initialize previous selection of parameters weights = list(map(lambda x: 10**x, [0,1,2,3,4])) ## weights for the hash function hash_values=set() for iter in range(maxiter): print('\nIteration = {:5d} T = {:12.6f}'.format(iter,T)) ## find next selection of parameters not visited before while True: cur_choice=next_choice(prev_choice) ## first selection or selection-neighbor of prev_choice ## indices of the selections in alphabetical order of the parameters  indices=[tune_dic[name].index(cur_choice[name]) for name in sorted([*tune_dic.keys()])] ## check if selection has already been visited hash_val = sum([i*j for (i, j) in zip(weights, indices)]) if hash_val in hash_values: print('\nCombination revisited - searching again') # tmp=abs(results.loc[:,[*cur_choice.keys()]] - list(cur_choice.values())) # tmp=tmp.sum(axis=1) # if any(tmp==0): ## selection has already been visited # print('\nCombination revisited - searching again') else: hash_values.add(hash_val) break ## break out of the while-loop ## train the model and obtain f-score on the validation dataset f_score,model=do_train(cur_choice, param, dtrain,'train',trainY,dvalid,'valid',validY) ## store the parameters results.loc[iter,[*cur_choice.keys()]]=list(cur_choice.values()) print(' F-Score: {:6.2f} previous: {:6.2f} best so far: {:6.2f}'.format(f_score, prev_f_score, best_f_score)) if f_score > prev_f_score: print(' Local improvement') ## accept this combination as the new starting point prev_f_score = f_score prev_choice = cur_choice ## update best parameters if the f-score is globally better if f_score > best_f_score: best_f_score = f_score print(' Global improvement - best f-score updated') for (key,value) in prev_choice.items(): best_params[key]=value else: ## f-score is smaller than the previous one ## accept this combination as the new starting point with probability exp(-(1.6 x f-score decline)/temperature)  rnd = random.random() diff = f_score-prev_f_score thres=np.exp(1.3*diff/T) if rnd <= thres: print(' Worse result. F-Score change: {:8.4f} threshold: {:6.4f} random number: {:6.4f} -> accepted'. format(diff, thres, rnd)) prev_f_score = f_score prev_choice = cur_choice else: ## do not update previous f-score and previous choice print(' Worse result. F-Score change: {:8.4f} threshold: {:6.4f} random number: {:6.4f} -> rejected'. format(diff, thres, rnd)) ## store results results.loc[iter,'F-Score']=f_score results.loc[iter,'Best F-Score']=best_f_score if iter % 5 == 0: T=0.85*T ## reduce temperature every 5 iterations and continue  print('\n{:6.1f} minutes process time\n'.format((time.clock() - t0)/60)) print('Best variable parameters found:\n') print(best_params)

Iteration = 0 T = 0.400000 Parameters: eta : 0.05 colsample_bytree : 0.9 scale_pos_weight : 300 gamma : 0.2 max_depth : 20 subsample : 0.5 F-Score: 0.80 previous: -1.00 best so far: -1.00 Local improvement Global improvement - best f-score updated Iteration = 1 T = 0.340000 selected move: gamma : from 0.20 to 0.15 Parameters: eta : 0.05 colsample_bytree : 0.9 scale_pos_weight : 300 gamma : 0.15 max_depth : 20 subsample : 0.5 F-Score: 0.80 previous: 0.80 best so far: 0.80 Worse result. F-Score change: 0.0000 threshold: 1.0000 random number: 0.1169 -> accepted Iteration = 2 T = 0.340000 selected move: eta : from 0.05 to 0.10 Parameters: eta : 0.1 colsample_bytree : 0.9 scale_pos_weight : 300 gamma : 0.15 max_depth : 20 subsample : 0.5 F-Score: 0.80 previous: 0.80 best so far: 0.80 Worse result. F-Score change: -0.0087 threshold: 0.9672 random number: 0.9110 -> accepted Iteration = 3 T = 0.340000 selected move: max_depth : from 20.00 to 15.00 Parameters: eta : 0.1 colsample_bytree : 0.9 scale_pos_weight : 300 gamma : 0.15 max_depth : 15 subsample : 0.5 F-Score: 0.80 previous: 0.80 best so far: 0.80 Local improvement Iteration = 4 T = 0.340000 selected move: eta : from 0.10 to 0.05 Parameters: eta : 0.05 colsample_bytree : 0.9 scale_pos_weight : 300 gamma : 0.15 max_depth : 15 subsample : 0.5 F-Score: 0.80 previous: 0.80 best so far: 0.80 Worse result. F-Score change: -0.0001 threshold: 0.9998 random number: 0.6716 -> accepted Iteration = 5 T = 0.340000 selected move: eta : from 0.05 to 0.01 Parameters: eta : 0.01 colsample_bytree : 0.9 scale_pos_weight : 300 gamma : 0.15 max_depth : 15 subsample : 0.5 F-Score: 0.81 previous: 0.80 best so far: 0.80 Local improvement Global improvement - best f-score updated Iteration = 6 T = 0.289000 selected move: eta : from 0.01 to 0.05 Combination revisited - searching again selected move: scale_pos_weight: from 300.00 to 400.00 Parameters: eta : 0.01 colsample_bytree : 0.9 scale_pos_weight : 400 gamma : 0.15 max_depth : 15 subsample : 0.5 F-Score: 0.81 previous: 0.81 best so far: 0.81 Local improvement Global improvement - best f-score updated Iteration = 7 T = 0.289000 selected move: colsample_bytree: from 0.90 to 1.00 Parameters: eta : 0.01 colsample_bytree : 1.0 scale_pos_weight : 400 gamma : 0.15 max_depth : 15 subsample : 0.5 F-Score: 0.81 previous: 0.81 best so far: 0.81 Worse result. F-Score change: -0.0043 threshold: 0.9809 random number: 0.0174 -> accepted Iteration = 8 T = 0.289000 selected move: eta : from 0.01 to 0.05 Parameters: eta : 0.05 colsample_bytree : 1.0 scale_pos_weight : 400 gamma : 0.15 max_depth : 15 subsample : 0.5 F-Score: 0.81 previous: 0.81 best so far: 0.81 Local improvement Global improvement - best f-score updated Iteration = 9 T = 0.289000 selected move: scale_pos_weight: from 400.00 to 500.00 Parameters: eta : 0.05 colsample_bytree : 1.0 scale_pos_weight : 500 gamma : 0.15 max_depth : 15 subsample : 0.5 F-Score: 0.82 previous: 0.81 best so far: 0.81 Local improvement Global improvement - best f-score updated 

...

Iteration = 96 T = 0.015504 selected move: scale_pos_weight: from 500.00 to 600.00 Parameters: eta : 0.1 colsample_bytree : 1.0 scale_pos_weight : 600 gamma : 0.1 max_depth : 15 subsample : 0.5 F-Score: 0.81 previous: 0.82 best so far: 0.86 Worse result. F-Score change: -0.0029 threshold: 0.7819 random number: 0.4103 -> accepted Iteration = 97 T = 0.015504 selected move: max_depth : from 15.00 to 20.00 Parameters: eta : 0.1 colsample_bytree : 1.0 scale_pos_weight : 600 gamma : 0.1 max_depth : 20 subsample : 0.5 F-Score: 0.82 previous: 0.81 best so far: 0.86 Local improvement Iteration = 98 T = 0.015504 selected move: scale_pos_weight: from 600.00 to 500.00 Parameters: eta : 0.1 colsample_bytree : 1.0 scale_pos_weight : 500 gamma : 0.1 max_depth : 20 subsample : 0.5 F-Score: 0.82 previous: 0.82 best so far: 0.86 Worse result. F-Score change: -0.0055 threshold: 0.6282 random number: 0.5258 -> accepted Iteration = 99 T = 0.015504 selected move: subsample : from 0.50 to 0.60 Combination revisited - searching again selected move: gamma : from 0.10 to 0.15 Parameters: eta : 0.1 colsample_bytree : 1.0 scale_pos_weight : 500 gamma : 0.15 max_depth : 20 subsample : 0.5 F-Score: 0.82 previous: 0.82 best so far: 0.86 Worse result. F-Score change: 0.0000 threshold: 1.0000 random number: 0.2742 -> accepted 19.2 minutes process time Best variable parameters found: {'eta': 0.4, 'colsample_bytree': 1.0, 'scale_pos_weight': 50, 'gamma': 0.1, 'max_depth': 15, 'subsample': 0.5} 

The evaluation on the test dataset results to an F-Score of 0.86 which is considered good given the high imbalance in the classes. The run time was 19 minutes on a 64-bit Windows system with 6GB RAM and an Intel Core i3 processor 2.30GHz.

The best hyper-parameters found are in the ranges expected to be good according to all sources. One can then proceed this way:

• Narrowing the ranges of these hyper-parameters,
• Possibly adding others which are not used here (for example, regularization parameters),
• Possibly doing some variable selection on the basis of the variable importance information,
• Importantly, combining different models, ensemble-like.

print('\nBest parameters found:\n') print(best_params) print('\nEvaluation on the test dataset\n') best_f_score,best_model=do_train(best_params, param, dtrain,'train',trainY,dtest,'test',testY,print_conf_matrix=True) print('\nF-score on the test dataset: {:6.2f}'.format(best_f_score)) f, (ax1,ax2) = plt.subplots(nrows=1, ncols=2, sharey=False, figsize=(8,5)) ax1.plot(results['F-Score']) ax2.plot(results['Best F-Score']) ax1.set_xlabel('Iterations',fontsize=11) ax2.set_xlabel('Iterations',fontsize=11) ax1.set_ylabel('F-Score',fontsize=11) ax2.set_ylabel('Best F-Score',fontsize=11) ax1.set_ylim([0.7,0.9]) ax2.set_ylim([0.7,0.9]) plt.tight_layout() plt.show() print('\nVariables importance:\n') p = xgb.plot_importance(best_model) plt.show()

Best parameters found: {'eta': 0.4, 'colsample_bytree': 1.0, 'scale_pos_weight': 50, 'gamma': 0.1, 'max_depth': 15, 'subsample': 0.5} Evaluation on the test dataset Parameters: eta : 0.4 colsample_bytree : 1.0 scale_pos_weight : 50 gamma : 0.1 max_depth : 15 subsample : 0.5 confusion matrix ---------------- tn: 85265 fp: 18 fn: 25 tp: 134 F-score on the test dataset: 0.86 

Variables importance: