I'd like to use the output of an xgboost BDT model in a code base without having to rely explicitly on xgboost or otherwise. Using a modified version of this script, xgb2cpp I am able to generate a cpp function which, to my eye, takes exactly the output of a trained network dumped to a txt file and converts to a cpp function. However, when running this function with the exact same inputs, I cannot replicate probabilities output by xgboost itself.
Here's a minimal working example of the code used to make the BDT:
import xgboost as xgb import numpy as np # Create dummy data for training and testing X_train = np.random.rand(100, 4) y_train = np.random.randint(0, 2, size=100) X_test = np.random.rand(20, 4) clf = xgb.XGBClassifier(n_estimators=2, min_child_weight=5, gamma=0.5, max_depth=2) clf.fit(X_train, y_train) y_pred_proba = clf.predict_proba(X_test) # dump model to txt file clf.get_booster().dump_model('test_minimal/dump.raw.txt') print(X_test) print(y_pred_proba) The dumped model looks like this:
booster[0]: 0:[f0<0.460618258] yes=1,no=2,missing=2 1:leaf=-0.10681767 2:[f2<0.684740245] yes=3,no=4,missing=4 3:leaf=0.000418715936 4:leaf=0.176613688 booster[1]: 0:[f3<0.255429506] yes=1,no=2,missing=2 1:leaf=-0.136423871 2:[f0<0.607831895] yes=3,no=4,missing=4 3:leaf=-0.00219070958 4:leaf=0.125992939 with corresponding cpp:
float classify(std::vector<float> &sample) { float sum = 0.0; if (sample[0] <0.460618258) { sum += -0.10681767; } else { if (sample[2] <0.684740245) { sum += 0.000418715936; } else { sum += 0.176613688; } } if (sample[3] <0.255429506) { sum += -0.136423871; } else { if (sample[0] <0.607831895) { sum += -0.00219070958; } else { sum += 0.125992939; } } return sum; } When testing for probabilities I pass the output of the cpp function to the sigmoid function to convert to a probability:
float sigmoid(float x) { return 1.0 / (1.0 + exp(-x)); } But this result does not match the result from xbgoost... so what gives? Does xgboost do something I am not realizing? I checked out their code and it seems it does exactly what I've done. Others have pointed out that a starting value of 0.5 is used by default, so I even accounted for this:
float sigmoid(float x) { return 1.0 / (1.0 + exp(-x + 0.5)); } Either I'm missing something or this isn't currently possible. Any help is appreciated!
