I build an SVM classifier but get an inverse ROC curve. The AUC is only 0.08. I've used the same datasets to build a Logistic Regression classifier and a Decision Tree classifier, and the ROC curves for them look good.
Here are my codes for SVM:
from sklearn.svm import SVC svm = SVC(max_iter = 12, probability = True) svm.fit(train_x_sm, train_y_sm) svm_test_y = svm.predict(X = test_x) svm_roc = plot_roc_curve(svm, test_x, test_y) plt.show() Could anyone tell me what is wrong in my codes?
For any classification problem if AUC<0.5, you are not performing better than random(0.5).
Reason could be:
A valid strategy is to simply invert the predictions as:
invert_prob=1-prob Reference: ROC
One potential fix is to remove max_iter = 12 (which would set it to the scikit learn default of max_iter=-1). Using such a low value can lead to bad scores as you can see from the following example:
from sklearn.model_selection import train_test_split from sklearn.svm import SVC from sklearn.metrics import plot_roc_curve from sklearn.datasets import load_breast_cancer data = load_breast_cancer() X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2) model = SVC(max_iter=12, probability = True) model.fit(X_train, y_train) plot_roc_curve(model, X_test, y_test) results in
However, executing exactly the same code (max_iter=12 still) again gives a totally different result:
After removing max_iter=12 the code consistently produces higher AUCs around $0.95$ to $0.99$.
max_iter=12 your results can fluctuate a lot. What happens if you use rbf as kernel and fit the model 10 times? Do you always get a low AUC? $\endgroup$