I have been playing with a toy problem to compare the performance and behavior of several scikit-learn classifiers.
Brief, I have one continuous variable X (which contains two samples of size N, each drawn from a distinct normal distributions) and a corresponding label y (either 0 or 1).
X is built as follows:
# Subpopulation 1 s1 = np.random.normal(mu1, sigma1, n1) l1 = np.zeros(n1) # Subpopulation 2 s2 = np.random.normal(mu2, sigma2, n2) l2 = np.ones(n2) # Merge the subpopulations X = np.concatenate((s1, s2), axis=0).reshape(-1, 1) y = np.concatenate((l1, l2)) n1, n2: number of data points in each sub-population; mu1, sigma1, mu2, sigma1: mean and standard deviation of each population from which the sample is drawn.
I then split X and y into training and test set:
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.25) And then I fit a series of models, for instance:
from sklearn import svm clf = svm.SVC() # Fit clf.fit(X_train, y_train) or, alternatively (full list in the table at the end):
from sklearn.ensemble import RandomForestClassifier rfc = RandomForestClassifier() # Fit rfc.fit(X_train, y_train) For all models, I then calculate the accuracy on the training and the test sets. For this I implemented following function:
def apply_model_and_calc_accuracies(model): # Calculate accuracy on training set y_train_hat = model.predict(X_train) a_train = 100 * sum(y_train == y_train_hat) / y_train.shape[0] # Calculate accuracy on test set y_test_hat = model.predict(X_test) a_test = 100 * sum(y_test == y_test_hat) / y_test.shape[0] # Return accuracies return a_train, a_test I compare the algorithms by changing n1, n2, mu1, sigma1, mu2, sigma1 and checking the accuracies of the training and test sets. I initialize the classifiers with their default parameters.
To make a long story short, the Random Forest Classifier always scores 100% accuracy on the test test, no matter what parameters I set.
If, for instance, I test the following parameters:
n1 = n2 = 250 mu1 = mu2 = 7.0 sigma1 = sigma2 = 3.0, I merge two completely overlapping subpopulations into X (they still have the correct label y associated to them). My expectation for this experiment is that the various classifiers should be completely guessing, and I would expect a test accuracy of around 50%.
In reality, this is what I get:
| Algorithm | Train Accuracy % | Test Accuracy % | |----------------------------|------------------|-----------------| | Support Vector Machines | 56.3 | 42.4 | | Logistic Regression | 49.1 | 52.8 | | Stochastic Gradien Descent | 50.1 | 50.4 | | Gaussian Naive Bayes | 50.1 | 52.8 | | Decision Tree | 100.0 | 51.2 | | Random Forest | 100.0 | *100.0* | | Multi-Layer Perceptron | 50.1 | 49.6 |
I don't understand how this is possible. The Random Forest classifier never sees the test set during training, and still classify with 100% accuracy.
Thanks for any input!
Upon request, I paste my code here (with only two of the originally tested classifiers and less verbose outputs).
import numpy as np import sklearn import matplotlib.pyplot as plt # Seed np.random.seed(42) # Subpopulation 1 n1 = 250 mu1 = 7.0 sigma1 = 3.0 s1 = np.random.normal(mu1, sigma1, n1) l1 = np.zeros(n1) # Subpopulation 2 n2 = 250 mu2 = 7.0 sigma2 = 3.0 s2 = np.random.normal(mu2, sigma2, n2) l2 = np.ones(n2) # Display the data plt.plot(s1, np.zeros(n1), 'r.') plt.plot(s2, np.ones(n1), 'b.') # Merge the subpopulations X = np.concatenate((s1, s2), axis=0).reshape(-1, 1) y = np.concatenate((l1, l2)) # Split in training and test sets from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.25) print(f"Train set contains {X_train.shape[0]} elements; test set contains {X_test.shape[0]} elements.") # Display the test data X_test_0 = X_test[y_test == 0] X_test_1 = X_test[y_test == 1] plt.plot(X_test_0, np.zeros(X_test_0.shape[0]), 'r.') plt.plot(X_test_1, np.ones(X_test_1.shape[0]), 'b.') # Define a commodity function def apply_model_and_calc_accuracies(model): # Calculate accuracy on training set y_train_hat = model.predict(X_train) a_train = 100 * sum(y_train == y_train_hat) / y_train.shape[0] # Calculate accuracy on test set y_test_hat = model.predict(X_test) a_test = 100 * sum(y_test == y_test_hat) / y_test.shape[0] # Return accuracies return a_train, a_test # Classify # Use Decision Tree from sklearn import tree dtc = tree.DecisionTreeClassifier() # Fit dtc.fit(X_train, y_train) # Calculate accuracy on training and test set a_train_dtc, a_test_dtc = apply_model_and_calc_accuracies(dtc) # Report print(f"Training accuracy = {a_train_dtc}%; test accuracy = {a_test_dtc}%") # Use Random Forest from sklearn.ensemble import RandomForestClassifier rfc = RandomForestClassifier() # Fit rfc.fit(X, y) # Calculate accuracy on training and test set a_train_rfc, a_test_rfc = apply_model_and_calc_accuracies(rfc) # Report print(f"Training accuracy = {a_train_rfc}%; test accuracy = {a_test_rfc}%") 
