2

i am using scikit-learn to classify my data, at the moment i am running a simple DecisionTree classifier. I have three classes with a big imbalanced problem. The classes are 0,1 and 2. The minor classes are 1 and 2.

To give you an idea about the number of samples of the classes:

0 = 25.000 samples 1 = 15/20 less or more 2 = 15/20 less or more

so minor classes are about 0.06% of the dataset. The approach that i am following to solve the imbalance problem is the UPSAMPLING of the minor classes. Code:

from sklearn.utils import resample, resample(data, replace=True, n_samples=len_major_class, random_state=1234)

Now comes the problem. I did two tests:

  1. If I upsample the minor classes and then divide my dataset in two groups one for training and one for testing... the accuracy is:
 precision recall f1-score support 0 1.00 1.00 1.00 20570 1 1.00 1.00 1.00 20533 2 1.00 1.00 1.00 20439 avg / total 1.00 1.00 1.00 61542

very good result.

  1. If I ONLY upsample the training data and leave the original data for testing, the result is:
 precision recall f1-score support 0 1.00 1.00 1.00 20570 1 0.00 0.00 0.00 15 2 0.00 0.00 0.00 16 avg / total 1.00 1.00 1.00 20601

as you can see the global accuracy is high, but the accuracy of the class 1 and 2 is zero.

I am creating the classifier in this way:

DecisionTreeClassifier(max_depth=20, max_features=0.4, random_state=1234, criterion='entropy')

I also have tried adding the class_weight with balanced value, but it makes no difference.

I should only upsample the training data, why am i getting this strange problem?

Improve this question

3 Answers 3

Reset to default
8

The fact that you obtain that behavior is quite normal when you do the re-sampling before the splitting; you are inducing a bias in your data.

If you oversample the data and then split, the minority samples in the test won't be anymore independent from the samples in the training set because they are generated together. In your case they are exact copies of the samples in the training set. Your accuracy is 100% because the classifier is classifying samples that have already been seen in the training.

Since your problem is strongly umbalanced I would suggest to use an ensemble of classifiers to handle it. 1) Split your dataset in training set and test set. Given the size of the dataset you can sample 1-2 samples from the minority class for test and leave the other for training. 2) From the training you generate N datasets containing all the remaining samples of the minority class and under-samples from the majority class (i would say 2*number of samples in the minority class). 3) For each one of the dataset obtained you train a model. 4) Use the test set to obtain the prediction; the final prediction will be the results of a majority vote of all the predictions of the classifiers.

To have robust metrics perform different iterations with different initial splitting test/training.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

3

You should not split the dataset after upsampling. You can do the upsampling within the training data.

Basically, you are leaking the test data into the training data.

Improve this answer

Comments

0

I have a function that resamples the dataset for each class to have the same amount of instance.

from sklearn.utils import resample import pandas as pd def make_resample(_df, column): dfs_r = {} dfs_c = {} bigger = 0 ignore = "" for c in _df[column].unique(): dfs_c[c] = _df[df[column] == c] if dfs_c[c].shape[0] > bigger: bigger = dfs_c[c].shape[0] ignore = c for c in dfs_c: if c == ignore: continue dfs_r[c] = resample(dfs_c[c], replace=True, n_samples=bigger - dfs_c[c].shape[0], random_state=0) return pd.concat([dfs_r[c] for c in dfs_r] + [_df])
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.