I am doing a random forest regression on my dataset (which has abut 15 input features and 1 target feature). I am getting a decently low R^2 of <1 for both the train and test sets (please do let me know if <1 is not a good-enough R^2 score).
import pandas as pd import numpy as np from sklearn.ensemble import RandomForestRegressor from sklearn.model_selection import train_test_split # load dataset df = pd.read_csv('Dataset.csv') # split into input (X) and output (Y) variables X = df.drop(['ID_COLUMN', 'TARGET_COLUMN'], axis=1) Y = df.TARGET_COLUMN # Split the data into 67% for training and 33% for testing X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33) # Fitting the regression model to the dataset regressor = RandomForestRegressor(n_estimators = 100, random_state = 50) regressor.fit(X_train, Y_train.ravel()) # Using ravel() to avoid getting 'DataConversionWarning' warning message print("Predicting Values:") y_pred = regressor.predict(X_test) print("Getting Model Performance...") # Get regression scores print("R^2 train = ", regressor.score(X_train, Y_train)) print("R^2 test = ", regressor.score(X_test, Y_test)) This outputs the following:
Predicting Values: Getting Model Performance... R^2 train = 0.9791000275450427 R^2 test = 0.8577464692386905 Then, I checked the difference between the actual target column values in the test dataset versus the predicted values, like so:
diff = [] for i in range(len(y_pred)): if Y_test.values[i]!=0: # a few values were 0 which was causing the corresponding diff value to become inf diff.append(100*np.abs(y_pred[i]-Y_test.values[i])/Y_test.values[i]) # element-wise percentage error I found that the majority of the element-wise differences were between 40-60% and their mean was almost 50%!
np.mean(diff) >>> 49.07580695857447 So, which one is correct? Is the regression score correct and my model is good for this data, or is the element-wise error I calculated correct and the model didn't do well for this data? If its the latter, please advise on how to increase the prediction accuracy.
I also checked the rmse score:
import math rmse = math.sqrt(np.mean((np.array(Y_test) - y_pred)**2)) rmse >>> 3.67328471827293 This seems quite high for the model to have done a good job, but please correct me if I'm wrong.
And I also checked the R^2 scores for different number of estimators:
import matplotlib.pyplot as plt model = RandomForestRegressor(n_jobs=-1) # Try different numbers of n_estimators estimators = np.arange(10, 200, 10) scores = [] for n in estimators: model.set_params(n_estimators=n) model.fit(X_train, Y_train) scores.append(model.score(X_test, Y_test)) plt.title("Effect of n_estimators") plt.xlabel("n_estimator") plt.ylabel("score") plt.plot(estimators, scores) 
Please advise.
I tried using linear regression first, and got a very high MSE (which is why I was trying out random forest):
from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_squared_error, r2_score lr = LinearRegression() lr.fit(X_train, y_train) y_pred = lr.predict(X_test) # The coefficients print('Coefficients: \n', lr.coef_) # The mean squared error print("Mean squared error: %.2f" % mean_squared_error(y_test, y_pred)) # Explained variance score: 1 is perfect prediction print('Variance score: %.2f' % r2_score(y_test, y_pred)) Coefficients: [ 1.93829229e-01 -4.68738825e-01 2.01635420e-01 6.35902010e-01 6.57354434e-03 5.13180293e-03 2.84015810e-01 -1.31469084e-06 1.95335035e+00] Mean squared error: 86.92 Variance score: 0.08 
