My Neural Network has an $R^2$ score of 0.30. What I know is that it is not zero and not negative so it's not super bad and it's not close to one so it's not perfect but it's hard to find any more useful information other than that.
Is $R^2$ even a good measurement for classification? Simplified my network tells if something is A or B so what does an $R^2$ score of 0.3 say about it?
I use this command from sklearn for r2_score: r2_score sklearn
$R^2$ is not a good measure to assess goodness of fit for a classification.
$R^2$ is suitable for predicting continuous variable. When dependent variable is continuous $R^2$ usually takes values between $0$ and $1$ (in linear regression for example it is impossible to have $R^2$ beyond these boundaries), and it is interpreted as share of variance of dependent variable that model is able to correctly reproduce. When $R^2$ equals $1$ it means that the model is able to fully recreate dependent variable, when it equals $0$, it means that the model completely failed at this task.
When the dependent variable is categorical it makes no sense, because $R^2$ uses distances between predicted and actual values, while distances between let say '1' meaning class 'A', '2' meaning class 'B' and '3' meaning class 'C' make no sense.
Use other measures, for example AUC for classification with two classes and Logarithmic Loss for classification with more classes. And make sure that you are using appropriate parameters in your model: in many machine learning models you must declare if problem is of classification or regression nature and it can affect results dramatically.
