Hey, I used it in python 2.7 and it seems to be working fine. Maybe some changes in the versions are giving rise to the warnings. Isn't the predictions working still?

use train_test_split of sklearn, it shuffles and splits

Oh sorry, yes - the one with the better test accuracy normally is what I use. But say your data is vary varied and you get very fluctuating validation accuracies, comparing the average of accuracies can also be tried.

once we have used cross-validation to select the better performing model(for instance you have 2 models-linear regression or neural network), we train that model (whether it be the linear regression or the neural network) on all the data

when you say test, you mean validation dataset's test right?

do you want the frequency of the phrase in the document? If that is the case, you can just use a dictionary counter right?

So when you say you don't use tf-idf, what are your features then? Just a simple BOG?

Maybe the dataset you contain is vary varied and the frequencies of the words don't matter to come to the conclusion?

@Richard, sentiment analysis for 2 classes is also a well received binary classification problem, did you try it with your datasets?