I currently have a model that has a pretty large dataset (50ishMB) and was performing pretty well with a 80:20 split. However, when I tried changing it up to a 50:50 split, the model performed 28% better than the 80:20 split. Note this is a time-series problem.
Since I have evaluated the model to perform better in that data split, is it fine to more forward with it or should I continue with the 80:20 split? Why?
Assigning more examples for training allows your model to be exposed to more types of hidden patterns in your data & presumably learn a better representation. While assigning more examples for testing gives you a better fidelity of the evaluation over these data patterns. Ultimately this is a project specific tradeoff you make.
CONSIDERATIONS
Does the evaluation need to be precise for a specific trained model or could it be a range approximation? If the latter then cross validation let’s you best use all the data to learn & evaluate.
Is the model used classical or deep learned? The latter techniques often require way more data so I would favor a 99:1 train/test split over the typical 70:30 or 80:20 splits.
Is the problem domain simple or extremely difficult? ie: are there sub patterns, many edge cases, lots of classes to learn with high human disagreement? You can plot learning curves to see how well your model learns with different volumes of data. Once it plateaus you can allocate excess data for evaluation.
How expensive is it to get quality & representative labeled data? If cheap consider creating ship-gates (unit test equivalent) datasets to capture various data scenarios.
Is the data heavily imbalanced, very noisy or very repetitive? if so I would apply smarter splitting techniques like stratified sampling, imbalance handling, etc over just random splitting.
It is common to allocate more data for training. I would attribute the perceived improved performance in your 50:50 data split simply to producing a less over-fitted model due to having fewer learning examples.
Using a 50:50 split is usually not recommended. People usually keep more data for training and less data for testing/validation.
The more train data you have, the better the model captures different scenarios of data and the more test data you have, the better your trained model will be evaluated. So it's a trade off between the two, and ultimately you have to decide which do you prefer more.
Since you mention you have a large dataset, choosing 50:50 wouldn't be that much of a problem compared to if you has a small dataset. But there would still be some data patterns you would be missing which in turn will make your model less generalizable so keep that in mind!
A possible solution to this trade off is cross-validation (preferably nested cross-validation). That way even if you have less training data, your model will use all of that data in the best possible way.
Cheers!
