As far as i know theoretically our model tend to be drifting/shifting as time goes on and need to be retrained. i wonder if its acceptable that instead of retraining the classification model, we keep everything of the model as it was and only adjust the probability threshold based on latest/newest data?
1 Answer
$\begingroup$ $\endgroup$
3 Retraining the model accounts for changes in the relationship between features and outcomes, or probabilities of outcomes.
Changing the threshold(s) accounts for changes in the cost of decisions relative to outcomes: Reduce Classification Probability Threshold
The two are conceptually distinct. Your data generating process may change without changes in the cost structure: retrain your model. Or your costs may change without changes in the DGP: adapt your decision process. Or both: do both.
- $\begingroup$ Thank you for the insightful and easy-to-understand explanation. I have a better grasp of how to distinguish the objectives between these 2 approaches. If I may, I'd like to validate my understanding. Assume a churn model with the expected cost of decisions to maintain the churn rate below 10% quarterly. After some time, the model consistently fails to achieve the unchanged expected cost result. In this case, retraining is wiser than just adjusting the threshold. Am I on the right track? $\endgroup$raffo– raffo2024-03-06 18:32:30 +00:00Commented Mar 6, 2024 at 18:32
- $\begingroup$ Probably so. It does sound like your underlying DGP may have changed, since your cost structure apparently hasn't. Then again, it's always good to investigate first and try to understand whether the DGP has indeed changed. Churn prediction is inherently hard, because we have a feedback loop: we take actions to influence our customers based on the model, so we consciously try to change the outcome based on the model. $\endgroup$Stephan Kolassa– Stephan Kolassa2024-03-07 07:48:18 +00:00Commented Mar 7, 2024 at 7:48
- $\begingroup$ Thank you for the discussion, I really appreciate your time. As an entry-level data officer, the insights you provided are quite helpful. $\endgroup$raffo– raffo2024-03-08 09:45:13 +00:00Commented Mar 8, 2024 at 9:45