0
$\begingroup$

I am trying to implement stacking model for a ML problem and having hard time figuring out the cross validation strategy. So far I have used 10-fold cross validation for all my models and would like continue using that stacking as well. Here's what I came up with but not sure if it makes sense,

  1. At each iteration of 10-fold CV, you will have 9 folds for training (training dataset) and 1 fold for testing (testing dataset).

  2. Divide the training dataset into 3 parts - F1, F2 and F3.

  3. Train base classifiers on F1, use F2 for early stopping and get out of fold predictions on F3 -> F3' (F3' is the set of predictions made by base-classifiers on F3)

  4. Train base classifiers on F2, use F3 for early stopping and get out of fold predictions on F1 -> F1'

  5. Train base classifiers on F3, use F1 for early stopping and get out of fold predictions on F2 -> F2'

  6. Train meta-classifier on (F1' + F2' + F3'), train base-classifiers on any two folds and use remaining fold for early-stopping.

  7. Validate meta-classifier on test dataset

Repeat these steps for every fold in 10 fold CV.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

In the diagram below that you can find here, it is shown the general approach to train a stacking model:

Thes steps are:

Given $T$ a train set of shape $mxp$

  1. Divide $T$ into $k$ folds
  2. Fit classifiers $M_1$, $M_2$,...,$M_n$ on $k-1$ and predict on fold $k$
  3. Save the predictions of the $M$ classifiers on fold $k$ (predictions on step 2)
  4. Repeat steps 2 and 3 $k$ times

After those steps, you will end up with a dataset $D$ of shape $mxn$

  1. Train a model $\Phi$ (meta model) on $D$ and retrain the $M$ different classifiers on $T$

Then you can use $\Phi$ to make predictions on unseen data

enter image description here

Hope it helps!

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer. My base-classifiers are neural nets so I need a validation set for early stopping. Can I use same set for validation of base-classifier and building out-of-fold predictions for meta-classifiers? $\endgroup$ Commented Feb 11, 2022 at 3:46
  • $\begingroup$ As far as I know, this approach is model agnostic, so you should be ok with neural networks as first-level classifiers. You could also have a look of this post $\endgroup$ Commented Feb 11, 2022 at 16:54

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.