2
$\begingroup$

I am using the forward feature selection algorithm from MATLAB. The code is as follows:

 X=combine_6_non; y=target; c = cvpartition(y,'k',10); opts = statset('display','iter'); [fs,history] = sequentialfs(@fun,X,y,'cv',c,'options',opts)

The function fun is as follows:

function err = fun(XT,yT,Xt,yt) model = svmtrain(XT,yT, 'Kernel_Function', 'rbf', 'boxconstraint', 1); err = sum(svmclassify(model, Xt) ~= yt); end

Now for different runs of the selection algorithm I am getting different feature sets. How should I zero down to the best feature set?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Is it possible to set a seed or random_state? If yes, then that should solve the issue :) $\endgroup$ Commented Aug 5, 2016 at 8:18
  • $\begingroup$ @Dawny33 what are these things? $\endgroup$ Commented Aug 5, 2016 at 8:23
  • 1
    $\begingroup$ Some algorithms are (partly) stochastic, thus randomly choosing value to explore the space of solutions. However, computers cannot generate random numbers but pseudo-random numbers that are simulated based on an initial seed. If you fix it, you should get the same results each time. There is no randomness in the SVM, thus I would guess it is in the splitting part $\endgroup$ Commented Aug 5, 2016 at 8:38
  • $\begingroup$ @Dawny33 so the partitioning is leading to this right? $\endgroup$ Commented Aug 5, 2016 at 8:40
  • $\begingroup$ @Rishika Yes. So, pl read up on how to set a random_state or a seed in Matlab, similar to Python :) $\endgroup$ Commented Aug 5, 2016 at 8:47

2 Answers 2

Reset to default
0
$\begingroup$

You can set a seed or a random state for the splitting process. This helps in generating a fixed random number everytime, which helps you get the same data everytime you do the CV split.

This can be done by set_seed(...) in R, and adding the random_state = ... in relevant functions parameters in Python. So, please add the relevant parameter for Matlab, and your CV would do good.

Improve this answer
$\endgroup$
0
$\begingroup$

Setting the seed will not entirely solve the problem of obtaining different optimal feature sets. It will result in an arbitrarily optimal one. The feature selection algorithm obtaining different optimal feature sets for different cv-fold setups or initialisations, implies that your ML problem is sensitive to the data and hyperparameters.

Ideally you should increasing your model capacity or the number of examples until the optimal feature sets are stable between runs. If that's not an option or does not improve the issue the conservative approach would be to use the intersection of these feature sets, or less conservatively, the union.

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.