2
$\begingroup$

I'm using k-means in every class of a binary classification problem and remove samples that have high distance from center of my features (21 features so 21 dimensions problem) before inserting data set to a neural network. After designing neural network model, now i want use this model for a new data set (out sample).

As you know we must use outlier detection parameters in per-process stage for out sample data (like normalization x-min(x)/max(x)-min(x) that will use max(x) and min(x) for normalization of out sample). what parameters of k-means algorithm should i use for out sample and how can I do that in MATLAB ?

Thanks.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

You have to calculate the distance of your test samples (out sample) from the previously defined centroids (second output in Matlab kmeans function) using the same metric (see pdist) and then apply the same cutoff values on the distances obtained.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for your fast answer. So there isn't any pre-defined function for that in MATLAB and we should manually do that. Can you add a brief code in your answer for more description? $\endgroup$ Commented Aug 25, 2014 at 11:38
  • $\begingroup$ I think we should use pdist2. Is that true? pdist2(out_sample_observations,centers_from_training_set_kmeans_output) $\endgroup$ Commented Aug 27, 2014 at 13:03
  • $\begingroup$ + We are using kmeans on every class (class 1 and class 2) in training_sample but in out sample data we only have one database without any labeling. So I have two center sets for every class in training sample and now i have only one data set (without labels) as out samples. How can I do outlier detection in this problem? $\endgroup$ Commented Aug 27, 2014 at 13:09
  • $\begingroup$ + I remove highest 5% distances after sorting distances in training sample. Is that true?. If yes now I should remove highest 5% distances in out sample (5%-after sorting) ? $\endgroup$ Commented Aug 27, 2014 at 14:00
  • $\begingroup$ pdist2 is indeed more handy here, about your 5% I'm not sure to follow you, once you have the distance you can remove anything above the threshold (the maximal distance allowed to a centroid) you defined on your training $\endgroup$ Commented Aug 28, 2014 at 7:43

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.