I have used MATLAB code and get the two different row vectors A=1×18 and B=1×350. From both row vectors separately I need to remove the noisy data by using standard deviation. But the problem is that data in both row vectors are NOT normally distributed. Is there any way that I used standard deviation for reducing noise from non normally distributed data. Any guidance will be appreciated. Thanks
1 Answer
$\begingroup$ $\endgroup$
1 First, good practice to raise validity concerns here when removing outliers and/or filtering data. This may have strong affects of validity of results. An intro is here: When to remove outlier in preparing features for machine learning algorithm .
Second, is it possible to address the small dataset problem -- Can you collect more data? Redefine the population to produce more data? Use another data set that is similar in developing the model?
Lastly, this seems to be a filter problem, to get started on a solution, check MATLAB documentation for filters.
If the results are going to be used anywhere, probably a good idea to document all of your decisions and the first two concerns. Absent much more detail, experience says there is a pretty high risk to any conclusions based on this model.
- $\begingroup$ @ davmor thanks for your guidance. First I will try what you have mentioned in your answer, then will discuss. thanks $\endgroup$user57546– user575462018-09-10 14:30:20 +00:00Commented Sep 10, 2018 at 14:30