Select top numerical features for a classification problem

First, let's break the problem:

First, What do we have?

• Models (which are "classes"): A, B, and C. This is the target variable for the classification problem.
• Tasks: Task 1 and Task 2. These are essentially different "data points" or "samples" for each model.
• Features: Feature 1, Feature 2, and Feature 3. These are the numerical measured attributes.

Second, what is the goal?

The goal is to build a classifier that takes the features of a new, unseen task and predicts which model (A, B, or C) would be the best fit.

Third, how can we do it?

Two options you can do which are:

Option 1: Statistical Feature Selection

Now, as per the question - the solution thoughts were to select features Statistically.

Nothing wrong with that, but you need to select the statistical methodology based on your data analysis where you need to build a hypothesis, calculate its correlation with other features, run classification, compare accuracy/precision/recall/f1-scores.

It is ongoing cycle where you might select different features/algorithms/models/evaluations based on your needs. Consideration such as features information loss tolerance, data type, data size, performance, or any other constraint must be studied within the hypothesis to make a rational decision on which best statistical algorithm to use.

For example, Chi-square test is typically used with categorical data. It measures the association between two categorical variables. While you can adapt Chi-Square for numerical data by binning it (i.e., converting it to categorical data), this can be tricky and may lead to a loss of information. Therefore, you have decided to select other algorithm based on xyz because of xyz.

As for aggregating the data, you need to aggregate per features since your ultimate goal is to select best-fit features for the classification purpose.

Here's how you can structure your data for feature selection:

A. Visualize your overall data.

 Sample Model (Target) Feature 1 Feature 2 Feature 3 Task 1 Model A 20 40 55 Task 2 Model A 77 30 22 Task 1 Model B 10 70 33 Task 2 Model B 88 79 97 Task 1 Model C 45 65 75 Task 2 Model C 30 40 99 

B. Aggregate your data based on features.

Assuming you are analyzing Feature1:

 Values for Model A: [20, 77] Values for Model B: [10, 88] Values for Model C: [45, 30] 

C. Calculate feature scores across all metrics.

Example: Giving x rational and instead of using Chi-Square, I will be using F-score and p-value.

 Feature F-Score p-value Feature2 5.2 0.045 Feature3 2.1 0.210 Feature1 0.8 0.505 

In this example, Feature 2 would be selected as the top feature because its p-value and F-score fits the criteria assigned earlier.

Option 2: Feature Selection using Machine Learning Models

In this option, you can use Model-Based Feature Importance or wrapper methods such as Recursive Feature Elimination.

The idea is not so far from option1. You select a classifier, train it, get importance scores, eliminate features, select best features, and repeat.

You can use Random Forest/Gradient Boosting Machines/Decision Tree.

Again, your selection here is solely depends on your data analysis, accuracy, and any constraint on your way.