Weighted K-NN

Weighted K-NN

Weighted k-Nearest Neighbors (k-NN) is a variation of the k-NN algorithm where votes are weighted by the inverse of the distance between the test point and each of its k neighbors. Essentially, nearer neighbors influence the classification more than distant ones.

The primary motivation behind weighted k-NN is to improve the performance of the k-NN method by taking into account the distance from the test sample to the training samples.

How does Weighted k-NN work?

  1. For a given test sample, compute the distance to every training sample.
  2. Sort the training samples by their distance to the test sample and select the top k.
  3. Compute weights for each of the k samples. Typically, the weight is set as the inverse of the distance. For example, if distance d is used, the weight could be 1/d or 1/(d^2).
  4. For classification, sum the weights of each class label and classify the test sample as the label with the most weight.
  5. For regression, predict the output as the weighted average of the k samples' values.

Python Implementation

Let's see how you can implement the weighted k-NN for classification using Python and Scikit-Learn:

from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.neighbors import KNeighborsClassifier # Load iris dataset as an example data = load_iris() X = data.data y = data.target # Split data into training and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Use KNeighborsClassifier with weights set to 'distance' clf = KNeighborsClassifier(n_neighbors=3, weights='distance') clf.fit(X_train, y_train) # Predict for the test set y_pred = clf.predict(X_test) # Compute accuracy accuracy = (y_pred == y_test).mean() print(f"Accuracy: {accuracy:.4f}")

Here, the weights parameter in KNeighborsClassifier is set to 'distance', which means neighbors will be weighted by the inverse of their distance.

In Scikit-Learn, you can also provide a user-defined function for the weights parameter to compute custom weights based on distances, if desired.

More Tags

missing-data desktop sqldatareader capistrano3 mozilla samesite tensorboard rdbms mysql-event chisel

More Programming Guides

Other Guides

More Programming Examples