Return to Answer

The points you raised apply to K Nearest Neighbors Regression too.

Let me start by rephrasing what you wrote in terms of K-nearest neighbors regression:

"""

Given that K nearest neighbor (KNN) regression relies on a distance metric with specific hyperparameters [Number of neighbors $K$, Parameter $p$ and dimension wieghts $w_p$ in $\sum_{d=1}^D w_d (|x_{1,d}-x_{2,d}|^p)^{\frac{1}{p}}$] that control the dissimilarities between points, can KNN regression truly be considered a non-parametric technique? I have the following points.

1. A KNN regressor's behavior is largely determined by the distances parametric structure, which restricts the functional form to be consistent with certain assumptions (e.g., smoothness, periodicity). This makes it a deterministic process governed by hyperparameters, not an inherently flexible, parameter-free model.

2. Like any parametric technique (e.g., DFT or AR models), as the amount of data grows, KNNs allow for more precise estimates of hyperparameters that better fit the data. This improvement with data could be considered similar to reducing sub-optimality in parameter estimation rather than reflecting an open, parameter-free structure.

In summary, should KNN regression not be considered a parametric method due to the distance metric/$K$'s role as a definitive, parameterized structure that guides model behavior? In my opinion, it is indeed a parametric estimator.

"""

[Note: I have excluded Point 3 and other text about the fact that a GP is a probabilistic model, which is not relevant.]

So if GPs are parametric, so is KNN regression. Are they indeed both parametric?

How to tell the difference.

In my view, none of what you have mentioned has to do with parametric vs nonparametric. It might be confusing that "nonparametric" methods still have parameters, but they all do.

So how do we tell the difference? I like to use the following demarcation. A method is nonparametric if, after estimating the parameters, we still need the data to make predictions. Otherwise, if having the parameters means we can throw away the data and still make predictions, it is parametric.

Some examples. If I throw away the data, can I still make predictions at point $\mathbf{x}$?

1. Linear regression: yes, I just calculate $\beta^\top\mathbf{x}$. It must be parametric!
2. Neural network: yes, I just push $\mathbf{x}$ through my network. It must be parametric!
3. K nearest neighbors: no; just knowing $K$ and the distance metric is not enough to make any predictions: I need to know the average at the $K$ closest points to $\mathbf{x}$ under my metric. It must be nonparametric.
4. Gaussian processes: knowing a kernel function's parameters are not enough to make any predictions: I need to know the correlation between all training points and $\mathbf{x}$ as well as the inter-training correlations. Hence it is nonparametric.

The points you raised apply to K Nearest Neighbors Regression too.

Let me start by rephrasing what you wrote in terms of K-nearest neighbors regression:

"""

Given that K nearest neighbor (KNN) regression relies on a distance metric with specific hyperparameters [Number of neighbors $K$, Parameter $p$ and dimension wieghts $w_p$ in $\sum_{d=1}^D w_d (|x_{1,d}-x_{2,d}|^p)^{\frac{1}{p}}$] that control the dissimilarities between points, can KNN regression truly be considered a non-parametric technique? I have the following points.

1. A KNN regressor's behavior is largely determined by the distance's parametric structure, which restricts the functional form to be consistent with certain assumptions (e.g., smoothness, periodicity). This makes it a deterministic process governed by hyperparameters, not an inherently flexible, parameter-free model.

2. Like any parametric technique (e.g., DFT or AR models), as the amount of data grows, KNNs allow for more precise estimates of hyperparameters that better fit the data. This improvement with data could be considered similar to reducing sub-optimality in parameter estimation rather than reflecting an open, parameter-free structure.

In summary, should KNN regression not be considered a parametric method due to the distance metric/$K$'s role as a definitive, parameterized structure that guides model behavior? In my opinion, it is indeed a parametric estimator.

"""

[Note: I have excluded Point 3 and other text about the fact that a GP is a probabilistic model, which is not relevant.]

So if GPs are parametric, so is KNN regression. Are they indeed both parametric?

How to tell the difference.

In my view, none of what you have mentioned has to do with parametric vs nonparametric. It might be confusing that "nonparametric" methods still have parameters, but they all do.

So how do we tell the difference? I like to use the following demarcation. A method is nonparametric if, after estimating the parameters, we still need the data to make predictions. Otherwise, if having the parameters means we can throw away the data and still make predictions, it is parametric.

Some examples. If I throw away the data, can I still make predictions at point $\mathbf{x}$?

1. Linear regression: yes, I just calculate $\beta^\top\mathbf{x}$. It must be parametric!
2. Neural network: yes, I just push $\mathbf{x}$ through my network. It must be parametric!
3. K nearest neighbors: no; just knowing $K$ and the distance metric is not enough to make any predictions: I need to know the average at the $K$ closest points to $\mathbf{x}$ under my metric. It must be nonparametric.
4. Gaussian processes: knowing a kernel function's parameters are not enough to make any predictions: I need to know the correlation between all training points and $\mathbf{x}$ as well as the intra-training correlations. Hence it is nonparametric.

The points you raised apply to K Nearest Neighbors Regression too.

Let me start by rephrasing what you wrote in terms of K-nearest neighbors regression:

"""

Given that K nearest neighbor (KNN) regression relies on a distance metric with specific hyperparameters [Number of neighbors $K$, Parameter $p$ and dimension wieghts $w_p$ in $\sum_{d=1}^D w_d (|x_{1,d}-x_{2,d}|^p)^{\frac{1}{p}}$] that control the dissimilarities between points, can KNN regression truly be considered a non-parametric technique? I have the following points.

1. A KNN regressor's behavior is largely determined by the distances parametric structure, which restricts the functional form to be consistent with certain assumptions (e.g., smoothness, periodicity). This makes it a deterministic process governed by hyperparameters, not an inherently flexible, parameter-free model.

2. Like any parametric technique (e.g., DFT or AR models), as the amount of data grows, KNNs allow for more precise estimates of hyperparameters that better fit the data. This improvement with data could be considered similar to reducing sub-optimality in parameter estimation rather than reflecting an open, parameter-free structure.

In summary, should KNN regression not be considered a parametric method due to the distance metric/$K$'s role as a definitive, parameterized structure that guides model behavior? In my opinion, it is indeed a parametric estimator.

"""

[Note: I have excluded Point 3 and other text about the fact that a GP is a probabilistic model. This has nothing to do with parametric vs nonparametric.]

Why GP regression is Nonparametric.

In my view, none of what you have mentioned has to do with parametric vs nonparametric. It might be confusing that "nonparametric" methods still have parameters, but they all do.

So how do we tell the difference? I like to use the following demarcation. A method is nonparametric if, after estimating the parameters, we still need the data to make predictions. Otherwise, if having the parameters means we can throw away the data and still make predictions, it is parametric.

Some examples. If I throw away the data, can I still make predictions at point $\mathbf{x}$?

1. Linear regression: yes, I just calculate $\beta^\top\mathbf{x}$. It must be parametric!
2. Neural network: yes, I just push $\mathbf{x}$ through my network. It must be parametric!
3. K nearest neighbors: no; just knowing $K$ and the distance metric is not enough to make any predictions: I need to know the average at the $K$ closest points to $\mathbf{x}$ under my metric. It must be nonparametric.
4. Gaussian processes: knowing a kernel function's parameters are not enough to make any predictions: I need to know the correlation between all training points and $\mathbf{x}$ as well as the inter-training correlations. Hence it is nonparametric.

The points you raised apply to K Nearest Neighbors Regression too.

Let me start by rephrasing what you wrote in terms of K-nearest neighbors regression:

"""

Given that K nearest neighbor (KNN) regression relies on a distance metric with specific hyperparameters [Number of neighbors $K$, Parameter $p$ and dimension wieghts $w_p$ in $\sum_{d=1}^D w_d (|x_{1,d}-x_{2,d}|^p)^{\frac{1}{p}}$] that control the dissimilarities between points, can KNN regression truly be considered a non-parametric technique? I have the following points.

1. A KNN regressor's behavior is largely determined by the distances parametric structure, which restricts the functional form to be consistent with certain assumptions (e.g., smoothness, periodicity). This makes it a deterministic process governed by hyperparameters, not an inherently flexible, parameter-free model.

2. Like any parametric technique (e.g., DFT or AR models), as the amount of data grows, KNNs allow for more precise estimates of hyperparameters that better fit the data. This improvement with data could be considered similar to reducing sub-optimality in parameter estimation rather than reflecting an open, parameter-free structure.

In summary, should KNN regression not be considered a parametric method due to the distance metric/$K$'s role as a definitive, parameterized structure that guides model behavior? In my opinion, it is indeed a parametric estimator.

"""

[Note: I have excluded Point 3 and other text about the fact that a GP is a probabilistic model, which is not relevant.]

So if GPs are parametric, so is KNN regression. Are they indeed both parametric?

How to tell the difference.

In my view, none of what you have mentioned has to do with parametric vs nonparametric. It might be confusing that "nonparametric" methods still have parameters, but they all do.

So how do we tell the difference? I like to use the following demarcation. A method is nonparametric if, after estimating the parameters, we still need the data to make predictions. Otherwise, if having the parameters means we can throw away the data and still make predictions, it is parametric.

Some examples. If I throw away the data, can I still make predictions at point $\mathbf{x}$?

1. Linear regression: yes, I just calculate $\beta^\top\mathbf{x}$. It must be parametric!
2. Neural network: yes, I just push $\mathbf{x}$ through my network. It must be parametric!
3. K nearest neighbors: no; just knowing $K$ and the distance metric is not enough to make any predictions: I need to know the average at the $K$ closest points to $\mathbf{x}$ under my metric. It must be nonparametric.
4. Gaussian processes: knowing a kernel function's parameters are not enough to make any predictions: I need to know the correlation between all training points and $\mathbf{x}$ as well as the inter-training correlations. Hence it is nonparametric.

The points you raised apply to K Nearest Neighbors Regression too.

Let me start by rephrasing what you wrote in terms of K-nearest neighbors regression:

"""

Given that K nearest neighbor (KNN) regression relies on a distance metric with specific hyperparameters [Number of neighbors $K$, Parameter $p$ and dimension wieghts $w_p$ in $\sum_{d=1}^D (w_d |x_{1,d}-x_{2,d}|^p)^{\frac{1}{p}}$] that control the dissimilarities between points, can KNN regression truly be considered a non-parametric technique? I have the following points.

1. A KNN regressor's behavior is largely determined by the distances parametric structure, which restricts the functional form to be consistent with certain assumptions (e.g., smoothness, periodicity). This makes it a deterministic process governed by hyperparameters, not an inherently flexible, parameter-free model.

2. Like any parametric technique (e.g., DFT or AR models), as the amount of data grows, KNNs allow for more precise estimates of hyperparameters that better fit the data. This improvement with data could be considered similar to reducing sub-optimality in parameter estimation rather than reflecting an open, parameter-free structure.

In summary, should KNN regression not be considered a parametric method due to the distance metric/$K$'s role as a definitive, parameterized structure that guides model behavior? In my opinion, it is indeed a parametric estimator.

"""

[Note: I have excluded Point 3 and other text about the fact that a GP is a probabilistic model. This has nothing to do with parametric vs nonparametric.]

Why GP regression is Nonparametric.

In my view, none of what you have mentioned has to do with parametric vs nonparametric. It might be confusing that "nonparametric" methods still have parameters, but they all do.

So how do we tell the difference? I like to use the following demarcation. A method is nonparametric if, after estimating the parameters, we still need the data to make predictions. Otherwise, if having the parameters means we can throw away the data and still make predictions, it is parametric.

Some examples. If I throw away the data, can I still make predictions at point $\mathbf{x}$?

1. Linear regression: yes, I just calculate $\beta^\top\mathbf{x}$. It must be parametric!
2. Neural network: yes, I just push $\mathbf{x}$ through my network. It must be parametric!
3. K nearest neighbors: no; just knowing $K$ and the distance metric is not enough to make any predictions: I need to know the average at the $K$ closest points to $\mathbf{x}$ under my metric. It must be nonparametric.
4. Gaussian processes: knowing a kernel function's parameters are not enough to make any predictions: I need to know the correlation between all training points and $\mathbf{x}$ as well as the inter-training correlations. Hence it is nonparametric.

The points you raised apply to K Nearest Neighbors Regression too.

Let me start by rephrasing what you wrote in terms of K-nearest neighbors regression:

"""

Given that K nearest neighbor (KNN) regression relies on a distance metric with specific hyperparameters [Number of neighbors $K$, Parameter $p$ and dimension wieghts $w_p$ in $\sum_{d=1}^D w_d (|x_{1,d}-x_{2,d}|^p)^{\frac{1}{p}}$] that control the dissimilarities between points, can KNN regression truly be considered a non-parametric technique? I have the following points.

1. A KNN regressor's behavior is largely determined by the distances parametric structure, which restricts the functional form to be consistent with certain assumptions (e.g., smoothness, periodicity). This makes it a deterministic process governed by hyperparameters, not an inherently flexible, parameter-free model.

2. Like any parametric technique (e.g., DFT or AR models), as the amount of data grows, KNNs allow for more precise estimates of hyperparameters that better fit the data. This improvement with data could be considered similar to reducing sub-optimality in parameter estimation rather than reflecting an open, parameter-free structure.

In summary, should KNN regression not be considered a parametric method due to the distance metric/$K$'s role as a definitive, parameterized structure that guides model behavior? In my opinion, it is indeed a parametric estimator.

"""

[Note: I have excluded Point 3 and other text about the fact that a GP is a probabilistic model. This has nothing to do with parametric vs nonparametric.]

Why GP regression is Nonparametric.

In my view, none of what you have mentioned has to do with parametric vs nonparametric. It might be confusing that "nonparametric" methods still have parameters, but they all do.

So how do we tell the difference? I like to use the following demarcation. A method is nonparametric if, after estimating the parameters, we still need the data to make predictions. Otherwise, if having the parameters means we can throw away the data and still make predictions, it is parametric.

Some examples. If I throw away the data, can I still make predictions at point $\mathbf{x}$?

1. Linear regression: yes, I just calculate $\beta^\top\mathbf{x}$. It must be parametric!
2. Neural network: yes, I just push $\mathbf{x}$ through my network. It must be parametric!
3. K nearest neighbors: no; just knowing $K$ and the distance metric is not enough to make any predictions: I need to know the average at the $K$ closest points to $\mathbf{x}$ under my metric. It must be nonparametric.
4. Gaussian processes: knowing a kernel function's parameters are not enough to make any predictions: I need to know the correlation between all training points and $\mathbf{x}$ as well as the inter-training correlations. Hence it is nonparametric.