Questions tagged [universal-approximation-theorems]
For questions related to the (different) universal approximation theorems (UATs), for example, in the context of neural networks.
34 questions
2 votes
2 answers
184 views
How would you go from 1 to k hidden layers in Cybenko's result that neural networks are universal approximators?
Cybenko showed that if $\sigma$ is a sigmoidal, continuous function, then for any $\varepsilon > 0$, for any continuous function $f: [0, 1]^d \to \mathbb{R}$, there exists a function of the form $g:...
- 123
1 vote
1 answer
136 views
Why are neural networks optimized instead of just optimizing a high dimensional function?
I know that neural networks are universal approximators when given a sufficient number of neurons, but there are other things that can be universal approximators, such as a Taylor series with a high ...
- 11
2 votes
1 answer
105 views
100 layer neural network with 100 hidden units vs. 1 layer neural network with 100 hidden units
Suppose we have a neural network with 100 hidden layers. Each hidden layer has one hidden node, and the hidden nodes employ a universal basis function (e.g. tanh). Now we want to compare this network'...
1 vote
1 answer
147 views
Why doesn't the Kolmogorov-Arnold representation theorem imply an MLP-like structure?
Recently, Kolmogorov-Arnold Networks (KANs) generated a lot of hype, with "AI experts" throwing around terms like "ML 2.0" and "a new era of ML". KANs are supposedly ...
- 11
2 votes
1 answer
206 views
Neural Networks are universal approximators? - Exercice 20.1 UML
I'm working on this question which can be found at page 282 of "Understanding Machine Learning: From Theory to Algorithms" by Shai Shalev-Shwartz and Shai Ben-David. The statement is as ...
- 21
1 vote
1 answer
93 views
Why do we need the identify function when approximating a function with a neural network with multiple layers?
I have a question about the explanation of universal approximation theorem provided by wikipedia. https://en.wikipedia.org/wiki/Universal_approximation_theorem#cite_note-:0-29 It says, after a ...
- 21
5 votes
2 answers
335 views
What makes the approximation capabilities of neural networks different than something like, say, Fourier series?
People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. However, this seems unremarkable ...
- 153
2 votes
1 answer
270 views
Is there a mathematical proof of the universal approximation theorem for neural networks with binary weights?
Since the Universal approximation theorem shows that standard multilayer feedforward networks with as few as a single hidden layer, sufficient hidden units, and arbitrary bounded and nonconstant ...
- 43
2 votes
1 answer
803 views
Why do activation functions in neural networks have to be non-polynomial to approximate any function?
Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.
- 31
1 vote
0 answers
108 views
Is the capability of RNN more than the capability of MLP?
Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named ...
- 4,172
1 vote
0 answers
121 views
Does Godel's incompleteness theorems restricts the scope of connectionist-AI?
It is well-known that Godel's incompleteness theorems restricted the reachability of symbolic-AI, which is dependent on mathematical logic. But, I am wondering whether it has any impact on the ...
- 4,172
2 votes
2 answers
427 views
Are the capabilities of connectionist AI and symbolic AI the same?
The universal approximation theorem says that MLP with a single hidden layer and enough number of neurons can able to approximate any bounded continuous function. You can validate it from the ...
- 4,172
5 votes
2 answers
743 views
Why does the activation function for a hidden layer in a MLP have to be non-polynomial?
Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial. Is there a ...
- 53
2 votes
0 answers
208 views
Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?
The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
- 23
1 vote
1 answer
124 views
Is there any paper that shows that multi-channel neural networks are universal approximators?
Lately, I have been reading a lot about the universal approximation theorem. I was surprised to find only theorems about "single-channel" standard networks (multi-layer perceptrons), where ...
- 13