LaTeX doesn’t gobble whitespace after optional (or obligatory) arguments – at least not any more than it always does (i.e. multiple consecutive simple spaces/tabs are collapsed to a single space):
\documentclass{article} \newcommand{\foo}[1][]{foo#1} \begin{document} % output: fooab \foo[a]b % output: fooa b \foo[a] b % output: fooa b \foo[a] b % output: fooa b \foo[a]\ \ b \end{document}
Presumably \newtheorem internally defines the resulting command in such a way that it always has a constant amount of whitespace after it, but that’s not part of the default behaviour of TeX commands.
The gobbling of whitespace after option-less commands is simply due to a problem of delineation: Without a space as in \aab LaTeX would have no way of knowing that the command name ends after the second a so there needs to be some marker to signal the end of the command. That can be whitespace but you can also use {}: \aa{}b (and using both, \aa{} b, gives you whitespace in the output because the {} already signals the end of the command to LaTeX). But if you have any kind of arguments, optional or obligatory, the end of the command name is unambiguous so there is no need to gobble whitespace.
(EDIT: Actually \aa{}b is not exactly equivalent to \aa b because the former adds a zero-width token after \aa. Most of the time the result is the same but the token can make a difference at times, for example for ligatures (fi looks different from f{}i), for kerning, or for certain commands which work on the following token such as \' or \^. Thanks to David Carlisle for this insight!)
Also as a side note, as far as I know the recommended way of using theorems is with environments:
\begin{defn}[Gauss] Y \end{defn}
But I guess you just used \defn for demonstration purposes?
\documentclass{article}\begin{document}Hello World\end{document}for example.