TeX maintains styles when processing math formulas; the styles are
- display, D
- text, T
- script, S
- scriptscript, SS
together with their cramped counterparts (denoted with a prime). You can read the TeXbook or TeX by Topic for the precise rules when the cramped styles are used, here, for the sake of simplicity, I'll not distinguish between cramped and non cramped. With C I'll denote the current style, and with C↕ the style for subscripts or superscripts relative to the current style
D↕ = S, T↕ = S, S↕ = SS, SS↕ = SS
When a formula is started in line, the initial style is T; if it is a display, the initial style is D.
Clearly, a subscript or superscript is typeset in style C↕; thus $1^2$ will have 1 in style T and 2 in style S. Similarly, $1^{2^3}$ will have the same choices for 1 and 2, but 3 will be in style SS. Further exponents will continue in style SS. Similarly for subscripts.
The difference with $$1^2$$, $$1^{2^3}$$ is just that 1 will be in style D. Of course, this will make no difference in case of 1, but will in case of \sum, because all “big operators” have a larger variant which is used in style D.
There is no distinction when the subscript/superscript is eventually set above or below an operator such as \lim: the same rules will apply.
What about fractions?
- C=D: numerator T, denominator T
- C=T: numerator S, denominator S
- C=S: numerator SS, denominator SS
- C=SS: numerator SS, denominator SS
This also holds for “generalized fractions” such as \above or \atop.
For radicals, the object under the radical sign will be set in the same style as the current one.
The current style can be overridden with the declarations \displaystyle, \textstyle, \scriptstyle and \scriptscriptstyle, whose scope holds for the current subformula (possibly delimited by { and }, not by \begingroup and \endgroup).
These declarations are used in \mathpalette (see The mysteries of \mathpalette), because in general it is not known in advance what the current style will be.
For each math family, TeX maintains a \textfont (used in styles D and T), a \scriptfont (used in style S) and a \scriptscriptfont (used in style SS). For instance, plain TeX has
\textfont0=\tenrm \scriptfont0=\sevenrm \scriptscriptfont0=\fiverm
It's the user's responsibility to assign the fonts. LaTeX does it automatically with \DeclareSymbolFont or \DeclareMathAlphabet.