I've inferred from the context in which it's used that "site" refers to a multi-user environment, as opposed to a particular user's section of that environment. But it's hard to find a formal explanation or origin for such a generic term.
Is it a holdover from an earlier time, unique to Emacs, or a widespread term that I was simply unaware of before diving into Emacs?
It is not specific to Emacs. And yes, it came from a time when a multi-user site (organization, school, company,...) installed Emacs for multiple-user use.
And yes, this is still the case sometimes. A company or other organization can well provide one or more company installations of Emacs, and provide company-specific customizations in site-lisp.
Furthermore, even if, say, a company has a giant farm of virtual GNU/Linux machines, and provides each of its developers with one such machine, it may well want to pre-install Emacs on each such machine or in a shared location, and it might provide company-specific code in site-lisp (also on each machine or in a shared location).
site-lisp. Here, your computer is a site.