The scheme used is actually much more ingenious. Each class is compiled against a particular API version, with symbolic references to other classes. In other words, each class is compiled in its native version, with the runtime guaranteeing interoperability with various versions. This allows us, for example, to depend on a particular flavor of the JSON standard library (to include or exclude null values). The standard system libraries are versioned, and each class is compiled against the native version.
I posted another answer about how you should consistently use the same version across all classes, for your own sanity, but the system makes a very strong effort to protect developers from version skew; a class should always perform according to the documentation for that class and version.
As for managed packages, only one version is in play at a time. The usual restrictions on upgrading packages allows most packages to be forwards-compatible. For example, a class that depends on Foo v1.2 will also run on Foo v1.8, because ISVs are generally prohibited from deleting or overwriting things that a consumer may depend on.
There are some limited ways around this restriction, but that's non-trivial in most cases. What's there today will be there in six months, guaranteed*.
*Guaranteed in the sense that the ISV needs to go out of their way to delete a feature, and also burdens them with updating all of their clients so they have a chance to adapt to the deletion.