In computer science, formal language is just a set of strings, usually infinite and often described using rules (two common versions of those rules are regular expressions and formal grammars).
Note that this means that all a language needs is syntax, language doesn't need to describe what each valid string means (that's called semantics).
Now, this means that programming languages are formal languages that also have semantics, which describes some computation. And for example XHTML is a formal language, whose semantics describe (roughly and informally) how a hypertext document looks and behaves.
XML is still a language, even though it doesn't have semantics itself (but many languages derived from XML do, like XHTML and XAML).
Technically, binary formats are also languages, but they're not called that way. The term "language" is reserved for human-readable formats.