A fundamental aspect of Mathematica is that everything is an expression and evaluating an expression is just applying rules to transform the expression into another expression. You really can't rely on any sort of intuitive semantics to short circuit this whole evaluation process.
When you evaluate
Subsets[Dot[a,b]] (* {Dot[], a, b, Dot[a, b]} *)
The result isn't due to some semantics related to Dot, but instead is due to simple rewrite rules. Since a and b don't have any values attached to them, the result of Dot[a,b] is just Dot[a,b]. Subsets can work with heads other than List, so
Subsets[Dot[a, b]]
gives us initially
{Dot[], Dot[a], Dot[b], Dot[a, b]}
Now, Dot applied to an "atom" just returns that "atom". Dot applied to an empty argument list has no rewrite rules, so it just returns unevaluated. Thus the final result is
{Dot[], a, b, Dot[a, b]}
And of course, the {1,2} argument you used filters that result.
It might help to consider this:
Subsets[f[a, b, c]] (* {f[], f[a], f[b], f[c], f[a, b], f[a, c], f[b, c], f[a, b, c]} *)
Subsets doesn't know the difference between Dot and f. When Subsets is done, the evaluation engine continues to evaluate that result. Any Dot-related expressions get rewritten based on Dot-related rules, turning things like Dot[a] into a.
For your next example, the Dot evaluated to an actual matrix:
Dot[Xpauli, Zpauli] (* {{0, -1}, {1, 0}} *)
That matrix is now what Subsets goes to work on. Subsets has no idea that Dot was involved. The Dot leaves no semantic traces behind. So Subsets just does its thing with this expression. We expect it to generate lists made from {0,-1} and {1,0}, and that's exactly what the result is.