David's answer is correct and the one you need to solve your specific problem. I thought nonetheless that it is worth providing some additional information that might help explain how to diagnose similar issues.
Matrix/tensor operations like Dot and Inverse are designed to work with lists, that is, expressions with a Head of List. It also works with SparseArray objects. From the documentation:
When its arguments are not lists or sparse arrays, Dot remains unevaluated.
You can check whether your expressions have compatible Heads using FullForm. It is common for people to use postfix (//) notation to check this. As you can see, in your version of the code, a has head List while b has head MatrixForm. So they can't combine.
a//FullForm
List[List[1,0,1,0],List[2,1,1,1],List[1,2,1,0],List[0,1,1,1]]
b//FullForm
MatrixForm[List[List[0],List[0],List[0],List[1]]]
If you discover you have erroneously created a matrix wrapped in MatrixForm, you can change it back to a list using First.
FullForm[First[b]]
List[List[0],List[0],List[0],List[1]]
As an aside, I don't see any point assigning the variable inv to represent the Inverse of a. Unless your real problem uses $a'$ more than once (especially if it is expensive to calculate), you can just as easily do:
Inverse[a].(First@b)
Note that I have mixed using of @ and [] style notation for pedagogical reasons.
MoreInformationsection ofMatrixFormin the docs:MatrixFormacts as a "wrapper", which affects printing, but not evaluation. $\endgroup$MatrixExp. $\endgroup$TraditionalForm. You can do this easily from the preferences menu, Evaluation tab. $\endgroup$