Assume ideal logic gates (no propagation delay) like this (image from wikipedia):

We know that the output of NOR gate is 1 if and only if both inputs are 0; and 0 otherwise.
When S = 1, Q = 1 and therefore \$\bar{Q} = 0\$; when R = 1, Q = 0 and \$\bar{Q} = 1\$.
But if you set both R and S to 1 we have that Q = 0 and \$\bar{Q} = 0\$ at the same time. This contradicts the relation \$Q = not Q\$\$Q = \bar{Q}\$. In the real world one of the gates will reach the 1 state first and the result will be unpredictable.
For the NAND-based RS flip-flop the same can be shown when R = S = 0, by writing the logic equations appropriately.