The operator - has greater depth than the surrounding numerals. As such, the second smallmatrix is "taller" than the first, causing the delimiters to stretch further. The following highlights this somewhat:

\documentclass{article} \usepackage{mathtools}% http://ctan.org/pkg/mathtools \setlength{\fboxsep}{-\fboxrule} \newcommand{\boxit}[1]{\text{\fbox{$#1$}}} \begin{document} \[ \left(\begin{smallmatrix*}[r] \boxit{1} & \boxit{2} \\ \boxit{5} & \boxit{7} \end{smallmatrix*}\right) \quad {\def\boxit#1{#1} \left(\begin{smallmatrix*}[r] \boxit{1} & \boxit{2} \\ \boxit{5} & \boxit{7} \end{smallmatrix*}\right)} \qquad \left(\begin{smallmatrix*}[r] \boxit{1} & \boxit{-1} \\ \boxit{2} & \boxit{3} \end{smallmatrix*}\right) \quad {\def\boxit#1{#1} \left(\begin{smallmatrix*}[r] \boxit{1} & \boxit{-1} \\ \boxit{2} & \boxit{3} \end{smallmatrix*}\right)} \] \end{document}
\boxit puts a bounding box around the element. Note how the elements in the first row of the second smallmatrix are not vertically aligned.
The marginal increase in height is enough to increase the delimiters. This behaviour can be adjusted via setting elements like \delimitershortfall and/or \delimiterfactor. In the following example, I've set \delimitershortfall to 0pt:

\documentclass{article} \usepackage{mathtools}% http://ctan.org/pkg/mathtools \setlength{\fboxsep}{-\fboxrule} \newcommand{\boxit}[1]{\text{\fbox{$#1$}}} \begin{document} \setlength{\delimitershortfall}{0pt} \[ \left(\begin{smallmatrix*}[r] \boxit{1} & \boxit{2} \\ \boxit{5} & \boxit{7} \end{smallmatrix*}\right) \quad {\def\boxit#1{#1} \left(\begin{smallmatrix*}[r] \boxit{1} & \boxit{2} \\ \boxit{5} & \boxit{7} \end{smallmatrix*}\right)} \qquad \left(\begin{smallmatrix*}[r] \boxit{1} & \boxit{-1} \\ \boxit{2} & \boxit{3} \end{smallmatrix*}\right) \quad {\def\boxit#1{#1} \left(\begin{smallmatrix*}[r] \boxit{1} & \boxit{-1} \\ \boxit{2} & \boxit{3} \end{smallmatrix*}\right)} \] \end{document}
Read some more about these settings in Automatic size adjustment for nested parentheses.
\vphantomseems to give the desired size:\item $A = \left( \begin{smallmatrix*}[r] 1 & \vphantom{-}2 \\ 5 & 7 \end{smallmatrix*} \right)$mathtoolsbut (as someone else has said) the fact that the minus has extra depth. this is built into the font (at least for computer modern), the reasoning being that plus and minus should have the same dimensions and spacing.