I would like to put an underbrace below the matrix like this:
$$\left(\underbrace{ \left[\begin{array}{ccc} a&b&c\\ d&e&f\\ g&h&i \end{array}\right]}_{A} \left[\begin{array}{c} x_1\\ x_2\\ x_3 \end{array}\right]\right)$$ 
But I don't want the outer brackets \left( and \right) to stretch vertically. I want them to remain as if there was no underbrace. How to achieve that behaviour?
You could \smash the underbrace. Of course, you may have to add some \vspace after the equation to compensate for the underbrace that is perceived to be of zero height.
Note that, with this usage, the whole matrix is actually smashed. But because there follows a vector of identical height, the outer parens are properly sized.
\documentclass{article} \begin{document} $$\left(\smash{\underbrace{ \left[\begin{array}{ccc} a&b&c\\ d&e&f\\ g&h&i \end{array}\right]}_{A}} \left[\begin{array}{c} x_1\\ x_2\\ x_3 \end{array}\right]\right)$$ \end{document} 
\newcommand{\smashedunderbrace}[2]{\smash{\underbrace{#1}_{#2}}\vphantom{#1}} in order to size the outer brackets correctly, even if the vector has a smaller size or is not there at all. Works well! Usage: \smashedunderbrace{what should be underbraced}{underbrace text} \vspace, since the underbrace was \smashed. Most of the time \left and \right make delimiters that are too large for my tastes so I almost always use the amsmath versions of \big, \Big, \bigg and \Bigg.
In your example
$$\Bigg(\underbrace{ \left[\begin{array}{ccc} a&b&c\\ d&e&f\\ g&h&i \end{array}\right]}_{A} \left[\begin{array}{c} x_1\\ x_2\\ x_3 \end{array}\right]\Bigg)$$ looks OK. You'll need \usepackage{amsmath} to get the best effects. See About big parenthesis larger than Bigg if you want to make your own custom sizes. For example, I sometimes use:
\newcommand\Bigger[2][7]{\left#2\rule{0mm}{#1truemm}\right.} By default this creates a 7mm high delimiter but \Bigger[10]( makes a 10mm bracket etc.
Take the \left( inside the underbrace. Then you need \right. and \left. to close and open the brace.
Code:
$$\underbrace{\left( \left[\begin{array}{ccc} a&b&c\\ d&e&f\\ g&h&i \end{array}\right]\right.}_{A} \left.\left[\begin{array}{c} x_1\\ x_2\\ x_3 \end{array}\right]\right)$$ Sometimes it's simpler just to create a local definition.
\documentclass{article} \begin{document} Let \[ A = \left[\begin{array}{ccc} a&b&c\\ d&e&f\\ g&h&i \end{array}\right] \] in \[ A\left[\begin{array}{c} x_1\\ x_2\\ x_3 \end{array}\right] \] \end{document} With {NiceArray} of nicematrix.
\documentclass{article} \usepackage{nicematrix} \begin{document} $\begin{NiceArray}{([ccc][c])} a & b & c & x_1 \\ d & e & f & x_2 \\ g & h & i & x_3 \CodeAfter \UnderBrace[yshift=1mm]{3-1}{3-3}{A} \end{NiceArray}$ \vspace{1cm} $\begin{NiceArray}{([ccc][c])} a & b & c & x_1 \\ d & e & f & x_2 \\ g & h & i & x_3 \CodeAfter \UnderBrace[shorten,yshift=1mm]{3-1}{3-3}{A} \end{NiceArray}$ \end{document} 
$$...$$syntax: please, have a look to Why is [ … ] preferable to $$ … $$? for reference.equationenvironment anyway, I just put$$...$$for this example. However I didn't know that, thanks!