How can continue the equation of the rectangle red in next line?
\documentclass[11pt,a4paper,twocolumn]{article} ... \begin{equation*} \left| \dot{r} \right| = \sqrt{{- \, {e}^{-t} \left( \cos t + \sin t \right)}^{2} \, + \, {{e}^{-t} \left( \cos t + \sin t \right)}^{2} \, + \, {(- \, {e}^{-t})}^{2}} \end{equation*} 
I suggest using the multlined environment from mathtools to break the radicand . You also can choose not to break it, but to type in medsize (~80 % of \displaystyle) with the \mediummath command from nccmath. I took the opportunity to clean your code:
\documentclass[11pt,a4paper,twocolumn]{article} \usepackage[utf8]{inputenc} \usepackage{mathtools, nccmath} \begin{document} \begin{equation*} \bigl| \dot{r} \bigr| = \sqrt{\begin{multlined}[b] -\!\bigl(e^{-t} ( \cos t + \sin t )\bigr)^2 + \\ \bigl(e^{-t} (\cos t + \sin t)\bigr)^2 + \bigl(-e^{-t}\bigr)^2 \end{multlined}} \end{equation*} \begin{align*} \bigl| \dot{r} \bigr| & = \sqrt{\medmath{ -\!\bigl(e^{-t} ( \cos t + \sin t )\bigr)^2 + \bigl(e^{-t} (\cos t + \sin t)\bigr)^2 + \bigl(-e^{-t}\bigr)^2}} \\ & = \sqrt{ -\!\bigl(e^{-t} ( \cos t + \sin t )\bigr)^2 + \bigl(e^{-t} (\cos t + \sin t)\bigr)^2 + \bigl(-e^{-t}\bigr)^2} \end{align*} \end{document} 
e^{-t}(\cos t + \sin t) should be squared? The OP's code seems to suggest that only (\cos t + \sin t) should be squared. (Of course, the OP's code might be in error...) { … }. Anyway, it's only a demo. As the equation* environment doesn't permit line breaks, consider using a multline* environment. And, instead of creating a multi-line surd expression with a \sqrt instruction, I suggest you use (...)^{1/2} notation.
Two additional comments about your code: Do try to use \left and \right less frequently, and don't needlessly encase various terms in curly braces.
\documentclass[11pt,a4paper,twocolumn]{article} \usepackage{amsmath} % for 'multline*' env. \begin{document} ... \hrule % just to illustrate column width \begin{multline*} | \dot{r} | = \bigl\{ - e^{-t} ( \cos t + \sin t )^{2} \\ + e^{-t} ( \cos t + \sin t )^{2} + (-e^{-t})^{2} \bigr\}^{1/2} \end{multline*} \end{document} If you are interested in a square root expression and not a power notation that Mico's answer provides, have a look at the following code:
\begin{equation*} \left| \dot{r} \right| = \sqrt{ \begin{aligned} &{- \, {e}^{-t} \left( \cos t + \sin t \right)}^{2} \, + \, \\ &{{e}^{-t} \left( \cos t + \sin t \right)}^{2} \, + \, {(- \, {e}^{-t})}^{2} \end{aligned} } \end{equation*} 
\quad at the beginning of the second line. Edit: Also note that the mathtools package provides a multlined environment (maybe you already knew… :-) I faced this situation today and the simplest solution is to use {align*} within the equation environment and double back slash \\ command at the location you want to start the next line. Although not automatic, it looks neat.
\begin{equation} \begin{align*} \Pi( \phi,\eta)= \int_0^L \left[ N \cdot \Lambda^T \left( \frac{d \eta_0}{ds} - \theta \times \frac{d \phi_0}{ds} \right)+M \cdot \Lambda^T \frac{d \theta}{ds} \right]ds \\ -\int_0^L \left( \bar{n} \cdot \eta_0 + \bar{m} \cdot \theta \right) ds = 0 \end{align*} \end{equation} 
alignedenvironment and, the symbol&and\\. Have a look herealignat*frommathtools(oramsmath) package?breqn?