I am going through Expectation Maximization (EM) algorithm derivation for Gaussian Mixture models. I understand it except for a small detail.
So, the general idea of EM is to maximize the expectation of complete log-likelyhood of the data $\log L(\theta;\mathbf{x},\mathbf{Z})$, where $\mathbf{x}$ is observed and $\mathbf{Z}$ is latent variables, $\theta$ are the model parameters. The expectation is taken over the posterior distribution of the latent variables: $\operatorname{E}_{\mathbf{Z}\mid\mathbf{X};\mathbf{\theta}^{(t)}} [\log L(\theta;\mathbf{x},\mathbf{Z})]$, where ${\theta}^{(t)}$ represents the parameters of the previous algorithm iteration.
Then, the derivation follows (copy-paste from Wikipedia): : \begin{align}Q(\theta\mid\theta^{(t)}) &= \operatorname{E}_{\mathbf{Z}\mid\mathbf{X}=\mathbf{x};\mathbf{\theta}^{(t)}} [\log L(\theta;\mathbf{x},\mathbf{Z}) ] \\ &= \operatorname{E}_{\mathbf{Z}\mid\mathbf{X}=\mathbf{x};\mathbf{\theta}^{(t)}} [\log \prod_{i=1}^{n}L(\theta;\mathbf{x}_i,Z_i) ] \\ &= \operatorname{E}_{\mathbf{Z}\mid\mathbf{X}=\mathbf{x};\mathbf{\theta}^{(t)}} [\sum_{i=1}^n \log L(\theta;\mathbf{x}_i,Z_i) ] \\ &= \sum_{i=1}^n\operatorname{E}_{Z_i\mid X_i=x_i;\mathbf{\theta}^{(t)}} [\log L(\theta;\mathbf{x}_i,Z_i) ] \\ &= \sum_{i=1}^n \sum_{j=1}^2 P(Z_i =j \mid X_i = \mathbf{x}_i; \theta^{(t)}) \log L(\theta_j;\mathbf{x}_i,j) \\ &= \sum_{i=1}^n \sum_{j=1}^2 T_{j,i}^{(t)} \big[ \log \tau_j -\tfrac{1}{2} \log |\Sigma_j| -\tfrac{1}{2}(\mathbf{x}_i-\boldsymbol{\mu}_j)^\top\Sigma_j^{-1} (\mathbf{x}_i-\boldsymbol{\mu}_j) -\tfrac{d}{2} \log(2\pi) \big]. \end{align}
I don't understand how we get the forth line from the third. Namely, why does the expectation value over joint distribution for all observed samples $\operatorname{E}_{\mathbf{Z}\mid\mathbf{X};\mathbf{\theta}^{(t)}}$ nicely disintegrate into expectations over individual samples $\operatorname{E}_{Z_i\mid X_i = x_i;{\theta}^{(t)}}$
