in this question if I have to find max compression in spring then if I consider both block and spring as system and then apply work energy theorem then as I have written above that spring force is internal and work done by it will cancel out... So according to this change in kinetic energy should come out to be zero but this is not true as block initially had velocity v. Explain where I am wrong
Here is the force diagram:
.
If one defines mass $m$ as the system the force on the mass due to the spring $\vec F_{\rm ms}$ is an external force.
The work done on the system in stopping the mass, with the spring compression being $x$, is $-\frac 12 kx^2$.
Applying the work-energy theorem one gets the "expected" result.
$\displaystyle \int _0^x \vec F_{\rm ms}\cdot d \vec x = \int _0^x -kx\, d x=-\frac 12 kx^2 = 0 -\frac 12 mv^2$.
Now consider the system as the spring and the mass.
$\vec F_{\rm ms}$ and $\vec F_{\rm sm}$ are internal forces equal in magnitude and opposite in direction (N3L).
$\vec F_{\rm sw}$ is the external force on the system but as it does not move it does no work.
Think of the effect of the two internal forces this way:
Force $\vec F_{\rm ms}$ does work and stops the mass moving.
Force $\vec F_{\rm sm}$ does work on the spring and compresses it. It is a conservative force which increases the elastic potential energy of the spring from zero to $\frac 12 kx^2$.
For conservative forces, the external work done on a system is the sum of the change in the kinetic energy of the system plus the change in the potential energy of the system.
In the case, $\underbrace{0}_{\text{external work done}}=\underbrace{-\frac 12mv^2}_{\text{change in kinetic energy}} +\underbrace{\frac 12 kx^2}_{\text{change in potential energy}} $
There is no ambiguity. You are just wrong in applying. There is only one interaction, where energy is being transferred from the kinetic energy of the object into the compression elastic potential energy of the spring.
When you want to use the work done concept, you have to consider which system is doing the work on which other system. Your current fallacy is in double-counting the one single work done as two.
The work energy theorem (WET) is fairly confusing, and often it is taught somewhat haphazardly.
For me, the key is to remember that the derivation of the WET is based on Newton’s 2nd law (N2). For a system, N2 relates the net force $\vec F_{net}$ to the acceleration of the center of mass $\vec a_{com}$. So for clarity the WET formula should be written $$\vec F_{net} \cdot \vec d_{com}=\Delta\left(\frac{1}{2}m \vec v_{com}^2\right)$$
In your problem, the forces between the spring and the block are internal, so they do not contribute to $\vec F_{net}$. However, you have neglected the force between the spring and the wall. This is an external force and, for this system, $\vec F_{net}=\vec F_{wall}\ne 0$.
Now, although the wall does not move, the center of mass does. So $\vec d_{com}\ne 0$. This means that $$\vec F_{net} \cdot \vec d_{com}\ne 0$$ and therefore $$\Delta\left(\frac{1}{2}m \vec v_{com}^2\right)\ne 0$$
The work energy theorem (WET) says the net work done on a system equals the change in the KE of the system.
However, It doesn’t necessarily follow that a change in KE of a system is always the result of net work done on the system. It can also result from conversions between KE and PE within the system. That’s what is happening here since the net external force exerted by the wall doesn’t do work on the system.
There are many other examples where this occurs. Consider an explosion of a stick of dynamite. The KE of its particles is the result of converting chemical potential PE to KE. No external work is involved.
Hope this helps.
The work-energy theorem (WET) is not very useful in this problem, because springs are deformable bodies that introduce very many internal forces whose net work is not zero, and this complicates the analysis. It is much easier to get the result from the law of conservation of energy (which is a different law than WET).
Nevertheless, WET can be applied correctly to the whole system.
Solution based on the work-energy theorem (WET)
The error in your argument is that when the system includes a deformable spring, there are many more forces and works involved, acting on the spring from within. You considered only two works - that due to the block on the spring $\frac{1}{2}k(x-x_0)^2$, and that due to the spring on the block $-\frac{1}{2}k(x-x_0)^2$ (these are the same magnitude but opposite sign). But there are many more internal works done on the spring, due to its internal forces, and sum of these internal works is a non-zero value, due to spring deformation.
In case of an ideal spring with zero mass, sum of these internal works $W^{internal}_{on~spring}$ turns out to be equal to minus the sum of works of the external forces on the spring $W^{external}_{on~spring}$:
$$ W^{internal}_{on~spring} = -W^{external}_{on~spring}. $$
Why? A massless spring that gets deformed but maintains finite velocity also maintains zero kinetic energy. We can apply WET to such a spring, and then sum of all works on it must be zero; any excess of work would manifest as kinetic energy of the spring, and the spring would get launched into very fast motion. When that doesn't happen, kinetic energy of the spring is maintained at zero, and thus the sum of internal works must equal minus the sum of external works.
The work-energy theorem states $$ W^{internal}_{on~spring} + W^{external}_{on~spring} ~~+ ~~W^{internal}_{on~block} + W^{external}_{on~block} = \Delta E_k. $$ Since the spring does not change its kinetic energy, change of kinetic energy of the system is $\Delta E_k = -\frac{1}{2}mv_0^2$.
Using the above results, the first two terms drop out, and we get
$$ W^{internal}_{on~block} + W^{external}_{on~block} = -\frac{1}{2}mv_0^2. $$
Since the block is assumed to not deform, $W^{internal}_{on~block}=0$, and at the moment when the block stops, and the spring is maximally compressed, we obtain
$$ W^{external}_{on~block} = -\frac{1}{2}mv_0^2. $$ We could have obtained this much more quickly by applying the WET directly to the block.
$W^{external}_{on~block}$ is the work due to the spring; when the spring gets compressed to length $x_{max}$, the work can be shown to be
$$ W^{external}_{on~block} = -\frac{1}{2}k(x_{max}-x_0)^2. $$ Combining the last two results, we get
$$ \frac{1}{2}k(x-x_0)^2 = \frac{1}{2}mv_0^2. $$
Now we can express $x_{max}$ : $$ x_{max} = x_0 - \sqrt{\frac{m}{k}}v_0. $$
As you can see, applying WET to the whole system to solve this problem is a bit more difficult than assumed. It is because WET does not refer to potential energy of the spring and conservation of total energy, and we had to find works of certain internal and external forces to get the same result.
We could have gotten the solution much more quickly, if we applied WET to the solid block only right frmo the start. Here, I wanted to show how WET applies to the whole system and we can get to the correct result.
Solution based on conservation of energy
Since there is no friction assumed, we can use the law of conservation of energy: sum of kinetic and potential energy of the system is constant. Using this, we can find the resulting length $x_{max}$ without having to find values of those works.
Potential energy stored in the spring is
$$ \frac{1}{2}k(x-x_0)^2, $$ where $x$ is the length of the spring in the state considered, and $x_0$ is its free length, when it is neither extended nor compressed.
Conservation of energy means $$ \frac{1}{2}k(x-x_0)^2 + \frac{1}{2}mv^2 = E = const. $$
Initially, the spring has length $x_0$, and the block has velocity $v_0$, so $$ E = \frac{1}{2}mv_0^2, $$ that is, $E$ is equal to initial kinetic energy of the block.
Then, when the spring is compressed maximally,
$$ E = \frac{1}{2}k(x_{max}-x_0)^2. $$ From the last two equations, we find the spring length $x_{max}$:
$$ x_{max} = x_0 - \sqrt{\frac{m}{k}}v_0. $$
