What does it mean for work to be done ON a system?
For a given system, e.g. an apple, an external force can be applied to the system, e.g., the force of gravity. We say that gravitational force is a force "on" the apple "by" the earth. We could make this explicit with ordered subscripts like $\vec F_{a,e}$, indicating "on" the apple ("a"), "by" the earth ("e"). When the apple moves a distance $d\vec x$ the force of gravity performs work "on" the apple $$ W_{a,e} = \int d\vec x \cdot \vec F_{a,e} $$
Suppose we have a system consisting of an ideal gas enclosed by a cylinder and a movable piston.
The system of interest in pedagogical questions of this form is usually the gas, not the cylinder. The cylinder is just there to contain the gas and the piston is there to allow the volume of the gas to change.
We say that if we were to compress the gas by letting the piston extert a net force on it (perhaps by increasing the pressure outside the piston), we are doing pressure-volume work ON the system.
Yes, if we apply a force $\vec F$ on the piston, which we assume thereby applies a force to compress the gas by an amount $dV = dx A$, where $A$ is the area of the piston, the net force on the gas is in the direction which causes compression and the work done "on" the gas is $$ dW_{on} = |dx||F| = |dx||A P| = |dV||P|= -PdV\;, $$ where I put absolute value symbols based on the fact that the cylinder and piston cause the force $\vec F$ to be in the exact same direction as the compression displacement $\delta\vec x$ so that $\vec F\cdot d\vec x = |F||dx|$;.
Suppose instead, we were to scrape the surface of the cylinder enclosing the gas with our finger against friction.
You are now changing the thing you are considering the "system." Previously the gas was the system of interest and the cylinder was a convenient way enclose the gas and to apply a force of a given direction on the gas via the piston.
We would still be doing work, and we could calculate its magnitude and all properties we might be interested in, but we have not done work ON the system.
You might still be doing work on some system, but it is not the same system that you initially considered. The usual setup of a pedagogical cylinder full of gas does not specify how frictional heating of the cylinder can cause a change in energy of the gas. Certainly, if the only way for the gas volume to change is via the piston then the scraping of the cylinder has no clear way to do work.
If you want to further specify that the cylinder is not insulating and that the scraping and friction causes heat $d Q$ to flow into the gas, then the energy of the gas can change by $$ dU_{no work} = dQ_{into} $$
If you allow for both heat transfer into the gas and work to be done on the gas then: $$ dU = dQ_{into} + dW_{on} = dQ_{into} - PdV = TdS - PdV\;. $$ where the last equation holds when the heat transfer $dQ$ can be written in terms of the entropy change $dS$ as $dQ=TdS$, and that last equation is called the "fundamental thermodynamics relation."
What is the principal difference between these types of work? How do we know one constitutes work done on the system while one doesn't?
We know based on the problem setup and the fact that we consistently use the word "system" to refer to the same thing in a given problem.