In an SMPS on the DC power supply side, Common Mode (CM) noise occurs when current exits the circuit and returns through the ground. Typically a DC supply is not grounded in two places (although it could be) as shown in the circuit above, this creates a ground loop and causes problems.

Typically noise exits a device through capacitance and then returns through the ground, which is entirely possible with an SMPS because it also produces AC noise that can readily radiate to nearby conductors from the circuit (Vn can also be located in the supply as not shown below, but has the same result):

Source: https://micro.rohm.com/en/techweb/knowledge/emc/s-emc/01-s-emc/6899

As shown in the diagram above, if it truly is the way you have set things up with a ground loop, there would be a few ways to measure the CM noise. One would be to put a scope ground below A (on A's ground), and the other on D. The differential could be measured from point A to D.

Or if you had a good current probe you could use it to probe the CM anywhere around the other loop (if there are not to grounds, you could probe through A's ground).

One thing to keep in mind is Vcm will change with the load, because the switching period and current changes, the noise will also change. So if you are testing for noise, you may want to test over a range of load conditions.

Maybe Regulatory?

If this is for regulatory testing, then typically AC conducted emissions are tested with an SMPS (typically EN55011, non-paywalled description here). The test setup usually involves an LISN and then measuring currents coming to and from the LISN.

Source: https://www.researchgate.net/publication/224130690_Systematic_Electromagnetic_Interference_Filter_Design_Based_on_Information_From_In-Circuit_Impedance_Measurements

Edit:

If you want to measure the common mode and differential mode without a load, then this is how it could be done with a two channel scope:

The scope ground needs to be connected to the chassis ground of the supply, this is where the noise currents return to the supply. To measure common mode noise, the supply needs a chassis ground.

The differential mode is:

\$V_{DM}=Ch1-Ch2\$

The common mode is:

\$V_{CM}=\frac{Ch1+Ch2}{2}\$

If you didn't have two probes then you could measure and subtract on paper. If you had two probes then you could see both simultaneously which would be better.

Better than that: many digital scopes have averaging and subtraction functions built in. If so then look at the special functions of your scope. You could average Ch1 and Ch2 together and get the Common Mode voltage. Subtracting both would give you the Differential Mode voltage.

My simulation works just fine, the only caveat is the pk-pk voltages don't average and you get 3V instead of 2.5V because one of the AC signals is riding on top of the other one for the Vcm measurement.

Vcm = 1 Vdm = 4

Vcm measured = 3 (should be 2.5) Vdm measured = 4

When you measure by ordinary 10:1 probe method you get Vdm and Vcm and stray noise pickup on coax and ground clip from LC resonance from inputs < 20ns rise time. Thus resulting in poor signal capture test method.

1st try to measure the Vdm (Diff mode) noise accurately, then when you capture Vdm +Vcm with a single 10:1 to your reference earth ground you can see the Vcm noise.

Suggested methods for Vdm:

-1 Using two passive 10:1 probes balanced to flat line in A-B (with 20MHz BW on)
-2 Using 200 MHz Fet Buffered Diff Probes - expensive but accurate
-3 Using above but with tip & clip replaced with 5mm spring probes ( easiest solution)
-4 Add small ceramic cap and 50 ( or 49.9) R to test point or direct 1:1 coax connection to DSO using internal 50R term (opt.) or external BNC-T and 50 Ohm BNC terminator. (adapt source with suitable connections:BNC,SMA,solder wire pins (logic diagram shown below)

^{simulate this circuit – Schematic created using CircuitLab}

Then test using active or dummy loads for load test 10~90%, 10~50%, 50~0% etc for Vdm dynamic load error. (use DSO 20 MHz filter to analyze if fuzzy)

To investigate Vcm thru 2 wire AC-DC converter leakage and Vcm introduced say by 3 prong charger and transformer C pf Leakage through laptop USB cable now this is external Vcm noise. This can also be 3 pronged EMI Line filter leakage current to Earth Gnd.

To investigate Vcm noise thru 2 wire AC-DC converter, you may be observing Vcm noise from any other source connected to load such as leakage from a Laptop USB cable with AC-charger on or a PC Tower with a USB cable to DUT with PSU Vcm issues. Vcm introduced say by 3 prong charger will have a CLC PI EMI line filter leakage to Earth Gnd.

Use Twisted pairs for all noise transient egressors (inductive) and high impedance ingress points, and coil cables to see if there is cable ingress and orient at 90 deg.

Additional Vcm reduction can be done with CLC baluns selected for the spectrum of noise observed. Since these span <3 decades of f, choosing the right Baluns makes a difference on DC cables and high impedance signals to ADC or DSO. Having 120 dB CMRR in an INA is wasted if your interface wires are balanced to 1% with 40 dB CMRR. Ref samples: https://www.schaffner.com/products/emcemi/

Note that when we use Earth grounds, they are only considered 0V at line frequency and are inductive for RF, so depending on layout , never assume all earth grounds are the same in your RF spectrum.

Based on your schematic, if you want to measure \$V_{cm}\$, there is no reason to connect anything except for a load to terminal B of the supply at all, just measure terminal A relative to the ground terminal of the supply.

Even single-ended passive probes usually have both a signal and a ground connection. You would connect the probe ground with as short a wire as possible to the supply ground.

Measurement of \$V_{DM}\$:

^{simulate this circuit – Schematic created using CircuitLab}

Measurement of \$V_{CM}\$:

^{simulate this circuit}