As I was trying in vain to fix a faulty ethernet controller here, one thing I tried was running tcpdump on the machine.

I found it interesting that tcpdump was able to detect that some of the ICMP packets the ping application thought it was sending were not actually going out on the wire, even though it was running on the same machine. I have reproduced those tcpdump results here:

14:25:01.162331 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 1, length 64 14:25:02.168630 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 2, length 64 14:25:02.228192 IP 74.125.224.80 > debian.local: ICMP echo reply, id 2334, seq 2, length 64 14:25:07.236359 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 3, length 64 14:25:07.259431 IP 74.125.224.80 > debian.local: ICMP echo reply, id 2334, seq 3, length 64 14:25:31.307707 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 9, length 64 14:25:32.316628 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 10, length 64 14:25:33.324623 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 11, length 64 14:25:33.349896 IP 74.125.224.80 > debian.local: ICMP echo reply, id 2334, seq 11, length 64 14:25:43.368625 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 17, length 64 14:25:43.394590 IP 74.125.224.80 > debian.local: ICMP echo reply, id 2334, seq 17, length 64 14:26:18.518391 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 30, length 64 14:26:18.537866 IP 74.125.224.80 > debian.local: ICMP echo reply, id 2334, seq 30, length 64 14:26:19.519554 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 31, length 64 14:26:20.518588 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 32, length 64 14:26:21.518559 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 33, length 64 14:26:21.538623 IP 74.125.224.80 > debian.local: ICMP echo reply, id 2334, seq 33, length 64 14:26:37.573641 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 35, length 64 14:26:38.580648 IP debian.local > 74.125.224.80: ICMP echo request, id 2334, seq 36, length 64 14:26:38.602195 IP 74.125.224.80 > debian.local: ICMP echo reply, id 2334, seq 36, length 64 

Notice how the seq number jumps several times... that indicates packets the ping application generates that are not actually leaving the box.

Which brings me to my question: how was tcpdump able to detect that the ICMP packets weren't actually going out? Is it able to somehow directly monitor what is on the wire?

If it does accomplish this, I assume it is by interfacing to some part of the kernel, which in turn interfaces to some hardware that is a standard part of a network controller.

Even so, that's pretty cool! If that is not actually how tcpdump functions, can someone explain to me how it detected the missing packets in software?