The RTL-SDR interfaces to your computer via USB. The USB interface and driver will buffer the data and send it to your software at the requested sample rate on average (baring underflow due to software glitches), but with varying timing due to interface jitter. The RTL2832U USB is likely sending larger chunks of data over USB in blocks of far more than 512 samples at a time, and the driver is chopping up these large chunks at slightly random times. So the per 512 sample time stamps are irrelevant except to indicate if you are losing data long term. Use the sample count divided by the sample rate instead.
Even the time stamps will have OS latency jitter.