Arduino is poor in terms of calculation, and it is not made for motor control. If you need a precise velocity, then you need a MCU or DSP with QEI - Quadrature Encoder Interface. It measures frequency and period of pulse train, then uses the information of the method which is more plausible.
For motion control you would need a fast MCU, not a legacy 8-bit MCU. An integrated FPU is welcome, but not mandatory if it has enough speed.
You better find some other platform.
P.S.:
You probably use the interrupt to calculate the speed from difference of counted pulses, so called frequency measuring method.
There is also a period measurement method. The error of frequency method is small at high frequencies and large at low frequencies, meanwhile the error of period measurement method is small at low frequency and large at high frequency.
Your control, is jerking at low speed because the speed measurement has a large error at encoder low frequency output (low motor speed).