There are two methods: period and frequency measurement. You have chosen the frequency method. My old books say:
$$f_x = \dfrac{N}{T_M} \pm \dfrac{1}{T_M}$$
Where \$T_M\$ is a measuremnt period (0.1s in your case) and \$N\$ is the number of counts.
$$f_x = \dfrac{RPM_x}{60s}\cdot 4000\ inc$$
$$RPM = \dfrac{60}{4000}\lbrace \dfrac{N}{0.1} \pm \dfrac{1}{0.1} \rbrace$$
$$RPM = 0.15\cdot N \pm 0.15$$
check:
26640*0.15 = 3966, actually it should be 26666 counts to mach my formula.
EDIT:
At 4000 RPM the frequency is 4000*4000/60= 266,666 kHz, so within 1/10s you should count 26666 pulses. Multiply with 0.15 and you get 3999.9 RPM. Next time you will count 26667 pulses that gives 4000.05 RPM, the difference is exactly 0.15 RPM which is the measurement error.
EDIT2:
This is a processor I have developed on the FPGA and the ALU can only divide by powers of 2 so yeah I am not able to divide by 20
That would be easily solved if you would be using an encoder with PPR with power of 2, that's why they do exist. For example 1024 PPR, then you could change the measurement interval so you could get 512/4096. Still you can change the measurement interval to 0.12s then you get:
$$RPM = \dfrac{60}{4000}\lbrace \dfrac{N}{0.12} \pm \dfrac{1}{0.12} \rbrace$$
$$RPM = 0.125\cdot N \pm 0.125$$
Et voila: $$RPM = \dfrac{N}{8} \pm 0.125$$
Set to 120ms and shift right 3 times.