how to enable pager for the colored output of git diff

Calling less with -r or -R it will iterpret color codes, so creating an alias for less is a possible solution:

alias less="less -r" 

The alternative is explained with Git 2.33 (Q3 2021), where the documentation for the "color.pager" configuration variable has been updated.

See commit a84216c (19 May 2021) by Jeff King (peff).
^{(Merged by Junio C Hamano -- gitster -- in commit 7ce7a61, 10 Jun 2021)}

doc: explain the use of color.pager

^{Signed-off-by: Jeff King}

The current documentation for color.pager is technically correct, but slightly misleading and doesn't really clarify the purpose of the variable.
As explained in the original thread which added it:

the point is to deal with pagers that don't understand colors.

And hence it being set to "true" is necessary for colorizing output to the pager, but not sufficient by itself (you must also have enabled one of the other color options, though note that these are set to "auto" by default these days).

git config now includes in its man page:

A boolean to specify whether auto color modes should colorize output going to the pager. Defaults to true;

Set this to false if your pager does not understand ANSI color codes.

From the thread:

I do not really understand why you would enable color at all if you want to disable it when paging.
Do you have many instances when you want a color diff which is short enough not to be paged?

When I use a pager that escapes the escape character or highlights the content itself the output of git diff without the pager should have colors but not with the pager.

For example using git diff with a pathspec is quite short most of the time.
For git diff, I have to enable paging manually and run git diff | $PAGER usually but git log uses the pager automatically and should not use colors with it.