If you want to restrict only by file extension you can use grep --include option:

 grep -R --include="*.txt" "pattern" /path/to/dir/

Another approach is to eliminate files which are not text but it will include html and js files, which after update are excluded with option --exclude for example:

 find /path/to/dir -type f -print | xargs file | grep --exclude=\*.{js,html} text | cut -f1 -d: | xargs grep "pattern"