Question: If I downloaded your complete repository, would these files be any good to me? If I changed them to fit on my machine, put them into the repository, and you checked them out, would that interfere with your work?
It doesn't matter if you are only one person. A single person can download a private repository on two computers, say on your desktop and on your laptop computer. So whether it is private or not shouldn't affect your decision. And things that are private today might not be private tomorrow.
Get rid of them, with git ignore or any other way.