The first Game Engine that used the term Engine was, to my knowledge, the Doom Engine by id Software in 1993.
Database Engines, however, used this term much earlier. dBase and, especially Foxpro, later also Borland, used the term Database Engine in their advertisements basically all through the 80ies.
Actually, I was trying to find Software ads and marketing material before 1980 and failed - interestingly, nobody seemed to see the need for software ads before the 1980ies