Okay, so the specific ask is how to avoid loading plugins if Vim is launched but not interacted with by a user or, IOW, how to do lazy-loading of plugins, triggered by user activity.
The most obvious way that comes to mind requires that you move all of the applicable plugin directories to pack/plugins/opt under ~/.vim (or wherever you keep your personal vim configuration). Vim does not automatically load plugins unless they are in pack/*/start. It will load plugins on-demand if they are located in pack/*/opt. This is done with the packadd command.
So move the plugins and then add something like this autocommand to your vimrc...
autocmd CursorMoved,CursorMovedI * \ packadd plugin-name-1 | \ packadd plugin-name-2 | \ ... packadd plugin-name-n
...meaning you have a packadd line for each plugin that was moved. Specifically, replace plugin-name-# with an actual plugin directory name (e.g. vim-fugitive).
It should be pretty clear what this does. As soon as one of the specified events occurs (i.e. the user moves the cursor) each listed plugin will be loaded. You might need to play around with the actual event or events you use...the two CursorMoved events seem like a good start.
There are some refinements that you'll probably want to do if this involves a large number of plugins. For instance, put the packadd calls in a function and call the function from the auto command. Then you can guard the packadd calls with a flag and prevent them from being executed more than once. (Something to consider if there's a noticeable performance hit from running the group of packadd commands repeatedly.)
See :h packadd, :h autocommand, :h autocommand-events for some of the major pieces here.