First of all, a bullet-proof digital restriction management (DRM) system which reliably stops a user from executing a program which is already on their computer is impossible. You can put barriers in the users path to make it harder for them, but any such barrier can be overcome by a determined user. So forget about creating a solution which is completely impervious to any conceivable attack. All you can do is a solution which is difficult enough to overcome that the average user won't bother.
In the case of Steam, you don't run the game, you ask Steam to run the game. If you look at the properties of any of the shortcuts for Steam games on your desktop, you will notice that they are not actually pointing at the game executable. They are links in the form steam://rungameid/1234567890 . What happens when you double-click on one of these shortcuts (or start the game from the Steam client) is
- Steam is launched in the background (if it isn't already running).
- Steam asks the user to authenticate with username and password (if they are not already logged in).
- Steam contacts the Valve server to confirm that this user has a valid license for this game (some games allow offline play, but they still demand that the last license validation wasn't too long ago).
- When the server confirms that the user has a valid license, then and only then does Steam launch the game executable.
Now it would of course still be possible for a technically versed user to locate the game executable in their Steam files and run it directly. In order to prevent this, the game executable might also try to confirm that the user owns the game on Steam on its own. But that's something the game developers would have to implement. For example, by using the Steamworks API in their game (When the user does not own the game on Steam, then even initializing the Steam API will fail). Note that the Steamworks API is not a stand-alone library. It requires a running instance of the Steam client with the user being logged in. Most of the functionality of the library will just contact the Steam client and asks it to fulfill the request. The Steam client might then delegate that request further to the Valve servers.
So the whole Steam DRM scheme hinges on one crucial component: The Steam client. If someone would hack their copy of the Steam client to bypass the license verification, then they could play games they downloaded through Steam and then refunded or copied from another Steam user.
It would also be possible to hack the game executable itself to no longer confirm the license by dummying out any Steamworks API calls. Now it would be possible to create a copy of it and run it without Steam.