If you're really sure that the slowness you notice is due to the time spent by Visual Studio packaging the application in an MSI file, then you can:
either turn off the building of the setup every time you compile the solution,
or simply unload, like you already do, the installer project. This is perfectly fine and is not limited to installer projects; you can unload anything you're not using right now, and reload those projects later when you need them. Note that you can even unload the projects which are referenced in other projects, as far as they were built successfully.
For example I'm currently working on a project which has 35 small projects. Most of the time, only two of them (the code I'm working on and the corresponding unit tests) are available, every other being unloaded.
Unloading projects you're not using right now has also a benefit on time spent by Visual Studio when loading the solution.
Note that companies working on large projects are using nightly builds, which may often take hours. If you're feeling that you're waiting for too long while the code is compiling, consider using nightly builds (and/or continuous integration in general; it's never too soon to start using it).

Once you have continuous integration, you may heavily optimize Visual Studio performance by loading from the version control only the projects you need.