Hey this was driving me mad as well, but I figured it out. When you're browsing on Chrome (and hangouts is open) you'll see a little icon for hangouts on the top right corner next to the menu button. Right click on it and there's an option to remove! Yay.

When you're browsing from Chrome (and Hangouts is open) you'll see a little icon for Hangouts on the top right corner next to the menu button. Right click on it and there's an option to remove!