I have this local debug window that is so annoying

How can I hide it? What is the purpose of it?

(Or uncheck Debugger to turn off the entire apparatus.)

Mathematica 10 (and later) seems to have some strange problems regarding the debugger tools window. In 10.0 and 10.1 under Windows once I turn on the debugger I experience the window persisting even after the menu is unchecked; it reappears when I close a Notebook window. I suspect this has to do with the change to individual window menus rather than a menu bar. Fortunately in my case restarting Mathematica removes the tools window, however other people have experienced it persisting between sessions; see:
In version 10, we need to Evaluation->Quit Kernel and then Start Kernel to prevent the debugger tool window from reappear.
Evaluation > Debugger$\endgroup$