You need to have consistent [normals][1] on your objects - normals determine what is outside and what is inside.

To recalculate normals go into Edit mode (<kbd>Tab</kbd>), select all geometry (<kbd>A</kbd>) and perform "Recalculate normals outside" (<kbd>Ctrl</kbd>+<kbd> N</kbd>)

After all your pieces have correct normals (you will also notice consistent shading of all objects) - booleans will work as expected.

You might also want to set the object that you do boolean with as wireframe (you do that under Object Display options):

[![enter image description here][2]][2]


 [1]: https://en.wikipedia.org/wiki/Normal_(geometry)
 [2]: https://i.sstatic.net/t7P5U.png