You need to have consistent normals on your objects - normals determine what is outside and what is inside.
To recalculate normals go into Edit mode (Tab), select all geometry (A) and perform "Recalculate normals outside" (Ctrl+ N)
After all your pieces have correct normals (you will also notice consistent shading of all objects) - booleans will work as expected.
You might also want to set the object that you do boolean with as wireframe (you do that under Object Display options):
