What you want to do for this kind of game, is not directly change the transform of the ball but change the velocity of the rigidbody.
Set friction to 0 in the rigidbody of the ball and lock the movements of the panels.
Have a script that detects if the ball collides with the panel, on collision delete the panel and let the physics response do the work.
Set the physics material of the ball/walls/panels to an elastic collision.
If you want to add some speed when the ball hits a panel/player, increment/alter the velocity or apply a force to the rigidbody.
The player should not be static. Static is used to optimize non moving GameObjects. The player should have a its rigidbody x and y positions locked and moved with a script on user input. Lock the rigidbody under constraints in the inspector.
Hope this helps!