What you want to do for this kind of game, is not directly change the transform of the ball but change the velocity of the rigidbody. Set friction to 0 in the rigidbody of the ball and lock the movements of the panels. Have a script that detects if the ball collides with the panel, on collision delete the panel and let the physics response do the work. Also set the physics material of the ball/walls/panels to an elastic collision. If you want to add some speed when the ball hits a panel, increment/alter the velocity or apply a force to the rigidbody.

[Tutorial][1]


 [1]: https://www.google.ca/url?sa=t&rct=j&q=&esrc=s&source=video&cd=1&ved=0CBwQtwIwAA&url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DRrdzLwZYz8Q&ei=k_j4VMPJDoiXgwTXzoPQCw&usg=AFQjCNFGMHKc9Pjf14BWyebIM4-kXUC72g&sig2=UwxcBWFmNubp192QTgyjoA&bvm=bv.87611401,d.eXY&cad=rja