A common progressive-enhancement approach is to make your anchor an actual anchor link... if JS is not available, clicking the link will just bring the panels (which you can place down below, in the flow of the document, and hide on dom-ready/load when JS is available) to the top.
<a href="#panels" id="panelToggler">Toggle Panels</a> <div id="panels"><!-- your panels--></div>
Then in your click handler for #panelToggler, first use e.preventDefault() so it won't try to pull the anchor to the top, then include the logic to toggle the panels.
If you don't care about users without JS being able to use whatever is in the panels, then don't even show them the toggle panels control at all. Even if it doesn't look like a link, it is really janky to just have a non-working "toggle panels" line of text sitting there in your UI. In this case, it really doesn't much matter what element you hang the functionality on for the JS-enabled users... button is appropriate, but a is generally more flexible with styling options. Take a look at most of the buttons in GMail... they're clusters of nested divs.