There are different levels of element customization in WPF, depending what class you extend from. Each has its own uses and is implemented differently. It is not clear from your question if you are asking about a specific type of control or about all of them in general. So, I will tell you what I think about different ones.
UIElement or FrameworkElement
Extending UIElement gives you the lowest level custom control where you have complete control over the layout and rendering. FrameworkElement is slightly higher level as it does most of the common layout stuff for you while also allowing you to override key parts of it. The main idea with these is that they do their own rendering rather than composing other elements together.
I have made a number of custom FrameworkElements over the years. One example is a ruler similar to one you might find in a program like Photoshop. It has a bunch of properties providing customization for how it is displayed as well as showing markers indicating mouse position relative to the ruler (and a number of other little optional features). I have used it in two different professional projects. I think the main benefit is that it is extremely easy to drop in and set properties/bindings on wherever desired. Build it once, use it over and over.
Control
Extending Control introduces the concept of compositing multiple elements/controls into one reusable component via control templates.
I have used this one less often, but still find it very valuable in the right circumstances. Again, the main benefit here is reusability. You create a control with properties that make sense for what you want to do, then hook up those properties to the properties of the controls in it's control template. Really, this is the same as applying a new template to an existing control, with the added feature of being able to define your own dependency properties. You also have the ability to perform custom logic in the control's code if you need to.
I may be misreading some of your text, but you seem to imply that making a custom control is considerably more difficult than making a control template for an existing control. I have found that the two are nearly identical in most cases using this approach, the only difference being whether you have a code behind you can use.
User Control
A user control is really only slightly different from a custom control practically speaking. Only, instead of defining a control template, you define the visual content directly.
This is probably the most common type of custom control. It is basically the standard method for making XAML based content in a WPF application. These can be reused like other controls, but are more suited for single use such as the content of a dialog or window or something else that is specific to a single application.
Some Other Control
You can also extend an existing control to add additional functionality to it. This way, you still get all the features the control offers and only have to implement the additional bit.
For example, I have a custom control called an AutoScrollRichTextBox that extends RichTextBox. So, it does everything a RichTextBox can do. It also has the ability to automatically scroll to the bottom when content is added to the text box (which it only does if the text box was already scrolled to the bottom before the addition content was added).
I could have implemented that feature as an attached property instead of an extension of the control (and maybe I should have), but it works, and I have used it in three different applications (as an output window and as a chat log). So, I am happy with it.
In the end, it really is just a matter of how self-contained, reusable, and easy to drop in you want a control to be. If there is already a control that does what you want, and you just want it to look different, then you should definitely use styles and templates to achieve that. However, if you want to make something that doesn't already exist, limiting yourself to using only styles and templates will make the implementation work harder and make the end result less reusable and more difficult to set up additional instances (unless all instances are identical).
The examples of making things like buttons that look different are not examples of what you should use a custom control for. They are just examples of how someone would go about making a custom control for the purpose of teaching the details of the process. If you actually want a customized button, just customize a button.
oxyplot. can you do smth like that using styles and templates for standard wpf controls?<</>>buttons. Creating custom controls in WPF is reasonably simple, though, so I wonder if you didn't look at the wrong documentation.