It seems like you understand why there are unique minifigs, but I'll point it out explicitly anyway. Unique minifigs add character to sets, and add to playability. For example, I always wanted this guy as a kid:

Without him, my pirates were just a leaderless band, but once they have a fearless leader with a peg leg and hook, things become a lot more exciting. As you pointed out, unique figs are also a necessary part of the licensed themes. What would Star Wars be without Darth Vader?

The core of your questions seems to be why these unique figs end up in so many sets such that you have duplicates and triplicates of them. I'm sure this isn't TLG's goal, but there are some marketing reasons for doing this.

First, TLG ideally wants each set to be a whole product. This generally means that the sets should be able to stand on their own, and upon seeing a set, a customer would actually want to buy it. Keep in mind that your rationality as a parent isn't always the same as others who may be buying Lego for your child, or children themselves. Here's an example of set to help me explain what I mean:

That's Gandalf Arrives (set 9469). As you can see, it is a pretty small set with Gandalf and Frodo. These are both core characters, and they each appear in several sets, but this set wouldn't be able to stand on its own without them. While this set may have less appeal to you or your child because you might already have these minifigs, the set has greater appeal to children who don't already have these minifigs. It also has greater appeal to your sister-in-law who needs to pick up a Lego set for your child's birthday party. She won't know what minifigs your child has already, and she's just looking for a nice reasonably cheap set with good playability.

Second, TLG uses unique minifigs to create desire to collect the whole product line. While there is duplication among the major characters, if you want to have all of the minor characters available, you pretty much have to buy every major set in the theme. This makes a lot of sense from a marketing point of view. For example, if you already own Helm's Deep (9474), does a small add-on really have much appeal to you? Uruk-Hai Army (9471) is a small set which attaches to the side of Helm's Deep:

This set may not have much appeal. It's just a basic castle wall with some men fighting some bad guys. However, this is the only set which contains the guy (Eomer) who brings in the cavalry and ultimately makes the battle a victorious one. This adds to this otherwise somewhat boring set, and makes it more appealing to consumers.

It seems like you understand why there are unique minifigs, but I'll point it out explicitly anyway. Unique minifigs add character to sets, and add to playability. For example, I always wanted this guy as a kid:

Without him, my pirates were just a leaderless band, but once they have a fearless leader with a peg leg and hook, things become a lot more exciting. As you pointed out, unique figs are also a necessary part of the licensed themes. What would Star Wars be without Darth Vader?

The core of your questions seems to be why these unique figs end up in so many sets such that you have duplicates and triplicates of them. I'm sure this isn't TLG's goal, but there are some marketing reasons which cause it to happen.

First, TLG ideally wants each set to be a whole product. This generally means that the sets should be able to stand on their own, and upon seeing a set, a customer would actually want to buy it. Keep in mind that your rationality as a parent isn't always the same as others who may be buying Lego for your child, or children themselves. Here's an example of set to help me explain what I mean:

That's Gandalf Arrives (set 9469). As you can see, it is a pretty small set with Gandalf and Frodo. These are both core characters, and they each appear in several sets, but this set wouldn't be able to stand on its own without them. While this set may have less appeal to you or your child because you might already have these minifigs, the set has greater appeal to children who don't already have these minifigs. It also has greater appeal to your sister-in-law who needs to pick up a Lego set for your child's birthday party. She won't know what minifigs your child has already, and she's just looking for a nice reasonably cheap set with good playability.

Second, TLG uses unique minifigs to create desire to collect the whole product line. While there is duplication among the major characters, if you want to have all of the minor characters available, you pretty much have to buy every major set in the theme. This makes a lot of sense from a marketing point of view. For example, if you already own Helm's Deep (9474), does a small add-on really have much appeal to you? Uruk-Hai Army (9471) is a small set which attaches to the side of Helm's Deep:

This set may not have much appeal. It's just a basic castle wall with some men fighting some bad guys. However, this is the only set which contains the guy (Eomer) who brings in the cavalry and ultimately makes the battle a victorious one. This adds to this otherwise somewhat boring set, and makes it more appealing to consumers.

As an aside, many Lego fans want to get a complete set of minifigs, but can't afford all the sets, so your extra Vader's have good resale value on sites like Bricklink where they sell for about $10 used.