I would first convert the RGB values into Lab space, then use the l,a,b values in the same 3d space distance calculation I have outlined above for the rgb values. This is known as the CIE76 algorithm (it's not the best, but probably one of the easier ones to implement... see my note below). That numeric difference value should get you close to understanding how similar two colors are to the human eye. Then based on some tests, I would pick a threshold value that you're comfortable with.
For converting RGB to Lab I found this page: http://cookbooks.adobe.com/post_Useful_color_equations__RGB_to_LAB_converter-14227.html
NOTE: This is basically the quickest (albeit, a bit dirty) way to get to color difference. I say dirty because there are more sophisticated algorithms out there that doesn't get distorted at the more saturated end of the color spectrum. If you want to see the more up-to-date algorithms, take a look at this wikipedia: http://en.wikipedia.org/wiki/Color_difference.