So, not knowing what your images really look like, you may want to average as much of the background as you can to come up with a good representation of the background color.
I would consider a couple things:
* Read in the pixels of each of the four edges. If there's little variance in the pixel color, then you may be done, just take the average.
* Do the same, but also read in lines from the edge to the middle until you hit a pixel that has a rather different color than your running average. Do this for all edges.
Those would be the cheapest things that I can think of to cover variances in background color. Depending on the images you're working with, you may have to get fancier.
A BufferedImage should get you your image data.
Mark