How do I find an image contained within an image?

Question

I'm currently building what basically amounts to a cross between a search engine and a gallery for web comics that's focused on citing sources and giving authors credit.

I'm trying to figure out a way to search an image to find characters within it.

For example:

cyanide and happiness

Assuming I have the red character and the green character saved as Red Man and Green Man how do I determine if an image contains one or the other.

This doesn't need to have 100% recognition or anything is this is more of an added feature I'd like to create, I'm just not sure where to start. I've done a lot of googling for image recognition but haven't found much helpful.

For what it's worth, I'd prefer to do this using Python.

I don't see how sikuli would be used for this. Sikuli looks like it's strictly for GUI interfaces. These are user uploaded pictures on a web server. — Adam
– Adam, Commented Oct 21, 2011 at 18:46
Sikuli is not only for GUI's. You can technically use it to run any Jython script you want. You could write a script that would open the user images then look for the image pattern you want to find. That's just a first blush idea. — Snaxib
– Snaxib, Commented Oct 21, 2011 at 19:24
However, Sikuli doesn't support scaling or rotation in images.... — Adam
– Adam, Commented Oct 21, 2011 at 20:00

Benedikt Köppel · Accepted Answer · 2023-02-10 10:30:28Z

As Moshe's answer only covers matching a template that is contained only once in the given picture. Here's how matching several at once:

import cv2 import numpy as np img_rgb = cv2.imread('mario.png') template = cv2.imread('mario_coin.png') w, h = template.shape[:-1] res = cv2.matchTemplate(img_rgb, template, cv2.TM_CCOEFF_NORMED) threshold = .8 loc = np.where(res >= threshold) for pt in zip(*loc[::-1]): # Switch columns and rows cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (0, 0, 255), 2) cv2.imwrite('result.png', img_rgb)

(Note: I changed and fixed a few 'mistakes' that were in the original code)

Result:

Source: https://opencv24-python-tutorials.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_template_matching/py_template_matching.html#template-matching-with-multiple-objects

FWIW, I used your code here to test something else and it appears to me that the line w, h = template.shape[:-1] should be h, w = template.shape[:-1], at least with my test images (it is consistent across 3 sets of images)
Just grab the source code from the source link in his answer; it is up to date and works.
Both the docs and this answer fail to mention this key detail - NOTE that there are MANY overlapping rectangles here! If not having overlaps matters to you, there are apparently methods of excluding them stackoverflow.com/a/52458784/9890027

DanBrezeanu · Accepted Answer · 2019-06-13 22:09:01Z

For anyone who stumbles across this in the future.

This can be done with template matching. To summarize (my understanding), template matching looks for an exact match of one image within another image.

Here's an example of how to do it within Python:

import cv2 method = cv2.TM_SQDIFF_NORMED # Read the images from the file small_image = cv2.imread('small_image.png') large_image = cv2.imread('large_image.jpeg') result = cv2.matchTemplate(small_image, large_image, method) # We want the minimum squared difference mn,_,mnLoc,_ = cv2.minMaxLoc(result) # Draw the rectangle: # Extract the coordinates of our best match MPx,MPy = mnLoc # Step 2: Get the size of the template. This is the same size as the match. trows,tcols = small_image.shape[:2] # Step 3: Draw the rectangle on large_image cv2.rectangle(large_image, (MPx,MPy),(MPx+tcols,MPy+trows),(0,0,255),2) # Display the original image with the rectangle around the match. cv2.imshow('output',large_image) # The image is only displayed if we call this cv2.waitKey(0)

I agree with Moshe but I believe it should be cv2.matchtemplate(large_image, small_image, method). Also here is another good source of information for template matching in python.
Weirdly enough from cv2 import cv raises ImportError: cannot import name 'cv' while import cv2 works just fine…
SOLUTION: Ok so as I'm using Py3, it actually uses OpenCV3 despite it still imports as cv2 so some stuff have changed places/names.
Will it work if one of the image is present as low opacity in other image.(On of the input images is watermarked in other image.)
I had to sudo apt-get install python3-opencv for this to work on Ubuntu 20.04

mevatron · Accepted Answer · 2011-10-21 22:53:13Z

OpenCV has a Python interface that you could look at. If the characters, don't change too much you could try to use the matchTemplate function.

Here is their official tutorial on it (the tutorial is written using the C++ interface, but you should be able to get a good idea of how to use the function in Python from it).

Maxim Terleev · Accepted Answer · 2021-09-09 11:50:41Z

Important note: matchTemplate is even able to detect resized and rotated templates. Here's the code and outputs.

import matplotlib.pyplot as plt import numpy as np import cv2 image = cv2.imread('/content/picture.png') template = cv2.imread('/content/penguin.png') heat_map = cv2.matchTemplate(image, template, cv2.TM_CCOEFF_NORMED) h, w, _ = template.shape y, x = np.unravel_index(np.argmax(heat_map), heat_map.shape) cv2.rectangle(image, (x,y), (x+w, y+h), (0,0,255), 5) plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

Image: picture Template: penguin Result: detected

Detailed explanaition over here (my blog): simple-ai.net/find-and-replace-in-image

For this simplified example it may work with resize and rotation, but not in even a little more complex scenarios. Claiming this is misleading.

Collectives™ on Stack Overflow

How do I find an image contained within an image?

4 Answers 4

4 Comments

5 Comments

1 Comment

2 Comments

Linked

Hot Network Questions