Building a Cloud-Based Image Matching System with Python and OpenCV
In this blog, we’ll explore the fascinating world of image matching algorithms and demonstrate how to build an efficient cloud-based image matching system using Python and OpenCV. Discover the magic behind feature detection, descriptor extraction, and how to implement a powerful image matcher that can find the best matches between an input image and a database of images. Get ready to dive into the world of computer vision and unleash the potential of image matching in real-world applications!
Introduction:
In the ever-expanding landscape of digital technology, image recognition stands at the forefront, transforming industries and shaping innovative solutions. Imagine a system capable of not only recognizing images but also matching them efficiently against a vast database. This blog post is your gateway into the world of cloud-based image recognition.
This robust image matching system holds immense potential. Its impact resonates across diverse sectors, from revolutionizing e-commerce platforms by enhancing product recommendations to optimizing content management systems through intelligent image sorting. By harnessing the cloud’s scalability and the precision of image recognition, our system stands poised to elevate user experiences and streamline processes.
Cloud Functions is a server-less compute platform provided by Google Cloud Platform (GCP). It allows you to run your code in a fully managed environment without the need to provision or manage servers. Cloud Functions lets you focus on writing the code for your application logic while GCP handles the infrastructure, scaling, and operational aspects. Cloud Functions can be triggered by various events, such as HTTP requests, Pub/Sub messages, Cloud Storage events, or Firestore changes. When a function is triggered, GCP automatically provisions the necessary resources, executes the function, and scales it based on the incoming workload. Learn more about it: https://cloud.google.com/functions/docs.
Getting Started:
Setting Up Your Google Cloud Project
- Create a Google Cloud Project:
a. Navigate to the Google Cloud Console:
— Open your web browser and go to [Google Cloud Console (https://console.cloud.google.com/).
b. Select or Create a Project:
— Click on Select a project in the top right corner of the console.
— Click New Project.
— Enter a unique project name.
— Ensure your billing account is linked to the project.
— Click Create to initialize your new Google Cloud Project. - Enable the Cloud Functions API:
a. Access the API Library:
— In your project dashboard, navigate to APIs & Services >Library.
b. Search for Cloud Functions API:
— In the search bar, type Cloud Functions API.
c. Enable the API:
— Click on the search result.
— Click Enable to activate the Cloud Functions API for your project. - Setting Up a Cloud Function:
a. Navigate to the Cloud Functions Section:
— Create a New Function, By clicking on Create Function.
— Provide a name for your function.
— Choose the runtime environment as Python.
b. Now Start write Your Function Code.
Let us see its implementation in Python:
1. But first things first, add the requriments.txt file:
functions-framework==3.*
certifi==2023.5.7
charset-normalizer==3.1.0
idna==3.4
jmespath==1.0.1
Pillow==9.5.0
python-dateutil==2.8.2
requests==2.30.0
s3transfer==0.6.1
six==1.16.0
urllib3==1.26.15
google-cloud-storage
opencv-python-headless2. main.py contains the primary logic for your Cloud Function. It handles HTTP requests, processes the source image, and matches it with a set of images provided in the form data.
Function Handler (function_handler):
Input:
— Receives an HTTP request with a source image file and image URLs as form data.
Processing:
i. Extracts the source image and image URLs from the request.
ii. Computes descriptors for the source image using get_ratings_by_image.
iii. Matches the descriptors with images specified by the URLs using image_matcher.
iv. Returns the best-matching image’s details as JSON response.
import logging
import functions_framework
import json
from rating import get_ratings_by_image
from matcher import image_matcher
@functions_framework.http
def function_handler(request):
try:
source_image_file = request.files.get("source_image")
image_urls_json = request.form.get("image_urls")
if not (source_image_file and image_urls_json):
return "Invalid form data. Please provide 'source_image' and 'image_urls' as form data.", 400
# Convert image_urls_json to a Python list of dictionaries
image_urls = json.loads(image_urls_json)
# Read the image data from the source image file
# source_image_data = source_image_file.read()
source_descriptors = get_descriptors_by_image(source_image_file)
if source_descriptors is None:
return "Failed to get ratings and descriptors.", 500
matched_image = image_matcher(source_descriptors["descriptors"], source_descriptors["shape"], keys=image_urls)
if matched_image is None:
return "Failed to find any matching image.", 404
return json.dumps(matched_image)
except Exception as e:
logging.error(str(e))
return "An error occurred while processing the request.", 5003. gcs_util.py contains utility functions for working with Google Cloud Storage (GCS).
Functions:
i. download_file_and_get_location(file_url): Downloads a file from a URL and returns its local location.
import os
import requests
from google.cloud import storage
def download_file_and_get_location(file_url):
file_name = file_url.split('/')[-1]
file_location = '/tmp/' + file_name
r = requests.get(file_url, stream=True)
with open(file_location, 'wb') as f:
for chunk in r.iter_content(chunk_size=1024 * 1024):
if chunk:
f.write(chunk)
return file_location4. matcher.py contains functions for matching images based on their descriptors.
Functions:
i.perform_matching(dv1, shape1, dv2, shape2, photo_url, id): Matches descriptors of two images and returns match details.
ii. image_matcher(dv, shape, keys): Performs image matching for a list of images and returns the best match.
# Standard Library Imports
import logging
import json
from datetime import datetime
from cv2 import (
FlannBasedMatcher,
resize,
DMatch
)
from numpy import asarray, float32, fromstring, uint8
from rating import get_rating_by_url
FLANN_INDEX_KDTREE = 0
index_params = dict(algorithm=FLANN_INDEX_KDTREE, trees=5)
search_params = dict(checks=100) #for better result increase checks value or pass empty dictionary
flann = FlannBasedMatcher(index_params, search_params)
def image_matcher(dv, shape, keys):
try:
logging.info("Performing Image matching")
dv1 = asarray(dv, dtype=float32)
outputs = []
logging.info(datetime.now())
for image in keys:
logging.info("Performing Image matching on image")
image_data = get_descriptors_by_url(id=image["id"], url=image["url"])
if image_data is None or not image_data["descriptors"]:
logging.info(f"No descriptors found for image with id: {image['id']}")
continue
dv2 = asarray(image_data["descriptors"], dtype=float32)
shape2 = image_data["shape"]
res = perform_matching(
dv1=dv1,
shape1=shape,
dv2=dv2,
shape2=shape2,
photo_url=image_data["photo_url"],
id=image_data["id"],
)
outputs.append(res)
logging.info("Matching completed for image with id: {}".format(image['id']))
logging.info(datetime.now())
if not outputs:
return None
sorted_objects = sorted(outputs, key=lambda obj: obj["good_match"], reverse=True)
if sorted_objects[0]["good_match"] > 40:
return sorted_objects[0]
return None
except Exception as e:
logging.error(str(e))
return None
def perform_matching(dv1, shape1, dv2, shape2, photo_url, id):
dv1, dv2, shape1, shape2 = (
dv1,
dv2,
shape1,
shape2,
)
# Ensure that dv1 and dv2 are numpy arrays of type float32
if dv1.dtype != "float32":
dv1 = dv1.astype("float32")
if dv2.dtype != "float32":
dv2 = dv2.astype("float32")
# Check if descriptors are not empty
if dv1 is None or dv1.shape[0] == 0 or dv1.shape[1] == 0:
logging.warning("dv1 descriptors are empty.")
return None
if dv2 is None or dv2.shape[0] == 0 or dv2.shape[1] == 0:
logging.warning("dv2 descriptors are empty.")
return None
# Check if dimensions of descriptors are compatible
if dv1.shape[1] != dv2.shape[1]:
logging.warning("Dimensions of dv1 and dv2 descriptors are not compatible.")
return None
# Convert descriptors to list of DMatch objects for knnMatch
matches = flann.knnMatch(dv1, dv2, k=2)
MIN_MATCH_COUNT = 100
# Store all the good matches as per Lowe's ratio test.
good_matches = [m for m, n in matches if m.distance < 0.9 * n.distance]
if len(good_matches) > MIN_MATCH_COUNT:
logging.info(f"Image Match with: {len(good_matches)}")
else:
logging.info("Not matched: {}/{}".format(len(good_matches), MIN_MATCH_COUNT))
return {
"url": photo_url,
"good_match": len(good_matches),
"id": id,
}4. descriptors.py contains functions for processing image descriptors and determining shapes.
Functions:
i. resize_with_aspect_ratio(image, shape, maxReso=800): Resizes an image while maintaining its aspect ratio for calculating the descriptors very fast because high quality images takes long time to calculate the descriptors.
ii. get_descriptors_by_image(file): Computes descriptors and shape for an input image file.
iii. get_descriptors_by_url(file): Computes descriptors and shape for an input image urls.
import logging
from pickletools import uint8
from datetime import datetime
import urllib
from numpy import asarray, fromstring, uint8
from cv2 import (
AKAZE_create,
IMREAD_COLOR,
imdecode,
resize,
)
def resize_image_with_aspect_ratio(image, shape, maxReso=720):
width, height = int(shape[1]), int(shape[0])
size = width, height
if height >= width:
perc = maxReso / height
width_new = int(perc * width)
size = width_new, maxReso
print("new height and width is ", maxReso, " ", width_new, " perc ", perc)
else:
if width > height:
perc = maxReso / width
height_new = int(perc * height)
size = maxReso, height_new
print("new height and width is ", height_new, " ", maxReso, " perc ", perc)
img = resize(image, size)
return img, img.shape
def get_descriptors_by_image(file):
try:
akaze = AKAZE_create(descriptor_type=3, threshold=0.001299)
contents = file.read()
logging.info(f"size of file: {len(contents)}")
nparr = fromstring(contents, uint8)
img = imdecode(nparr, IMREAD_COLOR)
# Assuming resize_with_aspect_ratio is defined elsewhere
img, shape = resize_image_with_aspect_ratio(image=img, shape=img.shape)
logging.info(f"New shape: {shape}")
_, dv = akaze.detectAndCompute(img, None)
if len(dv) <= 100:
logging.info(f"Image shape: {img.shape}, Descriptors shape: {dv.shape}")
except Exception as e:
logging.error(f"Error: {e}")
return None
return {
"descriptors": dv.tolist() if len(dv) > 100 else [],
"shape": shape,
}
def get_descriptors_by_url(id: str, url: str):
try:
resp = urllib.request.urlopen(url)
image = asarray(bytearray(resp.read()), dtype="uint8")
img = imdecode(image, IMREAD_COLOR)
akaze = AKAZE_create(descriptor_type=3, threshold=0.001299)
print("read image", "and shape", img.shape)
img, shape = resize_image_with_aspect_ratio(
image=img, shape=img.shape
) # Resize the image
print("New shape ", shape)
# akaze.setThreshold(3e-4)
_, dv = akaze.detectAndCompute(img, None)
if len(dv) <= 100:
print("image", img.shape, dv.shape)
except Exception as e:
logging.error(f"Error: {e}")
return None
return {
"descriptors": dv.tolist() if len(dv) > 100 else [],
"shape": shape,
"photo_url": url,
"id": id
}In this blog series, we embarked on a transformative journey into the realm of cloud-based image recognition using Python and Google Cloud services. Beginning with the foundational setup of our Google Cloud project and enabling essential APIs, we seamlessly integrated serverless computing through Google Cloud Functions.
The heart of our system lies in the meticulously crafted Python code. By leveraging powerful libraries such as OpenCV, we explored advanced image processing techniques, including descriptor computation and image matching. The intricate dance of data validation, error handling, and response generation ensured a seamless user experience.
In essence, our journey illuminated the path to crafting intelligent, cloud-based solutions. As we conclude, we stand at the threshold of innovation, armed with the knowledge to shape a future where the fusion of Python, cloud services, and image recognition transforms the way we interact with digital content. The impact of this system is profound, promising efficiency, accuracy, and unprecedented user engagement in the vast landscape of technology.
