Enhancing our search with the analyze image cognitive service

Last week we used the Bing image search cognitive service to help us to find images on the internet we didn’t previously have. If the first image wasn’t relevant, we used the search results to allow us to display and pick the next best image. What if we could use AI for this manual picking process, to find a more relevant image auto-magically? Today we are going to use another cognitive service to achieve that.

Analyzing images with AI

Microsoft Azure has an “analyze an image” cognitive service, that can return a lot of different pieces of information. In particular for us, we noticed that it can identify Lego.

In contrast, one of our sets returned the following image as the top/first image last week. These tags do not contain the Lego tag, as it’s clearly a property listing photo.

Given this information, we know that Lego models have a distinct tag identifier – the name “LEGO”. We are going to perform our Bing image search, and then analyze each image. If the image doesn’t contain the “LEGO” tag, we can discard that image. This will help guarantee that we only select and show pictures that contain the Lego tag. This is much more efficient than opening the image picker and removing the images ourselves – a very manual process for us.

Setting up the Analyze image API call

We start by reading the API for the Analyze image call. Since we are looking at images online, we will use the imageUrl call. We pass in tags, and plan to look for the tag “Lego”. In the example below, in theory, this should filter out the last two images. We are using the same cognitive services resource we created last week, so we use the same subscription key too, (in blue below). You’ll note in the query string we request for “Tags” to be returned in the response, in English, (in red below). The image we are processing, needs to be passed inside a Json document, which we can create and serialize using the JsonConvert command, (in green below). We then send the request, and process the response, returning each tag and the confidence in that tag (as a percentage). Here is the completed code.

With this new code, we also update our image picker page. Instead of getting the first 10 images, we get the first 20 images, throwing away any images without Lego until we get to 10 images. (Note that it’s possible we could end up with less than 10 remaining images). Here is the current page for an X-Wing, note the two false-positive pictures of houses.

After the update, we can now see it is only showing pictures that contain Lego.

Wrap up

We’ve automated our main image selection and image picker to only display Lego images with a powerful automatic filter! While the cognitive services are useful by themselves, we can see that combining two or more cognitive services together is super powerful. We will continue to look for other opportunities to use cognitive services in the future.

References

• Interactive cognitive service test page: https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/
• Analyze cognitive service API documentation: https://eastus.dev.cognitive.microsoft.com/docs/services/5adf991815e1060e6355ad44/operations/56f91f2e778daf14a499e1fa
• Featured image credit: https://jj09.net/wp-content/uploads/2018/12/ai-search.jpg