Google brings multimodal capabilities in Lens to AI Mode

Google shared that multimodal capabilities in its Lens feature would be coming to its AI mode that is available for Google One AI Premium subscribers, while also expanding AI Mode to millions of Google Labs experimental users in the U.S.

The feature will allow eligible Google Lens users to captures images of objects and then take an AI-powered deep dive with Gemini to learn more, ask complex questions, or get relevant links. This is what Google called AI Mode’s new multimodal understanding.

The company further noted that AI Mode queries were “twice as long” when compared to the standard search queries on Google.

“With Gemini’s multimodal capabilities, AI Mode can understand the entire scene in an image, including the context of how objects relate to one another and their unique materials, colors, shapes and arrangements. Drawing on our deep visual search expertise, Lens precisely identifies each object in the image,” said Google in its blog post explaining the new update.

A range of users with diverse needs, such as shopping, planning vacations, carrying out research, or even just someone going on a walk, can use AI Mode through Lens to explore their surroundings in greater detail than before.

“Using our query fan-out technique, AI Mode then issues multiple queries about the image as a whole and the objects within the image, accessing more breadth and depth of information than a traditional search on Google,” noted the company.

AI Mode is still being tested and improved by Google. Interested users with both Android and iOS systems can sign up in Labs to try out the AI Mode in the Google app and share their feedback.