Announced at Google I/O, Ask Photos is based on Google’s Gemini AI model. It massively expands Google Photos’ search abilities, using contextual info like how many times you’ve taken pictures of a particular person or object, and where you took them too.

One example shown off during the I/O keynote was asking for your car’s license plate – instead of just showing a bunch of random cars, it knows which car appears most often in your library, and which one is most often seen parked at your home. It also gives you a text summary as well as the images.

Other examples include asking for a timeline of your child’s birthday party themes over the years, with a descriptive list along with the images themselves, and when your child had their first swimming lesson.

Google photos has been around for nine years now, and today sees six billion photo and video uploads every twenty-four hours. That’s a whole lot of data to train Gemini’s face and object detection algorithms. In terms of search accuracy, Google reckons Gemini will nail even the most complex of search queries.

Ask Photos is set to roll out later in 2024. It will work for videos as well as photos, with text and voice input options. It should be integrated into Google Photos, rather than its own separate app.

Right now it’s unclear if you’ll need a Google One subscription; features like Magic Eraser and Magic Editor were initially limited to subscribers only, before Google made them free to all users.

• Related: Best AI phones – which smartphone has the best AI features?

• Read more: What is Google Ask Photos? New Google Photos AI search explained

1. Ask Photos

Google Photos is introducing a new experimental feature called Ask Photos, powered by Gemini AI models. Ask Photos allows users to search their photo library more intuitively, using natural language queries like “Show me the best photo from each national park I’ve visited.” Ask Photos also helps with tasks like curating trip highlights and generating captions — simply ask it for the best photos from your trip abroad, and it’ll instantly provide you with a curated selection of some of your best shots. And if your hats are of the tinfoil variety,  Google emphasises the privacy protections in place, noting that personal data is never used for ads and is safeguarded with industry-leading security measures. The Ask Photos feature will begin rolling out to users in the coming months.

2. Veo: AI-generated videos

Image generation has already blown minds with its rapid development over recent years, and the same appears to be on the horizon for video. At I/O 2024, Google unveiled Veo — a powerful new video generation model capable of creating high-quality 1080p videos over a minute long, in various cinematic styles. Veo has an advanced understanding of natural language and visual semantics, allowing it to accurately capture the tone and details of a prompt, while sticking to the laws of physics for realistic and natural motion. Very impressive indeed. Google is collaborating with filmmakers and creators to experiment with Veo and improve how they design, build and deploy the model to best support the creative storytelling process. Veo is currently available for select creators as a private preview, with plans to bring its magic to YouTube Shorts and other apps in future.

3. Imagen 3: Google’s best text-to-image model to date 

Google also unveiled Imagen 3, its most advanced text-to-image model to date, capable of whipping up highly detailed, photorealistic images with significantly fewer visual artefacts compared to previous versions. One thing we’re particularly looking forward to testing out, is the improvement in AI-rendered text within images, which is something that current models struggle with. If it works as well as advertised, it could open up a whole new world of content generation. Bespoke AI birthday cards, here we come.

4. Circle to Search rescues your homework

Circle to Search, a feature already available on various Pixel and Samsung devices, allows users to search for anything on their phone using a simple gesture without switching apps. AT I/O 2024, Google announced that it’s expanded Circle to Search’s capabilities to help students with homework directly from their mobile devices. By circling a challenging prompt, students can receive step-by-step instructions for solving physics and math word problems. Later this year, the feature will be enhanced to tackle more complex problems involving symbolic formulas, diagrams, and graphs, powered by LearnLM, Google’s new family of learning-focused models. Circle to Search is set to double its availability by the end of the year, potentially giving teachers some time to prepare for the onslaught of AI-generated homework.

5. Gemini on Android

Google is also enhancing its Gemini AI assistant on Android so that it can better understand the context of what’s being shown in the current app. and the app they are currently using. This generative AI-powered experience, which is integrated into the Android operating system, is set to become more versatile and user-friendly. Soon (a specific time frame hasn’t been specified), Android users will be able to access Gemini’s overlay on top of the app they are using, allowing for seamless interaction with the AI assistant, unlocking actions like dragging and dropping generated images into Gmail, Google Messages, and other apps. Gemini Advanced subscribers will also be able to take advantage of the “Ask this PDF”  feature, which automatically mines answers from PDF documents without the need to scroll through multiple pages. This update is expected to roll out over the next few months. Lastly, Google is also testing a new AI-powered feature which could detect red flags during phone calls, warning you if it sounds like the person you’re speaking to could be a scammer.

6. Gemini for Google Search

A new Gemini model, customised for Google Search, combines advanced capabilities like multi-step reasoning and planning, with Google’s existing search systems. In other words, it’s Google Search, but better. AI Overviews — a Labs feature which provides quick answers and overviews to user queries by curating sources from multiple sites — is now rolling out to everyone in the US, with plans to expand worldwide by the end of the year. Soon, users will be able to adjust AI Overviews by simplifying the language (useful for answering children’s queries), or breaking down the information in more detail. Google Search will also offer planning capabilities, starting with meals and vacations. Users can create customised meal plans and easily export them to Docs or Gmail, and later this year, additional categories like parties, date nights, and workouts will be added. 

7. Ask questions with a video

Ever dreamed of using AI to search through video content? Your time has come, thanks to the new Search with video feature. One example Google provides is using the video search feature to troubleshoot a broken record player. Instead of struggling to find the right words to describe the problem, the user can simply record a video of the record player’s unexpected behaviour, such as the metal piece with the needle drifting unexpectedly. Searching with video will be available soon for Search Labs users in English in the US, and will expand to more regions in the (hopefully) near future.

8. Gemini 1.5 Pro reads all the things

Google also unveiled Gemini 1.5 Pro, its most advanced AI model, to Gemini Advanced subscribers. Its main draw is its significantly expanded context window which starts at one million tokens, making it the longest of any widely available consumer chatbot worldwide. With such a lengthy context window, Gemini Advanced can comprehend multiple large documents totalling up to an impressive 1500 pages, or summarise 100 emails. In the near future, it will also be capable of processing an hour of video content or codebases exceeding 30,000 lines.

To fully utilise this extensive context window, Gemini Advanced now allows users to upload files directly from their devices or via Google Drive. This feature enables users to quickly obtain answers and insights from dense documents, such as understanding the specifics of a pet policy in a rental agreement, or comparing key arguments from multiple lengthy research papers. It can even create custom visualisations and charts, based off of information from spreadsheets.

9. A real Gem

Google I/O 2024 also introduced a new feature for Gemini Advanced subscribers called Gems, which will let users to create personalised versions of the Gemini AI assistant. With Gems, users can tailor their AI companion to suit their specific needs and preferences, whether they’re looking for a gym buddy, sous chef, coding partner, or creative writing guide, to name but a few optimistic examples. Users simply need to describe what they want their Gem to do and how they want it to respond. You could, for example, request, “You’re my running coach, give me a daily running plan and be positive, upbeat and motivating.” Gemini will then take these instructions and, with a single click, enhance them to create a Gem that meets your requirements.

10. Turbocharged Gmail

The Gmail mobile app also has an exciting update on the way which is, you guessed it, Gemini related. The new Gemini icon in Gmail will offer helpful options, such as summarising emails, listing the next steps, or suggesting replies. Users can also use the open prompt box for more specific requests, like finding a particular document or asking for discussion questions for an upcoming meeting.

• Read more: Google Pixel 9 preview: specs, release date and everything we know

• Read more: Google Pixel 8a preview: specs, release date and everything we know

Google I/O 2024: watch now

Here’s what was announced at Google I/O 2024:

Features

What is Google Ask Photos? the new Google Photos AI search explained

Gemini-based AI search coming to Google Photos later in 2024

19 May 2024 / 06:00 BST19 May 2024 / 07:10 BST

Features

10 things we learned from Google I/O 2024

All the tantalising announcements from Google’s Big Tech Show — aka, AI In Everything

14 May 2024 / 19:53 BST14 May 2024 / 19:53 BST

Our original article continues below

The Google I/O keynote starts at10am PT local for Google on America’s West Coast, which is 1pm ET on the other side of the US, and 6pm BST in the UK. Things typically last for around a couple of hours, so tune in with us. If you’d rather check the live stream out via YouTube, that’s cool too. You’ll find it on Google’s YouTube page, where there will also be a version available in American Sign Language for accessibility. Anyone who’s really into Google will also find it easy to stream other sessions and talks from Google I/O 2024. The opening keynote is the main draw for most people, but there’s a full programme running from 14-15 May, and anyone registered with the Big G as a developer can stream along.

You’ll find a handy big ‘Register’ button in the top right-hand corner of the official Google I/O website, where the opening shindig is also being aired in its entirety.

Join us to watch all the action unfold, but if you don’t have time, check back after the event for all the latest news and easy-to-understand analysis from the show, which we’ll be covering in full. In the meantime…

Google I/O 2024: what we expect to see

One of the only guaranteed hardware announcements we expect to see at I/O will be the long-awaited Pixel 8a, which, from everything we know so far, is shaping up to be one of the best-value handsets of the year. The Pixel ‘a’ range has long been praised for delivering a pure, long-updated Android experience with quality camera hardware and the latest Google features to boot, and we expect this year to be no different.

Android 15 will also, unsurprisingly, take up a fair chunk of stage time at Google I/O 2024. Currently available in beta, it includes all manner of new features focused on productivity, privacy, and security. And, of course, this leads us on to the topic of AI — a trend that refuses to slow down its momentum.

We could see updates on Google Gemini (formerly Bard), and we’re hoping to see some powerful new functionality that brings the fight straight to the likes of ChatGPT et al. Improvements to existing AI features like Circle to Search could also be on the cards. CSS Insight’s Principal Analyst & Director Americas, Leo Gebbie believes that “AI will be the major theme for Google I/O and we expect updates from every part of the business, focusing on how Google’s Gemini platform will become more integrated across the board. With Google I/O taking place just ahead of WWDC, Google will be keen to get ahead of Apple in terms of explaining why Google has a stronger position in AI given its extensive investment in Gemini. It will also be interesting to see whether Google transitions Google Assistant to Gemini Assistant given all the chatbot capabilities that Gemini offers combined with Google’s desire to keep building the Gemini brand.”

In addition, we could see a teaser for the next-gen Pixel Fold, and/or an update on the Wear OS smartwatch front, with some other surprises thrown in for good measure. Join us at kickoff on 14 May, and we’ll be sure to update you with all the latest news, as it happens.

• Read more: I think the Google Pixel 8a is the best value phone in 2024, and it’s not even close