Enhanced visual search on the iPhone improves the Photos app and protects your privacy.

The Apple app uses several features to help you search for images in your library and learn more about the content being displayed. One of these features is Enhanced Visual Search. Here's how it works, and how Apple protects your privacy when you use it.

Visual Search vs. Enhanced Visual Search

An important difference to note is how Apple’s "Visual Search" feature differs from its enhanced visual search function. Visual Search was introduced with iOS 15 and allows users to identify objects, landmarks, plants, and much more in the Photos app.

For example, you can perform a screen rotation in the Photos app to find out which breed of dog is depicted in the image. It can even recognize things like cleaning instructions on your clothes and the meaning of random symbols on your car’s instrument panel.

Enhanced Visual Search exists independently of visual search. While visual search helps you identify details on a photo you are already looking at, the enhanced visual search assists you in searching for a landmark or place by finding all photos in your library. The function works even if these photos do not have geolocation data.

For example, you can search your library for "Golden Gate Bridge" and see relevant images from your library. The function also works when the landmark in the background of an image is blurred or unclear.

How does Enhanced Visual Search protect your privacy?

Last week, there was a debate about whether the enhanced visual search sends your location information to Apple to find these places and interests. In the Settings app on Apple, it says: "Allow this device, which compares images with those in its database, to locate landmarks." Apple claims that this helps improve the accuracy of their results.

However, Apple does not send any personal information or photos to third parties for analysis. Instead, they use advanced algorithms and encryption techniques to protect user privacy.

Apple uses differential privacy and an OHTTP relay operated by a third party to hide device IP addresses before reaching Apple's servers. They also send "fake queries" along with their real queries so that the server cannot determine which ones are real. Additionally, the queries are routed through an anonymous network to ensure that the server can connect to multiple requests from the same client.

Apple claims that this process ensures that they do not learn any personal information about the images in your library.

You can disable the enhanced visual search by going to the Settings app, tapping on "Apps," and then selecting "Photos." Scroll down and there is a toggle switch for Enhanced Visual Search. Apple says that this switch should mainly be used in areas with low data traffic.

For more information about advanced visual search:

• Mixture of Machine Learning and Homomorphic Encryption in the Apple ecosystem
• Images and Privacy
• OHTTP Relay
• Differential Privacy

Follow Chance: Themes, Bluesky, Instagram, and Mastodon.