Article · Privacy and on-device AI

How on-device Vision works, in plain language

By ClearWhirl EditorialPublished 6 min read

On-device AI means a trained model runs on your iPhone's own chip, so a photo is read on the phone and not sent anywhere to be understood. The model turns each picture into things software can use: what is in it, any text inside it, and a signature for comparing it with other photos.

On-device AI in one paragraph

A model is a file full of numbers, produced by training on a very large number of examples. Once training is finished, the model can be copied to a phone. When an app wants to understand a picture, it hands the picture to the model, the iPhone's processor runs the numbers, and the answer comes back to the app. No server is needed for that step. That is the whole idea, and everything else on this page is detail.

What the model actually sees

To a computer, a photo is a grid of colored dots, each one stored as numbers. The model does not "see" a receipt the way you do. It runs those numbers through many layers of arithmetic and produces an output: a set of likely labels, a block of recognized text, or a short list of numbers that summarizes how the picture looks. We are keeping this conceptual on purpose. The exact internals vary by model, and the details are not needed to use the result.

Recognizing what is in a photo

One kind of output is a label with a confidence: this looks like a document, a screenshot, a ticket, a QR code, a meme. The model has learned patterns that tend to go with each of these, such as layout, text blocks and edges. It is giving you a best guess, not a fact, and some labels are too broad to be useful on their own. "Outdoor" or "building" fit thousands of photos, so a cleaner that searches by such labels alone will not narrow anything down.

Reading text inside images

A different task is reading the words in a picture, often called OCR (optical character recognition). The model finds regions that look like text and converts them into characters. That is what lets you search a library for "boarding pass" or "payment receipt" and find pictures that never had those words in a filename or caption. It is an ordinary text search, applied to text that was read out of the image.

This works best on clear, printed, well-lit text. It struggles with handwriting, very small print, blur, glare and fancy lettering such as the script on a greeting card. A result that is missing does not mean the words are not there.

Comparing photos to find near-duplicates and a likely best shot

A third output is a compact numeric summary of how a picture looks, sometimes called a fingerprint or feature vector. Two photos of the same scene taken a second apart produce similar numbers. Software can compare these summaries and group pictures that are close. That is how a cleaner can show you a stack of near-identical shots.

Picking a "best" one is a further step. It can use hints such as sharpness, whether eyes are open or whether the subject is centered. It is a suggestion about technical quality. It cannot know which shot has the expression you love, which is why you stay in charge of the final choice. If you want to do this by hand, how to find and delete duplicate and similar photos shows the steps.

Where the model lives, and why that matters for privacy

Two things are easy to blur together, so keep them apart:

  • Training happens elsewhere, before the model reaches you, using large amounts of data. That is how the model learned what a receipt tends to look like.
  • Running the model happens on your phone, on your photos, when you use the app.

Running on the phone is what matters for privacy, because your pictures do not have to leave the device for the model to look at them. It does not by itself prove an app is private. An app could run a model locally and still upload other things, which is why you should read the privacy label and policy, as we describe in the privacy checklist for iPhone cleaners and in what "local" should mean.

What on-device models do badly

Be skeptical in these situations:

  • Odd lighting. Dark, backlit or colored light can fool a model.
  • Tiny or blurry text. Small print on a receipt may be read wrongly or not at all.
  • Handwriting and decorative lettering. Much harder than clean print.
  • Unusual documents. Layouts the model has not seen much of can be mislabeled.
  • Context. The model does not know that a blurry photo is the last one you took of someone.

For these reasons, results are suggestions. The sensible workflow is to let the model do the sorting and then look at each item yourself before anything is removed. A fast wrong answer is still wrong.

Where Apple's on-device model fits

Apple provides on-device image analysis in its Vision framework, which apps can use for tasks such as finding text in images. Separately, iPhones that support Apple Intelligence have an on-device language model. ClearWhirl uses that one, only on supported iPhones, to understand the words you type into Ask search. On other iPhones, ClearWhirl works without it. These are two different things, and having one does not mean you have the other. Our piece on Apple Intelligence and photo clean-up goes further.

What ClearWhirl does (and does not do)

ClearWhirl is coming soon to the App Store. Its analysis runs on the iPhone, and your photos are not uploaded. It needs no account, shows no ads, does no tracking and works in flight mode. You can read how the pieces fit together on the how it works page.

It also has limits. It does not learn from your library on a server, since nothing is sent anywhere. It does not understand who people are or what a moment means to you. Scanning uses the phone's processor and takes time, especially the first time through a large library. And it frees storage only: it does not speed up the iPhone, clear memory or clear other apps' caches. Everything it suggests is reviewed by you first, and iOS shows its own confirmation for deletions.

Frequently asked questions

Does on-device AI send my photos to the cloud?

No, not for the analysis itself. The model runs on the iPhone, so the photo is read there. An app can still do other things with a connection, so read its privacy label and policy to see what it collects.

Does it use battery or heat the phone while it works?

It can. On-device analysis uses the phone's processor, and a large first scan may take a while. We do not put a number on it, because it varies by iPhone, library size and what else the phone is doing.

Is on-device AI the same as Apple Intelligence?

No. On-device AI is a general idea: a model that runs on the phone. Apple Intelligence is one set of Apple's features, available only on supported iPhones. Many on-device image tasks do not need it.

Can on-device AI make mistakes with my photos?

Yes. It can mislabel a photo, miss small or blurry text and group pictures that are not really alike. That is why you should review suggestions before deleting anything.

Does it need the internet?

Not to analyze a photo. The model is already on the phone. Downloading an app or checking a purchase may need a connection, but reading your library does not.