I built a face recognition system on FaceNet as my undergraduate thesis. The part that stayed with me wasn't getting it working — with a pretrained model and a few hundred lines that takes an afternoon — it was how completely the benchmark number failed to describe what the thing actually did when real people stood in front of it.
That gap is the subject here. Every model page quotes an accuracy figure above 99%. Almost nobody explains why yours will be worse, and the reasons are structural rather than a matter of picking a better model.
What the model actually gives you
FaceNet's contribution, in the 2015 CVPR paper, was to stop treating recognition as classification. Instead of training a network to output "which of these 500 people is this", it learns to map any face to a point in a fixed-dimensional space — an embedding — such that distance in that space means similarity. Two photos of you land close together; you and a stranger land far apart.
It's trained with triplet loss: an anchor image, a positive of the same person, a negative of someone else, and a penalty that pushes the positive closer than the negative by some margin. The elegance is that recognising someone the model has never seen becomes a distance computation rather than retraining.
The field has moved since. ArcFace and the angular-margin family that followed it replaced triplet loss with a margin applied directly in angular space, which removes the fiddly problem of mining useful triplets and produces cleaner separation. If you're starting today, start there. But the shape of the system — encode to a vector, compare by distance, threshold — is the same, and so is everything below.
1. There is no accuracy. There is an operating point.
This is the one that reframes everything else.
The model gives you a distance. You choose the number below which two faces count as the same person. That threshold is not a technical detail to tune once — it is the product decision, and it trades two errors against each other that cannot both be reduced:
- False match — a stranger is accepted as you. Tighten the threshold and this falls.
- False non-match — you are rejected as a stranger. Tighten the threshold and this rises.
Which one you'd rather have is a question about consequences, not accuracy. For unlocking a phone, a false match is a security breach and a false non-match is a minor annoyance, so you bias hard toward rejection. For an attendance system, a false non-match means a student marked absent who was present, and a false match means someone's friend signed them in — and honestly the second is more tolerable than standing at a door that won't let you in.
When a vendor says "99.8% accurate", they've chosen an operating point for you and not told you which. Ask for the false match rate and the false non-match rate at the threshold they're quoting, or the numbers mean nothing.
2. Searching a crowd is a different problem to checking one person
Verification is one-to-one: this person claims to be Anthony, does this face match Anthony's stored face? Identification is one-to-many: who is this, out of everyone we know? They sound like the same feature. They fail completely differently.
In a one-to-many search you run the comparison against every entry in the gallery, so every entry is another chance to be wrong. Suppose your false match rate is a respectable 1 in 100,000 per comparison, and your gallery has 10,000 people in it. Each search now averages about 0.1 false matches — roughly one search in ten returns somebody who isn't there.
Nothing about the model degraded. The arithmetic did. This is why a system that demonstrates beautifully on a team of 30 starts producing nonsense at 10,000, and why "it worked in the pilot" is not evidence about the rollout. If you're doing identification at scale, the per-comparison threshold has to be far tighter than intuition suggests, and that tightening costs you false non-matches on exactly the people the system is meant to recognise.
3. The benchmark is not your data
Labeled Faces in the Wild is the number everyone quotes, and it is effectively saturated — the top models sit around 99.8%, which mostly means the benchmark has stopped discriminating between good systems.
More importantly it isn't your input. LFW is built from news photographs: mostly frontal, reasonably lit, professionally shot, and skewed heavily toward the people who appear in western press coverage. Your camera is a webcam at a bad angle in a corridor with a window behind it. Nothing about 99.8% on LFW predicts what happens to that frame.
The corollary is that you cannot evaluate a face recognition system on public benchmarks. You have to build a test set from your own cameras, your own lighting, your own population, and measure both error rates on that. This is tedious and it is the only part of the process that tells you the truth.
4. The errors are not evenly distributed
NIST's FRVT Part 3 report on demographic effects measured this across a large number of commercial algorithms and found error rates varying substantially by demographic group — with false match rates in one-to-one comparison differing by orders of magnitude between groups on some algorithms.
Two things are true about this and both matter. The more recent NIST evaluations show the strongest algorithms have much smaller differentials than the 2019 picture — this is not a fixed property of the technology. And the algorithm you can actually deploy, on your hardware, at your budget, is frequently not one of the strongest.
So the practical instruction is: measure your error rates per group on your own test set, not just in aggregate. An aggregate number averages away exactly the failure you most need to find, and "it works fine" is a claim about the majority of your test set rather than about the system.
5. A photograph of a face is a face, unless you check
A face recognition model answers "whose face is this". It does not answer "is this a person". Hold up a phone displaying a photo and a naive system will match it happily, because it is, in the only sense the model understands, a correct face.
Defeating that is presentation attack detection — liveness — and it is a separate problem with separate models: depth, texture, micro-movement, challenge-response, infrared. If your system controls access to anything worth attacking, recognition without liveness is not a security control. It is a convenience feature that looks like one, which is worse than an obvious gap.
6. Enrollment quality sets the ceiling
The system's accuracy for a given person is capped by the reference image you stored for them. A bad enrollment photo — half-shadowed, low resolution, unusual expression — produces an embedding that sits slightly off from where that person's faces normally land, and every future comparison inherits that error. That one individual will be mysteriously hard to recognise forever, and no threshold tuning fixes it.
Which makes enrollment a UI problem more than a modelling one: check quality at capture time, reject and retake rather than storing something poor, and store multiple references per person where you can. It's unglamorous, it is not what anyone wants to work on, and it moves real-world accuracy more than swapping the model does.
7. This data is legally different
Not a footnote, and not something to leave to the end of the project.
A face embedding is biometric data. Under GDPR that is a special category with a higher bar for processing it at all. Illinois' Biometric Information Privacy Act has produced substantial damages awards and grants individuals a private right of action, which is unusual and is why so many settlements come out of it. Under the EU AI Act, building facial recognition databases by untargeted scraping of the internet or CCTV footage is a prohibited practice — that provision has been in force since 2 February 2025, and the wider timetable has been moving, so check the current implementation timeline rather than trusting a date in any article, including this one.
None of that is legal advice and I'm not qualified to give it. The engineering point is narrower and I'm confident of it: store embeddings rather than photographs where the design allows, be specific about retention and deletion before you collect anything, and treat "can we get the same outcome without biometrics" as a real question. On the attendance system I worked on separately, a card and a reader answered the actual requirement without any of this, and that is often the right answer.
The short version
Getting a face recognition demo working is genuinely easy now, and that is exactly what makes it deceptive — the difficulty was never in the model. It's in choosing an operating point you can defend, understanding that searching a gallery multiplies your error, measuring on your own data broken down by group, checking you're looking at a person rather than a picture, and being deliberate about collecting a category of data that's expensive to hold.
The demo takes an afternoon. Everything above is the project.