How Do AI Detectors Work? Perplexity, Burstiness and Token Probability

Your university has started running submissions through an AI detector, and a report has come back with a number you were not expecting. Before you try to dispute it, it helps to know what that number means. How do AI detectors work? They do not read machine writing the way a person does. They measure how predictable your text is to a language model, then turn that predictability into a score. Nothing in that process looks at what you mean, at your argument, or at whether you actually wrote the thing.

I have spent years building tools that pull apart text statistically, and the single most useful thing to understand about detection is this: it is a measurement of typicality, not of origin. Once that clicks, both the technology and its failures make a lot more sense.

What a detector is actually measuring

A detector does not know who wrote anything. What it has is a language model and a scoring function. The model reads your text from left to right, and at each point it produces a probability distribution over what word should come next. Give it the words "the results of the" and it will rank "study" highest, "experiment" a little lower, and "marmalade" somewhere near zero.

So the detector asks one question of your document: how often did this text choose the word the model expected? Writing that keeps landing on high-probability words looks machine-made, because that is exactly what a text generator does. The model samples from the top of its own distribution, over and over, for thousands of tokens.

This is why the whole category is on shakier ground than the marketing suggests. A detector is not finding a watermark or a fingerprint. It is noticing that your prose is statistically unsurprising, and unsurprising prose has many causes besides a chatbot.

Perplexity: how surprised the model is

The headline measure is called perplexity, which is easier than it sounds. Take the probability the model assigned to each word that actually appeared, average the logarithms of those probabilities, and exponentiate the result. What comes out is a single number describing how surprised the model was by your document. You can check the perplexity of your own draft rather than guess at it.

Low perplexity means the text went where the model expected. High perplexity means it kept taking turns the model didn't expect. Human writing tends to sit higher, because people reach for the odd specific noun, the unusual construction, the sentence that starts somewhere unexpected. Generated text tends to sit lower, because the decoding process is built to avoid exactly those moves.

Compare two ways of opening a methods section. "The results of the study were statistically significant" is a sequence almost any model would predict with high confidence, because it carries no unexpected information. "Two of the three cohorts drifted before week six, which we had not planned for" is far less predictable, because it carries specific, awkward information that could not be guessed from the rest of the paper. The second costs the detector real surprise, and that surprise is what registers as human.

Burstiness: the variance, not the average

Perplexity on its own is not enough, because a document can average out to a perfectly reasonable number while being suspiciously uniform underneath. What matters is the variation across your text, and that is what burstiness measures: how much sentence length and sentence-level perplexity swing as the writing moves along. A burstiness checker will show you that spread directly.

We speed up when we're confident and slow down when we're hedging, and that rhythm survives into the prose. People write in bursts. A paragraph might open with a short declarative sentence of eight words, run into a thirty-four word sentence that carries three clauses and a parenthetical, then drop back to something brisk.

Model output does not do this reliably. Sentences cluster near the mean length. Paragraphs come in neat bundles. Each section begins by restating its own heading. It reads smoothly and scores badly, because low variance across sentences is one of the strongest signals a detector has. It is not any single sentence that gives it away. It is the uniformity.

Signal	What it measures	Why machine text scores low
Perplexity	How surprised the model is by your word choices, averaged over the document	Decoding samples from high-probability tokens by design
Burstiness	How much sentence length and predictability vary across the text	Output clusters near the mean instead of swinging
Token probability	The per-word likelihood the other two are computed from	Long runs of individually unremarkable choices

Token probability and where the score comes from

Both of those numbers are built from a third one: the per-token log-likelihood, the raw material everything else is calculated from. Some tools show it to you directly, highlighting the stretches where your text was most predictable. Those highlights are not accusations about specific sentences, whatever the interface implies. They are the parts that contributed most to a document-level number.

Research approaches have moved past simple thresholds. DetectGPT and its descendants look at curvature: they perturb your text slightly and check whether the likelihood drops off in the pattern generated text tends to show. Others compare the same passage under two different models and use the disagreement as a signal.

One consequence matters for anyone being assessed. Because the metrics are computed against a particular reference model, a score is always relative to that model. Two detectors can read the same paragraph and disagree completely, and neither is malfunctioning. They are answering the same question with different reference points.

The second consequence is noted less often. A model can only find a passage predictable if that passage resembles what it was trained on, so detection quality degrades as the gap widens between the reference model and whatever produced the text. Newer generators, unusual disciplines and languages other than English all open that gap. It is part of why accuracy measured in a controlled benchmark rarely survives contact with a real pile of submissions.

What commercial detectors add on top

The tools institutions actually buy do not run textbook perplexity. Turnitin, GPTZero, Originality and the rest train supervised classifiers on large labelled sets of human and machine text, using statistical features alongside stylistic ones. What the classifier learns is whatever separates those two piles in its training data, which is a narrower and more historical thing than "AI writing" in general.

Training dependency is the quiet problem. A classifier trained primarily on one generation of model output has to generalize to newer models, to unfamiliar disciplines, and to writers whose English doesn't resemble its training distribution. Vendors publish accuracy figures from their own evaluations, and independent benchmarks routinely land well below those numbers on the same tools.

Reading a score without over-reading it

A detector report usually arrives as a percentage, and that percentage is routinely misread. It is not the proportion of your document written by a machine. Depending on the tool it is a classifier confidence, or the share of sentences that crossed some internal threshold, and vendors are often vague about which. Two reports both showing sixty percent can mean quite different things.

It is also worth knowing what moves a score for reasons that have nothing to do with who wrote it. Short documents are noisier, because there is less text for the averages to settle on, and a five hundred word essay can swing a long way on the strength of two or three unusual sentences. Quoted material counts too unless the tool strips it out, so a literature review packed with block quotes inherits the predictability of whatever it quotes. A technical passage laden with standard terminology behaves the same way.

If you are on the receiving end of a flag, the useful response is evidence rather than an argument about the metric. Version history, drafts, notes and search records all speak to process, and process is what a misconduct panel can actually evaluate. There is no threshold anyone can point to that separates a careful writer from a model.

Why detectors flag writing that no model produced

False positives are not a rare malfunction. They follow directly from measuring typicality. Any writing that is genuinely predictable will score as predictable, regardless of who produced it.

Non-native English writers are affected most. Stanford researchers found that detectors misclassified a large share of essays by non-native speakers as machine-generated, while classifying native-speaker essays correctly. The reason is mechanical rather than malicious: writers working in a second language tend to rely on well-worn constructions and a more common vocabulary. That is exactly the profile of low perplexity, which is why ESL writers get flagged for writing carefully.

Academic conventions cause the same problem. A methods section is supposed to be formulaic. Legal writing, technical documentation and clinical reporting all reward the standard phrasing over the inventive one. Heavy editing compounds it further, since polishing a draft usually means removing the irregularities that carried your statistical signature.

Institutions have noticed. Vanderbilt disabled Turnitin's AI detection feature rather than act on scores it could not verify, and other universities have restricted how the numbers may be used in misconduct proceedings. Treat a detector score as one weak piece of evidence, not a verdict.

What this means for your own writing

The practical advice that follows from the mechanics is not exotic, and none of it involves gaming anything. Vary your sentence length deliberately, because uniformity is what registers. Keep the specific detail: the actual figure, the actual limitation, the thing that went wrong in week six. Generic competence is what scores as machine-made, and specificity is both better writing and a stronger signal.

Keep your own idiom too. If you have a way of putting things, leave it in. The instinct to sand a draft down until it sounds like every other paper in the field is the instinct that lowers your perplexity.

One thing to avoid: some tools deliberately introduce grammatical errors to raise perplexity. It works on the metric and it is a terrible idea for anything graded or peer-reviewed, since you are exchanging a suspicion for a certainty. The goal is writing that reads as yours, not writing that has been damaged on purpose. That is the whole difference between rewriting for a human reader and breaking text to fool a scorer.

If you want to see these numbers for your own draft rather than take a black box at its word, the underlying measurements are not secret. The free text analysis tools will show you where your prose sits, and you can decide for yourself whether the flat stretches are a stylistic problem worth fixing.