Token Probability and Log-Likelihood in AI Detection

If you ask a detector how it came up with that score, most of its user interfaces will give you the same response: a paragraph with some words shaded darker than others. The shading isn't a guess. It comes from a number attached to every token in the passage, calculated before the detector ever touches perplexity, burstiness, or a final percentage.

It is a token probability, and every token probability ai detection is based on that, from the ground up. Any number of metrics a detector will report (including the two we cover elsewhere on this site) is just arithmetic done to a series of these numbers. The layer beneath the layer most explanations stop at is where you have to think about what a token actually is, what the model's distribution over it means, and why the arithmetic runs in log space rather than raw probability. Most explanations stop there.

None of it needs to stay opaque. A token, a probability distribution and a logarithm are ordinary tools, not proprietary secrets, and the same three ideas explain how AI detectors actually work, from raw text to a single number on a report.

Token Probability AI Detection: From Distribution to Score

Token probability ai detection works in three stages. A language model assigns a probability to every possible next token given what came before. Those per-token probabilities are converted to logarithms and summed across the passage, which sidesteps a numerical problem raw multiplication runs into almost immediately. The resulting sum, averaged and rescaled, is what eventually surfaces as a perplexity figure or feeds a classifier's decision. Each stage is a specific, checkable piece of arithmetic, not a black box.

What a Token Actually Is

A token is not a word. Modern language models split text into subword pieces with an algorithm such as byte-pair encoding, so a common word like 'the' is usually one token, a less common word like 'perplexity' splits into two or three pieces, and an unfamiliar name can fragment down to individual characters. A passage of ordinary English reliably tokenizes into more pieces than it has words, which is worth knowing before trusting any word count a tool reports back to you.

The model never sees words the way a reader does. It sees a sequence of integers, each one an index into a fixed vocabulary the model was trained with. Everything token probability ai detection does from this point on operates on that integer sequence, not on the original string of letters, which is part of why a detector's internal workings can feel disconnected from how a person actually reads a sentence.

The Model's Distribution Over the Next Token

At every position in a sequence, an autoregressive language model does not output one prediction. It outputs a full probability distribution across its entire vocabulary, tens of thousands of numbers that sum to one, ranking every possible next token from most to least likely given everything that came before. Feed a model the phrase 'the results of the' and it distributes most of that probability mass across a handful of plausible nouns, 'study,' 'experiment,' 'analysis,' while assigning a vanishingly small share to something like 'marmalade.'

The token that actually appears next in a real document lands somewhere in that ranking, and its assigned probability is the raw material everything else is built from. A model producing its own text can lean on this distribution directly, repeatedly picking from near the top. A detector reading someone else's finished text can only ask, after the fact, how much probability the model would have assigned to the choice that was actually made.

The probability of a whole sequence is the product of each token's probability given everything before it; if we chain that per-token question across an entire document using the standard rule for joint probability, the final result is a single number describing how expected the entire passage was. This product is where the arithmetic starts to break down, which is the reason the next section exists.

Why Log-Likelihood Replaces Raw Probability

Multiplying probabilities together is mathematically correct and practically useless past a few dozen tokens. Every individual probability is a fraction less than one, so a running product shrinks every time another token is multiplied in, and it shrinks fast. A few hundred tokens in, a raw product can fall so far below the smallest number a computer can represent that it rounds down to exactly zero, a failure mode called underflow.

Once that happens, the number carries no information at all. A document that was merely unlikely and a document that was wildly, chaotically unlikely both collapse to the identical zero, with no way to tell them apart afterward. Logarithms fix this because the logarithm of a product equals the sum of the logarithms, an identity from basic algebra. Summing a string of ordinary negative numbers does not underflow the way multiplying a string of small fractions does, and it is cheaper to compute besides.

Passage length (illustrative)	Raw probability, a running product	Sum of log-probabilities
12 tokens at an illustrative 0.1 each	1 x 10 to the power of -12, still representable	roughly -27.6
350 tokens at an illustrative 0.1 each	1 x 10 to the power of -350, underflows to exactly 0	roughly -805.9

This is a simplified illustration. Real per-token probabilities vary enormously rather than sitting at a flat 0.1, but the direction of the problem is exactly right. Once a raw product underflows to zero, the comparison a detector actually needs, whether this document was more or less expected than that one, becomes impossible. The sum of log-probabilities never fails that way. It stays a precise, ordinary, comparable number no matter how long the passage runs, which is the actual reason detectability gets measured in log space rather than in raw probability space.

From Per-Token Scores to a Detection Score

Summing log-probabilities across a passage produces the document's total log-likelihood under the reference model. Dividing by the number of tokens removes the effect of length and leaves the average log-likelihood per token, a number finally comparable between a five-hundred-word essay and a five-thousand-word thesis chapter. Negate that average and exponentiate it, and what comes out is the perplexity score, a metric this cluster covers in full in a separate look at what perplexity actually measures.

The same per-sentence figures, tracked for variation across a document rather than flattened into one average, are what burstiness is built from, a related but separate signal from perplexity. Both start from the identical per-token numbers described above. They just do different arithmetic on top of them.

A simple average is not the only way to use this raw material, and research has moved well past it. DetectGPT, published at ICML 2023 by Mitchell, Lee, Khazatsky, Manning and Finn, works from a different property of the same probability function: text sampled from a language model tends to sit in a region where small rewordings make the log-probability drop rather than rise, a pattern the paper calls negative curvature.

Rather than training a classifier or using a watermark, this method perturbs a passage, producing little variations in how it's reworded, and then rescores each variation with the same model. The paper compares this to the best zero-shot baseline and finds that, on text generated by a 20-billion-parameter model, this method achieves 0.95 AUROC while the best zero-shot baseline achieves 0.81 AUROC.

The Same Distribution, Used to Watermark Instead of Detect

Everything so far has treated the probability distribution as something read after the fact. It can also be touched at the moment of generation, which flips the problem around entirely. A 2023 method from Kirchenbauer, Geiping, Wen, Katz, Miers and Goldstein selects a randomized shortlist of allowed tokens before each word is produced, based on a hash of what came before, and softly steers sampling toward that shortlist. The bias is invisible to a reader and later recoverable with a statistical test on a short span of text, without needing access to the original model.

Google DeepMind's SynthID puts a version of this idea into production: it adjusts next-token probability scores at generation time and is live today in the Gemini app and web experience. OpenAI built a comparable system and has not shipped it. Reporting attributes the decision partly to a survey in which close to 30 percent of ChatGPT users said watermarking would make them use the product less, alongside concerns about how well a watermark would survive paraphrasing.

A detector's report shows a shaded word or a single percentage and stops there, without ever showing the distribution any of it came from. TextPulse's free perplexity checker runs a simplified version of the same per-token scoring against your own draft, which is a more direct way to see where a passage sits than reading a verdict secondhand.

Two neighbouring pages pick this up. Whether detectors still rely on burstiness has moved since 2023, and how GPTZero turns these same probabilities into a score is described on its own page.

It sits alongside the rest of TextPulse's free tools, so the same draft can be checked for rhythm and structure as well as word-level predictability, without opening a dozen separate tabs to do it.

Related research: whether an AI model can be told to write like a person is tested in Do AI Models Speak Human?, a TextPulse Research working paper in which four flagship models were given a detailed style brief and a human example, then scored on a stylometric spectrum and on GPTZero against real journal prose.