AI Humanizer Comparison Study: What Happened to the Same Academic Texts

In our AI humanizer comparison study, every humanizer moved the same 48 academic texts toward a human writing style, and the tools differed most in what they kept. Some kept the meaning, citations and numbers close to the source. Others added invented numbers and names or new grammar errors, and those outputs were often the ones a detector labeled human.

TextPulse Research ran the AI humanizer comparison study in August 2026 and published it as a working paper on 29 August. TextPulse is one of the tools in it, so every tool got the same inputs and every measure is a public script. The full working paper has every table, figure and script, and all 432 rewritten texts are public.

What the AI humanizer comparison study tested

The study gave every humanizer the same 48 texts. Each one was the Results and Discussion section of an invented empirical study, written by an AI model and about 301 words long. Four model families wrote 12 texts each: OpenAI, Google, DeepSeek and Anthropic. The texts covered four disciplines and three citation styles, so each tool met numbers, author names and reference markers it had to keep.

Part of the design	What the study used
Source texts	48 AI-written Results and Discussion sections, 264 to 344 words each
Who wrote them	12 texts each from OpenAI, Google, DeepSeek and Anthropic models
Disciplines	Medicine and health, physical and computational sciences, social sciences, humanities
Citation styles	APA author-year, numeric brackets, no citations
Checked facts in the texts	404 technical terms, 887 numbers, 124 citations
How each tool ran	Pasted once into its web page at the default setting, output kept as returned
Dates	26 to 28 August 2026
Detectors	Turnitin, one document per tool, and GPTZero, each text on its own

A humanizer's job is to rewrite AI output, so the study started from AI output with a verified list of facts. Before any rewriting, each model listed the terms, numbers and citations it had used. Every listed item was compared with the text, and any item missing from it was dropped. That left a fixed list for each text, so the study could count exactly what each humanizer kept, changed or invented.

Each text went into each tool's web page once, at the tool's default setting, and the output was saved exactly as it came back. Nobody edited an output. Some tools processed fewer than 48 because free tiers limit how many texts a tool will take. A core set of 24 texts was the same for each tool, in the same order. TextPulse processed all 48.

How the study measured a humanized text

A detector score tells you how a text looked to one detector on one day. It says nothing about whether the rewrite still reports the same results. The study scored every output on the text itself first and on detectors second. For students or researchers, it's important to have a paper that still says what the data says, in prose that sounds like a person wrote it.

Measure	What it checks	Why it matters in a paper
Meaning	How close the rewrite is to the source, as a similarity score from 0 to 1	A lower score means the paragraph now says something else
Citations kept	Share of in-text citations still there with the same author and year	A changed year breaks the match with the reference list
Numbers kept	Share of the checked numbers still in the text	A lost sample size or p-value changes the result
Invented items	Numbers and names in the output that the source never had	An invented figure is a false statement about the study
Grammar errors added	New errors per 1,000 words, counted with LanguageTool	Errors cost marks and reviewer trust
Reading grade	Change in Flesch-Kincaid grade	AI text has a higher grade level than published human prose
Style position	Position of the text between typical human and typical AI writing	Stylometric detectors respond to these features
Length	Output length against the source; 0.70 to 1.30 times counts as kept	A tool that halves a text has dropped content

Style position comes from a scale TextPulse developed from 47 style features from 121,092 academic texts (described in our stylometric classification paper). Typical human academic text is at 0 and typical AI text is at 100. We call the middle half of human texts (-36 to 33) the human range.

Vocabulary was scored against a list of 1,057 words from our AI vocabulary paper. Of those, 581 are words AI models use more than people do, and 476 lean the other way. The study counted how many AI-leaning words each tool removed per 1,000 words.

What every humanizer did to the text

Every tool in the study moved its outputs toward the human end of the style scale. The untreated AI texts averaged 138, past the typical AI text at 100. After rewriting, some tools reached the human range and others stopped about halfway. Every tool also removed AI-leaning vocabulary and lowered the reading grade, so the direction of change was the same across the field.

Two kinds of features move a text along the scale most. The first pair is the mean word length and the proportion of long words. A rewrite from "help" to "facilitate" will make the text more AI-like because it increases mean word length and increases the proportion of long words. The second is varied sentence length, which pushes toward human. A paragraph where all sentences have 22 to 26 words is flat, but a paragraph that mixes a 9-word sentence with a 31-word one is varied.

The idea is to get to a lower reading grade. AI-written academic text has a higher Flesch-Kincaid grade than published human academic prose. So, if it goes down, that gets the writing more like the plain style that researchers write in. That's what stylometric detectors respond to. Those are also the changes that make a draft easier to read. The two goals line up as long as the content stays intact.

Our post on how humanizers change a sentence shows these edits one sentence at a time. The study adds the measurement: how far each change moved the text, and what it cost in meaning and facts.

Where the tools differed: meaning, citations and numbers

The tools differed in their retention rates. For example, across the study, between 29 and 100 percent of citations were retained. Between 0.06 and 4.44 invented items appeared per output. Between 0.65 and 4.91 new grammar errors appeared per 1,000 words. The invented items were sample sizes, percentages, confidence bounds, author names and institutions that the source never mentioned.

Names and units went missing in quieter ways. When a tool reworded a statistic, it often dropped the unit, so "Cohen's d" became an effect size and "Celsius" became degrees. Author names went missing along with their citations. Species, epoch and instrument names were replaced by a description. A reader who wants to check a result needs those names, so each of these losses counts.

Rewrite depth and faithfulness push in opposite directions. A tool that changes a third of the words keeps the meaning more easily. A tool that changes most of them has more chances to lose something. The share of words changed ranged from 35 to 68 percent. The result a researcher wants is a deep rewrite that still keeps the meaning. That combination was rare.

The study also looked for hidden characters: invisible characters, non-standard spaces and look-alike letters from the Cyrillic or Greek alphabets. Some humanizers add these to change a detector's result. No tool in this study added any, so no output failed the quality gate for that reason.

Why a detector result alone is a weak humanizer benchmark

A detector result on its own is an incomplete humanizer benchmark. In the AI humanizer comparison study, the outputs GPTZero labeled human most often came from tools that invented more numbers and names, dropped more citations or added more grammar errors. The easiest way to change a detector's estimate is to change what the text says. A rewrite like that fails the paper it was meant to help.

To measure both sides at once, the study set a quality gate. To clear it, an output needed a meaning score of at least 0.95 and at least 90 percent of its citations. It also needed no invented numbers or names, no hidden characters and a length within tolerance. A GPTZero label of human then counted only for outputs that cleared the gate. The gate was the same for every tool.

Each tool's output went into one document with one text per page. The study used Turnitin's one AI percentage for the document. Turnitin uses an asterisk instead of a number for less than 20 percent. The detectors' results are estimates from one detector version on the dates of the study. We wrote about reading your Turnitin AI score in our guide on what a Turnitin AI score means.

How TextPulse did in the humanizer comparison

TextPulse came first in the study's composite index (which gives equal weight to rewrite depth, meaning, style position, grammar and content integrity). TextPulse's scores were 0.811 for those five measures and 0.843 when the gated detector measure was added. It did well because it did not have any weak measures. It rewrote more than half of each text and kept the facts a paper depends on.

Measure	TextPulse result
Words changed	52 percent
Meaning kept (similarity to the source)	0.971, about 97 percent
Citations kept	99 percent
Numbers kept	87 percent
Named entities kept	95 percent
Grammar errors added	0.65 per 1,000 words, the fewest in the study
Invented items	0.48 per output; an invented name in one output in fifty
Reading grade	3.6 grades lower, the largest drop
AI-leaning words removed	39 per 1,000 words, the most in the study
Style position	29, inside the human range
Outputs within length tolerance	47 of 48
Outputs that cleared the quality gate	67 percent
Turnitin AI score, one document of all outputs	24 percent, the lowest in the study (tied)
GPTZero human label on outputs that cleared the gate	10.4 percent (5 of 48), the highest in the study

Two results matter most for academic use. Among tools that changed at least half the words, TextPulse kept the meaning best. Among tools with a meaning score of 0.97 or higher, it changed the most words. No other tool in the study was in both groups. A humanizer for academic writing needs that combination, since a light rewrite leaves the AI style in place and a deep rewrite that drops facts changes the findings.

GPTZero labeled 9 of 48 TextPulse outputs human, and 5 of those also cleared the quality gate, the highest gated result in the study. The Turnitin document of TextPulse outputs scored 24 percent AI, tied for the lowest score. The other documents scored from 28 to 74 percent. Detector results are estimates, and the study reports them next to the text measures for that reason.

What the study cannot tell you

The AI humanizer comparison study lists its own limits. The texts were written by AI models to a fixed prompt, and real drafts are longer and less uniform. Each tool ran once, at its default setting, between 26 and 28 August 2026. Vendors update their tools, so the results describe those dates and should be read as a snapshot.

Different tools have different numbers of texts as they're limited to a certain amount of free texts that the tool will process. This also results in larger confidence intervals for the tools with fewer texts. Turnitin generates one score per document and therefore does not have a spread of scores per text. GPTZero is one detector, one model version (2026-08-09-base).

TextPulse made the study and is one of the tools in it. Each tool was given the same inputs. By protocol, all measures are publicly available scripts, and all raw results are shared. This means that anyone can run the numbers again, or weight them differently. The source texts, all 432 outputs, the detector replies and the scripts are public.

How to use the findings on your own draft

The findings turn into a short check you can run after any humanizer, ours included. Compare the output with your draft for the things the study counted. On a single section it takes a few minutes.

1. Check every number against your draft: sample sizes, percentages, p-values and confidence intervals.
2. Check every in-text citation for the same author and year, then match it to your reference list.
3. Look for names that changed: instruments, species, datasets, units and author names.
4. Read for claims that got stronger, such as "may indicate" rewritten as "shows".
5. Compare the length. An output much shorter than the source has probably dropped content.
6. Read the text aloud to catch grammar errors the rewrite added.

Length is the quickest of these checks. In the study, an output between 0.70 and 1.30 times the source length counted as kept. TextPulse kept 47 of 48 outputs inside that band, at an average of 1.14 times the source, so its outputs ran slightly longer than the drafts.

Citations take the longest to check by hand. In the study, TextPulse kept 99 percent of them, and its changes were in presentation, such as a parenthetical citation turned into a narrative one. Our post on an AI humanizer that preserves citations explains what to look for in your own reference markers.

The free burstiness checker measures how much the sentence lengths in your draft vary. Varied sentence length is one of the features the study's scale weighs toward human writing. For a ranked guide with prices, see our guide to the best AI humanizer for academic writing. When you're ready to rewrite, the TextPulse AI humanizer is built to keep your numbers, names and citations, and reports an estimated Human Score for the result.