Open studies on AI generated text and how machines judge it.
We study the questions our users face on a daily basis. How AI detectors behave, how human and AI writing styles differ linguistically, and what happens when the two are fused. Every study comes with a publicly available corpus, score data, and analysis code, so extended work is encouraged.
Studies that measure the output of more than one AI language model consistently find that the models write differently, in the same manner that human authors have different writing styles. We coin the term modelometry for the measurement and attribution of the inter-model writing style of AI systems, and modelolect for the style itself, formed on the pattern of idiolect and sociolect. The study asks whether the modelolect of a flagship model family is strong enough for a classifier that reads only surface stylometric features to classify the AI family that originally wrote a text. We test seven families (GPT, Claude, Gemini, DeepSeek, Grok, Llama and Qwen) against human text, in two registers (formal academic text, and informal chat text) and with three tiers of features. Every AI text used is from corpora generated for our prior studies or from open datasets. On academic text, a gradient boosting classifier over 47 interpretable stylometric features attributes 7 classes (human and six families) at 73.7% accuracy against a 14.3% chance rate, and 6 classes on a second corpus at 79.2%. On chat responses from the LMArena preference dataset, the same 47 features attribute all seven families at 63.9%, and a character n-gram model with model names masked reaches 87.0%, so the small interpretable feature set accounts for about three quarters of the attributable signal. The confusion structure is also informative, since DeepSeek is rarely confused with GPT (3% in the corpus where DeepSeek is most identifiable), so the writing-style evidence does not support the claim that 'DeepSeek behaves as a distillation of GPT'. The largest confusion in the chat register is between Qwen and GPT, at about 16%. Claude is the most identifiable family in both registers. A classifier trained on academic AI generated rewrites achieves only 18.6% on chat text from the same families, so a modelolect is specific to *register*, and to *model version*, and attribution requires training data from the register it will judge. As a byproduct we release inter-family vocabulary lexicons built with the log-odds method of our study on AI vocabulary. As a final experiment, after one TextPulse humanization, *p(human)* under the family classifier increases for 87 to 98% of the texts, a majority of the humanized texts classify as human, and the source family is recovered for at most 3% of them. All features, statistics, scores and code are made publicly available for future work.Studies that measure the output of more than one AI language model consistently find that the models write differently, in the same manner that human authors have different writing styles. We coin the term modelometry for the measurement and attribution of the inter-model writing style of AI systems, and modelolect for the style itself, formed…
Users who want an AI model to write like a human usually assume that the machine register is a default that an instruction can lift. We tested that assumption in this study. Ten passages of human-written published academic prose gave ten topics. Four flagship models (GPT-5.6 Sol, Claude Opus 5, Gemini 3.7 Flash and DeepSeek V4 Pro) wrote about 300 words on each topic under three prompts, namely, (1) a plain request; (2) a detailed brief on the properties that separate human academic prose from assistant prose (uneven sentence length, plain words, no triads, no connective openers, hedging, asides); and (3) the same brief with a human passage as an example. Every text was placed on the frozen 47-feature stylometric human-to-AI spectrum from our earlier work and scored by GPTZero, along with the ten human passages. The pooled spectrum position fell from 148 under the plain prompt to 30 under the detailed one (human passages 10), 14 of the 40 instruction-loaded texts were inside the middle half of the human anchors, and Claude Opus 5 was positioned past the human median. The models overshot every property the prompt named, writing more unevenly and at a lower reading grade than the journals, and left the one property it did not name, vocabulary range. On GPTZero, all 120 AI texts, under every prompt and from every model, received a document AI probability of 1.00, while all ten human passages received 0.00. The human example text in the third prompt format added no help. One pass through the TextPulse humanizer positioned the same texts a mean of 47 points toward the human end, while preserving 97 percent of the meaning, and passed GPTZero in cases where the AI rewritten texts did not. Supervised fine-tuning and RLHF heavily limit the distribution a model samples from, an instruction is a condition on that limited distribution rather than a replacement for it, and a probability-based detector relies more on the underlying process and distribution, and less on the surface-level stylometric properties. The results obtained demonstrate that prompting cannot undo preference tuning. All texts, prompts, detector replies, scores and code are made publicly available for future work.Users who want an AI model to write like a human usually assume that the machine register is a default that an instruction can lift. We tested that assumption in this study. Ten passages of human-written published academic prose gave ten topics. Four flagship models (GPT-5.6 Sol, Claude Opus 5, Gemini 3.7 Flash and DeepSeek…
AI text humanizers rewrite machine-generated text so that AI detectors classify it as human. Humanizer vendors advertise pass rates, but no published study measures what the rewriting does to the output text itself. We passed 48 AI-generated academic texts (about 300 words each, four disciplines, three citation styles, four AI model generator families) through eleven humanizers under one setting per tool, and scored 432 outputs on a fixed set of metrics, namely, position on a stylometric human-to-AI spectrum; semantic faithfulness to the source; retention of technical terms, numbers and citations; fabricated numbers and named entities; Flesch-Kincaid grade shift; grammar errors introduced; length change; rewrite depth; hidden-character injection; and vocabulary shift against a 1,057-word AI vocabulary lexicon generated from our prior study. Detection was measured with Turnitin and GPTZero, the gold standard university-grade detectors. Every humanizer tool pushed the text toward the human end of the spectrum, reduced AI-leaning vocabulary and lowered the reading grade, and the tools differed most in what they preserved. TextPulse rewrote 52 percent of the words while keeping 97 percent of the document meaning, 99 percent of citations and 87 percent of numbers, introduced 0.65 grammar errors per 1,000 words, fabricated almost no named entities, and removed more AI-leaning vocabulary than any tool that kept the text intact. The tools with the highest raw GPTZero pass rates had the weakest content: Walter Writes passed 60 percent of its outputs but also fabricated 4.4 numbers or names per each (~300 word) text, and Phrasly passed 38 percent but introduced 3.6 grammar errors per 1,000 words. On a strict experiment, when a detector pass is counted only for outputs that also preserve meaning, citations and facts, TextPulse has the highest rate at 10.4 percent (95 percent CI 2.1 to 18.8), Phrasly and GPTinf have 2.1 percent, and eight tools have none. TextPulse is also first on an equal-weight composite of rewrite depth, faithfulness, spectrum position, grammar cleanliness and content integrity, with or without the detector component. On Turnitin, a document holding only TextPulse's outputs scored 24 percent AI, the lowest score in the field together with StealthWriter; the other tools scored from 28 to 74 percent. Detector pass rate on its own is an incomplete measure of a humanizer, since the easiest approach to pass is to change what the text says, or even intentionally inject grammar errors. Across all experiments, all of the humanizer tools had the same treatment to allow for an even level playing field. Every metric was scripted, and all humanization outputs and code are made publicly available for easy replication of the same experiments in future work.AI text humanizers rewrite machine-generated text so that AI detectors classify it as human. Humanizer vendors advertise pass rates, but no published study measures what the rewriting does to the output text itself. We passed 48 AI-generated academic texts (about 300 words each, four disciplines, three citation styles, four AI model generator families) through eleven…
A widely cited study found that perplexity-based AI text detectors flag the majority of essays by non-native English writers as AI-generated while sparing native writers, and the finding has impacted the debate on AI detection in education. This study measures whether the bias is true for a transparent stylometric classifier. We score all 5,600 essays of the ICNALE corpus, written under controlled conditions by college students from ten Asian countries and regions at known proficiency levels and by English native speakers, with the human-versus-AI classifier of our prior studies, trained on 50,701 academic texts and rewrites and never on any essay. Every essay predates modern AI writing tools, so every flag is a false positive. The logistic regression classifier shows no bias against non-native writers. It flags 2.7 percent of learner essays and 3.3 percent of native essays at the default threshold, statistically indistinguishable rates, and both far below the 10.3 percent it flags on full-length human academic texts and the 24.3 percent on length-matched academic excerpts. The proficiency gradient runs opposite to the published bias, as the most proficient learners are flagged the most, namely, 5.0 percent at B2 against 1.9 percent at A2, since proficient writing drifts toward the formal register the classifier associates with an AI writing style. Learner limitations, namely, simple everyday vocabulary, repetitive word choice, and uneven sentence lengths, are the opposite of the AI signature under a stylometric analysis, while a perplexity (surprisal) analysis reads the same limitations as machine-like predictability. The protection is not intrinsic to the feature set, however. A gradient boosting model trained on identical features and data flags learners at 12.5 percent against 3.8 percent for natives, a threefold gap with the least proficient flagged most. Whether AI detection discriminates against non-native writers is a property of the detector itself, not of the human writing style. Per-essay scores and all analysis code are made publicly available.A widely cited study found that perplexity-based AI text detectors flag the majority of essays by non-native English writers as AI-generated while sparing native writers, and the finding has impacted the debate on AI detection in education. This study measures whether the bias is true for a transparent stylometric classifier. We score all 5,600 essays…
Real documents are often partly AI-processed, with a few sections AI-rewritten and the rest left as human-written. This study measures how a stylometric human-versus-AI binary responds as the AI-rewritten share of a document increases from 0 to 100 percent. From 25,561 content-aligned pairs of human-written academic texts and their AI rewrites, we splice documents in which a controlled fraction of the words, in increments of 10 percent, has been rewritten by one of eight AI model configurations, and score all 382,993 resulting documents with a classifier trained only on fully human and fully AI-rewritten texts, under grouped cross-validation throughout. The classifier reads the AI share as a proportion on a continuum, rather than detecting its presence and categorizing it into a discrete class of AI or Human. The mean score increases almost linearly from 0.17 to 0.83, and the share of documents flagged at the default threshold crosses 50 percent almost exactly at half AI content. Scattering the AI sentences through the document instead of concentrating them in one block leaves the response unchanged. Small AI shares are close to invisible. A document with 10 percent of its words AI-rewritten is separated from fully human documents with an AUC of 0.57, barely above chance, and under a strict one percent false positive budget only 15 percent of half-rewritten documents are flagged. Every feature mixes near linearly with the AI share except sentence-length variation, which is above the value linear mixing predicts, since the style change at the splice point itself adds variation, and 36 percent of half-rewritten documents show more sentence-length variation than the original human text. Distinguishing half-rewritten documents from pure ones as a class reaches an AUC of only 0.59 for a linear model and 0.76 for gradient boosting, so partial rewriting is detectable mainly as a lower score, not as a recognizable discrete category. Per-document scores and all analysis code are made publicly available.Real documents are often partly AI-processed, with a few sections AI-rewritten and the rest left as human-written. This study measures how a stylometric human-versus-AI binary responds as the AI-rewritten share of a document increases from 0 to 100 percent. From 25,561 content-aligned pairs of human-written academic texts and their AI rewrites, we splice documents in…
Our previous study classified human-written and AI-rewritten academic texts from interpretable stylometric features with an AUC of 0.936 on texts with a median length near 290 words. This study measures how that reliability depends on text length. From 25,561 text pairs in which both the human text and its AI rewrite contain at least 300 words, we cut every text to 25, 50, 100, 150, 200, 250, and 300 words, re-extract 45 stylometric features at each length, and retrain the classifier at each length under grouped five-fold cross-validation. The composition of the sample is identical at every length, so the results isolate text length from every other property of the texts. Classification reliability increases monotonically with length. The AUC of a logistic regression increases from 0.817 at 25 words to 0.940 at 300 words and 0.950 on the untruncated full texts, and most of the increase is at the 150-word length. Reliable individual verdicts degrade faster than the AUC suggests. Identifying 95 percent of AI rewrites requires falsely flagging 34 percent of human texts at 300 words, but 68 percent at 25 words, and a strict one percent false positive budget identifies only 17 percent of AI rewrites at 25 words. A classifier trained on full-length texts and applied to short texts at a fixed threshold flags 70 percent of 25-word human texts as AI while its detection of AI rewrites hardly changes, since short texts of any authorship have mechanically high lexical diversity, which the classifier reads as an AI characteristic. Word-level features preserve most of their distinguishing power at short lengths, and sentence-level variation carries almost no signal under 50 words, where texts average fewer than three sentences. All per-text scores and analysis code are made publicly available.Our previous study classified human-written and AI-rewritten academic texts from interpretable stylometric features with an AUC of 0.936 on texts with a median length near 290 words. This study measures how that reliability depends on text length. From 25,561 text pairs in which both the human text and its AI rewrite contain at least 300…
This study treats the separation of human-written and AI-rewritten academic text as a plain text classification task. The data are 60,306 human-written academic texts and 60,786 AI rewrites of those texts, produced by eight AI model configurations across six model families, each text described by 47 interpretable stylometric features such as word length, passive voice, lexical diversity, punctuation rates, and sentence openers. A logistic regression on these features separates the two classes with an AUC of 0.936 and an accuracy of 86.4 percent under grouped five-fold cross-validation, and a gradient boosting model increases this to an AUC of 0.961 and an accuracy of 89.5 percent. Most of the distinction between human and AI text is therefore carried by simple, countable linguistic properties of the writing. The errors have a clear structure. Identifying 99 percent of AI rewrites requires falsely flagging 68 percent of human texts, and keeping false flags at 1 percent detects only 47 percent of the rewrites. When 10 percent of a text population is AI, fewer than half of the flagged texts at the default threshold are actually AI. The misclassified human texts are the most formal ones, with longer words, more passive voice, and higher lexical diversity, and the false positive rate varies almost four times across academic disciplines. Applied without modification to an independent corpus of fully AI generated and hybrid texts, the classifier identifies all 30 generated texts, passes 26 of 30 human texts, and correlates at 0.70 with the mean verdict of nine commercial detectors. Per-text classifier scores and all analysis code are made publicly available.This study treats the separation of human-written and AI-rewritten academic text as a plain text classification task. The data are 60,306 human-written academic texts and 60,786 AI rewrites of those texts, produced by eight AI model configurations across six model families, each text described by 47 interpretable stylometric features such as word length, passive voice…
When a language model rewrites a human text, it changes more than the words. This study measures what occurs to the rest of the linguistic style. We compared 60,786 human-written academic texts with AI rewrites of the same texts produced by eight AI models across six model families, and computed 49 stylometric features for every text, namely, punctuation rates, passive voice, nominalization, first person, contractions, sentence openers, lexical diversity, word length, and a set of hedge and connective phrases. The paired design controls topic and content, so every change is caused by the model itself. The shifts are consistent. AI rewrites use longer words (Cohen's d = 1.57 for words of seven or more letters), more nominalizations (d = 1.12), higher lexical diversity (d = 1.31), more passive voice, and more commas, while first person, contractions, questions, and repeated sentence openers are mostly nonexistent. Each model has its own version of this profile. A multinomial logistic regression reading only the 49 features identifies which of six models produced a rewrite with 64.3 percent accuracy against a 16.7 percent chance rate, and a gradient boosting model raises this only to 66.4 percent, which shows the fingerprint is carried almost entirely by simple, interpretable features. DeepSeek is the most identifiable model and OpenAI is the least. Qwen changes the style of the human source text least and DeepSeek changes it most. Per-text feature data and all analysis code are made publicly available.When a language model rewrites a human text, it changes more than the words. This study measures what occurs to the rest of the linguistic style. We compared 60,786 human-written academic texts with AI rewrites of the same texts produced by eight AI models across six model families, and computed 49 stylometric features for every…
Burstiness, the uneven rhythm of sentence lengths in a text, is one of the most explicit differences between human and AI writing, and among the least precisely measured. This study measures it at scale. We compared 60,779 human-written academic texts with rewrites of the same texts generated by eight AI model configurations across six model families, so that every burstiness change is caused by the model rather than by topic or content. Human academic writing in this corpus has a mean coefficient of variation of 0.449; the AI rewrites average 0.376, and 79.3 percent of all AI rewrites are flatter than their human written source. The flattening is universal across models but varies eight times in strength. Models trim long sentences and remove the short, direct sentences human writers use, converging on a uniform spectrum of medium-length sentences that reads monotonous and stiff. A threshold that catches 62 percent of rewrites also flags 39 percent of genuine human texts, and flags science authors most. Burstiness is a strong population-level signal, but an unsafe individual-level verdict. Per-text sentence statistics and all code are made publicly available.Burstiness, the uneven rhythm of sentence lengths in a text, is one of the most explicit differences between human and AI writing, and among the least precisely measured. This study measures it at scale. We compared 60,779 human-written academic texts with rewrites of the same texts generated by eight AI model configurations across six model…
We compared 60,786 human-written academic texts with AI rewrites of the same texts, produced by eight model configurations across six model families, 40.4 million tokens in total, and measured which words the models inject and suppress. The signature is consistent: formal connectives and Latinate substitutions such as "thereby" (13 times the human rate), "consequently" (11 times), and "utilized" (7 times), while plain words humans prefer are suppressed, with "used" cut to one ninth of its human rate. The famous chatbot words behave differently in rewriting: "meticulously" is enriched 214-fold, yet "delve", the most famous marker of all, is not enriched at all. We release the studied lexicon of 1,057 words with per-word corpus statistics and AI scores, and a full-vocabulary lexicon that scores the entire English vocabulary, every single-word WordNet lemma (90,520 entries), of which 3,371 pass a two-signal evidence check.We compared 60,786 human-written academic texts with AI rewrites of the same texts, produced by eight model configurations across six model families, 40.4 million tokens in total, and measured which words the models inject and suppress. The signature is consistent: formal connectives and Latinate substitutions such as "thereby" (13 times the human rate), "consequently" (11…
We audited 60,786 paired passages, each a human-written academic excerpt and a machine rewrite of that same excerpt, produced by eight model configurations across six model families. The pairs contain 213,881 in-text citation marks, and every rewrite citation can be checked exactly against its source. 96.9 percent of citation marks survived rewriting unchanged. 2.26 percent were corrupted, including silently dropped citations, corrupted author names, and shifted years. Corruption varied 28-fold across models on identical inputs, from 0.30 percent to 8.34 percent, and a small class of rewrites asserted author-year attributions that the source text never stated.We audited 60,786 paired passages, each a human-written academic excerpt and a machine rewrite of that same excerpt, produced by eight model configurations across six model families. The pairs contain 213,881 in-text citation marks, and every rewrite citation can be checked exactly against its source. 96.9 percent of citation marks survived rewriting unchanged. 2.26 percent…
Five current model families (DeepSeek, Mistral, OpenAI, Anthropic, and Gemini, 2026) were audited under one protocol across 30 academic topics. Writing in prose, the models embedded 194 author-year citations and none was an outright fabrication. Asked for full reference lists, the same models produced 1,500 references, of which 15.2 percent were fabricated or attributed a real title to the wrong authors, with family rates from 9.0 percent (OpenAI) to 29.7 percent (Mistral). A further 20.9 percent referred to real works with wrong details, and one in four supplied DOIs either failed to resolve or resolved to a different publication. Invention concentrates exactly where bibliographic detail is demanded.Five current model families (DeepSeek, Mistral, OpenAI, Anthropic, and Gemini, 2026) were audited under one protocol across 30 academic topics. Writing in prose, the models embedded 194 author-year citations and none was an outright fabrication. Asked for full reference lists, the same models produced 1,500 references, of which 15.2 percent were fabricated or attributed a…
Nine commercial AI text detectors (Turnitin, GPTZero, Originality.ai, Pangram, Copyleaks, ZeroGPT, Winston, Sapling, and QuillBot) were treated as independent raters of 90 academic texts: 30 purely human-written before 2022, 30 AI-generated by five model families, and 30 hybrid (human + AI) splices. Overall agreement is substantial (Krippendorff's alpha 0.71, Fleiss' kappa 0.78), but on hybrid human-AI texts agreement falls to chance level (mean pairwise kappa 0.02). In total, 28 of 30 hybrid texts received both a human verdict and an AI verdict from different tools, and the median per-text score range was 100 points. On mixed human-AI writing, the classification a text receives relies more on the choice of detector than on the text itself.Nine commercial AI text detectors (Turnitin, GPTZero, Originality.ai, Pangram, Copyleaks, ZeroGPT, Winston, Sapling, and QuillBot) were treated as independent raters of 90 academic texts: 30 purely human-written before 2022, 30 AI-generated by five model families, and 30 hybrid (human + AI) splices. Overall agreement is substantial (Krippendorff's alpha 0.71, Fleiss' kappa 0.78), but on hybrid…