Why Scientific Papers Should Still Be Written by Humans

Often one gets asked in many different variations stuff like

• Oh but the remarks of the reviewers for the paper, you just paste them into an AI and let it revise the paper, right?
• Oh but why writing takes time? You can use AI for this, right?
• Writing in science is now done entirely by AI, correct?
• Writing a Thesis? I can use an LLM for this, correct?

The strongest argument against AI-generated scientific papers is not that AI makes mistakes. It is that scientific literature should remain an independent record of how humans think about science. It is an argument about the structure and patterns, less directly about the facts.

The use of generative AI for writing scientific publications is controversial, and in some institutions and journals it is restricted or prohibited. The reasons usually given are predictable: large language models hallucinate, they may invent citations, and they can confidently produce statements that are simply incorrect.

These are real failure modes of language models. But they are also becoming surprisingly weak arguments against the use of modern AI systems in science.

A language model asked from memory to “give me ten references supporting this statement” may indeed produce plausible-looking references that do not exist. But this is no longer representative of how a well-designed AI-assisted research workflow has to operate. A modern agentic system can search literature databases, retrieve the actual papers, inspect their contents, extract bibliographic metadata from the source, compare a claim against the cited material, and iteratively correct its output. The workflow is no longer simply

[ \text{prompt}\rightarrow\text{LLM}\rightarrow\text{answer}. ]

It can instead look much more like

[ \text{task} \rightarrow \begin{cases} \text{search}\\ \text{retrieve}\\ \text{reason}\\ \text{calculate}\\ \text{execute} \end{cases} \rightarrow \text{verify} \rightarrow \text{revise} \rightarrow \text{human review}. ]

Hallucination in such a system becomes an intermediate failure mode rather than necessarily an error in the final result.

This is not fundamentally different from other scientific workflows. A numerical optimizer can converge to a nonsensical solution. A nonlinear fit can produce physically meaningless parameters. A simulation can contain a programming error. A researcher can misremember a paper. None of these observations imply that optimization, simulation, literature searches, or human researchers should be prohibited.

They imply that results must be verified, in the same instance by the generating researcher himself. The same standard should be applied to AI. Ironically, as AI systems become better at verification, the most interesting argument for keeping generative AI out of the final writing process becomes stronger, not weaker.

• A Scientific Paper Is More Than a Container for Facts
• Generative Reformulation Changes the Distribution
• Scientific Literature Is an Unusually Valuable Training Corpus
• The Feedback Loop
• The Internet Is Already Contaminated - Science Does Not Have to Be
• There Is an Epistemic Provenance Problem Too
• This Is Not an Argument Against AI in Science
• A Perfect AI Does Not Solve the Problem
• Preserve the Independent Samples

A Scientific Paper Is More Than a Container for Facts

It is tempting to model a scientific paper as a collection of propositions (and I personally admit this is how I use such papers consciously myself when reading and refering to them):

[ S=\{s_1,s_2,\ldots,s_n\}. ]

Under this view, it should not matter whether a human or a machine generated the sentences, provided every proposition is correct, which is also how I usually personally think about the content: It doesnt matter who has written it, just the content is important.

But scientific literature contains another form of information. A paper is also an observation or measurement of how its authors organize, interpret and communicate their understanding of a subject.

It is the result of the projection from reality through a researcher onto the publication. A researcher decides which observation deserves emphasis. They choose whether two effects should be explained together or separately. They choose an analogy. They choose terminology. They decide that an unexpected result is “interesting”, “surprising”, “consistent with”, or merely “not inconsistent with” a particular interpretation. Even awkwardness can contain information.

An unusual explanation may indicate an unusual mental model. A peculiar term may expose intellectual ancestry between research groups. Different communities develop characteristic ways of structuring arguments. Researchers vary in how cautiously they formulate uncertainty and how they connect experimental observations to theoretical interpretations.

Consequently, scientific prose can be considered a sample from a distribution

[ P_{\mathrm{human}}(x), ]

where $x$ represents much more than grammar. It contains wording, structure, emphasis, terminology, argumentation, hedging, analogy and ultimately traces of the author’s conceptual model.

That distribution is scientifically and historically interesting in its own right.

Generative Reformulation Changes the Distribution

Now consider what happens when researchers routinely pass their manuscripts through the same families of language models. The published text is no longer sampled directly from the distribution of human scientific expression. Instead,

[ P_{\mathrm{published}}(y) = \int P_{\mathrm{AI}}(y\mid x) P_{\mathrm{human}}(x) \,dx. ]

Even when the semantic content is perfectly preserved, the transformation is not statistically neutral.

Language models have preferences.

They prefer some sentence structures over others. They normalize terminology. They remove unusual formulations. They tend toward particular ways of organizing explanations and transitions. Given many possible ways of expressing approximately the same proposition, a model does not select uniformly between them. They have preferences in the sampler.

This is precisely what makes AI rewriting useful: it maps unusual, awkward or improbable formulations onto formulations that are statistically more conventional.

But from another perspective, exactly that is also the problem. If sufficiently many researchers apply the same transformation, scientific writing becomes more homogeneous. One might loosely describe this as a reduction in entropy:

[ H(P_{\mathrm{published}}) < H(P_{\mathrm{human}}). ]

Of course, not every reduction in linguistic entropy is undesirable. Removing spelling mistakes probably destroys very little scientific information. But generative reformulation operates at a considerably higher semantic level. It can change argument structure, terminology, emphasis and explanatory style while apparently preserving the meaning.

Some of the variance being removed is therefore not noise. It is human variance, the actual patterns of the thought process.

Scientific Literature Is an Unusually Valuable Training Corpus

This becomes particularly important when considering what happens to the resulting literature.

Scientific publications are among the most valuable corpora available for training future artificial intelligence systems. They are curated, relatively information-dense, associated with identifiable authors, embedded in explicit citation networks and subject to various degrees of review.

Ideally, their provenance resembles

[ \text{physical world} \rightarrow \text{experiment} \rightarrow \text{human interpretation} \rightarrow \text{human-written paper}. ]

When generative AI becomes part of the publication channel, another transformation is introduced:

[ \text{physical world} \rightarrow \text{experiment} \rightarrow \text{human interpretation} \rightarrow \boxed{\text{model transformation}} \rightarrow \text{paper}. ]

This would perhaps be unremarkable if the story ended there.

It does not.

Those papers become training material for subsequent models.

The Feedback Loop

Suppose a model $M_0$ is trained predominantly on human-generated scientific literature. Researchers then use $M_0$ to reformulate their papers. These papers become part of the corpus $D_1$ from which the next generation $M_1$ is trained.

Researchers subsequently use $M_1$ to produce more scientific text:

[ M_0 \rightarrow D_1 \rightarrow M_1 \rightarrow D_2 \rightarrow M_2 \rightarrow \cdots ]

A feedback channel has been created. A simplified representation of the training distribution might be

[ P_{n+1} = (1-\alpha)P_H + \alpha P_{M_n}, ]

where $P_H$ represents independently generated human scientific expression and $P_{M_n}$ represents text influenced by previous generations of models. As $\alpha$ increases, future models increasingly learn from distributions partly generated by their predecessors.

The dramatic version of this problem is sometimes discussed in terms of model collapse. But complete collapse is not necessary for the feedback loop to matter.

A much more mundane outcome is distributional homogenization. Rare terminology becomes slightly rarer. Unusual explanatory structures become slightly rarer. Uncommon ways of conceptualizing a problem become slightly less visible.

The language preferred by existing models becomes increasingly common in the training corpus. The next generation therefore learns that this language is even more representative of scientific writing and reproduces it with correspondingly greater probability. A small statistical preference can thereby become self-amplifying.

The Internet Is Already Contaminated - Science Does Not Have to Be

There is an obvious objection: this process is already occurring across the public Internet.

That is true.

AI-generated product descriptions, articles, forum posts, documentation, summaries, social-media posts and automatically generated websites are increasingly mixed with human-created material. Maintaining the entire Internet as a pristine corpus of human-generated text is probably impossible. Also the idea of filtering by watermarking to avoid the self amplification will fail.

But that is an argument for preserving high-quality uncontaminated corpora, not for abandoning the distinction altogether.

Scientific literature is unusually suitable for this purpose.

It is comparatively small. It is curated. Its provenance is important already. Authors and publication dates are known. Version histories frequently exist. Scientific communities already maintain elaborate standards concerning attribution and provenance. Maintaining scientific publications as a predominantly human-generated corpus could therefore provide something increasingly scarce:

[ D_{\mathrm{science}}^{\mathrm{human}}, ]

a continuing stream of independently generated samples of human scientific reasoning and expression.

One might think of it almost as maintaining an uncontaminated reference sample.

There Is an Epistemic Provenance Problem Too

The issue is not relevant only to future AI training. Consider encountering the following sentence in a paper:

We interpret this behaviour as evidence for a transition between the two regimes.

Normally, the reader implicitly attributes this formulation to the authors. The sentence is evidence not merely that the authors endorse proposition $X$, but also about how they conceptualize $X$. There is an implicit measurement chain:

[ \text{author cognition} \rightarrow \text{written expression}. ]

If an AI generates the paragraph from notes or a rough draft, the chain becomes

[ \text{author cognition} \rightarrow \text{prompt/draft} \rightarrow \text{generative model} \rightarrow \text{written expression}. ]

A transfer function has been inserted between the quantity of interest and its observable representation. This does not mean that the resulting statement is false. The authors may inspect it carefully and agree completely with it. But the statistical provenance of the text has changed.

Science is normally extremely interested in knowing the transfer functions between something being observed and the signal eventually recorded. There is no obvious reason why scientific communication itself should be exempt from the same consideration.

This Is Not an Argument Against AI in Science

Quite the opposite.

Once the distinction is made between scientific assistance and generating the final expression of scientific thought, there is an enormous space in which AI can be used. An AI agent can search for relevant literature, retrieve papers, check whether citations actually support claims, reproduce calculations, execute analysis code, search for inconsistencies, identify missing controls, challenge interpretations, compare a manuscript against existing literature, look for dimensional errors, test reproducibility and act as an endlessly patient adversarial reviewer. It can even criticize the writing.

There is an interesting distinction between an AI saying

“The antecedent of ‘it’ in the third sentence is ambiguous between the detector and the sample.”

and saying

“Here is a better version of the paragraph.”

In the first case, the machine supplies information to the researcher, who subsequently generates another human sample. In the second case, the machine supplies the sample itself.

Schematically, the former retains

[ P(\text{text}\mid\text{author}), ]

whereas the latter increasingly produces

[ P(\text{text}\mid\text{author},M). ]

This suggests a much more interesting norm than simply declaring AI unsuitable for scientific work.

One could use AI extremely aggressively behind the publication:

[ \text{AI search} + \text{AI verification} + \text{AI calculation} + \text{AI criticism} + \text{AI reproducibility testing} \rightarrow \boxed{\text{human scientific expression}}. ]

In fact, sufficiently capable AI verification might eventually make scientific publications more reliable than they are today.

A Perfect AI Does Not Solve the Problem

A useful thought experiment is to imagine that all of the current technical objections disappear. Suppose an AI never invents a citation. Suppose it retrieves every source correctly, understands every paper it reads, performs calculations flawlessly, distinguishes evidence from speculation, and never introduces a scientific error while reformulating a manuscript.

Would there still be a reason not to let it write the scientific literature?

Yes.

Because

[ \text{scientifically correct AI-generated text} \neq \text{independently human-generated scientific text} ]

when the distribution itself is something worth preserving. Indeed, the problem becomes more important as AI improves.

A poor writing assistant will be used reluctantly. An excellent writing assistant that consistently produces clear, elegant scientific prose will be used everywhere. The better the model becomes, the greater the pressure toward a common model-mediated linguistic distribution.

The argument for preserving human scientific writing therefore does not depend upon AI being bad. It becomes stronger precisely when AI becomes good.

Preserve the Independent Samples

This leads to a rather different justification for restrictions on generative AI in scientific writing. Scientific papers should not remain human-written because machines are incapable of producing correct scientific prose.

They increasingly are.

Nor should scientists avoid AI because language models occasionally make mistakes. Humans make mistakes too, and both can be subjected to increasingly powerful verification procedures.

The more interesting reason is that scientific literature constitutes a continuously growing dataset describing not only what humanity knows, but how different humans independently understand and express what they know. Every independently written paper is another sample from that population. If those samples are increasingly passed through the same generative models, we gradually replace

[ P(\text{scientific expression}\mid\text{humanity}) ]

with

[ P(\text{scientific expression}\mid \text{humanity},M_0,M_1,\ldots). ]

That distinction may matter enormously to the artificial intelligences trained on this material, to historians studying the development of scientific ideas, to scientometric analyses, and simply to the intellectual diversity of science itself.

There is therefore no contradiction in advocating both far more AI in scientific research and less generative AI in the final writing of scientific papers.

• Use machines to search.
• Use them to calculate.
• Use them to simulate.
• Use them to criticize.
• Use them to reproduce results.
• Use them to hunt relentlessly for errors.
• Use them to tell us where our explanations are unclear.

And then let the scientist decide what all of it means - and write it down.

The scientific literature may become one of the few large, continuously maintained corpora in which we can still reasonably ask:

How did humans themselves describe what they discovered?

As generative AI becomes ubiquitous everywhere else, preserving an answer to that question may become more valuable, not less.

This article is tagged: Opinion, Science, Artificial Intelligence, Large Language Models, Machine learning, LLM