Teaching a model to summarize from raw text alone
The problem
Training a summarizer normally needs people: paired summaries, instruction labels or preference rankings. That is expensive to collect and ties the model to one way of annotating. And in many places it is not available at all: proprietary or low-resource corpora, fast-moving domains, languages without annotation pipelines. So what can a model learn from the text alone?
The idea: a game with no judge
The working hypothesis is that a summary should preserve information that helps reconstruct its source. This is a proxy: a useful summary deliberately omits details, and reconstruction overlap alone does not establish quality. The summarizer writes a summary, a second pass tries to reconstruct the text from it, and the summarizer is rewarded for how much the reconstruction overlaps with the original. There is no learned reward model and no human label in that post-training loop. The starting models are pretrained; reference summaries and LLM judges are used for evaluation, not as training rewards.
Why a reward alone is not enough
Left alone, a model finds ways to raise this reward that do not make better summaries: it drifts away from fluent language and games the overlap. The fix is an anchor: the summarizer is kept close to a frozen copy of the pretrained model, so the quality of the language never has to enter the reward.
The difference is clear in the experiments. Without the anchor, every softer alternative I tried raised the reward while quality on held-out text collapsed. With it, the model drifted about 16 times less, and quality went up.
Results
I trained five models from three families, from 0.3B to 8B parameters, on three datasets: CNN/DM and XSum (news) and SAMSum (dialogue).
- Consistent gains. ROUGE-1 and BERTScore improved in all 15 model and dataset settings. On average over the settings, the gain is +1.46 ROUGE-1 points and +0.89 BERTScore points, and every paired confidence interval excludes zero.
- LLM judges prefer it. Three independent LLM judges compared the trained model with its starting model on the same texts, blind to which was which. Most pairs are ties, because a summary often changes only a little. When a judge picked a side, it picked the trained model 62% to 68% of the time. Where two judges both picked a side, they agreed on the winner 96% to 99% of the time.
- Output diagnostics. The outputs are shorter and less repetitive, with no detected change in copying from the source or in factual grounding under the diagnostics used. These checks do not rule out every form of reward gaming or factual error.
The judges are language models, not people, so I do not call this human preference.
What this is, and is not
This is a study of a mechanism, not a state-of-the-art summarizer. The point is narrower: raw text alone gives a usable training signal, and a fixed anchor is what keeps that signal from being exploited. The same safeguard also held up when I deliberately corrupted the reward, which suggests it may be useful for other imperfect rewards too.
Further extension
Can the model also create the problems it learns from? I explore this extension in Self-challenging language models: endless verifiable problems at the edge of ability, where a pretrained model plays both task writer and solver, using executable verification to supply the training signal without a task dataset or judge model.