Does a better readability score mean an AI summary is clearer?
A readability score can improve without showing that anyone understood more. That matters when you ask an AI writing tool to make a research summary ‘accessible’. A March 2026 PLOS One paper compared human-written research blogs with versions generated using ChatGPT-4. The AI versions had lower Flesch–Kincaid grade levels: 14.6 versus 16.9. That score uses features such as sentence length and syllables; it isn't a test of what readers learned. The experiment involved 366 respondents recruited through a policy research institute's networks across 11 countries. More than 80% had at least a master's degree. It measured quality ratings and intended engagement, not whether people could correctly explain the findings. The study ran in 2023–24, so this isn't a verdict on today's writing tools either. For a summary meant to help someone make a decision, I'd give a few intended readers a different check: what does this finding let you decide, and what would you still need to know? Compare their answers with the source. That's a test I'd propose, not a result from this paper. Plain language is worth the effort. A nicer score shouldn't let us call the explanation finished while the person who needs it is still guessing.
Comments
I'd check what got easier by disappearing. In a hypothetical rewrite, ‘may help some patients’ becomes ‘helps patients’: shorter, smoother, and a larger promise. Before accepting an AI simplification, compare who the finding applies to and how certain it is with the original. Keep those limits in ordinary words, rather than banishing them to a footnote. The reader shouldn't have to choose between a sentence they can understand and a claim they can trust. That's an editing check I'd use, not a finding from this study.
I'd also let the rewrite get longer. For a hypothetical council research brief, ‘green infrastructure reduced runoff’ could become ‘Rain gardens held some of the rainwater that would otherwise have flowed into the drains.’ More words, but now I can picture what happened. I'd ask the AI to explain the unfamiliar term with a concrete example supported by the source, rather than just replace it with ‘better planning’. Then check that the example really belongs to the study. Short sentences can still leave the reader doing all the imagining.
I'd make that reader check private, and start with ‘I'm testing whether this explanation works,’ not ‘Let's see what you understood.’ Imagine a new employee reading an AI-written briefing while their manager waits for an answer. Saying ‘yes, that makes sense’ may feel safer than pointing out the sentence that lost them. Let them mark that sentence or ask a question without having to perform understanding in front of the team. Then fix the explanation and try again. That's how I'd run the check, not a finding from this study. The person helping you find a confusing passage shouldn't leave feeling they failed a reading test.