Why Story Discovery is the Killer App for Generative AI in Journalism
An argument for why story discovery, rather than story telling is a better use of GenAI’s capabilities and reflection of its limitations
The Generative AI in the Newsroom’s aim is to collaboratively figure out when — and when not — to use generative AI in news production. As a writer for the site, and having worked on applying generative AI to a variety of different newsroom tasks over the last three years, I am becoming increasingly more comfortable with recommending one use beyond any other: using generative AI to surface story ideas.
This is where the exciting — and most journalistically valuable — stuff is happening. Take my coursemate at the AI Journalism Lab at Newmark J-School, Rune Ytrberg, who impressed me with his use of large language models (LLMs) to uncover a health scandal in his local hospital. He asked LLMs to find the most severe cases of neglect from more than 1000 pages of Norwegian hospital documents. This unearthed the case of a radiologist who had been giving suspected cancer patients’ X-Rays a fleeting glance, to devastating consequences.
Let me elaborate my argument for why newsrooms should prioritise tasks to help with finding stories when using generative AI. The crux of it boils down to this core limitation of the technology: Generative AI often makes errors, or “hallucinations” as the technologists like to say. This affects everything for the news industry where a fundamental goal is striving for accuracy. We have already seen factual errors creep into the news, thanks to misjudged uses of generative AI that don’t appear to prioritize accuracy at all. I wrote previously on this site about Apple withdrawing its Apple Intelligence news summaries after multiple inaccuracies. For instance, Apple Intelligence wrongly summarized that the man charged with the murder of UnitedHealthcare CEO Brian Thompson, Luigi Mangione, had shot himself outside court when the BBC had reported he Mr Mangione shouted “completely unjust” while on his way to court.
Journalist Karen Hao’s simple framework has helped me work out a safe use of generative AI. She has two questions:
- Does the task that you’re trying to perform with AI need high accuracy, or does it not need high accuracy?
- Will your output be internally facing for your own research, or will it be audience-facing as a final product?
“If it’s low accuracy, and it’s going to be internally facing research, then it’s much safer to adopt Gen AI” she says in a Reuters Institute article.
Using generative AI to get tips for potential stories fits into Karen’s framework for a safe use of generative AI since it is an internally facing activity and doesn’t absolutely require high accuracy since someone will check the lead before advancing it. I split the news-making process broadly into two main blocks: story finding and storytelling. Story finding is the furthest away, in the production process, from the audience with the most opportunities for journalists to catch any mistakes before an audience member sees.
If you use generative -AI for storytelling, such as reformatting, summarizing, or wording for a younger age range then, as a responsible news organization, you will then have to check that copy for any factual errors the model may have introduced. I know what this fact checking looks like up close. I used to manage a team of 15 reporters at the UK publisher Newsquest. Their job was to check the AI-generated first draft of a news story. Some called our team the humans in the loop. The task of spotting the factual errors AI introduces is subtly different to a traditional subeditor. In very non-technical terms, it was as if a little AI pixie would dance around the copy sprinkling falsehoods such as mixing up names, making up dates and, now and again, changing a fundamental fact in the story.
It was the AI-assisted reporters’ job to catch the pixie’s mischief by reading the original input, then the AI generated copy and then weed out the differences. They completed this task multiple times a day and largely succeeded in producing stories quicker than other reporters.
Another experience, however, leads me to argue that newsrooms should instead focus on applying generative AI to transformative tasks rather than efficiencies. By that I mean, focus on tasks you didn’t or couldn’t do before rather than on finding ways to make the current tasks you do quicker and/or cheaper. That’s because efficiency can be an illusion when you don’t have access to details on all the costs. As a product manager for a small audio production company, Overcoat Media, I have just finished making a prototype for a generative AI tool called PodMorph which reformats podcast content for various platforms.
One of the things I loved about this project was that I had rare access to a fuller picture of the costs. I was struck that it would have been entirely possible to spend more using generative AI than the savings you might make on labor. There are a lot of variables and increasingly more ways of doing things. But some costs for us included the obvious generative AI tools as well as previously hidden costs of cloud databases and server costs. We also spent money on the user research, the user testing (both of which used the content makers’ time) and we spent money on a software engineer. Also, I was aware that we took up the time of some of perhaps the most expensive people at Overcoat Media — the directors — because bringing in generative AI forces you to ask a profound question about a company’s aims. I also found that when I analysed one particular manual task we were part-automating (generating a first draft of a text programme description) the difference between the cost of the maintenance of our tool and the cost of the time we were saving were almost identical. For all these reasons I believe it is better to aim to use generative AI to provide your audience with something you previously couldn’t, like uncovering more original and impactful stories.
In sum, using generative AI to tell stories (1) risks lower quality output, (2) can increase the proportion of repetitive work needed for checking outputs before publication, and (3) may not even be cheaper than your previous way of working. Instead, I argue, newsrooms should consider the opportunities in using generative AI for finding stories. That’s what my coursemate Rune did to uncover a scandal putting lives at risk.
Rune is the head of the data journalism lab at the northern Norwegian newspaper iTromsø. Through the second half of 2024, his team published a series of articles which revealed a radiologist had rushed his job checking for tumours on X-rays. Rune says he wouldn’t have found this story if he hadn’t used generative AI.
Rune’s team didn’t know of the existence of this story when they started looking into shortages at the University Hospital in Tromsø. They had first started looking into whether understaffing at Tromsø’s hospital was costing lives after Renate and Anders lost their son, Erik, in childbirth in July 2023. A year after Erik’s death, iTromsø reported that the state administrator concluded that the hospital had violated the requirement of a duty to provide proper health services. On reading this story, another patient contacted iTromsø and told them that he had been sent home with unbearable back pain after analysis of an X-ray didn’t detect a cyst which was then found a year later. This led Rune to do a wider search on hospital documents. Using Norway’s freedom of information law, he requested documents regarding the X-ray department sent to the state administrator and the health inspectorate. He received almost 1000 pages. He says this was too big to search in the traditional ways. This is where he started experimenting with generative AI.
By using LLMs from both OpenAI and Anthropic plugged into Anything LLM (an interface which allows users to, among other things, upload documents and get generative AI to search within them) he was able to interrogate the documents without knowing exactly what he was looking for. The breakthrough was when he asked for the five most severe cases from all the documents. This brought up documents regarding a substitute doctor and he requested more documents regarding a substitute doctor.
From this he discovered that a temporary doctor had been allowed to work from home on his own computer with little control from the hospital. With coordination from other newspapers, they worked out that the doctor had actually been working across Norway, Denmark and Sweden, which don’t coordinate work schedules so could have doubled up on shifts. In one instance, a document revealed that he had spent no more than 11 seconds looking at an X-ray. They found that five patients died under his watch.
CREDIT: Magnus Eriksen
Rune and his team went on to be recognised for their work by winning the Data-SKUP award (an award for data journalism given by the SKUP investigative journalism association in Norway) in October 2024 and were commended by the jury for making it “possible to absorb large amounts of unstructured information and give journalists the opportunity to find narratives and stories”. In other words, they had applied generative AI to find a needle in a haystack.
Rune’s metaphorical haystack was documents he obtained through freedom of information requests. Journalists are starting to experiment with using generative AI to surface stories from other heaps of information too. These range from US education publication Chalkbeat monitoring public meetings to Filipino journalist Jaemark Tordecilla interrogating his government budget. What other stories are waiting to be found in various haystacks of documents?
