Teaching a Custom GPT to Read Audit Reports and Support Watchdog Journalism
For the longest time, journalists in the Philippines have been relying on reports from the government’s Commission on Audit (COA) to help uncover potential graft in government agencies. The audit reports usually provide leads that reporters could explore further to see if there are anomalies involving how public funds are spent.
A recent example was the purchase by the country’s Department of Education (DepEd) of P2.4 billion ($134 million) worth of laptops that state auditors deemed to be “pricey” and “outdated” in 2021 — the agency had procured computers with Intel Celeron chips for about PhP58,300 or more than a thousand dollars each, about P23,000 above the approved prices, and definitely much more expensive than market prices. The information from the state audit report was uncovered by GMA News Online reporter Llanesca T. Panti, whose story was published on August 3, 2022. (Disclosure: I was editor-in-chief of GMA News Online at the time.) The story quickly became viral, and public outrage was swift. The next month, the Senate’s blue ribbon committee began a probe into the anomaly. By January, the Senate panel recommended filing charges of graft and perjury against government officials involved in the procurement.
Panti was the first journalist to publish the overpriced laptop procurements, and while other news outlets later reported further on it, the issue might not have made it to public consciousness if not for her eagle eyes. The Executive Summary for the DepEd 2021 audit report was 46 pages long, and the observation about laptops was buried in page 26. Only a couple of lines provided details about the laptop purchases:
Teachers’ laptops procured thru the DBM-PS are pricey for an entry-level type laptop
We recommend that Management:
a) explain why the amount of ₱58,300.00 price per unit was accepted by DepEd as part of DBM-PS’ recommended ABC despite the fact that the estimated price based on DepEd’s approved APP and APR is ₱35,046.50 only;
If Panti hadn’t spotted it, maybe another reporter from another publication would have spotted the detail. Or maybe not. It was easy to miss.
The Center for Media Freedom and Responsibility, a media watchdog in the Philippines, notes that “COA’s many audit reports are agency-specific and may be difficult to read and understand.” It is usually up to reporters and editors from each newsroom to go through the stacks of pages and try to figure out what would be newsworthy from each report.
Because journalists have limited attention and newsrooms increasingly have even more limited resources, I investigated whether AI could help augment journalists’ capacity to go through audit reports looking for clues that could uncover anomalies.
Using ChatGPT
To start, I tried using the vanilla ChatGPT’s GPT-4 model from OpenAI. With a subscription of $20 a month to ChatGPT Plus, it allows you to upload a PDF document that it can scan and summarize.
Asking the model to summarize the document — the Executive Summary of a COA report from the Philippines’ National Economic and Development Agency — came back with a description of the contents of the document, which might be nice for a grade school term paper on the subject but is virtually useless for an investigative journalist.
Asking ChatGPT if there are red flags in the document provides a little more insight, but many of the points in the summaries are still too vague for a reporter looking for a story. For example, ChatGPT said the audit “found errors affecting asset, liability, and equity accounts. Such errors can indicate weaknesses in the accounting processes and systems, possibly leading to inaccurate financial reporting.”
I tried another prompt: “I’m an investigative reporter. Are there any newsworthy items from this report?” Again, it gave vague responses.
At this point, a reporter would probably realize that it might be easier to read the audit report themselves rather than trying to wrestle with ChatGPT to find the exact prompt that would uncover newsworthy information.
Enter Custom GPTs
In November, OpenAI announced that it was allowing users to create custom agents on top of ChatGPT. I explored whether a custom GPT — helpfully named COA Beat Assistant — could actually help an investigative reporter deal better with audit reports.
After testing different configurations for the COA Beat Assistant, I came up with the following guidelines:
- To try to find meaningful information in the Executive Summary of COA reports, I had to limit what I wanted COA Beat Assistant to look for. Each document contains a section called “Summary of Significant Audit Observations and Recommendations”, and I wanted the GPT to have a short summary of each item there, which would serve as sort of a menu for the reporter.
- I wanted COA Beat Assistant to use only the information found in the corpus of the document, and not include anything outside that document such as from previous training data.
- After summarizing the Significant Audit Observations and Recommendations, I wanted the GPT to guide the user with prompts for the next steps.
- After uploading the report, I wanted COA Beat Assistant not to respond immediately. Instead, I wanted it to ask the user if they wanted to summarize the Significant Audit Observations and Recommendations. The technical reason for this is that the token limit for ChatGPT counts both input and output for the same prompt. This means that if you upload a 46-page document for the GPT, it might not have enough space to generate a useful reply.
Given this, I ended up giving COA Beat Assistant the following instructions:
Upon the upload of an audit report, COA Beat Assistant will process the report, which contains a list of Significant Audit Observations. It will not yet summarize the report. Instead, it will ask the user if it wants a summary of all the observations in one sentence, followed by the Observation Number. After displaying the summary of observations, it will prompt the user with possible options. It will use only information from this document and not use any previous training data.
I tried using COA Beat Assistant with the Executive Summary of the 2021 COA Report for DepEd, the same report that contains the item about the overpriced laptops. After uploading the document, it asked me if I wanted a summary of observations, to which I said yes.
It returned with a summary of the observations. It was helpful, albeit incomplete and riddled with formatting errors. For example, it didn’t include all the observation numbers in its summary:
But the information it generated would immediately be useful to any reporter looking for leads. At the very least, it included the issues related to overpricing and budgets — including for teachers’ laptops. Asking COA Beat Assistant for Observation №4, it led to the following insight:
I also asked it about Observation №5, flawed budget utilization:
At each of these points, there should be enough leads for the investigative reporter to decide whether they should do a deep dive on the report for a potential story. The COA Beat Assistant had already fulfilled its mission.
But of course, I already knew what I was looking for with the DepEd 2021 audit report. I tried out the tool on reports about other government agencies that I had not read before:
- Department of Agriculture
- Department of Tourism
- Cebu Provincial Government
- Pampanga Provincial Government
- Quezon City Government
In each of these cases, the assistant successfully summarized the audit observations from COA and prompted the user about which items they wanted more detailed responses. To my eyes, this was already a very useful tool for doing an intensive task.
COA Beat Assistant performed better with smaller documents. Interestingly, for some of those documents, the bot completely ignored my instruction not to put out a summary and went ahead and replied with summaries. (The summaries were, at least, accurate.)
All told, I spent a total of about 16 hours working on customizing COA Beat Assistant, trying different sets of instructions and prompts, to finally get it to behave close to the way I wanted it to behave. Given that there are dozens of audit reports published every year and each would ordinarily take at least an hour or maybe two to comb through, this kind of up-front investment could save a lot of time in the long run.
Using COA Beat Assistant in the Field
I wanted to test how useful COA Beat Assistant was for actual reporters working on audit reports. I gave access to the tool to Panti, the beat reporter who originally discovered the overpriced laptops issue, to see if the tool would be useful to her.
After a few hours of testing, it quickly became apparent how much it would make life easier for someone like her. When dealing with COA reports, Panti says, one of the biggest pain points was getting the PDF reader on her laptop to play nice with the documents, which always came with different formatting depending on the auditor producing the report. The summaries that COA Beat Assistant provides are useful at first glance, as it gives an experienced journalist like her a good idea whether to spend more time on a particular agency or to move on to another report that might yield something more newsworthy.
Without additional guidance from me, Panti also discovered further uses for the COA Beat Assistant. Because she was familiar with the structure of the full COA report, she would take insights from bullet points generated by the COA Beat Assistant off the Executive Summary, and let that guide where she would go next. She would then feed other documents from the full report where further information might be, to gain further insights. In one instance, she tried feeding it with the expanded Observations documents. In another instance, she fed it with an Annex document containing notes from the auditors, where more important details might be buried.
She also found shortcomings with how the COA Beat Assistant processed documents. In one instance, she noticed that the figure that the custom GPT returned didn’t seem quite right. It turned out that the bot was returning that figure from the first column of a financial report, rather than the total at the rightmost column.
Because of issues like these and her own concerns about GAI, she doesn’t see herself relying solely on the insights and summaries generated by the COA Beat Assistant when it comes time to finally write her story. Instead, she only uses them to find the sections that would be the logical starting point for her report, after which she would still do her deep dive on the document. She also wouldn’t trust a bot to help her write the story, preferring instead to do everything from scratch the old-fashioned way.
Still, she sees the tool as a useful timesaver, estimating that it could cut down on the time she spends on one report by as much as 80 percent. Panti, who also covers the House of Representatives beat for GMA News Online, sees a tool like this being helpful for journalists who do a lot of multitasking. “I could be monitoring a committee hearing while feeding audit reports to the COA Beat Assistant, and then dive deep into them later if there’s something that might be newsworthy,” she told me.
I introduced the tool in a workshop for journalists organized by the Philippine Center for Investigative Journalism (PCIJ). Sheila Coronel, the director of the Toni Stabile Center for Investigative Journalism at Columbia J-School and the founding executive director of the PCIJ, tried the tool out on the audit reports for provinces in Mindanao in the southern Philippines, some which are among the poorest areas in the country. Immediately, she noticed an item about the millions of pesos being spent by one area on Gender and Development, and flagged it as a potential investigative story.
Meanwhile, Francis Allan Angelo, the editor-in-chief of the regional newspaper Daily Guardian, was delighted by how the COA Beat Assistant cut through the jargon of COA reports to provide the summaries, even as he acknowledged that “the deep dive is best left to the reporter.” He compared the tool to using pivot tables in spreadsheets to get order and clarity for data.
Key Insights, and What’s Next for Custom GPTs
The introduction of custom GPTs is a potential game-changer for newsrooms, as it allows news managers and developers to limit the scope of work for the bot: it doesn’t have to think critically, it just needs to find the right information and summarize it. This allows for the development of new GPT apps customized to do the job within the parameters and guardrails set by the newsroom. It’s also impressive how fast it is to develop and deploy a useful tool for a specific purpose. Previously, you had to do costly and intensive machine learning training to deploy a similar application, and even then you might not get as effective a tool.
There’s also a low barrier given the $20 monthly fee to use ChatGPT Plus. However, it may become prohibitive for news organizations from developing countries, especially smaller ones, to get all their reporters a ChatGPT Plus account, even at that low monthly rate.
The tool tends to struggle with larger documents. Additionally, there are still restrictions on ChatGPT 4 that limit users from using it heavily; even paying users are limited to a certain number of queries over a given period of time. But other tools such as Copilot on the Microsoft Edge browser do not work as well with guardrails as the custom GPTs.
Meanwhile, even just the example of the COA Beat Assistant inspires possibilities within Philippine newsrooms — custom GPTs could be created for environmental reports, court decisions, and all sorts of corpuses of documents to make life easier for investigative journalists. If you or your newsroom are experimenting with custom GPTs for news tasks, let us know in this form so we can try it out!
