Analyzing interviews, drafting reports and auditing expense reports are among AI use cases for auditors and risk managers.
At recent peer group meetings of NeuGroup for Enterprise Risk Management and NeuGroup for Internal Audit Executives, members interested in harnessing the power of generative artificial intelligence tools discussed use cases as well as potential data security risks posed by AI. Several member companies have contracted with OpenAI—creator of ChatGPT—or Microsoft to set up in-house, large language models (LLMs) while others are considering the move.
Extra work now for less later. One member of the ERM group is using an in-house LLM to streamline his review of the nearly 100 risk interviews he is responsible for annually. He’s working with his company’s data analytics team to train the AI on past data and a glossary of jargon. All identifying information about individuals in the company was stripped out before uploading interview transcripts.
- Wringing new efficiencies from the tool required increased legwork on the front end, the member said. “The problem is I just created more work for myself this year because I’ve got to manually go through and look at what the tool says versus what we say and see if it’s reasonable.
- “Otherwise, do you just trust it? How do you audit it unless you do all the manual work? So now we’re doing twice as much work in hopes that next year we’ll be doing half as much work.”
- In an ideal world, the member said, “I’m hoping it will be near real-time so that we can do an interview, and either right after the interview or within a few hours, the interviewee gets a note that says, ‘Here are the meeting notes. This is what we believe the key points were. Please comment.’ That would save me a whole lot of time.”
Drafting reports. Across the two groups, the most common use case being discussed involves the drfafting of audit or risk reports. While reports produced by generative AI will require revision and correction, they will help get the ball rolling, members say.
- One member explained why he’s optimistic about the tool. “It seems like the most obvious use case for internal audit is the audit reports or the drafting process.
- “If ChatGPT was able to pass the graduate management exams at an 80 percent rate, then surely it can come up with [an outline of] a draft audit report that would be meaningful in a way that would save enough time and mindshare to be useful.”
- He views this as low-hanging fruit but sees more applications including audit planning if it delivers what’s been promised. “Pretty mind-blowing,” he said.
Expense reports. Other use cases include using the technology to assist or fully take over some of the painstakingly detailed work of combing through data that needs to be audited—like expense reports. One member sees a future use case in his company’s continuous audit program, which includes going through mountains of credit card expense reports.
- He said, “Ideally it would identify things with precision, rather than producing a list of 500 things that could be fraud” that someone would then need to review manually. Another member wistfully added, “You kind of think AI should do your T&E audit for you.”
A hallucinating black box. Several members across the two groups acknowledged that GPT tools function as a black box, raising questions about how to audit outputs from them. As has been widely reported, generative AI also occasionally introduces errors, a phenomenon that has been dubbed hallucinating.
- The member using the tool to streamline the risk interview process reported back in a recent monthly ERM session on the results of his first foray. “No one knows how this works, so it’s really hard to audit,” was one comment.
- He also described issues in the initial outputs. “The AI was hallucinating. We tried to limit the creativity factor to make it be more literal. But if you say, ‘here’s a transcript, tell me what the top 10 risks are in this discussion,’ in a 45-minute discussion, some people didn’t mention at least 10. Some mention two over and over again. Well, the tool will try to figure out something from that text to make up 10.”
Managing the risks. In the past few months, some corporates, such as Samsung, have banned the use of generative AI chatbots for employees. The Samsung ban came after an engineer uploaded sensitive internal source code to ChatGPT.
- And several member companies in the ERM and internal audit groups are prohibiting employees from using ChatGPT or other LLMs until they have a closed system—with most companies eying enterprise licenses from LLM providers including OpenAI. The goal is mitigating the potential risk of IP or other proprietary information finding its way onto the internet.
- Several companies have some sort of AI committee. One member has an AI council through which business leaders within the organization can make requests to use AI in their function. She discussed with her team possible use cases and said report writing seemed like the most natural choice. “We’re going to log a request to get some kind of approval to just get the discussion going,” she said.
The biggest risk: missing out? Almost every member involved in these conversations sees that perhaps the biggest risk is getting left behind by competitors who adopt and leverage the tool quicker than they do.
- One ERM member put it bluntly: “Can a cut-rate, mediocre competitor become a power competitor if they figure out how to use these tools quicker than we do?” Another said, “AI has replaced ESG and work-from-home as a topic that you can’t escape.”
