The race for leveraging generative AI as a strategic advantage is on, but the ground must be tilled before the field yields results.
Large language models (LLMs) like ChatGPT and Microsoft Copilot have swept through the corporate landscape as many companies have licensed or built their own internal instance of the technology. Early adopters in organizations are getting their hands dirty with this new tool, but those in the know realize that their house—the underlying data on which AI models are built—must be in order before they can make the AI revolution a reality.
- Ed Barrie, a finance technology expert and the co-founder of Treasury4, recently told NeuGroup Insights, “AI and machine learning are only as good as the underlying datasets they’re being applied against. I’m skeptical about some of the use cases that do get talked about in the industry—unless you have a really rich dataset and much more granularity to then drive those insights.”
- At a recent session, many members of NeuGroup for IT Auditors shared that this point rings true: for now, the use cases for generative AI are limited. Corporates must have a plan to obtain large quantities of high-quality data before they can fully realize the efficiency gains promised by AI and machine learning.
- One member named two places where “the rubber should hit the road” for companies: “In the automation of task work, especially in the building out of sampling and testing methodologies; and in building the LLM using audit data to provide summaries, analysis and trending risk assessment data.”
Current use cases. Members are finding ways to gain small victories from the technology—for some, these victories are won by significantly cutting down research time for audits.
- “Right now, I see AI as a knowledge tool similar to Google search that can be used to increase efficiency and effectiveness,” one member said. AI can help to cut down the time researching and writing reports by 10% to 50%, he added.
- The same member that after challenging audit leaders to use the company’s internal LLM, multiple team members could grasp the unique details of specific risk frameworks in less time and with a better understanding than from a Google search.
- “What I’m going to try to do in our model in the near future is basically try to string together several documents,” another member said. “Right now I can get an extract of tickets from our project management software, put it in our platform and basically ask it, ‘Get me these several fields from these tickets and put it in a table format in this order.’ And it’ll kick out a table you can copy and paste into your work paper.”
- One member’s team has found use cases like data entry that can make a 60-minute task take 30 or 45 minutes. But for anything beyond that, the conclusion is, “I don’t have the datasets. I have talked to others who have put five years of their audit work in, and it still wasn’t enough to train the models.”
Training the models. Some IT auditors shared that they are focusing on automating manual work, which one member called “dumb” work, using tools like Power Automate or UiPath; then capturing that work in datasets to train their models.
- “We’re going to devote full resources to look at every process, automate, and then put AI over that automation, particularly when it comes to controls,” one member said. There are a lot of areas where there are manual controls—we’re trying to automate there. We want to fix the processes before we just jump into putting AI in something that is broken.”
- In identifying what should be automated, one member suggested that “as you conduct your normal course of work, identify the pieces of evidence your auditor is asking for. Take inventory of those things because later on, when you want to get to automation and translate that automation into actual action for the business, you’re going to need to understand where the datasets sit.”
Future use cases. The goal, many IT auditors in the session shared, is to get internal LLMs to the point that they can provide better, company-specific outputs—applying this tool to more value-added, strategic work.
- A member described his team’s process for auditing IT help desk tickets, using random number generators and dozens of tabs in spreadsheets. “That’s time-consuming work, it takes hours,” he said.
- Once a process like this is automated, he will be able to test and validate a much broader swath of data that can train an AI model to do value-adding work like identifying key trends and themes, providing risk assessment data and issuing summaries.
- The same member’s “pie in the sky” goal is to use an LLM to comprehensively generate a risk assessment, which he said will make the audit planning process more effective and efficient. “You want to be able to ask a library, ‘where do I have governance issues?’ or ‘how long did it take to close out this management action plan that was high risk?’”
