Microsoft’s Copilot helps pave the way for more use of AI by auditors and others at corporates embracing change.
One NeuGroup member who is an early adopter of Copilot, Microsoft’s assistant for using generative AI within its Office suite of products, demonstrated the usefulness, efficiency and power of large language models (LLMs) like ChatGPT by presenting at a recent NeuGroup meeting with a slide deck generated in part by—you guessed it—AI.
- The presentation, delivered at the fall peer group meeting of NeuGroup for Internal Audit Executives, sparked a conversation about AI use cases, best practices, risks and governance. It was one of dozens held across the NeuGroup Network this year about leveraging ChatGPT and other LLMs.
- One of those sessions focused on the implications of AI for IT auditors. A few companies are setting up labs to give IT auditors a deeper understanding of LLMs and the risks they present.
Audit use cases. The presenter provided a hypothetical use case of a team member designing an audit program entering the prompt, “Pretend you are an auditor assessing X; what are the key risks? And what test steps would you suggest?” Members who have done similar prompts have been impressed with the tool’s ability to help teams get started in the early stages of a long process, freeing up users to spend more time on activities that add value.
- Another relatively straightforward use case is employing AI to record interviews or meetings. The Copilot tool can produce a transcript from Teams, Microsoft’s communications platform, summarize the discussion in a work paper and produce a list of to-dos for meeting participants. One member said, “the summarization tools are amazing.”
- With a Copilot license, a user can also access any referenced document or chat to produce a PowerPoint presentation. And they can search an entire database more efficiently depending on their access permissions.
- Auditors using Copilot can ask Excel to write formulas, create pivot tables and identify outliers from any dataset. Suggested prompts include, “summarize this data and identify three key trends” and “create a model to show what if X happens.”
Tips for prompting. The presentation included a list of tips for interacting with ChatGPT or other AI interfaces. They include:
- Define the tool’s role: “pretend you are an IT auditor assessing…”
- Use “chained prompting” by breaking up a complex prompt into multiple, simpler tasks.
- Define the output by supplying the format you want: “create a document” or “provide a list.”
- Change the tone or format: “make this less formal” or “summarize this.”
- Provide context and be as specific as possible. Example: “Pretend you are an IT auditor in a large tech company assessing access controls in SAP S/4HANA. Provide a list of three key risk areas I should focus on with suggested audit test steps in each, in a form that a non-expert would understand.”
Risks. Access control for companies using AI was a key risk identified in the discussion. As one member put it, “It seems if you’re going to put this into place, then your data governance needs to be really strong.” If it’s not, companies run the risk of users accessing confidential or privileged information.
- A section of the member’s presentation on AI governance recommended establishing a clear and accountable structure; developing and implementing AI policies and procedures; conducting regular AI risk assessments and audits.
- Several members discussed the risk of employees’ personal information being accessed in error, and fallout from that. When asked about whether a rogue user could ask the chat to pull the CEO’s Social Security number, one member said, “In theory, it could find it.” This is where a company’s IT and IT audit teams come into play.
IT auditors setting up AI labs. At the inaugural meeting of a NeuGroup pilot peer group for IT auditors, members discussed how they are handling the use of AI across organizations. To do their jobs, they need to understand AI technology. Or as one member put it, they need to be able to “use their own sticks to make fire.” His company is among a small number setting up labs where experts will instruct IT auditors.
- The member wants IT auditors to get hands-on experience with what they’re auditing and told his team, “This is your opportunity to learn how AI and ML works in detail.” He said having the ability to audit AI tools without relying on a third party is imperative.
- Another member, who is not setting up a lab, pointed out that AI advancements may mean that IT auditors will have to add advanced skillsets to their teams. “Now you’re talking about finding a qualified data scientist. How are you solving the people problem on this?” he asked.
- That generated much discussion among participants. Most agreed that finding the perfect person—a data scientist with audit experience—was unlikely. But looking for a data scientist who likes challenges and is a leader is a good first step.
