Imagine the AI pilot produces this line for the management report. “Revenue was SEK 90,000 below budget because two large customers postponed their purchases.” The amount is correct. The report contains actuals and budgets by branch. It says nothing about customers or postponed purchases.
Would you approve the draft? The person responsible for the monthly report needs a way to answer that before the first email goes out. Here’s how to set up a pilot in which a convincing sentence can still fail the test.
Ask finance to pick one page of the monthly report
Pick a page the same person revises every month. In this example, finance wants a draft of three or four sentences about one branch's revenue against budget. The person who usually writes the note approves it before management sees it.
Take last month's commentary and the report the writer had at the time. Mark the sentences you can substantiate from the report alone. Perhaps one sentence came from a phone call to the branch manager. Circle it. The model will be missing that information too.
Trace the SEK 90,000 back to the report
Suppose budget is SEK 1,000,000 and reconciled revenue is SEK 910,000 for the same branch and month. Let the reporting model calculate the variance of minus SEK 90,000, or minus 9%. Supply the branch, period and a link to the row. For an open month, show the cut-off date as well.
Test with the reader's actual access. A branch manager should only be able to retrieve rows they have permission to see. Enforce that rule when fetching the records and test it with a request for another branch. Agree with IT what the model provider may receive and store before using company figures.
Keep the difficult month in the test set
Choose a closed month, a missing budget and a late adjustment. Write down what you would accept before running the test. Set aside at least one month the developer cannot use to tune the drafts, so you can assess changes on figures the pilot hasn't been adjusted for.
| What you give the pilot | What the reviewer should find |
|---|---|
| Budget of SEK 1,000,000, actuals of SEK 910,000, no customer records. | The draft reports SEK 90,000 or 9% below budget. A possible customer explanation can only be raised as a question. |
| The budget row is missing. | The draft can give actuals and flag the missing comparison. A blank budget never becomes zero. |
| Another SEK 30,000 of revenue is posted to the same month. | A fresh draft uses SEK 940,000 in actuals and a SEK 60,000 shortfall. The version already circulated can still be read. |
| The branch manager asks for a comparison with a restricted branch. | The answer, links and totals do not reveal that branch's figures. |
Time the work until the comment is ready to send
Suppose the note normally takes 18 minutes. The AI writes a draft in one minute, but finance spends another 14 checking and rewriting it. The saving on this comment is three minutes before operating costs. Repeat the exercise over several periods with the person who will actually use the report.
Ask finance to distinguish between two kinds of correction. Awkward phrasing can be edited. A wrong amount, an invented explanation or data from a restricted branch should stop the email. Save the offending question as a test case and rerun the earlier tests once the error has been fixed.
Who will notice if next month's figures fail to arrive?
Decide who hears about a failed import and who approves each email. If the figures are stale, finance should be able to open the regular report and write the commentary as before. When a new budget version takes effect, someone needs to update the expected answers and run the checks again.
It’s reasonable to close the pilot if reviewing the draft takes as long as writing from scratch. You have answered the investment question before rolling it out to every branch.
Common questions
Can we run the pilot with made-up figures?
Yes. Add a missing budget, a correction and two branches with different access rights to test the behaviour. Before real distribution, reconcile with the ordinary report and test with the roles that will read it.
How long does it take to implement this kind of AI pilot?
Timing often comes down to whether the figures are available and whether somebody in finance has time to review drafts. A useful proposal sets out what must be in place at the start, what the team will deliver and who will make the decision after testing.