When Can You Trust AI with Invoices and Contracts?

You can trust AI with invoices and contracts when it extracts a single piece of information from a document. In our test, the best models did that with 98-100 percent accuracy. When the AI had to apply a condition, such as deciding whether a payment is late, the best model answered 83 percent of questions correctly and mid-tier models at most one in five. That is why the human check should follow the type of task, not the type of document.

This article continues our test in which six locally run language models read 50 Finnish invoices, calls for tender, contracts and meeting minutes. The first article compared the models. Here we break the results down by task and by document type. When you plan automation, the key question is not which model is best, but which steps you can hand to AI and where you need a person. When AI extracts and processes information from documents automatically, it is called intelligent document processing (IDP).

Which tasks can AI handle reliably?

AI handles extraction almost without error, and the best models also calculate and combine information reliably. Applying a condition is the only task where no model reached 90 percent.

Task	What it requires	Gemma 4 12B	Qwen 3.8 27B	Phi-4	Mid-tier models
Extraction	Finding one value	100%	100%	98%	58-98%
Calculation	Arithmetic on figures in the document	96%	87%	70%	2-48%
Cross-reference	Combining two parts of a document	98%	76%	84%	6-43%
Applying a condition	Applying a rule to a situation	83%	71%	66%	4-20%

Extraction does not separate the models. Of the mid-tier models (Qwen3 8B, Mistral 7B and Llama 3.2 3B), the 7-billion-parameter Mistral found single values 94 percent of the time, almost as well as Qwen, which is four times its size. The differences appear only when the information has to be processed.

Each model answered 41-50 questions of each task type, so individual figures are indicative: the margin of error on an 83 percent result is about ±11 percentage points. The ranking is still clear. Applying a condition was the weakest task for five models out of six, and the second weakest for the sixth.

Why is applying a condition the hardest task?

Applying a condition involves three steps in a row, and each one can fail. The question "The buyer pays the invoice on 10 June 2026. Is the payment late, and what interest rate applies?" requires finding the due date, counting the days and choosing the right condition.

A chain erodes the result quickly. As a worked example: if each of three steps succeeds 95 percent of the time, the whole answer succeeds about 86 percent of the time. That is why a model that is flawless at extraction can still fail one reasoning question in six, as the best model in our test did.

The same pattern shows up across document types. Short invoices were harder for the best models than long contracts:

Document	Gemma 4 12B	Qwen 3.8 27B	Phi-4
Contracts	98%	97%	89%
Calls for tender	98%	94%	87%
Meeting minutes	92%	78%	84%
Invoices	93%	83%	75%

The explanation is in the questions. Invoice questions required calculation, such as line totals, VAT amounts and late-payment interest. Minutes required combining times and agenda items. The figures for contracts and calls for tender may be slightly optimistic, because some models could not process the longest documents in time. The practical conclusion: a document's length does not tell you how hard it is for AI. The type of question does.

What mistakes does AI make with invoices and contracts?

The best models' mistakes were not invented facts but unfinished or overly cautious conclusions. The type of mistake tells you what your check needs to look for.

Mistake	Example from the test	What the check looks for
Right rule, missing conclusion	Asked for the earliest acceptable date for certificates, Gemma replied that certificates must not be older than three months and did not calculate the date (24 May 2026).	Does the model answer the question or just repeat the rule
Needless refusal	"Left at 16:34, the meeting ended at 19:08. Was the member present at the end?" Qwen 27B: "Information not found in the document."	Refusals where the answer can be calculated
Wrong formula	Phi-4 calculated the VAT rate by dividing the net amount by the gross total.	The intermediate steps of the calculation
Calendar error	Qwen3 8B: the offer is valid "until 31 November 2026".	Whether dates are valid
Right total, wrong conclusion	Mistral added up the invoice lines correctly but stated that the total did not match the net amount.	The conclusion against the calculation

What these have in common is that the wrong answer looks careful. Gemma's answer about the certificates is true, but it does not answer the question. A simple format check catches this kind of error: if you asked for a date, the answer must contain a date.

Needless refusal was the most common mistake among the best models. We covered its link to invented answers separately in our article on AI hallucination in business documents.

How should you split invoice and contract review between AI and people?

Split the work by task: extraction to AI, calculation to AI with a programmatic check, and applying conditions to a person for approval. That way a person does not review everything, only the points where the likelihood and the cost of an error are highest.

Process	AI handles	A person approves
Purchase invoices	Supplier, business ID, totals, VAT and due date	Invoices whose total does not match the order
Accounts receivable	Due dates, payment dates and open balance	The late-payment decision and interest before a reminder goes out
Calls for tender	Deadlines, requirements and references to attachments	Eligibility conditions, such as how old certificates may be
Contracts	Parties, term and notice period	Applying a condition to a situation, such as whether the contract can still be terminated
Meeting minutes	Decisions, attendees and agenda items	Conclusions based on times and attendance

The test did not measure the following practices, but they follow directly from the mistakes we observed:

1. Calculate days, interest and taxes in code. Let the model extract the figures and let ordinary code do the arithmetic. Calendar errors and wrong formulas are then impossible.
2. Ask for the answer as fields, not free text. When the answer takes the form "late: yes, days: 6", missing information is caught automatically and a contradictory answer cannot mislead the reader.
3. Test again when the model changes. Phi-4 added the payment term on top of the due date in the same way in five questions. A new model version can bring a new systematic error in its place, which is why AI automation degrades without ongoing monitoring.

If you are wondering where to start, pick a process with a lot of extraction and little applying of conditions. You can find a suitable candidate with the same scoring used to choose your first automation target.

The results apply to six locally run models, quantised and run on CPU only with the same instruction. The invoices were created for the test, and the comparison covers short and medium-length documents. The test did not measure cloud-hosted models.

If you want to know which steps of your invoice and contract processing AI can handle on its own and where approval is needed, a fixed-price Automation Assessment reviews your processes and calculates the value in euros for each opportunity.

---

Empirica Finland is a Finnish provider of operational AI and automation that builds automations from the data produced by a company's systems as well as its devices and sensors, and is responsible for keeping those automations running. Empirica is a Claude Partner Network member and a Microsoft partner.

← Back to homepage