When AI is the wrong answer to a document problem

Many AI project requests start with a document problem. Invoices arrive in different formats. Contracts need key terms pulled out. Staff spend hours copying fields from PDFs into a system. Somebody has seen a demo in which a model reads a document and fills in a form, and it looks like exactly the fix.

Sometimes it is. Often a cheaper, more reliable fix exists one step earlier in the process. The questions below help tell the two apart before any money goes into models, prompts or infrastructure.

Question 1: Do you control where the document comes from?

If the documents are produced by your own staff, your own website or a form you designed, the cheapest solution is usually to capture the data as data in the first place.

A customer uploading a PDF of a form that your own website generated is a strong sign. So is an internal team typing information into a Word template that another team then retypes into a system. In both cases, replacing the document with a structured form — with validation, dropdowns and required fields — removes the extraction problem entirely.

If you control the source, fix the source. A model reading documents you could have avoided creating is an expensive workaround.

Question 2: How much do the documents really vary?

Extraction problems range from “the same three layouts every time” to “anything a stranger might send”.

If a handful of suppliers send invoices in a handful of fixed layouts, template-based extraction or the structured e-invoice formats many suppliers can already produce may be enough, and they behave predictably. If documents arrive from hundreds of senders in unpredictable formats, with handwriting, scans and free text, a language model becomes much more attractive.

Count the layouts before choosing the tool. Teams often assume more variety than actually exists.

Question 3: What does a wrong answer cost?

Models make mistakes, and some of those mistakes look confident. The right design depends entirely on what happens when a field is wrong.

• A wrong product category on an internal report is annoying and easy to fix.
• A wrong bank account number on a payment, or a wrong expiry date on a contract, can be very expensive.

Where errors are costly, the workflow needs a human review step, clear confidence signals, and checks against other data (does this supplier’s bank account match the one on file?). That review step is part of the cost and has to be in the plan from day one, not added after the first incident.

Question 4: Is the volume large enough to matter?

Automation has fixed costs: building, testing, monitoring, maintaining prompts as models change, handling edge cases. If a person spends two hours a week on the task, those costs may never pay back. If a team spends most of its week on it, the case is very different.

Measure the current time spent, per document, including corrections. It is the only honest baseline for the business case.

Question 5: Can you build an evaluation set?

This is the most important question, and the one most often skipped.

Before building anything, collect a set of real documents — fifty to a hundred is a sensible start — together with the correct extracted values, checked by someone who knows the domain. Include the awkward ones: poor scans, unusual layouts, documents in other languages.

This set lets you test any approach, AI or not, and measure accuracy field by field. Without it, you cannot tell whether a demo that looked impressive will work on your documents, and you cannot tell whether a later model update made things better or worse.

If you cannot assemble an evaluation set, you are not ready to build the system yet.

When AI is the right tool

A language model is usually a good fit when:

• documents come from third parties you do not control
• formats vary widely and keep changing
• the information needed is described in text, not in fixed positions
• volume is high enough to justify the investment
• errors can be caught by review or by checks against other data

In those situations it can remove a large amount of manual work that no template could handle.

What to settle before any model spend

• The evaluation set, as above, and a target accuracy per field.
• A cost ceiling. Model usage is priced per request or per volume of text; estimate monthly cost at realistic volume, including retries.
• Data protection. Documents often contain personal data. Know where it is processed and stored, what your agreement with the model provider says, and whether the processing fits your GDPR obligations.
• The human step. Who reviews what, and how corrections feed back into improving the system.
• A fallback. What happens when the model or provider is unavailable.

Sometimes the answer is a smaller project

The most useful outcome of scoping an AI request is often a narrower project: two document types that carry most of the manual work, automated well, instead of a general assistant that does everything approximately.

That is how we scope AI work at ANASBRENT — use case first, evaluation set second, model last. If you have a document process you are considering automating, describe it to us and we will tell you honestly whether AI belongs in the answer.