Build vs buy: should you build contract extraction in-house with an LLM?

Guide · 7 min read · Updated July 2026

Here is the part vendors usually skip: yes, you can build contract extraction in-house, and the first version will be genuinely impressive. Paste a signed contract into a frontier LLM with a decent prompt and you will get the parties, the term, the pricing and the renewal date back in clean JSON — in an afternoon, with one engineer. That demo is real, it is not a trick, and it is exactly why every CTO who sees it asks the reasonable question: why would we pay for this?

The honest answer is that the demo and the product are different things. The model is roughly 20% of a production extraction system. The other 80% is everything that makes the output trustworthy enough for finance, billing and CRM to run on — and that 80% is where build-vs-buy actually gets decided.

What the afternoon demo proves — and what it doesn't

The demo proves that modern LLMs read contract language well. It does not prove any of the things a production system has to prove:

• Accuracy you can state, per field, per document type. "It looked right on the five contracts we tried" is not a number. Production means a labelled evaluation set, measured field-level accuracy on order forms vs MSAs vs amendments, and regression testing every time the prompt or model changes. Building the evaluation harness is often more work than building the extraction itself.
• Confidence and review workflows. The model will sometimes be wrong, and it will be wrong confidently. A production system scores its own confidence per field, routes low-confidence values to a human, and gives that reviewer the source clause alongside the extracted value so review takes seconds, not a re-read. Without this, every output needs full manual checking — which quietly recreates the manual process you were automating.
• The hard 20% of documents. Scanned signatures over text, multi-page pricing tables that split across page breaks, amendments that override clause 4.2 of an order form from two years ago, contracts in German that amend contracts in English. The demo used your cleanest PDF; production gets your worst.
• Matching, not just extracting. "Enterprise Licence — Tier 2 (annual)" in the contract has to map to an actual product in your CRM and ERP catalogues, or the extracted line items create duplicates instead of records. Entity resolution against your systems is its own project.
• Write-back integrations. Getting structured data out of the model is the easy half; landing it on the right Salesforce Opportunity, the right NetSuite record, without overwriting fields a human already corrected, with conflict flagging — that is integration engineering, and it is permanent. The system-level detail is a whole topic on its own: see extracting contract data into Salesforce.
• Reconciliation, not one-shot extraction. Extraction gets the data in once. The commercial value mostly comes from noticing when the systems stop matching the contract — the uplift that billing never applied, the renewal date that drifted. Continuous reconciliation is a second product on top of the first.
• Monitoring and drift. Models get deprecated and replaced. Document formats change when sales adopts a new order-form template. Someone has to notice when accuracy degrades — before finance does.

The maintenance burden is the real cost

The build cost people estimate is the initial build. The cost that actually accrues is ownership: prompt and pipeline updates when the model provider ships a new version, re-running evaluations, handling the novel document format that broke parsing, extending the schema when finance asks for payment terms as well as pricing, and keeping the integrations alive as your CRM admin renames fields. This is a standing engineering commitment, not a project with an end date — and it competes for the same engineers who are supposed to be building your actual product.

When building is the right call

Sometimes it genuinely is. Build in-house if:

• Extraction is core IP. If understanding documents is your product — not an input to your revenue operations — then you should own it, evaluation harness and all.
• You have real in-house ML capacity. Not "an engineer who is good with APIs" but a team that already ships and maintains ML systems in production, with evaluation infrastructure that exists today.
• Your documents are genuinely unusual. If your contracts are a document type no vendor handles well — highly bespoke, regulated, or structurally unlike commercial agreements — a vendor's head start may not transfer, and a narrow in-house build can win.
• You only need it once. A one-off migration of a few hundred legacy contracts, with humans checking every output, is a scripting task. Buy makes sense for a living pipeline, not a single batch.

If two or more of those describe you, take the build option seriously. Most companies evaluating this are in the other case: contract data is critical input to their business, not their business.

Total cost of ownership, counted in engineer-time

Currency figures date quickly; engineer-time does not. A realistic in-house build looks like this:

• Prototype: days. This is the afternoon demo, tidied up.
• Production pipeline (parsing, evaluation set, confidence scoring, review UI, catalogue matching, one or two write-back integrations): several engineer-months, usually across more than one engineer, because it spans ML, backend and frontend work.
• Reconciliation and monitoring: several more engineer-months, if it gets built at all — in practice this is the piece in-house builds defer indefinitely.
• Ongoing ownership: a meaningful fraction of an engineer, permanently — model migrations, format changes, schema extensions, integration breakage.

Against that, weigh what those engineer-months would have produced pointed at your own roadmap. That opportunity cost, not the vendor invoice, is usually the deciding line in the spreadsheet.

What buying should get you — the checklist

If you do buy, hold the vendor to the 80%: measured per-field accuracy demonstrated on your document types, confidence scoring with human review built in, source-linking from every field to the clause behind it, product matching against your live catalogues, write-back to your actual systems, and continuous reconciliation after go-live. A vendor that only does the extraction step has built the same 20% you could. The wider evaluation criteria are covered in how to choose contract extraction software.

Settle it with evidence, not speculation

The good news is you do not have to decide this on whiteboard arguments. Run your twenty ugliest contracts — the scanned ones, the amended ones, the multi-language ones — through a vendor sandbox and through your prototype, and compare field-level results. An afternoon of evidence beats a quarter of debate. TrustedIQ will run that sandbox on your documents: book a demo.