How to pick the right AI approach — buy the product, write the rule, call an API, prompt a model, add retrieval, or train your own — with a selector you can use in a meeting, the arithmetic that settles the argument, and the vocabulary that stops four incompatible proposals from being debated as one.
Motivation
Someone says: "We should use AI to read our invoices." Ten minutes later there are four proposals on the table: buy a document API, train a model on five years of history, prompt a language model, or write rules against the ERP.
The room debates them as competing answers. They are not. Two are different layers of the same pipeline and belong together. One is a runtime choice. One is a training choice nobody has costed. The meeting ends in a pilot that proves nothing, because no one wrote down what "works" would mean.
This is a missing procedure, not a missing skill. "AI" is not one technology with one adoption decision — it is nine approaches with different cost shapes, different failure modes, and different reasons to be wrong. This RFC is the procedure for choosing between them.
| Principle | Why |
|---|---|
| Decompose before deciding | Every "AI project" is a pipeline; each layer has its own answer |
| Constraints prune, economics ranks | Latency, residency, and auditability are gates — not trade-offs |
| Take the lightest option that clears the bar | Simplicity is moving parts you must keep alive, not technique |
| Prompting sets the task, retrieval supplies the facts, training fixes the form | Most "train it on our data" requests are retrieval requests |
| No selection without a scored evaluation set | Otherwise selection is a preference argument between senior people |
| Every selection expires | Capability and price move faster than systems get rewritten |
Pick in Five Minutes
Step 1 — Name the layer. "Read our invoices" is not a task, it is a pipeline. Choose per layer, never per project.
| Layer | Example question | This layer is |
|---|---|---|
| Source | Could this arrive structured instead? | Process design |
| Capture | What characters are on this page? | Solved perception |
| Normalize | Is this date, currency, ID canonical? | Deterministic code |
| Extract | Which number is the total? | Reading under variation |
| Classify | Which cost center is this line? | Fixed label space |
| Validate | Do the lines sum? Is the VAT rate legal? | Arithmetic and rules |
| Match | Which purchase order is this? | Search and scoring |
| Decide | Approve, hold, or escalate? | Policy under uncertainty |
| Act | Post to the ledger | Transaction code |
Six of those nine are not model problems. That is the point of the step.
Step 2 — Read the answer off the table. For the layer you named:
| If the layer is… | And… | Start here |
|---|---|---|
| A document that could arrive as data | Suppliers or systems are negotiable | Change the source |
| An entire standard workflow (AP, payroll, e-signature) | A mature product category exists | Buy the product |
| An exact, stable, writable rule | Arithmetic, validation, formatting | Code |
| A solved perception task | OCR, speech, translation, embeddings | Bought API |
| Reading, judging, or writing text | Under ~1M items/year | General LLM with a schema |
| Reading text about your own facts | Answers depend on internal data | LLM + retrieval |
| Reading text and taking action | Multi-step, path varies per item | LLM + tools |
| Text work under a residency or offline rule | Data cannot reach a provider | Self-hosted open-weight LLM |
| Numeric or tabular scoring | High volume, sub-100 ms, labeled history | Classical ML |
| Fixed labels at extreme volume | Above ~1M items/year, or a hard latency floor | Distill a small model — after running the LLM first |
Step 3 — Check the disqualifiers. These remove options before any comparison; establishing them after the build invalidates the build.
| Constraint | Threshold | Removes |
|---|---|---|
| Latency | Under ~100 ms per decision | Hosted LLM calls |
| Residency | Data may not leave a network or jurisdiction | Any provider without a compliant regional deployment — usually points at self-hosted open weights, not at training |
| Reproducibility | Same input must yield same output on audit | Generation as the final authority |
| Explainability | Someone can demand the reason for a decision | Opaque scoring as the sole basis |
| Untrusted input | Documents arrive from outside | Any design where model output can act without a deterministic check |
| Unit cost ceiling | A per-item cost the business case can't exceed | Large-model inference on the high-volume path |
| Offline | No network at inference | Everything hosted |
Step 4 — Get a number before you argue. If nobody can say what "working" means, the meeting's output is not a decision — it is an assignment to build a 200-item evaluation set and reconvene. That week routinely saves a quarter.
The default answer for text work is a general model with a strict output schema. It has no fixed cost, no training data requirement, and no model to own. Everything else on the list needs a reason: a constraint it satisfies, a workflow it replaces wholesale, or volume arithmetic that beats it.
The Approach Menu
Ordered by how much you end up owning — the first rung that clears the bar wins, because every rung down adds permanent operational surface.
| # | Approach | Wins when | You own | Fails by |
|---|---|---|---|---|
| 0 | Change the source | Input format is negotiable | Nothing | Third parties you don't control |
| 1 | Buy the finished product | The workflow is a mature commercial category | A contract | Ceiling on fit; integration and lock-in |
| 2 | Rules and code | The rule is exact, stable, writable | Code and tests | Brittleness as variation grows |
| 3 | Bought capability API | Perception is solved — OCR, speech, embeddings | A contract | A vendor ceiling you can't raise |
| 4 | General LLM with a schema | The work needs reading or judgment | A prompt and an eval set | Non-determinism; linear cost forever |
| 5 | LLM + retrieval | Answers depend on facts the model can't know | An index | Retrieval becomes the accuracy ceiling |
| 6 | LLM + tools | The work requires taking action | Orchestration and authorization | Compounding error; hardest to govern |
| 7 | Self-hosted open-weight LLM | Residency, offline, or extreme volume rules out hosted | Inference infrastructure | Capacity, upgrades, and quality gap vs. frontier |
| 8 | Classical ML on features | Numeric signal, high volume, hard latency floor | Labels, features, training pipeline | Silent decay; needs labeled history |
| 9 | Trained or distilled task model | Fixed labels, extreme volume, proven taxonomy | Everything in 8, plus a registry and rollback | Retraining forever |
Two rungs get skipped for bad reasons. Rung 2 feels unfashionable — yet validation and arithmetic belong in code, where they can be unit-tested, not in a model that is right 97% of the time. Rung 8 gets skipped because attention moved to language models — yet tabular scoring (fraud, credit, churn) is still won decisively by gradient-boosted trees, at lower cost and far better latency.
"Simpler is cheaper" — true, if you define simple correctly
The instinct is right; the usual definition is wrong. Simplicity is parts you must keep alive, not technical modesty.
| Technically simple | Operationally simple | Must be kept alive | |
|---|---|---|---|
| Rules engine | Yes | Only while the rules are few | Rule set, owner, regression tests, exception backlog |
| Classical ML / trained model | No | No | Labels, features, training, hosting, drift monitoring, retraining, rollback |
| General LLM with a schema | No | Yes | A prompt, a schema, an eval set |
An LLM call is the sophisticated technique and the operationally simple choice. A hand-written rules pipeline is the modest technique that quietly becomes 4,000 rules and a full-time maintainer. Counting complexity in moving parts rather than cleverness reverses the ranking most teams start with — and the reversed ranking is the correct one.
Is a frontier LLM expensive compared to classical ML?
Per item, yes — by two to four orders of magnitude. In total cost of ownership at ordinary volume, no.
| Classical ML / trained model | General LLM | |
|---|---|---|
| Per item | ~$0.00001 (CPU milliseconds) | ~$0.001–$0.05 |
| To reach production | $25,000–$80,000 (labels, features, training, deployment) | Near zero — a prompt and a schema |
| Annual ownership | $15,000+ (retraining, monitoring, hosting, on-call) | Zero beyond usage |
| Time to first result | Weeks to months | Hours |
Both columns are true at once, which is why unit price ranks the options backwards. And they are frequently not substitutes at all: classical ML cannot read an unstructured PDF, and an LLM cannot score a transaction in five milliseconds. Where both genuinely apply — fixed labels, high volume — volume decides, not preference.
The Words That Hide the Decision
Circulate this before the meeting, not after. Most selection arguments are two people using one phrase for different things.
| Said | Usually means | Actually requires |
|---|---|---|
| "Train a model on our data" | "It should follow our conventions" | Retrieval or a better prompt; occasionally fine-tuning |
| "We need our own model" | "Our data must not leave our control" | A hosting and contract decision — often self-hosted open weights |
| "It will learn from corrections" | "It improves as we use it" | Nothing automatic. A labeled feedback loop somebody owns and funds |
| "RAG" | "It can use our documents" | A retrieval design: filters, ranking, permissions, citations |
| "AI agent" | Anything from a chatbot to write access in production | An authorization decision, separate from the model decision |
| "It must be 100% accurate" | "I have not set an error budget" | An error budget and an escalation path per decision type |
| "Our data is too specialized" | An untested assumption about vocabulary | Twenty hard examples and an hour |
| "It works in the demo" | The easy third of the distribution | A re-run on the frozen holdout, tail included |
Six things called "training"
| Name | Changes | Needs | Order of cost | Who should |
|---|---|---|---|---|
| Pre-training | Creates a base model | Trillions of tokens | Tens of millions | Frontier labs only |
| Continued pre-training | Domain language | Billions of tokens | Hundreds of thousands | Genuinely alien vocabulary |
| Fine-tuning (incl. LoRA) | Output form, schema, tone | 1,000–50,000 examples | Thousands, plus upkeep | Proven, stable, high-volume tasks |
| Task model training | A dedicated classifier | 2,000–50,000 labels | Tens of thousands | Fixed labels at extreme volume |
| Preference tuning | Ranking among acceptable outputs | Thousands of comparisons | Tens of thousands | A graded quality signal exists |
| In-context learning | Behavior for one call | 0–50 examples in the prompt | Free | Everyone, first |
Fine-tuning changes how a model answers, not what it knows. A model tuned on last year's contracts writes confidently about clauses that changed in January. Facts belong in retrieval, where they can be updated, cited, and revoked.
Retrieval Is Not One Thing
"Add RAG" names a goal, not a design. Vector search is one method among several and is frequently the wrong one — most retrieval failures are method mismatches, not model weaknesses.
| Method | Wins when | Cost |
|---|---|---|
| Deterministic lookup by ID, date, owner, or scope | The right document is known, not searched | Near zero. No index |
| Lexical search (BM25) | Exact tokens matter — SKUs, error codes, names, citations | Cheap, explainable, no re-embedding on model change |
| Hybrid lexical + vector, then a reranker | General document corpora | The realistic production default |
| Vector search alone | Paraphrase and concept matching | Index sync, embedding-version lock-in, chunk-level permission risk |
| Structured query (text-to-SQL) | The answer is a fact in a table, not prose | No index; needs schema context and query guardrails |
| Live API call | Data changes faster than any index refresh | Zero staleness — the most under-used option |
| Knowledge graph | Multi-hop questions across entities | Expensive to build and to keep current |
| Long-context stuffing | The corpus is small — one contract, one handbook | No retrieval infrastructure at all |
Three rules regardless of method. Permissions are applied at query time, not after retrieval — filtering afterwards leaks. Freshness is a property of the method: every indexed approach is stale by its refresh interval; only live calls aren't. Embedding a number is worse than matching it — spend on the retrieval method before spending on a better embedding model.
The Arithmetic of Training Your Own
Training trades a large fixed cost for a lower marginal cost. The break-even is nearly always larger than people expect.
V* = (C_fixed + C_upkeep) / (c_general − c_specialized)
Worked with deliberately favorable assumptions — routing invoice lines to cost centers:
| Item | Assumption |
|---|---|
| Labeling 5,000 examples (model-assisted, human-reviewed) | $4,000 |
| Engineering: build, evaluate, deploy (3–4 weeks) | $25,000 |
| Retraining, monitoring, on-call | $15,000 / year |
| Inference hosting | $3,600 / year |
| Per item: small general model | $0.002 |
| Per item: self-hosted specialized model | $0.0001 |
| Horizon | Break-even volume |
|---|---|
| Year one ($29,000 fixed + $18,600 upkeep) | ~25,000,000 items / year |
| Steady state, year two onward ($18,600 upkeep) | ~9,800,000 items / year |
| Three-year total cost of ownership | ~14,000,000 items / year |
At a realistic 50,000 items per year, the general model costs about $100 annually and the specialized one about $47,600 in year one — to save $95. The gap only closes if the per-item delta is large, which is why the first optimization is always within the general-model rung:
| Per-item saving | Year-one break-even | Situation |
|---|---|---|
| $0.0019 | ~25,000,000 / year | Small general model → specialized |
| $0.02 | ~2,400,000 / year | Large general model → specialized |
| $0.10 | ~480,000 / year | Large model, long prompt → specialized |
Before training anything, try the cheap moves. A smaller model, a shorter prompt, a cached stable prefix, batching, and a rule that filters the easy cases routinely cut per-item cost by an order of magnitude — in days, with no fixed cost and no model to own.
When specializing does pay, the sequence is not "train instead of prompt." Run the general model in production, keep its inputs and verified outputs, then train the small model on that corpus. The general model becomes the labeling engine and the quality reference; the small model becomes the runtime. This avoids the classic failure of labeling a taxonomy up front that production then invalidates.
Re-derive these numbers with current prices. The structure is stable; the figures are not.
Evidence
No approach can be selected without a scored evaluation set — and the set outlives every choice made with it. Models get replaced; the eval set is what makes replacement a one-day decision.
| Build it in a week | |
|---|---|
| Sample | 150–300 real items, stratified across supplier, format, language, layout, amount band |
| Include the tail | A third from the awkward cases: handwriting, credit notes, multi-page, foreign currency, poor scans |
| Label per layer | Extraction and classification separately, not one end-to-end verdict |
| Freeze a holdout | A third, never inspected during development |
| Score abstention | "I don't know" is a correct answer at low confidence — it routes to a human instead of causing an error |
| Assign owners | Domain expert owns labels, engineering owns the harness, both in version control |
Where output is genuinely subjective — drafting, summarizing — the score is a human rubric, not an exact match. Rubric scoring is slower and still beats opinion. The only unacceptable answer is having no score at all.
| Metric | Why it decides |
|---|---|
| Accuracy per layer | Locates the weak stage instead of averaging it away |
| Escalation rate | Human review is usually the dominant cost, not inference |
| Cost per correct decision, including error cost | The only figure that ranks approaches against each other |
| p95 latency | Averages hide the failures users actually experience |
A system that abstains on 20% of items with 99.9% accuracy on the rest usually beats one that answers everything at 95%. The first has a known cost — review of a fifth of the volume. The second has an unknown liability spread across every downstream system.
Five Problems, Five Answers
| Problem | Decisive facts | Verdict |
|---|---|---|
| Invoice capture and coding, 50k/year | Volume ~500x below break-even; capture is commodity; VAT rules are arithmetic | Buy capture, code the rules, general LLM with a schema for extraction and coding, escalate low confidence. Training saves $95/year and costs a permanent obligation |
| Ticket routing, 50k/day, 12 queues, 200 ms | Fixed labels, extreme volume, hard latency floor | Start on the LLM to establish the taxonomy and generate labels in production; distill to a small classifier once the taxonomy holds for a quarter; keep the LLM for the low-confidence tail |
| Contract clause review, 400/year | Low volume, high judgment, standards shift with case law | LLM + retrieval over the clause library, mandatory citations, no autonomous decision. Fine-tuning here bakes in last year's positions |
| Fraud scoring, 5 ms, regulator asks why | Latency and explainability disqualify rungs 4–7 outright | Gradient-boosted trees with feature attribution. The LLM's role is summarizing a flagged case for the analyst — not scoring |
| Knowledge search, 30k documents, constant change | Needs current facts with provenance | Retrieval with citations and an explicit "not in the sources" answer. The request to "fine-tune on our wiki" is asking training to do retrieval's job |
The first row generalizes: for most enterprise document work, the honest answer is buy the commodity layer, code the deterministic layer, prompt the judgment layer, and train nothing.
Deciding, and Re-Deciding
Run the meeting in this order. No approach may be named before question four.
- What are the layers? Nine boxes on the board, not one.
- Which layer are we discussing? Every sentence after this is scoped to one box.
- What are the hard constraints and the error budget for that box?
- What does the evaluation set say?
Decision rights follow the same split: the domain owner sets the error budget and owns the labels, engineering owns the harness and the measured numbers, the budget holder picks among options that cleared the constraints — and nobody overrules a constraint, because a constraint is a fact, not a preference.
Record it where it can be revisited.
task: invoice-line-coding
layer: classify
date: 2026-09-06
constraints:
latency_p95_ms: 5000
residency: EU
error_budget: "2% miscoded, caught by monthly reconciliation"
volume_per_year: 50000
evaluated:
- approach: rules-on-supplier-master
accuracy: 0.71
cost_per_correct_decision: 0.052 # includes rework on 29% misroutes
rejected: "fails on new suppliers and split lines"
- approach: general-llm-with-schema
accuracy: 0.94
escalation_rate: 0.09
cost_per_correct_decision: 0.019
- approach: trained-classifier
projected_accuracy: 0.93
break_even_volume_per_year: 25000000
rejected: "volume is 500x below break-even"
selected: general-llm-with-schema
eval_set: evals/invoice-line-coding-v3 (247 items, 82 held out)
review_on: 2027-03-06
review_trigger: "volume above 2M/year, or p95 breach, or eval accuracy below 0.90"
owner: finance-operations
Every selection expires. Two years ago several tasks now handled by a prompt required a trained model, and several now handled by a small model required a large one. Systems that aged well are not the ones that picked correctly — they are the ones that kept re-picking cheap.
| Practice | Effect |
|---|---|
| Prompts, schemas, and evals in version control | The asset survives any model it was written against |
| One interface in front of every model call | Swapping providers is configuration, not a refactor |
| Re-run the eval quarterly and on any price or model change | Selection stays a standing question |
| A review trigger recorded with the decision | Someone reconsiders on a number, not a news cycle |
| No business logic that exists only inside trained weights | Logic in weights cannot be read, diffed, or reviewed |
The durable assets are the decomposed pipeline, the labeled evaluation set, and the recorded decision — not the model. Models are the most replaceable component in the system, and any architecture that treats one as permanent has locked in the fastest-moving part of the stack.