Why Companies Are Overpaying for Routine AI Work

Routine AI work is often sent to the most capable model a company has approved. The employee choosing it rarely sees the price difference, so the most expensive option becomes the default before the company has shown that the task needs it.

One request barely registers. The same choice repeated across teams and millions of rows becomes a material operating expense. We wanted to know how much of that premium buys a better result.

The test: 13 models, one task

The task in this study is single-label classification: read a piece of text, pick one label from a fixed list. It is the quiet workhorse behind support routing, review tagging, survey coding, and taxonomy work, and it runs at volumes where model choice shows up on the bill. It is also unusually measurable. Every row has a right answer a person can verify, so accuracy is a score, not an opinion.

Cost is only one part of the model decision. Customers also need to know where company text is processed. A managed API sends that text to a provider or host. Open weights can run inside an approved environment, but the company then takes on the infrastructure and its operating cost.

To measure the tradeoff between accuracy and cost, we compared 13 models from Anthropic, Google, OpenAI, and Qwen on the same 200 labelled rows. Every model received the same prompt and label definitions, and we counted every token.

The comparison ran proprietary models against open-source and open-weight models. We sampled 50 rows from each of four datasets, for 200 unique texts and at least 2,600 primary classification decisions across the 13 models. Separate paired runs tested the effect of model reasoning.

We counted input, visible output, and hidden reasoning tokens, then applied prices captured on 1 August 2026. The benchmark did not measure response time, throughput, or security. With only 50 rows in each dataset, one answer moves a dataset score by two points. Small gaps should be treated as ties.

User:
You are an expert text classifier. Analyze the given text and assign the ONE label that best fits.

## AVAILABLE LABELS
[the dataset's labels, one line each with a short gloss]

## INSTRUCTIONS
1. Read the text carefully.
2. Choose the SINGLE most appropriate label from the available labels.
3. Assign exactly one label — never zero, never more than one.
4. Be specific but concise in your reason.

## OUTPUT (JSON)
Return a JSON object with an "assigned_labels" array containing EXACTLY ONE item (assign the single best-fitting label). The item must have these fields in this exact order:
- reason: Why this label fits (ONE sentence, max 25 words)
- confidence: Integer 0-100
- label_name: The exact label name from the available labels

Example:
{ "assigned_labels": [{ "reason": "The text describes the topic", "confidence": 92, "label_name": "ExactLabelName" }] }

## TEXT TO CLASSIFY
[the row's text]

Cheaper models held up on routine text

Mean accuracy ranged from 74% to 89%. Ten models finished within four points of Opus, the leader. Their estimated hosted bills looked nothing alike: the lowest was $51 per million rows and Opus was about $9,100.

Hosted gpt-oss-120B scored 83.5% for about $60 per million rows. Qwen 3.6 27B and GPT-5.6 Luna each scored 85.5%, at roughly $382 and $250. Those averages trailed Opus by 5.5 and 3.5 points, respectively. The map below shows how much providers charged for those differences.

Long complaints changed the answer

On the three shorter datasets, the cheap end of the field kept up with the most expensive model. Qwen 3.6 27B and GPT-5.4 Nano beat Opus on AG News, 86% to 84%. GPT-5.6 Luna, at $250 per million rows, beat it on Banking77, 90% to 88%. On DBpedia, eleven of the thirteen models scored 100%, including gpt-oss-120B at $60. The CFPB complaint dataset broke the pattern. Its texts had a median length of 812 words, some ran beyond 2,400, and accuracy spread from 66% to 84%. Opus led alone, four points ahead of the next model.

Opus cost an estimated $14,600 per million CFPB rows. That is a large bill, but this was also the one dataset where the strongest model created a clear accuracy advantage. A company processing long complaints may decide the reduction in routing errors is worth the price. The shorter datasets did not support the same decision.

The result suggests a two-tier system: use a cheaper model for routine rows and send the difficult slice to a stronger one. We did not test that combined system. It needs its own labelled evaluation, including a reliable rule for deciding which rows move to the second tier.

Thinking added cost, not much accuracy

Some models reason internally before returning an answer. Those tokens are hidden from the user, but providers still bill for them as output. Every reasoning model used its largest reasoning budget on the long CFPB complaints.

Across eight paired runs, mean accuracy rose 0.4 points. Six models moved one point or less. Qwen MoE gained 3.0 points; Qwen 27B lost 2.5. Cost rose in every pair, from 6% for Sol and Terra to 442% for Qwen MoE. For this classification task, thinking should be an evaluated option rather than an automatic setting.

Things to call out in this study

• →The sample is small. Each dataset contains 50 rows, so half-point differences should not choose a vendor. Treat the wide cost range and the 18-point CFPB gap as reasons to run a larger test on your own text, not substitutes for one.
• →This measured one task. Single-label classification results say nothing about summarization, writing, or extraction. Those workflows need their own labelled or human-rated evaluation.
• →Qwen3.8-Max-Preview arrived as this report closed. It is a hosted preview model and did not run through our harness, so we have no classification evidence yet. It enters the table after it completes the same test.
• →Model origin is a policy question the table cannot answer. Customers have asked whether future rules could restrict Chinese-developed models. Origin, licence, deployment host, and data location sit with legal and security teams.
• →Self-hosting carries its own bill. Every cost here is a hosted API price. Running an open-weight model yourself adds GPUs, engineering, and on-call time, and below a certain volume that overhead can cancel the per-row savings from switching models.

A framework for choosing a model

Use the same process for every recurring AI workflow. Start with the business error you can tolerate and a labelled sample that reflects production. Model names and provider rankings come after that work.

Start with the least expensive deployment that meets the accuracy bar. Pay for a stronger model only where measured errors justify it.