Fireworks: measure cost per completed task, not per token

On October 3, 2026, DEV Community's AI section published an analysis arguing that comparing model prices says little about completed work. The piece is built around a customer comparison released by Fireworks: total token use on a task fell from 49.3K to 29.9K, a 39% reduction, while the task score moved only from 0.751 to 0.753.

The article treats those figures as useful evidence for token efficiency, but not as sufficient evidence for choosing a model in a real tool-using system. Its framing is that model pricing is easy to compare, whereas completed work is harder to compare, and that the missing unit of measurement is an accepted work product. Token counts and task scores, on this account, do not fill that gap.

The report does not identify the models involved, when the comparison was run, the tool-calling setup, or the task type. It also does not specify which side produced the 0.751 and the 0.753. Beyond the two numbers, no further statement from Fireworks is quoted, and the post, timestamped 12:58 on that date, is the only report in this set.

As of this report, the discussion stops at a question about measurement rather than a settled result: token efficiency for the task in question has been quantified, while a unit for accepted work output has not been established, and the article does not propose one.