Per-use cost changes the economics
An AI feature incurs a variable cost on every interaction: model tokens, and often retrieval, speech or embedding infrastructure on top. A subscription earns a fixed amount per period whatever the usage. Plotted against usage, the two curves can cross earlier than a product plan assumes, and beyond the crossing point the most engaged users are the least profitable ones.
For that reason allowances are a design element of an AI feature, not a restriction added later.
Controls to build before the first prompt
- A cost model per user journey, with the assumptions written down: tokens in and out per step, retries, tool calls, and a worst case as well as an average.
- Hard allowances per tenant and per user, with a clear message when an allowance runs out and a defined route to more capacity.
- A token budget per request and a circuit breaker that falls back to a non-AI path or a cheaper model instead of failing or continuing to spend.
- Usage metering per tenant, so that cost is attributable, reportable and alertable.
- An evaluation set that runs in CI, so that a prompt or model change cannot degrade quality silently. Recording cost per case in the same run makes cost regressions visible as well.
In its simplest form the cost model sums, over every step of a journey, the input tokens at the input rate and the output tokens at the output rate, then adds an allowance for retries. Context that grows with each turn makes later steps more expensive than earlier ones, which is why the worst case has to be modelled alongside the average.
Prompt work starts once these controls exist, because they define the constraints the prompts have to meet.
Levers that lower the cost per journey
- Route by difficulty. A smaller, cheaper model handles classification, extraction and simple replies; the larger model is reserved for the steps that need it.
- Bound the context. History and retrieved material are trimmed or summarised, because input tokens are billed again on every call.
- Cache what repeats. Stable prompt prefixes suit provider-side prompt caching, and repeated lookups suit application-side result caching.
- Move non-interactive work off the request path. Work that does not need an immediate answer can run as a batch, which providers usually price lower.
Human oversight for decisions about people
When an output affects a person's access to an opportunity, such as a grade, progression or admission, the system should assist and a person should decide. Regulation points in the same direction: Article 22 of the GDPR restricts decisions based solely on automated processing that significantly affect an individual, and the EU AI Act classes AI used to determine access to education, or to evaluate learning outcomes, as high-risk and requires human oversight of such systems.
A workable pattern has four parts. The model drafts an assessment together with the reasons for it. A named person reviews the draft and decides. The audit log records the draft, the model and prompt version, the decision, the decider and the time. Where automation is used at all, it is limited to favourable or neutral actions, such as moving a case forward; an adverse decision stays with a person.
Before adding AI to an existing product
Two questions decide readiness. What does the feature cost at ten times the current usage? What does a user see when the budget is exhausted? If either has no answer, the feature is not ready to ship. When the feature influences decisions about people, a third question belongs alongside them: who is accountable for the outcome?
This reasoning draws on Jversity's interview-practice tool, where each automated interview journey incurs a measurable cost in model tokens while subscription revenue stays flat.