LLM cost and capacity: deployment, serving and routing

A support service answers short policy questions during the day and classifies long document batches overnight. One workload needs a quick response; the other can wait in a queue. Choosing a model by its token price misses that difference. The useful comparison is the cost and capacity required to complete authorized business tasks at the accepted quality and latency.

Count all the work needed for a valid outcome#

Define a successful task first. For the illustrative support service, it is a supported answer using permitted evidence within the agreed response window, or an appropriate escalation when evidence is absent. A fluent but unsupported answer is not a low-cost success. Failed and retried work remains part of the bill even if it produces no useful result.

Over a stated period, divide the total allocated cost by the number of valid completed tasks. State how shared infrastructure and staff time are allocated so comparisons remain meaningful. Also report completion rate and the kinds of failures: the same cost per success can conceal a design that fails many more incoming requests.

Apply eligibility before choosing the cheaper model#

First determine which routes are allowed: data region, provider approval, sensitivity, context fit, required modalities, tool support and output constraints. Then choose among eligible routes. A cheaper fallback must not silently send confidential data to an unapproved service or remove a required validation step. Keep these conditions in application policy rather than asking the model to choose its own permissions.

Routing can be a deterministic rule based on task type, a trained classifier, a model-selected decision or a sequential fallback after a first attempt. These have different overhead and failure modes. RouteLLM studies quality-cost routing between model choices using preference data; it does not establish that a router trained elsewhere will save money for your documents or risk categories.

Work through the first-attempt trap#

Consider a simplified route that always calls a router and a small model, then escalates some tasks to a large model. Its expected inference cost is C_router + C_small + p_escalate × C_large. The expression deliberately omits validation, caching changes, retries and human correction; include those in a real budget. It is not a formula for every routing architecture.

Under these toy assumptions and equal accepted quality, the routed design costs less only when escalation is below 75%. That boundary changes when any omitted cost changes. A missed escalation can look cheap while returning a wrong answer; an unnecessary escalation can waste both calls. Sequential fallback also makes escalated requests wait for the first attempt before the second begins.

Bound retries by attempts, elapsed time and overall task budget. Retry only errors that the operation contract makes safe. During a provider outage, synchronized retries can increase queue pressure. Use controlled backoff, admission limits and an approved fallback or explicit temporary failure rather than allowing a model loop to keep spending indefinitely.

Compare complete routes at the same quality floor#

Evaluate a direct stronger model, a direct smaller model and the routed candidate on the same held-out tasks. Stratify by sensitivity, difficulty, input size, missing evidence and required tools. Measure supported-answer quality, missed and unnecessary escalations, total accepted-task cost, timeout rate and end-to-end latency. Keep the router, prompt, model and corpus versions with the results.

Freeze acceptance rules before observing cost improvements. Recheck routing after changes to either model, because the old classifier’s boundary may no longer match their strengths. Inspect tasks close to the routing threshold and failures where the cheaper model confidently produced unsupported output. A router should improve the system outcome, not merely move traffic away from an expensive endpoint.

Choose deployment by workload and ownership#

These are architectural patterns, not a universal ranking or an official list of fixed deployment formats. Amazon Bedrock’s batch documentation illustrates asynchronous processing with model and region constraints. It does not imply that every model has the same completion deadline or that batch is appropriate for a live support chat.

Self-hosting can reduce marginal serving cost at sufficient useful utilization, but idle capacity, engineers, upgrades and failure recovery still cost money. A managed API can simplify operations while creating quota and dependency constraints. Compare both under the same arrival pattern and quality requirements; do not compare an API’s actual bill with an optimistically saturated GPU estimate.

Price only routes that satisfy the data contract#

Document where prompts, outputs, uploaded files, cache state and logs are processed and retained. Review the provider account configuration and contractual controls for the actual service and region. A model available in one region may not be available in another, and a fallback may use different retention settings. Do not infer data isolation from a marketing label such as private deployment.

For self-hosting, control operator access, network egress, storage encryption, secrets, patching and deletion of derived data. For hosted services, minimize transmitted context and avoid unnecessary personal information in telemetry. Both require application-level tenant and document authorization; the hosting choice does not replace it.

Plan beyond the model weights#

Serving memory includes weights, KV cache, runtime and activation overhead, batching requirements and headroom. Longer contexts and more concurrent sequences compete for KV capacity. vLLM’s optimization documentation discusses memory pressure, preemption and scheduling; the right settings depend on the installed version, hardware, model and request distribution. A claim that weights fit on a device is not a capacity plan.

Quantization can reduce storage and memory use, but bit width alone does not guarantee speed or preserve every task behavior. Hugging Face’s documentation describes multiple methods with runtime and hardware considerations. Evaluate the quantized artifact on the actual business tasks and serving kernels before treating memory savings as an operational win.

Measure time to first token, output generation rate and full response latency separately. Use representative input and output lengths, bursts, concurrency, cold starts and warm steady state. Report tail latency and queue depth, not only average throughput. A benchmark with tiny prompts cannot predict a queue of long contract analyses.

Cache reuse changes work, not truth#

vLLM prefix caching reuses compatible prefill computation; it does not reduce the work of generating new output tokens. Claude’s prompt-caching documentation illustrates provider-specific write, read and lifecycle rules. Account for cache population, refresh and eviction instead of assuming every hit is free or equivalent to an end-to-end latency reduction.

A source update or permission revocation may correctly lower the hit rate. Preserve tenant, source and policy compatibility before reusing private answers or context. Semantic answer caching has an additional correctness question: similar wording may refer to a different account or effective date. Measure false reuse as a defect, not as an attractive cost saving.

Build a worksheet from the workload#

Record arrival rates by task type, input/output distributions, acceptable wait time, peak concurrency, quality floor, refresh frequency and expected escalation. Measure service time under that mix, then test overload behavior. Quota is a provider limit, reserved capacity is a commercial allocation and observed throughput is a measurement; do not treat them as interchangeable numbers.

Add per-tenant quotas, admission control, timeouts, spend alerts and a stop control for costly routes. Ensure one long-running customer task cannot consume all available concurrency. Define what the user sees when capacity is exhausted: a queue position, a bounded retry window or a controlled failure. An unbounded hidden queue simply converts insufficient capacity into unpredictable latency.