[USE CASE]

Serve LLM Inference in Production

High-throughput, low-latency serving for 8B–70B models.

Production LLM inference is dominated by KV-cache memory, not compute. For 70B models use H200/B200 or MI300X; for 7B–13B models L40S delivers the best $/token. Plan for vLLM or TensorRT-LLM with continuous batching.

Recommended
8× NVIDIA H200 (70B) or 4× L40S (7B–13B)
Est. cost
$0.40 – $2.10 per million output tokens
Duration
Steady-state

Step-by-step

  1. 1. Pick GPU by KV cache
    Compute KV bytes per token × context × concurrency. Pick the cheapest GPU that fits.
  2. 2. Deploy vLLM or TensorRT-LLM
    Continuous batching + paged attention. FP8 quantization on H200/B200.
  3. 3. Route by region
    Servers.computer routing layer picks the lowest-latency endpoint per request region.
  4. 4. Autoscale
    Scale by p95 latency, not GPU utilization.

FAQ

What's the cheapest GPU for Llama-3 70B inference?
AMD MI300X and NVIDIA H200 are roughly tied at ~$0.50 per million output tokens at scale. L40S can't fit 70B; use H200 minimum.
Should I use vLLM or TensorRT-LLM?
TensorRT-LLM wins on raw throughput on NVIDIA. vLLM wins on developer velocity and supports more model families. Both are production-grade.
Definition

What is Servers.Computer?

Servers.Computer is an AI compute routing and procurement layer that benchmarks, compares, and deploys GPU clusters (NVIDIA H100, H200, B200 and AMD MI300) across global cloud providers in real time.

Other use cases
Train a 70B Parameter LLMDeploy AI Agents in Production