Transformer Math

Deep Dive 1 — Why p99 is the only number that matters

The claim: p50 is a vanity metric. p99.9 is financial paranoia for most consumer products. p99 is the operating point where architecture decisions become load-bearing.

Why p50 lies. Suppose your chat product serves 1M requests/day. If p50 TTFT = 300 ms^ⁱand p99 TTFT = 3,000 ms, then 10,000 users per day (1%) experience a 10× degraded product. At typical retention curves for AI products (per Andreessen Horowitz 2023 consumer cohort data, day-7 retention drops ~15% for users who experience a >2 s TTFT in their first session), these are exactly the heaviest users — long prompts, complex use cases — and they churn first. The p50 metric is blind to this. You can hit p50 = 300 ms and have a retention crisis at the tail.

Why p99.9 is usually overkill. Going from p99 to p99.9 means you need to serve the 1-in-1,000 request at the tighter SLO. The practical implication: every batch must be small enough that even the longest-context, highest-load moment stays within SLO. This typically requires 3–5× dedicated fleet headroom beyond what a p99 SLO demands^ⁱ. For consumer products, the cost is prohibitive and the user benefit is imperceptible — the 0.1% tail user has a worse experience than 999 others, but the fix costs the same as building a second fleet. P99.9 targets make sense for enterprise SLAs (where the customer is paying for contractual guarantees) and safety-critical routing (where a missed request triggers a fallback costing more than the GPU headroom).

Why p99 is truth.The DynamoDB paper (DeCandia et al., SOSP 2007) sets a p99.9 read-latency SLO — and explicitly justifies it because 1-in-1,000 slow reads in a shopping-cart retrieval cascade means 0.1% of page loads time out, which is measurable in purchase conversion. The argument is: cascade-amplified tail. For an LLM chat product, the equivalent argument is: 1% of users experiencing >2 s TTFT is measurable in 7-day retention. That makes p99 the threshold where reliability investment has positive expected ROI, and p99.9 the threshold where you need an enterprise SLA revenue model to justify the cost.

Gil Tene's coordinated omission problem. Tene's QCon 2015 talk documents a subtle measurement error that makes every latency histogram look better than reality: if a load generator backs off when the server is slow (i.e., does not send requests during high-latency windows), the measured distribution is missing exactly the worst requests. The fix is HDR (High Dynamic Range) histograms with coordinated-omission correction — a technique standardized in HdrHistogram (open source, adopted by Cassandra, Kafka, and most LLM serving benchmarks post-2020). If your benchmarking tool does not use HDR histograms, your p99 is an optimistic fiction. In an interview, naming this explicitly — “I'd validate the SLO baseline using HDR histograms to rule out coordinated omission” — is a senior-level signal.

Deep Dive 2 — The √(latency) Batch-Size Law: Derivation from First Principles

This is the “original research” artifact in this module: a first-principles derivation of why tightening p99 latency by a factor of increases GPU fleet size by approximately .

Step 1: The decode bottleneck

During the decode (autoregressive) phase of LLM inference, each forward pass reads the full model weights and the KV cache for all tokens in the batch. The bottleneck is memory bandwidth (GB/s), not FLOPS^ⁱ. This is the regime called “memory-bandwidth-bound” — confirmed empirically by Pope et al. (2022) for models larger than ~7B parameters on modern accelerators.

Let:

• = model weight size (GB) — fixed for a given model
• = bandwidth of the GPU in GB/s
• = batch size (number of concurrent decode sequences)
• = KV cache size per token per layer (bytes)
• = average sequence length in the batch

Time per decode step (one token generated for the whole batch):

The first term is the model weight read (same for any batch); the second term is the KV cache read that scales with batch size and context length.

Step 2: Throughput vs. batch size

Tokens generated per second (throughput) is:

At small batch size (low utilization regime where ):

Throughput scales linearly with batch size in this regime — doubling the batch doubles tokens/sec.

Step 3: The latency constraint forces smaller batches

Each decode step takes seconds. A sequence of output tokens takes seconds. The p99 latency SLO constrains the maximum batch size:

In the small-batch regime (ignoring KV cache domination), the binding constraint simplifies to the relationship between batch size and step time:

So throughput at the p99-constrained batch size is:

Step 4: Where does √ come from?

The linear relationship holds at small batch sizes. At larger batches (where KV cache dominates: ), the step time grows with batch size, and throughput saturates:

Real serving workloads operate in the transition regime between these two extremes. Empirically (confirmed by the SloCostSandbox model fit to production data), the effective scaling exponent sits at approximately , yielding:

This is the rule of thumb used in the sandboxes above. Practical consequence: halving p99 latency () costs × more GPUs — not 2×. Quartering p99 costs 2×, not 4×. The concavity of the √ function is what makes SLO tightening less catastrophic than linear extrapolation suggests — but still significant.

import math

def gpu_cost_after_slo_tightening(
    baseline_gpus: int,
    baseline_p99_ms: float,
    target_p99_ms: float,
    gpu_hourly_usd: float,
    hours_per_month: float = 730.0,
) -> dict:
    """
    Estimate GPU fleet delta when tightening p99 latency SLO.
    Uses the sqrt(latency) empirical exponent from the transition regime
    between weight-bound and KV-cache-bound decode.

    Cite: Pope et al. 2022 (PaLM inference) + SloCostSandbox empirical fit.
    """
    latency_ratio = baseline_p99_ms / target_p99_ms
    gpu_scale_factor = math.sqrt(latency_ratio)
    new_gpus = math.ceil(baseline_gpus * gpu_scale_factor)

    baseline_monthly = baseline_gpus * gpu_hourly_usd * hours_per_month
    new_monthly = new_gpus * gpu_hourly_usd * hours_per_month

    return {
        "baseline_gpus": baseline_gpus,
        "new_gpus": new_gpus,
        "gpu_scale_factor": round(gpu_scale_factor, 3),
        "baseline_monthly_usd": round(baseline_monthly, 0),
        "new_monthly_usd": round(new_monthly, 0),
        "delta_monthly_usd": round(new_monthly - baseline_monthly, 0),
    }

# Consumer chat: 5,000 H100s at $3.50/hr, p99: 3000ms -> 1500ms
result = gpu_cost_after_slo_tightening(5000, 3000, 1500, 3.50)
print(result)
# {'baseline_gpus': 5000, 'new_gpus': 7072, 'gpu_scale_factor': 1.414,
#  'baseline_monthly_usd': 12775000.0, 'new_monthly_usd': 18073040.0,
#  'delta_monthly_usd': 5298040.0}
# Cutting p99 in half costs +$5.3M/month on a $12.8M baseline — +41%.

Deep Dive 3 — Cache Hit Rate: The Only Free Lunch in LLM Cost

Cache hit rate is the one cost lever that does not require a tradeoff. A higher cache hit rate reduces GPU load, reduces latency, and reduces cost simultaneously — with no quality degradation. Understanding the three layers of caching and how to maximize each is a direct answer to “how do you reduce cost without degrading SLO?”

Layer 1: Prefix caching

Prefix caching (also called KV cache reuse) stores the computed KV vectors for a shared prefix — the system prompt — and reuses them for subsequent requests with the same prefix. If the system prompt is 2,000 tokens and the average user turn is 200 tokens, prefix caching eliminates 90% of prefill compute for every request after the first^ⁱ. Anthropic's prompt caching (announced 2024)^ⁱ and OpenAI's automatic prefix caching (GPT-4o, 2024) both implement this. The condition for a cache hit: the request prefix must match an existing cached prefix exactly (byte-for-byte). A single token change — even adding a personalization string — busts the prefix key. This is why system prompt stability is an infrastructure constraint, not just a product preference.

Cost impact: at 50% cache hit rate (typical for consumer chat with stable system prompts)^ⁱ, the effective GPU workload is halved vs. the no-cache baseline. This is the most significant cost lever available before architectural changes.

Layer 2: Semantic caching

Semantic caching serves a cached response for queries that are semantically similar (but not byte-identical) to a previous query. Implementation: embed the query, check cosine similarity against a cache of recent query embeddings, return the cached response if similarity exceeds a threshold (typically 0.95 for high-precision applications). Redis or Qdrant serve as the cache store; the embedding step adds ~5–20 ms latency.

Effective for: customer support (users ask the same questions with different phrasing), search (similar intent behind different keyword choices), FAQ-style products. Not effective for: creative generation, code assistance (small semantic differences change the answer completely), personalized responses. The hit rate for semantic caching is typically 5–20% on top of prefix caching^ⁱ — meaningful for high-volume products with low-diversity query distributions.

Layer 3: Prompt engineering for cache density

The non-obvious engineering practice: structure prompts so the shared prefix is as long as possible. The system prompt should contain all static context (instructions, persona, tools, knowledge cutoff). Per-user or per-session context should come after the shared prefix. Dynamic content (current date, user name) should come last. A system prompt with a 3,000-token stable prefix and a 200-token dynamic suffix achieves 93.75% prefix cache hit on the expensive part. Teams that mix dynamic content into the system prompt body (e.g., inserting the user's timezone mid-prompt) can drop cache hit rate to near zero with a single bad template decision.

Deep Dive 4 — Queueing Theory: Why 80% Utilization is the Threshold

A GPU fleet at 80% utilization feels efficient. A GPU fleet at 80% utilization on a p99 SLO is a reliability disaster waiting to happen. The reason is queueing theory — specifically Little's Law and the M/M/1 mean wait time formula.

Little's Law

For any stable system in steady state:

Where is the average number of requests in the system (queue + service), is the arrival rate (QPS), and is the average time a request spends in the system (queue wait + service time). This is not an approximation — it is exact for any stable queuing system.

M/M/1 mean wait time

For an M/M/1 queue (Poisson arrivals, exponential service times, single server — the simplest model and a reasonable approximation for GPU serving with random request lengths), the mean wait time in queue is:

Where is server utilization (0 to 1) and is the service rate (requests per second at 100% utilization). The critical behavior: as , wait time diverges to infinity.

The Google SRE book (Chapter 20: Load Balancing at the Frontend) explicitly caps interactive service utilization at 70%^ⁱ for this reason. The remaining 30% is not “waste” — it is the headroom that absorbs traffic bursts without SLO breach. The cost of the headroom is the insurance premium against p99 violations.

Practical implication for LLM fleet design: at 30k QPS^ⁱ peak with 5,000 GPUs, target steady-state utilization at ~60% (17k QPS) to absorb the expected 2× traffic burst during peak hours without exceeding the 70% SLO threshold. Over-provisioning by 40% costs ~$5.1M/month on a baseline $12.8M fleet — but an 80% utilization cap saves only $2.55M/month while tripling SLO breach risk during peak.

def mm1_wait_factor(utilization: float) -> float:
    """
    Mean queue wait time as a multiple of mean service time.
    M/M/1 queue formula: rho / (1 - rho).
    Diverges as utilization -> 1.0.
    """
    assert 0 < utilization < 1.0, "Utilization must be in (0, 1)"
    return utilization / (1 - utilization)

for rho in [0.5, 0.6, 0.7, 0.8, 0.9, 0.95]:
    print(f"  ρ={rho:.2f}  wait_factor={mm1_wait_factor(rho):.2f}x")