Transformer Math

Deep dive A — DiT architecture and step-budget economics

Sora represents a departure from U-Net diffusion architectures (used in Stable Diffusion 1.x/2.x and DALL-E 2) toward the Diffusion Transformer (DiT) family introduced by Peebles and Xie (2023, arXiv:2212.09748). The architectural choice is load-bearing for every infra decision downstream.

• Spatial-temporal patch tokenization. Sora compresses raw video frames into a lower-dimensional latent space using a video VAE (spatial compression 8× in H and W, temporal compression 4× in T^ⁱ — per OpenAI research notes). The DiT then treats non-overlapping spacetime patches of this compressed latent as tokens, analogous to how a ViT treats image patches. For a 10 s clip at 24 fps compressed to 60 temporal frames at 128×72 spatial resolution, the token count is approximately 60 × (128/8) × (72/8) = 60 × 16 × 9 = 8,640 tokens per forward pass. Compare to SDXL's image DiT operating on roughly 256–1,024 tokens: Sora's forward pass processes 8–34× more tokens, which dominates GPU memory bandwidth requirements. (Arithmetic verified; VAE compression factors per OpenAI research notes.)
• Why transformers beat U-Nets at video scale. U-Net architectures use a hierarchical encoder-decoder structure designed for spatial 2D data. Extending to 3D video requires factored spatial-temporal attention (separate attention over space and time) or full 3D convolutions — both of which either sacrifice cross-frame coherence or scale poorly in memory. A transformer backbone with full self-attention over the spacetime-patch token sequence naturally captures long-range temporal dependencies (a camera panning over a scene can reference a frame 8 seconds prior) without architectural surgery. DiT also scales more predictably: the original paper (Table 1) demonstrates that FID improves log-linearly with model size, enabling confident compute-quality tradeoffs at design time.
• Step-budget economics — the 50-step multiplier. Each of the 50 denoising steps^ⁱ runs one full DiT forward pass over all 8,640 tokens. An H100 80 GB has ~2 TB/s memory bandwidth; at FP16, reading a 7B-parameter DiT once takes roughly 7 GB ÷ 2 TB/s = 3.5 ms — but with batch size 1 and attention over 8,640 tokens, attention alone consumes O(n²) memory proportional to ~75 M attention cells per layer × 48 layers ≈ 3.6B operations per step. Empirically (community estimate from DiT benchmark data), a single step for a 10 s clip on H100 takes approximately 2–3 s. 50 steps × 2.4 s/step = 120 s total^ⁱ — consistent with the observed ~2 min generation time. This is 60× slower than a typical SDXL image generation (50 steps × ~0.04 s/step), reflecting the ~34× token count increase plus sequential attention costs. (Community estimate; verify against your target hardware.)
• The “fail at step 48” problem is much worse for video. At 2.4 s/step, a failure at step 48 wastes 48 × 2.4 s = 115 s of H100 time — roughly $0.11 per failure at $3.50/hr. At 20 concurrent generations with a 0.5% failure rate, the expected wasted GPU cost is 0.005 × 20 × $0.11/failure = $0.011/s = $660/hr of pure waste. Step-level checkpointing every 10 steps reduces the expected waste to 0.005 × 20 × (10/50) × $0.11 = $132/hr — a 5× reduction. (Arithmetic verified: 660 ÷ 5 = 132.) This is a direct business case for checkpointing that management can evaluate.
• Real-world example — DiT scaling.The original DiT paper (Peebles & Xie, 2023) evaluated 12 DiT configurations from DiT-S/8 (33M params) to DiT-XL/2 (675M params)^ⁱ and demonstrated that FID on ImageNet drops from 68.4 to 2.27^ⁱ as model size increases — a 30× quality improvement for a 20× parameter increase. Sora's production model is substantially larger than DiT-XL/2 (community estimate: several billion parameters), but the scaling relationship gives confident a priori design guidance: if quality is insufficient, scaling the model is the highest-leverage intervention, not adding denoising steps past 50 (diminishing returns above ~40 steps per DDPM ablations). See: arxiv.org/abs/2212.09748, Table 1.

Deep dive B — Latent diffusion vs pixel diffusion at video scale

The choice between latent-space diffusion (Rombach et al., 2022 — the architecture behind Stable Diffusion and Sora) and pixel-space diffusion (DDPM, DALL-E 2) is the most consequential architectural decision in video generation, and the reasoning is different from the image case because the data dimensionality is orders of magnitude higher.

• Why pixel diffusion fails at video scale.A 1080p 24 fps 10 s clip is 240 frames × 1920 × 1080 × 3 channels = ~1.5 GB of raw pixel data. Running the diffusion noise schedule directly on this space would require processing 1.5 GB per denoising step × 50 steps = 75 GB of GPU memory throughput per generation — far exceeding H100's 80 GB VRAM for a single clip, let alone a batch. Pixel-space diffusion is computationally infeasible for full-resolution video.
• The latent diffusion solution. A video VAE (variational autoencoder) compresses the clip to a much smaller latent representation (spatial 8× + temporal 4× = 32× total compression^ⁱ). The same 1.5 GB clip compresses to approximately 47 MB of latent^ⁱ — small enough to fit multiple clips in H100 VRAM simultaneously. Diffusion runs entirely in this compressed latent space; only the final decode step (the VAE decoder) maps back to pixels. The decoder is a single pass, not 50 steps, so its compute cost is negligible relative to the denoising loop. (Arithmetic verified: 1.5 GB ÷ 32 ≈ 47 MB. VAE compression factors per OpenAI research notes.)
• Tradeoff: VAE bottleneck artifact rate. The 32× compression^ⁱ introduces a fidelity ceiling — fine-grained details (text in video, individual hair strands, fast motion) that fall below the VAE's spatial resolution are permanently lost and cannot be recovered by more denoising steps. This manifests as blurry text and motion blur artifacts in Sora outputs — a known limitation acknowledged in the OpenAI research notes. From an eval perspective: add a text-legibility test to the weekly quality eval (generate a clip containing on-screen text; score whether the text is legible at 1× zoom). A regression in this metric points to a VAE quality issue, not a DiT quality issue.
• The serve-time implication. The VAE decoder is a fixed-cost operation at the end of generation (~2–5 s on H100 for a 10 s clip at 1080p — community estimate). It is not parallelizable with the denoising loop and is always on the critical path. Include it in end-to-end latency budgets. A single H100 that takes 120 s for denoising + 5 s for decode has a p99 of 125 s, not 120 s — relevant when every second matters for Plus-tier SLO.
• Real-world example — latent diffusion scalability. Rombach et al. (2022, §4.3) demonstrate that latent diffusion reduces training compute by 2.7× and inference memory by 4× vs pixel-space diffusion at 256×256 resolution. At 1080p these savings are proportionally larger because the pixel-space dimensionality grows with resolution squared. The paper also benchmarks the VAE reconstruction quality (PSNR, SSIM) as a function of compression ratio, showing that 8× spatial compression achieves near-lossless reconstruction on natural images. The practical limit is text rendering, which requires fine spatial structure the VAE cannot preserve at 8×. See: arxiv.org/abs/2112.10752, §4.3.

Deep dive C — Video safety failure taxonomy (original research)

Video safety has five distinct failure modes that do not all appear in image-generation safety stacks. Each requires a different classifier architecture and a different eval golden set.

The key infra insight: CSAM requires every-frame scanning (non-negotiable, adds ~1 s latency for 240 frames on GPU); celebrity likeness and NCII can be sampled (every 5th frame) to contain latency; disinformation deepfakes cannot be caught at generation time and require platform-level provenance infrastructure (C2PA) as the primary mitigation. Design the post-filter pipeline accordingly: CSAM gate runs first and is blocking; other classifiers run in parallel on the sampled-frame stream.