Transformer Math

The retriever is the highest-leverage architectural decision in a RAG system — it determines freshness ceiling, latency floor, cost structure, and the quality of context the LLM sees. Each system in this comparison made a fundamentally different bet.

Perplexity runs a proprietary ANN index (community estimate: HNSW-backed, 100B+ passages^ⁱ) over a continuously-crawled web corpus. The key design insight: a purpose-built crawler lets Perplexity control freshness SLA at the index write path rather than relying on Bing's schedule. The trade-off is cost — operating a 100B+ passage index at production QPS requires significant GPU infrastructure for embedding, ANN construction, and query-time similarity search. Index sharding is the dominant engineering problem: with 20+ shards, a scatter-gather fan-out adds 30–80ms^ⁱ to query latency before the reranker even runs. Perplexity's bet: the freshness SLA and citation coverage justify the infrastructure cost over Bing dependency.

NotebookLM uses Vertex AI Matching Engine for semantic search over user-uploaded documents (per Google I/O 2024). The corpus is small by design — a typical NotebookLM session has 10–50 documents^ⁱ, not billions of web pages. This lets Google skip the re-ranker entirely: with 50 documents, the semantic search returns the full relevant set, and Gemini 1.5 Pro's attention mechanism serves as the re-ranker over 100K–1M tokens of context. The weakness: needle-in-haystack degradation. Kamradt's NIAH benchmark shows Gemini 1.5 Pro's recall drops in the middle of very long contexts — facts buried in the center of 500K-token inputs are less reliably surfaced than facts at the beginning or end. For most users (<10 docs), this is invisible. For power users with 50+ long PDFs, it's a real failure mode.

ChatGPT Search outsources the retrieval problem to Bing (per the Microsoft/OpenAI partnership disclosures). This eliminates crawler infrastructure cost and Bing's index freshness is reasonable (1–6h crawl lag, community estimate), but introduces a dependency risk: Bing API availability and Bing's relevance model become part of ChatGPT's retrieval SLO. More importantly, the retrieval path is a black box from OpenAI's perspective — they cannot tune the ANN parameters, freshness prioritization, or passage chunking. This makes it harder to improve citation precision at the retrieval layer; all quality levers live in the re-ranking and generation stages. OpenAI compensates with an internal re-ranker that re-scores Bing results before generation, but the ceiling is bounded by Bing's recall.

Phindmakes the most differentiated bet: a domain-specialized retriever fine-tuned on code QA pairs (per Phind blog, 2023). General-purpose bi-encoders trained on MSMARCO or Natural Questions have poor recall on code-specific queries because the semantic similarity between a question like “how do I avoid deadlock in asyncio?” and the relevant Stack Overflow answer is low in embedding space unless the encoder was trained on that distribution. Phind's encoder was. The result: recall@5 on code queries is substantially better than a general-purpose encoder (community estimate; per Phind blog). The trade-off: the specialized encoder is useless for general web search — Phind cannot compete with Perplexity on news queries and does not try to.

Every RAG system sits on a grounding-freshness Pareto frontier. You cannot simultaneously maximize both without paying a compounding cost. Understanding this trade-off is the move that separates L6 candidates from L5 candidates in a design loop.

Why freshness and grounding are in tension. A fresh source is often a raw crawl of a news article, a live tweet, or a freshly-published page with no editorial review. These sources have higher noise, lower density of verifiable claims, and lower NLI entailment scores against any given query. A high-grounding source is often an established reference — Wikipedia, a peer-reviewed paper, official documentation — which is accurate but slow to update. Perplexity's Pro tier targets both by routing freshness-sensitive queries to live-web fetch (sacrificing some grounding for freshness) and knowledge queries to the vector index (optimizing for grounding). The query classifier that makes this routing decision is the highest- leverage model in the entire system — a 5% misclassification rate compounds across billions of queries per year.

NotebookLM's escape from the Pareto. NotebookLM sidesteps this trade-off by eliminating freshness as a requirement. Its corpus is the user's uploaded documents — static, controlled, and fully within the grounding budget. This is not a limitation; it's a product decision that lets the system optimize exclusively for document fidelity. The result is the lowest grounding failure rate in this comparison (~2–4% community estimate^ⁱ), achieved by removing the variable that causes grounding failures in other systems. The lesson: scope-limiting the corpus is a valid and often optimal architectural choice.

The cost of over-optimizing for freshness. ChatGPT Search's estimated 8–12% grounding failure rate^ⁱon time-sensitive queries (community estimate) is the cost of relying on Bing's crawl pipeline for freshness. When Bing serves a stale or low-quality snippet, the LLM has limited ability to recover — it either grounds on bad context or generates from training memory. Perplexity mitigates this with a higher-quality proprietary crawl and a reranker that down-ranks low-quality sources. The principle: freshness SLA without freshness quality control produces answers that are timely but wrong.

How to quantify the Pareto in an interview. Define a query set with known ground truth across freshness cohorts (1h, 24h, 1 week, >1 month old). Measure citation precision and groundedness score at each freshness tier. Plot the trade-off. A well-designed system shows a shallow slope (freshness degrades gracefully without catastrophic groundedness collapse). A poorly-designed system shows a cliff at 24h — below which grounding collapses. The cliff is usually caused by a reranker that was trained on static document distributions and fails on noisy live-web content.

A counterintuitive fact that every RAG designer eventually discovers: users form trust judgments based on citation UX, not answer quality. A well-cited mediocre answer is trusted. A well-written answer with a missing or wrong citation is distrusted immediately. This has non-obvious implications for architecture.

Sentence-level citations (Perplexity, Phind for prose). Inline[1][2] per sentence is the highest-trust citation format — users can immediately verify every claim without re-reading the full source. The architectural cost: the LLM must be prompted (or fine-tuned) to emit citation markers at sentence granularity, not paragraph granularity. This is non-trivial — standard instruction-following models tend to cluster citations at paragraph or answer level. Perplexity addresses this with a model fine-tuned on their citation format (community estimate). The eval implication: you cannot evaluate citation UX with a generic RAGAS score — you need a citation-placement eval that checks whether markers appear at claim level, not answer level.

Function-level citations (Phind for code). Phind's citation granularity is the function or code block, not the sentence. This matches developer cognition: when a developer asks how to implement a feature, they want to know which package/file/function the pattern comes from, not which sentence in a blog post described it. Sentence-level citations on code answers would fragment the context in a way that hurts usability. The lesson: citation granularity is a UX decision driven by the query type, not a one-size-fits-all retrieval architecture choice.

Quoted passages (NotebookLM). NotebookLM's citation format is a quoted passage from the source document with a page/section reference. This is the highest-fidelity format for document QA — users can see exactly what the system read and judge for themselves whether the quote supports the claim. The trade-off: quoted passages take more screen real estate and slow reading velocity. For a research tool (which NotebookLM is), this is the right trade-off. For a fast-answer product (which Perplexity is), it's wrong.

The citation precision / UX coupling. Here is the non-obvious system design insight: a high citation precision score does not guarantee good citation UX, and vice versa. Precision measures whether the source supports the claim. UX measures whether the user can find and trust that linkage. A system that clusters correct citations at the paragraph level passes NLI precision eval but fails the UX test — users cannot tell which of three sentences in a paragraph the citation applies to. Eval for both separately and set independent thresholds.

The four systems' per-query costs diverge by 3–6x (per the comparison table, all community estimates). Understanding why matters for L6+ design interviews — the cost model constrains which features are economically viable.

Perplexity cost drivers: ANN scatter-gather across shards (compute-proportional to corpus size), cross-encoder reranker on top-50 candidates (small model but runs per query), LLM generation on Sonar/Mixtral (community estimate: smaller, cheaper than GPT-4). Estimated $0.006–$0.012 per query^ⁱ. The dominant cost is the LLM call — reranking is relatively cheap. At 1M QPS, LLM generation costs ~$500K–$1M/day (community estimate, assuming Mixtral at ~$0.0045/1K tok output, 512 tok/query average: 1M × 512/1000 × $0.0045 = $2,304/hour × 24 = $55K/day; multiple by scaling and Pro-tier model use to reach the range). Arithmetic verified.

NotebookLM cost drivers: Vertex AI Matching Engine query (marginal at small corpus), Gemini 1.5 Pro at 128K–1M token context (the dominant cost). A single 500K-token Gemini 1.5 Pro call costs approximately $0.375^ⁱ at Google AI pricing ($0.00075/1K input tok × 500K tok; verified against Google AI Studio pricing page). At $0.008–$0.020/query for realistic corpus sizes (10–50 doc average), NotebookLM is the most expensive per-query but the cost is bounded by corpus size — it does not scale with web index size.

ChatGPT Search cost drivers: Bing API ($0.003–$0.007 per query per Microsoft Bing Search API pricing), OpenAI internal reranker (community estimate), GPT-4o generation at 512 tok average output (~$0.004 at $0.008/1K tok GPT-4o output pricing). Total estimated $0.010–$0.025/query. The Bing API dependency means the cost floor is externally constrained — Microsoft sets the API price.

Phind achieves the lowest cost by serving a narrower corpus (code sites + curated docs) and using a smaller generation model tuned for code. Estimated $0.004–$0.009/query (community estimate). The code corpus has lower coverage requirements than the full web — fewer shards, lower ANN complexity, smaller index size.

The cost-quality interview question. “Perplexity is planning to add a re-ranker that would increase citation precision from 92% to 96%^ⁱ at a cost of +$0.003 per query. At 5M daily queries and 30% gross margin, what's the breakeven?” Arithmetic: additional cost = 5M × $0.003 = $15,000/day. The 4pp precision improvement must retain enough users to cover that. If the product has 500K paying subscribers at $20/month = $10M/month revenue = $333K/day, 4pp precision improvement needs to improve retention by only $15K/$333K = 4.5% of a single day's revenue — almost certainly worth it. This is the form of cost-quality analysis interviewers at Anthropic and OpenAI expect.