RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving

Paper Title: RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
Reviewer: The Guardian

---

Summary

The authors present RAGO, a framework for optimizing the performance of Retrieval-Augmented Generation (RAG) serving systems. The paper first introduces RAGSchema, a structured abstraction to describe the diverse landscape of RAG pipelines. Using this abstraction, the authors characterize the performance of four representative RAG paradigms, identifying shifting bottlenecks between retrieval and inference components under various configurations. They then propose the RAGO framework, which performs an exhaustive search over a design space of task placement, resource allocation, and batching policies to find Pareto-optimal system configurations. The evaluation, conducted using a custom simulation framework, claims that RAGO can achieve up to a 2x increase in QPS/chip and a 55% reduction in TTFT compared to a baseline extended from LLM-only serving systems.

Strengths

1. Problem Formulation: The paper correctly identifies a timely and critical problem. As RAG systems move into production, the complexity of their serving pipelines—composed of heterogeneous computational stages—presents a significant optimization challenge. The work does a commendable job of structuring this problem.
2. Workload Abstraction (RAGSchema): The proposed RAGSchema (Section 3.2, page 4) is a logical and useful abstraction. It provides a structured vocabulary for defining and comparing complex RAG pipelines, which is a necessary first step for any systematic analysis.
3. Performance Characterization: The analysis in Section 5 offers valuable insights into RAG system behavior. The identification of shifting bottlenecks (e.g., retrieval dominating in hyperscale scenarios for small models, Section 5.1, vs. encoding dominating in long-context scenarios, Section 5.2) is well-articulated and highlights the core challenge that the paper aims to solve. The sensitivity analyses regarding model size, query numbers, and sequence lengths are particularly informative.

Weaknesses

My primary concerns with this paper center on the methodological rigor of the evaluation and the novelty of the proposed optimization technique. The claims of optimality and significant performance gains rest on a foundation that appears non-verifiable and potentially fragile.

1. Reliance on a Non-Validated, In-House Simulator: The paper's entire quantitative analysis is predicated on an "in-house calibrated XPU simulator" (Section 4, page 6). The authors state it is "well-correlated with the production-grade XPU accelerators" but provide absolutely no evidence to substantiate this claim. There are no correlation plots, error analyses, or quantitative comparisons to real hardware measurements. Similarly, the retrieval performance model is "calibrate[d]... using internal production datasets." This lack of transparency and validation makes it impossible to assess the credibility of the results. The findings could be artifacts of the simulator's specific assumptions rather than reflections of real-world system behavior. Without rigorous validation, the results are fundamentally irreproducible and untrustworthy.

2. "Optimization" via Brute-Force Search: The core of the RAGO framework is an "exhaustive search" (Algorithm 1, page 11). While functional, this is the most basic possible approach to exploring a design space. To present brute force as a novel optimization framework is a significant overstatement. The paper lacks any discussion on the scalability of this search. What is the size of the configuration space for the evaluated workloads? How long does RAGO take to find the "optimal" schedule? A framework that requires hours or days of search to configure a system is impractical. The contribution here appears to be the enumeration of a search space, not a sophisticated method for navigating it.

3. Potentially Weak Baseline: The claimed 2x performance improvement is measured against a baseline described in Section 7.1 (page 11). This baseline collocates RAG components with the LLM's prefix stage and uses a "carefully tune[d]" 1:1 prefix:decode resource ratio. While not a naive strawman, it is questionable whether this represents a strong, state-of-the-art deployment. A skilled systems engineer would likely already consider disaggregating a component known to be a bottleneck (like the long-context encoder in Case II). The significant gains reported may be partially inflated by comparing against a configuration with obvious, well-understood inefficiencies. The paper does not sufficiently justify that this baseline represents a legitimate, production-quality manual optimization.

4. Oversimplification of System Dynamics: The performance models, based on roofline principles (Figure 4, page 6), inherently simplify complex system interactions. For instance, the analysis of iterative retrieval stalls in Section 5.3 (page 9) explicitly assumes zero latency for retrieval and prefix stages to isolate the batching effect. This is an unrealistic condition that likely magnifies the observed slowdown. The model does not appear to account for system-level effects such as network contention, OS scheduler jitter, or nuanced cache interactions, all of which can significantly impact end-to-end performance and invalidate the clean separation of stages assumed by the simulator.

Questions to Address In Rebuttal

1. Simulator Validation: Can the authors provide quantitative evidence validating their XPU and retrieval simulators against real hardware? This should include correlation plots and an analysis of prediction error (e.g., MAPE) across a set of representative model and retrieval configurations.
2. Scalability and Practicality of RAGO: What is the runtime of the RAGO exhaustive search for the case studies presented in the paper? How does the search time scale with the number of pipeline stages and the granularity of resource allocation options? At what point does this brute-force approach become intractable?
3. Baseline Justification: Please provide a stronger justification for why the chosen baseline represents a state-of-the-art, manually optimized system. Why is this specific collocation strategy and 1:1 resource split the correct point of comparison, as opposed to other plausible heuristic-based configurations?
4. Generality of Architectural Conclusions: The analysis is based on a specific family of "XPU" accelerators with parameters detailed in Table 2 (page 5). How would the key findings—particularly the bottleneck locations for each paradigm—change if run on an accelerator with a different compute-to-memory-bandwidth ratio or a different interconnect topology (e.g., an NVIDIA H100 GPU)?
5. Impact of Placement Heuristics: In Section 6.1 (page 10), you state that RAGO restricts collocation to "consecutive neighbors to avoid excessively complicating the search space." How do you know that an optimal configuration does not involve collocating non-neighboring stages? What is the potential performance loss introduced by this simplifying heuristic? This constraint seems to contradict the claim of finding a truly optimal schedule.

Review Form: The Synthesizer

Summary

This paper presents RAGO, a systematic framework for understanding, characterizing, and optimizing the serving performance of Retrieval-Augmented Generation (RAG) systems. The authors' core contribution is a three-pronged approach to tackle the immense complexity of modern RAG pipelines. First, they introduce RAGSchema, a structured abstraction to formally describe the diverse landscape of RAG workloads. Second, using this schema, they conduct a comprehensive performance characterization of four representative RAG paradigms, revealing non-obvious performance bottlenecks that shift dramatically depending on the workload. Finally, they build RAGO, a system optimization framework that uses these insights to explore a vast scheduling policy space—spanning task placement, resource allocation, and batching—to find Pareto-optimal configurations. The authors demonstrate that RAGO can achieve up to a 2x increase in queries-per-second (QPS) per chip and a 55% reduction in time-to-first-token (TTFT) latency compared to a strong baseline extended from LLM-only serving systems.

Strengths

The most significant strength of this work is its conceptual contribution of bringing a principled, systematic methodology to the chaotic and rapidly evolving domain of RAG system optimization. This paper elevates the conversation from ad-hoc tuning of individual components to a holistic, co-design problem.

1. The RAGSchema Abstraction: The introduction of RAGSchema (Section 3.2, page 4) is a standout contribution. In a field where new RAG variants emerge constantly, this abstraction provides a much-needed canonical language for describing and comparing workloads. By capturing key performance-relevant attributes (pipeline stages, model sizes, retrieval parameters), it creates a foundation for reproducible research, benchmarking, and systematic optimization that was previously lacking. It effectively tames the complexity of the problem space.

2. Insightful Workload Characterization: The performance characterization in Section 5 is excellent and provides immense value to the community on its own. By analyzing four distinct paradigms (hyperscale, long-context, iterative, and rewriter/reranker), the authors demonstrate that there is no "one size fits all" solution. The findings—such as the database encoder becoming the bottleneck in long-context RAG (Section 5.2, page 8) or the subtle idleness effects of batched iterative retrievals (Section 5.3, page 9)—are non-obvious and critical for practitioners and system designers. This analysis effectively maps the problem terrain that RAGO is designed to navigate.

3. Holistic Optimization Space: RAGO addresses the full, coupled optimization problem. It doesn't just tune batch sizes; it considers the interplay between task placement (collocation vs. disaggregation), resource allocation across heterogeneous components (CPU servers and ML accelerators), and batching policies. This holistic view is crucial, as the paper shows that decisions in one dimension profoundly impact the others. This connects disparate research threads from LLM serving (prefix/decode splitting) and distributed systems into a unified framework for RAG.

4. Contextualization and Future-Looking Implications: This work provides a clear bridge between the worlds of ML model design, information retrieval, and computer systems/architecture. The finding that retrieval becomes a dominant bottleneck as ML accelerators improve (Figure 7a, page 8) offers a concrete directive for future hardware design, making a strong case for co-designing retrieval and inference accelerators (as explored in works like Chameleon [50]). RAGO provides the analytical framework needed to reason about such future systems.

Weaknesses

The weaknesses of the paper are less about flaws in the execution and more about the boundaries of its scope and potential areas for future expansion.

1. Abstraction of Quality: RAGSchema and RAGO primarily operate on system performance metrics (latency, throughput). While the authors acknowledge that RAG parameters (e.g., number of retrieved documents, percentage of database scanned) affect model quality (recall), this critical quality-performance trade-off is outside the core optimization loop. In a real-world deployment, a user might specify a target recall, which would constrain the search space. Integrating this dimension would make the framework even more powerful.

2. Scalability of the Optimization Search: RAGO relies on an exhaustive search to find the Pareto frontier (Section 6.2, page 11). While feasible for the paradigms explored, this approach may face scalability challenges as RAG systems evolve into more complex, dynamic, and conditional agentic workflows. A discussion on how this framework might incorporate heuristic or learning-based search strategies for more complex future workloads would be valuable.

3. Static Pipeline Assumption: The framework appears to assume a relatively static RAG pipeline defined at the outset by RAGSchema. Agentic systems may involve dynamic, data-dependent execution paths (e.g., deciding to call a tool or perform another retrieval based on the content of a generated token). The current framework doesn't seem to explicitly model this dynamism, which represents the next frontier of complexity.

Questions to Address In Rebuttal

1. The decoupling of performance and quality is a pragmatic choice. Could the authors elaborate on how they envision RAGO being used in a production setting where a product owner might provide a quality constraint, such as a minimum retrieval recall? Would this simply prune the search space, or would it require a more fundamental change to the optimization objective?

2. Your baseline system is a thoughtfully tuned extension of an LLM-only system, which is a strong point of comparison. Could you comment on why the hybrid task placement strategies explored by RAGO (Figure 17b, page 12) are so effective compared to a more naive "collocate everything with the prefix" strategy that one might intuitively adopt?

3. Given the exhaustive search methodology, could you speculate on the computational cost of running RAGO itself? How long does it take to generate a Pareto frontier for one of the case studies, and how do you see this scaling as RAG pipelines incorporate more optional stages and components?

Paper Title: RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
Reviewer Persona: The Innovator (Novelty Specialist)

---

Summary

The authors present a framework for optimizing the serving performance of Retrieval-Augmented Generation (RAG) systems. The work makes three primary claims of contribution: (1) RAGSchema, a structured taxonomy for describing diverse RAG workloads; (2) a detailed performance characterization of four RAG paradigms using this schema, which reveals significant performance variability and shifting bottlenecks; and (3) RAGO, a system optimization framework that performs a systematic search over scheduling policies—specifically task placement, resource allocation, and batching—to find Pareto-optimal configurations for a given RAG workload.

Strengths

The primary novel contribution of this work lies in the synthesis of well-known systems optimization principles and their application to the specific, emerging domain of RAG serving. While the individual techniques employed by the RAGO framework are not new, their holistic integration to navigate the complex, heterogeneous (CPU for retrieval, XPU for inference), and multi-stage pipeline of RAG systems is a novel endeavor.

Specifically, the paper's novelty rests on:

1. A Unified Optimization Framework for a Novel System Class: Prior work has optimized LLM serving (prefix/decode splitting) or retrieval systems (ANN algorithms) in isolation. Other RAG-specific works, such as Chameleon [50] or PipeRAG [51], have proposed point solutions for specific bottlenecks (retrieval acceleration, iterative retrieval stalls). This paper is the first I am aware of to propose a generalizable and systematic framework that co-optimizes the entire end-to-end RAG pipeline, considering the interplay between all its optional and mandatory stages (encoder, rewriter, retrieval, reranker, prefix, decode).

2. Codification of the RAG Search Space: The RAGSchema abstraction, while fundamentally a taxonomy, serves as a necessary and novel contribution in the context of this work. It formalizes the configuration space of RAG pipelines, which is a prerequisite for any systematic optimization. By defining this structure, it enables the RAGO framework to operate methodically, a step beyond ad-hoc optimizations.

Weaknesses

While the application of the framework is novel, its constituent components and underlying methodology lack fundamental novelty. The work is more a feat of rigorous systems engineering and integration than one of conceptual invention.

1. Lack of Algorithmic Novelty in the Optimization Framework: The core of RAGO is a systematic, exhaustive search over a discretized space of scheduling policies, guided by an analytical cost model (Algorithm 1, page 11). This methodology is well-established and conceptually identical to design-space exploration frameworks in other domains. For instance, Timeloop [88] and MAESTRO use this exact approach (analytical modeling + systematic search) to find optimal dataflows for DNN accelerators. The authors have effectively built a "Timeloop for RAG serving scheduling," which is a valuable engineering contribution but not a new optimization paradigm.

2. The "Novel" Decisions are Applications of Prior Art: The key scheduling decisions RAGO explores are direct extensions of known techniques:

   • Task Placement (Collocation vs. Disaggregation): The central placement decision explored by RAGO is whether to group (collocate) or separate (disaggregate) pipeline stages. This directly mirrors the "phase splitting" of prefix and decode stages, a concept already thoroughly explored in prior LLM serving literature such as Splitwise [89] and DistServe [132]. RAGO merely applies this known principle to a pipeline with more stages.
   • Batching Policies: The use of techniques like continuous batching for the decode stage is considered standard practice in modern LLM serving systems like Orca [120] and vLLM [62]. RAGO incorporates this as a known best practice rather than introducing a new batching methodology.

3. Characterization as the Primary Insight: Much of the paper's intellectual weight rests on the characterization study (Section 5, pages 7-9), which demonstrates how bottlenecks shift depending on the RAGSchema. While insightful, a characterization study's novelty is contingent on it revealing profoundly counter-intuitive truths. The findings here—that retrieval can be a bottleneck at hyperscale (Case I), or that a small encoder can be a bottleneck on a huge input (Case II)—are logical consequences of Amdahl's Law applied to a new pipeline structure. They are valuable confirmations but not paradigm-shifting discoveries.

In summary, the paper's novelty is confined to its specific application domain. It does not introduce a new type of algorithm, a new systems primitive, or a fundamentally new theoretical insight into system performance. The contribution is the creation of the first such systematic optimizer for RAG, not the invention of a new kind of optimizer.

Questions to Address In Rebuttal

1. The search methodology in RAGO appears to be an exhaustive search over a pre-defined and discretized policy space. How is this approach fundamentally different from prior design-space exploration frameworks like Timeloop [88], beyond the target domain? Could the authors elaborate on any novel search or pruning strategies that were required to make this exploration tractable for the RAG domain?

2. The concept of disaggregating compute stages with different workload characteristics (compute-bound vs. memory-bound) is central to recent LLM serving systems [89, 132]. Can the authors clarify what new principles of task placement RAGO introduces beyond applying this known heuristic to a wider array of pipeline stages (e.g., rewriter, reranker)?

3. RAGSchema is presented as a key contribution. While it is a clear and useful abstraction, taxonomies themselves are not always considered novel research contributions. Could the authors argue why this abstraction is more than a descriptive framework and constitutes a novel scientific contribution in its own right, perhaps by demonstrating how it enables insights or optimizations that would be impossible otherwise?