Apple and CMU Cut Agent Reasoning Latency by 90% with SSR Framework

Inference latency per round in multimodal search reduced by over 90%.

In typical agentic workflows, the model generates a segment of free-form thinking before executing each action: analyzing current observations, evaluating existing evidence, and then deciding on the next step. This "thinking" process is central to the ReAct paradigm but incurs significant inference overhead.

A research team from Apple and Carnegie Mellon University recently proposed a new framework called SSR (Selection-based Structured Reasoning), which redesigns this component. Instead of generating reasoning token-by-token, the model "selects an answer" from a set of predefined reasoning options. Experimental results show that this method reduces per-round inference latency by over 90% across seven multimodal search benchmarks while maintaining task success rates comparable to free-form thinking.

• Paper Title: Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents
• Affiliations: Apple, Carnegie Mellon University
• Paper Link: https://arxiv.org/abs/2610.01892

I. Motivation: Is Small Model "Fluff" Worth It?

Currently, most search agents follow an interaction paradigm similar to ReAct: Observation → Reasoning → Action, repeating this cycle until the task is complete.

Consider a multimodal search question: The model sees a photo of a building, and the query is, "What exhibition will be held in this building next?" A typical agent first generates free-form thinking: "This image looks like some museum... I need to identify the building first... then search for its recent exhibitions..." Subsequently, it constructs a search query based on this thought and calls the tool.

However, this "think-before-act" pattern reveals a clear cost-benefit issue with small models. Researchers observed that edge-side small models (e.g., 2B, 4B parameters) sometimes generate lengthy reasoning content that does not necessarily contribute to generating the next action or making the final decision.

More critically, by analyzing numerous search trajectories, the research team discovered that although entities, queries, and observations vary significantly across questions, the high-level reasoning logic determining the next action recurs frequently. For example, two seemingly unrelated questions—"Where is a certain AI conference being held?" and "When was LEGO founded?"—share the same underlying reasoning logic: "The entity is known, but the target information is missing; use the entity name plus the target attribute to search."

This implies that the reasoning space for search agents does not need to cover the entire natural language space. Rather than generating from scratch every time, it is better to predefine a set of reusable reasoning options and let the model perform reasoning as if answering multiple-choice questions.

Switching Reasoning to Multiple-Choice Significantly Improves Inference Speed

II. Methodology: Selection-Based Structured Reasoning

2.1 Reasoning Library and Selection-Based Reasoning

The core of SSR is a reasoning library containing six reasoning options:

1. Use image-to-image search to identify entities when uncertain about them in the image.
2. Use text search to query specific facts when needed.
3. Crop or zoom into local regions of the image when detailed inspection is required.
4. Answer based on collected evidence once sufficient evidence is gathered.
5. Directly read the image to answer if the answer is present within it.
6. Answer directly using common sense if applicable.

During each interaction round, the Agent no longer generates reasoning word-by-word. Instead, it selects one option from the reasoning library as the current round's reasoning, and then generates specific actions based on the selected reasoning, such as what query to use for this search or which bounding box to crop. This design decouples reusable reasoning options from instance-specific execution details: the options provide the rationale for "why do this," while action generation handles "how specifically to do it."

2.2 Using the Model Itself for Scoring, No Additional Classification Head Needed

The key question is: How to select? The research team’s approach is to let the LLM itself act as the classifier. For each reasoning option, they calculate the average probability based on the current context; then apply softmax to obtain a categorical distribution over the reasoning library. Sampling from this distribution completes the reasoning selection. The entire selection process requires no additional task heads and no SFT warm start. Scoring and action generation share the same model parameters and system prompt. This effectively uses the base model's inherent semantic priors as the reasoning selector: leveraging the model's "understanding capability" to choose the most appropriate reasoning option based on the current context.

2.3 Parallel Reasoning Decoding: "Parallel Decoding" for Reasoning

This is the direct source of SSR's efficiency advantage. Since all reasoning options are predefined, scoring can be fully parallelized. At the token level, the entire option is prefilled at once via teacher forcing, allowing the likelihood of all tokens to be computed in a single batched pass. Consequently, the entire reasoning selection process does not require sequential token generation.

For example, consider a scenario where a 2B model needs to determine whether to use image search or text search. Traditional free-form reasoning might take hundreds of milliseconds to generate intermediate thoughts. With SSR, the model evaluates both predefined options simultaneously in parallel decoding, reducing the decision time to mere milliseconds.

2.4 Training Strategy

SSR employs a lightweight training strategy. The model is fine-tuned on data consisting of (context, selected reasoning option, subsequent action) triplets. During training, the loss function focuses on maximizing the probability of the correct reasoning option given the context, while also ensuring the generated actions align with the selected reasoning. This joint optimization ensures that the model learns to map contexts to appropriate reasoning strategies efficiently.

III. Experimental Results

The researchers evaluated SSR on seven multimodal search benchmarks, including InfoSeek, SimpleVQA, and others. Key findings include:

1. Latency Reduction: SSR achieved over 90% reduction in per-round inference latency compared to baseline ReAct agents using free-form thinking.
2. Accuracy Maintenance: Task success rates remained comparable to those of full free-form reasoning baselines, demonstrating that structured selection does not compromise performance.
3. Scalability: The benefits were particularly pronounced for smaller models (2B–7B parameters), where the overhead of verbose reasoning is proportionally higher.

Table: Performance comparison across different model sizes.

IV. Conclusion

SSR demonstrates that efficient multimodal search agents do not necessarily require extensive free-form reasoning. By structuring the reasoning process into a selection problem over predefined options, we can drastically reduce latency without sacrificing accuracy. This approach is particularly valuable for deploying intelligent agents on resource-constrained devices, offering a practical path toward real-time, responsive AI systems.

Future work may explore expanding the reasoning library to cover more complex scenarios and integrating dynamic option generation to handle novel situations not covered by static templates.

SSR is a general-purpose alternative to inference interfaces that can be integrated into existing SFT and RL training pipelines. A key advantage of SSR is that, since inference is constrained within the reasoning library, it can be trained directly via RL without requiring an SFT warm start. Many existing approaches first use SFT to teach the model "specific thinking patterns" before applying RL. SSR not only eliminates the data construction and training time associated with the SFT stage but also makes model behavior more predictable and controllable.

III. Experiments: Comparable Accuracy, 10x Faster Inference

The research team evaluated performance on Qwen3-VL 2B/4B models using H100 GPUs and the SGLang inference engine across seven multimodal search benchmarks: MMSearch, HR-MMSearch, FVQA-test, InfoSeek, SimpleVQA, LiveVQA, and MAT-Search.

• Comparable Search Accuracy: The average success rate for 4B SSR (GRPO) reached 61.37%, matching state-of-the-art (SOTA) results at the same scale. The 2B SSR achieved 51.26%, comparable to zero-shot 8B models. Across four training methods—GRPO, GSPO, SAPO, and SFT—SSR maintained success rates equivalent to free-form thinking versions.
• Significant Speed Improvements: Per-round inference latency decreased by over 90%, while total model inference latency per question dropped by 28–54%. Compared to baselines such as Chain-of-Draft, Sketch-of-Thought, efficiency-reward RL, and Probe & Prefill, SSR achieved the highest success rate with a p95 inference latency of just 0.07 seconds (versus 0.8–1.9 seconds for baselines), demonstrating greater stability.

Key findings from ablation studies include:

• Semantics Matter. If the model is restricted to selecting indices (e.g., "[1]") instead of full text, the average success rate drops by 3.9%, with performance degradation observed across all seven benchmarks. This indicates that natural language scoring leverages the model's semantic understanding for context matching, rather than learning a pure index classifier.
• Monotonic Gains from Reasoning Library Size. Increasing options from one generic choice to six raised the average success rate monotonically from 37.24% to 61.37%. Notably, expanding from one to two options (distinguishing between "answering" and "tool invocation") yielded a 17.27 percentage point jump, proving that more diverse and fine-grained reasoning options better guide action generation.
• Adaptive Selection. The same checkpoint exhibits different selection distributions across benchmarks: "Answer based on image" dominates in SimpleVQA (which leans toward simple factual QA), while "Inspect image details" is most frequent in HR-MMSearch (focused on high-resolution images).
• Comparison with Autoregressive Versions. Experiments revealed that training a model to generate reasoning library options token-by-token autoregressively requires SFT to enforce format compliance. RL alone, using prompts and format rewards, cannot prevent the model from "hallucinating" or deviating from constraints. In contrast, SSR mechanistically guarantees that inference results must originate from the predefined reasoning library.

IV. Conclusion: Redefining the Reasoning Space

The research team suggests that the significance of this work extends beyond simply "shortening CoT." It poses a more fundamental question: Must the representation of reasoning rely exclusively on unrestricted natural language generation?

For open-ended mathematical reasoning, theorem proving, or code design, the expressive power of free-form thinking is crucial. However, in multi-turn search and tool-use scenarios, agents typically face only a few high-level decisions at each step: Is current information sufficient? Do entities in the image need identification? Should searching continue? Does a local region need zooming? The parts highly dependent on specific inputs—such as search queries, tool arguments, crop coordinates, and final answers—are still generated freely by the model. Strategic-level reasoning, however, does not necessarily require regeneration within the full language space at every step.

Therefore, SSR should be understood as a reorganization of the reasoning space: transforming from a completely open language space into a compact, reusable, and semantically clear strategy space. The model has not stopped thinking; it has merely replaced part of its token-by-token generative reasoning with selection and inference within a structured reasoning space.

© THE END

Please contact our official account for authorization to reprint.

Submissions or media inquiries: [email protected]