Scalable LLM Agent Tool Access in the Cloud
Mingxin Li, Enge Song, Yueshang Zuo, Xiaodong Liu, Rong Wen, Qiang Fu, Gianni Antichi, Jian He, Jing Tie, Zhou Shao, Xiaobo Xue, Xiong Xiao, Luyao Zhong, Shaokai Zhang, Jiangu Zhao, Jianyuan Lu, Shize Zhang, Xiaoqing Sun, Changgang Zheng, Zihao Fan, Haonan Li, Tian Pan, Xiaomin Wu, Yang Song, Xing Li, Biao Lyu, Meng Li, Haipeng Dai, Guihai Chen, Shunmin Zhu
Why It Matters
What makes this one worth your time
This work is relevant for AI engineers and researchers focusing on improving the efficiency and scalability of LLM agents interacting with external tools in cloud environments.
A cloud-scale gateway system enhances LLM agent tool access by optimizing MCP service integration and tool selection.
Summary
The paper presents a cloud-scale gateway system for the Model Context Protocol (MCP) service, addressing challenges in tool access for LLM agents. It integrates legacy services, consolidates MCP variants, and optimizes tool selection, achieving significant improvements in recall, selection time, and token usage.
Key contributions
- Development of a cloud-scale gateway system for MCP service.
- Optimization of tool selection time and token usage for LLM agents.
- Integration of legacy services and consolidation of MCP variants.
Notable insights
- The gateway system breaks the direct-connect model, offloading legacy service integration and consolidating incompatible MCP variants.
- Hybrid retrieval achieves 98% Top-15 recall, significantly improving tool selection efficiency.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.15593v1 Announce Type: cross Abstract: LLM agents increasingly rely on tool calling to act on external systems, and the Model Context Protocol (MCP) has quickly become its de facto interface. Operating MCP at cloud scale, however, becomes difficult. On the tool provider side, legacy services are not directly callable through MCP; the rapid protocol development also creates ongoing compatibility cost. On the agent side, the number of accessible tool is limited by the LLM context window and inference overhead; mounting a large tool set increases token usage and inference latency and can reduce task success rate. Moreover, for stateful MCP backends with multiple replicas, preserving session affinity increases client-side complexity. We present a cloud-scale gateway system for MCP service. It breaks the direct-connect model on the data plane and offloads legacy service integration, consolidating incompatible MCP variants, access control, tool recommendation, and session-aware routing to the gateway. Hybrid retrieval sustains 98% Top-15 recall; it scales agent tool access to 3,000+ with high tool selection accuracy, and reduces tool selection time by $8.9\times$ and token usage by $23.8\times$, with low per-call overhead, stable under scale-out. Finally, we share the lessons learned from deploying the gateway system in production.