Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy
This article discusses the Cosmos LLM Gateway created at PayPal to manage enterprise AI traffic across multiple language model vendors. It outlines its architecture, governance policies, and operational efficiency, addressing challenges such as credential management and cost control while ensuring high availability for developers and business applications.