The emergence of generative language models (LLMs) has forced API and AI managers to rethink how they manage the exposure, consumption, and security of these services. The nature of LLMs introduces technical peculiarities, such as token limitations, the need for semantic moderation and prompt engineering, which traditional API managers were not designed to cover. In this context, Kong and WSO2 have evolved their platforms to respond to the new demands, albeit with different approaches and levels of maturity.
Kong Gateway: ecosistema de plugins para LLMs
Kong has opted to extend its gateway through an ecosystem of specialised plugins. Its AI Proxy allows interaction with multiple providers (OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Mistral, Hugging Face, among others) to be standardised, creating a layer of abstraction that reduces technological dependency and allows models to be changed without impacting consumers. For more complex environments, the AI Proxy Advanced version adds intelligent balancing based on latency, cost or request semantics, as well as fallback capabilities.
In terms of consumption control, AI Rate Limiting Advanced introduces token- and cost-based quotas, with explicit headers to inform the customer of their consumption status. Content moderation reaches a higher level with AI Semantic Prompt Guard and integration with Azure Content Safety, allowing prompts to be blocked based on semantic similarity. In terms of observability, Kong exposes detailed metrics in Grafana and offers unified logging for all AI interactions. Finally, it highlights prompt engineering using templates and decorators, AI Semantic Cache to optimise repetitive responses, and request and response transformers that enable dynamic mediation with AI.
Discover our comprehensive ebook on managing LLMs in API Managers
WSO2 API Manager: initial integration with LLMs
For its part, WSO2 has introduced support for AI APIs in version 4.4.0, albeit in a more basic form. It allows you to create AI APIs from providers such as OpenAI, Azure OpenAI and Mistral, as well as configure customised vendors. Unlike Kong, it does not apply universal standardisation to requests, which means a greater integration burden in each case.
Rate limiting is managed through AI Policies, which set quotas for tokens and number of requests, aligning with WSO2’s usual business plan model. In terms of observability, the system captures detailed information on token and model consumption, facilitating granular analysis. However, critical functionalities such as content moderation, prompt engineering, caching, and mediation are not available natively, relying on custom mediators to fill these gaps.
The two leaders in LLM API management
Below is a comparative table between Kong Gateway and WSO2 API Manager regarding the main capabilities for managing LLMs APIs
Conclusions
Managing LLMs APIs requires going beyond traditional API governance mechanisms. While Kong is positioned as the most advanced option, with a robust ecosystem covering standardisation, security, moderation and optimisation, WSO2 offers a solid starting point, but is still limited in critical capabilities that are essential for secure and scalable enterprise deployment.
For those responsible for APIs and AI in enterprises, the decision between Kong and WSO2 will depend on the maturity of their use cases. Organisations requiring advanced, multi-vendor governance and semantic capabilities should lean towards Kong, while those seeking simpler, more progressive integration may consider WSO2, provided they are willing to develop custom extensions to cover its shortcomings.
We help you make the leap you need
At CloudAPPi, we are experts in APIs and AI integration in business environments, and we support organisations throughout the cycle: from the evaluation of use cases to the implementation of governed and secure LLM-based agents.
Integrate AI into your processes now
Schedule a meeting with us and we’ll explain everything
