Downloaded 10 times
Uploaded byCallon Campbell
Elevating AI Workflows: Integrating Azure API Management and Azure Functions with OpenAI
Explore the seamless integration of Azure API Management and Azure Functions with OpenAI to create intelligent, scalable, and secure applications. This presentation will delve into how API Management can enhance control, security, and monitoring of AI API calls, while Azure Functions provide event-driven processing and efficient handling of data flows. I will demonstrate practical scenarios where this combination optimizes AI-driven solutions, including chatbot development, data processing automation, and more. Attendees will gain insights into best practices for setting up API Management policies, writing Azure Functions, and leveraging OpenAI's powerful AI capabilities. Join me to unlock the full potential of Azure services in your AI projects.
Elevating AI Workflows: Integrating Azure API Management and Azure Functions with OpenAI
- 1. Elevating AI Workflows: IntegratingAzure API Management and Azure Functions with Azure OpenAI Callon Campbell Microsoft MVP | Azure @flying_maverick
- 2. Sponsors Microsoft is aproud sponsor of Global Azure in Toronto on April 20th . We’re passionate about supporting the developer community and invite you to access valuable documentation and training resources by visiting docs.microsoft.com and the Microsoft Reactor. We are a team of senior infrastructure specialists, software developers and data engineers who are experts in the Microsoft Azure Cloud. We partner with you to deliver innovative business solutions using Agile, DevOps and advanced Software Automation. Twenty years in business. Headquartered in Toronto. Work for customers across Canada and the US. Work across multiple industries and sectors. www.objectsharp.com
- 3. About me 25years enterprise development with Microsoft technologies – .NET (C#), Azure, ASP.NET, Desktop, SQL, and Mobile Passionate about serverless and cloud-native application development, with focus on app migration and modernization, app integration and data analytics Blog at https://TheFlyingMaverick.com, @flying_maverick Speaker at community events and meetups Organizer of “Canada’s Technology Triangle .NET User Group” in Kitchener, Ontario Callon Campbell Azure Architect | Developer Adastra Microsoft MVP | Azure (2018-2025)
- 4. Agenda What isAPI Management and why it remains critical in the era of AI How to govern runtime of AI APIs Integrations with Azure OpenAI Demos Q&A
- 5. Why are wehere • Generative AI has ignited a remarkable range of possibilities • All industry sectors are embracing AI advancements • Most AI services are utilized and accessed via APIs • It’s essential to have a well-planned API management strategy to ensure the effective use of AI services • Approaches driven by experimentation are the enablers that pave the road to success
- 6. APIs are thebackbone of digital transformation, modern apps, and AI interfaces.
- 7.
- 8. What is APIManagement? Comprehensive platform for managing APIs across all environments. Provides tools for: Creating APIs Publishing APIs Securing APIs Analyzing APIs Helps organizations streamline their API strategies.
- 9. API Management Features API Gateway: Acts as a front door for APIs, handling all incoming requests and routing them to the appropriate backend services. Developer Portal: A customizable portal for API consumers to discover, learn about, and use APIs. Management Plane: Tools for administrators to manage API lifecycle, policies, and analytics. Security: Protects APIs with built-in security features like authentication, authorization, and rate limiting. (securely expose your Azure OpenAI endpoints) Scalability: Supports scaling APIs to meet varying demand levels. Monitoring & Analytics: Provides insights into API usage, performance, and health.
- 10. Runtime governance ofAI APIs with API Management
- 11. GenAI development runson APIs Intelligent Apps Conversational Agents Personalized Content Content Generation Chat on your Data Voice Assistants Your own Copilot But these APIs must be Managed Secured Governed AI Services Azure AI Services OpenAI Mistral LLaMa Azure AI Search Hugging Face Cohere and more!
- 12. Unmanaged AI APIs increaserisk and hinder potential Unpredictable & unattributable costs Reliability concerns Security risks Developer friction Governance challenges
- 13. Azure API Managementenables AI APIs Intelligent Apps Conversational Agents Personalized Content Content Generation Chat on your Data Voice Assistants Your own Copilot Cost efficiency High reliability Robust security Developer enablement Enhanced governance Native Azure integration Defender for APIs Policy Monitor … and more AI Services Azure AI Services OpenAI Mistral LLaMa Azure AI Search and more! Hugging Face Cohere
- 14. Maximize potential andtake control of AI APIs with Azure API Management Cost Efficiency Control and attribute costs with token monitoring, limits, and quotas Return cached responses for semantically similar prompts High Reliability Enable geo- redundancy and automatic failovers with load balancing and circuit breakers Robust Security Isolate and manage user credentials Secure APIs with built-in controls and Microsoft Defender for Cloud Developer Enablement Replace custom backend code with built-in policies Publish AI APIs for consumption Gain insights with comprehensive logs Enhanced Governance Enforce runtime policies Centralize monitoring and audit logs
- 15. Scaling Up: MultipleApps, Multiple OpenAI Endpoints Intelligent App Intelligent App Intelligent App Azure OpenAI Endpoints Scaling Challenges Track token usage Multiple OpenAI endpoints Authentication and authorization Assign token-based limits
- 16. GenAI gateway capabilitiesin API Management Intelligent App Intelligent App Intelligent App Azure API Management Token-based limiting GenAI Gateway Load balancing Semantic caching Observability Azure OpenAI Endpoints Managed identity
- 17. Demo API Management |Azure OpenAI
- 18.
- 19. Request forwarding APIMuses the managed identity (user or system assigned). APIM is authorized to consume the Azure OpenAI API through Role Based Access Controls. Zero impact on consumers using the API directly, with SDKs or orchestrators like LangChain. Just need to update the endpoint to use the APIM endpoint instead of Azure OpenAI endpoint. Keyless approach: API consumers use the APIM subscription keys, and the Azure OpenAI keys are never used. Request forwarding
- 20. Token limit policy •Manage and enforce limits per API consumer based on the usage of Azure OpenAI Service tokens. • Set a rate limit, expressed in tokens-per-minute (TPM). • Set a token quota over a specified period, such as hourly, daily, weekly, monthly, or yearly.
- 21. Token limit policy <azure-openai-token-limitcounter-key="@(context.Subscription.Id)" tokens-per-minute="500" estimate-prompt-tokens="false" remaining-tokens-variable-name="remainingTokens"> </azure-openai-token-limit>
- 22. Emit token metricpolicy • Sends metrics to Application Insights about consumption of LLM tokens through Azure OpenAI Service APIs. • Helps provide an overview of the utilization of Azure OpenAI Service models across multiple applications or API consumers. • Useful for chargeback scenarios, monitoring, and capacity planning.
- 23. Emit token metricpolicy <azure-openai-emit-token-metric namespace="openai"> <dimension name="Client IP" value="@(context.Request.IpAddress)" /> <dimension name="API ID" value="@(context.Api.Id)" /> <dimension name="User ID" value="@(context.Request.Headers.GetValueOrDefault("x-user-id", "N/A"))" /> </azure-openai-emit-token-metric>
- 24. Backend circuit breaking Azure OpenAI endpoint is configured as an APIM backend, promoting reusability across APIs and improved governance. Circuit breaking rules define controlled availability for the OpenAI endpoint. When the circuit breaks, APIM stops sending requests to OpenAI. Handles the status code 429 (Too Many Requests) and any other status code sent by the OpenAI service. Doesn’t need any policy configuration. The rules are just properties of the backend. New product feature built-in backend circuit breaker functionality Backend circuit breaking
- 25. Backend load balancing Spread the load to multiple backends, which may have individual backend circuit breakers. Shift the load from one set of backends to another for upgrade (blue-green deployment). Currently, the backend pool supports round-robin, weighted, and priority- based load balancing. Doesn’t need any policy configuration. The rules are just properties of the backend. built-in load balancing backend pool functionality New product feature Backend load balancing
- 26. Semantic caching policy •Optimize token use by storing completions for similar prompts. • Helps reduce token consumption and improves response performance.
- 27. Well-Architected Framework principles •Zero-trust approach and keyless strategy for Azure OpenAI • Redundancy and capable of handling variable usage spikes • Elasticity and mechanisms to distribute the load to multiple endpoints • Observability to continuously improve quality and user experience • Cost control mechanisms to track token usage and allocate costs https://learn.microsoft.com/en-us/azure/well-architected/
- 28. Demos API Management |Backends, Policies, Chat App with Azure OpenAI
- 29.
- 30.
- 31. Why this extension? Comparedwith standard Azure OpenAI API call, the extension would give: • Capability to work with large variety of triggers and bindings offered by function apps. • Function Apps would have pre-defined triggers to allow developers control event-driven or routine-based tasks. This extension would work well with the current offered types of functions. • Flexibility in the development phase when multiple Azure products are engaged. Different bindings allow function apps to listen and respond when certain Azure product changes. With host.json file inside Function Apps, settings would be easier to adjust and test. • Essentially, this extension would help you make API calls to the Azure OpenAI endpoint with a smoother experience.
- 32. Integration With the integrationbetween Azure OpenAI and Functions, you can build functions that can:
- 33. How does thiswork with API Management? Essentially replace the Azure OpenAI endpoints with the APIM endpoints.
- 34. AI Hub GatewayLanding Zone accelerator aka.ms/apim-genai-lza
- 35. GenAI gateway reference architecture •Use APIM to create a GenAI gateway. • Integrates with Azure OpenAI services in the cloud and any on- premises custom LLMs that are deployed and available as REST endpoints. • The architecture incorporates elements that are engineered for batch use cases, with the aim of optimizing PTU utilization. GenAI gateway reference architecture using APIM
- 36. Wrapping up Effective APIgovernance Ensure compliance, reliability, and security while accelerating innovation instead of creating roadblocks. AI runtime governance Use API Management capabilities to maximize the potential of AI APIs, incl. increased cost efficiency, reliability, security, and governance.
- 37. Useful resources API Management aka.ms/apim/openai-docs | Documentation aka.ms/apim/genai/sample-app | GenAI gateway guide aka.ms/apim/genai/labs | GenAI gateway labs aka.ms/apim-genai-lza | GenAI gateway accelerator Designing and implementing a GenAI gateway solution | Microsoft Learn GenAI gateway capabilities in Azure API Management | Micrososft Learn Azure Functions Azure OpenAI extension for Azure Functions | Microsoft Learn azure-functions-openai-extension/samples
- 38. Let’s connect callon@cloudmavericks.ca @flying_maverick https://linkedin.com/in/calloncampbell https://github.com/calloncampbell
- 39.
Editor's Notes
- #2 Hi everyone, a warm welcome to this session on agentic AI and how to get started with the Azure AI Agent Service
- #5 Industries are embracing AI advancements. AI services are mostly accessed via APIs. A solid API management plan is crucial for effective AI use. Experimentation is key to success.
- #6 APIs are the backbone of digital transformation, modern apps, and AI interfaces.
- #7 Who here has used APIM?
- #11 On the left we have your GenAI development / apps which runs on the AI Services APIs on the right. So we need to consider how we’re these AI APIs are going to be managed, secured and governed.
- #12 Unmanaged AI APIs can lead to unpredictable and difficult-to-trace expenses, impacting budget management. The performance and availability of services may be inconsistent without proper management. Unmanaged AI APIs may expose systems to vulnerabilities, leading to unauthorized access and data breaches. Without effective management, challenges can arise, hindering smooth collaboration and development processes. Ensuring compliance and oversight with unmanaged AI APIs can be complex and time-consuming.
- #15 One of the main resources you have in generative AI services is tokens. Azure OpenAI Service assigns quota for your model deployments expressed in tokens-per-minute (TPM) which is then distributed across your model consumers - for example, different applications, developer teams, departments within the company, etc.
- #16 A "GenAI gateway" is an intelligent middleware that dynamically balances incoming traffic across backend resources to optimize resource utilization. It can also address challenges related to billing and monitoring.
- #20 This policy provides flexibility to assign token-based limits on any counter key, such as subscription key, originating IP address, or an arbitrary key defined through a policy expression. The policy also enables precalculation of prompt tokens on the Azure API Management side, minimizing unnecessary requests to the Azure OpenAI Service backend if the prompt already exceeds the limit.
- #21 This policy provides flexibility to assign token-based limits on any counter key, such as subscription key, originating IP address, or an arbitrary key defined through a policy expression. The policy also enables precalculation of prompt tokens on the Azure API Management side, minimizing unnecessary requests to the Azure OpenAI Service backend if the prompt already exceeds the limit.
- #22 This policy captures prompt, completions, and total token usage metrics and sends them to an Application Insights namespace of your choice. Moreover, you can configure or select from predefined dimensions to split token usage metrics, so you can analyze metrics by subscription ID, IP address, or a custom dimension of your choice.
- #23 This policy provides flexibility to assign token-based limits on any counter key, such as subscription key, originating IP address, or an arbitrary key defined through a policy expression. The policy also enables precalculation of prompt tokens on the Azure API Management side, minimizing unnecessary requests to the Azure OpenAI Service backend if the prompt already exceeds the limit.
- #24 One of the challenges when building intelligent applications is to ensure that the applications are resilient to backend failures and can handle high loads. The backend circuit breaker features dynamic trip duration, applying values from the Retry-After header provided by the backend. This ensures precise and timely recovery of the backends, maximizing the utilization of your priority backends. // other if needed Circuit Breaker Pattern: Stops operations likely to fail. Returns fallback response during failures. Resumes normal operations when stable. API Management: Prevents backend overload. Trips circuit based on retry-after header. Azure OpenAI: Enforces rate limiting (429 response code). Configurable circuit breaking in API Management. Marks backend as unhealthy when circuit is open.
- #25 By configuring your Azure OpenAI Service endpoints using backends in Azure API Management, you can balance the load across them. You can also define circuit breaker rules to stop forwarding requests to the Azure OpenAI Service backends if they're not responsive. The backend load balancer supports round-robin, weighted, and priority-based load balancing, giving you flexibility to define a load distribution strategy that meets your specific requirements.
- #26 Enable semantic caching by using Azure Redis Enterprise or another external cache compatible with RediSearch and onboarded to Azure API Management. By using the Azure OpenAI Service Embeddings API, the azure-openai-semantic-cache-store and azure-openai-semantic-cache-lookup policies store and retrieve semantically similar prompt completions from the cache. This approach ensures completions reuse, resulting in reduced token consumption and improved response performance.
- #29 Sample app Frontend: Two files, index.html and app.js, that make requests to the backend. Backend: A Node.js Express app that serves the frontend and makes requests to the Azure Open AI instance. Azure OpenAI Service: Two instances of Azure OpenAI models, one primary endpoint and one secondary/ failover endpoint. Azure API Management: Manages the Azure OpenAI instances and exposes them to the frontend.
- #30 The Azure OpenAI extension for Azure Functions is currently in preview.
- #35 The AI Hub Gateway Landing Zone is a solution accelerator that provides a set of guidelines and best practices for implementing a central AI API gateway to empower various line-of-business units in an organization to leverage Azure AI services. The Azure API Management(APIM) Landing Zone accelerator provides a comprehensive solution to deploy a GenAI gateway using Azure API Management with best practices around security and operational excellence.
- #36 Centralized AI API Gateway: A central hub which provides a single point of entry for AI services that can be shared among multiple use-cases in a secure and governed approach. Seamless integration with Azure AI services: Ability to just update endpoints and keys in existing apps to switch to use AI Hub Gateway. AI routing and orchestration: The AI Hub Gateway Landing Zone provides a mechanism to route and orchestrate AI services, based on priority and target model enabling the organization to manage and govern AI services in a consistent manner. Granular access control: The AI Hub Gateway Landing Zone does not use master keys to access AI services, instead, it uses managed identities to access AI services while consumers can use gateway keys. Private connectivity: The AI Hub Gateway Landing Zone is designed to be deployed in a private network, and it uses private endpoints to access AI services. Capacity management: The AI Hub Gateway Landing Zone provides a mechanism to manage capacity based on requests and tokens. Usage & charge-back: The AI Hub Gateway Landing Zone provides a mechanism to track usage and charge-back to the respective business units with flexible integration with existing charge-back & data platforms. Resilient and scalable: The AI Hub Gateway Landing Zone is designed to be resilient and scalable, and it uses Azure API Management with its zonal redundancy and regional gateways which provides a scalable and resilient solution. Full observability: The AI Hub Gateway Landing Zone provides full observability with Azure Monitor, Application Insights, and Log Analytics with detailed insights into performance, usage, and errors. Hybrid support: The AI Hub Gateway Landing Zone approach the deployment of backends and gateway on Azure, on-premises or other clouds.
- #37 Provisioned Throughput Units - PTU Generative AI - GenAI Large Language Model - LLM
- #41 All the demos are here!