Azure Enterprise AI Architecture: Production Decisions vs Demo Decisions

Azure Enterprise AI Architecture: Production vs Demo Decisions

Every enterprise AI project starts with a demo that works. The demo runs on a laptop or a basic cloud deployment, shows the capability, and gets buy-in. Then the team tries to deploy it to production and discovers that the demo architecture does not survive contact with real scale, real users, and real security requirements.

This post covers the specific Azure AI architecture decisions that differ between demo and production deployments — the ones that matter but are rarely discussed in tutorials.

1. API Key vs Managed Identity Authentication

Demo decision: Store the Azure OpenAI API key in an environment variable or config file. It works. Production decision: Use Azure Managed Identity. The application authenticates to Azure OpenAI using its Managed Identity — no API key, no secret to rotate, no secret to leak. The RBAC assignment (Cognitive Services User role) controls what the identity can access. Why it matters: API keys in environment variables get committed to git repositories, included in container images, and exposed in cloud-init logs. Managed Identity eliminates the secret entirely.

2. Public Endpoint vs Private Endpoint

Demo decision: Deploy Azure OpenAI with the public endpoint enabled (the default). Production decision: Deploy with a private endpoint, disabling public internet access. Requests route through your Azure Virtual Network. Why it matters: Your LLM calls may include sensitive business data (customer records, financial data, PII). Routing these calls through a public endpoint means they traverse the public internet. Private endpoints keep all traffic inside the Azure network, eliminating that exposure.

3. Rate Limit Handling

Demo decision: Call the Azure OpenAI API and handle 429 errors by retrying after a fixed delay. Production decision: Implement a proper rate limit management strategy: Request queuing with Azure Service Bus to smooth out traffic spikes. Circuit breaker pattern to fail fast when rate limits are sustained. Multiple Azure OpenAI deployments across regions with routing logic (PTU deployments in primary region, pay-as-you-go fallback in secondary). Azure API Management in front of Azure OpenAI to enforce per-client rate limits. Why it matters: Under production load, naive retry logic causes cascading failures. Every retry is another request that hits the rate limit, causing a retry storm.

4. Observability

Demo decision: Print statements and console logging. Production decision: Structured observability stack for AI workloads: Azure Monitor and Application Insights with custom dimensions for each LLM call (model, prompt tokens, completion tokens, latency, success/failure). Azure Log Analytics workspace with KQL queries for LLM usage patterns. Custom metrics for: average tokens per request by endpoint, P95 latency by model, error rate by error type, cost per request by application feature. Alerts on: latency spikes (P95 > 10s), error rate > 2%, token consumption > 120% of budget.

5. Content Filtering Configuration

Demo decision: Use the default Azure OpenAI content filters. Production decision: Configure content filters appropriate for your use case and risk tolerance. Azure OpenAI offers configurable content filtering across four harm categories (hate, sexual, violence, self-harm) at four severity levels. For enterprise applications processing business documents, the default filters may block legitimate content. For consumer-facing applications, you may need stricter filtering. Document your content filter configuration and the business justification — regulators may ask.

6. Prompt Injection Defence

Demo decision: Concatenate user input directly into the prompt. Production decision: Implement prompt injection defences: Use the system/user message separation — never concatenate user input into the system prompt. Validate and sanitise user input before including it in prompts. Use Azure OpenAI's prompt shield feature to detect injection attempts. Implement output validation — if the model's output doesn't match expected format or content patterns, flag it for review rather than using it directly.

7. Cost Controls

Demo decision: No cost controls — run until budget is exhausted. Production decision: Azure Budgets with alerts at 80% and 100% of monthly budget. Per-deployment token quotas in Azure OpenAI to prevent any single deployment consuming all available quota. Application-level token budgets: maximum tokens per user per day, maximum tokens per request. Logging of all requests with cost attribution to feature/tenant for showback/chargeback.

Techseria designs and deploys production-grade Azure AI architectures. Book a Strategy Session — we'll review your current AI architecture and identify the production gaps before they become incidents.