6 posts tagged with "aks"

Sometimes, you need to add authentication to an application without modifying its source code. It might be off-the-shelf software or an older in-house application built before modern authentication protocols such as OpenID Connect or SAML 2.0 became commonplace. So how can you protect it with modern authentication without changing the application itself?

This is where a reverse proxy can handle authentication on the application's behalf.

In this post we'll show how to protect the AKS Store Demo with an Istio Gateway (using the Gateway API), oauth2-proxy, and Microsoft Entra ID. Istio service mesh delegates every protected request to oauth2-proxy through Envoy external authorization. Envoy's external authorization filter delegates authorization decisions to an external HTTP or gRPC service, allowing flexible and centralized access control. Oauth2-proxy uses the authorization code flow with PKCE (Proof Key for Code Exchange) to authenticate the user. It uses AKS Workload Identity to authenticate the app registration when redeeming the authorization code.

There is no Entra application password to create, store in Kubernetes, or rotate.

Most inference autoscalers react to CPU or GPU utilization. But for large language models the metric that actually matters to users is Time To First Token (TTFT) — how long they wait before the response starts streaming. A GPU can be 60% utilized and still be delivering 30-second TTFT under a burst of long-context requests.

In this post I'll show how to wire NVIDIA Dynamo disaggregated inference together with KEDA on AKS so that the system autoscales the decode worker fleet directly on TTFT p99 — using Azure Managed Prometheus as the metric source and AKS-managed GPU drivers so there is no NVIDIA GPU Operator to maintain.

You've got your AKS cluster, your GPU quota is approved, and you're ready to serve large language models. But picking the right inference stack — vLLM, TensorRT-LLM, SGLang, disaggregated vs. unified — can cost you days before your first token lands.

That's the gap NVIDIA Dynamo fills.

You deploy your workloads on AKS and collect metrics with Prometheus and logs with Loki. But when latency spikes hit, you stare at dashboards knowing something is slow without knowing where in your code the time is being spent.

That's the gap continuous profiling fills.

You open a shell inside a pod, run ping 8.8.8.8, and nothing happens. Silence. Then you run curl https://www.microsoft.com and it succeeds immediately. DNS resolution also works normally.

So why does ping fail?

Welcome to "Tools of the Trade" - a series where we share the tools and workflows that help us work more effectively. In this first post, I'll show you how I manage multiple AKS clusters without losing track of which cluster I'm working on. If you've ever accidentally deployed to the wrong cluster, this one's for you.