Most inference autoscalers react to CPU or GPU utilization. But for large language models the metric that actually matters to users is Time To First Token (TTFT) — how long they wait before the response starts streaming. A GPU can be 60% utilized and still be delivering 30-second TTFT under a burst of long-context requests.

In this post I'll show how to wire NVIDIA Dynamo disaggregated inference together with KEDA on AKS so that the system autoscales the decode worker fleet directly on TTFT p99 — using Azure Managed Prometheus as the metric source and AKS-managed GPU drivers so there is no NVIDIA GPU Operator to maintain.

You've got your AKS cluster, your GPU quota is approved, and you're ready to serve large language models. But picking the right inference stack — vLLM, TensorRT-LLM, SGLang, disaggregated vs. unified — can cost you days before your first token lands.

That's the gap NVIDIA Dynamo fills.

You deploy your workloads on AKS and collect metrics with Prometheus and logs with Loki. But when latency spikes hit, you stare at dashboards knowing something is slow without knowing where in your code the time is being spent.

That's the gap continuous profiling fills.

In this Tool of the Trade, we take a quick look at how fuzziness with fzf can enhance your day-to-day life in the terminal. This is one of those tools that you did not know you really needed until you try it.

You create an Azure Red Hat OpenShift cluster, and minutes later, you notice something interesting in the managed resource group: two storage accounts with cryptic names like cluster1a2b3c4d5e and imageregistry1a2b3c4d5e.

What are they for? Why two? And what happens if you accidentally delete one?

You open a shell inside a pod, run ping 8.8.8.8, and nothing happens. Silence. Then you run curl https://www.microsoft.com and it succeeds immediately. DNS resolution also works normally.

So why does ping fail?

Welcome to "Tools of the Trade" - a series where we share the tools and workflows that help us work more effectively. In this first post, I'll show you how I manage multiple AKS clusters without losing track of which cluster I'm working on. If you've ever accidentally deployed to the wrong cluster, this one's for you.

When deploying Azure Red Hat OpenShift (ARO) clusters, managing authentication and authorization for various cluster components traditionally relies on service principals or other credential-based approaches. This introduces operational overhead and potential security risks related to credential rotation and management.

In this walk through, we'll set up a very basic AKS cluster to get you quickly up and running with the Anyscale platform using AKS as the compute backend. We'll run this cluster in our own Azure Virtual Network and have it connected to an Azure Blob Storage Account on that VNet. Finally, we'll execute the basic Anyscale 'Hello World' demo on that compute.

We all know this, even if we don't like to admit it: modern infrastructure can scale infinitely, but human understanding doesn't.

We've all seen it happen—organizations going from managing dozens of servers to thousands of containers, from deploying weekly to deploying hundreds of times per day, from serving thousands of users to millions. The technology handled the scale beautifully. The humans? Not so much.

This is the first industry issue that platform engineering should be addressing: how do we manage infrastructure complexity that has outgrown not just individual cognitive capacity, but our collective ability to communicate and transfer knowledge as teams?