7 posts tagged with "kubernetes"

Most inference autoscalers react to CPU or GPU utilization. But for large language models the metric that actually matters to users is Time To First Token (TTFT) — how long they wait before the response starts streaming. A GPU can be 60% utilized and still be delivering 30-second TTFT under a burst of long-context requests.

In this post I'll show how to wire NVIDIA Dynamo disaggregated inference together with KEDA on AKS so that the system autoscales the decode worker fleet directly on TTFT p99 — using Azure Managed Prometheus as the metric source and AKS-managed GPU drivers so there is no NVIDIA GPU Operator to maintain.

You've got your AKS cluster, your GPU quota is approved, and you're ready to serve large language models. But picking the right inference stack — vLLM, TensorRT-LLM, SGLang, disaggregated vs. unified — can cost you days before your first token lands.

That's the gap NVIDIA Dynamo fills.

You deploy your workloads on AKS and collect metrics with Prometheus and logs with Loki. But when latency spikes hit, you stare at dashboards knowing something is slow without knowing where in your code the time is being spent.

That's the gap continuous profiling fills.

In this Tool of the Trade, we take a quick look at how fuzziness with fzf can enhance your day-to-day life in the terminal. This is one of those tools that you did not know you really needed until you try it.

You create an Azure Red Hat OpenShift cluster, and minutes later, you notice something interesting in the managed resource group: two storage accounts with cryptic names like cluster1a2b3c4d5e and imageregistry1a2b3c4d5e.

What are they for? Why two? And what happens if you accidentally delete one?

You open a shell inside a pod, run ping 8.8.8.8, and nothing happens. Silence. Then you run curl https://www.microsoft.com and it succeeds immediately. DNS resolution also works normally.

So why does ping fail?

Welcome to "Tools of the Trade" - a series where we share the tools and workflows that help us work more effectively. In this first post, I'll show you how I manage multiple AKS clusters without losing track of which cluster I'm working on. If you've ever accidentally deployed to the wrong cluster, this one's for you.