How AI labs interview — Question to Offer

1. The shape of the loop

The labs run shorter loops than the rest of this book. You'll usually see four rounds for senior, five for staff. A coding round, two system-design rounds, one behavioral. Sometimes a paper-reading or research-flavored round if the team is closer to the model. Most teams are not.

The two-hour onsite design block is where you win or lose. The labs hire mostly for infrastructure right now, not research, and they grade infra at a bar that is closer to AWS than to a university lab. The interviewer wants you to talk about LLM serving the way you'd talk about a stateful database fleet, not the way you'd talk about a notebook.

They are small companies. The whole engineering org is the size of one Microsoft team. Your interviewer is probably an engineer who joined eighteen months ago, who is now on-call for a system that serves a meaningful fraction of the world's inference. They are tired. They want to hire someone who would let them sleep.

2. What the labs grade

Three signals dominate the design rounds.

Distributed systems literacy at depth. The labs do not want a candidate who treats inference as a black box. They want a candidate who knows that the dominant cost is GPU memory pressure from the KV cache, that batching matters at the decoding-step level (not the request level), and that prompt-prefix sharing is the single biggest win in the stack right now. If you can't talk about these without prompting, you'll score depth-at-mid, not depth-at-senior.
Ownership signal. Small team. You're going to be on-call for a service that costs the company millions of dollars an hour to run. The behavioral round asks about a time you owned an incident end to end. "My manager handled the runbook" is a no-hire answer.
Safety reasoning. This is the one that's different from every other company in this book. The labs care, structurally, whether you can reason about prompt injection, abuse vectors, content policy. Even on an infra round, the senior interviewer often slips in "and how would you detect a customer using this for spam at scale?" Have an answer ready.

3. What's different here

Younger company, smaller org. Fewer levels. The bar at L5 here looks more like a Google L6 than a Google L5 — because at a 200-engineer company, a senior engineer owns a whole subsystem, not a slice of one.
Infra teams and research teams interview differently. An infra team rounds you on system design and gives a coding question that's more like a Stripe coding question (build a clean API). A research-leaning team (alignment, evals, post-training) gives a research-flavored round that looks more like a PhD oral than a coding interview. Find out which team you're interviewing for before you start prep.
The serving stack is a distributed-systems problem, not an ML problem. The labs explicitly do not want you to talk about model architecture. You don't need to know what attention is. You need to know that the attention key-value tensors live in GPU memory for the duration of a decoding loop, that they grow linearly with sequence length, and that this is the dominant capacity planner in the inference fleet.
Tokens, not requests. The unit of work is the token. Rate limits are tokens-per-minute. Latency targets are tokens-per-second. Pricing is per million tokens. If you scope an API design and your rate limit is requests-per-minute, you will be corrected immediately and you'll lose the scoping signal in the first ten minutes.
Cold starts have a different shape. Loading a multi-hundred-gigabyte set of model weights into GPU memory is the cold start. Container start is nothing compared to it. The mitigations are different too. See [[llm-serving-at-scale-kv-cache-batching]] and compare to [[azure-functions-cold-start]] (Microsoft) for the FaaS-flavored version.

4. The level bar

The labs are still settling on their level system. As of writing, both companies use a small number of bands that map roughly to:

L4 (mid) — design at a component level. You own one piece of a service. The bar here is closer to Microsoft L62 or Google L4. You're expected to be fluent in standard distributed-systems vocabulary and able to defend one deep dive.
L5 (senior) — design at a subsystem level. You own the whole serving stack for one model variant, or the whole evals pipeline, or the whole fine-tune orchestration layer. Two deep dives, each surviving three pushes. You're expected to bring up safety and abuse concerns unprompted.
L6 (staff) — design at a platform level. The bar shifts to "how does this primitive interact with five other primitives the company runs?" You're expected to know the prompt cache, the KV cache, the rate limiter, the evals system, and the fine-tune system well enough to design any of them in detail and explain how they connect. The chapter targets L6 by default with explicit notes on what shifts at L5.

5. The lab-flavored mistakes

Watch for:

Treating the LLM as a black box behind an HTTP server. The depth seam at the lab is inside the serving stack. If your design has "call the model" as a single box on the diagram, you've already lost half the round. Open the box. Talk about KV cache, continuous batching, prompt prefix reuse, paged attention.
Using request-shaped vocabulary. Token-shaped vocabulary. Tokens per minute. Tokens per second. Cost per million tokens. Memory per token of KV state. If you say "queries per second" in this round you sound like you've never run an inference fleet.
Forgetting that the same prompt prefix appears across millions of requests. Every system prompt the customer wrote is reused on every one of their requests. The biggest single optimization in the entire stack right now is sharing the KV cache for the prefix. Bring it up in the first ten minutes of any serving question.
Ignoring the abuse vector. The senior tell here is whether you scope abuse and content-policy as first-class concerns. A weak candidate treats them as someone else's problem. A strong candidate puts a rate limiter and a content classifier in the request path on the first diagram.
Pretending the model is deterministic. It isn't. Same input, same parameters, different output across runs because of floating-point non-determinism on the GPU. This matters for caching, for evals, for reproducibility. Senior interviewers love to push on this.
Naming model names or version numbers. The teams change them every quarter and your interviewer is bored of hearing them. Talk about the model generically. Talk about the serving fleet, the training cluster, the evals harness. You'll sound like you understand the platform, not the marketing.

6. The seven scenarios in this chapter

Each unit dissects one question end to end. The chapter covers:

24.1 LLM serving at scale — KV cache and batching. The core inference question. Continuous batching, KV cache memory, paged attention. The most-asked lab question.
24.2 Prompt cache invalidation. How shared prefixes get reused, and the painful problem of when to throw the cache away. Asked at L6.
24.3 Rate limiting and abuse detection for LLM APIs. Token-based rate limits, per-customer per-model quotas, and the abuse classifier. The intersection of capacity and safety.
24.4 Fine-tune job orchestration. Customer uploads training data, lab runs the job, customer gets a new model. Multi-tenant GPU scheduling and a job state machine.
24.5 Evals infrastructure. A new model variant ships; you run it against a battery of evals and catch regressions before it goes to production. The experimentation layer of the lab.
24.6 Tool use and function-calling infrastructure. The model returns a structured tool call, your service invokes the tool, returns the result, model continues. The orchestration loop and the sandbox.
24.7 Take-home — LLM proxy service. A design doc for a proxy that sits between customer apps and the lab's API, providing rate limiting, prompt caching, observability, and audit logs.

7. The next page

The first scenario is the one every lab asks: design the serving stack. KV cache, continuous batching, paged attention. The infra round that sorts senior from staff.