How to pass the OpenAI system design interview

OpenAI's system design interview feels completely different from your standard big-tech loop. You aren't going to be designing generic social media feeds, ride-sharing apps, or standard e-commerce backends. Instead, you will need to architect the actual infrastructure powering large language models at a massive scale. The evaluation bar leans heavily on high-throughput token streaming, agentic tool execution, strict resource isolation, and managing incredibly expensive GPU compute. Passing means you have to show a deep, practical understanding of real-time networking, asynchronous concurrency, and hardware-aware system architecture.

The OpenAI engineering interview process

This hiring process intentionally tests how well you handle ambiguity while building scalable AI infrastructure. The exact structure might shift slightly depending on the specific team you are targeting—like the API team, the ChatGPT product team, or the core infrastructure group. Even so, the standard software engineering loop follows a fairly predictable pattern. Interviewers expect you to drive the conversation, identify edge cases before they become problems, and anchor every single architectural decision to the end-user experience.

Core system design questions and architectures

Let's look at the major architectural challenges candidates actually face in the OpenAI system design round. I have phrased these exactly as interviewers present them. We will break down the expected technical depth, the trade-offs you have to weigh, and the optimal solutions for every single scenario. You cannot just draw a few boxes and arrows; you have to explain the exact data flow, the network protocols, and the failure states.

"How do you stream tokens to millions of users with low latency?"

This specific question tests your grip on real-time communication protocols and connection management. Standard REST APIs completely fail here. If a user asks for a 2,000-word essay, waiting for the full response to generate causes an unacceptable Time to First Token (TTFT). The user will stare at a blank screen for twenty seconds, assume the app is broken, and leave. You have to design a streaming gateway that pushes tokens to the client the millisecond they leave the GPU.

• Protocol choice: Server-Sent Events (SSE) is the industry standard for token streaming. Unlike WebSockets, which maintain a complex full-duplex connection, SSE operates over standard HTTP and is unidirectional from server to client. This perfectly matches the LLM response pattern, carries less network overhead, and is vastly easier to load balance across L7 proxies.
• Connection management: Long-lived connections demand specialized load balancers. You have to design for connection draining and graceful degradation during deployments so you do not drop active streams. If an inference node goes down, the gateway must catch the broken stream, retry the generation from the last known state, and seamlessly resume the SSE connection without the client noticing.
• Buffering and chunking: Emitting every single character over the network is incredibly inefficient because of TCP packet overhead. Sending three bytes of text requires a forty-byte TCP header. Grouping tokens into small chunks—like 50-millisecond buffers—optimizes network utilization. The human eye cannot perceive a 50ms delay, but your network stack will handle the reduced packet volume much more effectively.

When you walk through this flow, trace the exact path of a request. The user's client opens an HTTP connection with an 'Accept: text/event-stream' header. The request hits an edge CDN, passes through your API gateway, and lands on an inference routing service. The router holds the connection open while it talks to the backend GPU cluster. As the GPU spits out tokens, the router buffers them slightly, formats them as SSE data payloads, and flushes them down the open socket directly to the user.

"How do you optimize GPU memory and manage the KV cache?"

Large language models are inherently stateless. If you ask a model a question, it processes the text and generates an answer. If you ask a follow-up question, the model has no memory of the previous interaction. To maintain a conversation, you have to send the entire history back to the model every single time. Recomputing the internal attention states for this history is computationally expensive and adds massive latency. Interviewers want to see how you solve this physics problem.

• The Prefill vs. Decode phase: When a request hits the GPU, the model first processes the entire prompt in parallel. This is the prefill phase, which generates Key-Value (KV) tensors for every token. Then, it enters the decode phase, generating one new token at a time sequentially. The prefill phase is compute-bound, while the decode phase is memory-bandwidth-bound.
• PagedAttention: Storing these KV tensors takes up a huge amount of VRAM. Standard memory allocation leads to severe fragmentation. You should design your system to use PagedAttention, which breaks the KV cache into fixed-size blocks, much like virtual memory in an operating system. This nearly eliminates memory fragmentation and allows you to serve significantly more concurrent users on the same GPU.
• Sticky routing: If a user is in a long chat session, you want to avoid recomputing the prefill phase for every single message. Design your API gateway to hash the conversation ID and route the request to the exact same GPU node that handled the previous turn. If the KV cache is still in VRAM, the model can skip the prefill phase entirely and immediately start decoding the new response.
• Cache eviction: VRAM is strictly limited. You need a strategy for when the GPU memory fills up. Implement a Least Recently Used (LRU) eviction policy. If a user stops typing for five minutes, their KV cache gets evicted to make room for active users. When they return, the system simply recomputes the cache from scratch.

"How do you securely execute untrusted Python code generated by the model?"

Advanced models regularly write and execute code to solve math problems, format spreadsheets, or analyze data. This is an absolute security nightmare. Interviewers want to see exactly how you isolate this untrusted execution from your internal network and your other tenants. You have to assume the model will eventually generate malicious code, either by hallucination or via a user's prompt injection attack.

• MicroVM isolation: Standard Docker containers share the host operating system's kernel. If a model writes a Python script that exploits a kernel vulnerability, it can break out of the container and access the host machine, compromising other users' data. You must propose true hardware-level virtualization. AWS Firecracker or gVisor creates a microVM with a dedicated, isolated kernel for every single execution environment.
• Snapshotting for cold starts: Booting a full Linux kernel takes time. If a user is waiting for an answer, adding a one-second boot delay ruins the experience. Solve this by booting thousands of microVMs in the background, creating a memory snapshot of their initialized state, and pausing them. When a code execution request arrives, you clone the snapshot and resume the VM in single-digit milliseconds.
• Resource limits: Rely on cgroups to strictly limit CPU time, memory consumption, and disk I/O. If a user asks the model to calculate a massive prime number, the resulting Python script could easily hit an infinite loop. Strict resource limits prevent these loops from turning into denial-of-service (DoS) attacks that drain your compute budget.
• Network egress: The sandbox must be entirely cut off from your internal network. If the code tries to query 169.254.169.254 (the AWS metadata service), it should fail immediately. Use Linux network namespaces (netns) and tap devices to route all sandbox egress traffic through an isolated NAT gateway. Only allow traffic to specific external domains on port 443, and apply strict bandwidth limits to prevent the sandbox from being used in DDoS attacks.

"How do you handle long-running agent loops and async tool execution?"

When an LLM decides to call an external API or search the web, the generation process pauses, waits for the tool to finish, and then resumes. The inference engine cannot sit around waiting for a third-party API to respond; GPUs are too expensive to sit idle. Handling this back-and-forth requires robust state management and solid asynchronous concurrency.

• State persistence: When the model decides to call a tool, it emits a structured JSON payload and halts. Your gateway must immediately take this payload, save the current conversation state to a fast database like Redis or DynamoDB, and free up the GPU for the next user in the queue.
• Asynchronous event-driven architecture: Drop the tool execution payload into a message broker. Kafka is ideal here because it handles high throughput and guarantees message delivery. A separate fleet of CPU-only worker nodes consumes these messages. These workers handle the slow, unpredictable work of calling external APIs, scraping websites, or querying databases.
• The resumable loop: Once the CPU worker gets the API response, it writes the result back to the database under the same conversation ID. It then publishes a tool-complete event. Your gateway subscribes to this event, grabs the updated state, appends the tool's output directly to the system prompt, and places the request back into the GPU inference queue. The model sees the new data and resumes generating the final answer for the user.

"How do you ingest and process multi-modal inputs like video and audio?"

Text is lightweight, but modern models ingest massive files, including high-resolution images, hour-long audio files, and PDFs. Designing a system that can reliably transfer gigabytes of data from a user's mobile device to a backend GPU cluster requires a completely different architectural approach than standard JSON APIs.

• Direct-to-storage uploads: Never pass large binary files through your API gateway. Doing so saturates your proxy's memory and bandwidth. Instead, the client should request a presigned URL from your backend. The client then uploads the file directly to an S3 bucket or blob storage. Once the upload finishes, the client sends the file's reference ID to the LLM gateway.
• Chunking and embedding: Large documents and videos cannot fit entirely into a model's context window. You need an asynchronous pipeline that picks up the uploaded file, extracts the text or frames, chunks the data into smaller segments, and runs them through an embedding model. These embeddings are stored in a vector database like Pinecone or Milvus, allowing the LLM to perform Retrieval-Augmented Generation (RAG) rather than reading the entire file at once.
• Real-time audio streaming: If you are building a voice assistant, you need bidirectional streaming. While SSE works for text output, real-time voice requires WebRTC or WebSockets. You capture tiny chunks of audio on the client, stream them to a transcription service (like Whisper), and pipe the resulting text directly into the LLM prompt queue.

Reliability and token cost accounting

"How do you implement distributed rate limiting for LLM requests?"

Traditional rate limiting usually just counts HTTP requests. In LLM infrastructure, however, you have to rate limit based on compute complexity—specifically, tokens. A single request asking for a 4,000-token essay costs vastly more compute time and VRAM than ten separate requests asking for a one-word answer. If you only limit by request count, a handful of users running max-length prompts will completely take down your cluster.

• Token bucket algorithm: Run a Redis-backed token bucket where the 'cost' of a request is estimated upfront. When a request hits the gateway, you calculate the prompt tokens using a fast tokenizer and add the requested max_tokens parameter. You deduct this total from the user's bucket via a Redis Lua script to guarantee atomicity. If they lack the tokens, you return a 429 status code.
• Reconciliation: The upfront deduction is an estimate. If the model stops generating early and only uses 50 tokens instead of the 1,000 max_tokens requested, you must perform a reconciliation step. After the generation finishes, you refund the unused 950 tokens back to the user's bucket.
• Dynamic backpressure: If your GPU cluster is saturated, the system has to return an HTTP 429 early, regardless of the user's token bucket balance. Implement a global queue depth metric that your load balancers check before they accept any new connections. If the queue is too deep, shed the load at the edge.
• Cost accounting: Emit asynchronous events to a message broker like Kafka containing the exact prompt and completion token counts for billing. Never put billing database writes in the critical path of a user request. If the billing service goes down, users should still be able to generate text.

"How do you balance latency (TTFT) and throughput?"

This is the ultimate trade-off in AI infrastructure. If you optimize entirely for latency, you process one request per GPU at a time. The user gets their first token instantly, but your throughput plummets, and your compute costs skyrocket. If you optimize entirely for throughput, you wait to batch 100 requests together before processing them. Throughput is massive, but the first user waits ten seconds for their response.

• Continuous batching: Static batching forces the GPU to wait until every request in the batch is completely finished before accepting new ones. Because some generations take 10 tokens and others take 1,000, static batching wastes massive amounts of compute. You must implement continuous batching (iteration-level scheduling). As soon as one request finishes generating its final token, the system instantly injects a new request into the very next decode iteration.
• Dynamic batch sizing: The maximum batch size should not be a hardcoded number. It should dynamically adjust based on the current queue depth and the available KV cache memory. If traffic is light, use smaller batches to minimize TTFT. If traffic spikes, increase the batch size to maximize overall cluster throughput, accepting a slight hit to latency.

Demonstrating senior engineering judgment

Passing the OpenAI system design interview takes a lot more than just drawing a few boxes on a screen and naming AWS services. Interviewers are evaluating your ability to navigate ambiguity, justify your trade-offs, and protect system reliability under extreme load. Always anchor your design decisions in the user experience. That means minimizing Time to First Token (TTFT), maximizing Inter-Token Latency (ITL) consistency, and ensuring that a failure in one component does not cascade into a total system outage. If you can confidently discuss the trade-offs between SSE and WebSockets, architect a secure Firecracker sandbox, manage distributed token buckets, and explain the intricacies of KV cache routing, you will easily stand out as a highly capable infrastructure engineer ready to build at the absolute cutting edge.