How to Pass the OpenAI Decomposition Interview

The OpenAI decomposition—or "decomp"—interview has a reputation for being incredibly ambiguous. Standard system design rounds usually ask you to build a clone of Twitter or Uber, where the constraints and architecture patterns are well-documented across dozens of prep books. Decomp takes a totally different approach. It tests how well you can take a messy, open-ended product requirement and hammer it into a concrete technical architecture. If you're interviewing for a Software Engineer or Forward Deployed Engineer role, expect this round to push your limits. The interviewers want to see how you navigate that initial fog, define strict API boundaries, and make pragmatic trade-offs when the pressure is on.

OpenAI engineers operate in an environment where the product playbook hasn't been written yet. You are often building features that rely on probabilistic models, unpredictable latency, and massive data pipelines. The decomp round simulates this reality. Your interviewer isn't looking for a perfect, textbook architecture. They are looking for your ability to logically sequence a problem, identify the riskiest assumptions, and build a system that won't collapse when edge cases inevitably hit.

The OpenAI Engineering Interview Process

Before diving into the specifics of decomp, it helps to understand where it fits into the broader interview loop. The decomposition interview is typically the linchpin of the onsite stage.

What is the Decomposition Interview?

Think of the decomp interview as the bridge between product management and system architecture. Your interviewer will toss out a high-level business problem—like deploying a custom LLM solution for a Fortune 500 client, or building a system to evaluate model regressions—and then they'll sit back. They expect you to grab the wheel and drive the conversation.

You will need to dig up hidden constraints, define the data models, outline API contracts, and sketch out a viable backend architecture. It's not about just drawing boxes on a whiteboard. Instead, it focuses heavily on logical sequencing. How exactly do you take a massive, unruly problem and slice it up into deliverable, technically sound milestones? If you jump straight to drawing a microservices diagram before you know the expected QPS or the shape of the data, you will fail the round.

The 4-Stage Decomp Framework

To prevent yourself from spinning out in the ambiguity, you need a repeatable framework. Treat the interview as a collaborative working session with a staff engineer. Follow these four stages linearly, and do not move to the next stage until the current one is rock solid.

• Scoping and Constraint Gathering: Start by asking targeted questions to put boundaries around the problem. What is the expected QPS? What kind of latency requirements are we looking at? Do we need to worry about specific data privacy constraints like HIPAA or SOC2? Are we optimizing for high availability or strict consistency?
• API and Interface Design: Nail down the exact inputs and outputs before you even start talking about the backend. Take the time to write out the REST or gRPC endpoints. Lay out the exact JSON request payloads, and define the response structures. If you are building a batch processing system, define the webhook payload that signals completion.
• Data Modeling: Sketch out the database schema. You will need to choose between a relational setup like PostgreSQL if you need transactional guarantees, a NoSQL store like DynamoDB for high-throughput key-value lookups, or a Vector database like Pinecone or Milvus for handling unstructured data and embeddings.
• Architecture and Trade-offs: Now you connect all those components. Talk through caching layers using Redis, or message queues like Kafka for asynchronous processing. Explicitly call out the trade-offs for every choice you make. If you choose Kafka, acknowledge the operational overhead. If you choose Redis, discuss cache eviction policies.

Deep Dive Scenario 1: Enterprise RAG for Legal Documents

Here is a classic scenario you might face, especially if you are going for a Forward Deployed Engineer role. The prompt: 'A massive law firm wants an internal ChatGPT that can securely search and summarize their 10 million historical case files and contracts. How do you build this?'

A weak candidate will immediately start talking about vector databases. A strong candidate starts with the ingestion pipeline. 10 million PDFs require a robust, asynchronous processing architecture. You should propose an event-driven ingestion flow: documents are uploaded to an S3 bucket, which triggers an event to an SQS queue. A fleet of worker nodes (perhaps running Celery or a Go routine pool) picks up these tasks to extract the text. You must explicitly mention how you handle OCR failures or malformed PDFs, perhaps routing them to a Dead Letter Queue (DLQ) for manual review.

Next, discuss chunking. Naive fixed-size chunking will destroy legal context—cutting a sentence in half right before a crucial 'unless' clause. Propose a semantic chunking strategy, such as recursive character text splitting with a 10-15% overlap, to ensure context bleeds across chunk boundaries. Once chunked, the text is sent through an embedding model (like text-embedding-3-small) and stored.

For retrieval, rely on hybrid search. Vector search is great for semantic meaning ('breach of contract'), but terrible for exact matches (like a specific case number, e.g., 'TX-2023-8912'). Combining dense vector search with sparse BM25 guarantees you catch both semantic intent and exact keyword matches. You would merge these results using a Reciprocal Rank Fusion (RRF) algorithm before passing the context to the LLM.

Finally, tackle security. Legal documents have strict access controls. Explain how you would implement role-based access control (RBAC) at the vector database level. You must use pre-filtering—attaching metadata tags (like 'access_group: partners_only') to the vector payload. If you use post-filtering (retrieving the top 10 vectors and then filtering out unauthorized ones), you risk returning zero context to the LLM if the user lacks permissions for all 10 retrieved chunks.

Deep Dive Scenario 2: Rate Limiting the OpenAI API

This scenario tests your grasp of distributed systems, concurrency, and low-latency design. The prompt: 'Design the rate limiting service that sits in front of the OpenAI API. It needs to handle millions of requests per minute, enforcing both Requests Per Minute (RPM) and Tokens Per Minute (TPM) limits per user.'

Start by defining the algorithm. The Token Bucket algorithm is the industry standard for this. You have a bucket with a maximum capacity, and tokens are added at a fixed rate. When a request comes in, you check if the bucket has enough tokens. For TPM, the number of tokens required equals the estimated token count of the prompt. If the bucket is empty, you return an HTTP 429 Too Many Requests status code.

The implementation is where candidates stumble. You cannot just read a value from a database, decrement it in application memory, and write it back. In a high-throughput environment, concurrent requests will read the same initial value, bypassing the limit entirely. You need atomicity. Propose a distributed architecture relying on Redis, specifically utilizing Redis Lua scripts. A Lua script executes atomically on the Redis server, ensuring that the 'check balance and deduct' operation cannot be interrupted by race conditions.

Your interviewer will likely push back: 'What happens when Redis becomes a bottleneck?' This is your cue to discuss scaling and trade-offs. You can partition the Redis cluster, sharding the data by user ID so no single node takes all the traffic. If latency is still too high, you can introduce local in-memory buckets on the API gateway nodes. These local buckets sync with the central Redis cluster asynchronously. Be upfront about the trade-off here: you are trading strict consistency for high availability and low latency. During the brief synchronization windows, a user might slightly exceed their quota, which is usually an acceptable business risk compared to taking down the entire API via Redis exhaustion.

Deep Dive Scenario 3: Asynchronous Batch Processing System

OpenAI offers a Batch API where users can upload massive JSONL files containing thousands of prompts, which are processed asynchronously at a 50% discount within 24 hours. The prompt: 'Design the system that powers the asynchronous Batch API.'

Begin with the API contract. The user hits a `POST /v1/batches` endpoint with a file ID. The immediate response should be a 200 OK returning a batch ID and a status of `validating`. This fast synchronous response is critical for a good developer experience.

The architecture relies heavily on state management. The uploaded file lives in an object store (S3). A background worker streams the file, parsing each JSON line to validate the schema and estimate token counts. If valid, the batch state in your primary database (PostgreSQL) updates to `in_progress`. The individual prompts are then pushed onto a distributed message queue (Kafka or RabbitMQ).

Here, you must discuss resource allocation. How do you ensure batch jobs don't starve real-time API traffic? You run a separate pool of LLM inference workers specifically dedicated to batch processing. These workers pull from the queue, execute the prompts, and write the results back to a temporary datastore. Once the queue for a specific batch ID is empty, a final aggregator service stitches the results back into a single JSONL file, uploads it to S3, updates the database state to `completed`, and fires off a webhook to notify the user.

Common Red Flags

Interviewers are trained to spot certain anti-patterns. Avoid these mistakes to keep the interview on track.

• Jumping straight into architecture: Proposing some massive, complex microservices setup before you have even defined the API contracts or figured out the user scale. Always define the 'what' (the data and API) before the 'how' (the infrastructure).
• Ignoring failure modes: Distributed systems fail constantly. Forgetting to talk about retries with exponential backoff, dead-letter queues, or fallback mechanisms for when external APIs—like the LLM itself—inevitably time out shows a lack of production experience.
• Over-engineering the solution: Recommending Kafka, Cassandra, and Kubernetes for a tiny internal system that only needs to handle 10 requests per second. You should always justify your technology choices using the constraints you gathered earlier. A simple Postgres database and a monolith is often the correct starting point.
• Silent assumptions: Designing a system assuming it is read-heavy without explicitly confirming that with the interviewer. If you guess wrong, your entire architecture will be optimized for the wrong bottleneck.

How to Prepare for the Decomp Round

The best way to practice for the decomposition interview is to reverse-engineer existing OpenAI products. Take a feature like Custom GPTs or the Assistants API. Sit down with a blank document and try to write out the exact API endpoints required to make it work. What does the JSON payload look like when a user uploads a file to an Assistant? How does the backend track conversation history threads?

Once you have the API, sketch the database schema. Then, draw the architecture. Finally, write down three things that could break under heavy load (e.g., database connection exhaustion, cache stampedes, token limit miscalculations) and how you would mitigate them. Practicing this end-to-end flow will build the mental muscle memory you need to confidently navigate the ambiguity of the actual interview.