What OpenAI learned scaling ChatGPT's storage layer on Cosmos DB

OpenAI has published Part II of its series on scaling Habitat, the online storage platform behind ChatGPT, the API, Codex and internal services. According to the post, Habitat answers in milliseconds for over 1 billion users across 40 geographic regions and holds more than 500 petabytes. Part II covers the layer underneath: Azure Cosmos DB, and how OpenAI and the Azure team made a multi-tenant design hold up. It is an infrastructure case study, not a model release, and the failure modes it describes show up in any shared datastore once enough teams depend on it.

A circular journey

The authors (Jon Lee, Hrachya Hakobyan and Chaomin Yu) describe tenancy going in a circle: one shared container, then single-tenant containers, then multi-tenant by default again.

Phase I relied on Cosmos DB primitives: one default container per account, with the documented 10K RU/s limit per logical partition key as back pressure. Three problems followed. Early API choices let clients pick logical partition keys directly, so common keys such as user_id and workspace_id collided across unrelated products and shared one key's limits. Limits are actually enforced on the physical partition, so noisy neighbors running niche features disrupted login and signup. And partitions got weaker over time.

Provisioned throughput is spread roughly evenly across a container's physical partitions. When data grows, partitions split, but total RU/s stays the same, so each partition's share shrinks. For storage-heavy, blob-like workloads, growth alone can push a partition's allocated capacity below its steady demand, and requests start returning 429s with no change in traffic. As a physicist I read this as a conservation problem: a fixed budget divided by a growing denominator.

Phase II gave critical new workloads their own containers, plus a script that kept provisioned throughput in step with partition count. That "mostly avoided" the Phase I problems, but every container needed its own capacity planning, and legacy critical workloads stayed in the default container because nobody migrated them.

Phase III returns to multi-tenant by default, with several mechanisms working together: the ability to move data between containers, hot-key throttling so a single hot key cannot overwhelm a shared partition, quality-of-service priorities so critical requests are protected under contention, serverless mode with improved partition scaling, and co-management with Azure. The original covers each in detail.

Global reads

The second half covers latency. The policy is clear from its structure: store data near users when capacity allows, broadly replicate only the smallest and most latency-sensitive datasets, rely on integrated caching for everything else, reduce proxy overhead with Rust, and scale change data capture. OpenAI reports global p50 read latency falling from roughly 120 ms to about 10 ms, the 12x improvement in the headline. The direction matches what I would expect: most reads in a system like this should never reach the primary store.

My take

The lesson is not "use Cosmos DB" or "go multi-tenant." It is that tenancy is an operating discipline, not a decision you make once. Shared containers became workable here because hot-key throttling, request priorities and better serverless partition scaling contained interference, and data movement meant an early layout mistake no longer had to be permanent. No single mechanism did it alone.

Two smaller points are worth taking:

1. Design partition keys so workloads cannot collide. Namespace keys by use case, or have the platform derive them, so one product's user_id never shares limits with another's.
2. Watch throughput per partition, not just per container. Weak partitions are invisible on a container-level dashboard. A workload with flat traffic and growing storage is exactly the one that will page you.

For teams building agentic products this matters more than it looks. Memory, conversation state and tool traces are growing, unevenly keyed, storage-heavy datasets: the same profile that produces weak partitions and noisy neighbors.