In-depth analysis · About 7 min read
How OpenAI reworked Habitat storage: shared capacity, regional caches and a qualified 12× read gain
OpenAI explains Cosmos DB tenancy, CDC invalidation and Rust in Habitat; its reported median read gain combines several changes.
- Published here
- Source published
- Source checked
In an October 23 engineering article, OpenAI describes how Habitat evolved its Azure Cosmos DB storage and read path. The account covers operational tradeoffs and measurements reported by OpenAI, rather than a new model release or a public storage product.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- OpenAI attributes a roughly 120 ms to 10 ms reduction in global read p50 to combined data placement, regional caching and a Rust rewrite; the slowest uncached requests still face distant storage.
- After trying shared and then isolated containers, Habitat returned to shared tenancy by default with Azure co-engineering. A separate exabyte-scale system remains in development.
Why a shared database could throttle unrelated features
OpenAI describes Habitat as a storage layer supporting more than 500 petabytes and products serving over 1 billion users across 40 geographic regions. These are the author’s scale claims, not an independently measured user count or an announcement that this many users joined. The article’s practical problem is how to pool capacity without allowing one feature to damage another. Initially, each Cosmos DB account used one default container, with clients selecting logical partition keys. Common identifiers could therefore make different use cases share the same key’s limits.
RU/s denotes provisioned request units per second, rather than a fixed number of queries. Although the initial design relied on a stated 10,000 RU/s logical-key limit for backpressure, physical partitions actually enforce shared limits. Unrelated keys on one partition can compete. There is another trap: if storage growth splits a physical partition while total container throughput stays fixed, each child gets less throughput. Storage-heavy workloads can thus lose effective capacity without an increase in request rate. The diagrams explain these mechanisms; their proportions are not production measurements.
Why isolation helped, then became an operating burden
Dedicated containers gave critical services separate physical partitions and reduced the blast radius of noisy neighbors. OpenAI also synchronized provisioned throughput with physical-partition counts to manage weak partitions. The price was a capacity-planning obligation at each service boundary. A launch from internal testing to large public traffic could overwhelm a small isolated container, while the same increment would barely disturb an already large shared pool. Isolation moved failure boundaries, but did not remove the need to anticipate launch demand.
Habitat eventually returned to shared containers by default, together with hot-key throttling, lower-priority treatment of bulk work and a serverless variant built with Azure. The article contrasts an earlier 5,000 RU/s physical-partition ceiling with a tailored mode allowing up to 10,000 RU/s and splits triggered by RU usage as well as storage. This is a described collaboration, not evidence that every Azure customer can enable the same configuration. Our analysis: pooling becomes useful again only when enforcement protects interactive work and the capacity control loop can react to demand.
Why a managed service still needed shared failure testing
OpenAI says its largest Cosmos DB account holds tens of petabytes and serves more than 20 million queries per second. A power outage in late 2024, followed by a failback incident around a month later, exposed operating assumptions at that scale. A power outage led to a failover that would take several hours for the largest accounts; around a month later, failing them back caused a connection stampede that overwhelmed the original region’s gateway. The client reconnected there before all relevant physical partitions had returned. The account describes different failure stages, not one uninterrupted outage.
The response extended beyond buying more capacity. Habitat and Azure teams jointly monitor layers, test against a production-sized nonproduction account, iterate on SDKs, and maintain direct engineering escalation. The article also describes gateway metadata caching, load protection, connection reuse and separation of critical and offline workloads. For other teams, our inference is to test restoration as well as failover at a representative size. Provider ownership of the database does not by itself validate a customer’s connection behavior, recovery duration or acceptable blast radius.
What the 12× read improvement actually includes
Reads represent more than 90% of Habitat requests, according to OpenAI. Its strategy first places data near users where physical capacity permits, then broadly replicates a few small, latency-sensitive datasets, and uses nearby Valkey caches for other hot objects and edges. Some critical accounts replicate to all 40 served Azure regions, but this is not a statement that all data is copied everywhere. Storage competes with serving compute for regional power, cooling and space. Placement therefore remains a product and capacity tradeoff, not an unlimited locality guarantee.
Cache invalidation comes from Cosmos DB change feeds: CDC consumers invalidate local copies after updates, and a later read refills them. OpenAI reports propagation globally within seconds, supporting freshness without relying only on TTL expiration; this is not instantaneous consistency. A Rust rewrite also reduced service overhead. Together these changes brought global read p50 from roughly 120 ms to roughly 10 ms. The chart marks nearby replication, in-region caching and Rust as separate changes. The 12× figure cannot be assigned to Rust alone, generalized to write latency or read as every user’s total ChatGPT response time.
CDC scale and the boundary between deployed and planned systems
CDC also feeds secondary indexes, movement pipelines and analytics. OpenAI reports millions of captured changes per second across thousands of containers and millions of container partitions, through more than a hundred Kafka topics. A control plane assigns regions and workers; replica reads and lower priority protect interactive traffic. Stream isolation, lease handling, dead-letter queues and switching readers during regional failures address different failure modes. Dead-letter handling prevents a permanently failing delivery from blocking the stream; it does not prove every downstream consumer is already current.
The remaining constraints matter when applying this account. Slow cache misses still reach remote storage, and the article does not supply a controlled comparison separating every optimization or a universal tail-latency guarantee. OpenAI is building an exabyte-scale system using Rockset’s RocksDB Cloud core with Raft replication and a Cosmos-compatible document model. Disaggregated storage and physical replication aim to reduce flash needs in exchange for lower peak QPS. It remains in development, not a completed Cosmos replacement. We read Part II and its read-latency chart; linked Part I, SDK sources and service guarantees were not independently verified.
Why it matters
For infrastructure teams, the useful comparison is between workload isolation, aggregate spare capacity and the cost of operating each boundary. Cache freshness, failover tests and tail latency need separate acceptance criteria.
Limits and availability
These are OpenAI’s engineering claims, not an AISubHub reproduction or an Azure service guarantee. The article does not establish universal access to its customized serverless mode, new subscription benefits or application-wide latency gains.
Sources and evidence
Source published: · Source checked:
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.