Postgres LISTEN/NOTIFY: Does it really scale?
Postgres LISTEN/NOTIFY has a bad reputation, partly due to a popular post claiming it doesn't scale. If that were true, it would be a shame, as LISTEN/NOTIFY is a powerful tool, allowing you to use your Postgres database for durable, low-latency notifications, streams, and pub/sub (publish/subscribe, a communication pattern where message senders don't send them directly to specific receivers, but rather categorize them into topics). The accusations aren't wrong: NOTIFY has unintuitive and undocumented performance characteristics, stemming from the use of a global lock. But "unintuitive behavior" is not the same as "unscalable."
In this article, we'll show how we optimized LISTEN/NOTIFY-based streams at scale, achieving 60 thousand writes per second on a single Postgres server with millisecond latency.
The basic architecture of Postgres-based streams is simple: you create a streams table where each "chunk" of the stream (for example, an LLM response token – small pieces of text generated by large language models) is a new row, and then you write to the streams by inserting into the table.
The tricky part is reading from the stream, as you don't know when the next chunk will arrive. One solution is polling (a method where one system repeatedly checks another at regular intervals to see if there's new information): having each reader query the end of the stream for new chunks. However, polling scales poorly. If the polling interval is too high, latency is too high for interactive use cases (like online chats). But if the polling interval is too low, concurrent pollers overload the database.
The better solution is LISTEN/NOTIFY. This allows readers to block, waiting for a notification from a writer that a new chunk has been published to the stream. This way, readers don't waste resources with polling, but wake up immediately when a new stream chunk arrives.
In our initial implementation of LISTEN/NOTIFY-based streams, a trigger on the streams table fired a function that sent a notification every time a new stream chunk was written. Readers waited for these notifications and woke up to a new stream chunk.
This implementation was correct and delivered low latency, but at scale, its throughput was low. Even using a large Postgres database, it couldn't sustain more than 2.9 thousand stream writes per second. Interestingly, it hit the bottleneck without visibly consuming any Postgres resources (CPU, memory, or IOPS). As you might have guessed, the root cause was the original "LISTEN/NOTIFY doesn't scale" problem: a global lock that Postgres acquires during NOTIFY. But why does Postgres do this, and how can we optimize it without losing the benefits of Postgres notifications?
Unraveling the Knot: The LISTEN/NOTIFY Exclusive Lock
To understand the problem, we need to examine how Postgres LISTEN/NOTIFY actually works.
The root cause of the poor performance is that, in Postgres, committing a transaction that calls NOTIFY requires acquiring a global exclusive lock (a global exclusive lock, which prevents other operations from accessing or modifying a shared resource in the database). This lock is taken when the transaction begins to commit and is not released until the transaction is fully committed and its contents have been written to disk with fsync() (a system call that ensures cached data is physically written to disk).
This lock is necessary because Postgres guarantees that notifications are sent in transaction commit order (the order in which transactions are permanently saved to the database). To enforce this, it stores all outgoing notifications in a global internal queue whose order must exactly match the commit order of the transactions sending those notifications. Adding notifications to this queue must be done transactionally as part of the commit. However, Postgres doesn't assign a commit order to transactions until they have finished committing, as committing can take a variable amount of time.
This creates an ordering problem: transactions containing notifications must add themselves to the queue in commit order, but the commit order isn't defined until the commit is complete. The solution is the global lock, which serializes the commits of transactions containing notifications, so their commit order is defined in advance and they can correctly order themselves in the internal notification queue.
This exclusive lock explains the poor performance we observed. Since we called NOTIFY from a trigger on the streams table, each stream write includes a call to NOTIFY. To be committed, each stream write needs to acquire the global lock and hold it for the entire duration of its commit, including writing to disk. This means that stream writes need to be committed sequentially, preventing usual Postgres optimizations like group commit (which commits many transactions together in a single fsync()). As a result, stream writes cannot be completed faster than Postgres can commit transactions, which leads to this bottleneck. This also explains why we didn't see significant consumption of any Postgres resources, such as CPU or disk: there wasn't any, because all transactions were serialized by a global lock.
Incidentally, there has been some online discussion about a Postgres patch related to this issue. This patch (to be released in Postgres 19) neither removes the global lock nor fixes the bottleneck we observed. Instead, it optimizes the more restricted case where there are many notification channels and each listener is only waiting on a specific channel.
The Solution: Optimizing LISTEN/NOTIFY at Scale
To make LISTEN/NOTIFY-based streams faster, we need to bypass this bottleneck. The key observation is that, for streams, and for many other LISTEN/NOTIFY applications, notifications are not, in themselves, a source of truth. Instead, they merely "ping" a reader to check a database table (the true source of truth) for new data. As a result, notifications don't need to be globally ordered or perfectly durable, so we can optimize NOTIFY by buffering (temporarily storing in memory) the notifications and periodically writing them in a single batch transaction (a transaction that groups several smaller operations into a single larger operation), significantly reducing contention on the global lock.
Buffering and batching NOTIFYs avoid the bottleneck because the global lock only needs to be acquired when the buffer is written, and not for each individual stream write. This means that individual stream writes can proceed quickly, taking advantage of Postgres optimizations like group commit to achieve high throughput, while the buffer writes in the background.
Adopting a buffer introduces a new complication: a process failure could lead to the loss of notifications that haven't yet been written. However, given that notifications are just a "ping" to check the source of truth, this is generally an acceptable risk, as the reader can simply re-synchronize if it misses a notification.
Why This Matters To You
The lesson here is clear: powerful tools like Postgres LISTEN/NOTIFY, even with their quirks, can be effectively used in high-performance scenarios. Instead of discarding a technology based on myths or superficial understandings, it's worth diving into the details of its implementation.
For developers and architects, understanding the internal mechanisms of the database, such as the nature of locks and commit optimizations, is crucial for building scalable systems. With the right strategy, it's possible to transform a supposed "bottleneck" into a high-throughput component, paving the way for real-time applications such as LLM response streaming, chat, and other interactive experiences that demand low latency and high capacity. Don't let a bad reputation prevent you from exploring the full potential of your tools.
Sources
📬 Enjoyed this? Subscribe to the Vaccari's Code newsletter for the next wave of software & AI trends, straight to your inbox: Subscribe here