⚡️TLDR;
Discord has a server with 10,000,000 concurrent users and keeps things under control by
Sending events only to people who actively check that server
Splitting work from a single process to multiple sub-processes
Being efficient with in memory databases
🔥 Too many users is a good problem. Fanning out is not.
The biggest issue Discord has for a server with 10,000,000 users is fanning out notifications to them.
When someone uses a call like “@everyone”, everyone in that humongous server is notified. That’s 10 million notifications in a single message!
🔧 Here’s the requirements
Coming straight from the official article:
Ensuring that we can keep almost all operations quick is important to the server feeling like it's responsive:
When a message is sent, others should see it right away;
When someone joins a voice channel, they should be able to start participating right away.
Dropping the “@everyone” command isn’t an option so we need to figure other ways to make it work.
👏 Optimization #1: Do less fam
The simplest way to handle to much throughput is to put less through
Before doing crazy technical solutions, Discord figured out if how much fanning out it actually needed to do.
Turns out not everyone is active in every server at all times of the day.
So their first optimization was to just send notifications when someone peeked into that server. Lazy loading the notification was a huge first step.
How good was this optimization?
Turns out 90% of users aren’t active in discord servers at the same time. This means fan-out work got reduced by 90% for any given server wide broadcast.
That’s pretty good… but that eventually this fan-out resource problem will still be an issue if a server grows 10x.
⚖️ Optimization #2: Split the work
For simplicity, a “Guild” is mostly interchangeable with discord server. (Details can be found in the official article.)
If there’s work that needs to be done on a server, it can send it to multiple sub-processes, or Elixir “Relays”. These relays handle the sub-work for individual sessions like if a user is on mobile, desktop, or an app created by another developer.
Summary: Take notifications and split that work up.
🔨 Optimization #3: Use worker processes
If the server is busy doing other things, a worker process can be spun up to handle other expensive tasks.
For things that need to be commonly referenced like all the members in a server, it’s expensive to keep getting an entire list from a database.
So Discord used an in-memory database instead to store all the member data. Now every worker process can read from the same in-memory database.
It also means every “@everyone” ping becomes more performant and somewhat annoying if misused by trolls.
💰 HELP WANTED
This newsletter has grown to 9600 → 10,500 AMAZING READERS. It’s grown to a scale that a single person can’t maintain all of it on their own.
If you’re interested in being a byte-sized design writer, apply here!
📝 Official Article
(Links to official article and sources are available to paid subscribers. They help maintain and support this newsletter!)
Keep reading with a 7-day free trial
Subscribe to Byte-Sized Design to keep reading this post and get 7 days of free access to the full post archives.