How Discord Reduced WebSocket Traffic by 40%

Compressing the Chaos: The Engineering Secrets Behind Discord’s Optimizations

🚀 TL;DR

Discord reduced WebSocket traffic by 40% using a combination of Zstandard compression and Passive Sessions V2. They leveraged streaming compression, tuned parameters, and replaced redundant data snapshots with delta updates. Here’s how Discord pulled it off—and what engineers can learn.

---

This Article is Powered By Inspector!

Add Real-Time Debugging To Your Web Application

If you are looking for HTTP monitoring, database query insights, and the ability to forward alerts and notifications to your preferred messaging environment try Inspector for free.

Inspector offers first party libraries for frameworks like Laravel, Symfony, and Spring Boot for a full featured experience with zero configurations needed.

Learn more on the website: https://inspector.dev

---

🔧 What Discord Needed To Solve

Handling real-time updates for millions of users is no small feat. Discord's gateway service delivers live updates, but this resulted in heavy bandwidth usage, particularly on mobile devices. The challenge?

1. Reduce bandwidth without compromising real-time responsiveness.

2. Improve compression while keeping memory and CPU usage under control.

3. Handle evolving patterns in data efficiently.

---

🌐 Discord’s Two-Part Solution

1️⃣ Streaming Compression with Zstandard

• Background: Since 2017, Discord used zlib for payload compression, but zstandard offered potential improvements in speed and compression ratios.

• Initial Experiment: Early attempts with zstandard showed worse results due to lack of streaming compression, which enables reuse of previous data.

• The Breakthrough: By modifying an open-source Elixir library, Discord enabled zstandard to support streaming, dramatically improving performance:

  • Payload sizes shrank from 270 bytes (zlib) to 166 bytes (zstandard).

  • Compression speed doubled, with zstandard taking 45 microseconds per byte compared to zlib’s 100.

2️⃣ Passive Sessions V2

• What are Passive Sessions? For users not actively viewing a server, Discord sends periodic updates to keep their client in sync without overwhelming them with unnecessary data.

• The Problem: The previous system sent full snapshots of server updates, even if only one element had changed.

• The Fix: With Passive Sessions V2, Discord only sends delta updates (what changed), reducing passive session traffic from 35% to just 5% of total bandwidth.

---

🚀 Key Challenges and Solutions

• False Starts with Zstandard: Initial experiments failed because Discord wasn’t using streaming compression. By adding support for this, they unlocked the real potential of zstandard.

• Complexity of Dictionaries: While zstandard dictionaries promised marginal improvements, the added complexity of managing them across clients outweighed the benefits.

• Memory Optimization: Efforts to optimize memory usage during off-peak hours faced challenges with memory fragmentation, highlighting the need to balance system intricacies with operational gains.

---

🌟 The Results

Through a combination of zstandard streaming and Passive Sessions V2, Discord slashed its gateway bandwidth usage by 40%. Here's how the improvements break down:

1. Zstandard Streaming: Improved compression ratios and speeds across all platforms (desktop, iOS, and Android).

2. Passive Sessions V2: Reduced redundant data transmission for inactive users, cutting traffic by 30%.

---

🧠 Advanced Insights: Engineering Learnings from Discord’s Optimizations

When it comes to large-scale systems, the devil is in the details. Discord’s journey to reduce WebSocket traffic by 40% offers a treasure trove of insights for senior engineers building distributed systems, optimizing compression algorithms, or solving high-scale data challenges. Let’s dive deeper into the engineering nuances that made this optimization possible and extract actionable takeaways for your own projects.

---

1. The Power of Streaming Compression

Streaming compression proved to be the turning point for Discord. Traditional compression algorithms work well for one-off tasks, but when dealing with long-lived connections—such as WebSockets—streaming compression becomes a game-changer. Here’s why:

• Efficiency through Context Reuse: Streaming compression allows the algorithm to leverage previously seen data, creating a rolling context that improves efficiency. Without streaming, each payload starts fresh, losing the opportunity to build on past patterns.

• Small Payloads, Big Impact: Discord’s payloads are small and frequent, making them the perfect candidate for streaming compression. If your system handles a similar workload, consider streaming options like zlib or zstandard.

• Lesson: Not all workloads benefit from streaming. If your data is mostly unique or large (e.g., video streams), the benefits may not outweigh the added complexity.

Key Takeaway: Understand your data's size and patterns before deciding on a compression strategy. Streaming compression works best when payloads are frequent, small, and repetitive.

---

2. Balancing Trade-Offs in Compression Settings

Discord's engineers tuned zstandard’s parameters—like chainlog, hashlog, and windowlog—to strike a balance between compression speed, memory usage, and efficiency. These parameters directly impact your system’s performance:

• Memory vs. Compression Ratio: Increasing parameters like chainlog improves compression but at the cost of higher memory usage. For instance, a larger chainlog gives the algorithm a longer history to reference, but this requires more memory.