✏️ tldr;
Levels.fyi is a website where users can see anonymous and publicly sourced salary information.
The website used Google Sheets as their backend in the beginning but eventually needed to scale to use an actual server.
They did this by duplicating database queries to a real backend server before migrating completely.
🧠 Give me Context!
When Levels.fyi was being built, they wanted to move quickly without a backend. To save time, they used Google Sheets as their backend to read, write, and parse their salary data on JSON files (you read that correctly).
In between was using Amazon Lambda functions, a server less framework to call their API's. This outsourced any infrastructure management to Amazon and Google.
📋 What's the Problem?
They started to experiencing the following problems
1️⃣ Slower page load time
2️⃣ Timed out lambda functions
3️⃣ No good SQL data analysis
4️⃣ Google sheets API limits reached
5️⃣ Salary data is vulnerable to scrape
❌ Requirements and Must Do's
You can call this a backend migration and here's what they had to do.
The new backend cannot be restricted by API limiting (AWS, Google)
Move salary data to a real database server
Seamless migration from old to new backend without affecting live traffic
🚧 Design something pretty
While the requirements are restrictive, the solution is actually simple.
Since this was a completely overhaul of their old Google Sheets backend, they had the freedom to build a real backend server and set up a real database server without problems.
The challenge was to migrate reads and writes over.
Levels.fyi did this by
Writes - Duplicating new write calls to their database
Reads - Slowly call new read API's as data was being migrated.
💰 HELP WANTED
This newsletter has grown to 16,000 → 17,500 AMAZING READERS. It’s grown to a scale that a single person can’t maintain all of it on their own.
If you’re interested in being a byte-sized design writer, apply here!
📝 Official Article
(Links to official article and sources are available to paid subscribers. They help maintain and support this newsletter!)
Keep reading with a 7-day free trial
Subscribe to Byte-Sized Design to keep reading this post and get 7 days of free access to the full post archives.