372K views · Sep 10, 2012 · 1:08:17 · 5 HN recommenders · 8 mentions · discussed 2014–2023
Dropbox engineer Kevin Modzelewski gives a Stanford colloquium on Dropbox's history and the technical problems it faced scaling, including detailed database schema decisions.
Why Hacker News points people here: Recommended as a detailed, candid case study of scaling a real system from zero users to millions.
“I found the Dropbox lecture [1] at Stanford one of the most riveting things ever.”HN comment, 2018
“I found the Dropbox lecture [1] at Stanford one of the most riveting things ever.”
Riveting look at the technology behind Dropbox.
I found the Dropbox lecture [1] at Stanford one of the most riveting things ever. There is just so much technology behind Dropbox, it is staggering. There is a reason why it is so much better than iCloud sync, Google Drive, Box or OneDrive. [1] this video
“It is incredibly detailed (down to the exact sql schema they use for file metadata etc).”
Incredibly detailed talk on Dropbox architecture evolution, down to SQL schema.
This is a talk about the evolution of Dropbox's architecture from 2012: this video . It is incredibly detailed (down to the exact sql schema they use for file metadata etc).
“there are a lot of good talks on youtube from places like dropbox and facebook that explain the problem and solution.”
Case study on scaling problems and solutions at Dropbox.
Applications dont really need to be well architected until they are hitting scale. Then the parts of their system that need to relieve pressure will need to be re-architected. This is almost like a case study and there are a lot of good talks on youtube from places like dropbox and facebook that explain the problem and solution. Example: this video If you dont want to do youtube case studies there are also books to read about distributed systems. Also reading about cloud architecture can help.
“You might enjoy this talk”
Shows Dropbox is far more than a GUI on rsync.
Dropbox is a lot more than a GUI on top of rsync. Even purely from an engineering standpoint (ignoring product & design) that's incorrect. You might enjoy this talk: this video
“Here's some of my favorite talks that go from 0 users to millions.”
Talk going from zero users to millions.
imo the best way is to look at what other's have built. Here's some of my favorite talks that go from 0 users to millions. Dropbox - this video Instagram - link Slack (bonus. not as applicable but good reminder of why initial architecture does matter) - link
“there are a lot of good talks on youtube from places like dropbox and facebook that explain the problem and solution.”
Example scaling case study.
> Applications don't really need to be well architected until they are hitting scale. very True, 'a system Well architected' before hitting scale is considered OVER Engineering > Then the parts of their system that need to relieve pressure will need to be re-architected. > This is almost like a case study and there are a lot of good talks on youtube from places like dropbox and facebook that explain the problem and solution. Example: this video
“At 48:36 they explain why they changed from PRIMARY KEY (id) to PRIMARY KEY (ns_id, latest, id).”
Explains Dropbox's change to composite primary key for data locality.
An important detail which this guide leaves out: The "actual location of the row" is usually the leaf node of another B-Tree. That is the primary B-Tree and it's indexed by the primary key. The major consequence is every non-primary index query involves dereferencing N pointers, which could mean loading N pages from disk/SSD to get leaf nodes spread out across the primary tree. Whereas if you query a range in the primary B-Tree directly, the rows are consecutive so you would only load N/M consecutive pages based on the ratio of rowsize to pagesize. That's why some people use composite keys for primary key, to get better data locality in the primary B-Tree index. See "How We've Scaled Dropbox"[1] to hear the same thing. At 48:36 they explain why they changed from PRIMARY KEY (id) to PRIMARY KEY (ns_id, latest, id). ns_id is kinda like user_id. So that change groups journal entries for users together on disk Specifically. PRIMARY KEY (id) orders things by creation date whereas PRIMARY KEY (ns_id, latest, id) orders things by ns_id primarily. [1]: this video
“Illustration from Dropbox scaling talk below.”
Illustrates primary key clustering for range queries.
Postgres is >50x slower for range queries(example below) and is akin to using array-of-pointers (ie Java) whereas MySQL supports array-of-struct (C). Illustration from Dropbox scaling talk below. Sneak peek photo [1] (from [2]). Just imagine its literally 500-1000x more convoluted per B-tree leaf node. That's every Postgres table unless you CLUSTER periodically. [1]: link [2]: link Mind boggling how many people aren't aware of primary indexes in MySQL that is not supported at all in Postgres. For certain data layouts, Postgres pays either 2x storage (covering index containing every single column), >50x worse performance by effectively N+1 bombing the disk for range queries, or blocking your table periodically (CLUSTER). In Postgres the messiness loading primary data after reaching the B-tree leaf nodes pollutes caches and takes longer. This is because you need to load one 8kb page for every row you want, instead of one 8kb with 20-30 rows packed together. Example: Dropbox file history table. They initially used autoinc id for primary key in MySQL. This causes everybodys file changes to be mixed together in chronological order on disk in a B-Tree. The first optimization they made was to change the primary key to (ns_id, latest, id) so that each users (ns_id) latest versions would be grouped together on disk. Dropbox scaling talk: this video If a dropbox user has 1000 files and you can fit 20 file-version rows on each 8kb disk page (400bytes/row), the difference in performance
