Design Harmful Content Detection
On this page
Problem Description
In this interview, we're going to design the system that finds harmful posts on a large social network, similar to how Facebook and Instagram find posts that break their rules. People publish posts with text, photos and videos, and a small share of them break the rules: hate speech, violence, nudity, scams. When a post is published, the system checks it and decides whether to leave it up, show it to fewer people, remove it, or send it to a human moderator. People can also report posts they think are harmful, and the system looks at those again. Moderators review the posts sent to them and make the final call, and the system keeps learning from their decisions. You can use existing building blocks: a posts service that stores posts and their photos and videos, counts how many people saw each post, and can remove a post or show it to fewer people; and a review tool that moderators already use to look at a post and record a decision.
Out of Scope
- Writing the rules themselves – the policy team decides what counts as hate speech; we enforce it.
- Appeals by the people whose posts were removed – a separate flow on top of the review tool; a good follow-up question.
- Fake accounts and spam bots – a different problem about accounts, not posts.
- Private messages – different privacy rules and a different product.
- Live video – it needs checks while the stream is running, a separate interview.
Good Clarifying Questions
- How many posts, and how many moderators? – About 500 million new posts a day; a third have photos and 1 in 20 is a video. About 5 million reports a day. 10,000 moderators review about 4 million posts a day in total.
- How much is harmful, and which harms matter most? – Under 1% of posts. Child safety and terrorism are the most severe: they must be removed even when we're less sure. Spam and nudity are the most common.
- How fast must a post be checked? – Text and photo posts within 10 seconds of posting, before most people see them; videos within 5 minutes; reported posts within 1 minute.
- Can the model remove posts on its own? – Yes, when it's very sure: the policy team wants at least 95% of automatic removals to be right. Everything else waits for a moderator.
- What if checks fall behind or the models are down? – Posts still publish. The checks catch up, and nothing is removed without a model or a moderator deciding.
Requirements
Functional Requirements
- Check Every New Post – When a post is published, the system leaves it up, shows it to fewer people, removes it, or sends it to a moderator.
- Look Again at Reported Posts – When people report a post, the system checks it again with the reports and sends it to a moderator if it may be harmful.
- Review Posts – Moderators get the posts sent to them, one at a time, and their decision is applied to the post.
- Keep Learning – The system trains a new model from moderators' decisions every week and uses it only after it proves at least as good as the current one.
Non-Functional Requirements
- Scale – 500M new posts a day: about 6K/s on average, 15K/s at peak; 5M reports and 4M moderator reviews a day.
- Latency – Text and photo posts checked within 10 s, videos within 5 min, reported posts within 1 min.
- Freshness – A moderator's decision applies to the post at once and reaches training within a day; a new model every week.
- Availability – 99.9%. If checks fall behind or the models are down, posts still publish and are checked as soon as possible; nothing is removed without a decision.
Try it yourself first
You have the full problem. Sketch your answer, then sign up free to read the complete walkthrough.
- The complete design, flow by flow, with pictures
- Deep dives into the hard parts
- What mid, senior and staff candidates are expected to say
- Full company question lists
Free forever. No card needed.