Skip to content

Twitter search

Problem Statement

Design a scalable, low-latency system to index and search billions of tweets in near real time for a global user base. The system must support search, trending topics, and autocomplete, while handling massive write and read traffic with high availability.

You are expected to design this as if it were going into production at Twitter scale.

Functional Requirements

Your design must support:

  1. Tweet Search

  2. Search by:

    • Keywords and phrases
    • Hashtags
    • User IDs
    • Filters: time range, geo/location, language
    • Trending Topics

  3. Identify and rank trending hashtags and phrases:

    • Globally
    • Per region
    • Trends must reflect near real-time activity
    • Autocomplete

  4. Provide autocomplete suggestions while users type

  5. Suggestions should be based on:

    • Popular queries
    • Recent searches
    • Trending terms
    • APIs

  6. Search API

  7. Trending topics API
  8. Autocomplete API

  9. Ranking

  10. Rank search results using:

    • Recency
    • Popularity (engagement)
    • Optional personalization signals

Non-Functional Requirements

Your system must meet the following constraints:

  1. Scale

  2. Billions of tweets stored

  3. Thousands to tens of thousands of queries per second globally

  4. Latency

  5. P99 latency ≤ 200 ms for search and autocomplete

  6. Freshness

  7. Trending topics freshness < 1 minute

  8. Newly created tweets should appear in search within seconds

  9. Availability

  10. ≥ 99.99% uptime

  11. Tolerant to regional and data-center failures

  12. Consistency

  13. Eventual consistency is acceptable across shards and regions

What You Should Deliver

Provide a practical, production-oriented design that includes:

  1. Requirement clarification & assumptions

  2. High-level architecture

  3. Core services

  4. Data flow (tweet ingestion → indexing → search)

  5. Data storage choices

  6. Indexing strategy

  7. Sharding and partitioning

  8. Search architecture

  9. How queries are executed and ranked

  10. How latency targets are met

  11. Trending topics computation

  12. Windowing strategy

  13. Real-time vs batch trade-offs

  14. Autocomplete design

  15. Data sources

  16. Update frequency

  17. Scalability strategy

  18. Horizontal scaling

  19. Hot partition mitigation

  20. Failure handling

  21. Regional failover

  22. Backpressure, retries

  23. Rough capacity estimates

  24. Tweets/day

  25. Index size
  26. QPS assumptions

  27. Trade-offs

    • Explicitly explain what you choose not to optimize and why

Expectations

  • Be concrete (name technologies or categories: inverted index, stream processor, cache, etc.)
  • Avoid academic theory unless it directly impacts production behavior
  • Prefer simple, reliable designs over clever ones
  • Assume this system will be maintained by hundreds of engineers over many years

Interview Kit

Read first: Solution · Sharding §9 secondary indexes · Stream processing · Indexing §8 full-text

Curveballs. The interviewer changes one thing mid-design. The hint in italics is what a strong answer reaches for:

  1. A user blocks someone. Within how many seconds must the blocked person's tweets disappear from their results, and where is that enforced? (At query time, per viewer, not at index time.)
  2. An election night sends query volume 20× for one hour, all for five terms. (Result cache with short TTL, request coalescing, and shedding of expensive queries.)
  3. Legal requires a tweet to be withheld in one country only. (A visibility class evaluated with the viewer's region.)
  4. Autocomplete suggests a private medical query typed by one user. How did that happen, and what threshold prevents it?

Must answer (security, privacy, operations):

  • Deleted tweets: the SLO for disappearing from results vs. from the index, and how tombstones bridge the gap
  • Query-log retention and anonymization. Why user text never reaches query_string

Phase it (MVP → Growth → Scale): MVP: Postgres full-text search plus Redis for trends. Growth: Kafka to Elasticsearch with time-sharded indexes. Scale: earlybird-style in-memory real-time tier plus archive tier, multi-region.

Score yourself with the rubric.