Design the Twitter API

Context#

The Twitter (now X) API is the textbook read-heavy fan-out problem. Roughly 100 reads per write at planet scale: a single tweet by a popular account is read by tens of millions of timelines within seconds. The interviewer wants to see whether you can design an API that survives that asymmetry without trying to solve every problem in the data plane.

The classic interview overlap with high-level design is real: fan-out-on-write vs fan-out-on-read, follow-graph storage, search indexing, hot-key mitigation. We acknowledge the overlap up front and cut decisively — this writeup is about the public API contract, not the internal Manhattan / Heron / Mesos topology that delivers it.

The interviewer’s hidden objectives:

• Can you defend timeline fan-out as a hybrid (push-on-write for normal users, pull-on-read for celebrities)?
• Can you write a streaming endpoint (real-time filter / sample / rules) without conflating it with WebSockets-for-chat?
• Can you handle pagination for an infinite, write-mutating timeline (cursor-based, with since_id and max_id)?
• Can you separate the public read API (timelines, search, lookup) from the firehose / streaming API (sampled or filtered)?
• Can you make the rate-limit story legible — per-app, per-user, per-endpoint?

Scope cuts: trends ranking, ads, content moderation algorithms (Trust & Safety), the recommendation timeline (“For You”). We design the chronological-and-curated timeline contract plus the rest of the public surface.

Requirements (functional and non-functional)#

Functional — in scope:

• Post a tweet (POST /tweets). Reply, quote, retweet — same shape with a referenced tweet id.
• Delete a tweet (DELETE /tweets/{id}).
• Fetch a user’s home timeline (GET /users/{id}/timelines/reverse_chronological).
• Fetch a user’s profile timeline (GET /users/{id}/tweets).
• Look up tweets by id, batch or single.
• Follow / unfollow (POST/DELETE /users/{source_id}/following).
• Search recent tweets (last 7 days for standard; full archive for academic / enterprise tier).
• Real-time stream: filtered, sampled (the firehose-lite), or rules-based.

Functional — out of scope:

• The “For You” recommendation timeline (a model, not a contract).
• Trends API ranking (separate concern; its own surface).
• Ads / promoted tweets (separate API).
• Direct messages (a different state machine; see messenger writeups).
• Content moderation surface (Trust & Safety internal).

Non-functional:

• Timeline read latency: <= 150 ms p95. The user is staring at a refresh.
• Tweet creation: <= 300 ms p95 (includes write replication; fan-out is async).
• Stream connection: maintain millions of concurrent long-lived connections; sub-second delivery from tweet creation to stream subscribers.
• Availability: 99.95% on the read path; 99.9% on writes.
• Throughput: 5k tweets/s steady-state, 200k tweets/s during major sporting / political events. 500k timeline reads/s.

Use case diagram#

   ┌──────────────┐                              ┌───────────────────┐
   │   End user   │                              │ Partner / Bot dev │
   └──────┬───────┘                              └────────┬──────────┘
          │                                                │
   ┌──────┼──────────────┐                                 │
   ▼      ▼              ▼                                 ▼
 [post] [scroll] [follow / search]                  [stream rules]
   │      │              │                                 │
   ▼      ▼              ▼                                 ▼
 ┌──────────────────────────────────────────────────────────────┐
 │                   Twitter Public API                          │
 └──────────────────────────────────┬───────────────────────────┘
                                    │
              ┌─────────────────────┼─────────────────────┐
              ▼                     ▼                     ▼
        ┌──────────┐         ┌──────────────┐     ┌──────────────┐
        │ Timeline │         │   Search     │     │  Streaming   │
        │ service  │         │   index      │     │   gateway    │
        └──────────┘         └──────────────┘     └──────────────┘

Two actor types: end users (web / mobile, indirectly via Twitter clients) and partner / developer programs (direct API consumers — bots, journalism tools, academic research).

Class diagram#

   ┌──────────────────────┐        ┌──────────────────────┐
   │        User          │   *  * │       Follow         │
   ├──────────────────────┤◄───────┤──────────────────────┤
   │ id : Snowflake       │        │ source_id : User     │
   │ handle : string      │        │ target_id : User     │
   │ name : string        │        │ created_at : ts      │
   │ verified : bool      │        └──────────────────────┘
   │ followers_count : int│
   │ following_count : int│
   │ celebrity_flag : bool│        ── derived: followers_count > 1M
   └──────────┬───────────┘
              │ 1
              ▼ *
   ┌──────────────────────┐        ┌──────────────────────┐
   │       Tweet          │ 1    * │     Engagement       │
   ├──────────────────────┤◄───────┤──────────────────────┤
   │ id : Snowflake       │        │ tweet_id : Tweet     │
   │ author_id : User     │        │ user_id : User       │
   │ text : string ≤280   │        │ kind : enum (like,  │
   │ created_at : ts      │        │      retweet, quote,│
   │ in_reply_to? : Tweet │        │      reply)         │
   │ quoted? : Tweet      │        └──────────────────────┘
   │ media : Attachment[] │
   │ status : enum        │        ── active | deleted | hidden
   │ lang : iso639        │
   │ entities : Entities  │        ── extracted hashtags, mentions, urls
   └──────────────────────┘

Tweet is the central record. Engagement is a thin join — likes, retweets, quotes, replies are not separate tables in the API contract; they’re all foreign-keyed off (tweet_id, user_id, kind). The celebrity_flag is derived (not a stored column) — it drives the fan-out strategy for that user’s posts.

Sequence diagram (key flows)#

The post-and-fan-out flow:

 Client      API Gateway     TweetWrite      Snowflake       FanOut        Kafka       Timeline
   │ POST /tweets │              │              │              │              │             │
   │─────────────►│              │              │              │              │             │
   │              │ auth, rate   │              │              │              │             │
   │              │─────────────►│              │              │              │             │
   │              │              │  new id      │              │              │             │
   │              │              │─────────────►│              │              │             │
   │              │              │   id         │              │              │             │
   │              │              │◄─────────────│              │              │             │
   │              │              │  persist     │              │              │             │
   │              │              │              │              │              │             │
   │              │              │  emit event  │              │              │             │
   │              │              │─────────────────────────────►│             │             │
   │  201 Created │              │              │              │              │             │
   │◄─────────────│              │              │              │              │             │
   │                                            │              │              │             │
   │                                            │  celeb?      │              │             │
   │                                            │ ◄─decision   │              │             │
   │                                            │  if YES → don't push, mark for pull       │
   │                                            │  if NO  → push to N follower timelines    │
   │                                            │              │              │             │
   │                                            │  per-follower write          │             │
   │                                            │─────────────────────────────►│            │

The hybrid: normal users (≤1M followers) get push fan-out — the tweet is materialised into each follower’s per-user timeline. Celebrities (>1M followers) skip the push; their tweets sit in a “celebrity feed” that’s pull-merged at timeline read time.

The read flow:

 Client      API Gateway     TimelineRead     UserTimeline      CelebFeed
   │ GET /home_timeline │            │              │              │
   │──────────────────────────►      │              │              │
   │                  │ auth, rate   │              │              │
   │                  │──────────────►              │              │
   │                  │              │  fetch top-K │              │
   │                  │              │──────────────►              │
   │                  │              │   200 tweets │              │
   │                  │              │◄─────────────│              │
   │                  │              │ followed celebs?            │
   │                  │              │ ─────────────────────────► │
   │                  │              │   their recent tweets       │
   │                  │              │ ◄─────────────────────────── │
   │                  │              │  merge sort  │              │
   │                  │              │              │              │
   │  ranked page     │              │              │              │
   │◄─────────────────│              │              │              │

The merge step is where the celebrity tail joins the materialised push timeline. Cost is bounded by “how many celebs does this user follow” — usually <50, often <10.

Activity diagram (for non-trivial state)#

The tweet has a small state machine — but the fan-out has interesting structure:

                  [tweet created]
                        │
                        ▼
                ┌───────────────┐
                │ author celeb? │
                └──┬──────────┬─┘
                   │ no       │ yes
                   ▼          ▼
            ┌──────────┐  ┌──────────────────┐
            │ enqueue  │  │ write to celeb   │
            │  fan-out │  │ feed only;       │
            └────┬─────┘  │ pull-merge later │
                 │        └──────────────────┘
                 ▼
        ┌──────────────────┐
        │ per follower:    │
        │ INSERT into      │
        │ follower's       │
        │ home_timeline    │
        └──────┬───────────┘
               │
               ▼ (eventually consistent within ~5s)
        [followers see it]

State of tweet: active ──delete──► deleted (tombstone)
                        ──admin──► hidden  (visible to author only)

When a tweet is deleted, a tombstone propagates the same fan-out paths. Followers who already received the tweet in their materialised timeline see it disappear on the next read.

API implementation#

Endpoint catalogue#

Twelve endpoints across three resource families. The streaming endpoints use HTTP long-lived connections with chunked transfer encoding (newline-delimited JSON — NDJSON) — not WebSockets.

OpenAPI schema (excerpt)#

paths:
  /2/tweets:
    post:
      operationId: createTweet
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [text]
              properties:
                text: { type: string, maxLength: 280 }
                reply:
                  type: object
                  properties:
                    in_reply_to_tweet_id: { type: string }
                quote_tweet_id: { type: string }
                media:
                  type: object
                  properties:
                    media_ids:
                      type: array
                      items: { type: string }
                      maxItems: 4
      responses:
        '201':
          description: Tweet created
          content:
            application/json:
              schema:
                type: object
                properties:
                  data:
                    type: object
                    properties:
                      id: { type: string }
                      text: { type: string }

  /2/users/{id}/timelines/reverse_chronological:
    get:
      operationId: getHomeTimeline
      parameters:
        - { name: id, in: path, required: true, schema: { type: string } }
        - { name: max_results, in: query, schema: { type: integer, minimum: 5, maximum: 100, default: 100 } }
        - { name: pagination_token, in: query, schema: { type: string } }
        - { name: since_id, in: query, schema: { type: string } }
        - { name: until_id, in: query, schema: { type: string } }
        - { name: tweet.fields, in: query, schema: { type: string }, description: "Comma-separated; e.g. created_at,author_id,public_metrics" }
        - { name: expansions, in: query, schema: { type: string }, description: "Comma-separated; e.g. author_id,referenced_tweets.id" }
      responses:
        '200':
          description: A page of the home timeline
          content:
            application/json:
              schema: { $ref: '#/components/schemas/TimelinePage' }

  /2/tweets/search/stream:
    get:
      operationId: filteredStream
      description: "Long-lived HTTP. Each line is a JSON-encoded tweet matching the active rules."
      responses:
        '200':
          description: NDJSON stream
          content:
            application/x-ndjson:
              schema: { $ref: '#/components/schemas/Tweet' }

components:
  schemas:
    TimelinePage:
      type: object
      properties:
        data:
          type: array
          items: { $ref: '#/components/schemas/Tweet' }
        meta:
          type: object
          properties:
            result_count: { type: integer }
            newest_id: { type: string }
            oldest_id: { type: string }
            next_token: { type: string, nullable: true }
        includes:
          type: object
          properties:
            users:
              type: array
              items: { $ref: '#/components/schemas/User' }
            tweets:
              type: array
              items: { $ref: '#/components/schemas/Tweet' }
    Tweet:
      type: object
      required: [id, text]
      properties:
        id: { type: string }
        text: { type: string }
        author_id: { type: string }
        created_at: { type: string, format: date-time }
        in_reply_to_user_id: { type: string }
        referenced_tweets:
          type: array
          items:
            type: object
            properties:
              type: { type: string, enum: [replied_to, quoted, retweeted] }
              id: { type: string }
        public_metrics:
          type: object
          properties:
            like_count: { type: integer }
            retweet_count: { type: integer }
            reply_count: { type: integer }
            quote_count: { type: integer }

Wire-level: the filtered stream#

The streaming endpoint uses NDJSON over a long-lived HTTP/2 connection. Each line is a tweet that matched one of the configured rules.

GET /2/tweets/search/stream?tweet.fields=created_at,author_id HTTP/2
Authorization: Bearer eyJhbGciOi...
Accept: application/x-ndjson

HTTP/2 200 OK
Content-Type: application/x-ndjson
Transfer-Encoding: chunked

{"data":{"id":"1789012345","text":"breaking ...","author_id":"123"}}
{"data":{"id":"1789012346","text":"another ...","author_id":"456"}}
...

Rules are managed by a separate endpoint:

POST /2/tweets/search/stream/rules HTTP/2
Content-Type: application/json

{
  "add": [
    {"value": "from:nasa has:media", "tag": "nasa-with-media"},
    {"value": "(rocket OR launch) lang:en -is:retweet", "tag": "english-launches"}
  ]
}

Each rule has its own tag. When a tweet matches multiple rules, the response includes matching_rules so the consumer can route appropriately.

Client samples — three languages#

Post a tweet, then fetch the home timeline.

import os, requests

API = "https://api.twitter.example.com"
TOKEN = os.environ["TWITTER_BEARER"]
HEADERS = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}

def post_tweet(text, reply_to=None):
    body = {"text": text}
    if reply_to:
        body["reply"] = {"in_reply_to_tweet_id": reply_to}
    r = requests.post(f"{API}/2/tweets", json=body, headers=HEADERS, timeout=5)
    r.raise_for_status()
    return r.json()["data"]

def home_timeline(user_id, since_id=None, page_size=100):
    params = {"max_results": page_size, "tweet.fields": "created_at,author_id,public_metrics"}
    if since_id:
        params["since_id"] = since_id
    r = requests.get(
        f"{API}/2/users/{user_id}/timelines/reverse_chronological",
        params=params,
        headers={"Authorization": f"Bearer {TOKEN}"},
        timeout=5,
    )
    r.raise_for_status()
    return r.json()

t = post_tweet("designing the twitter api in the engineering playbook")
print("posted", t["id"])
page = home_timeline("self")
for tw in page["data"]:
    print(tw["created_at"], tw["text"][:60])

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
    "net/url"
    "os"
)

const API = "https://api.twitter.example.com"

func headers() http.Header {
    h := http.Header{}
    h.Set("Authorization", "Bearer "+os.Getenv("TWITTER_BEARER"))
    h.Set("Content-Type", "application/json")
    return h
}

func postTweet(text string) (map[string]any, error) {
    body, _ := json.Marshal(map[string]any{"text": text})
    req, _ := http.NewRequest("POST", API+"/2/tweets", bytes.NewReader(body))
    req.Header = headers()
    resp, err := http.DefaultClient.Do(req)
    if err != nil { return nil, err }
    defer resp.Body.Close()
    var out map[string]any
    json.NewDecoder(resp.Body).Decode(&out)
    return out, nil
}

func homeTimeline(userID, sinceID string) (map[string]any, error) {
    u, _ := url.Parse(fmt.Sprintf("%s/2/users/%s/timelines/reverse_chronological", API, userID))
    qs := u.Query()
    qs.Set("max_results", "100")
    qs.Set("tweet.fields", "created_at,author_id,public_metrics")
    if sinceID != "" { qs.Set("since_id", sinceID) }
    u.RawQuery = qs.Encode()
    req, _ := http.NewRequest("GET", u.String(), nil)
    req.Header = headers()
    resp, err := http.DefaultClient.Do(req)
    if err != nil { return nil, err }
    defer resp.Body.Close()
    var out map[string]any
    json.NewDecoder(resp.Body).Decode(&out)
    return out, nil
}

const API = "https://api.twitter.example.com";
const TOKEN = process.env.TWITTER_BEARER;

const headers = (json = false) => ({
  Authorization: `Bearer ${TOKEN}`,
  ...(json ? { "Content-Type": "application/json" } : {}),
});

export async function postTweet(text, replyTo) {
  const body = { text, ...(replyTo ? { reply: { in_reply_to_tweet_id: replyTo } } : {}) };
  const r = await fetch(`${API}/2/tweets`, {
    method: "POST",
    headers: headers(true),
    body: JSON.stringify(body),
  });
  if (!r.ok) throw new Error(`HTTP ${r.status}`);
  return (await r.json()).data;
}

export async function homeTimeline(userId, sinceId) {
  const params = new URLSearchParams({
    max_results: "100",
    "tweet.fields": "created_at,author_id,public_metrics",
    ...(sinceId ? { since_id: sinceId } : {}),
  });
  const r = await fetch(
    `${API}/2/users/${userId}/timelines/reverse_chronological?${params}`,
    { headers: headers() },
  );
  if (!r.ok) throw new Error(`HTTP ${r.status}`);
  return r.json();
}

Fan-out trade-off in one table#

The hybrid resolves both: push for the common case, pull for the long tail of celebrities. The Manhattan database and Heron streaming topology behind it are infrastructure; the API contract just promises a timeline.

Trade-offs and extensions#

Likely follow-up extensions and the shape of the answer:

• Long-form posts (X Premium). Same POST /2/tweets, accepts text up to 25k chars when the calling user is Premium. Schema validation diverges by entitlement.
• Spaces (audio rooms). Separate resource (Space); its own state machine; uses WebRTC for media; API exposes only metadata + recording downloads.
• Communities. Tweets scoped to a community id; same read endpoints, additional community_id filter.
• Embedded analytics. Augment public_metrics with non_public_metrics (impressions, profile clicks) gated by user-context auth.
• Webhook on mention. POST /2/account_activity registers a webhook URL; Twitter delivers tweet events that mention the registered user. Apache Kafka behind the scenes; the merchant gets a clean POST.

Mock interview follow-ups#

• “How do you paginate a timeline that’s growing in real time?” — Cursor by tweet id (Snowflake ids are time-sortable). The since_id parameter gives “new since I last saw” — append-only at the head. The until_id parameter goes the other direction. No page numbers; pagination is anchored to ids.
• “What’s the rate-limit story?” — Per-app and per-user, separately, per-endpoint. The headers x-rate-limit-limit, x-rate-limit-remaining, x-rate-limit-reset are returned on every response. Sliding window of 15 minutes for most endpoints. Streaming endpoints are limited by connection count, not request count.
• “How does the search endpoint handle planet-scale data?” — Inverted index, sharded by time (each shard is a few hours wide). Recent search hits the last few shards; full archive search walks them all. The API contract is the same; latency differs.
• “How do you handle a tweet that goes viral mid-stream — every consumer sees it 10x?” — Each tweet appears in a consumer’s stream at most once, even if it matches multiple rules. The matching_rules array tells which rules matched. Deduplication is the gateway’s job, not the consumer’s.
• “What happens to followers when an account is suspended?” — Tombstone propagation to follower timelines. The follow relationship persists in the graph (audit + appeal recovery) but the tweets disappear from timelines.
• “How do you handle a celebrity who follows back 10M people?” — Their read path is the slow case. Pull-merge across 10M outboxes is a non-starter. Solution: their “home timeline” gets a degraded experience — top-K sampled from followed accounts, computed offline every few minutes. Most celebrity accounts do not use the home timeline anyway.
• “How do you guarantee ordering in the stream?” — Per-rule, per-partition ordering only. Global ordering across all of Twitter is not promised; it’d require a single point of contention. Apache Kafka partitions by tweet id; consumers tolerate a few hundred ms reordering across partitions.

Related#

• Design a Search Service API — the search endpoint family on Twitter rests on this shape.
• Design a Pub-Sub Service API — the streaming endpoints have this shape internally (Apache Kafka partitions).
• Event-Driven Architecture Protocols — webhooks for account activity, streaming for firehose.
• WebSockets — Bidirectional Streaming — the alternative push transport; here we picked NDJSON over HTTP.
• The API-Design Walk-through — the seven-step recipe this writeup followed.