Whatsapp Chat Messaging

Requirements

Functional
  1. Two Users can send communicate using chat messages.
  2. Group chats supported.
  3. Users can see chat history
Non Functional
  1. Minimum Latency, should be near real time messaging
  2. Reliable. Messages should not be lost

BOE

DAU:
- World population 7 billion
- 20-30% actively use 210 Million
Incoming bandwidth
- User sends(chat,image,video) approx(100kb/day)
- 210Mx100K = 210,000 GB/day = 210TB
- 5 years = 210TB x 30 x 12 x 5 = 378 PB
- Per second = 2.4GB/sec
Storage estimates:
- horizontally scalable nosql. recycled data

HLD

Simple 2 Users Design

User Authentication

User installs a WhatsApp messaging app on his phone, the app comes with a pre-configured host name.
When user enters his username and password and clicks on login, app gets the IP address from the DNS, IP resolves to the AWS API Gateway or Nginx LB.
Password Hash travels on wire once the user is authenticated, a JWT token is provided to the app, which is stored in the secure store.

User-A sending "Hi" to User-B

User-A finds User-B from phone contacts stored, clicks chat on the WhatsApp and sends "Hi"
{src=User-A, dst=User-B, Msg=Hi, timestamp=..}
HTTPS message will reach API gateway, ingestion will receive HTTP message.
Ingestion will store message on the DB.
{src=User-A, dst=User-B, Msg=Hi, timestamp=.., delivered=false}
Fan-out service checks the DB every 2 seconds Once it finds delivered=false it will get ExpoPush Token from ExpoTokens table and send notification(via ExpoNotification Service) to User-B 

sequenceDiagram
autonumber

    actor ClientA as WhatsApp Client A
    box AWS
        participant Gateway as API Gateway
        participant Ingestion@{"type":"control"} as Ingestion Service
        participant DB@{"type":"database"} as DB
        participant FanOut@{"type":"control"} as Fan-out Service
    end
    participant ExpoServer as Expo Push Server
    actor ClientB as WhatsApp Client B


    ClientB->>ExpoServer: Register for push notifications
    ExpoServer-->>ClientB: Return Expo Push Token
    ClientB->>Ingestion: POST /api/v1/expo-token (Token, User-B ID)
    Note over DB: ExpoTokens Table
    Ingestion->>DB: Store token
    
    Note over ClientA: Finds User-B from contacts
types "Hi" & sends
    ClientA->>Gateway: HTTPS POST Message payload
{src: User-A, dst: User-B, Msg: "Hi", timestamp}
    Gateway->>Ingestion: HTTP Payload
    Note over DB: Message table
    Ingestion->>DB: Insert record
{src: User-A, dst: User-B, Msg: "Hi", timestamp, delivered: false}
    Ingestion-->>Gateway: 200 OK (Message Received)
    Gateway-->>ClientA: 200 OK (Sent)

    Note over FanOut: Check DB every 2 sec
    Note over DB: Message Table
    FanOut->>DB: Select * from MessageTable where delivered = false
    DB-->>FanOut: {src: User-A, dst: User-B, Msg: "Hi", ...}
    Note over DB: Expo Token Table
    FanOut->>DB: select from ExpoToken where dst=userB
    DB-->>FanOut: ExpoPush Token
    FanOut->>ExpoServer: HTTP POST expoURL {Token, URL}
    ExpoServer-->>ClientB: Message

Scaling to 100 M Users

      flowchart LR

    A["User A Phone"]
    B["User B Phone"]

    GLB["Global / Regional 
Load Balancer"]
    API["API Gateway
Nginx"]

    subgraph K8S["Kubernetes Cluster"]

        ING["Ingress
Controller"]

        subgraph INGEST["Message Ingestion Service"]
        direction TB
            I1["Pod 1"]
            IN["Pod N"]
            I1 ~~~ IN
        end

        RP[["Kafka / Redpanda"]]

        subgraph HISTORY["Messge Storage Service"]
        direction TB
            H1["Pod 1"]
            HN["Pod N"]
            H1 ~~~ HN
        end

        subgraph FANOUT["Fanout Service"]
        direction TB
            F1["Pod 1"]
            FN["Pod N"]
            F1 ~~~ FN
        end

        MSGDB[("Message History DB
ScyllaDB / Cassandra")]

        REDIS[("Redis
Connection Registry")]

        subgraph WS["WebSocket Service"]
        direction TB
            W1["Pod 1"]
            WN["Pod N"]
            W1 ~~~ WN
        end

        subgraph NOTIFY["Notification Service"]
        direction TB
            N1["Pod 1"]
            N2["Pod N"]
            N1 ~~~ N2
        end

        DEVICE[("Device / Push Token DB")]
    end

    EXPO["Expo / FCM / APNs"]

    A -->|"HTTPS POST /message"| GLB
    GLB --> ING
    ING --> INGEST

    INGEST -->|"Topic:
MessageCreated"| RP

    RP -->|"Topic:
MessageCreated"| HISTORY
    HISTORY -->|"Store Message"| MSGDB

    RP -->|"Topic:
MessageCreated"| FANOUT

    FANOUT --> REDIS

    REDIS -->|"B connection info"| FANOUT

    FANOUT -->|"Online delivery"| WS
    WS -->|"WebSocket frame"| B

    FANOUT -->|"B offline"| NOTIFY
    NOTIFY --> DEVICE
    NOTIFY --> EXPO
    EXPO -->|"Push notification"| B

Sequence Diagram

Topics, Partitions are created at Kafka boot 

deploy.sh
#!/usr/bin/env bash
BROKER="${KAFKA_BROKER:-kafka.ib.svc.cluster.local:9092}"
create() {
  local topic=$1 parts=$2
  echo "Creating topic: $topic ($parts partitions)"
  kubectl exec -n ib deploy/kafka -- \
    rpk topic create "$topic" -p "$parts" --brokers "$BROKER" 2>/dev/null || \
}
create identity-events 24
$ ./deploy/local/create-topics-local.sh

Redpanda
Redpanda over kafka: Due to predictable latencies

ScyllaDB(NoSQL DB)
  History messages would be huge, SQL cannot scale, even messages are immutable
  Over SQL: Because noSQL(implemented using LSM Tree) can scale horizontally while SQL(implemented using BTree) cannot
  Over Cassandra: Seastar Framework is fast wrt Java JVM

Redis
  For storage of User to connection-id mapping
  User can connect to any pod(out of 20), fanout service should know the pod to which message to be sent

Websocket vs Expo
  Expo / FCM(Firebase Cloud Messaging) is used for sending notifications to device which is offline. it holds the notification for 30 days
  When device is connected, Web socket is used to send message to the mobile device.

Sequence Diagram