What is Box
Box.com is a cloud file storage and collaboration platform. Users upload, share, and manage files through a web or mobile client.
Box is AP from CAP
Strong Consistency is hard to achieve.
User-A in US writes and within 1 sec, User B in India find same data.
Availablity: Required. users can continue uploading, downloading,
and editing files even if all replicas are not immediately identical
Requirements
Functional:
- User can upload/download file of any size
- Users can modify existing files (partial updates, overwrite,
append)
- Users can share files and folders with other users with
permissions.
Non Functional:
- Highly Available 99.99% (System is there for RW)
- Eventual Consistent
- Latency low
- Scalability: Support billions of files and petabytes of storage.
HLD
Block Diagram
%%{init: {'themeVariables': {'fontSize': '13px'}, 'sequence': {'actorMargin': 45, 'messageMargin': 30, 'boxMargin': 8, 'useMaxWidth': true}}}%%
flowchart LR
Client["Web / Mobile Client"]
subgraph Edge
API["API Gateway"]
end
subgraph Control_Plane
APP["Application Service"]
META["Metadata Service"]
AUTH["Auth Service"]
NOTIFY["Notification Service"]
CACHE["Redis Cache"]
WORKER["Async Workers"]
end
subgraph Data
DB[(Metadata DB)]
S3[(Object Storage)]
MQ["Kafka / SQS"]
end
Client --> API
API --> AUTH
API --> APP
APP --> META
META --> CACHE
CACHE --> DB
APP --> S3
S3 --> MQ
MQ --> WORKER
WORKER --> META
WORKER --> NOTIFY
META --> DB
Flow Diagram(10 GB Upload)
Design-1. upload through the Application Server
Design-2: Using Pre-signed URL (Direct Upload to Object Storage)
Instead of streaming file bytes through the Application Server, Box.com
uses a pre-signed URL so the client uploads directly to object
storage (e.g. Amazon S3). The Application Server only issues a
time-limited, signed permission to upload to a specific path.
Reference:
AWS S3 — Sharing objects with presigned URLs. The same pattern applies to uploads: the server signs a
PUT URL; the client uses it without holding the app server
connection open.
# Before (slow)
Browser/Client ──10 GB──► App Server ──10 GB──► S3(Object Store)
# After (pre-signed URL)
Browser/Client ──POST /upload (metadata only)──► App Server
App Server ──returns pre-signed PUT URL──► Browser/Client
Browser/Client ──10 GB PUT (direct)──────────────► S3
Browser/Client ──POST /upload/complete───────────► App Server (update metadata)
How do Pre-signed URLs work? (Box.com upload)
A pre-signed URL embeds temporary security credentials in the query
string (access key, signature, expiry). Object storage validates the
signature and expiry before accepting the upload — no separate auth call
needed during transfer.
# Example pre-signed PUT URL (conceptual)
https://box-uploads.s3.amazonaws.com/acme-corp/user42/report.zip
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=...
&X-Amz-Date=20260412T120000Z
&X-Amz-Expires=3600
&X-Amz-SignedHeaders=host
&X-Amz-Signature=abc123...
# Client uploads directly:
PUT https://box-uploads.s3.amazonaws.com/acme-corp/user42/report.zip?...signature...
Content-Type: application/zip
Body: <10 GB file bytes>
How metadata is stored and used by the Application Server? The Application Server owns the metadata database. It never stores the 10 GB blob — only information about the file:
# Metadata record (stored in DB by Application Server)
{
"file_id": "f_8a3b2c1d", <<<<<<<<<<<
"user_id": "user_42",
"tenant_id": "acme-corp",
"folder_id": "folder_reports",
"filename": "Q4-report.zip", <<<<<<<<<<<
"size_bytes": 10737418240,
"content_type": "application/zip",
"storage_key": "s3://box-uploads/acme-corp/user42/f_8a3b2c1d",
"status": "pending | complete | failed",
"etag": "d41d8cd98f00b204e9800998ecf8427e",
"created_at": "2026-04-12T12:00:00Z",
"uploaded_at": "2026-04-12T12:45:00Z",
"permissions": ["owner: user_42", "shared_with: team_eng"]
}
Sequence Diagram: 10 GB Upload (Q4-report.zip)
Example: Bob (user_42) uploads
Q4-report.zip (10 GB) to folder
folder_reports.
Application Server generates file_id = f_8a3b2c1d and a
pre-signed PUT URL. Client uploads directly to S3 — app server handles
only metadata (~KB).
Step-by-step (Box.com file upload):
1. Client sends upload request to Box Application Server:
filename, size (10 GB), folder ID, content-type. Actual file is not
uploaded to App Server(only Meta data is sent)
2. Application Server authenticates the user, checks quota and
folder permissions, generates a unique file_id and storage
key (e.g. s3://bucket/tenant/user/file_id).
3. Application Server calls object storage SDK to generate a
pre-signed PUT URL (valid for e.g. 60 minutes).
4. Application Server writes a pending metadata record to
the database and returns the pre-signed URL + file_id to
the client.
5. Client uploads 10 GB directly to S3 using the
pre-signed URL (optionally multipart). App server is not involved.
6. Client (or S3 event) notifies Application Server:
POST /upload/complete { file_id, etag, size }.
7. Application Server verifies the object exists in S3, updates
metadata status to complete, and the file appears in the
user's Box folder.
%%{init: {'themeVariables': {'fontSize': '13px'}, 'sequence': {'actorMargin': 45, 'messageMargin': 30, 'boxMargin': 8, 'useMaxWidth': true}}}%%
sequenceDiagram
autonumber
participant CDN as CDN
actor Bob
participant Browser as Browser
participant App as Application Server
participant DB as Metadata DB
participant S3 as Object Storage (S3)
Bob->>Browser: Open https://box.com
Browser->>CDN: https://box.com
CDN->>Browser: box.js, images, HTML
Note over Browser: js mentions to Send Metadata to AppServer
Then upload to pre-signed url
Bob->>Browser: Select report.zip (10 GB)
Note over Browser: javascript calculates Metadata
{filename=report, size=10GB, filetype=zip, folder}
See box.js below
Note over Browser, App: File Need not travel on Network to App Server
Browser->>App: POST /upload
filename=Q4-report
MetaData={filename=report, size=10GB, filetype=zip, folder}
user_id=user_42
App->>App: Authenticate user_42
Check quota & folder_reports permission
App->>App: Generate file_id=f_8a3b2c1d
storage_key=acme-corp/user42/f_8a3b2c1d
App->>DB: INSERT metadata
file_id=f_8a3b2c1d
filename=Q4-report.zip
size_bytes=10737418240
status=pending
App->>S3: generate_presigned_url(PUT)
bucket=box-uploads
key=acme-corp/user42/f_8a3b2c1d
expires=3600s
S3-->>App: presigned PUT URL
App-->>Browser: 200 OK
file_id=f_8a3b2c1d
presigned_url=https://box-uploads.s3.amazonaws.com
/acme-corp/user42/f_8a3b2c1d
?X-Amz-Signature=abc123...
Note over Browser,S3: Browser directly uploads 10 GB file to S3(Object Store)
Browser->>S3: PUT Q4-report.zip (10 GB)
multipart upload
using presigned URL
S3-->>Browser: 200 OK
etag=9f86d081884c7d659a2feaa0c55ad015
Note over Browser: Browser updates the Metadata
to App server
Browser->>App: POST /upload/complete
file_id=f_8a3b2c1d
etag=9f86d081...
size=10737418240
Note over App: App server checks
Is file present?
App->>S3: HEAD acme-corp/user42/f_8a3b2c1d
verify object exists
S3-->>App: 200 OK size=10737418240
Note over App: App server updates
metadata in DB
App->>DB: UPDATE metadata
status=complete
etag=9f86d081...
uploaded_at=2026-04-12T12:45:00Z
App-->>Browser: 200 OK
Q4-report.zip ready in folder_reports
Browser-->>Bob: Upload complete — file visible in Box
// box.js
const response = await fetch("/upload", {
method: "POST",
headers: {
"Content-Type": "application/json"
},
body: JSON.stringify({
filename: file.name,
size: file.size,
contentType: file.type
})
});
// Server returns pre-signed URL
const { uploadUrl } = await response.json();
// Step 2: Upload file directly to S3
await fetch(uploadUrl, {
method: "PUT",
body: file,
headers: {
"Content-Type": file.type
}
});
Design-1: 10 GB upload through the Application Server
Problems
In an early design, when a user uploaded a 10 GB file to Box.com,
the entire file passed through the Application Server before
reaching object storage. That has these issues:
1. Memory pressure, OOMs — buffering 10 GB stream in RAM per
upload exhausts server memory; many concurrent uploads can cause OOM
(out-of-memory) crashes.
2. Thread / worker exhaustion — Each large upload occupies a
worker for the full upload duration. New requests wait or time out.
3. Network saturation/Bandwidth bottleneck — App server NIC
becomes the choke point; egress/ingress limits are hit before object
storage limits are.
4. Single point of failure — If the app server crashes
mid-upload, the entire 10 GB transfer must restart from zero.
5. Scaling cost: To handle file bytes you must scale out fat app
servers, which is expensive
6. Timeout errors — Load balancers, proxies, and HTTP gateways
often have 60–120 second idle timeouts — far shorter than a multi-GB
upload requires.
Client ──(10 GB file bytes)──► Application Server ──► Object Storage (S3)
Solution: Pre-signed URL (Direct Upload to Object Storage)
1. App server handles a small JSON request (~KB), not 10 GB of bytes.
2. Upload bandwidth goes directly from client → object storage
(optimized for large transfers, multipart upload, CDN edge).
3. App server workers are freed in milliseconds; they can serve
thousands of other users while the 10 GB upload runs in the
background
4. Object storage supports multipart upload — the client splits
10 GB into chunks, retries failed parts, and resumes without restarting
the whole file
5. Pre-signed URLs expire (e.g. 1 hour), limiting exposure if the URL is
leaked.
Delta Update
Use case: File is uploaded using presigned URL, Owner adds a text="test" on line number 10, page number 10. How this delta update flows from the Browser to the box server and how this gets replicated to all the read replicas.
Offset Calcuation
Offests is bytes from the beginning of the file, not the word or line
number
If we want to modify line 10 the editor scans the file until it reaches
the 10th newline (\n) and computes something like: Line 10 starts at
byte 7432
hello.txt //10 KB
Byte Offset
0 H
1 e
2 l
3 l
4 o
5 \n
6 A
7 B
8 C
PATCH /file //Send
{
offset:7432,
length:15,
data:"New sentence"
}
What Application Server should do
The Application Server is a control plane, not a data plane for
large files:
1. Authentication & authorization — verify user identity, folder
access, sharing rules.
2. Issue pre-signed URLs — generate time-limited PUT/GET URLs for
upload and download.
3. CRUD metadata — create, read, update, delete file and folder
records in the database.
4. Sharing & collaboration — manage share links, team folders,
access control lists.
5. Search & indexing — query and index file metadata for fast
folder listing and search.
6. Quota & billing — track storage used per user/tenant; enforce
limits before upload.
7. Upload lifecycle — track pending → complete → failed states;
handle completion callbacks.
8. Audit & compliance — log who uploaded, downloaded, or shared
which file and when.
9. Notifications — notify collaborators when a file upload
completes or is shared.