Log management Service

Design a service that collects Logs from network devices(firewalls, routers, switches) and supports full text search over ingested log corpus.

Requirements

Functional

1. Inject (POST): Accep a batch of log events(JSON) from devices, return an injestion acknowledgment.
2. Search: Accept time range, filter parameters and full text query, return matching log entries with parameters
3. Alerts (Management): Register or update an alert rule(eg: regex match or threshold over time window); return alert
4. GET alerts/[alert id]/status: Return current status of the alert(active/inactive), recent firing timestamp

Non Functional

High Throughput: System should injest logs from tens of thousands of devices, sending millions of events per second

HLD

A. Injestion Layer

A1. Ingestion Service (The "Validator & Router")

Receive a raw string, turn it into a structured object(JSON)+add meta data unique uuid
Ingestion Service writes to Kafka(not the database) This is because no database can handle the raw, unbuffered "firehose" of a million writes per second without a queue in front of it.
Functions of Injestion Service:
1. Parse: It breaks the string into a JSON object
2. Enrich: It adds metadata (like the IP address of the device or a unique log_id)
3. Route: It pushes the log to a Kafka topic.

Input= "tenant-A 1707312000 ERROR auth-mod Login failed for user 123"
Output:
{
  "tenant": "tenant-A",
  "timestamp": "2026-02-07T12:00:00Z",
  "level": "ERROR",
  "module": "auth-mod",
  "message": "Login failed for user 123",
  "ingest_id": "uuid-987"
}
            

B. Processing & Storage Layer

B1. Indexer Service (The "Search Engine Writer")

It consumes from Kafka topic and prepares data for OpenSearch(SSD-based).
- Since SSD is expension, Index all logs for the last 7 days.
- OR, index only Critical/Error/Warn for 30 days and ignore "Debug" entirely.

B2. Storer Service (The "Archivist")

Stores everything. It writes the raw JSON to S3 (Cold Store) in compressed formats like Parquet.
Why store in cold Store/S3? If OpenSearch crashes or you need to perform a "Forensic Audit" from a year ago, you go to S3. It is 1/10th the cost of OpenSearch.
It waits until it has a large chunk of data (e.g., 100MB or 5 minutes of logs) and uploads a single file to: s3://logs/tenant-A/2026/02/07/logs_1200.parquet.

C. Alerting Layer

Alerting usually happens in one of two ways: Real-time or Scheduled.

C1. Stream Processing Service(Flink/Spark Streaming)

For "Real-time" alerts (e.g., a regex match), a service consumes directly from Kafka, evaluates the rule, and triggers an alert immediately.

C2. Query-Based Alerting Service

For "Threshold" alerts (e.g., "more than 500 errors in 5 minutes"), a service periodically runs a count query against the Search Engine.

C3. Alert State Store

A fast Key-Value store (like Redis or DynamoDB) to track the [alert_id]/status (active/inactive) and firing timestamps.