Data Analytics Pipeline

This accepts data from many heterogeneous sources, stores the original data, cleans and standardizes it, transforms it into business-ready datasets, and exposes those datasets to analytics, BI, data warehouse, and ML systems. What data can come?

csv
files
json
pdf
Application logs

Requirements

Functional requirements
1. Multiple ingestion sources
2. Preserve original data
3. Data validation: Detect missing fields, invalid types, duplicate records, malformed records, invalid timestamps, invalid business values
4. Analytics: Users should be able to ask: How many sales in city X
Non-functional requirements
1. Scalability: upto 10 TB/day
2. Availability: A failure in one pipeline should not destroy data already ingested
3. Security

HLD