Design an error message counting service
PremiumData pipelineMedium
- Error fingerprinting
- Log template mining
- Pre aggregation
Quick answer
This data pipeline interview question asks you to design an error message counting service. Aim for a design that moves data from sources to consumers reliably and on time, then explain ingestion, processing, storage and trade-offs. Interviewers want to see how you handle volume, late or bad data and failures along the way.
The problem
Design an error message counting service that tells engineers how often each distinct error happens, like the error grouping in error-tracking tools. The same error appears with different request IDs, user IDs and timestamps, so raw message strings cannot be counted directly. The hard part is grouping those variants into one error and keeping counts exact during incident bursts.
For example, Timeout calling inventory for order 7c9e6679-7425-40de-944b-e07fc1f90ae7 after 3021 ms and the same line with another order and duration must count as one error.
What must an error message counting service do?
- Group errors: Turn each ERROR line into a fingerprint, so messages that differ only by IDs, numbers, timestamps, emails, IPs or hex strings count as one error per service.
- Exact counts: Count every error line per fingerprint per minute, and return totals for any range from 1 minute to 30 days.
How well must it do it?
- Burst absorption: Handle 50 times the normal error volume for 30 minutes without dropping counts or slowing the applications.
- Accuracy: Retries never double count; at most the last 10 seconds of a crashed pod's counts may be lost.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Error fingerprinting
- Log template mining
- Pre aggregation
- Kafka
- Aho corasick
- Exactly once counting
How you’ll practise
- Requirements
- Data volume
- Ingestion
- Storage and processing
- Serving and freshness
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any data pipeline design round.
- Clarifies sources, consumers and freshness needs
- Chooses batch, streaming or both with reasons
- Handles late, duplicate and malformed records
- Makes each stage safe to retry
- Explains storage formats and partitioning
- Plans monitoring, backfills and data quality checks
Questions people ask about this interview question
- Which design type is this question?
- It is a data pipeline question. You design an error message counting service and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Medium. Entry-level engineers can practise it, but it expects you to reason about requirements and trade-offs, not only list components.
- What should I cover first?
- Start with the data sources, the consumers and how fresh the data must be. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team