Log Volume Cost Reducer With Pattern Mining for Self-Hosted Loki and Vector Pipelines
Build a tool that groups a company's logs into templates using log parsing algorithms, finds the noisy patterns nobody ever searches, and generates sampling and drop rules for the log pipeline. Teams running Grafana Loki or sending logs to a paid service often find that a handful of chatty patterns make up most of the volume. The tool shows the savings before anything changes and keeps rare and error logs intact.
Intermediate
A small team, or one strong student willing to learn something new.1 to 2 semesters · 2 students
7 to ship
4 optional extrasSuggested stack
What you should ship
- A streaming pattern miner that groups log lines into templates per service with Drain
- Volume statistics per template: lines, bytes and trend over time
- Query usage analysis from Loki query logs: which templates people actually search for
- A recommender that proposes drop, sample or keep per template with the expected saving
- Generated pipeline rules for Vector (and optionally Fluent Bit) with a dry-run mode
- Safety rules that never drop errors, rare patterns or logs needed for audit
- An evaluation on public log datasets and a demo microservice system under load
If you have time left
- Turning repetitive log lines into metrics instead of dropping them
- Alerts when a new chatty pattern appears after a deployment
- Support for OpenTelemetry logs
- Per-team cost reports
The problem: paying to store logs nobody reads
Logs are often the largest observability cost. Services print the same health check, cache hit or "request completed" line millions of times a day. Teams pay to ship, index and store it, whether on a hosted platform priced by volume or on their own Loki cluster's disks and object storage. When someone asks which logs can go, nobody knows, because the logs are free text and nobody wants to be the person who deleted the line needed in the next incident.
Startups in Pakistan serving international customers feel this sharply, since observability bills are in dollars. Smaller teams self-host to save money and then run out of disk.
What already exists and the gap
- Grafana Cloud Adaptive Logs groups incoming logs into patterns using the Drain algorithm, checks which patterns are queried, and recommends drop rates for low-value ones. It is a paid cloud feature.
- Hosted log platforms offer pattern views and exclusion filters within their products.
- Vector and Fluent Bit can filter and sample logs, but someone must write the rules.
- Drain3 is an open source streaming implementation of Drain.
The gap is an open, self-hostable version of the idea for teams running their own pipelines: mine patterns, join them with query usage, recommend rules, generate the pipeline config and prove the savings in a dry run, with strict safety rules.
Inspiration
"Drain: An Online Log Parsing Approach with Fixed Depth Tree" by Pinjia He, Jieming Zhu, Zibin Zheng and Michael R. Lyu (ICWS 2017) parses logs into templates in a streaming way using a fixed depth tree, and showed the highest accuracy on four of five large real-world datasets. Grafana's Adaptive Logs, which uses Drain to find low-value patterns, shows the approach works commercially. This project builds an open version and studies its safety.
What you build
- Platform engineers see a ranked list: "this template is 31% of all log bytes and was searched twice in 30 days".
- Service owners approve or reject recommendations for their service.
- The pipeline applies approved rules, and a dry-run report shows what would have been dropped.
The technically interesting part
- Pattern mining at volume. Run Drain in streaming mode per service, tune similarity thresholds, and handle variables such as IDs, timings and URLs without exploding the number of templates.
- Joining with usage. Loki query logs record LogQL queries, not templates. Work out which templates each query would have matched, so you know which patterns people actually use.
- Safe recommendations. Never drop error levels, rare templates, or anything matched by alerts. Prefer sampling (keep 1 in 100) over dropping, and keep every first occurrence after a deployment.
- Generating config. Turn approved recommendations into Vector transforms that match templates efficiently, and test that the generated rules match exactly what the miner intended.
- Drift over time. Services change their log lines with each release, so templates appear and vanish. Track template lifetimes, retire rules for templates that no longer occur, and flag new chatty templates that arrive with a deployment.
- Explaining value. For each template show sample lines, volume, cost estimate in bytes and the queries that used it, so an owner can decide in seconds.
How to evaluate it
- Parsing accuracy: on public labelled log datasets such as those in the LogPAI benchmarks.
- Savings: bytes stored before and after on a demo microservice system under realistic load.
- Safety: inject rare error events and incident-like sequences, and check none are lost.
- Overhead: CPU and latency added to the pipeline.
Suggested plan
- Weeks 1 to 3: run a demo microservice system with Vector and Loki; read the Drain paper and try Drain3.
- Weeks 4 to 7: streaming pattern mining and volume statistics.
- Weeks 8 to 11: query usage join and the recommender with safety rules.
- Weeks 12 to 15: generated rules, dry run, evaluation and report.
Who it is for, concretely
A ten-person startup running Kubernetes with Loki on object storage, or an agency hosting thirty client sites with logs shipped to one server. Both have more log volume than they can afford to keep and no time to read every pattern.
Path to a product
Log cost is a constant complaint, and a vendor charging for the same feature shows demand. An open source tool with a paid hosted dashboard, or consulting for teams cutting observability bills, are realistic paths.
Scope warning
Use your own demo system and public datasets, not real company logs, unless you have written permission and the logs are cleaned of personal data. Support one pipeline tool first. Recommendations need human approval; never drop logs automatically.
Learn what you need on Sefism
- Platform engineering and SRE roadmap: observability and its costs.
- Platform engineering and SRE cheat sheet: logs, metrics and traces in one place.
- Grafana Documentation: Loki, LogQL and dashboards.
- Awesome Site Reliability Engineering: further reading on observability practice.
- How do I move from DevOps to site reliability engineering?: where this kind of project leads.
Get help with this idea
The hard part is joining query usage to templates and keeping drops safe. Tauseef can help you set up the demo system, design the safety rules and measure savings credibly. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.