Command-Line Log Analyser with a Scheduled Report

The problem small teams have with their logs

Every web server and app writes logs, and in most small teams nobody reads them until something breaks. A startup with one server, a freelancer hosting a few client sites, or a university lab running a web service rarely has a logging platform. When a customer says the site was slow last night, someone SSHs in and types half-remembered grep one-liners.

Reading logs quickly is one of the most practical skills an engineer can have, and the classic Unix tools are still the fastest way to do it on a single machine. The pain is that the one-liners live in people's shell history, break on the next log format, and never run unless someone remembers to run them.

What already exists and the gap

• GoAccess is an open-source real-time log analyser that runs in a terminal or browser and understands Nginx, Apache, Caddy, CloudFront and ELB formats.
• lnav is a terminal log viewer that detects formats, merges files and lets you query logs with SQL.
• Grafana Loki and hosted logging services collect logs centrally and let you search and alert on them.

GoAccess focuses on web traffic and does not read application JSON logs or the auth log. lnav is excellent for interactive investigation but is not a scheduled report. Central logging is the right answer at scale, but it is another system to run and pay for. The gap is a small, tested, scriptable tool that answers the same five questions every morning across several log types, and tells you only when something needs attention.

What you build

One command, for example logscan, with subcommands:

• top-errors: most frequent status codes and error messages.
• slow-requests: slowest endpoints by p95 response time, if the log records it.
• busiest-hours: requests per hour with a small text histogram.
• top-ips: most active clients, with a flag for suspicious patterns such as many 404s.
• failed-ssh: failed login attempts by user and source from the auth log.
• report: all of the above for a time window, written as Markdown.

A cron job or systemd timer runs report daily and an alert mode hourly.

The technically interesting part

Three things make this more than a script collection.

First, correct parsing of messy input. Real logs have quoted fields with spaces, user agents with quotes inside them, truncated lines and mixed time zones. Decide how each subcommand treats a malformed line (skip and count it, never crash) and prove it with tests.

Second, time windows across rotated files. "The last 24 hours" usually spans access.log, access.log.1 and a gzipped access.log.2.gz. Reading only the files you need, in order, without decompressing everything, is a small but real engineering problem.

Third, knowing when Bash stops being the right tool. Measure each subcommand on a large file. Where awk is fast, keep it; where logic becomes unreadable or slow, move that part to Python and explain the choice with numbers.

How to evaluate it

• Correctness: at least 30 Bats test cases on sample logs with known answers, including malformed lines, empty files and missing files, all passing in CI.
• Performance: a timing table for 100 MB and 1 GB generated logs per subcommand, compared with an equivalent GoAccess or lnav query where one exists.
• Quality: ShellCheck clean, consistent exit codes, and --help for every subcommand.
• Usefulness: run it on a real server (yours or a friend's) for two weeks and report one thing it found that nobody had noticed.

Suggested plan

1. Week 1: collect or generate sample logs, define the subcommands and expected outputs.
2. Weeks 2 to 3: access log and JSON parsers, first four subcommands, tests.
3. Week 4: auth log, time windows across rotated files, report and alert modes.
4. Week 5: scheduling, timings on large files, the Python rewrite where justified.
5. Week 6: two-week real-server run, README and demo.

Path to a product

This is an open-source tool first. Publish it on GitHub with a clear README, a licence and an install script, and ask two or three developers who host their own servers to try it. Adoption, issues from strangers and a good write-up are the realistic outcome, and they make a strong portfolio for DevOps and SRE internships. If freelancers find it useful, a small paid add-on (hosted report delivery or a dashboard across several servers) is possible later, but do not plan around it.

Scope warning

Do not build a log collection platform, a database or a web UI. Do not try to support every log format; support three well. Use generated or public sample logs, and mask IP addresses and anything personal in screenshots, because logs often contain personal data.

Learn what you need on Sefism

• DevOps and cloud roadmap : where shell scripting and log analysis sit in the wider path.
• Greg's Wiki: BashGuide : write Bash that handles quoting, errors and arguments correctly.
• jq Manual : the reference for parsing JSON application logs.
• ShellCheck : catches the bugs that make scripts fail on odd input.
• roadmap.sh: Shell / Bash : a checklist of the shell skills this project uses.

Get help with this idea

This is a good first DevOps project if you keep it small and test it seriously. Tauseef can help you choose the subcommands, decide where Bash ends and Python begins, and plan an evaluation that shows real engineering. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session