Files and text tools

Linux tools treat text as lines. Small programs that each do one thing well can be chained to answer most questions about logs and data.

Viewing files

Also called cat, less, head, tail -f

What: cat prints a file, less pages through it (search with /, quit with q), head and tail show the first or last lines, and tail -f follows a file as it grows.
Why: Reading configs and following logs live is most of what you do on a server.

Shell

less /var/log/syslog
head -n 20 data.csv
tail -n 100 app.log
tail -f /var/log/nginx/access.log

Watch out

cat on a huge log file floods the terminal; use less or tail instead.

find

Also called locate, fd

What: find walks a directory tree and matches files by name, type, size, age or owner, then prints them or runs an action on each.
Why: It answers "where is that config?", "what filled the disk?" and "which files changed today?".

Shell

find . -name '*.log'
find /var -type f -size +100M
find . -type f -mtime -1
find /tmp -type f -mtime +7 -delete
find . -name '*.js' -not -path './node_modules/*'

Watch out

Leaving the pattern unquoted, so the shell expands *.log in the current directory before find ever sees it.

grep

Also called ripgrep, rg, search text

What: grep prints lines matching a pattern. Useful flags: -i ignore case, -r recurse, -n line numbers, -v invert, -c count, -E extended regex, -o only the match, -A/-B/-C lines of context.
Why: It is the fastest way to find an error in a log or a setting in a codebase.

Shell

grep -rn 'DATABASE_URL' --include='*.ts' src/
grep -i 'error' app.log | grep -v 'healthcheck'
grep -c ' 500 ' access.log
grep -E 'timeout|refused' -C 2 app.log

sed

Also called stream editor, find and replace

What: sed edits text as it streams past, most often substituting with s/old/new/g. -i edits files in place; -n with p prints only selected lines.
Why: Quick replacements across files and config tweaks in scripts, without opening an editor.

Shell

sed 's/http:/https:/g' links.txt
sed -i 's/^PORT=.*/PORT=8080/' .env
sed -n '10,20p' big.log
sed '/^#/d' config.ini

Watch out

sed -i differs on macOS, which needs sed -i ''. Scripts that must run on both break on this.

awk

Also called columns, fields

What: awk splits each line into fields ($1, $2, $NF for the last) and runs a small program per line, with patterns, variables and sums.
Why: It pulls columns out of logs and command output and totals them in one line.

Shell

awk '{print $1}' access.log
awk -F, '{print $2, $5}' data.csv
awk '$9 >= 500 {print $7}' access.log
awk '{sum += $10} END {print sum / NR}' access.log

sort, uniq, cut and wc

Also called counting, top N

What: sort orders lines (-n numeric, -r reverse, -k by column), uniq collapses adjacent duplicates (-c counts them), cut extracts fields by delimiter, and wc -l counts lines.
Why: Chained together they answer "top 10 IPs", "how many distinct users" and similar questions instantly.

Shell

cut -d' ' -f1 access.log | sort | uniq -c \
  | sort -rn | head -10
cut -d: -f1 /etc/passwd | sort
wc -l *.csv

Watch out

uniq without sort first: it only removes duplicates that sit next to each other.

xargs

Also called batch arguments

What: xargs reads items from standard input and passes them as arguments to a command, in batches. -0 pairs with find -print0 to handle names with spaces; -P runs in parallel.
Why: Many commands take arguments, not input; xargs connects a list from one command to another that needs arguments.

Shell

find . -name '*.tmp' -print0 | xargs -0 rm -f
grep -rl 'oldName' src/ | xargs sed -i 's/oldName/newName/g'
cat urls.txt | xargs -n 1 -P 4 curl -sO

Watch out

Piping plain find output into xargs: filenames with spaces split into several arguments. Use -print0 and -0.

diff and comparing files

Also called diff -u, cmp, sha256sum

What: diff -u shows line differences between two files in unified format. sha256sum prints a checksum to confirm two files, or a download, are identical.
Why: Comparing a working config with a broken one, or checking a download is intact, is a quick way to find what changed.

Shell

diff -u nginx.conf.bak nginx.conf
sha256sum release.tar.gz

Go deeper