Text tools

cat, less, head and tail

Also called tail -f, pager

What

cat prints a file, less lets you scroll and search it, head and tail show the start and end, and tail -f follows a growing file.

Why

Picking the right viewer lets you read a huge log or watch one live without flooding the terminal.

less app.log      # /word to search, n next, q quit
head -20 data.csv
tail -50 app.log
tail -f app.log   # Ctrl+C to stop
wc -l app.log

grep

Also called ripgrep, rg

What

Prints lines that match a pattern, in files or in piped input.

Why

The fastest way to find where something is used, what went wrong in a log or which files mention a word.

grep -rn "TODO" src/
grep -i "error" app.log | tail
grep -v "DEBUG" app.log
grep -rl "apiKey" .
grep -E "404|500" access.log | wc -l

find

Also called find command

What

Walks a folder tree and matches files by name, type, size or modification time, optionally running an action on each.

Why

Finds files by their properties rather than their content, such as old logs or huge files.

find . -name "*.log"
find . -type d -name node_modules -prune
find ~ -size +500M
find . -mtime -1 -type f
find . -name "*.tmp" -delete   # run without -delete first

sort, uniq, cut, tr and wc

Also called top 10 pipeline

What

Small tools that reorder, count, slice and clean lines of text. uniq only merges adjacent duplicates, so it usually follows sort.

Why

Together they answer "what are the most common..." questions about any text in seconds.

cut -d' ' -f1 access.log | sort | uniq -c \
  | sort -rn | head -10
cut -d, -f2 users.csv | sort -u
tr 'A-Z' 'a-z' < words.txt
wc -l *.csv

Regular expressions

Also called regex, regexp

What

A pattern language: . any character, * zero or more, + one or more, ? optional, ^ and $ start and end of line, [0-9] a set, ( ) a group.

Why

grep, sed, awk, editors and programming languages all use them to match and extract text.

Example

^ERROR [0-9]{3} matches lines that begin with ERROR, a space and three digits (with grep -E).

sed

Also called stream editor, s/old/new/g

What

Edits text as it streams through, most often with substitution s/old/new/g. -i edits files in place.

Why

Quick, repeatable edits in scripts, such as changing a setting or version number in many files.

sed 's/http:/https:/g' config.txt      # preview
sed -i 's/http:/https:/g' config.txt   # in place
sed -n '10,20p' app.log

awk

What

Processes text line by line, splitting each line into fields $1, $2 and so on, with -F to choose the separator.

Why

The quick tool for columns: printing, filtering and summing fields in logs and CSV files.

awk '{print $1}' access.log
awk -F, '$3 > 100' orders.csv
awk -F, '{ total += $3 } END { print total }' orders.csv
ps aux | awk '$3 > 50 {print $2, $11}'

jq

Also called JSON processor

What

A command-line tool for reading, filtering and reshaping JSON.

Why

APIs, cloud CLIs and config files speak JSON, and jq lets you question them from the shell and in scripts.

curl -s https://api.github.com/repos/cli/cli | jq .
jq '.name' package.json
jq -r '.[] | .login' users.json
jq '.[] | select(.role == "admin") | .email' users.json
jq 'length' users.json