Sefism.Linux, terminal and Git: Linux

Tauseef Fayyaz

Chapter 07

Shell scripting and automation

Small, safe Bash scripts, checked by ShellCheck, run on a schedule with cron, and your own dotfiles.

Shebang and executable scripts

Also called shebang, #!, chmod +x, ~/.local/bin

What: The first line #!/usr/bin/env bash chooses the interpreter. chmod +x makes the file runnable, and ./script.sh runs it from the current folder.
Why: Scripts in a folder on your PATH, such as ~/.local/bin, become commands you can run anywhere.

Shell

#!/usr/bin/env bash
set -euo pipefail
echo "Hello from $(hostname)"

Watch out

Writing Bash features in a script with #!/bin/sh, which may be a smaller shell.

Exit codes

Also called $?, &&, ||

What: Every command returns a number: 0 means success, anything else failure. $? holds the last one; && runs the next command only on success and || only on failure.
Why: Scripts, cron, CI and other programs decide what happened from exit codes, not from printed text.

Shell

grep -q "ready" status.txt && echo "ready"
cd "$dir" || exit 1
mkdir -p out || { echo "cannot create out" >&2; exit 1; }

set -euo pipefail

Also called strict mode, set -e

What: -e stops on a failing command, -u treats unset variables as errors, and pipefail makes a pipeline fail if any part fails.
Why: Stops a script before a failed step turns into a destructive one, such as deleting in the wrong folder.

Watch out

Relying on set -e alone; it has exceptions, so still check important steps explicitly.

Variables and arguments

Also called $1, $@, $#, ${var:-default}

What: $1, $2 are positional arguments, "$@" is all of them, $# is the count, ${var:-default} gives a fallback and $(cmd) captures output.
Why: Arguments make one script reusable for many inputs.

Shell

#!/usr/bin/env bash
set -euo pipefail
if [[ $# -lt 1 ]]; then
  echo "usage: $0 <folder> [days]" >&2
  exit 1
fi
folder="$1"
days="${2:-7}"
stamp="$(date +%F)"

Watch out

Unquoted $@ or $1, which splits file names containing spaces.

Conditionals and tests

Also called if, [[ ]], case

What: if [[ ... ]]; then ... fi runs code based on a test: -f file exists, -d folder exists, -z empty string, == string match, -gt number comparison. case handles several options.
Why: Lets scripts check their inputs and environment before acting.

Shell

if [[ -f .env ]]; then
  echo "found .env"
elif [[ -z "${API_KEY:-}" ]]; then
  echo "API_KEY is not set" >&2
  exit 1
fi

Loops

Also called for, while read

What: for loops over a list or glob; while read -r line reads input line by line.
Why: Repeating an action over files or lines is the most common job a script does.

Shell

for f in ./*.csv; do
  echo "rows in $f: $(wc -l < "$f")"
done

while read -r host; do
  ping -c1 "$host" > /dev/null && echo "$host up"
done < hosts.txt

Watch out

Looping over $(ls), which breaks on spaces; loop over the glob directly.

Functions and script structure

Also called shell function, local, main

What: Named blocks of commands, called like programs, with local variables. A readable script has a usage message, argument checks, small functions and a main at the bottom.
Why: Structure makes a script easy to read, test and change months later.

Shell

log() { echo "[$(date +%T)] $*" >&2; }

backup() {
  local src="$1" dest="$2"
  log "backing up $src"
  tar -czf "$dest/$(date +%F).tgz" "$src"
}

main() { backup "$@"; }
main "$@"

Watch out

Keeping a growing script in Bash when Python would make it clearer and safer.

ShellCheck

Also called SC2086, linting shell scripts

What: A static analyser for shell scripts that warns about quoting bugs, ignored errors, wrong test syntax and more, each with a numbered explanation page.
Why: Catches the classic shell mistakes before they run on real files.

Shell

shellcheck backup.sh
shellcheck scripts/*.sh

Watch out

Silencing warnings with disable comments instead of reading why they fired.

cron

Also called crontab, scheduled jobs

What: Runs commands on a schedule defined by five fields: minute, hour, day of month, month, day of week. crontab -e edits your jobs.
Why: Backups, reports and clean-ups run themselves.

Shell

# m  h  dom mon dow  command
30 9  *   *   1    ~/bin/report.sh >> ~/logs/report.log 2>&1
0  2  *   *   *    /usr/bin/find ~/tmp -mtime +7 -delete
*/15 * *  *   *    ~/bin/check-site.sh

Watch out

Relying on your interactive PATH and aliases; cron has a minimal environment, so use full paths and log output.

Dotfiles repository

Also called dotfiles, symlink, ln -s

What: Your configuration files kept in a Git repository, with an install script that links them into your home with ln -s.
Why: A new machine or server is set up in minutes, and your configuration is versioned and documented.

Shell

#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")"
for f in bashrc gitconfig; do
  ln -sfn "$PWD/$f" "$HOME/.$f"
done
echo "linked" >&2

Watch out

Committing tokens in your shell config; source them from an untracked file instead.

Go deeper