Text Processing: grep, sed & awk

Text Processing: grep, sed & awk

In a production environment, raw text is everywhere — application logs containing thousands of lines, nginx access logs growing at tens of thousands of requests per minute, CSV exports from monitoring systems, configuration files spanning hundreds of servers. The ability to search, extract, transform, and summarise that text from the command line — without writing a Python script, without opening a file in an editor, without waiting for a dashboard to load — is one of the highest-leverage skills a DevOps engineer can possess. Three tools do the heavy lifting: grep, sed, and awk. Each has a distinct purpose and the three compose cleanly through pipes.

grep — Searching Text at Scale

grep prints every line from its input that matches a pattern. Its name comes from the ed editor command g/re/p (globally match a regular expression and print). The default pattern language is basic regular expressions (BRE); the -E flag enables extended regular expressions (ERE), and -P enables Perl-compatible regular expressions (PCRE) on systems that support it.

Flags you will use daily in production:

• -i — case-insensitive match
• -v — invert match (print lines that do NOT match)
• -r / -R — recursive search through directories
• -l — print only file names that contain a match
• -n — prefix each output line with its line number
• -c — print a count of matching lines per file
• -A N / -B N / -C N — print N lines After / Before / around each match (context)
• -o — print only the matching portion of the line, not the whole line
• -E — extended regex (alternation |, quantifiers +, ?, grouping ())
• --color=auto — highlight matches (set this in your shell profile)

sed — Stream Editing for Transformation

sed (stream editor) reads input line by line, applies a script of editing commands, and writes to standard output. It never modifies files in place unless you pass -i. The single most-used sed command is s/pattern/replacement/flags — substitute. Understanding sed means understanding this one command deeply.

The substitution flags that matter:

• g — replace all occurrences on the line (not just the first)
• i — case-insensitive match (GNU sed)
• p — print the line after substitution (useful with -n to print only changed lines)
• 2, 3 — replace only the Nth occurrence on the line

awk — Field-Oriented Data Processing

awk is a complete programming language designed around the concept of records and fields. By default, it splits each input line (record) on whitespace into fields ($1, $2, ..., $NF for the last field, $0 for the whole line). It runs a pattern-action program against every record. The canonical form is awk '/pattern/ { action }'.

Built-in variables that appear constantly in real scripts:

• NR — current line (record) number across all files
• NF — number of fields in the current record
• FS — field separator (default: whitespace; set with -F)
• OFS — output field separator (default: space)
• $0 — the entire current line
• $1 ... $NF — individual fields
• BEGIN { } — runs once before any input is read
• END { } — runs once after all input is consumed

Composing the Three Tools: Real-World Pipelines

The real power of these tools emerges when you pipe them together. Each tool in the chain is a specialist that does one thing extremely well. You read the chain left-to-right as a data-processing narrative.

Performance Considerations at Scale

When processing log files that are gigabytes in size — common in production — tool choice and order have real performance implications:

• Put grep first in the pipeline. It rejects lines early, reducing the data that sed and awk have to process. Filtering 100 million lines down to 10,000 before handing them to awk is orders of magnitude faster than feeding all 100 million to awk.
• Use grep -F for fixed strings. Regex matching has overhead; plain string matching (Boyer-Moore algorithm) is significantly faster when you do not need regex.
• Prefer awk over a shell loop for column arithmetic. A shell loop that processes one line per iteration forks a subprocess per line; awk processes millions of lines in a single process invocation.
• For very large files, consider LC_ALL=C. Prefixing with LC_ALL=C disables multi-byte character handling and can make grep and awk 3-5x faster on ASCII-dominant logs.

Common Failure Modes

• grep returns exit code 1 (no match) and halts a pipeline with set -e. In scripts that use set -e (covered in the next lesson), a grep that finds nothing exits with code 1, which aborts the script. Guard against this with grep ... || true when zero matches is an acceptable outcome.
• sed in-place on a symlink follows the link, not the link itself. When your config file is a symlink (common in Ansible-managed environments), sed -i replaces the symlink target. Verify the result with ls -la after editing.
• awk field numbering is 1-based, not 0-based. $0 is the whole line; $1 is the first field. Engineers from Python or Go backgrounds assign the wrong field number and silently get garbage output — always verify with a small sample first.
• Locale-sensitive regex in awk. Character classes like [[:alpha:]] match differently depending on the locale. Set LC_ALL=C for predictable, ASCII-only matching in production scripts.

With grep, sed, and awk in your toolkit — and an understanding of how to compose them — you can answer in seconds questions that would otherwise require writing a Python script, loading data into a database, or waiting for a BI dashboard to refresh. At scale, that speed is the difference between a 10-minute incident and a 10-second diagnosis.