Format Command Output In Linux

The short answer

In Unix-like operating systems such as Linux and macOS, you can scan patterns and process the output of commands using the awk command as follows:

$ <command> | awk '/<pattern>/ {<action>}'

Run in Warp

Where:

• <command> is the command whose output you want to process or filter.
• <pattern> is a text string or a regex pattern used to filter the output of the specified command.
• <action> is the action to perform on the lines matching the specified pattern.

Alternatively, you can also pass one or more file paths as arguments:

$ awk '/<pattern>/ {<action>}' <file ...>

Run in Warp

For example, the following command will print to the standard output all the lines of the server.log file containing the error string:

$ cat server.log | awk '/error/ {print}'

Run in Warp

And the following command will print to the standard output all the lines of the server.log file starting with the 2024-05 string:

$ awk '/^2024-05/ {print}' server.log

Run in Warp

Splitting the output into columns

By default, the awk command splits every line of its input into multiple columns, using the space character as a delimiter.

To print one or more specific columns instead of the entire line, you can use the following syntax:

$ <command> | awk '{print $<column>[,$<column>]}'

Run in Warp

Where:

• $<column>[,$<column>] is a list of comma-separated column numbers prefixed with a dollar sign ($).

For example, this command will only print the first column of the output generated by the ps command:

$ ps | awk '{print $1}'
PID
46686
46697
19849
5274

Run in Warp

And this command will print the first and fourth columns of the output generated by the ps command:

$ ps | awk '{print $1,$4}'
PID CMD
46686 npm
46697 node
19849 awk
5274 bash

Run in Warp

Note that you can quickly reference the last column using the pre-defined $NF variable as follows:

$ <command> | awk '{print $NF}'

Run in Warp

Using custom delimiters

To split the output of commands into columns based a specific delimiter instead of the default space character, you can use the -F flag as follows:

$ <command> | awk -F '<delimiter>' '{<action>}'

Run in Warp

Alternatively, you can specify multiple delimiters at once by encapsulating them in square brackets as follows:

$ <command> | awk -F '[<delimiter>...]' '{<action>}'

Run in Warp

For example, considering the following data.csv file:

Name,Age,City
Sarah,32,New York
John,28,Chicago
Emily,35,Seattle

Run in Warp

The following command will use the , character to split each line of the data.csv file into separate columns and only output the third column:

$ cat data.csv | awk -F ',' '{print $3}'
City
New York
Chicago
Seattle

Run in Warp

Summing up and counting columns

Counting the occurrences of a pattern

To count the occurrences of a specified pattern with awk, you can increment a variable using the ++ operator as follows:

$ awk '/pattern/ { <variable>++ } END { print <variable> }' <file ...>

Run in Warp

For example, considering the following server.log file:

ERROR | User authentication failed
INFO | Backup completed successfully
ERROR | User authentication failed
WARNING | High memory usage
INFO | Data export completed

Run in Warp

This command will increment the count variable every time the "ERROR" string is found in the server.log file:

$ awk '/ERROR/ { count++ } END { print count }' server.log
2

Run in Warp

Summing up the data of columns

To add up the values of a column and print their sum with awk, you can use the following syntax:

$ <command> | awk '{<variable> += $<column>} END {print <variable>}'

Run in Warp

Where:

• <variable> is the name of the variable the sum of the specified column will be accumulated in.
• <column> is the column number to add up.
• END is used to specify a command to execute once all the lines are processed.

For example, considering the following data.csv file:

bash
Name,Salary,Position
John,33000,Developer
Jack,36000,Manager

Run in Warp

This command will print the sum of the values of the second column:

bash
$ cat data.csv | tail -n +2 | awk -F ',' '{sum += $2} END {print sum}'
69000

Run in Warp

Where:

• cat data.csv is used to output the file's content.
• tail -n +2 is used to remove the first line of the output.
• awk -F ',' is used to split each line of the output into columns using , as a separator.
• {sum += $2} is used to add the value of the second column to the sum variable.
• END {print sum} is used to print the sum variable.

Manipulating text

Extracting a substring

To extract a substring from a line with awk, you can combine the print and substr functions as follows:

$ <command> | awk '{print substr($<column>, <start>, <length>)}'

Run in Warp

Where: 

• <column> is the column number of the string you want to extract a substring from. Note that the full input string can be accessed through the column number 0.
• <start> is a number representing the starting position within the specified string.
• <length> is a number representing the length of the substring.

For example, this command will take as input the entire "Hello World" string and print 5 characters starting at position 7:

$ echo "Hello World" | awk '{print substr($0, 7, 5)}'
World

Run in Warp

And this command will take as input the second column and print 5 characters starting at position 0:

$ echo "Hello World" | awk '{print substr($2, 0, 5)}'
World

Run in Warp

Replacing the occurrences of a string or pattern

To replace all the occurrences of a string or pattern with awk, you can use the gsub function as follows:

$ <command> | awk '{gsub(<pattern>, <string>, <column>); print}'

Run in Warp

Where:

• <pattern> is the string or pattern you want to replace.
• <string> is the string you want to replace the matched pattern with.

For example, this command will replace all the occurrences of the letter l with x:

$ echo "Hello World" | awk '{gsub("l", "x", $0); print}'
Hexxo Worxd

Run in Warp

Easily retrieve this command using the Warp’s AI Command Suggestions

If you’re using Warp as your terminal, you can easily retrieve this command using the Warp AI Command Suggestions feature:

Entering awk replace all occurrences into the AI Command Suggestions will prompt a awk command that can then be quickly inserted into your shell by doing CMD+ENTER.

Conditional statements

To execute an action based on the evaluation of a condition with awk, you can use the following syntax:

$ <command> | awk '{if (<condition>) {<action>} else if (<condition>) {<action>} else {<action>}}'

Run in Warp

For example, let's consider the following students.txt file:

Alice 45
Bob 30
Charlie 25

Run in Warp

This command will check if the value of the second column is greater or equal to 40 and print the first column with “Passed” if true, or “Failed” otherwise:

$ cat students.txt | awk '{if ($2 >= 40) { print $1, "Passed" } else { print $1, "Failed" }}'
Alice Passed
Bob Failed
Charlie Failed

Run in Warp