How To Filter The Output of Commands

Filtering the output with grep

To filter a command's output, you can feed the data it writes to the standard output to the grep command using the pipe operator (|) as follows:

$ <command> | grep <pattern>

Where:

• command is the command you want to filter the output of.
• pattern is either a string pattern or regular expression.

Alternatively, to filter the content of a file, you can pass it as an argument of the grep command as follows:

$ grep <pattern> <file>

Where:

• file is the path of the file you want to filter the output of.

For example, this command will filter the server logs at the information level contained in the logs file by only matching and outputting the lines containing the INFO string:

$ cat logs | grep "INFO"
2023-11-08 08:12:45 [INFO] User 'john_doe' logged in.
2023-11-08 08:27:50 [INFO] Service restarted successfully.
2023-11-08 08:40:05 [INFO] Server backup initiated.

Note that this command can also be written like this:

$ grep "INFO" logs

Specifying multiple patterns at once

To specify multiple patterns at once, you can use the grep command with the -e flag as follows:

$ <command> | grep -e <pattern> [-e <pattern> …]

Note that the -e flag can be repeated multiple times, for each pattern.

For example, this command will only output the Docker images starting with an n or u character using two regular expressions:

$ docker images | grep -e '^n' -e '^u'
ubuntu       latest    e4c58958181a   6 weeks ago    77.8MB
node         latest    add6f751ed2b   2 months ago   1.1GB
nginx        latest    f5a6b296b8a2   2 months ago   187MB

You can learn more about matching multiple patterns at once with our other articles on how to grep multiple strings, words, and patterns and how to grep across multiple lines.

Excluding lines based on a pattern

To invert the search and return lines that do not include a specific pattern or exclude a specific string, you can use the grep command with the -v flag as follows:

$ <command> | grep -v <pattern>

For example, this command will exclude all the listed entries with a .js or .json file extension:

$ ls | grep -v -E '\.(js|json)$'

Where:

• -E turns on Extended Regex Expressions for grep. For more information, see our section on common gotchas with grep.

You can learn more about excluding patterns with our article on how to exclude patterns or files with grep.

Easily filter commands using Warp Block Filtering feature

If you’re using Warp as your terminal, you can easily filter outputs and logs using the Warp Block Filtering feature:

To apply a dynamic filter to a block, you can either click on the filter icon in the top right corner of a block or press Option+Shift+F, and type in a string or pattern to match. You can learn more about this feature on the official Warp Block Filtering page.

Filtering and formatting the output with awk

The awk command is a powerful tool used for pattern scanning and text processing.

Similar to grep, you can filter the output of a command based on a pattern using the following awk command:

$ <command> | awk '/<pattern>/ {print}'

Where:

• command is the command you want to filter the output of.
• pattern is a string pattern or regular expression.
• print is an action (a command) used to define what to do with the filtered output.

For example, this command will filter the server logs at the information level contained in the logs file by only matching and printing the lines containing the INFO string:

$ cat logs | awk '/INFO/ {print}'
2023-11-08 08:12:45 [INFO] User 'john_doe' logged in.
2023-11-08 08:27:50 [INFO] Service restarted successfully.
2023-11-08 08:40:05 [INFO] Server backup initiated.

Outputting specific columns

To extract and print specific columns of the output, you can use the print action of the awk command followed by the columns numbers as follows:

$<command>|awk '{print $1,...,$N}'

Where:

• $N is the number of the column to output.

For example, this command will only display the permissions and names of the files listed by the ls -l command, which are respectively the first and ninth columns:

$ ls -l|awk'{print $1,$9}'
total
-rw-r--r-- index.js
drwxr-xr-x node_modules
-rw-r--r-- package-lock.json
-rw-r--r-- package.json

Replacing patterns in the output

To replace the first occurrence of a pattern in the output, you can use the sub action of the awk command as follows:

$ <command> | awk '{sub(/<pattern>/, <string> [, <column>]); print}'

Where:

• pattern is a regular expression.
• string is a replacement string.
• column is the number of the column the pattern should be replaced in.

For example, this command will substitute the domain names of the email addresses with three \ characters:

$ cat passwords.txt | awk '{sub(/@[a-z]+\.[a-z]+/, "@***", $3); print}'
john   h3ll0   john@***
jack   w015d   jack@***

Alternatively, to replace all the occurrences of a pattern in the output, you can use the gsub action.

For example, this command will substitute every single letter of the passwords with a \ character:

$ cat passwords.txt | awk '{gsub(/./, "*", $2); print}'
john   *****   john@email.com
jack   *****   jack@email.com

Removing duplicate lines with uniq

To remove duplicate lines from an output, you can use uniq command in combination with the sort command as follows:

$ <command> | sort | uniq

For example, this command will first sort the content of the logs files alphabetically using the sort command, then deduplicate and count each line using the uniq -c command:

$ cat logs | sort | uniq -c
   2 Error: Connection timeout
   1 Info: Server restarted
   3 Warning: Disk space low

Showing the first lines with head

To show the first 10 lines of a command output, you can use the head command as follows:

$ <command> | head

Alternatively, you can specify a file path using the following syntax:

$ head <file>

Showing the first N lines

To show a specific number of lines instead, you can use the -n flag as follows:

$ <command> | head -n <number>

For example, this command will sort the content of the logs file in reverse order and only output the first 2 lines:

$ cat logs | sort -r | tail -n 2
2023-11-08 08:15:21 [ERROR] Database connection failed.
2023-11-08 08:12:45 [INFO] User 'john_doe' logged in

Showing the last lines with tail

To show the last 10 lines of a command output, you can use the tail command as follows:

$ <command> | tail

Alternatively, you can specify a file path using the following syntax:

$ tail <file>

Showing the last N lines

To show a specific number of lines instead, you can use the -n flag as follows:

$ <command> | tail -n <number>

For example, this command will only show the last 3 lines of the logs file containing the ERROR string:

$ cat logs | grep "ERROR" | tail -n 3
2023-11-08 08:25:11 [ERROR] Out of memory. System halted.
2023-11-08 08:30:02 [ERROR] Disk space full. Critical alert.
2023-11-08 08:35:17 [ERROR] HTTP 500 Internal Server Error on /api/v1/user

Note that if the number is prefixed with a plus sign (+), the tail command will display the output from that specified line number onwards.

For example, this command will output the last two lines of the logs file:

$ tail -n 2 logs
2023-11-08 08:20:30 [INFO] File 'report.pdf' downloaded by 'marketing_team'.
2023-11-08 08:25:11 [ERROR] Out of memory. System halted.

And this command will output the content of the logs file from the second line onwards:

$ tail -n +2 logs
2023-11-08 08:15:21 [ERROR] Database connection failed.
2023-11-08 08:17:09 [WARNING] CPU temperature exceeds threshold.
2023-11-08 08:20:30 [INFO] File 'report.pdf' downloaded by 'marketing_team'.
2023-11-08 08:25:11 [ERROR] Out of memory. System halted.