How To Filter The Output of Commands

[#filter-outputs-with-grep] Filtering the output with [.inline-code]grep[.inline-code] [#filter-outputs-with-grep]

To filter a command's output, you can feed the data it writes to the standard output to the [.inline-code]grep[.inline-code] command using the pipe operator ([.inline-code]|[.inline-code]) as follows:

$ <command> | grep <pattern>

Where:

• [.inline-code]command[.inline-code] is the command you want to filter the output of.
• [.inline-code]pattern[.inline-code] is either a string pattern or regular expression.

Alternatively, to filter the content of a file, you can pass it as an argument of the [.inline-code]grep[.inline-code] command as follows:

$ grep <pattern> <file>

Where:

• [.inline-code]file[.inline-code] is the path of the file you want to filter the output of.

For example, this command will filter the server logs at the information level contained in the [.inline-code]logs[.inline-code] file by only matching and outputting the lines containing the [.inline-code]INFO[.inline-code] string:

 $ cat logs | grep "INFO"
2023-11-08 08:12:45 [INFO] User 'john_doe' logged in.
2023-11-08 08:27:50 [INFO] Service restarted successfully.
2023-11-08 08:40:05 [INFO] Server backup initiated.

Note that this command can also be written like this:

 $ grep "INFO" logs

[#specify-multiple-patterns] Specifying multiple patterns at once [#specify-multiple-patterns]

To specify multiple patterns at once, you can use the [.inline-code]grep[.inline-code] command with the [.inline-code]-e[.inline-code] flag as follows:

$ <command> | grep -e <pattern> [-e <pattern> …]

Note that the [.inline-code]-e[.inline-code] flag can be repeated multiple times, for each pattern.

For example, this command will only output the Docker images starting with an [.inline-code]n[.inline-code] or [.inline-code]u[.inline-code] character using two regular expressions:

$ docker images | grep -e '^n' -e '^u'
ubuntu       latest    e4c58958181a   6 weeks ago    77.8MB
node         latest    add6f751ed2b   2 months ago   1.1GB
nginx        latest    f5a6b296b8a2   2 months ago   187MB

You can learn more about matching multiple patterns at once with our other articles on how to grep multiple strings, words, and patterns and how to grep across multiple lines.

[#exclude-lines-with-a-pattern] Excluding lines based on a pattern [#exclude-lines-with-a-pattern]

To invert the search and return lines that do not include a specific pattern or exclude a specific string, you can use the [.inline-code]grep[.inline-code] command with the [.inline-code]-v[.inline-code] flag as follows:

$ <command> | grep -v <pattern>

For example, this command will exclude all the listed entries with a [.inline-code].js[.inline-code] or [.inline-code].json[.inline-code] file extension:

$ ls | grep -v -E '\.(js|json)$'

Where:

• [.inline-code]-E[.inline-code] turns on Extended Regex Expressions for grep. For more information, see our section on common gotchas with grep.

You can learn more about excluding patterns with our article on how to exclude patterns or files with grep.

Easily filter commands using Warp Block Filtering feature

If you’re using Warp as your terminal, you can easily filter outputs and logs using the Warp Block Filtering feature:

To apply a dynamic filter to a block, you can either click on the filter icon in the top right corner of a block or press [.inline-code]Option+Shift+F[.inline-code], and type in a string or pattern to match. You can learn more about this feature on the official Warp Block Filtering page.

[#filter-outputs-with-awk] Filtering and formatting the output with [.inline-code]awk[.inline-code] [#filter-outputs-with-awk]

The [.inline-code]awk[.inline-code] command is a powerful tool used for pattern scanning and text processing.

Similar to [.inline-code]grep[.inline-code], you can filter the output of a command based on a pattern using the following [.inline-code]awk[.inline-code] command:

$ <command> | awk '/<pattern>/ {print}'

Where:

• [.inline-code]command[.inline-code] is the command you want to filter the output of.
• [.inline-code]pattern[.inline-code] is a string pattern or regular expression.
• [.inline-code]print[.inline-code] is an action (a command) used to define what to do with the filtered output.

For example, this command will filter the server logs at the information level contained in the [.inline-code]logs[.inline-code] file by only matching and printing the lines containing the [.inline-code]INFO[.inline-code] string:

$ cat logs | awk '/INFO/ {print}'
2023-11-08 08:12:45 [INFO] User 'john_doe' logged in.
2023-11-08 08:27:50 [INFO] Service restarted successfully.
2023-11-08 08:40:05 [INFO] Server backup initiated.

[#output-specific-columns] Outputting specific columns [#output-specific-columns]

To extract and print specific columns of the output, you can use the [.inline-code]print[.inline-code] action of the [.inline-code]awk[.inline-code] command followed by the columns numbers as follows:

$<command>|awk '{print $1,...,$N}'

Where:

• [.inline-code]$N[.inline-code] is the number of the column to output.

For example, this command will only display the permissions and names of the files listed by the [.inline-code]ls -l[.inline-code] command, which are respectively the first and ninth columns:

$ ls -l|awk'{print $1,$9}'
total
-rw-r--r-- index.js
drwxr-xr-x node_modules
-rw-r--r-- package-lock.json
-rw-r--r-- package.json

[#replace-matching-patterns] Replacing patterns in the output [#replace-matching-patterns]

To replace the first occurrence of a pattern in the output, you can use the [.inline-code]sub[.inline-code] action of the [.inline-code]awk[.inline-code] command as follows:

$ <command> | awk '{sub(/<pattern>/, <string> [, <column>]); print}'

Where:

• [.inline-code]pattern[.inline-code] is a regular expression.
• [.inline-code]string[.inline-code] is a replacement string.
• [.inline-code]column[.inline-code] is the number of the column the pattern should be replaced in.

For example, this command will substitute the domain names of the email addresses with three [.inline-code]*[.inline-code] characters:

$ cat passwords.txt | awk '{sub(/@[a-z]+\.[a-z]+/, "@***", $3); print}'
john   h3ll0   john@***
jack   w015d   jack@***

Alternatively, to replace all the occurrences of a pattern in the output, you can use the [.inline-code]gsub[.inline-code] action.

For example, this command will substitute every single letter of the passwords with a [.inline-code]*[.inline-code] character:

$ cat passwords.txt | awk '{gsub(/./, "*", $2); print}'
john   *****   john@email.com
jack   *****   jack@email.com

[#remove-duplicate-lines] Removing duplicate lines with [.inline-code]uniq[.inline-code] [#remove-duplicate-lines]

To remove duplicate lines from an output, you can use [.inline-code]uniq[.inline-code] command in combination with the [.inline-code]sort[.inline-code] command as follows:

$ <command> | sort | uniq

For example, this command will first sort the content of the logs files alphabetically using the [.inline-code]sort[.inline-code] command, then deduplicate and count each line using the [.inline-code]uniq -c[.inline-code] command:

$ cat logs | sort | uniq -c
   2 Error: Connection timeout
   1 Info: Server restarted
   3 Warning: Disk space low

[#show-first-lines-with-head] Showing the first lines with [.inline-code]head[.inline-code] [#show-first-lines-with-head]

To show the first 10 lines of a command output, you can use the [.inline-code]head[.inline-code] command as follows:

$ <command> | head

Alternatively, you can specify a file path using the following syntax:

$ head <file>

[#show-the-first-n-lines] Showing the first N lines [#show-the-first-n-lines]

To show a specific number of lines instead, you can use the [.inline-code]-n[.inline-code] flag as follows:

$ <command> | head -n <number>

For example, this command will sort the content of the [.inline-code]logs[.inline-code] file in reverse order and only output the first 2 lines:

$ cat logs | sort -r | tail -n 2
2023-11-08 08:15:21 [ERROR] Database connection failed.
2023-11-08 08:12:45 [INFO] User 'john_doe' logged in

[#show-the-last-lines-with-tail] Showing the last lines with [.inline-code]tail[.inline-code] [#show-the-last-lines-with-tail]

To show the last 10 lines of a command output, you can use the [.inline-code]tail[.inline-code] command as follows:

$ <command> | tail

Alternatively, you can specify a file path using the following syntax:

$ tail <file>

[#show-the-last-n-lines] Showing the last N lines [#show-the-last-n-lines]

To show a specific number of lines instead, you can use the [.inline-code]-n[.inline-code] flag as follows:

$ <command> | tail -n <number>

For example, this command will only show the last 3 lines of the [.inline-code]logs[.inline-code] file containing the [.inline-code]ERROR[.inline-code] string:

$ cat logs | grep "ERROR" | tail -n 3
2023-11-08 08:25:11 [ERROR] Out of memory. System halted.
2023-11-08 08:30:02 [ERROR] Disk space full. Critical alert.
2023-11-08 08:35:17 [ERROR] HTTP 500 Internal Server Error on /api/v1/user

Note that if the [.inline-code]number[.inline-code] is prefixed with a plus sign ([.inline-code]+[.inline-code]), the [.inline-code]tail[.inline-code] command will display the output from that specified line number onwards.

For example, this command will output the last two lines of the [.inline-code]logs[.inline-code] file:

$ tail -n 2 logs
2023-11-08 08:20:30 [INFO] File 'report.pdf' downloaded by 'marketing_team'.
2023-11-08 08:25:11 [ERROR] Out of memory. System halted.

And this command will output the content of the [.inline-code]logs[.inline-code] file from the second line onwards:

$ tail -n +2 logs
2023-11-08 08:15:21 [ERROR] Database connection failed.
2023-11-08 08:17:09 [WARNING] CPU temperature exceeds threshold.
2023-11-08 08:20:30 [INFO] File 'report.pdf' downloaded by 'marketing_team'.
2023-11-08 08:25:11 [ERROR] Out of memory. System halted.