Perl one-liners are powerful for text processing, but by default the -p and -n options process input line by line. When you need to match patterns across multiple lines, you’ll hit limitations because \s doesn’t match newlines by default, and the record separator is set to \n. The solution is the -0777 flag, which changes…
You can extract and display a single file’s content from a tar archive without unpacking the entire archive using tar‘s standard output option. Basic Syntax tar -xOf archive.tar.gz path/to/file The key flags are: -x: Extract files from the archive -O (capital O): Write extracted files to standard output instead of disk -f: Specify the archive…
When processing text data, you often need to extract everything from a certain field onward. awk makes this straightforward once you understand the mechanics. Basic Syntax The fundamental approach uses awk’s field variables and looping: awk ‘{for(i=N;i<=NF;i++) printf “%s “, $i; print “”}’ file.txt Replace N with the field number you want to start from….
Several methods exist to prepend a prefix string to each line of text. The right approach depends on your input source and performance requirements. Using sed The sed stream editor is the most portable and efficient approach: sed ‘s/^/PREFIX/’ file.txt This uses a substitution command where ^ matches the start of each line and PREFIX…
BAM files store aligned sequence reads in a compressed binary format. Counting reads by chromosome is a common task in genomics workflows—useful for quality control, coverage analysis, and identifying potential sequencing biases. Using samtools idxstats The fastest and most straightforward approach is samtools idxstats, which requires an indexed BAM file: samtools index reads.bam samtools idxstats…
Bash doesn’t have a built-in split function, but you can achieve string splitting through several methods. The most common approach uses IFS (Internal Field Separator), though other techniques exist depending on your needs. IFS with read The simplest method for splitting a string by a delimiter: string=”apple,banana,cherry” IFS=’,’ read -ra ADDR <<< “$string” for i…
Extracting or removing trailing numbers from strings is a common task in shell scripting. Whether you’re parsing log files, processing configuration values, or cleaning up filenames, you’ll need reliable methods to handle these operations. Using Parameter Expansion The cleanest approach uses bash parameter expansion, which doesn’t require external commands and works in all POSIX-compliant shells….
The most common need is collapsing multi-line records into single lines. Here are the practical approaches: Using tr to remove newlines The simplest method removes all newlines: tr -d ‘\n’ < input.txt > output.txt This works well for data without embedded line breaks you want to preserve. For CSV or structured data where you need…
Grep handles tab characters in a few different ways depending on your needs. Here’s what actually works. Using \t with extended regex The most straightforward approach uses grep’s extended regular expressions (-E flag): grep -E ‘\t’ filename.txt This matches any line containing a literal tab character. The \t escape sequence is interpreted as a tab…
Most modern systems have thermal sensors built into the CPU and motherboard. Reading these sensors is essential for monitoring system health, diagnosing performance issues, and preventing thermal throttling or hardware damage. Install lm-sensors The standard tool for reading hardware sensors on Linux is lm-sensors. Install it on your system: Debian/Ubuntu: sudo apt install lm-sensors RHEL/CentOS/Fedora:…
Sorting text file lines by their length is useful for analyzing text patterns, finding anomalies, or preparing data for processing. Here are the most practical approaches. Using awk with sort The standard approach combines awk to calculate line length with sort to order them: awk ‘{ print length(), $0 | “sort -n” }’ /path/to/text/file This…
When you’re processing text with varying amounts of whitespace between fields, the standard cut command falls short. Given input like this: a b c d You need a way to reliably extract fields regardless of how many spaces separate them. Here are the practical approaches. Using tr with cut The tr command can normalize whitespace…
When you need the commit hash of the latest commit, there are several ways to get it. The most direct approach is using git log with format options rather than parsing output with grep. Using git log with –format The cleanest method is to use git log with the –format option: git log -1 –format=”%H”…
When processing delimited data in Bash, you often need to drop the last N fields from each line without knowing the total field count. The naive cut -d’.’ -f1 approach fails immediately when field counts vary. Given input like: systemd.3.gz systemd.mount.3.gz systemd.mount.f.3.gz You want to remove the last 2 fields (.3.gz) to get: systemd systemd.mount…
Converting text to lowercase is a common task in shell scripts and command-line workflows. Linux offers multiple approaches, each with different trade-offs in performance, portability, and Unicode handling. Using tr for simple conversions The tr command translates characters and is widely portable across Unix-like systems: echo “HELLO WORLD” | tr ‘[:upper:]’ ‘[:lower:]’ Output: hello world…
Detecting whether an image is predominantly blank, black, or contains minimal content is useful for automated workflows like batch processing, quality control, or identifying failed captures. Here’s how to do it reliably on Linux. Using ImageMagick ImageMagick’s convert command (or magick in newer versions) can extract image statistics to determine if an image is nearly…
awk is the most efficient approach for this task. It processes the file in a single pass and doesn’t load data into memory: awk ‘{ total += $5; count++ } END { print total/count }’ file.txt This assumes your numeric value is the 5th whitespace-delimited field. For a file like: Run 0: Time used: 22.235331711…
The sort command’s -n flag only handles decimal numbers. When you apply it to hexadecimal values, it falls back to lexicographic (alphabetical) sorting, which produces incorrect results since hex digits don’t follow numeric order alphabetically. Given a file with hex addresses: 400000000 __crt0 400000039 __newr0 400001B14 get_my_task_id 200000020 reg1 20400001000 status Running sort -n will…
The nl command is purpose-built for adding line numbers to text files: nl text-file By default, nl numbers all non-empty lines and skips blank lines. This behavior is controlled by the body section style, which defaults to t (text only). Common nl options Number all lines including blanks: nl -ba text-file Start numbering at a…
The cut command is designed for simple field splitting, but it has a hard limitation: the delimiter must be a single character. cut –delimiter=”delim” -f 1,3 # Error: cut: the delimiter must be a single character This is a deliberate design choice. cut prioritizes speed and simplicity for common use cases like splitting on tabs,…