By default, rsync connects as your standard user. When you need to write to protected directories like /etc, /var/www, or other root-owned locations, use the –rsync-path option to invoke rsync with sudo privileges on the remote side. Basic syntax rsync -avz –rsync-path=”sudo rsync” local-file user@remote:/protected/path This tells the remote server to execute rsync under sudo….
Sparse files optimize storage by storing only metadata for contiguous zero-filled ranges instead of allocating actual disk blocks. They’re essential for VM images, virtual block devices, and large databases where much of the allocated space remains unused. Most modern filesystems support sparse files: ext4, btrfs, and XFS on Linux; NTFS on Windows; and APFS on…
When a Bash script spawns child processes—say script A starts B, which starts C, which runs rsync—you need a way to terminate the entire process tree cleanly. Simply killing the parent leaves orphaned children running in the background. The most reliable approach uses session IDs (SID). By default, child processes inherit the same session ID…
Spaces in file and directory paths are a common source of rsync failures. When paths contain spaces, the shell interprets them as argument separators unless you properly escape or quote them. Here’s how to handle this correctly. The core issue When rsync receives an unquoted path with spaces, the shell splits it at each space….
SSH key management becomes critical when you’re handling multiple identities across different hosts. By default, ssh tries keys in a specific order, but you can explicitly configure which key to use for particular hosts through the SSH config file. Using ~/.ssh/config The cleanest approach is to define host-specific key configurations in ~/.ssh/config. This file lets…
The scp command supports bandwidth limiting through the -l flag, which caps the transfer rate in Kbit/s. This prevents large file transfers from saturating your network connection. Basic Usage The -l flag specifies the maximum bandwidth in kilobits per second (Kbit/s): # Limit to 1000 Kbit/s (approximately 125 KB/s) scp -l 1000 largefile.tar.gz user@remote:/path/to/destination/ #…
When backing up with rsync, you often need to skip directories like cache/, node_modules/, or .git/ that appear at multiple levels in your directory tree. Listing every instance individually is impractical, but rsync’s exclude patterns handle this elegantly. Basic exclude syntax The simplest approach uses –exclude with a directory name pattern: rsync -avxP –exclude=cache/ /path/to/src/directory/…
When you need to rsync against hosts behind NAT gateways or port-forwarded SSH services, the standard port 22 won’t work. The solution is the -e flag, which lets you specify a custom SSH command with your desired port. Basic usage Use -e to pass custom SSH arguments: rsync -avxP –delete -e “ssh -p 13022” username@example.com:/remote/path/…
Archiving or copying a 1TB sparse file containing only 32MB of actual data is impractical with naive approaches—you’ll waste time and I/O bandwidth copying zeros. The solution is using SEEK_HOLE and SEEK_DATA flags with lseek(), which allow tools to skip over hole regions entirely. How SEEK_HOLE and SEEK_DATA work The Linux kernel provides two additional…
Before writing code, get comfortable with fundamental Linux commands. You’ll use these daily: File operations: ls, cd, cat, less Text search: grep, ripgrep (rg) Remote access: ssh, scp, rsync Session management: tmux or screen for persistent terminal sessions Shell: bash (default on most distributions), though zsh and fish are popular alternatives Start with built-in documentation….
Hadoop 2.x reached end-of-life in 2016. This guide covers setup for learning purposes and legacy system maintenance only. For production deployments, use Hadoop 3.x or later, which includes performance improvements, better YARN scheduling, HDFS erasure coding, and improved security. Cloud-managed options like AWS EMR, Google Dataproc, and Azure HDInsight eliminate most operational overhead. This guide…
Symbolic links (symlinks) are aliases that point to other files or directories. Most operations treat them transparently—programs reading or writing through a symlink behave as if working with the target directly. Understanding how to create, test, and manage them correctly is essential for system administration. Creating Symbolic Links Create a basic symlink: ln -s ../target…
Building and maintaining Linux clusters requires coordinated management of user accounts, storage, networking, and services across multiple machines. This guide covers practical approaches to common clustering challenges, from unified authentication to network isolation. Account and Storage Management Unified Authentication and Home Directories Centralizing user management across cluster nodes eliminates the overhead of maintaining individual user…
rsync is a reliable choice for backing up your Linux home directory, especially when dealing with large files and incremental changes. Unlike git (which struggles with files >5GB), rsync efficiently handles large files and only transfers what’s changed on subsequent runs. Basic Backup Command The fundamental syntax is straightforward: rsync -avxP \ /path/to/directory/to/backup \ /path/to/directory/for/storing/backup…