Files with replication factor 1 are a liability in production HDFS clusters. They have no redundancy, meaning a single node failure results in data loss. Identifying and fixing these files should be part of your regular cluster maintenance. Using the HDFS CLI The most straightforward approach is using hdfs fsck with output parsing: hdfs fsck…
Several approaches exist to detect active file access in Linux, ranging from simple procfs inspection to sophisticated kernel-level tracking. Using lsof The most straightforward method is lsof (list open files), which shows all files currently open by running processes: lsof /path/to/file This returns process ID, user, file descriptor, and access mode. To check only write…
Monitoring server temperatures is essential for preventing hardware damage and maintaining system stability. Linux provides several tools to query thermal sensors and identify which components are running hot. Using lm-sensors The most reliable approach is lm-sensors, a userspace tool that reads temperature data from kernel drivers: # Install lm-sensors sudo apt install lm-sensors # Debian/Ubuntu…
The command journalctl -b -1 should show the previous boot’s journal, but on many systems it returns: Failed to look up boot -1: No such boot ID in journal This happens because persistent journal storage isn’t enabled by default. Without it, journald only keeps logs in volatile memory (/run/log/journal), which gets cleared on shutdown. Enable…
Understanding file descriptors is fundamental to shell scripting. By default, your shell has three standard streams: STDOUT (FD 1): Standard output, where commands print results STDERR (FD 2): Standard error, where commands print errors and diagnostics STDIN (FD 0): Standard input, where commands read data Basic Output Print to STDOUT: echo “This goes to stdout”…
To get the parent process ID (PPID) of the running process, use the os.getppid() function: import os ppid = os.getppid() print(f”Parent Process ID: {ppid}”) This returns the process ID of the parent process as an integer. On Unix-like systems, this is straightforward. On Windows, the behavior is consistent as well. Accessing Full Process Information For…
Bash has several native flags that help you trace and validate script execution before and during runtime. Trace mode (-x): Executes the script while printing each command before it runs. bash -x script.sh This prints expanded commands to stderr, showing variable substitution and command execution order. You can enable tracing for specific sections using set…
On Linux systems, the root filesystem (/) can be on various device types — a raw disk partition, an LVM logical volume, or other storage configurations. Finding the underlying physical disk requires checking the device hierarchy, which differs depending on the storage setup. Quick method with df and lsblk The simplest approach uses lsblk with…
Bash provides straightforward methods to convert Unix epoch timestamps into human-readable formats. This is essential for log parsing, system monitoring, and scripting across CI/CD pipelines and production environments. Using the date Command The standard approach is the date command with the -d flag (GNU coreutils) or -r flag (BSD/macOS): Linux (GNU date) date -d @1234567890…
Retrieving the hostname of the machine running your script is essential for logging, node identification in distributed systems, monitoring, and conditional logic based on where code executes. Bash offers multiple approaches with different tradeoffs. Using the $HOSTNAME Variable The quickest method is the $HOSTNAME environment variable, which Bash sets at shell initialization: echo $HOSTNAME host01…
The date command is your primary tool for retrieving Unix epoch timestamps. It’s available on virtually all Unix-like systems and integrates seamlessly into scripts and command pipelines. Basic Epoch Timestamp date +%s This returns the number of seconds since January 1, 1970 UTC. Output looks like 1735689600. Higher Precision with Nanoseconds For microsecond or nanosecond…
The standard sort command is single-threaded by default, which limits performance on modern multi-core systems. Several approaches exist to speed up sorting operations using parallelism. GNU Sort with Parallel Sorting Modern versions of GNU sort (8.23+) include built-in parallel sorting support. Enable it with the –parallel flag: sort –parallel=4 large_file.txt > sorted_output.txt The –parallel option…
The most straightforward way to convert SVG files to PNG is using ImageMagick or Inkscape. Both are widely available in standard repositories and handle the conversion reliably. Using ImageMagick (convert/magick) ImageMagick provides the magick command (the modern equivalent to the older convert utility): magick input.svg output.png For basic conversions, that’s all you need. To control…
There are several ways to retrieve the hostname on a Linux system from Python. The choice depends on your use case and whether you need the simple hostname or the fully qualified domain name (FQDN). Using socket.gethostname() The simplest approach uses the socket module from the standard library: import socket hostname = socket.gethostname() print(hostname) This…
When working with Hadoop Distributed File System (HDFS), files containing spaces in their names require special handling during the hdfs dfs -put command. Without proper escaping or quoting, the shell interprets spaces as delimiters, splitting the filename into multiple arguments. Direct Upload with Quoting The simplest approach is to wrap the filename in quotes: hdfs…
Disabling a user account is a common administrative task. Whether you’re offboarding an employee, removing an unused service account, or securing a compromised account, there are several approaches depending on your requirements. Lock the account with usermod The simplest method is to lock the password hash: sudo usermod -L username This prepends an exclamation mark…
Process priority management is essential for ensuring critical tasks get CPU resources when needed. Linux provides several mechanisms to control how the kernel scheduler allocates CPU time to processes. Understanding Priority Levels Linux uses two priority concepts: Nice value: ranges from -20 (highest priority) to 19 (lowest priority). Default is 0. Real-time priority: ranges from…
The most common need is collapsing multi-line records into single lines. Here are the practical approaches: Using tr to remove newlines The simplest method removes all newlines: tr -d ‘\n’ < input.txt > output.txt This works well for data without embedded line breaks you want to preserve. For CSV or structured data where you need…
The screen terminal multiplexer can capture window output in several ways. Here are the most practical approaches depending on your use case. Enable logging before starting screen Start a new screen session with the -L flag to log all output to a file: screen -L -S mysession This creates a log file named screenlog.0 in…
Spaces in file and directory paths are a common source of rsync failures. When paths contain spaces, the shell interprets them as argument separators unless you properly escape or quote them. Here’s how to handle this correctly. The core issue When rsync receives an unquoted path with spaces, the shell splits it at each space….