Filesystem corruption occasionally happens. When it does, you need fsck to run before the system fully mounts its filesystems. Here’s how to force a filesystem check during the next boot. Using touch to Mark Filesystems The traditional method is to create /forcefsck: sudo touch /forcefsck sudo reboot On the next boot, the system will check…
When running virtual machines with QEMU/KVM, you’ll frequently need to transfer files or share directories between the host and guest systems. Several methods work well depending on your use case and performance requirements. 9p (Virtio) Filesystem The 9p protocol is the most straightforward approach for sharing directories. It’s lightweight and doesn’t require additional guest agents….
The HDFS NameNode holds the entire filesystem namespace and block map in memory. Estimating memory requirements accurately is critical for cluster planning and preventing out-of-memory failures that can cripple your entire cluster. Core Memory Components The NameNode’s memory consumption breaks down into several key areas: Namespace Objects: The NameNode maintains an in-memory representation of the…
Samba defaults to asynchronous writes for performance reasons, but the behavior is fully configurable. Understanding these options is critical when data integrity matters more than speed. Default Async Behavior By default, Samba writes are asynchronous. When smbd receives a write request, it: Copies data to a memory buffer Returns success to the client immediately Flushes…
Atomicity is fundamental to distributed filesystems. When multiple processes across a cluster access the same files, operations must either complete fully or not at all—no partial states. HDFS provides specific atomic guarantees that applications can rely on for implementing distributed locks and coordinating access patterns. What HDFS Guarantees as Atomic As of Hadoop 3.2.1 (and…
On Linux systems, the root filesystem (/) can be on various device types — a raw disk partition, an LVM logical volume, or other storage configurations. Finding the underlying physical disk requires checking the device hierarchy, which differs depending on the storage setup. Quick method with df and lsblk The simplest approach uses lsblk with…
When working with Hadoop Distributed File System (HDFS), files containing spaces in their names require special handling during the hdfs dfs -put command. Without proper escaping or quoting, the shell interprets spaces as delimiters, splitting the filename into multiple arguments. Direct Upload with Quoting The simplest approach is to wrap the filename in quotes: hdfs…
You need to check whether a directory exists in your $PATH variable. This is useful when verifying that a tool installation succeeded, or when debugging shell initialization scripts. The regex approach Since $PATH uses colons as separators, a regex is the most straightforward method: if [[ “$PATH” =~ (^|:)”/usr/local/mytool/bin”(:|$) ]]; then echo “Directory found in…
Python provides several ways to check if files or directories exist, with different trade-offs depending on your use case. Using pathlib (recommended) The modern approach uses pathlib.Path, which is cleaner and more object-oriented: from pathlib import Path path = Path(‘/etc/passwd’) # Check if path exists (file or directory) if path.exists(): print(“Path exists”) # Check if…
When you try to unmount a filesystem, Linux sometimes refuses with a “device is busy” error. This safety mechanism prevents data loss when processes are actively using the filesystem. But there are legitimate cases where you need to force an unmount: a dead NFS server, a failed USB device, or a hung mount that won’t…
When accepting path input in Bash scripts, users might provide /path/to/dir/ or /path/to/dir. Normalizing these to a consistent format without the trailing slash is a common requirement. The Problem You need a reliable method that: Removes trailing slashes if present Works with both files and directories Handles paths that already lack a trailing slash Works…
The free command—along with ps, top, kill, pgrep, and dozens of other system utilities—is maintained as part of the procps-ng project. Source repository The source code is hosted at: https://gitlab.com/procps-ng/procps You can clone it with: git clone https://gitlab.com/procps-ng/procps.git cd procps Most distributions package procps-ng, so you likely already have it installed. You can verify the…
When you try to create a filesystem on a disk with mkfs, you may see this error: # mkfs -t ext4 /dev/sdd1 mke2fs 1.42.8 (20-Jun-2013) /dev/sdd1 is apparently in use by the system; will not make a filesystem here! Even though the device isn’t mounted, mkfs refuses to proceed. This safety mechanism prevents accidental data…
When you see output like this from hdfs dfsadmin -report, you need to understand what each metric means and how to respond: Under replicated blocks: 139016 Blocks with corrupt replicas: 9 Missing blocks: 0 Understanding block states in HDFS is essential for cluster health. The distinction between these categories determines how urgent your response needs…
When writing C programs on Linux that handle file paths, you need to allocate buffers safely. The kernel defines maximum path and filename lengths through standard macros in <linux/limits.h>. Getting the limits On Linux systems, <linux/limits.h> provides two key macros: NAME_MAX — maximum length of a single filename component (255 characters) PATH_MAX — maximum length…
On a Linux system, you often need to let regular users mount and unmount specific devices without requiring sudo or the root password. This is common for external drives, USB devices, or secondary internal disks. There are several approaches, each with different security and usability tradeoffs. Using /etc/fstab with the user Option The simplest approach…
When a filesystem listed in /etc/fstab fails to mount during boot, the system can hang or drop to an emergency shell, preventing normal startup. This happens because the kernel treats all mount entries equally by default. Non-critical filesystems—network shares, external drives, or optional data partitions—shouldn’t block your entire system from booting. Understanding fstab mount options…
In HDFS, files are split into blocks and replicated across multiple DataNodes for fault tolerance and parallelism. When you need to know exactly which DataNodes hold a specific file’s blocks, use hdfs fsck with the -locations and -blocks flags. Basic command hdfs fsck /path/to/file -files -locations -blocks This queries the NameNode and returns the block…
systutorials.com runs on WordPress with the Twenty Fourteen theme (now legacy) and a custom Q2A installation for the Q&A section. If you’re rebuilding something similar, the approach differs between the two platforms. WordPress Theme Strategy If starting fresh today, Twenty Fourteen is outdated. Modern alternatives include: Twenty Twenty-Five — the current WordPress default, lightweight and…
When a program appends data to a file, you need to know how much space remains on the partition containing that file. This determines whether the file can grow or if you’re about to hit a capacity limit. Quick answer To find available space for a file, use df: df -B1 /mnt/logs/app-log.log | tail -1…