I/O schedulers in Linux sit between application requests and physical disk hardware, deciding the order in which I/O operations execute. The fundamental difference between work-conserving and non-work-conserving schedulers comes down to how they handle idle periods. Work-Conserving Schedulers A work-conserving scheduler must dispatch a pending I/O request whenever one exists, regardless of physical disk positioning….
Filesystem corruption occasionally happens. When it does, you need fsck to run before the system fully mounts its filesystems. Here’s how to force a filesystem check during the next boot. Using touch to Mark Filesystems The traditional method is to create /forcefsck: sudo touch /forcefsck sudo reboot On the next boot, the system will check…
When you try to create a filesystem on a disk with mkfs, you may see this error: # mkfs -t ext4 /dev/sdd1 mke2fs 1.42.8 (20-Jun-2013) /dev/sdd1 is apparently in use by the system; will not make a filesystem here! Even though the device isn’t mounted, mkfs refuses to proceed. This safety mechanism prevents accidental data…
When a hard drive fails physically, data recovery becomes challenging but sometimes possible. If the drive isn’t recognized by the system or showing signs of mechanical failure, professional recovery services may be necessary—but they’re expensive. Before going that route, you can attempt recovery yourself using open-source tools. When to attempt DIY recovery You can safely…
Understanding the difference between reliability, durability, and availability is critical when designing or operating distributed storage systems like HDFS, Ceph, or cloud object storage. These terms are often confused, but they address distinct concerns. Durability Durability answers: If the system fails completely, will my data survive? Durability is about persistence to non-volatile storage. Data is…
When managing LVM storage, you need to know which physical disks support each logical volume — especially for risk assessment when hardware fails. This guide shows you how to trace those relationships. Using lvdisplay to map extents The lvdisplay command with the –maps flag shows how logical extents map to physical volumes: lvdisplay –maps This…
LVM (Logical Volume Management) abstracts physical storage into logical volumes, giving you flexibility to resize filesystems, snapshot data, and manage disk space without downtime. It sits between your physical disks and the filesystems you mount. The three core LVM concepts are: Physical Volumes (PVs): Physical disks or disk partitions Volume Groups (VGs): Collections of PVs…
When verifying file integrity on Linux, you’ll typically reach for one of several checksum utilities. A natural question: which one is fastest? The answer depends heavily on whether I/O or CPU is your bottleneck. Disk I/O Dominates on Most Systems In realistic scenarios with typical storage, disk I/O is almost always the limiting factor. Test…
When you repurpose disks from one RAID array to another system, Linux keeps the old RAID metadata intact as a safety measure. This prevents accidental data corruption, but it also blocks you from reusing the disk until the signatures are cleared. RAID metadata isn’t stored in one place — it’s scattered across the disk by…
GPT (GUID Partition Table) is required for disks larger than 2TB. The older MBR format maxes out at 2.2TB due to 32-bit sector addressing. Tools like fdisk will refuse to work properly with large drives and recommend switching to GPT. While parted has a steeper learning curve than some alternatives, it’s the standard for GPT…
RAID0 (striping) on enterprise RAID controllers like Broadcom MegaRAID is used to expose multiple disks as a single logical volume with no redundancy. This configuration is appropriate only for performance-critical workloads where data loss from a single disk failure is acceptable, or for temporary/cache storage. This guide covers the full setup process on servers with…
Andrew Fikes, Principal Engineer at Google, presented on storage architecture and the fundamental challenges that emerge when operating at scale. The presentation from the 2010 Faculty Summit remains relevant for understanding how distributed storage systems must be designed and the trade-offs inherent in large-scale deployments. Core Storage Challenges at Scale Google’s storage systems had to…
Colossus is Google’s successor to the Google File System (GFS), designed to handle the massive scale of data storage across Google’s infrastructure. While GFS was documented in detail in the 2003 SOSP paper, Colossus information remains sparse in academic literature—much of what we know comes from talks and interviews with Google engineers. Colossus serves as…
Historical Archive: This post lists conferences from 2011. The submission deadlines and venues are no longer current. For up-to-date conference information, check the official conference websites or resources like Conference Alert and CCF Deadlines. This post lists important conferences on Cloud Computing in year 2012. OSDI 2012 10th USENIX Symposium on Operating Systems Design and…