The default number of loop devices on most Linux systems is 8, which is often insufficient for scenarios like running multiple virtual machines with loop device-backed storage, containerized environments, or disk image management. If you need more than the default allocation, you’ll need to increase max_loop. First, check your current loop device count: losetup -f…
Xen remains a robust hypervisor for production environments—AWS EC2, Citrix Hypervisor, and many hosting providers still rely on it. However, KVM has become the default for most Linux distributions due to simpler management and better integration with standard tooling. For new deployments on standard Linux, evaluate KVM with libvirt or container solutions like Docker and…
Free virtualization on Linux has multiple solid options. While VMware Player is still available, QEMU combined with libvirt has become the standard approach for most sysadmins. This guide covers creating VM disk images and configurations that work across multiple hypervisors. Creating Virtual Disk Images The first step is creating a virtual disk. QEMU’s qemu-img tool…
Crontab controls what runs and when. Each crontab entry has this format: .—————- minute (0-59) | .————- hour (0-23) | | .———- day of month (1-31) | | | .——- month (1-12) OR jan,feb,mar,apr … | | | | .—- day of week (0-6) (Sunday=0 or 7) OR sun,mon,tue … | | | | |…
Jeff Dean’s “Designs, Lessons and Advice from Building Large Distributed Systems” is essential reading for anyone building or operating distributed infrastructure. This talk, presented at multiple conferences including LADIS and various tech events, distills decades of experience at Google and now at Google Research. If you’re designing systems that scale beyond a single machine, you…
TeraSort is Hadoop’s standard benchmark for measuring distributed sorting performance across clusters. It consists of three components: TeraGen generates random input data, TeraSort performs the distributed sort, and TeraValidate confirms the output is globally sorted. Modern Hadoop deployments typically run on Hadoop 3.x or higher, with improvements in YARN resource management, better performance tuning options,…
Colossus is Google’s successor to the Google File System (GFS), designed to handle the massive scale of data storage across Google’s infrastructure. While GFS was documented in detail in the 2003 SOSP paper, Colossus information remains sparse in academic literature—much of what we know comes from talks and interviews with Google engineers. Colossus serves as…
Hadoop 1.x reached end-of-life in 2014 and is no longer maintained. This post documents a deprecated architecture for historical reference only. For new deployments, use Hadoop 3.x+ with YARN, which offers significant improvements in resource management, multi-tenancy, and reliability. See the Apache Hadoop documentation for current versions. For managed services, consider AWS EMR, Google Dataproc,…
Understanding distributed systems and cloud computing starts with the literature. These papers form the foundation for anyone working in the space — they’re not just historical artifacts, they’re still actively referenced and their design principles remain relevant. Internet-Scale Systems and Datacenters The Google cluster architecture paper remains essential reading. It laid out the practical realities…
Historical Archive: This post lists conferences from 2012. The submission deadlines and venues are no longer current. For up-to-date conference information, check the official conference websites or resources like Conference Alert and CCF Deadlines. This post lists important conferences related to Cloud Computing in year 2013. SOSP 2013 SOSP’13: The 24th ACM Symposium on Operating…
Historical Archive: This post lists conferences from 2011. The submission deadlines and venues are no longer current. For up-to-date conference information, check the official conference websites or resources like Conference Alert and CCF Deadlines. This post lists important conferences on Cloud Computing in year 2012. OSDI 2012 10th USENIX Symposium on Operating Systems Design and…
Archived Content (2010): This post describes a research project from 2010. The tools and versions mentioned (Hadoop 0.20, MapReduce Streaming) are historically significant but have been largely superseded by modern distributed build systems like Bazel Remote Execution and cloud-native CI/CD pipelines. mrcc – A Distributed C Compiler System on MapReduce Original Project Date: January 2010…