LVM logical volumes are a practical choice for Xen guest storage. They support dynamic resizing, snapshots, and efficient cloning — capabilities that make capacity management and guest provisioning straightforward compared to fixed disk images. This guide covers creating and configuring LVM-backed virtual block devices (VBDs) for Xen DomU instances. Creating the logical volume Start with…
LVM snapshots provide an efficient way to duplicate virtual machines in seconds rather than minutes. Instead of copying an entire filesystem, snapshots leverage copy-on-write semantics—physical blocks are only duplicated when the source or snapshot is modified. This approach works well for any LVM-backed virtualization platform, though Xen is used here as an example. Note: Xen…
Xen remains a production-grade hypervisor used in enterprise environments (AWS EC2, Citrix Hypervisor), though KVM has become the default for most Linux distributions. This guide covers building a stable Xen Dom0 on current Fedora releases using modern kernel sources. Hardware Considerations This guide assumes server-class hardware with: Intel or AMD processor with virtualization extensions (VT-x…
Xen remains a critical hypervisor in production environments—AWS EC2, Citrix Hypervisor, and other cloud platforms rely on it—but most Linux distributions now default to KVM for general-purpose virtualization. If you’re working with Xen specifically, this guide covers Dom0 and DomU setup on modern distributions. When to Use Xen Xen is worth considering if you need:…
On Fedora/RHEL-based systems: dnf install ecryptfs-utils On Debian/Ubuntu: apt-get install ecryptfs-utils Load the kernel module modprobe ecryptfs To ensure the module loads automatically on boot, add it to /etc/modules-load.d/: echo “ecryptfs” | sudo tee /etc/modules-load.d/ecryptfs.conf Create directories You’ll need two directories: one for encrypted storage (.private) and one as the mount point (private). The .private…
sshfs is the standard tool for mounting remote filesystems over SSH. It’s available in most distribution repositories. Debian/Ubuntu: sudo apt install sshfs RHEL/CentOS/Fedora: sudo dnf install sshfs Alpine: apk add sshfs On some systems you may need to ensure the fuse kernel module is loaded: sudo modprobe fuse Basic Mount Mount a remote directory to…
Xen remains a robust hypervisor for production environments—AWS EC2, Citrix Hypervisor, and many hosting providers still rely on it. However, KVM has become the default for most Linux distributions due to simpler management and better integration with standard tooling. For new deployments on standard Linux, evaluate KVM with libvirt or container solutions like Docker and…
Andrew Fikes, Principal Engineer at Google, presented on storage architecture and the fundamental challenges that emerge when operating at scale. The presentation from the 2010 Faculty Summit remains relevant for understanding how distributed storage systems must be designed and the trade-offs inherent in large-scale deployments. Core Storage Challenges at Scale Google’s storage systems had to…
Modern datacenters rely on distributed storage and processing systems designed to handle massive datasets across clusters of commodity hardware. Understanding these systems is essential for anyone working with infrastructure at scale. Here’s an overview of the major architectures and implementations you’ll encounter. Storage Systems Google File System (GFS) and successors GFS established the foundation for…
Colossus is Google’s successor to the Google File System (GFS), designed to handle the massive scale of data storage across Google’s infrastructure. While GFS was documented in detail in the 2003 SOSP paper, Colossus information remains sparse in academic literature—much of what we know comes from talks and interviews with Google engineers. Colossus serves as…
Hadoop 1.x reached end-of-life in 2014 and is no longer maintained. This post documents a deprecated architecture for historical reference only. For new deployments, use Hadoop 3.x+ with YARN, which offers significant improvements in resource management, multi-tenancy, and reliability. See the Apache Hadoop documentation for current versions. For managed services, consider AWS EMR, Google Dataproc,…
Understanding distributed systems and cloud computing starts with the literature. These papers form the foundation for anyone working in the space — they’re not just historical artifacts, they’re still actively referenced and their design principles remain relevant. Internet-Scale Systems and Datacenters The Google cluster architecture paper remains essential reading. It laid out the practical realities…
Archived Content (2010): This post describes a research project from 2010. The tools and versions mentioned (Hadoop 0.20, MapReduce Streaming) are historically significant but have been largely superseded by modern distributed build systems like Bazel Remote Execution and cloud-native CI/CD pipelines. mrcc – A Distributed C Compiler System on MapReduce Original Project Date: January 2010…