You have internal hosts on a private network (like 10.0.3.0/24) that aren’t directly accessible from your workstation. They sit behind a gateway (bastion host) like gateway.example.org. You need seamless SSH access to these internal IPs without manually hopping through the gateway each time. Using SSH ProxyJump The modern, straightforward approach is SSH’s built-in ProxyJump feature….
When managing user accounts on a cluster or application, you’ll often need to generate temporary passwords programmatically. Here’s a practical approach using Scala’s built-in utilities. Basic implementation with scala.util.Random For non-security-critical use cases, here’s a straightforward generator: def randomString(len: Int): String = { val rand = new scala.util.Random() val chars = “0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz” (0 until len).map(_…
Domain registration pricing has commoditized significantly over the past few years. Most registrars cluster around similar price points, and permanent discounts are increasingly rare. Rather than chasing one-off coupons that expire or have restrictive terms, understanding how registrars price domains and what you actually need will serve you better. Current Market Rates As of 2026,…
When you need to rsync against hosts behind NAT gateways or port-forwarded SSH services, the standard port 22 won’t work. The solution is the -e flag, which lets you specify a custom SSH command with your desired port. Basic usage Use -e to pass custom SSH arguments: rsync -avxP –delete -e “ssh -p 13022” username@example.com:/remote/path/…
Redis is an in-memory data structure store designed for speed and simplicity. Understanding its architecture and consistency guarantees is essential for deploying it correctly in production systems. Core Architecture Redis operates as a single-threaded, event-driven server that uses an in-memory data store with optional persistence. The single-threaded nature simplifies reasoning about concurrent access — all…
Choosing the right number of mappers and reducers directly impacts Hadoop job performance. This isn’t a set-and-forget configuration—it depends on your cluster characteristics, data size, and task complexity. Mapper Configuration The number of mappers is primarily determined by the number of HDFS blocks in your input files. Each block typically generates one map task by…
Consistency models define the guarantees that a distributed system makes about the order and visibility of data modifications across nodes. Understanding these models is essential when designing systems, choosing databases, or debugging data anomalies in production. Strong Consistency Models Strict Consistency is the ideal but rarely achievable model. It guarantees that reads always reflect the…
SQL layers abstract traditional relational query interfaces over distributed NoSQL and big data systems, enabling applications to leverage SQL’s familiarity while benefiting from horizontal scalability. Here are the key approaches and systems that have emerged in this space. Distributed SQL on Key-Value Stores Phoenix (Apache Phoenix) provides a SQL interface over HBase. It compiles SQL…
MySQL Cluster uses NDB (Network Database) storage engine to provide high availability and data redundancy across multiple nodes. This guide covers installation and configuration on modern RHEL-compatible systems. Prerequisites Three or more servers (minimum: one management node, two data nodes, one SQL node) Rocky Linux 9 or AlmaLinux 9 Root or sudo access Static IP…
Data visualization transforms raw metrics into actionable insights. When you’re managing infrastructure metrics, application logs, business KPIs, or system performance data, visualization is how you extract signal from noise. The human brain processes visual information faster and more reliably than log files or spreadsheet rows—which is why dashboards have become essential infrastructure in modern operations….
Quartz is an open-source Java library for scheduling background jobs and tasks. Whether you need to execute work at specific times, on recurring intervals, or in response to events, Quartz provides a robust scheduling framework that integrates with standalone applications, application servers, and clustered environments. Core Features Embeds directly into standalone or web applications Supports…
Running commands across multiple servers is fundamental to cluster management and infrastructure automation. While orchestration tools like Ansible and Terraform dominate modern deployments, sometimes you need lightweight solutions that work without dependencies—or you’re already invested in SSH-based infrastructure. When managing clusters, you typically work in two modes: serial execution (one node at a time) and…
When SSH fails but the daemon still runs, or when kernel panics strike, you need automated recovery without physical access. This guide covers three essential techniques: forcing reboots when the system is unresponsive, auto-recovery from kernel panics, and alerting you when it happens. Force a reboot when SSH is unresponsive If a server stops responding…
Hadoop 2.x reached end-of-life in 2016. This guide covers setup for learning purposes and legacy system maintenance only. For production deployments, use Hadoop 3.x or later, which includes performance improvements, better YARN scheduling, HDFS erasure coding, and improved security. Cloud-managed options like AWS EMR, Google Dataproc, and Azure HDInsight eliminate most operational overhead. This guide…
SSH performance is constrained by cipher choice, especially for large file transfers over high-bandwidth links where CPU becomes the bottleneck. Selecting the right cipher, key exchange algorithm, and compression settings can yield substantial throughput improvements. Current Cipher Landscape Modern OpenSSH (7.4+) has deprecated or removed weak ciphers. Here’s what’s viable today: Secure and performant: chacha20-poly1305@openssh.com…
Benchmarks exist to answer one question: how fast can this system process data? The AMPLab Big Data Benchmark, developed by UC Berkeley, became a foundational effort to answer that question systematically across different distributed query engines. The Benchmark’s Original Scope The AMPLab benchmark evaluated five systems circa 2013: Redshift – Amazon’s columnar data warehouse (built…
This guide sets up OpenLDAP for centralized authentication and NFS with automount to provide unified home directories across a cluster. The setup uses modern OpenLDAP backends and current NFS4 best practices. System Environment Server: LDAP and NFS: 10.0.0.2, Fedora 41 x86_64 Base DN: dc=lgcpu1 Clients: Range: 10.0.0.1/24, Fedora 41 x86_64 LDAP Server Setup Package Installation…
MapReduce is a foundational distributed computing model, but it’s no longer the default choice for new projects. Hadoop’s MapReduce (via YARN in Hadoop 2.x+) remains production-stable and useful for specific scenarios, while Apache Spark, Flink, and cloud-native services handle most modern workloads more efficiently. When MapReduce Still Makes Sense Use MapReduce if you’re: Maintaining existing…
When managing Xen Dom0 hosts, shutting down guest VMs (DomUs) before powering off the physical server is essential to prevent data loss and filesystem corruption. Modern Xen distributions (including Citrix Hypervisor and AWS infrastructure) still rely on this principle, though the tooling has evolved. Using xl for VM Shutdown Current Xen installations use xl rather…
When managing multiple servers, you need reliable ways to verify which hosts are reachable. While orchestration tools like Kubernetes and Ansible handle monitoring at scale, a lightweight Bash script works well for quick spot checks, operational tasks, or smaller deployments. Basic ICMP Reachability Check The simplest approach uses ping to test ICMP reachability: ping -W…