I got this error when I tried to save a file with a space in its name into HDFS: $ hdfs dfs -put -f “/home/u1/testa/test a” “/u1/testa/test a” put: unexpected URISyntaxException while the HDFS seems allow spaces in its file names: https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-common/filesystem/model.html . How to achieve the effect of saving the files with spaces in…
Some time, Linux fails to unmount a filesystem reporting “device is busy”. I understand that this helps avoid data lost by disallowing unmouting a filesystem when it is being used. But for some situations, I am sure there is something wrong happened or I care not data lost, such as a NFS mounting while the…
Where can I find the source code for the free command on Linux? The source code for the free commands (and many more, like kill, ps, top, sysctl) can be found in procps-ng: https://gitlab.com/procps-ng/procps procps is a set of command line and full-screen utilities that provide information out of the pseudo-filesystem most commonly located at…
I have a disk from another server installed on a new server. However, when I try to make a filesystem on it, mkfs reports # mkfs -t ext4 /dev/sdd1 mke2fs 1.42.8 (20-Jun-2013) /dev/sdd1 is apparently in use by the system; will not make a filesystem here! This is a new disk to the new server….
One of HDFS cluster’s hdfs dfsadmin -report reports: Under replicated blocks: 139016 Blocks with corrupt replicas: 9 Missing blocks: 0 The “Under replicated blocks” can be re-replicated automatically after some time. How to handle the missing blocks and blocks with corrupt replicas in HDFS? Understanding these blocks A block is “with corrupt replicas” in HDFS…
On a Linux box, we may allow non-root users mount and umount certain disks, such as /dev/sde1. How to allow non-root users on Linux to mount and unmount disks? Please check the tutorial at Controlling Filesystem Mounting on Linux by Playing with /etc/fstab: Allow non-root users to mount and unmount filesystems.
Some entries in /etc/fstab may not not critical for booting Linux or even not available until Linux has booted. How to avoid the failures or unavailability of some mounting entries in /etc/fstab blocking the boot process of Linux? Please check the tutorial at Controlling Filesystem Mounting on Linux by Playing with /etc/fstab: Allow non-root users…
A file may be splitted to many chunks and replications stored on many datanodes in HDFS. Now, the question is how to find the DataNodes that actually store a file in HDFS? You may use the dfsadmin -fsck tool from the Hadoop hdfs util. Here is an example: $ hadoop fsck /user/aaa/file.name -files -locations -blocks…
How to import 3rd party libraries (e.g. not standard libraries) in OCaml? An answer by Gabriel Scherer on how to import Batteries is just great and answers this question with much information. Although it is for Batteries, the method is general. The OCaml compiler (or toplevel, etc.) will find with no additional information only the…
I backup my iPhone with iTune to my local computer. Where can I find out the backup files of my iPhone by iTune? And how to get the real files (like songs, notes, etc.) from the backup files? The location of the backup files from iTune It depends on the computer’s operating system: Mac: ~/Library/Application…
Linux may report “device is busy” when we try to umount a filesystem. This behavior is reasonable as it can help us avoid data loss by disallowing unmouting a filesystem when it is being used. But for situations when we are sure there is something wrong happened or we care not data lost such as…
RAID systems such as MegaRAID add signatures to disks to maintain the infomration on these didks. When we simply remove these disks and install them to another server, Linux on the new server may detect these RAID signature infomration and refuses to continue write to the disk. Here is one example that mkfs reports “apparently…
It is common for programs to write to files in filesystems on disks. However, what if the disk was almost full when your program writes to the filesystem on a disk? For systems software and mission critical programs, it is a better or must-to-do practice to check the available filesystem space before actually writing to…
Controlling the mounting of filesystems is a useful technique for managing Linux systems. The mounting configurations are mostly in the /etc/fstab file. In this post, we will discuss 2 common and useful techniques for controlling the filesystem mounting by playing with the /etc/fstab file: allowing non-root users to mount/unmount filesystems and avoiding mounting failures blocking…
SSD is common for VPS and PaaS virtual machines for higher I/O performance. Now, it is coming to shared hosting too. DreamHost states that “Now with solid state drives (SSDs), our standard web hosting loads pages 200% faster”. We ourselves are happy to know this performance improvement with the price kept the same. Good work,…
In Extending a Mounted Ext4 File System on LVM in Linux, we introduced how to extend a mounted ext4 filesystem on LVM Logical volume by adding a new physical volume. It is also common to shrink an ext4 file system as to spare some disk space. In this post, I will discuss how to shrink…
Hadoop 2 or YARN is the new version of Hadoop. It adds the yarn resource manager in addition to the HDFS and MapReduce components. Hadoop MapReduce is a programming model and software framework for writing applications, which is an open-source variant of MapReduce designed and implemented by Google initially for processing and generating large data…
LVM is a great tool to manage hard disks on Linux—you can abstract the hard drives away and manage logical volumes from volume groups, you can dynamically add or remove hard drives while the file systems on the logical volumes need not to backed up and recovered, and you may create many snapshots of the…
I tried to flash my HP Compaq CQ35-240TX’s BIOS to F.15 but unfortunately failed. The computer turned to white screen. When I restarted (powered off then boot again), this is no response any more: the hard disk and caps lock’s LED keep blinding. I thought “Oh my God! I must send it to HP to…
Creating file-backed virtual block device (VBD) for Xen virtual machines and installing Fedora 12 in Xen DomU via internet will be introduced. Note that this tutorial is based on a pretty old OS (Fedora 12). But the method here is still valid while some minor details may need to be changed for latest Xen and…