stop-dfs.sh report that there are no datanodes running on some nodes like hdfs-node-000208: no datanode to stop However, there are DataNode process running there. How to clean these processes on many (100s) of nodes? You may use this piece of bash script: for i in `cat hadoop/etc/hadoop/slaves`; do echo $i; ssh $i ‘jps | grep […]
Posts tagged Command
105 posts in total.
How to merge sam files on Linux?
The samtools merge can merge bam files while it can not work for sam files. How to merge sam files on Linux? According to the sam format specification, header lines start with @, while alignment lines do not. So you can use grep to merge sam files as follows efficiently. Assume the header is from […]
How to do diff like `git diff –word-diff` without git on Linux?
The result of git by git diff –word-diff is great. But how to do diff like git diff –word-diff without git on Linux? The plain diff command on Linux seems not accept options like –word-diff. The wdiff is for word-diff: wdiff program is a front end to diff for comparing files on a word per […]
How to convert PDF to text with format kept on Linux?
How to convert PDF to text with format kept on Linux? Many of the formatting in PDF will not be available in text. But better keep the text’s relative positions as the same. For example, the table columns should be kept. The pdftotext tool can convert PDF to text pretty well: pdftotext – Portable Document […]
How to simulate pressing multiple keys together on Linux?
This post introduces a key presser on Linux with xvkb. But how to simulate pressing multiple keys together on Linux? That is, I would like to have multiple keys such as “A” “B” “C” “D” pressed together and being hold. How to simulate this on Linux with programs/commands? You can make use of xdotool (xdotool: […]
How to cut by multiple spaces in Linux?
How to cut by multiple spaces in Linux? For example, if I get a string like a b c d where the spaces among the fields are not decided. How to get the four fields a, b, c and d out? Here, we take getting ‘b’ (the 2nd field) as the example. You can first […]
How to set the replication factor for one file when it is uploaded by `hdfs dfs -put` command line in HDFS?
When uploading a file by the hdfs dfs -put command line in HDFS, how to set a replication factor instead of the global one for that file? For example, HDFS’s global replication factor is 3. For some temporary files, I would like to save just one copy for faster uploading and saving disk space. The […]
How to iterate all files and directories under a directory in Python?
How to iterate all files and directories under a directory in Python? For example, I would like to iterate each file and directory under /mnt/data/ /mnt/data/ |– file.txt `– great That is: /mnt/data/files.txt /mnt/data/great You can use the os.walk to do this: root, dirs, files = os.walk(path).next() root will be the path.dirs will contain the […]
How to detect whether Linux runs in UEFI or BIOS mode inside the Linux?
How to detect whether Linux runs in UEFI or BIOS mode inside the Linux itself without needed to boot the the management console of the mother board? You can detect whether Linux runs in EFI mode by checking whether /sys/firmware/efi exist. In bash, you can test by [ -d /sys/firmware/efi/ ] This technique is used […]
How to make “tree” output consistent on Linux
I tried tree on different Linux boxes to verify the files by diff. However, I found the format can be a little bit different on different nodes. For examples, the tree result could be . |– test2 | |– test4 | `– test5 `– test3 1 directory, 3 files or . ├── test2 │ ├── […]
How to open the preferences dialog in Gthumb 3.4 on Linux?
How to open the preferences dialog in Gthumb 3.4 on Linux? There seems no options to open the dialog in the menu on the toolbar. It is in the application menu of gthumb under Gnome 3. Click gthumb application icon on the top left side of you desktop and in the menu you can find […]
How to find out the MX record for a domain on Linux?
How to find out a domain’s MX record for Email on Linux? For example, to find which domain/IP the email to gmail.com is delivered to. You may make use of the host command. The -t option is used to select the query type. type can be any recognized query type: CNAME, NS, SOA, SIG, KEY, […]
How to allow websites’ .htaccess files in Apache2?
How to allow websites to use .htaccess files in Apache2 web servers? The AllowOverride directive controls this: http://httpd.apache.org/docs/current/mod/core.html#allowoverride To allow AllowOverride in a directory, add the following rules to the /etc/httpd/conf/httpd.conf or similar configuration file for your apache installation: <Directory “/var/www/www.example.com”> Options FollowSymLinks # # AllowOverride controls what directives may be placed in .htaccess files. […]
How to force umount a NFS directory on Linux?
The NFS server is down. It is reported “Stale NFS file handle”. I tried ‘umount’ and ‘umount -f’. But neither succeeded. # umount /mnt/store umount.nfs: /mnt/store: Stale NFS file handle # umount -f /mnt/store umount.nfs: /mnt/store: Stale NFS file handle How to force umounting the NFS without rebooting the Linux server? I usually have to […]
How to wrap long lines in a text file on Linux?
I have some text files which contain some long lines. Now I would like to break lines to at most 80 characters. How to wrap long lines in a text file on Linux? The fold command can wrap lines. fold – wrap each input line to fit in specified width For your purpose, fold -w […]
How to merge 2 .a libraries to one .a library on Linux?
We have 2 static .a libraries and we would like to merge them into a single one for easier usage. How to merge 2 .a libraries to one .a library on Linux? With GNU ar, you can specify the single command-line option -M and control it with a script supplied via standard input, like the […]
How to make Fedora Linux not clean some files in /tmp/?
On my Fedora 20, I find that the system automatically clean up file under /tmp/. This is convenient. However, it cause some problems for some programs. For example, HDFS puts its DataNode pid file under /tmp/ by default like hadoop-hadoop-datanode.pid. After it is cleaned up, the hadoop-daemon.sh script will consider there is no DataNode running. […]
How to deactivate a LVM logical volume on Linux?
How to deactivate a LVM logical volume activated by #vgchange -aay on Linux You may need to make a LVM volume group inactive and thus unknown to the kernel. To deactivate a volume group, use the -a (–available) argument of the vgchange command. To deactivates the volume group vg, use this command # vgchange -a […]
How to encode spaces in curl GET request URL on Linux?
The problem is like this: I would like to send a GET request using curl and the content has spaces, such as curl “http://example.com/send?msg=hello world” The space and the “world” will be left away if this command is executed directory on Linux. How to encode spaces in curl request URL? You can use the –date-urlencode […]
How to run screen on a Linux host reporting “Cannot make directory ‘/var/run/screen’: Permission denied”?
I want to run screen on a remote host on which I do not have root/sudo privilege and screen is not installed. I can compile or copy the screen program that can run on the host. However, it reports: “Cannot make directory ‘/var/run/screen’: Permission denied” As screen is not installed on the host, the directory […]