diff on Linux is a very handy tool. Any good diff alternatives on Windows to find differences between 2 files? A GUI program will be better on Windows. I would recommend meld for Windows users. Meld is a visual diff and merge tool open source software. meld is available on Linux, Windows and Mac OS […]
In Python, os.makedirs() with 0777 mode can not give others write permission The code is as follows $ python Python 2.7.5 (default, Aug 4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux2 Type “help”, “copyright”, “credits” or “license” for more information. >>> import os >>> os.makedirs(“/tmp/test1/test2”, 0777) >>> The created dirs are not […]
HDFS SecondaraNameNode log shows 2017-08-06 10:54:14,488 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint java.io.IOException: Inconsistent checkpoint fields. LV = -63 namespaceID = 1920275013 cTime = 0 ; clusterId = CID-f38880ba-3415-4277-8abf-b5c2848b7a63 ; blockpoolId = BP-578888813-10.6.1.2-1497278556180. Expecting respectively: -63; 263120692; 0; CID-d22222fd-e28a-4b2d-bd2a-f60e1f0ad1b1; BP-622207878-10.6.1.2-1497242227638. at org.apache.hadoop.hdfs.server.namenode.CheckpointSignature.validateStorageInfo(CheckpointSignature.java:134) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:531) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:395) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:361) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:357) It seems the checkpoint […]
How to detect whether a file is being written by any other process in Linux? Before a program open a file to processes it, it wants to ensure no other processes are writing to it. Here, we are sure after the files are written and closed, they will not be written any more. Hence, one-time […]
HDFS stores the metadata of files and blocks in the memory of the NameNode. How to estimate the memory usage of HDFS NameNode for a HDFS cluster? Each file and each block has around 150 bytes of metadata on NameNode. So you may do the calculation based on this. For examples, assume block size is […]
I have a very large (e.g. 100GB) .gz file and would like to split it into smaller files like 8GB/each for storage/copying. How to split a gzip file to several small ones on Linux? You may use the tool split to split a file by sizes. An example to split a large.tgz to at most […]
How to get the mtime of a file on Linux from the file’s path? You can use stat to get the file status including the mtime: %y time of last modification, human-readable %Y time of last modification, seconds since Epoch As an example, $ stat -c %y ./file 2017-06-26 13:33:06.764042064 +0800 $ stat -c %Y […]
In Python, how to process read a file line by line to process it? Like those lines in Bash: while read line ; do echo $line done < ./input.txt In Python, you can process a file line by line by a for in the file like with open(“./input.txt”, “r”) as thefile: for line in thefile: […]
i need to create a program which is taking information from printer , so that i cannot use os namespace , so there is any way without using the os namespace You may use a way as follows. open the file seek to the end tell its position which is the file size An example […]
In the screen, copying the history of the window output is quite hard. How to save the screen easily to a file? First type Ctrl + A then : to get to command mode. In the command mode, execute hardcopy -h /path/to/file screen will save the window output to /path/to/file. Answered by Eric Z Ma.
Before reading or writing a file, access should be checked first. How to detect whether a file is readable and writable in Python? You can use the os.access(path, mode) library function https://docs.python.org/release/2.6.6/library/os.html#os.access like the Linux access library function for C. It returns True if access is allowed, False if not. For readable and writable, you […]
In the Go programming language, How to process a file line by line? An equivalent piece of Bash code could be while read line ; do echo $line done < ./input.txt You can make use of the bufio package’s Scan(). // open the file filepath f := os.Open(filepath) // create a scanner fs := bufio.NewScanner(f) […]
In Bash, the [ -f ] and [ -d ] tests can test whether a file or a directory exist. What are the corresponding python method for these tests? For -f: **os.path.isfile(path)** Return True if path is an existing regular file. This follows symbolic links. For -d: **os.path.isdir(path)** Return True if path is an existing […]
How to get a file’s size in Python from the full path of a file? You can use this piece of Python code: os.path.getsize(file_path) It returns the size in bytes. It raises os.error if the file does not exist or is inaccessible. Manual of os.path.getsize(): https://docs.python.org/2/library/os.path.html#os.path.getsize Answered by Eric Z Ma.
How to get the file extension from a filename in Python? For example, I would like to get “.txt” from “file.txt”. The Python code: >>> name, ext = os.path.splitext(‘file.txt’) >>> name ‘file’ >>> ext ‘.txt’ Manual of os.path.splitext: https://docs.python.org/2/library/os.path.html#os.path.splitext Answered by Eric Z Ma.
I would like to open files by open() directly. But how to get a FILE pointer from a file descriptor in C on Linux? And how to do the reverse direction: get a file descriptor from a FILE pointer? Get a FILE pointer from a file descriptor (e.g. fd) in C on Linux: FILE *file […]
How to efficiently archive a very large sparse file, say 1TB? The sparse file may contains a small amount of data, say 32MB. SEEK_HOLE and SEEK_DATA The SEEK_HOLE/SEEK_DATA functionalities play the trick and makes `tar` and `cp` handle the large sparse file very efficiently. `lseek` with `SEEK_HOLE` returns the offset of the start of the […]
How to find hard links to a file in Linux? To find out all hard links to file1: $ find /home -xdev -samefile file1 Read more: http://linuxcommando.blogspot.com/2008/09/how-to-find-and-delete-all-hard-links.html Answered by SA.