How to export my Google Chrome password on Linux to a human-readable text file? In newer versions of Chrome, the passwords are stored using the encrypted password storage provided by the system, either Gnome Keyring or KWallet. We need to force Chrome to use a temporary profile folder with unencrypted password storage. Step 1. Connect…
One of HDFS cluster’s hdfs dfsadmin -report reports: Under replicated blocks: 139016 Blocks with corrupt replicas: 9 Missing blocks: 0 The “Under replicated blocks” can be re-replicated automatically after some time. How to handle the missing blocks and blocks with corrupt replicas in HDFS? Understanding these blocks A block is “with corrupt replicas” in HDFS…
I do not want to have coredump for the developing machine using systemd on Linux? How to disable the coredump function of systemd? You can disable coredump function of systemd by adding this line Storage=none to /etc/systemd/coredump.conf For your reference: Storage= Controls where to store cores. One of “none”, “external”, “journal”, and “both”. When “none”,…
I can only connect to a Linux server by SSH. How to check whether a USB disk is connected to the Linux server? 2 ways I usuaully use: Method One: Check the print out of dmesg After the USB storage is connected and detected by Linux, it may print some messages containing “[sde]” (the new…
I prefer Linux Mint (I am using Cinnamon) not automatically mounting a portable storage such as USB disk or iPhone for me and I would like do the mounting manually if I need it. The default behavior of Linux Mint is to automatically mount the disks. How to disable this? The auto mounting on Linux…
We have a running HDFS cluster. Currently, the NameNode metadata data directory has only one directory configured in hdfs-site.xml: <property> <name>dfs.namenode.name.dir</name> <value>file:///home/hadoop/hdfs/</value> <description>NameNode directory for namespace and transaction logs storage.</description> </property> We would like to add a new directory for dfs.namenode.name.dir to make replicas of the metadata on a separated disk for higher data reliability….
How to remove .svn directories under hlfs dir recursively as follows. weiwei@weiwei-HP-Compaq-dx6128-MT-PX478AV:~/workshop1/hlfs > find ./ -name “.svn” ./test/build/.svn ./test/.svn ./output/conf/.svn ./output/lib32/.svn ./patches/.svn ./src/include/.svn ./src/include/api/.svn ./src/snapshot/.svn ./src/snapshot/unittest/build/.svn ./src/snapshot/unittest/.svn ./src/utils/.svn ./src/clean/Mapreducer/build/.svn ./src/clean/Mapreducer/.svn ./src/clean/.svn ./src/clean/unittest/.svn ./src/icache/.svn ./src/icache/unittest/.svn ./src/backend/.svn ./src/storage/.svn ./src/cache/.svn ./src/cache/unittest/.svn ./src/clone/.svn ./src/tools/.svn ./src/tools/unittest/.svn ./src/logger/.svn weiwei@weiwei-HP-Compaq-dx6128-MT-PX478AV:~/workshop1/hlfs > find ./ -name “.svn” | xargs rm -rf
As nodes are added and deleted in a Hadoop cluster. Storage usage across DataNodes may be different. Some DataNodes’ disks are almost used up while some others’ are almost empty. How to balance data across DataNodes in HDFS? Hadoop provides the balancer to redistribute the data. Brief introduction to balancer in Hadoop: balancer. The design…
Some important concepts in distributed system like Hadoop distributed file system, Google file system and so on. Answer from http://www.quora.com/Whats-the-difference-between-Reliability-Durability-and-Availability-for-data-storage-system The difference between durability and availability is fairly simple. Durability is about what happens when all power goes out everywhere. Has all data been written to stable storage that doesn’t require power (e.g. disk/flash), in…
offlineimap actually does “synchronization”—if an email is deleted from the server, the email will be deleted from the local Maildir (http://offlineimap.org). This is good for mail sync. However, for backing up emails,one would like to keep the email even if it is deleted from the server. How to achieve this? You may use `getmail` which…
libvirt uses files for VM images by default under /var/lib/libvirt/images/. LVM is a convenient tool for managing VM images. How to use LVM groups with libvirt? This post discusses the method to add LVM groups as a storage pool for libvirt: http://ailoo.net/2011/02/use-a-lvm-volume-group-with-libvirt/ For a general introduction of the storage management in libvirt, please check: http://libvirt.org/storage.html
I have a “t_form” table and a “t_attribute” table. It looks a little bit like below. Form table form_id | form_name | description ———————————– 1 | Laptop | Possible attributes are Model, Screen Size, OS, Cd Player 2 | Mobile | Possible attributes are Model, OS 3 | Tablet | Possible attributes are Model, Screen…
How to find out and change the storage engine of tables in MySQL databases? Find out the storage engine of a table in a database: SELECT ENGINE FROM information_schema.TABLES WHERE TABLE_SCHEMA = ‘database’ AND TABLE_NAME = ‘table’ Change the storage engine of a table: ALTER TABLE table ENGINE = type type can be innodb or…
Facebook uses lots MySQL databases. Any information about how Facebook scales MySQL? Some information on the Web: MySQL at Facebook’s page https://www.facebook.com/MySQLatFacebook?filter=1 A post by Ryan Thiessen, Database Operations at Facebook on Quora: http://www.quora.com/Facebook-Engineering/How-does-Facebook-structure-MySQL-so-that-it-is-robust-and-scalable And more: http://mashable.com/2011/12/15/facebook-timeline-mysql/ http://gigaom.com/2011/12/06/facebook-shares-some-secrets-on-making-mysql-scale/ http://www.wired.com/wiredenterprise/2011/12/facebook-timeline-anatomy “A lot of people are surprised that for this shiny new thing for Facebook, we’re using…
What are the SQL layer solution over NoSQL databases such as key/value stores? Phoenix: A SQL layer on HBase: https://github.com/forcedotcom/phoenix They also show some performance results: https://github.com/forcedotcom/phoenix/wiki/Performance F1 – The Fault-Tolerant Distributed RDBMS Supporting Google’s Ad Business: http://research.google.com/pubs/pub38125.html With F1, we have built a novel hybrid system that combines the scalability, fault tolerance, transparent sharding,…
ANSI C is a standardized version of the C programming language that was introduced in 1989 by the American National Standards Institute (ANSI). The ANSI C standard defines the syntax and semantics of the C language, including its grammar. In this post, we will explore the ANSI C grammar in more detail. The ANSI C…
Any tutorials or tips on how to manage LVM? LVM provides a flexible and easy way to management storage disks on Linux. It also provides a set of tools for management. Here are some tutorials and references for managing LVM: LVM man pages:Â https://www.systutorials.com/docs/linux/man/8-lvm/#lbAL Manage Disk Volume in Linux:Â http://www.hongkedavid.com/blog/disk_volume.html LVM with Xen:Â https://www.systutorials.com/b/linux/tag/lvm/ Remove missing phyiscal volumes…
Evolution is a great personal information management tool that provides Email, address book and calendar tools. Evolution provides many enterprise friendly feature such as native support to Microsoft Exchange connectivity for Emails, address books and calendars. Evolution uses various ways including plain files and dconf configuration systems. This post will give an introduction to the…
Amazon S3 is a widely used public cloud storage system. S3 allows an object/file to be up to 5TB which is enough for most applications. The AWS Management Console provides a Web-based interface for users to upload and manage files in S3 buckets. However, uploading a large files that is 100s of GB is not…
It is common practice to calculate the checksums for files to check its integrity. For large files, the checksum computation is slow. Now I am wondering why it is so slow and whether choosing another tool will be better. In this post, I try three common tools md5sum, sha1sum and crc32 to compute checksums on…