How to disable auto mounting on Linux Mint Cinnamon?

Posted on

I prefer Linux Mint (I am using Cinnamon) not automatically mounting a portable storage such as USB disk or iPhone for me and I would like do the mounting manually if I need it. The default behavior of Linux Mint is to automatically mount the disks. How to disable this? The auto mounting on Linux
Read more

How to add a new HDFS NameNode metadata directory to an existing cluster?

Posted on

We have a running HDFS cluster. Currently, the NameNode metadata data directory has only one directory configured in hdfs-site.xml: <property> <name>dfs.namenode.name.dir</name> <value>file:///home/hadoop/hdfs/</value> <description>NameNode directory for namespace and transaction logs storage.</description> </property> We would like to add a new directory for dfs.namenode.name.dir to make replicas of the metadata on a separated disk for higher data reliability.
Read more

how to remove specific directories recursively

Posted on

How to remove .svn directories under hlfs dir recursively as follows. weiwei@weiwei-HP-Compaq-dx6128-MT-PX478AV:~/workshop1/hlfs > find ./ -name “.svn” ./test/build/.svn ./test/.svn ./output/conf/.svn ./output/lib32/.svn ./patches/.svn ./src/include/.svn ./src/include/api/.svn ./src/snapshot/.svn ./src/snapshot/unittest/build/.svn ./src/snapshot/unittest/.svn ./src/utils/.svn ./src/clean/Mapreducer/build/.svn ./src/clean/Mapreducer/.svn ./src/clean/.svn ./src/clean/unittest/.svn ./src/icache/.svn ./src/icache/unittest/.svn ./src/backend/.svn ./src/storage/.svn ./src/cache/.svn ./src/cache/unittest/.svn ./src/clone/.svn ./src/tools/.svn ./src/tools/unittest/.svn ./src/logger/.svn weiwei@weiwei-HP-Compaq-dx6128-MT-PX478AV:~/workshop1/hlfs > find ./ -name “.svn” | xargs rm -rf

How to balance DataNode storage in HDFS?

Posted on

As nodes are added and deleted in a Hadoop cluster. Storage usage across DataNodes may be different. Some DataNodes’ disks are almost used up while some others’ are almost empty. How to balance data across DataNodes in HDFS? Hadoop provides the balancer to redistribute the data. Brief introduction to balancer in Hadoop: balancer. The design
Read more

What’s the difference between Reliability, Durability, and Availability for data storage system?

Posted on

Some important concepts in distributed system like Hadoop distributed file system, Google file system and so on. Answer from http://www.quora.com/Whats-the-difference-between-Reliability-Durability-and-Availability-for-data-storage-system The difference between durability and availability is fairly simple. Durability is about what happens when all power goes out everywhere. Has all data been written to stable storage that doesn’t require power (e.g. disk/flash), in
Read more

How to back up emails from the email server?

Posted on

offlineimap actually does “synchronization”—if an email is deleted from the server, the email will be deleted from the local Maildir (http://offlineimap.org). This is good for mail sync. However, for backing up emails,one would like to keep the email even if it is deleted from the server. How to achieve this? You may use `getmail` which
Read more

How to use LVM groups with libvirt

Posted on

libvirt uses files for VM images by default under /var/lib/libvirt/images/. LVM is a convenient tool for managing VM images. How to use LVM groups with libvirt? This post discusses the method to add LVM groups as a storage pool for libvirt: http://ailoo.net/2011/02/use-a-lvm-volume-group-with-libvirt/ For a general introduction of the storage management in libvirt, please check: http://libvirt.org/storage.html

HTML form generation from the database and store value into the database

Posted on

I have a “t_form” table and a “t_attribute” table. It looks a little bit like below. Form table form_id | form_name | description ———————————– 1 | Laptop | Possible attributes are Model, Screen Size, OS, Cd Player 2 | Mobile | Possible attributes are Model, OS 3 | Tablet | Possible attributes are Model, Screen
Read more

How to find out and change the storage engine of tables in MySQL

Posted on

How to find out and change the storage engine of tables in MySQL databases? Find out the storage engine of a table in a database: SELECT ENGINE FROM information_schema.TABLES WHERE TABLE_SCHEMA = ‘database’ AND TABLE_NAME = ‘table’ Change the storage engine of a table: ALTER TABLE table ENGINE = type type can be innodb or
Read more

MySQL at Facebook

Posted on

Facebook uses lots MySQL databases. Any information about how Facebook scales MySQL? Some information on the Web: MySQL at Facebook’s page https://www.facebook.com/MySQLatFacebook?filter=1 A post by Ryan Thiessen, Database Operations at Facebook on Quora: http://www.quora.com/Facebook-Engineering/How-does-Facebook-structure-MySQL-so-that-it-is-robust-and-scalable And more: http://mashable.com/2011/12/15/facebook-timeline-mysql/ http://gigaom.com/2011/12/06/facebook-shares-some-secrets-on-making-mysql-scale/ http://www.wired.com/wiredenterprise/2011/12/facebook-timeline-anatomy “A lot of people are surprised that for this shiny new thing for Facebook, we’re using
Read more

SQL layers on NoSQL databases

Posted on

What are the SQL layer solution over NoSQL databases such as key/value stores? Phoenix: A SQL layer on HBase: https://github.com/forcedotcom/phoenix They also show some performance results: https://github.com/forcedotcom/phoenix/wiki/Performance F1 – The Fault-Tolerant Distributed RDBMS Supporting Google’s Ad Business: http://research.google.com/pubs/pub38125.html With F1, we have built a novel hybrid system that combines the scalability, fault tolerance, transparent sharding,
Read more

Managing LVM

Posted on

Any tutorials or tips on how to manage LVM? LVM provides a flexible and easy way to management storage disks on Linux. It also provides a set of tools for management. Here are some tutorials and references for managing LVM: LVM man pages: https://www.systutorials.com/docs/linux/man/8-lvm/#lbAL Manage Disk Volume in Linux: http://www.hongkedavid.com/blog/disk_volume.html LVM with Xen: https://www.systutorials.com/b/linux/tag/lvm/ Remove missing phyiscal volumes
Read more

Where Does Evolution Save Its Data and Configuration Files on Linux?

Posted on

Evolution is a great personal information management tool that provides Email, address book and calendar tools. Evolution provides many enterprise friendly feature such as native support to Microsoft Exchange connectivity for Emails, address books and calendars. Evolution uses various ways including plain files and dconf configuration systems. This post will give an introduction to the
Read more

How to Upload Large Files to Amazon S3 with AWS CLI

Posted on

Amazon S3 is a widely used public cloud storage system. S3 allows an object/file to be up to 5TB which is enough for most applications. The AWS Management Console provides a Web-based interface for users to upload and manage files in S3 buckets. However, uploading a large files that is 100s of GB is not
Read more

Which Checksum Tool on Linux is Faster?

Posted on

It is common practice to calculate the checksums for files to check its integrity. For large files, the checksum computation is slow. Now I am wondering why it is so slow and whether choosing another tool will be better. In this post, I try three common tools md5sum, sha1sum and crc32 to compute checksums on
Read more

SSD Enabled For DreamHost Shared Hosting: Simple Performance Measurement

Posted on

SSD is common for VPS and PaaS virtual machines for higher I/O performance. Now, it is coming to shared hosting too. DreamHost states that “Now with solid state drives (SSDs), our standard web hosting loads pages 200% faster”. We ourselves are happy to know this performance improvement with the price kept the same. Good work,
Read more

Making GPT Partition Table and Creating Partitions Using parted in Linux

Posted on

My best favorite disk partition table manipulation tools are cfdisk/fdisk on Linux. However, for large disks, cfdisk/fdisk (of the versions by this post is written) will just give up with a message suggesting GPT partition table format and using GNU parted like WARNING: The size of this disk is 6.0 TB (6001042391040 bytes). DOS partition
Read more

Hadoop Installation Tutorial (Hadoop 2.x)

Posted on

Hadoop 2 or YARN is the new version of Hadoop. It adds the yarn resource manager in addition to the HDFS and MapReduce components. Hadoop MapReduce is a programming model and software framework for writing applications, which is an open-source variant of MapReduce designed and implemented by Google initially for processing and generating large data
Read more

RAII-like Error Handling and Resource Management in C

Posted on

Error handling and resource management are pervasive in programs. RAII originated in C++ is great. With RAII, it is much easier to write easy-to-read code that allocats/deallocats resources in the constructor/destructors. By representing a resource with a local object, we are sure that local object’s destructor will release the resource and will not forget to
Read more

Data Consistency Models of Public Cloud Storage Services: Amazon S3, Google Cloud Storage and Windows Azure Storage

Posted on

The public cloud storage services like Amazon S3, Google Cloud Storage and Windows Azure Storage replicate the data to ensure high availability. On the other hand, with data being replicated, the storage services exhibits certain data consistency models. Different cloud service providers employ different data consistency models nowadays. In this post, we survey the data
Read more