How to Compress/Uncompress Files in Linux Using gzip, bzip2, 7z, rar and zip

Posted on

Compress/uncompress files are frequent operations. The normal tools for compressing/uncompressing in Linux is gzip, bzip2, 7z, rar and zip. This post introduces how to compress and uncompress file in Linux using these tools. We use best compressing rate with all these tools and mark the options for “best rate” in bold fonts. We can delete
Read more

Hadoop TeraSort Benchmark

Posted on

TeraSort is one of Hadoop’s widely used benchmarks. Hadoop’s distribution contains both the input generator and sorting implementations: the TeraGen generates the input and TeraSort conducts the sorting. Here, we provide a short tutorial for using the Hadoop TeraSort benchmark. TeraGen generates random data that can be used as input data for a subsequent running
Read more

Large-scale Data Storage and Processing System in Datacenters

Posted on

Research on Cloud Computing has made big progresses and many excellent large-scale systems have been designed in recent years. I compiled a list of some large-scale data storage and processing systems in datacenters as follows. Storage systems Google File System (GFS): http://research.google.com/archive/gfs.html HDFS implementation: https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html Colossus (GFS2): Colossus: Successor to the Google File System (GFS)
Read more

Hadoop Installation Tutorial (Hadoop 1.x)

Posted on

Update: If you are new to Hadoop and trying to install one. Please check the newer version: Hadoop Installation Tutorial (Hadoop 2.x). Hadoop mainly consists of two parts: Hadoop MapReduce and HDFS. Hadoop MapReduce is a programming model and software framework for writing applications, which is an open-source variant of MapReduce that is initially designed
Read more

Setting Up a Git Server Using Gitosis

Posted on

Update: Since gitosis is not maintained and supported, please check out gitolite for setting up a new git server. (see the comment from Sitaram Chamarty, the gitolite author, the author of gitolite.) Gitosis is a piece of software writen by Tommi Virtanen for hosting git repositories. It manages multiple repositories under the same user account.
Read more

Hadoop Default Ports

Posted on

Hadoop’s namenode and datanodes expose a bunch of TCP ports used by Hadoop’s daemons to communicate to each other or listen directly to users’ requests. These ports information are needed by both the Hadoop users and cluster administrators to write programs or configure firewalls/gateways accordingly. A post written by Philip Zeyliger from Cloudera’s blog summarizes the
Read more

A Simple Sort Benchmark on Hadoop

Posted on

After [[hadoop-installation-tutorial|installing Hadoop]], we usually run some benchmark programs to test whether the system works well. In the post of the Hadoop install tutorial, we show a very simple to grep strings from a simple sets of files. In this post, we introduce the Sort for testing and benchmarking Hadoop. The Sort program is also
Read more

Pitfalls and Lessons on Configuing and Tuning Hadoop

Posted on

This post lists pitfalls and lessons learning when configuring and tuning Hadoop. Hadoop with IPv6 Hadoo doesn’t support IPv6 currently (up to 0.20.2 and 0.21.0): Hadoop and IPv6. The performance of the cluster may suffer from turning IPv6 on in clusters: mail archive. One good practice is to disable IPv6 on servers in the Hadoop
Read more

Setting Up Standalone (Local) Hadoop

Posted on

Hadoop is designed to run on [[hadoop-installation-tutorial|hundreds to thousands of computers]] inside cluster. However, Hadoop is configured to run things in a non-distributed mode as a single Java process by default. This is specially useful for debugging since distributed debugging is really a nightmare. This post introduces how to set up a standalone Hadoop environment.
Read more

mrcc – A Distributed C Compiler System on MapReduce

Posted on

The mrcc project’s homepage is here: mrcc project. Abstract mrcc is an open source compilation system that uses MapReduce to distribute C code compilation across the servers of the cloud computing platform. mrcc is built to use Hadoop by default, but it is easy to port it to other could computing platforms, such as MRlite,
Read more