Posts on Systems

A Beginners’ Guide to x86-64 Instruction Encoding

By Eric Z Ma,
The encoding of x86 and x86-64 instructions is well documented in Intel or AMD’s manuals. However, they are not quite easy for beginners to start with to learn encoding of the x86-64 instructions. In this post, I will give a list of useful manuals for understanding and studying the x86-64 instruction encoding, a brief introduction and an example to help you get started with the formats and encodings of the x86-64 instructions. » Read more

How to force a metadata checkpointing in HDFS

By Eric Z Ma,
The metadata checkpointing in HDFS is done by the Secondary NameNode to merge the fsimage and the edits log files periodically and keep edits log size within a limit. For various reasons, the checkpointing by the Secondary NameNode may fail. For one example, HDFS SecondaraNameNode log shows errors in its log as follows. 2017-08-06 10:54:14,488 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint java.io.IOException: Inconsistent checkpoint fields. » Read more

How To Debug Linux Kernel With Less Efforts

By Weiwei Jia,
Introduction In general, if we want to debug Linux Kernel, there are lots of tools such as Linux Perf, Kprobe, BCC, Ktap, etc, and we can also write kernel modules, proc subsystems or system calls for some specific debugging aims. However, if we have to instrument kernel to achieve our goals, usually we would not like to pay more efforts like above solutions since we’d like to achieve our aims quickly and easily. » Read more

QEMU/KVM Network Mechanisms

By Weiwei Jia,
Introduction As we know, network subsystems are important in computer systems since they are I/O systems and need to be optimized with many algorithms and skills. This article will introduce how QEMU/KVM [2] network part works. In order to put everything simple and easy to understand, we will begin with several examples and then understand how it works internally. Examples In this example, we will use TAP device [1] as QEMU/KVM host network device driver and VirtIO driver will be used to send/receive network packets/data between Host OS and Guest OS. » Read more

I/O Microscopy: Tasks’ Disk I/O Information with High Accuracy

By Weiwei Jia,
Abstract Most popular task monitor systems (such as top, iotop, proc, etc) can only get tasks’ disk I/O information like tasks’ I/O utilization percentage every seconds due to kernel timer/tick frequency and high time cost of system interfaces. This article presents I/O Microscopy, a new way to get tasks’ disk I/O information with high accuracy. Experiments show that I/O microscopy can filter out I/O intensive tasks effectively. » Read more

How does linux kernel collect task stats data

By Weiwei Jia,
Motivation ∞ Recently, I find it is hard to know the percentage of time that one process uses to wait for synchronous I/O (eg, read, etc). One way is to use the taskstats API provided by Linux Kernel [1]. However, for this way, the precision may be one problem. With this problem, I dig into Linux Kernel source codes to see how “blkio_delay_total” (Delay time waiting for synchronous block I/O to complete) is calculated. » Read more

Uploading Large Files to Amazon S3 with AWS CLI

By Eric Z Ma,
Amazon S3 is a widely used public cloud storage system. S3 allows an object/file to be up to 5TB which is enough for most applications. The AWS Management Console provides a Web-based interface for users to upload and manage files in S3 buckets. However, uploading a large files that is 100s of GB is not easy using the Web interface. From my experience, it fails frequently. » Read more

Hadoop Installation Tutorial (Hadoop 2.x)

By Eric Z Ma,
Hadoop 2 or YARN is the new version of Hadoop. It adds the yarn resource manager in addition to the HDFS and MapReduce components. Hadoop MapReduce is a programming model and software framework for writing applications, which is an open-source variant of MapReduce designed and implemented by Google initially for processing and generating large data sets. HDFS is Hadoop’s underlying data persistency layer, loosely modeled after the Google file system (GFS). » Read more

Big Data Benchmark from AMPLab of UC Berkeley

By Eric Z Ma,
Benchmarks are important to understand the performance and quantitative and qualitative comparison of different systems. Many analytic frameworks, such as Hive, Impala and Shark, are designed and implemented these years and become fundamental software for processing big data. How to benchmark these big data analytic systems is an interesting problem. The Big Data Benchmark ∞ The Big Data Benchmark from AMPLab, UC Berkeley provides quantitative and qualitative comparisons of five systems by the time this post is written: Redshift – a hosted MPP database offered by Amazon.com based on the ParAccel data warehouse, Hive – a Hadoop-based data warehousing system, Shark – a Hive-compatible SQL engine which runs on top of the Spark computing framework, Impala – a Hive-compatible* SQL engine with its own MPP-like execution engine and Stinger/Tez – Tez is a next generation Hadoop execution engine currently in development. » Read more

Data Consistency Models of Public Cloud Storage Services: Amazon S3, Google Cloud Storage and Windows Azure Storage

By Eric Z Ma,
The public cloud storage services like Amazon S3, Google Cloud Storage and Windows Azure Storage replicate the data to ensure high availability. On the other hand, with data being replicated, the storage services exhibits certain data consistency models. Different cloud service providers employ different data consistency models nowadays. In this post, we survey the data consistency models provided by the solutions from the three big players: Amazon S3 and DynamoDB, Google Cloud Storage and Windows Azure Storage. » Read more

Software Engineering Advice from Building Large-Scale Distributed Systems by Jeff Dean

By Eric Z Ma,
Software Engineering Advice from Building Large-Scale Distributed Systems by Jeff Dean. You can download the slides from Software Engineering Advice from Building Large-Scale Distributed Systems by Jeff Dean. These slides contain the “Numbers everyone should know” which everyone working on systems should be familiar with. Numbers Everyone Should Know ∞ L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 100 ns Main memory reference 100 ns Compress 1K bytes with Zippy 10,000 ns Send 2K bytes over 1 Gbps network 20,000 ns Read 1 MB sequentially from memory 250,000 ns Round trip within same datacenter 500,000 ns Disk seek 10,000,000 ns Read 1 MB sequentially from network 10,000,000 ns Read 1 MB sequentially from disk 30,000,000 ns Send packet CA->Netherlands->CA 150,000,000 ns » Read more

Hadoop MapReduce Tutorials

By Eric Z Ma,
Here is a list of tutorials for learning how to write MapReduce programs on Hadoop, the opensource MapReduce implementation with HDFS. MapReduce Tutorials ∞ The official tutorial on Hadoop MapReduce framework: http://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.html. Yahoo! Hadoop Tutorial ∞ A comprehensive tutorial on Hadoop from Yahoo! Developer Network: http://developer.yahoo.com/hadoop/tutorial/. More about MapReduce ∞ To better understand the design behind MapReduce, it is always good to read Jeff Dean and Sanjay Ghemawat’s MapReduce paper: http://research.google.com/archive/mapreduce.html. » Read more

PUMA: A MapReduce Benchmark Suite

By Eric Z Ma,
MapReduce is a well-known programming model designed for generating and processing large data. There are various MapReduce implementations. One widely known and used one may be Hadoop. Benchmarking MapReduce frameworks gets to be important. Faraz Ahmad et al. developed a benchmark suite: PUMA MapReduce Benchmark. During our work on MapReduce, we developed a benchmark suite which represents a broad range of MapReduce applications exhibiting application characteristics with high/low computation and high/low shuffle volumes. » Read more

Large-scale Data Storage and Processing System in Datacenters

By Eric Z Ma,
Research on Cloud Computing has made big progresses and many excellent large-scale systems have been designed in recent years. I compiled a list of some large-scale data storage and processing systems in datacenters as follows. Storage systems ∞Google File System (GFS): http://research.google.com/archive/gfs.html HDFS implementation: https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html Colossus (GFS2): Colossus: Successor to the Google File System (GFS) BigTable: http://research.google.com/archive/bigtable.html Megastore: http://research.google.com/pubs/pub36971.html Spanner: http://research.google.com/archive/spanner.html Dynamo: http://dl.acm.org/citation.cfm?id=1294281RAMCloud: http://dl.acm.org/citation.cfm?id=1965751 and http://dl.acm.org/citation.cfm?id=2043560Compute systems ∞MapReduce: http://research.google.com/archive/mapreduce.html Hadoop implementation: Hadoop MapReduce Tutorials Sawzall: http://research.google.com/archive/sawzall.html FlumeJava: http://dl.acm.org/citation.cfm?id=1806638 Pig latin: http://dl.acm.org/citation.cfm?id=1376726 Dryad/DryadLINQ: http://research.microsoft.com/en-us/projects/dryad/ Pregel: http://dl.acm.org/citation.cfm?id=1807184 and http://googleresearch.blogspot.com/2009/06/large-scale-graph-computing-at-google.html Dremel: http://research.google.com/pubs/pub36632.html Storm: https://blog.twitter.com/2011/a-storm-is-coming-more-details-and-plans-for-release and https://github.com/nathanmarz/storm/wiki Spark: https://www.usenix.org/conference/nsdi12/resilient-distributed-datasets-fault-tolerant-abstraction-memory-cluster-computing and http://spark-project.org/DVM: IEEE Transactions on Computers paper and VEE paperResource management ∞Mesos: http://mesos.apache.org/documentation/latest/architecture/ » Read more

Microsofts Cosmos Service

By Eric Z Ma,
Cosmos is “Microsoft’s internal data storage/query system for analyzing enormous amounts (as in petabytes) of data”. There is no paper/technical report about Cosmos published yet. I compiled a list of information about Cosmos on the Web as follows. What is Microsoft’s Cosmos service? by Yaron Y. Goland. Microsoft Cosmos: Petabytes perfectly processed perfunctorily by Seth Eliot. Cosmos Big Data and Big Challenges by Pat Helland. » Read more

Colossus: Successor to the Google File System (GFS)

By Eric Z Ma,
Colossus is the successor to the Google File System (GFS) as mentioned in the recent paper on Spanner on OSDI 2012. Colossus is also used by spanner to store its tablets. The information about Colossus is slim compared with GFS which is published in the paper on SOSP 2003. There is still some information about Colossus on the Web. Here, I list some of them. » Read more

Hadoop Installation Tutorial (Hadoop 1.x)

By Eric Z Ma,
Update: If you are new to Hadoop and trying to install one. Please check the newer version: Hadoop Installation Tutorial (Hadoop 2.x). Hadoop mainly consists of two parts: Hadoop MapReduce and HDFS. Hadoop MapReduce is a programming model and software framework for writing applications, which is an open-source variant of MapReduce that is initially designed and implemented by Google for processing and generating large data sets [1]. » Read more

mrcc – A Distributed C Compiler System on MapReduce

By Eric Z Ma,
The mrcc project’s homepage is here: mrcc project. Abstract mrcc is an open source compilation system that uses MapReduce to distribute C code compilation across the servers of the cloud computing platform. mrcc is built to use Hadoop by default, but it is easy to port it to other could computing platforms, such as MRlite, by only changing the interface to the platform. » Read more