Reading: Years in Big Data. Months with Apache Flink. 5 Early Observations With Stream Processing: https://data-artisans.com/blog/early-observations-apache-flink. The article suggest adopting the right solution, Flink, for big data processing. Flink is interesting and built for stream processing. The broader view and take away may be to solve problems using the right solution. We saw many painful […]
One of HDFS cluster’s hdfs dfsadmin -report reports: Under replicated blocks: 139016 Blocks with corrupt replicas: 9 Missing blocks: 0 The “Under replicated blocks” can be re-replicated automatically after some time. How to handle the missing blocks and blocks with corrupt replicas in HDFS? Understanding these blocks A block is called corrupt by HDFS if […]
After a node failure and restarting the HDFS, the NameNode reports: “The reported blocks 1968810 needs additional 5071 blocks to reach the threshold 0.9990 of total blocks 1975856. Safe mode will be turned off automatically.” in the log. Why this happens? And how to fix it? About why the NameNode stays in the safe mode: […]
What is the design of Snapshots in HDFS? This PDF documents the design of snapshot. Jing Zhao and Tsz-Wo Sze from Hortonworks gave a great talk on the design of HDFS snapshots. The slides can be downloaded at here. The development of snapshot is tracked by HDFS-2802. Answered by Eric Z Ma.
Technical discussions on Redis. Redis internal documentation: http://redis.io/topics/internals Redis manifesto, the philosophy behind Redis: http://oldblog.antirez.com/post/redis-manifesto.html Redis Architecture: Overview Of Redis Architecture Redis data model and eventual consistency: http://antirez.com/news/36 Answered by anonymous.
Which are the consistency models used for distributed systems? Papers that survey the consistency models Robert C. Steinke and Gary J. Nutt. 2004. A unified theory of shared memory consistency. J. ACM 51, 5 (September 2004), 800-849. DOI=10.1145/1017460.1017464 http://doi.acm.org/10.1145/1017460.1017464 David Mosberger. 1993. Memory consistency models. SIGOPS Oper. Syst. Rev. 27, 1 (January 1993), 18-26. DOI=10.1145/160551.160553 […]
I want to learn transactional memory technologies. Any suggestions on Transactional memory learning materials? Thanks! I highly suggest the Transactional Memory lecture by James R. Larus and Ravi Rajwar of Synthesis Lectures on Computer Architecture: The Transactional Memory lecture:http://www.morganclaypool.com/doi/abs/10.2200/S00070ED1V01Y200611CAC002 Link to the PDF:http://www.morganclaypool.com/doi/pdf/10.2200/S00070ED1V01Y200611CAC002 Answered by anonymous.