Do big data stream processing in the stream way

Reading: Years in Big Data. Months with Apache Flink. 5 Early Observations With Stream Processing: The article suggest adopting the right solution, Flink, for big data processing. Flink is interesting and built for stream processing. The broader view and take away may be to solve problems using the right solution. We saw many painful […]

How to handle missing blocks and blocks with corrupt replicas in HDFS?

One of HDFS cluster’s hdfs dfsadmin -report reports: Under replicated blocks: 139016 Blocks with corrupt replicas: 9 Missing blocks: 0 The “Under replicated blocks” can be re-replicated automatically after some time. How to handle the missing blocks and blocks with corrupt replicas in HDFS? Understanding these blocks A block is called corrupt by HDFS if […]

HDFS stays in safe mode because of reported blocks not reaching 0.9990 of total blocks

After a node failure and restarting the HDFS, the NameNode reports: “The reported blocks 1968810 needs additional 5071 blocks to reach the threshold 0.9990 of total blocks 1975856. Safe mode will be turned off automatically.” in the log. Why this happens? And how to fix it? About why the NameNode stays in the safe mode: […]

Consistency models for distributed systems

Which are the consistency models used for distributed systems? Papers that survey the consistency models Robert C. Steinke and Gary J. Nutt. 2004. A unified theory of shared memory consistency. J. ACM 51, 5 (September 2004), 800-849. DOI=10.1145/1017460.1017464 David Mosberger. 1993. Memory consistency models. SIGOPS Oper. Syst. Rev. 27, 1 (January 1993), 18-26. DOI=10.1145/160551.160553 […]

Transactional memory learning materials

I want to learn transactional memory technologies. Any suggestions on Transactional memory learning materials? Thanks! I highly suggest the Transactional Memory lecture by James R. Larus and Ravi Rajwar of Synthesis Lectures on Computer Architecture: The Transactional Memory lecture: Link to the PDF: Answered by anonymous.