Chapter 1. Meet Hadoop

pengjunling

　　1.      A zettabyte is 1021bytes, or equivalently one thousand exabytes, onemillion petabytes, or one billion terabytes.


2.      It has been saidthat “More data usually beats better algorithms,” which is to say that for someproblems (such as recommending movies or music based on past preferences),however fiendish your algorithms are, they can often be beaten simply by havingmore data (and a less sophisticated algorithm).
 
3.      While the storagecapacities of hard drives have increased massively over the years, accessspeeds—the rate at which data can be read from drives—have not kept up.
 
4.      The first problemof reading and writing data in parallel to or from multiple disks is hardwarefailure. The second problem is that most analysis tasks need to be able tocombine the data in some way; Various distributed systems allow data to becombined from multiple sources, but doing this correctly is notoriouslychallenging.
 
5.      Hadoop provides:a reliable shared storage and analysis system. The storage is provided by HDFSand analysis by MapReduce. There are other parts to Hadoop, but thesecapabilities are its kernel.
 
6.      MapReduce is abatch query processor, and the ability to run an adhoc query against your whole dataset and get the results in a reasonable timeis transformative.
 
7.      Seek time isimproving more slowly than transfer rate. Seeking is the process of moving thedisk’s head to a particular place on the disk to read or write data. Itcharacterizes the latency of a disk operation, whereas the transfer ratecorresponds to a disk’s bandwidth. If the data access pattern is dominated byseeks, it will take longer to read or write large portions of the dataset thanstreaming through it, which operates at the transfer rate. For updating a smallproportion of records in a database, a traditional B-Tree (which is limited bythe rate it can perform seeks) works well. For updating the majority of adatabase, a B-Tree is less efficient than MapReduce, which uses Sort/Merge torebuild the database.
 
8.      MapReduce can beseen as a complement to an RDBMS. MapReduce is a good fit for problems thatneed to analyze the whole dataset, in a batch fashion, particularly for ad hocanalysis. An RDBMS is good for point queries or updates, where the dataset hasbeen indexed to deliver low-latency retrieval and update times of a relativelysmall amount of data. MapReduce suits applications where the data is writtenonce, and read many times, whereas a relational database is good for datasetsthat are continually updated.
RDBMS compared to MapReduce

	Traditional RDBMS	MapReduce
Data size	Gigabytes	Petabytes
Access	Interactive and batch	Batch
Updates	Read and write many times	Write once, read many times
Structure	Static schema	Dynamic schema
Integrity	High	Low
Scaling	Nonlinear	Linear

 
9.      Anotherdifference between MapReduce and an RDBMS is the amount of structure in thedatasets that they operate on. MapReduce works well on unstructured orsemi-structured data, since it is designed to interpret the data at processingtime. In other words, the input keys and values for MapReduce are not anintrinsic property of the data, but they are chosen by the person analyzing thedata.
 
10.  One of thecentral assumptions that MapReduce makes is that it is possible to perform(high-speed) streaming reads and writes.
 
11.  MapReduce is alinearly scalable programming model. The programmer writes two functions—a mapfunction and a reduce function—each of which defines a mapping from one set ofkey-value pairs to another. These functions are oblivious to the size of thedata or the cluster that they are operating on, so they can be used unchangedfor a small dataset and for a massive one.
 
12.  The approach inHPC is to distributethe work across a cluster of machines, which access a shared filesystem, hostedby a SAN. This works well for predominantly compute-intensive jobs, but becomesa problem when nodes need to access larger data volumes (hundreds of gigabytes,the point at which MapReduce really starts to shine), since the networkbandwidth is the bottleneck and compute nodes become idle.
 
13.  MPI(MessagePassing Interface) gives great control to the programmer, but requires that heor she explicitly handle the mechanics of the data flow, exposed via low-levelC routines and constructs, such as sockets, as well as the higher-levelalgorithm for the analysis. MapReduce operates only at the higher level: theprogrammer thinks in terms of functions of key and value pairs, and the dataflow is implicit.