コンテンツにスキップ

英文维基 | 中文维基 | 日文维基 | 草榴社区

利用者:Distart/working/MapReduce

MapReduce is a programming model for processing large data sets, and the name of an implementation of the model by Google. MapReduce is typically used to do distributed computing on clusters of computers.[1]

The model is inspired by the map and reduce functions commonly used in functional programming,[2] although their purpose in the MapReduce framework is not the same as their original forms.[3]

MapReduce libraries have been written in many programming languages. A popular free implementation is Apache Hadoop.

Overview

[編集]

MapReduce is a framework for processing embarrassingly parallel problems across huge datasets using a large number of computers (nodes), collectively referred to as a cluster (if all nodes are on the same local network and use similar hardware) or a grid (if the nodes are shared across geographically and administratively distributed systems, and use more heterogenous hardware). Computational processing can occur on data stored either in a filesystem (unstructured) or in a database (structured). MapReduce can take advantage of locality of data, processing data on or near the storage assets to decrease transmission of data.

"Map" step: The master node takes the input, divides it into smaller sub-problems, and distributes them to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker node processes the smaller problem, and passes the answer back to its master node.

"Reduce" step: The master node then collects the answers to all the sub-problems and combines them in some way to form the output – the answer to the problem it was originally trying to solve.

MapReduce allows for distributed processing of the map and reduction operations. Provided each mapping operation is independent of the others, all maps can be performed in parallel – though in practice it is limited by the number of independent data sources and/or the number of CPUs near each source. Similarly, a set of 'reducers' can perform the reduction phase - provided all outputs of the map operation that share the same key are presented to the same reducer at the same time, or if the reduction function is associative. While this process can often appear inefficient compared to algorithms that are more sequential, MapReduce can be applied to significantly larger datasets than "commodity" servers can handle – a large server farm can use MapReduce to sort a petabyte of data in only a few hours. The parallelism also offers some possibility of recovering from partial failure of servers or storage during the operation: if one mapper or reducer fails, the work can be rescheduled – assuming the input data is still available.

Logical view

[編集]

The Map and Reduce functions of MapReduce are both defined with respect to data structured in (key, value) pairs. Map takes one pair of data with a type in one data domain, and returns a list of pairs in a different domain: Map(k1,v1)list(k2,v2)

The Map function is applied in parallel to every pair in the input dataset. This produces a list of pairs for each call. After that, the MapReduce framework collects all pairs with the same key from all lists and groups them together, thus creating one group for each one of the different generated keys.

The Reduce function is then applied in parallel to each group, which in turn produces a collection of values in the same domain:

Reduce(k2, list (v2))list(v3)

Each Reduce call typically produces either one value v3 or an empty return, though one call is allowed to return more than one value. The returns of all calls are collected as the desired result list.

Thus the MapReduce framework transforms a list of (key, value) pairs into a list of values. This behavior is different from the typical functional programming map and reduce combination, which accepts a list of arbitrary values and returns one single value that combines all the values returned by map.

It is necessary but not sufficient to have implementations of the map and reduce abstractions in order to implement MapReduce. Distributed implementations of MapReduce require a means of connecting the processes performing the Map and Reduce phases. This may be a distributed file system. Other options are possible, such as direct streaming from mappers to reducers, or for the mapping processors to serve up their results to reducers that query them.

Example

[編集]

The canonical example application of MapReduce is a process to count the appearances of each different word in a set of documents:

function map(String name, String document):
  // name: document name
  // document: document contents
  for each word w in document:
    emit (w, 1)

function reduce(String word, Iterator partialCounts):
  // word: a word
  // partialCounts: a list of aggregated partial counts
  sum = 0
  for each pc in partialCounts:
    sum += pc
  emit (word, sum)

Here, each document is split into words, and each word is counted by the map function, using the word as the result key. The framework puts together all the pairs with the same key and feeds them to the same call to reduce, thus this function just needs to sum all of its input values to find the total appearances of that word.

Dataflow

[編集]

The frozen part of the MapReduce framework is a large distributed sort. The hot spots, which the application defines, are:

  • an input reader
  • a Map function
  • a partition function
  • a compare function
  • a Reduce function
  • an output writer

Input reader

[編集]

The input reader divides the input into appropriate size 'splits' (in practice typically 16 MB to 128 MB) and the framework assigns one split to each Map function. The input reader reads data from stable storage (typically a distributed file system) and generates key/value pairs.

A common example will read a directory full of text files and return each line as a record.

Map function

[編集]

Each Map function takes a series of key/value pairs, processes each, and generates zero or more output key/value pairs. The input and output types of the map can be (and often are) different from each other.

If the application is doing a word count, the map function would break the line into words and output a key/value pair for each word. Each output pair would contain the word as the key and "1" as the value.

Partition function

[編集]

Each Map function output is allocated to a particular reducer by the application's partition function for sharding purposes. The partition function is given the key and the number of reducers and returns the index of the desired reduce.

A typical default is to hash the key and modulo the number of reducers. It is important to pick a partition function that gives an approximately uniform distribution of data per shard for load balancing purposes, otherwise the MapReduce operation can be held up waiting for slow reducers to finish.

Between the map and reduce stages, the data is shuffled (parallel-sorted / exchanged between nodes) in order to move the data from the map node that produced it to the shard in which it will be reduced. The shuffle can sometimes take longer than the computation time depending on network bandwidth, CPU speeds, data produced and time taken by map and reduce computations.

Comparison function

[編集]

The input for each Reduce is pulled from the machine where the Map ran and sorted using the application's comparison function.

Reduce function

[編集]

The framework calls the application's Reduce function once for each unique key in the sorted order. The Reduce can iterate through the values that are associated with that key and produce zero or more outputs.

In the word count example, the Reduce function takes the input values, sums them and generates a single output of the word and the final sum.

Output writer

[編集]

The Output Writer writes the output of the Reduce to stable storage, usually a distributed file system.

Distribution and reliability

[編集]

MapReduce achieves reliability by parceling out a number of operations on the set of data to each node in the network. Each node is expected to report back periodically with completed work and status updates. If a node falls silent for longer than that interval, the master node (similar to the master server in the Google File System) records the node as dead and sends out the node's assigned work to other nodes. Individual operations use atomic operations for naming file outputs as a check to ensure that there are not parallel conflicting threads running. When files are renamed, it is possible to also copy them to another name in addition to the name of the task (allowing for side-effects).

The reduce operations operate much the same way. Because of their inferior properties with regard to parallel operations, the master node attempts to schedule reduce operations on the same node, or in the same rack as the node holding the data being operated on. This property is desirable as it conserves bandwidth across the backbone network of the datacenter.

Implementations are not necessarily highly-reliable. For example, in Hadoop the NameNode is a single point of failure for the distributed filesystem.

Uses

[編集]

MapReduce is useful in a wide range of applications, including distributed pattern-based searching, distributed sort, web link-graph reversal, term-vector per host, web access log stats, inverted index construction, document clustering, machine learning,[4] and statistical machine translation. Moreover, the MapReduce model has been adapted to several computing environments like multi-core and many-core systems,[5][6] desktop grids,[7] volunteer computing environments,[8] dynamic cloud environments,[9] and mobile environments.[10]

At Google, MapReduce was used to completely regenerate Google's index of the World Wide Web. It replaced the old ad hoc programs that updated the index and ran the various analyses.[11]

MapReduce's stable inputs and outputs are usually stored in a distributed file system. The transient data is usually stored on local disk and fetched remotely by the reducers.

Criticism

[編集]

David DeWitt and Michael Stonebraker, experts in parallel databases and shared-nothing architectures, have been critical of the breadth of problems that MapReduce can be used for.[12] They called its interface too low-level and questioned whether it really represents the paradigm shift its proponents have claimed it is.[13] They challenged the MapReduce proponents' claims of novelty, citing Teradata as an example of prior art that has existed for over two decades. They also compared MapReduce programmers to Codasyl programmers, noting both are "writing in a low-level language performing low-level record manipulation."[13] MapReduce's use of input files and lack of schema support prevents the performance improvements enabled by common database system features such as B-trees and hash partitioning, though projects such as Pig (or PigLatin), Sawzall, Apache Hive[14], YSmart[15], HBase[16] and BigTable[16][17] are addressing some of these problems.

Another article, by Greg Jorgensen, rejects these views.[18] Jorgensen asserts that DeWitt and Stonebraker's entire analysis is groundless as MapReduce was never designed nor intended to be used as a database.

DeWitt and Stonebraker have subsequently published a detailed benchmark study in 2009 comparing performance of Hadoop's MapReduce and RDBMS approaches on several specific problems.[19] They concluded that databases offer real advantages for many kinds of data use, especially on complex processing or where the data is used across an enterprise, but that MapReduce may be easier for users to adopt for simple or one-time processing tasks. They have published the data and code used in their study to allow other researchers to do comparable studies.

Google has been granted a patent on MapReduce.[20] However, there have been claims that this patent should not have been granted because MapReduce is too similar to existing products. For example, map and reduce functionality can be very easily implemented in Oracle's PL/SQL database oriented language.[21]

Conferences and users groups

[編集]

See also

[編集]
  • Hadoop, Apache's free and open source implementation of MapReduce.

References

[編集]

Specific references:

  1. ^ Google spotlights data center inner workings | Tech news blog - CNET News.com
  2. ^ "Our abstraction is inspired by the map and reduce primitives present in Lisp and many other functional languages." -"MapReduce: Simplified Data Processing on Large Clusters", by Jeffrey Dean and Sanjay Ghemawat; from Google Research
  3. ^ "Google's MapReduce Programming Model -- Revisited" — paper by Ralf Lämmel; from Microsoft
  4. ^ Cheng-Tao Chu; Sang Kyun Kim, Yi-An Lin, YuanYuan Yu, Gary Bradski, Andrew Ng, and Kunle Olukotun. “Map-Reduce for Machine Learning on Multicore”. NIPS 2006. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  5. ^ Colby Ranger; Ramanan Raghuraman, Arun Penmetsa, Gary Bradski, and Christos Kozyrakis. “Evaluating MapReduce for Multi-core and Multiprocessor Systems”. HPCA 2007, Best Paper. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  6. ^ Bingsheng He, Wenbin Fang, Qiong Luo, Naga K. Govindaraju, Tuyong Wang. “Mars: a MapReduce framework on graphics processors”. PACT'08. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  7. ^ Bing Tang, Moca, M., Chevalier, S., Haiwu He and Fedak, G.. “Towards MapReduce for Desktop Grid Computing”. 3PGCIC'10. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  8. ^ Heshan Lin, Xiaosong Ma, Jeremy Archuleta, Wu-chun Feng, Mark Gardner, Zhe Zhang. “MOON: MapReduce On Opportunistic eNvironments”. HPDC'10. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  9. ^ Fabrizio Marozzo, Domenico Talia, Paolo Trunfio. “A Peer-to-Peer Framework for Supporting MapReduce Applications in Dynamic Cloud Environments”. In: Cloud Computing: Principles, Systems and Applications, N. Antonopoulos, L. Gillam (Editors), chapt. 7, pp. 113–125, Springer, 2010, ISBN 978-1-84996-240-7. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  10. ^ Adam Dou, Vana Kalogeraki, Dimitrios Gunopulos, Taneli Mielikainen and Ville H. Tuulos. “Misco: a MapReduce framework for mobile systems”. HPDC'10. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  11. ^ How Google Works”. baselinemag.com. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。 “As of October, Google was running about 3,000 computing jobs per day through MapReduce, representing thousands of machine-days, according to a presentation by Dean. Among other things, these batch routines analyze the latest Web pages and update Google's indexes.”
  12. ^ Database Experts Jump the MapReduce Shark”. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  13. ^ a b David DeWitt; Michael Stonebraker. “MapReduce: A major step backwards”. craig-henderson.blogspot.com. 2008年8月27日閲覧。
  14. ^ Apache Hive - Index of - Apache Software Foundation”. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  15. ^ Rubao Lee, Tian Luo, Yin Huai, Fusheng Wang, Yongqiang He and Xiaodong Zhang. “YSmart: Yet Another SQL-to-MapReduce Translator” (PDF). Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  16. ^ a b HBase - HBase Home - Apache Software Foundation”. Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  17. ^ Bigtable: A Distributed Storage System for Structured Data” (PDF). Template:Cite webの呼び出しエラー:引数 accessdate は必須です。
  18. ^ Greg Jorgensen. “Relational Database Experts Jump The MapReduce Shark”. typicalprogrammer.com. 2009年11月11日閲覧。
  19. ^ Andrew Pavlo; E. Paulson, A. Rasin, D. J. Abadi, D. J. Dewitt, S. Madden, and M. Stonebraker. “A Comparison of Approaches to Large-Scale Data Analysis”. Brown University. 2010年1月11日閲覧。
  20. ^ US Patent 7,650,331: "System and method for efficient large-scale data processing "
  21. ^ Curt Monash. “More patent nonsense — Google MapReduce”. dbms2.com. 2010年3月7日閲覧。

General references:

[編集]
Papers
Books
Educational courses