MRUNIT hadoop MapReduce逐步调试工具！

MRUnit简介

MRUnit是一款由Couldera公司开发的专门针对Hadoop中编写MapReduce单元测试的框架。可以用MapDriver单独测试Map，用ReduceDriver单独测试Reduce，用MapReduceDriver测试MapReduce作业。实战我们将利用MRUnit对本系列上篇文章MapReduce基本编程中的字数统计功能进行单元测试。加入MRUnit依赖<dependency><groupId>com.cloudera.hadoop</groupId><artifactId>hadoop-mrunit</artifactId><version>0.20.2-320</version><scope>test</scope></dependency>单独测试Mappublic class WordCountMapperTest { private Mapper mapper;private MapDriver driver;@Beforepublic void init(){ mapper = new WordCountMapper();driver = new MapDriver(mapper);}@Testpublic void test() throws IOException{ String line = "Taobao is a great website";driver.withInput(null,new Text(line)).withOutput(new Text("Taobao"),new IntWritable(1)).withOutput(new Text("is"), new IntWritable(1)).withOutput(new Text("a"), new IntWritable(1)).withOutput(new Text("great"), new IntWritable(1)).withOutput(new Text("website"), new IntWritable(1)).runTest();}}上面的例子通过MapDriver的withInput和withOutput组织map函数的输入键值和期待的输出键值，通过runTest方法运行作业，测试Map函数。测试运行通过。单独测试Reducepublic class WordCountReducerTest { private Reducer reducer;private ReduceDriver driver;@Beforepublic void init(){ reducer = new WordCountReducer();driver = new ReduceDriver(reducer);}@Testpublic void test() throws IOException{ String key = "taobao";List values = new ArrayList();

values.add(new IntWritable(2));

values.add(new IntWritable(3));driver.withInput(new Text("taobao"), values).withOutput(new Text("taobao"), new IntWritable(5)).runTest();}}上面的例子的测试Map函数的写法类似，测试reduce函数，因为reduce函数实现相加功能，因此我们假设输入为<taobao,[2,3]>，则期待结果应该为<taobao,5>.测试运行通过。测试MapReducepublic class WordCountTest { private Mapper mapper;private Reducer reducer;private MapReduceDriver driver;@Beforepublic void init(){ mapper = new WordCountMapper();reducer = new WordCountReducer();driver = new MapReduceDriver(mapper,reducer);}@Testpublic void test() throws RuntimeException, IOException{ String line = "Taobao is a great website, is it not?";driver.withInput("",new Text(line)).withOutput(new Text("Taobao"),new IntWritable(1)).withOutput(new Text("a"),new IntWritable(1)).withOutput(new Text("great"),new IntWritable(1)).withOutput(new Text("is"),new IntWritable(2)).withOutput(new Text("it"),new IntWritable(1)).withOutput(new Text("not"),new IntWritable(1)).withOutput(new Text("website"),new IntWritable(1)).runTest();}}这次我们测试MapReduce的作业，通过MapReduceDriver的withInput构造map函数的输入键值，通过withOutput构造reduce函数的输出键值。来测试这个字数统计功能，这次运行测试时抛出了异常，测试没有通过但没有详细junit异常信息，在控制台显示