189 8069 5689
本篇内容主要讲解“Hadoop如何打包和运行MapReduce程序”，感兴趣的朋友不妨来看看。本文介绍的方法操作简单快捷，实用性强。下面就让小编来带大家学习“Hadoop如何打包和运行MapReduce程序”吧!
创新互联主要从事网站设计制作、成都网站制作、网页设计、企业做网站、公司建网站等业务。立足成都服务安平,10年网站建设经验,价格优惠、服务专业,欢迎来电咨询建站服务:18980820575
主要内容：将 MapReduce 代码通过命令行打包成 jar 包，然后提交给 Hadoop 集群运行。示例的 WordCount.java、WordCount.txt 见最后面。
一、编译 Hadoop 的应用程序需要将所需的依赖包添加到 CLASSPATH，可以添加到 .bashrc 或者 /etc/profile。
# javac 编译相关包依赖 HADOOP_CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath) # 将 HADOOP_CLASSPATH 添加到 CLASSPATH export CLASSPATH=.:$HADOOP_CLASSPATH:$CLASSPATH
二、编译源代码
# 编译 没有设置 CLASSPATH 通过 -cp $($HADOOP_HOME/bin/hadoop classpath) javac WordCount.java # 打包 jar -cvf WordCount.jar *.class
三、提交到 Hadoop
# 上传 WordCount.txt 到 Hadoop hdfs dfs -mkdir input hdfs dfs -put WordCount.txt input # 提交任务 jar 包、main 所在的类、输入文件夹、输出文件夹 hadoop jar WordCount.jar WordCount input output # 查看运行结果 hdfs dfs -cat output/* # 删除输出结果目录 hdfs dfs -rm -r output
四、运行结果
and 1 bigdata 2 hadoop 2 hello 4 world 1
附录：
WordCount.txt，单词使用空格分隔
hello world hello hadoop hello bigdata hello hadoop and bigdata
WordCount.java
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordCount {
public static class TokenizerMapper extends Mapper {
private final static IntWritable ONE = new IntWritable(1);
private final Text word = new Text();
@Override
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, ONE);
}
}
}
public static class IntSumReducer extends Reducer {
private final IntWritable result = new IntWritable();
@Override
public void reduce(Text key, Iterable values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "WordCount");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
} 到此，相信大家对“Hadoop如何打包和运行MapReduce程序”有了更深的了解，不妨来实际操作一番吧！这里是创新互联网站，更多相关内容可以进入相关频道进行查询，关注我们，继续学习！