正文
Mapreduce 测试自带实例 wordcount
提示：扫一扫查出行【扫一扫了解最新限行尾号】
复制提示
2.7.3版本的hadoop：
jar程序所在目录：$HADOOP_HOME/shar/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar
1.本地创建测试文本：
mkdir /home/hadoop/data //创建data文件夹
touch testinput.txt //创建测试文本
vim testinput.txt //修改文本 //文本添加一行
this is a test log cat testinput.txt //检查txt文本
2.hdfs
hadoop fs -ls / //查看hdfs上的目录
hadoop fs -mkdir /input //创建input目录
hadoop fs -rm -r /output //如果有output目录，删除
hadoop fs -put /home/hadoop/data/testiinput.txt /input //把测试文本上传到input目录上
hadoop jar /home/hadoop/softwares/hadoop-2.7./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7..jar wordcount /input /output
//执行程序,wordcount为程序的主类名， /input 输入目录 /output 输出目录（输出目录不能存在） hadoop fs -ls /output //完成后查看输出目录
hadoop fs -cat /output/part-r- //查看输出结果
运行过程：
// :: INFO client.RMProxy: Connecting to ResourceManager at hadoop/192.168.30.129:
// :: INFO input.FileInputFormat: Total input paths to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1523884458275_0001
// :: INFO impl.YarnClientImpl: Submitted application application_1523884458275_0001
// :: INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1523884458275_0001/
// :: INFO mapreduce.Job: Running job: job_1523884458275_0001
// :: INFO mapreduce.Job: Job job_1523884458275_0001 running in uber mode : false
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: Job job_1523884458275_0001 completed successfully
// :: INFO mapreduce.Job: Counters:
File System Counters
FILE: Number of bytes read=
FILE: Number of bytes written=
FILE: Number of read operations=
FILE: Number of large read operations=
FILE: Number of write operations=
HDFS: Number of bytes read=
HDFS: Number of bytes written=
HDFS: Number of read operations=
HDFS: Number of large read operations=
HDFS: Number of write operations=
Job Counters
Launched map tasks=
Launched reduce tasks=
Other local map tasks=
Total time spent by all maps in occupied slots (ms)=
Total time spent by all reduces in occupied slots (ms)=
Total time spent by all map tasks (ms)=
Total time spent by all reduce tasks (ms)=
Total vcore-milliseconds taken by all map tasks=
Total vcore-milliseconds taken by all reduce tasks=
Total megabyte-milliseconds taken by all map tasks=
Total megabyte-milliseconds taken by all reduce tasks=
Map-Reduce Framework
Map input records=
Map output records=
Map output bytes=
Map output materialized bytes=
Input split bytes=
Combine input records=
Combine output records=
Reduce input groups=
Reduce shuffle bytes=
Reduce input records=
Reduce output records=
Spilled Records=
Shuffled Maps =
Failed Shuffles=
Merged Map outputs=
GC time elapsed (ms)=
CPU time spent (ms)=
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
Shuffle Errors
BAD_ID=
CONNECTION=
IO_ERROR=
WRONG_LENGTH=
WRONG_MAP=
WRONG_REDUCE=
File Input Format Counters
Bytes Read=
File Output Format Counters
Bytes Written=
结果：
a
is
log
test
this
