解决Hue/hiveserver2报错：java.io.IOException: Job status not available

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

sql是：select count(distinct col) from db.table;

排查过程中遇到过几个不同的报错：

1. beeline -u jdbc:hive2://0.0.0.0:10000 -e "select count(distinct col) from db.table;"

INFO  : Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1494385775332_0822

ERROR : Ended Job = job_1494385775332_0822 with exception 'java.io.IOException(Job status not available )'

java.io.IOException: Job status not available

        at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:333)

        at org.apache.hadoop.mapreduce.Job.getJobState(Job.java:358)

        at org.apache.hadoop.mapred.JobClient$NetworkedJob.getJobState(JobClient.java:295)

        at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:244)

        at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549)

        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437)

        at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)

        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)

        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)

        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)

....

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1)

Diagnostics:	Staging dir does not exist /mr-history/anonymous/.staging

2. beeline -u jdbc:hive2://0.0.0.0:10000 -n userxxx -e "select count(distinct col) from db.table;"

ERROR : Ended Job = job_1494385775332_0823 with exception 'java.io.IOException(java.io.IOException: Unknown Job job_1494385775332_0823

        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:218)

        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:281)

        at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)

        at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

)'

java.io.IOException: java.io.IOException: Unknown Job job_1494385775332_0823

Diagnostics:	Job init failed : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://xxx/mr-history/userxxx/.staging/job_1494385775332_0823/job.splitmetainfo

使用不同的用户执行还是报一样的错误。

3. hive -e "select count(distinct col) from db.table;" 却能执行成功

解决：

根据第二步Diagnostics的报错信息搜索到一篇文章：https://community.hortonworks.com/questions/17489/job-init-fail-job-splitmetainfo-file-does-not-exis.html