阿里 EMR3.2 Hbase 客户端 BUG

1. 错误现象

今早在 EMR3.2 的生产新集群上执行导出操作:

hbase org.apache.hadoop.hbase.mapreduce.Export safeclound.tb_ammeter /backup/tb_ammeter 1 1531843200000 1532016000000

2. 错误表现

• 执行 (RowCounter/Export/Import/...) MapReduce Job 时一直卡住半天没动静，查看日志显示如下错误

2018-07-19 17:07:26,113 INFO [main] mapreduce.Job: Task Id : attempt_1529895044112_7775_m_000002_1, Status : FAILEDError: java.io.IOException: Cannot create a record reader because of a previous error. Please look at the previous logs lines from the task\'s full log for more details. at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:174) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:515)*
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:758)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.IllegalStateException: The input format instance has not been properly initialized. Ensure you call initializeTable either in your constructor or initialize method
 at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getTable(TableInputFormatBase.java:585)
 at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:169)
...
8 more

• 然后查看日志

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory*
...

• 解决方法（经联系阿里工程师，提供以下方案）

• 创建客户端使用的hbase配置

mkdir /tmp/hbase-client-conf
cp /etc/ecm/hbase-conf/* /tmp/hbase-client-conf
vim /tmp/hbase-client-conf/hbase-site.xml

• 删除 phoenix 相关的配置，保存：

hbase.rpc.controllerfactory.class
hbase.master.loadbalancer.class
hbase.region.server.rpc.scheduler.factory.class
hbase.coprocessor.master.classes

• 手动指定使用新的临时客户端配置

hbase --config /tmp/hbase-client-conf org.apache.hadoop.hbase.mapreduce.Export safeclound.tb_ammeter /backup/tb_ammeter *1 1531843200000 1532016000000

3. 原因分析

从日志也可看出，是因为 EMR(hbase ) 默认配置了部分扩展处理类，但执行 hbase 命令行时没有加载到这些类，由于这些类不影响 MR(yarn) 执行，因此简单的做法就是拷贝到临时目录删掉配置的这些类，然后指定临时配置目录执行.