基于阿里E-MapReduce3.2(Hbase1.2+Phoenix4.10)增量导入导出采坑实录

错误现象：

今早在EMR3.2的生产新集群上执行导出操作：

hbase org.apache.hadoop.hbase.mapreduce.Export safeclound.tb_ammeter /backup/tb_ammeter 1 1531843200000 1532016000000

2. 错误日志：

执行job任务一直卡住半天没动静，查看日志显示如下错误：

2018-07-19 17:07:26,113 INFO [main] mapreduce.Job: Task Id : attempt_1529895044112_7775_m_000002_1, Status : FAILEDError: java.io.IOException: Cannot create a record reader because of a previous error. Please look at the previous logs lines from the task's full log for more details. at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:174) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:515)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:758)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.IllegalStateException: The input format instance has not been properly initialized. Ensure you call initializeTable either in your constructor or initialize method
        at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getTable(TableInputFormatBase.java:585)
        at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:169)
        ... 8 more

然后查看hbase日志发现：

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory
....

解决方法（经联系阿里售后工程师，提供以下方案）：

1. 创建客户端使用的hbase配置

mkdir /tmp/hbase-client-conf
cp /etc/ecm/hbase-conf/* /tmp/hbase-client-conf
vim /tmp/hbase-client-conf/hbase-site.xml
# 删除phoenix相关的配置，保存：
<property>
  <name>hbase.rpc.controllerfactory.class</name>
</property>
<property>
  <name>hbase.master.loadbalancer.class</name>
</property>
<property>
  <name>hbase.region.server.rpc.scheduler.factory.class</name>
</property>
<property>
  <name>hbase.coprocessor.master.classes</name>
</property>

2. 执行命令时指定客户端配置

# Export(ClusterA)
hbase --config /tmp/hbase-client-conf org.apache.hadoop.hbase.mapreduce.Export safeclound.tb_ammeter /backup/tb_ammeter 1 1531843200000 1532016000000

# Download from hdfs(ClusterA)
hdfs dfs -get /backup/tb_ammmeter /tmp

# Scp transfer from ClusterA to ClusterB
cd /tmp; zip -r tb_ammeter.zip tb_ammeter
scp /tmp/tb_ammter.zip clusterB:/tmp

# Upload to hdfs(ClusterB)
cd /tmp; unzip tb_ammeter.zip
hdfs dfs -put /tmp/tb_ammeter /backup

# Import(ClusterB)
hbase --config /tmp/hbase-client-conf org.apache.hadoop.hbase.mapreduce.Import safeclound.tb_ammeter /backup/tb_ammeter