自定义 NameNode Xmx等参数

参考1: 如何优雅设置hadoop相关组件的jvm Xmx

参考2: 合理设置Hdfs Xmx

总结

• 不建议直接修改hadoop-env.sh的HADOOP_HEAPSIZE，因为这样会导致其他组件如hive也被修改了（有些组件像yarn是有自己的变量的如YARN_HEAPSIZE），应该让各个组件参数独立，因此参考阿里的EMR这样设置：hadoop-env.sh结尾处增加:

###
# Advanced Users Only!
###
# ...
# ...

# --- BEGIN ADD ---
# Refer to Aliyun-EMR and add $Hadoop during startup $HEAPSIZE='HADOOP_NAMENODE_HEAPSIZE' to identify the 
# startup arg 'HADOOP_HEAPSIZE' Who does heapsize belong to, or dynamically identify whether the current 
# node matches the namenode according to the configuration. The setting cannot be written, because other 
# components such as NameNode/DataNode/HiveServer/etc will also be affected, this violates the principle 
# of component configuration isolation.
# see: https://blogs.wl4g.com/archives/1212
export HADOOP_NAMENODE_HEAPSIZE=2560
export HADOOP_DATANODE_HEAPSIZE=1152
#export HADOOP_SECONDARY_NAMENODE_HEAPSIZE=1024

# Gets and checking this host whether is namenodes/datanodes. 
# @Deprecated due to poor performance.
#NAMENODES=$($JAVA_HOME/bin/java -cp $HADOOP_PREFIX/etc/hadoop:$HADOOP_PREFIX/share/hadoop/common/lib/*:$HADOOP_PREFIX/share/hadoop/common/*:$HADOOP_PREFIX/share/hadoop/hdfs:$HADOOP_PREFIX/share/hadoop/hdfs/lib/*:$HADOOP_PREFIX/share/hadoop/hdfs/*:$HADOOP_PREFIX/share/hadoop/yarn/lib/*:$HADOOP_PREFIX/share/hadoop/yarn/*:$HADOOP_PREFIX/share/hadoop/mapreduce/lib/*:$HADOOP_PREFIX/share/hadoop/mapreduce/*:$HADOOP_PREFIX/contrib/capacity-scheduler/*.jar org.apache.hadoop.hdfs.tools.GetConf -namenodes 2>/dev/null)
#IS_NAMENODE=$([ -z "$(echo $NAMENODES|grep `hostname`)" ] && echo N || echo Y)
#[ "$IS_NAMENODE" == "$IS_DATANODE" ] && $(echo "Shouldn't be here"; exit 2) # Check

IS_DATANODE=$([ -z "$(cat $HADOOP_CONF_DIR/slaves|grep `hostname`)" ] && echo N || echo Y)
if [[ "$HADOOP_HEAPSIZE" == "HADOOP_NAMENODE_HEAPSIZE" || "$IS_DATANODE" == "N" ]]; then
  export HADOOP_HEAPSIZE=$HADOOP_NAMENODE_HEAPSIZE
fi
if [[ "$HADOOP_HEAPSIZE" == "HADOOP_DATANODE_HEAPSIZE" || "$IS_DATANODE" == "Y" ]]; then
  export HADOOP_HEAPSIZE=$HADOOP_DATANODE_HEAPSIZE
fi
# --- END ADD ---