学习Hadoop+spark集群错误笔记备忘

Hadoop+Spark错误i笔记：
环境：Hadoop2.9.1+Spark2.3.1+JKD1.8+centos6.9 32bit + hive2.3.3

Hadoop

Java HotSpot(TM) Client VM warning: You have loaded library /usr/local/bigdata/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

参考：https://blog.csdn.net/milhua/article/details/78711895
解决：
hadoop-env.sh和yarn-env.sh中添加如下两行：

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native  
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"

2.无论是32未机器还是64位机器都报这个错误，郁闷。。。。

18/08/06 20:05:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
stopping yarn daemons

参考：https://blog.csdn.net/l1028386804/article/details/51538611
解决：
在//usr/local/hadoop-2.5.2/etc/hadoop/log4j.properties文件中添加

log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

SPARK

1.spark on yarn

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-root/nm-local-dir/usercache/root/filecache/22/__spark_libs__1623923019181416508.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/bigdata/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

问题：jar包冲突
解决：find / -name “slf4j*”

[root@centos01 conf]# find / -name "slf4j*"
/usr/local/bigdata/hadoop/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/slf4j-log4j12-1.7.25.jar
/usr/local/bigdata/hadoop/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/slf4j-api-1.7.25.jar
/usr/local/bigdata/hadoop/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/slf4j-log4j12-1.7.25.jar
/usr/local/bigdata/hadoop/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/slf4j-api-1.7.25.jar
/usr/local/bigdata/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar
/usr/local/bigdata/hadoop/share/hadoop/common/lib/slf4j-api-1.7.25.jar
/usr/local/bigdata/spark/jars/slf4j-api-1.7.16.jar
/usr/local/bigdata/spark/jars/slf4j-log4j12-1.7.16.jar
[root@centos01 conf]# mv /usr/local/bigdata/spark/jars/slf4j-api-1.7.16.jar /usr/local/bigdata/spark/jars/slf4j-api-1.7.16.jar.bak
[root@centos01 conf]# mv /usr/local/bigdata/spark/jars/slf4j-log4j12-1.7.16.jar /usr/local/bigdata/spark/jars/slf4j-log4j12-1.7.16.jar.bak
[root@centos01 conf]# find / -name "slf4j*"
/usr/local/bigdata/hadoop/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/slf4j-log4j12-1.7.25.jar
/usr/local/bigdata/hadoop/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/slf4j-api-1.7.25.jar
/usr/local/bigdata/hadoop/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/slf4j-log4j12-1.7.25.jar
/usr/local/bigdata/hadoop/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/slf4j-api-1.7.25.jar
/usr/local/bigdata/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar
/usr/local/bigdata/hadoop/share/hadoop/common/lib/slf4j-api-1.7.25.jar
/usr/local/bigdata/spark/jars/slf4j-log4j12-1.7.16.jar.bak
/usr/local/bigdata/spark/jars/slf4j-api-1.7.16.jar.bak

java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream

原因：从spark1.4以后，所有spark的编译都是没有将hadoop的classpath编译进去的，所以必须在spark-env.sh中指定hadoop中的所有jar包。
参考： https://www.cnblogs.com/yanghuabin/p/8329205.html
解决：spark-env.sh：

export SPARK_DIST_CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath)

HIVE

Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bhive.session.id%7D_resources

解决：hive-site.xml
将$ {system:java.io.tmpdir}全部改为自定义路径，如：/home/tmpdir

接着：

hive> show tables;
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

都是版本惹的祸，谁让我总追求最新版本呢。
解决：退出hive, 执行：schematool -dbType mysql -initSchema 初始化元数据表。参考：https://blog.csdn.net/hhj724/article/details/79094138

ParseException line 3:0 cannot recognize input near 'timestamp' 'bigint' ',' in column name or primary key or foreign key

timestamp， date 字段为hive保留字，加上反斜杠即可保留，如：`date`, `timestamp` 。