目录

Oozie

Oozie

1 概述

2 操作手册

2.1 oozie share lib update



hadoop fs -mkdir /user/oozie/share/lib/lib_20161202183044/spark2
hadoop fs -put /data/cloudera/parcels/SPARK2/lib/spark2/jars/*  /user/oozie/share/lib/lib_20161202183044/spark2
#少了这个步骤,汇报spark class not find
hadoop fs -cp hdfs://swc-cdh/user/oozie/share/lib/lib_20200610220322/spark/oozie-sharelib-spark-4.1.0-cdh5.15.1.jar hdfs://swc-cdh/user/oozie/share/lib/lib_20200610220322/spark2/

 hadoop fs -put /etc/spark/conf/spark-defaults.conf /user/oozie/share/lib/lib_20200610220322/spark2
 hadoop fs -put /etc/hive/conf/hive-site.xml /user/oozie/share/lib/lib_20200610220322/spark2
 
 
 hadoop fs -chown oozie:oozie /user/oozie/share/lib/lib_20200610220322/spark2/*

[bokun.wang]$ oozie admin -sharelibupdate -oozie http://xxx:11000/oozie

[ShareLib update status]
	sharelibDirOld = hdfs://cdh/user/oozie/share/lib/lib_20190320212412
	host = http://xxxx:11000/oozie
	sharelibDirNew = hdfs://cdh/user/oozie/share/lib/lib_20190320212412
	status = Successful

oozie admin -oozie http://xxxx:11000/oozie -shareliblist

[@ bokun.wang]$ oozie admin -oozie http://xxxx:11000/oozie -shareliblist
[Available ShareLib]
hive
spark2
distcp
mapreduce-streaming
spark
oozie
hcatalog
hive2
sqoop
pig

2.2 oozie share lib spark to spark2

 <property>
    <name>oozie.action.sharelib.for.spark</name>
    <value>spark2</value>
</property>

hadoop fs -put spark2 /user/oozie/share/lib/lib_20190320212412/
hadoop fs -chown -R  oozie:oozie  /user/oozie/share/lib/lib_20190320212412/spark2

#然后执行2.1中的share lib update command 或者 restart oozie

2.3 oozie web console ext 2.2

wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip

creates=/var/lib/oozie/ext-2.2 unzip ./ext-2.2.zip -d /var/lib/oozie/
chown oozie:oozie -R /var/lib/oozie/ext-2.2

ll /data/cloudera/parcels/CDH/lib/oozie/

lrwxrwxrwx 1 root root    14 Aug 10  2018 libext -> /var/lib/oozie

Done.

##Q&&A

1 ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain

要让一个workflow使用ShareLib,只需要在job.properties里面加入oozie.use.system.libpath=true就行啦~
前提是oozie share lib中有对一个的版本目录,如果没有需要加一下哦,比如我上面的spark2,虽然是同样的错,但是实际上是jar包不存在,首先保证有jar在设置这个配置

#加上这一步
hadoop fs -cp hdfs://swc-cdh/user/oozie/share/lib/lib_20200610220322/spark/oozie-sharelib-spark-4.1.0-cdh5.15.1.jar hdfs://swc-cdh/user/oozie/share/lib/lib_20200610220322/spark2/