在CentOS6.9安装Azkaban3.50.0_centos6.9 azkaban-程序员宅基地
Azkaban简介
上图是Azkaban官网提供的架构图，包括三个组件：
Relational Database (MySQL)
AzkabanWebServer
AzkabanExecutorServer
组件分工：
元数据库：用于维护任务流信息以及执行情况信息。
Web服务器：用户交互的门户，同时也是管理任务流、工程，控制调度，任务分配的核心组件。
执行服务器：任务被真正调起的服务器，可扩展。
AzkabanWebServer对数据库的使用：
项目管理
追踪任务流的状态
查看任务执行结果以及历史日志
调度任务
sla
AzkabanExecutorServer 对数据库的使用：
从数据库获取工程的信息
从数据库获取要执行的任务流
储存任务流和任务的日志信息
从数据库获取内置任务流的任务依赖关系
一、环境
操作系统：CentOS6.9
软件版本：Azkaban3.50.0
二、安装
我这里是将Azkaban安装在hadoop用户，所以所有的linux操作都在hadoop用户。
1、编译Azkaban可执行文件
1.1 下载安装包
cd /data/install_pkg
wget https://github.com/azkaban/azkaban/archive/3.50.0.tar.gz
解压安装包：
tar -zxvf 3.50.0.tar.gz
1.2 修改邮件代码
vi /data/install_pkg/azkaban-3.50.0/azkaban-common/src/main/java/azkaban/utils/EmailMessage.java
找到邮件参数配置的地方
:/props
增加以下语句：
props.put("mail.smtp.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
1.3 安装git
因为使用gradle编译会用到git，所以要安装：
yum install git
1.4 安装JCE
如果不安装JCE，后面编译的时候会报错。安装方法见：
https://blog.csdn.net/andyguan01_2/article/details/88343641
1.5 编译文件
使用Gradle编译文件。Gradle是一个基于Apache Ant和Apache Maven的项目自动化构建工具。
cd /data/install_pkg/azkaban-3.50.0
./gradlew build installDist
编译报错，看着像是测试过程中出错，没找具体到是什么原因。
Starting a Gradle Daemon (subsequent builds will be faster)
Parallel execution with configuration on demand is an incubating feature.
> Task :az-core:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :azkaban-common:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :azkaban-web-server:npmSetup
/data/install_pkg/azkaban-3.50.0/azkaban-web-server/.gradle/npm/npm-v5.6.0/bin/npm -> /data/install_pkg/azkaban-3.50.0/azkaban-web-server/.gradle/npm/npm-v5.6.0/lib/node_modules/npm/bin/npm-cli.js
/data/install_pkg/azkaban-3.50.0/azkaban-web-server/.gradle/npm/npm-v5.6.0/bin/npx -> /data/install_pkg/azkaban-3.50.0/azkaban-web-server/.gradle/npm/npm-v5.6.0/lib/node_modules/npm/bin/npx-cli.js
+ [email protected]
added 476 packages in 14.414s
> Task :azkaban-exec-server:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :azkaban-common:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :azkaban-hadoop-security-plugin:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /data/install_pkg/azkaban-3.50.0/azkaban-hadoop-security-plugin/src/main/java/azkaban/security/HadoopSecurityManager_H_2_0.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :azkaban-common:test
azkaban.utils.UtilsTest > testValidCronExpressionV FAILED
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)EmailMessageCreatorTest
at org.junit.Assert.assertTrue(Assert.java:52)
at azkaban.utils.UtilsTest.testValidCronExpressionV(UtilsTest.java:63)
> Task :azkaban-web-server:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :azkaban-exec-server:compileTestJava
Note: /data/install_pkg/azkaban-3.50.0/azkaban-exec-server/src/test/java/azkaban/execapp/AzkabanExecutorServerTest.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
> Task :azkaban-web-server:npm_install
added 39 packages in 3.23s
> Task :azkaban-common:test
328 tests completed, 1 failed, 18 skipped
> Task :azkaban-solo-server:compileJava
Note: /data/install_pkg/azkaban-3.50.0/azkaban-solo-server/src/main/java/azkaban/soloserver/AzkabanSingleServer.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
> Task :az-hdfs-viewer:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
> Task :az-reportal:compileJava
Note: /data/install_pkg/azkaban-3.50.0/az-reportal/src/main/java/azkaban/reportal/util/StreamProviderHDFS.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
> Task :az-hadoop-jobtype-plugin:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /data/install_pkg/azkaban-3.50.0/az-hadoop-jobtype-plugin/src/main/java/azkaban/jobtype/HadoopSecureSparkWrapper.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':azkaban-common:test'.
> There were failing tests. See the report at: file:///data/install_pkg/azkaban-3.50.0/azkaban-common/build/reports/tests/test/index.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 2m 4s
71 actionable tasks: 66 executed, 5 from cache
于是改为使用命令./gradlew build installDist -x test，-x test表示跳过测试。
./gradlew clean
./gradlew build installDist -x test
1.6 复制可执行文件到安装目录
编译完成后，我们就可以获取到web，exec，db，solo的可执行文件tar包，拷贝到Azkaban安装目录：
ll /data/install_pkg/azkaban-3.50.0/azkaban-*/build/distributions/*.tar.gz
cp /data/install_pkg/azkaban-3.50.0/azkaban-*/build/distributions/*.tar.gz /u01/app/azkaban-3.50.0
ll /u01/app/azkaban-3.50.0
解压tar包到Azkaban安装目录：
cd /u01/app/azkaban-3.50.0
tar -zxvf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz -C /u01/app/azkaban-3.50.0
tar -zxvf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz -C /u01/app/azkaban-3.50.0
tar -zxvf azkaban-db-0.1.0-SNAPSHOT.tar.gz -C /u01/app/azkaban-3.50.0
tar -zxvf azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz -C /u01/app/azkaban-3.50.0
2、初始化MySQL
初始化MySQL的库、用户、权限、表、数据：
mysql -u root -p
CREATE DATABASE azkaban;
CREATE USER 'azkaban'@'%' IDENTIFIED BY '填实际密码';
GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;
flush privileges;
use azkaban;
source /u01/app/azkaban-3.50.0/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql;
3、配置SSL
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
要输入密码的地方，都填相同密码。几个问题的回答，what … country code … ?填CN，Is … correct?填y，其他问题可以直接回车不填。。
完成上述工作后，将在当前目录生成 keystore 证书文件，将keystore移动到azkaban web服务器根目录中。
mv keystore /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT
4、配置web-server
因为web-server和exec-server都没有默认配置文件，我们需要复制solo的配置文件。
cd /u01/app/azkaban-3.50.0
cp -r azkaban-solo-server-0.1.0-SNAPSHOT/conf azkaban-web-server-0.1.0-SNAPSHOT
cp /data/install_pkg/azkaban-3.50.0/azkaban-web-server/src/test/resources/log4j.properties azkaban-web-server-0.1.0-SNAPSHOT/conf
4.1 配置azkaban.properties
vi /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban.properties
配置以下内容：
# Azkaban Personalization Settings
#服务器UI名称,用于服务器上方显示的名字
azkaban.name=Test
#描述
azkaban.label=My Local Azkaban
#UI颜色
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
#默认根web目录建议最好写绝对路径，以免出现找不到文件错误
web.resource.dir=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/web
#默认时区，已改为亚洲/上海，默认为美国
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
#用户权限管理默认类
user.manager.class=azkaban.user.XmlUserManager
#用户配置文件所在位置
user.manager.xml.file=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban-users.xml
# Loader for projects
# global配置文件所在位置
executor.global.properties=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/conf/global.properties
azkaban.project.dir=projects
#数据库类型
database.type=mysql
#端口号
mysql.port=3306
#数据库连接IP
mysql.host=填实际IP
#数据库名
mysql.database=azkaban
#数据库用户名
mysql.user=azkaban
#数据库密码
mysql.password=填实际密码
#最大连接数
mysql.numconnections=100
# Azkaban Jetty server properties.
# Jetty服务器属性.
velocity.dev.mode=false
#最大线程数
jetty.maxThreads=25
#Jetty SSL端口
jetty.ssl.port=8443
#Jetty端口
jetty.port=8081
#SSL文件名
jetty.keystore=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/keystore
#SSL文件密码
jetty.password=填实际密码
#Jetty主密码 与 keystore文件相同
jetty.keypassword=填实际密码
#SSL文件名
jetty.truststore=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/keystore
# SSL文件密码
jetty.trustpassword=填实际密码
#jetty.use.ssl=false
jetty.maxThreads=25
# Azkaban Executor settings
executor.port=12321
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy
between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/plugins/jobtypes
4.2 配置commonprivate.properties
先创建配置文件：
mkdir -p /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/plugins/jobtypes
vi /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/plugins/jobtypes/commonprivate.properties
输入以下内容：
azkaban.native.lib=false
execute.as.user=false
4.3 配置azkaban-users.xml
修改用户配置文件azkaban-users.xml：
vi /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban-users.xml
增加admin用户：（在第4行）
<azkaban-users>
<user groups="azkaban" password="填实际密码" roles="admin" username="azkaban"/>
<user password="填实际密码" roles="metrics" username="metrics"/>
<user username="admin" password="填实际密码" roles="admin,metrics"/>
<role name="admin" permissions="ADMIN"/>
<role name="metrics" permissions="METRICS"/>
</azkaban-users>
5、配置executor
将 azkaban-web-server-0.1.0-SNAPSHOT目录下的conf拷贝到 azkaban-exec-server-0.1.0-SNAPSHOT
cd /u01/app/azkaban-3.50.0
cp -r azkaban-web-server-0.1.0-SNAPSHOT/conf azkaban-exec-server-0.1.0-SNAPSHOT
cp -r azkaban-web-server-0.1.0-SNAPSHOT/plugins azkaban-exec-server-0.1.0-SNAPSHOT
6、启动服务
6.1 启动Executor Server
进入azkaban-exec-server-0.1.0-SNAPSHOT下启动（注意不要进入bin目录启动）
cd /u01/app/azkaban-3.50.0/azkaban-exec-server-0.1.0-SNAPSHOT
./bin/start-exec.sh
会在当前目录生成.out的日志文件：
ll /u01/app/azkaban-3.50.0/azkaban-exec-server-0.1.0-SNAPSHOT/*.out
more /u01/app/azkaban-3.50.0/azkaban-exec-server-0.1.0-SNAPSHOT/*.out
日志内容为：
Using Hadoop from /u01/app/hadoop
Using Hive from /u01/app/apache-hive-2.3.4-bin
./bin/internal/../..
.:/usr/java/jdk1.8.0_121/lib/dt.jar:/usr/java/jdk1.8.0_121/lib/tools.jar:/u01/app/hadoop-2.6.5/etc/hadoop:/u01/app/hadoop/share/hado
op/common/lib/*:/u01/app/hadoop/share/hadoop/common/*:/u01/app/hadoop/share/hadoop/hdfs:/u01/app/hadoop/share/hadoop/hdfs/lib/*:/u01
/app/hadoop/share/hadoop/hdfs/*:/u01/app/hadoop-2.6.5/share/hadoop/yarn/lib/*:/u01/app/hadoop-2.6.5/share/hadoop/yarn/*:/u01/app/had
oop/share/hadoop/mapreduce/lib/*:/u01/app/hadoop/share/hadoop/mapreduce/*:/u01/app/hadoop/contrib/capacity-scheduler/*.jar:./bin/int
ernal/../../lib/activation-1.1.jar:./bin/internal/../../lib/aopalliance-1.0.jar:./bin/internal/../../lib/az-core-0.1.0-SNAPSHOT.jar:
./bin/internal/../../lib/azkaban-common-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/azkaban-db-0.1.0-SNAPSHOT.jar:./bin/internal/../
../lib/azkaban-exec-server-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/azkaban-hadoop-security-plugin-0.1.0-SNAPSHOT.jar:./bin/inter
nal/../../lib/azkaban-spi-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/c3p0-0.9.1.1.jar:./bin/internal/../../lib/commons-codec-1.9.ja
r:./bin/internal/../../lib/commons-collections-3.2.2.jar:./bin/internal/../../lib/commons-dbcp2-2.1.1.jar:./bin/internal/../../lib/c
ommons-dbutils-1.5.jar:./bin/internal/../../lib/commons-fileupload-1.2.1.jar:./bin/internal/../../lib/commons-io-2.4.jar:./bin/inter
nal/../../lib/commons-jexl-2.1.1.jar:./bin/internal/../../lib/commons-lang-2.6.jar:./bin/internal/../../lib/commons-logging-1.2.jar:
./bin/internal/../../lib/commons-math3-3.0.jar:./bin/internal/../../lib/commons-pool2-2.4.2.jar:./bin/internal/../../lib/gson-2.8.1.
jar:./bin/internal/../../lib/guava-21.0.jar:./bin/internal/../../lib/guice-4.1.0.jar:./bin/internal/../../lib/httpclient-4.5.3.jar:.
/bin/internal/../../lib/httpcore-4.4.6.jar:./bin/internal/../../lib/jackson-core-asl-1.9.5.jar:./bin/internal/../../lib/jackson-mapp
er-asl-1.9.5.jar:./bin/internal/../../lib/javax.inject-1.jar:./bin/internal/../../lib/jetty-6.1.26.jar:./bin/internal/../../lib/jett
y-util-6.1.26.jar:./bin/internal/../../lib/joda-time-2.0.jar:./bin/internal/../../lib/jopt-simple-4.3.jar:./bin/internal/../../lib/j
sr305-3.0.2.jar:./bin/internal/../../lib/kafka-clients-0.10.0.0.jar:./bin/internal/../../lib/kafka-log4j-appender-0.10.0.0.jar:./bin
/internal/../../lib/log4j-1.2.17.jar:./bin/internal/../../lib/lz4-1.3.0.jar:./bin/internal/../../lib/mail-1.4.5.jar:./bin/internal/.
./../lib/metrics-core-3.1.0.jar:./bin/internal/../../lib/metrics-jvm-3.1.0.jar:./bin/internal/../../lib/mysql-connector-java-5.1.28.
jar:./bin/internal/../../lib/quartz-2.2.1.jar:./bin/internal/../../lib/servlet-api-2.5-20081211.jar:./bin/internal/../../lib/slf4j-a
pi-1.7.21.jar:./bin/internal/../../lib/slf4j-log4j12-1.7.21.jar:./bin/internal/../../lib/snakeyaml-1.18.jar:./bin/internal/../../lib
/snappy-java-1.1.2.4.jar:./bin/internal/../../lib/velocity-1.7.jar:./bin/internal/../../extlib/*.jar:./bin/internal/../../plugins/*/
*.jar:/u01/app/hadoop/conf:/u01/app/hadoop/*:/u01/app/apache-hive-2.3.4-bin/conf:/u01/app/apache-hive-2.3.4-bin/lib/*
Starting AzkabanExecutorServer on port 12321 ...
2019/03/08 15:01:08.059 +0800 INFO [AzkabanExecutorServer] Starting Jetty Azkaban Executor...
2019/03/08 15:01:08.084 +0800 INFO [AzkabanServer] Loading azkaban settings file from ./bin/internal/../../conf
2019/03/08 15:01:08.084 +0800 INFO [AzkabanServer] Loading azkaban properties file
2019/03/08 15:01:08.094 +0800 ERROR [StdOutErrRedirect] SLF4J: Class path contains multiple SLF4J bindings.
2019/03/08 15:01:08.094 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/hadoop-2.6.5/share/hadoop/common/
lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:01:08.095 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/azkaban-3.50.0/azkaban-exec-serve
r-0.1.0-SNAPSHOT/lib/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:01:08.095 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/apache-hive-2.3.4-bin/lib/log4j-s
lf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:01:08.095 +0800 ERROR [StdOutErrRedirect] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanat
ion.
2019/03/08 15:01:08.097 +0800 ERROR [StdOutErrRedirect] SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2019/03/08 15:01:08.607 +0800 WARN [StorageCleaner] Project cleanup disabled. All artifacts will be stored.
2019/03/08 15:01:08.608 +0800 INFO [AzkabanExecServerModule] Event reporter is not enabled
2019/03/08 15:01:08.608 +0800 INFO [FlowRunnerManager] Execution dir retention set to 86400000 ms
2019/03/08 15:01:08.609 +0800 INFO [FlowRunnerManager] useNewThreadPool: false
2019/03/08 15:01:08.611 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:01:08.611 +0800 INFO [FlowRunnerManager] Cleaning recently finished
2019/03/08 15:01:08.612 +0800 INFO [FlowRunnerManager] Cleaning old execution dirs
2019/03/08 15:01:08.613 +0800 INFO [JobTypeManager] Loading plugin default job types
2019/03/08 15:01:08.616 +0800 INFO [JobTypeManager] Job type plugin directory set. Loading extra job types from /u01/app/azkaban-3.5
0.0/azkaban-web-server-0.1.0-SNAPSHOT/plugins/jobtypes
2019/03/08 15:01:08.616 +0800 INFO [JobTypeManager] Common plugin job props file /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SN
APSHOT/plugins/jobtypes/common.properties not found. Using only globals props
2019/03/08 15:01:08.616 +0800 INFO [JobTypeManager] Common plugin load props file /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-S
NAPSHOT/plugins/jobtypes/commonprivate.properties found. Attempt to load.
2019/03/08 15:01:08.626 +0800 INFO [log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2019/03/08 15:01:08.628 +0800 INFO [ExecJettyServerModule] Setting up connector with stats on: true
2019/03/08 15:01:08.628 +0800 INFO [ExecJettyServerModule] Jetty connector name: 0.0.0.0:12321, default header buffer size: 4096
2019/03/08 15:01:08.628 +0800 INFO [ExecJettyServerModule] Jetty connector name: 0.0.0.0:12321, (if) new header buffer size: 4096
2019/03/08 15:01:08.663 +0800 INFO [JmxJobMBeanManager] Initializing azkaban.execapp.jmx.JmxJobMBeanManager
2019/03/08 15:01:08.663 +0800 INFO [AzkabanExecutorServer] Job callback enabled? true
2019/03/08 15:01:08.665 +0800 INFO [JobCallbackManager] Initializing
2019/03/08 15:01:08.667 +0800 INFO [JobCallbackRequestMaker] responseWaitTimeoutMS: 5000
2019/03/08 15:01:08.668 +0800 INFO [JobCallbackRequestMaker] Global request configuration [expectContinueEnabled=false, proxy=null,
localAddress=null, cookieSpec=null, redirectsEnabled=true, relativeRedirectsAllowed=true, maxRedirects=50, circularRedirectsAllowed=
false, authenticationEnabled=true, targetPreferredAuthSchemes=null, proxyPreferredAuthSchemes=null, connectionRequestTimeout=3000, c
onnectTimeout=3000, socketTimeout=3000, contentCompressionEnabled=true]
2019/03/08 15:01:08.942 +0800 INFO [JobCallbackRequestMaker] Jobcall thread pool size: 10
2019/03/08 15:01:08.943 +0800 INFO [JobCallbackRequestMaker] Initialization for azkaban.execapp.event.JobCallbackRequestMaker is com
pleted
2019/03/08 15:01:08.944 +0800 INFO [JobCallbackManager] Initialization completed azkaban.execapp.event.JobCallbackManager
2019/03/08 15:01:08.944 +0800 INFO [JobCallbackManager] azkabanHostName oracle02.auditonline.prd.df.cn
2019/03/08 15:01:08.945 +0800 INFO [AzkabanExecutorServer] Registering MBeans...
2019/03/08 15:01:08.953 +0800 INFO [AzkabanExecutorServer] Bean azkaban.jmx.JmxJettyServer registered.
2019/03/08 15:01:08.956 +0800 INFO [AzkabanExecutorServer] Bean azkaban.execapp.jmx.JmxFlowRunnerManager registered.
2019/03/08 15:01:08.957 +0800 INFO [AzkabanExecutorServer] Bean azkaban.execapp.jmx.JmxJobMBeanManager registered.
2019/03/08 15:01:08.958 +0800 INFO [AzkabanExecutorServer] Bean azkaban.execapp.jmx.JmxJobCallback registered.
2019/03/08 15:01:08.958 +0800 INFO [AzkabanExecutorServer] No value for property: jmx.attribute.processor.class was found
2019/03/08 15:01:08.958 +0800 INFO [log] jetty-6.1.26
2019/03/08 15:01:08.992 +0800 INFO [log] Started [email protected]:12321
2019/03/08 15:01:09.792 +0800 INFO [AzkabanExecutorServer] Started Executor Server on oracle02.auditonline.prd.df.cn:12321
2019/03/08 15:01:09.831 +0800 INFO [AzkabanExecutorServer] Setting timezone to Asia/Shanghai
2019/03/08 15:02:08.614 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:03:08.614 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:03:08.615 +0800 INFO [FlowRunnerManager] Cleaning recently finished
2019/03/08 15:04:08.615 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:05:08.615 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:05:08.616 +0800 INFO [FlowRunnerManager] Cleaning recently finished
2019/03/08 15:06:08.616 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:07:08.616 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:07:08.617 +0800 INFO [FlowRunnerManager] Cleaning recently finished
查看JPS，会出现AzkabanExecutorServer：
jps
6.2 启动Web Server
进入azkaban-web-server-0.1.0-SNAPSHOT下启动（注意不要进入bin目录启动）
cd /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT
./bin/start-web.sh
会在当前目录生成.out的日志文件：
ll /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/*.out
more /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/*.out
我这边启动后报错：
java.lang.NoSuchMethodError:com.google.common.collect.ImmutableMap.toImmutableMap
解决方法见：
https://blog.csdn.net/andyguan01_2/article/details/88353889
问题解决后，重新启动Web Server，日志内容为：
Using Hadoop from /u01/app/hadoop
Using Hive from /u01/app/apache-hive-2.3.4-bin
./bin/internal/../..
.:/usr/java/jdk1.8.0_121/lib/dt.jar:/usr/java/jdk1.8.0_121/lib/tools.jar:/u01/app/hadoop-2.6.5/etc/hadoop:/u01/app/hadoop/share/hado
op/common/lib/*:/u01/app/hadoop/share/hadoop/common/*:/u01/app/hadoop/share/hadoop/hdfs:/u01/app/hadoop/share/hadoop/hdfs/lib/*:/u01
/app/hadoop/share/hadoop/hdfs/*:/u01/app/hadoop-2.6.5/share/hadoop/yarn/lib/*:/u01/app/hadoop-2.6.5/share/hadoop/yarn/*:/u01/app/had
oop/share/hadoop/mapreduce/lib/*:/u01/app/hadoop/share/hadoop/mapreduce/*:/u01/app/hadoop/contrib/capacity-scheduler/*.jar:./bin/int
ernal/../../lib/activation-1.1.jar:./bin/internal/../../lib/antlr-2.7.2.jar:./bin/internal/../../lib/aopalliance-1.0.jar:./bin/inter
nal/../../lib/az-core-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/az-flow-trigger-dependency-plugin-0.1.0-SNAPSHOT.jar:./bin/interna
l/../../lib/azkaban-common-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/azkaban-db-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/azkaba
n-spi-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/azkaban-web-server-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/c3p0-0.9.1.1.jar:./
bin/internal/../../lib/cglib-nodep-2.2.jar:./bin/internal/../../lib/commons-beanutils-1.7.0.jar:./bin/internal/../../lib/commons-cha
in-1.1.jar:./bin/internal/../../lib/commons-codec-1.9.jar:./bin/internal/../../lib/commons-collections-3.2.2.jar:./bin/internal/../.
./lib/commons-compress-1.2.jar:./bin/internal/../../lib/commons-dbcp2-2.1.1.jar:./bin/internal/../../lib/commons-dbutils-1.5.jar:./b
in/internal/../../lib/commons-digester-1.8.jar:./bin/internal/../../lib/commons-fileupload-1.2.1.jar:./bin/internal/../../lib/common
s-io-2.4.jar:./bin/internal/../../lib/commons-jexl-2.1.1.jar:./bin/internal/../../lib/commons-lang-2.6.jar:./bin/internal/../../lib/
commons-logging-1.2.jar:./bin/internal/../../lib/commons-math3-3.0.jar:./bin/internal/../../lib/commons-pool2-2.4.2.jar:./bin/intern
al/../../lib/commons-validator-1.3.1.jar:./bin/internal/../../lib/data-1.15.7.jar:./bin/internal/../../lib/data-transform-1.15.7.jar
:./bin/internal/../../lib/dom4j-1.1.jar:./bin/internal/../../lib/gson-2.8.1.jar:./bin/internal/../../lib/guava-21.0.jar:./bin/intern
al/../../lib/guice-4.1.0.jar:./bin/internal/../../lib/httpclient-4.5.3.jar:./bin/internal/../../lib/httpcore-4.4.6.jar:./bin/interna
l/../../lib/jackson-core-2.2.2.jar:./bin/internal/../../lib/jackson-core-asl-1.9.5.jar:./bin/internal/../../lib/jackson-mapper-asl-1
.9.5.jar:./bin/internal/../../lib/javax.inject-1.jar:./bin/internal/../../lib/javax.servlet-api-3.0.1.jar:./bin/internal/../../lib/j
etty-6.1.26.jar:./bin/internal/../../lib/jetty-util-6.1.26.jar:./bin/internal/../../lib/joda-time-2.0.jar:./bin/internal/../../lib/j
opt-simple-4.3.jar:./bin/internal/../../lib/json-20070829.jar:./bin/internal/../../lib/li-jersey-uri-1.15.7.jar:./bin/internal/../..
/lib/log4j-1.2.17.jar:./bin/internal/../../lib/mail-1.4.5.jar:./bin/internal/../../lib/metrics-core-3.1.0.jar:./bin/internal/../../l
ib/metrics-jvm-3.1.0.jar:./bin/internal/../../lib/mina-core-1.1.7.jar:./bin/internal/../../lib/mysql-connector-java-5.1.28.jar:./bin
/internal/../../lib/netty-3.2.3.Final.jar:./bin/internal/../../lib/oro-2.0.8.jar:./bin/internal/../../lib/parseq-1.3.6.jar:./bin/int
ernal/../../lib/pegasus-common-1.15.7.jar:./bin/internal/../../lib/quartz-2.2.1.jar:./bin/internal/../../lib/r2-1.15.7.jar:./bin/int
ernal/../../lib/restli-common-1.15.7.jar:./bin/internal/../../lib/restli-server-1.15.7.jar:./bin/internal/../../lib/servlet-api-2.5-
20081211.jar:./bin/internal/../../lib/slf4j-api-1.7.18.jar:./bin/internal/../../lib/slf4j-log4j12-1.7.18.jar:./bin/internal/../../li
b/snakeyaml-1.18.jar:./bin/internal/../../lib/snappy-0.3.jar:./bin/internal/../../lib/sslext-1.2-0.jar:./bin/internal/../../lib/stru
ts-core-1.3.8.jar:./bin/internal/../../lib/struts-taglib-1.3.8.jar:./bin/internal/../../lib/struts-tiles-1.3.8.jar:./bin/internal/..
/../lib/velocity-1.7.jar:./bin/internal/../../lib/velocity-tools-2.0.jar:./bin/internal/../../extlib/*.jar:./bin/internal/../../plug
ins/*/*.jar:/u01/app/hadoop/conf:/u01/app/hadoop/*:/u01/app/apache-hive-2.3.4-bin/conf:/u01/app/apache-hive-2.3.4-bin/lib/*
2019/03/08 15:02:55.688 +0800 INFO [AzkabanWebServer] Starting Jetty Azkaban Web Server...
2019/03/08 15:02:55.708 +0800 INFO [AzkabanServer] Loading azkaban settings file from ./bin/internal/../../conf
2019/03/08 15:02:55.709 +0800 INFO [AzkabanServer] Loading azkaban properties file
2019/03/08 15:02:55.718 +0800 ERROR [StdOutErrRedirect] SLF4J: Class path contains multiple SLF4J bindings.
2019/03/08 15:02:55.719 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/hadoop-2.6.5/share/hadoop/common/
lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:02:55.719 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/azkaban-3.50.0/azkaban-web-server
-0.1.0-SNAPSHOT/lib/slf4j-log4j12-1.7.18.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:02:55.719 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/apache-hive-2.3.4-bin/lib/log4j-s
lf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:02:55.719 +0800 ERROR [StdOutErrRedirect] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanat
ion.
2019/03/08 15:02:55.722 +0800 ERROR [StdOutErrRedirect] SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2019/03/08 15:02:56.157 +0800 INFO [WebServerProvider] Excluded Cipher Suites: []
2019/03/08 15:02:56.161 +0800 INFO [log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2019/03/08 15:02:56.161 +0800 INFO [WebServerProvider] Setting up connector with stats on: true
2019/03/08 15:02:56.163 +0800 INFO [WebServerProvider] Starting SSL server on port: 8443 # Max threads: 25
2019/03/08 15:02:56.679 +0800 INFO [ExecutorManager] Initializing local executor localhost:12321
2019/03/08 15:02:57.044 +0800 WARN [StorageCleaner] Project cleanup disabled. All artifacts will be stored.
2019/03/08 15:02:57.044 +0800 INFO [AzkabanProjectLoader] Using temp dir: /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/
temp
2019/03/08 15:02:57.044 +0800 INFO [AzkabanProjectLoader] Project version retention is set to 3
2019/03/08 15:02:57.046 +0800 WARN [XmlValidatorManager] Validator directory validators does not exist or is not a directory.
2019/03/08 15:02:57.046 +0800 WARN [XmlValidatorManager] Azkaban properties file does not contain the key project.validators.xml.fil
e
2019/03/08 15:02:57.103 +0800 INFO [TriggerManager] TriggerManager loaded.
2019/03/08 15:02:57.125 +0800 INFO [AzkabanWebServerModule] Loading user manager class azkaban.user.XmlUserManager
2019/03/08 15:02:57.233 +0800 INFO [XmlUserManager] Loading user azkaban
2019/03/08 15:02:57.233 +0800 INFO [XmlUserManager] Loading user metrics
2019/03/08 15:02:57.234 +0800 INFO [XmlUserManager] Loading user admin
2019/03/08 15:02:57.287 +0800 INFO [AzkabanWebServer] Loading built-in checker and action types
2019/03/08 15:02:57.290 +0800 INFO [CheckerTypeLoader] Registering checker BasicTimeChecker
2019/03/08 15:02:57.291 +0800 INFO [CheckerTypeLoader] Registering checker SlaChecker
2019/03/08 15:02:57.291 +0800 INFO [CheckerTypeLoader] Registering checker ExecutionChecker
2019/03/08 15:02:57.291 +0800 INFO [ActionTypeLoader] Registering action ExecuteFlowAction
2019/03/08 15:02:57.291 +0800 INFO [ActionTypeLoader] Registering action KillExecutionAction
2019/03/08 15:02:57.292 +0800 INFO [ActionTypeLoader] Registering action AlertAction
2019/03/08 15:02:57.292 +0800 INFO [ActionTypeLoader] Registering action CreateTriggerAction
2019/03/08 15:02:57.293 +0800 INFO [PluginCheckerAndActionsLoader] Loading plug-in checker and action types
2019/03/08 15:02:57.293 +0800 ERROR [PluginCheckerAndActionsLoader] plugin path plugins/triggers doesn't exist!
2019/03/08 15:02:57.314 +0800 INFO [AzkabanWebServer] Setting timezone to Asia/Shanghai
2019/03/08 15:02:57.314 +0800 INFO [AzkabanWebServer] Registering MBeans...
2019/03/08 15:02:57.325 +0800 INFO [AzkabanWebServer] Bean azkaban.jmx.JmxJettyServer registered.
2019/03/08 15:02:57.328 +0800 INFO [AzkabanWebServer] Bean azkaban.jmx.JmxTriggerManager registered.
2019/03/08 15:02:57.331 +0800 INFO [AzkabanWebServer] Bean azkaban.jmx.JmxExecutorManager registered.
2019/03/08 15:02:57.349 +0800 INFO [AzkabanWebServer] Bean org.apache.log4j.jmx.HierarchyDynamicMBean registered.
2019/03/08 15:02:57.349 +0800 INFO [AzkabanWebServer] ************* loginLoggerObjName is null, make sure there is a logger with nam
e azkaban.webapp.servlet.LoginAbstractAzkabanServlet
2019/03/08 15:02:57.350 +0800 INFO [ExecutorManager] Cleaning old logs from execution_logs
2019/03/08 15:02:57.353 +0800 INFO [AzkabanWebServer] Setting up web resource dir /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-S
NAPSHOT/web
2019/03/08 15:02:57.383 +0800 INFO [ExecutorManager] Cleaning old log files before 2018-12-14T15:02:57.350+08:00
2019/03/08 15:02:57.385 +0800 INFO [ExecutorManager] Cleaned up 0 log entries.
2019/03/08 15:02:57.385 +0800 INFO [ExecutorManager] log clean up time: 0 seconds.
2019/03/08 15:02:57.435 +0800 INFO [JdbcTriggerImpl] Loading all triggers from db.
2019/03/08 15:02:57.437 +0800 INFO [JdbcTriggerImpl] Loaded 0 triggers.
2019/03/08 15:02:57.438 +0800 INFO [log] jetty-6.1.26
2019/03/08 15:02:58.736 +0800 INFO [ProjectManagerServlet] downloadBufferSize: 8192
2019/03/08 15:02:58.879 +0800 INFO [RestLiAnnotationReader] Processed actions resource 'azkaban.restli.UserManagerResource'
2019/03/08 15:02:58.881 +0800 INFO [RestLiAnnotationReader] Processed actions resource 'azkaban.restli.ProjectManagerResource'
2019/03/08 15:02:59.057 +0800 INFO [log] Started [email protected]:8443
2019/03/08 15:02:59.057 +0800 INFO [AzkabanWebServer] Server started
查看JPS，会出现AzkabanWebServer：
jps
6.3 登录管理界面
打开浏览器，输入https://10.200.4.117:8443，登录admin用户：
完毕。
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
智能推荐
软件测试流程包括哪些内容？测试方法有哪些？_测试过程管理中包含哪些过程-程序员宅基地
文章浏览阅读2.9k次，点赞8次，收藏14次。测试主要做什么？这完全都体现在测试流程中，同时测试流程是面试问题中出现频率最高的，这不仅是因为测试流程很重要，而是在面试过程中这短短的半小时到一个小时的时间，通过测试流程就可以判断出应聘者是否合适，故在测试流程中包含了测试工作的核心内容，例如需求分析，测试用例的设计，测试执行，缺陷等重要的过程。..._测试过程管理中包含哪些过程
政府数字化政务的人工智能与机器学习应用：如何提高政府工作效率-程序员宅基地
文章浏览阅读870次，点赞16次，收藏19次。1.背景介绍政府数字化政务是指政府利用数字技术、互联网、大数据、人工智能等新技术手段，对政府政务进行数字化改革，提高政府工作效率，提升政府服务质量的过程。随着人工智能(AI)和机器学习(ML)技术的快速发展，政府数字化政务中的人工智能与机器学习应用也逐渐成为政府改革的重要内容。政府数字化政务的人工智能与机器学习应用涉及多个领域，包括政策决策、政府服务、公共安全、社会治理等。在这些领域，人工...
ssm+mysql+微信小程序考研刷题平台_mysql刷题软件-程序员宅基地
文章浏览阅读219次，点赞2次，收藏4次。系统主要的用户为用户、管理员，他们的具体权限如下：用户：用户登录后可以对管理员上传的学习视频进行学习。用户可以选择题型进行练习。用户选择小程序提供的考研科目进行相关训练。用户可以进行水平测试，并且查看相关成绩用户可以进行错题集的整理管理员：管理员登录后可管理个人基本信息管理员登录后可管理个人基本信息管理员可以上传、发布考研的相关例题及其分析，并对题型进行管理管理员可以进行查看、搜索考研题目及错题情况。_mysql刷题软件
根据java代码描绘uml类图_Myeclipse8.5下JAVA代码导成UML类图-程序员宅基地
文章浏览阅读1.4k次。myelipse里有UML1和UML2两种方式，UML2功能更强大，但是两者生成过程差别不大1.建立Test工程，如下图，uml包存放uml类图package com.zz.domain;public class User {private int id;private String name;public int getId() {return id;}public void setId(int..._根据以下java代码画出类图
Flume自定义拦截器-程序员宅基地
文章浏览阅读174次。需求：一个topic包含很多个表信息，需要自动根据json字符串中的字段来写入到hive不同的表对应的路径中。发送到Kafka中的数据原本最外层原本没有pkDay和project，只有data和name。因为担心data里面会空值，所以根同事商量，让他们在最外层添加了project和pkDay字段。pkDay字段用于表的自动分区，proejct和name合起来用于自动拼接hive表的名称为 ..._flume拦截器自定义开发 kafka
java同时输入不同类型数据,Java Spring中同时访问多种不同数据库-程序员宅基地
文章浏览阅读380次。原标题：Java Spring中同时访问多种不同数据库 多样的工作要求，可以使用不同的工作方法，只要能获得结果，就不会徒劳。开发企业应用时我们常常遇到要同时访问多种不同数据库的问题，有时是必须把数据归档到某种数据仓库中，有时是要把数据变更推送到第三方数据库中。使用Spring框架时，使用单一数据库是非常容易的，但如果要同时访问多个数据库的话事件就变得复杂多了。本文以在Spring框架下开发一个Sp..._根据输入的不同连接不同的数据库
随便推点
EFT试验复位案例分析_eft电路图-程序员宅基地
文章浏览阅读3.6k次，点赞9次，收藏25次。本案例描述了晶振屏蔽以及开关电源变压器屏蔽对系统稳定工作的影响， 硬件设计时应考虑。_eft电路图
MR21更改价格_mr21 对于物料 zba89121 存在一个当前或未来标准价格-程序员宅基地
文章浏览阅读1.1k次。对于物料价格的更改，可以采取不同的手段：首先，我们来介绍MR21的方式。 需要说明的是，如果要对某一产品进行价格修改，必须满足的前提条件是： ■ 1、必须对价格生效的物料期间与对应会计期间进行开启； ■ 2、该产品在该物料期间未发生物料移动。执行MR21，例如更改物料1180051689的价格为20000元，系统提示“对于物料1180051689 存在一个当前或未来标准价格”，这是因为已经对该..._mr21 对于物料 zba89121 存在一个当前或未来标准价格
联想启天m420刷bios_联想启天M420台式机怎么装win7系统(完美解决usb)-程序员宅基地
文章浏览阅读7.4k次，点赞3次，收藏13次。[文章导读]联想启天M420是一款商用台式电脑，预装的是win10系统，用户还是喜欢win7系统，该台式机采用的intel 8代i5 8500CPU，在安装安装win7时有很多问题，在安装win7时要在BIOS中“关闭安全启动”和“开启兼容模式”，并且安装过程中usb不能使用，要采用联想win7新机型安装，且默认采用的uefi+gpt模式，要改成legacy+mbr引导，那么联想启天M420台式电..._启天m420刷bios
冗余数据一致性，到底如何保证？-程序员宅基地
文章浏览阅读2.7k次，点赞2次，收藏9次。一，为什么要冗余数据互联网数据量很大的业务场景，往往数据库需要进行水平切分来降低单库数据量。水平切分会有一个patition key，通过patition key的查询能..._保证冗余性
java 打包插件-程序员宅基地
文章浏览阅读88次。是时候闭环Java应用了 原创 2016-08-16 张开涛 你曾经因为部署/上线而痛苦吗？你曾经因为要去运维那改配置而烦恼吗？在我接触过的一些部署/上线方式中，曾碰到过以下一些问题：1、程序代码和依赖都是人工上传到服务器，不是通过工具进行部署和发布；2、目录结构没有规范，jar启动时通过-classpath任意指定；3、fat jar，把程序代码、配置文件和依赖jar都打包到一个jar中，改配置..._那么需要把上面的defaultjavatyperesolver类打包到插件中
VS2015，Microsoft Visual Studio 2005,SourceInsight4.0使用经验，Visual AssistX番茄助手的安装与基本使用9_番茄助手颜色-程序员宅基地
文章浏览阅读909次。1.得下载一个番茄插件，按alt+g才可以有函数跳转功能。2.不安装番茄插件，按F12也可以有跳转功能。3.进公司的VS工程是D:\sync\build\win路径，.sln才是打开工程的方式，一个是VS2005打开的，一个是VS2013打开的。4.公司库里的线程接口，在CmThreadManager.h 里，这个里面是我们的线程库，可以直接拿来用。CreateUserTaskThre..._番茄助手颜色