NiFi日志调整_nifi1.15.3 日志調整-程序员宅基地

技术标签: NiFi日志太多  NiFi  nifi  

NiFi日志调整


默认的NiFi log配置会产生大量的日志,我们的生产环境多次出现硬盘写满的情况(NiFi集群只执行流式计算不存数据,所以硬盘不大)。

一, conf/logback.xml

默认配置:

<configuration scan="true" scanPeriod="30 seconds">
    <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
        <resetJUL>true</resetJUL>
    </contextListener>

    <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!--
              For daily rollover, use 'app_%d.log'.
              For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
              To GZIP rolled files, replace '.log' with '.log.gz'.
              To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{
    yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <immediateFlush>true</immediateFlush>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{
    40} %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="USER_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-user.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!--
              For daily rollover, use 'user_%d.log'.
              For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
              To GZIP rolled files, replace '.log' with '.log.gz'.
              To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-user_%d.log</fileNamePattern>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{
    40} %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="BOOTSTRAP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-bootstrap.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!--
              For daily rollover, use 'user_%d.log'.
              For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
              To GZIP rolled files, replace '.log' with '.log.gz'.
              To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-bootstrap_%d.log</fileNamePattern>
            <!-- keep 5 log files worth of history -->
            <maxHistory>5</maxHistory>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{
    40} %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{
    40} %msg%n</pattern>
        </encoder>
    </appender>

    <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->

    <logger name="org.apache.nifi" level="INFO"/>
    <logger name="org.apache.nifi.processors" level="WARN"/>
    <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>
    <logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/>
    <logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" />


    <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" />
    <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" />
    <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" />
    <logger name="org.apache.zookeeper.server.quorum" level="ERROR" />
    <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" />
    <logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" />

    <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" />

    <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" />
    <logger name="org.apache.curator.ConnectionState" level="OFF" />

    <!-- Logger for managing logging statements for nifi clusters. -->
    <logger name="org.apache.nifi.cluster" level="INFO"/>

    <!-- Logger for logging HTTP requests received by the web server. -->
    <logger name="org.apache.nifi.server.JettyServer" level="INFO"/>

    <!-- Logger for managing logging statements for jetty -->
    <logger name="org.eclipse.jetty" level="INFO"/>

    <!-- Suppress non-error messages due to excessive logging by class or library -->
    <logger name="org.springframework" level="ERROR"/>

    <!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) -->
    <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/>

    <!-- Suppress non-error messages due to Jetty AnnotationParser emitting a large amount of WARNS. Issue described in NIFI-5479. -->
    <logger name="org.eclipse.jetty.annotations.AnnotationParser" level="ERROR"/>

    <!-- Suppress non-error messages from SSHJ which was emitting large amounts of INFO logs by default -->
    <logger name="net.schmizz.sshj" level="WARN" />
    <logger name="com.hierynomus.sshj" level="WARN" />

    <!--
        Logger for capturing user events. We do not want to propagate these
        log events to the root logger. These messages are only sent to the
        user-log appender.
    -->
    <logger name="org.apache.nifi.web.security" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.web.api.config" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.authorization" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.cluster.authorization" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.web.filter.RequestLogger" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>


    <!--
        Logger for capturing Bootstrap logs and NiFi's standard error and standard out.
    -->
    <logger name="org.apache.nifi.bootstrap" level="INFO" additivity="false">
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>
    <logger name="org.apache.nifi.bootstrap.Command" level="INFO" additivity="false">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>

    <!-- Everything written to NiFi's Standard Out will be logged with the logger org.apache.nifi.StdOut at INFO level -->
    <logger name="org.apache.nifi.StdOut" level="INFO" additivity="false">
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>

    <!-- Everything written to NiFi's Standard Error will be logged with the logger org.apache.nifi.StdErr at ERROR level -->
    <logger name="org.apache.nifi.StdErr" level="ERROR" additivity="false">
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>


    <root level="INFO">
        <appender-ref ref="APP_FILE"/>MockProcessContext
    </root>

</configuration>

默认配置会保留30天的日志,每天的日志大小不受限制。其中APP_FILE appender 会产生大量日志。
在这里插入图片描述
只需要修改其回滚策略限制总日志大小。

 <totalSizeCap>3GB</totalSizeCap>

修改后

<configuration scan="true" scanPeriod="30 seconds">
    <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
        <resetJUL>true</resetJUL>
    </contextListener>

    <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!--
              For daily rollover, use 'app_%d.log'.
              For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
              To GZIP rolled files, replace '.log' with '.log.gz'.
              To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{
    yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
            <totalSizeCap>3GB</totalSizeCap>
        </rollingPolicy>
        <immediateFlush>true</immediateFlush>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{
    40} %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="USER_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-user.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!--
              For daily rollover, use 'user_%d.log'.
              For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
              To GZIP rolled files, replace '.log' with '.log.gz'.
              To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-user_%d.log</fileNamePattern>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{
    40} %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="BOOTSTRAP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-bootstrap.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!--
              For daily rollover, use 'user_%d.log'.
              For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
              To GZIP rolled files, replace '.log' with '.log.gz'.
              To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-bootstrap_%d.log</fileNamePattern>
            <!-- keep 5 log files worth of history -->
            <maxHistory>5</maxHistory>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{
    40} %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{
    40} %msg%n</pattern>
        </encoder>
    </appender>

    <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->

    <logger name="org.apache.nifi" level="INFO"/>
    <logger name="org.apache.nifi.processors" level="WARN"/>
    <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>
    <logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/>
    <logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" />


    <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" />
    <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" />
    <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" />
    <logger name="org.apache.zookeeper.server.quorum" level="ERROR" />
    <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" />
    <logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" />

    <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" />

    <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" />
    <logger name="org.apache.curator.ConnectionState" level="OFF" />

    <!-- Logger for managing logging statements for nifi clusters. -->
    <logger name="org.apache.nifi.cluster" level="INFO"/>

    <!-- Logger for logging HTTP requests received by the web server. -->
    <logger name="org.apache.nifi.server.JettyServer" level="INFO"/>

    <!-- Logger for managing logging statements for jetty -->
    <logger name="org.eclipse.jetty" level="INFO"/>

    <!-- Suppress non-error messages due to excessive logging by class or library -->
    <logger name="org.springframework" level="ERROR"/>

    <!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) -->
    <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/>

    <!-- Suppress non-error messages due to Jetty AnnotationParser emitting a large amount of WARNS. Issue described in NIFI-5479. -->
    <logger name="org.eclipse.jetty.annotations.AnnotationParser" level="ERROR"/>

    <!-- Suppress non-error messages from SSHJ which was emitting large amounts of INFO logs by default -->
    <logger name="net.schmizz.sshj" level="WARN" />
    <logger name="com.hierynomus.sshj" level="WARN" />

    <!--
        Logger for capturing user events. We do not want to propagate these
        log events to the root logger. These messages are only sent to the
        user-log appender.
    -->
    <logger name="org.apache.nifi.web.security" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.web.api.config" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.authorization" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.cluster.authorization" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.web.filter.RequestLogger" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>


    <!--
        Logger for capturing Bootstrap logs and NiFi's standard error and standard out.
    -->
    <logger name="org.apache.nifi.bootstrap" level="INFO" additivity="false">
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>
    <logger name="org.apache.nifi.bootstrap.Command" level="INFO" additivity="false">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>

    <!-- Everything written to NiFi's Standard Out will be logged with the logger org.apache.nifi.StdOut at INFO level -->
    <logger name="org.apache.nifi.StdOut" level="INFO" additivity="false">
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>

    <!-- Everything written to NiFi's Standard Error will be logged with the logger org.apache.nifi.StdErr at ERROR level -->
    <logger name="org.apache.nifi.StdErr" level="ERROR" additivity="false">
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>


    <root level="INFO">
        <appender-ref ref="APP_FILE"/>MockProcessContext
    </root>

</configuration>

在这里插入图片描述

版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/weixin_43172032/article/details/121713643

智能推荐

linux 路由跟踪表满错误 nf_conntrack: table full, dropping packet 原理解决方法_nf_conntrack:table full-程序员宅基地

文章浏览阅读5.2k次。netfilter/conntrack 相关内核参数往往是用 Linux 服务器的互联网小公司业务量上去之后遇到的第 3 个“新手怪”。(第 1 位:进程可用的 FD 不足,第 2 位:IP 临时端口不足 + TIME_WAIT 状态的连接过多导致无法建立新连接)很多人以为 Linux 经过这么多年优化,默认参数应该“足够好”,其实不是。默认参数面向“通用”服务器,不适用于连接数和访问量比较多..._nf_conntrack:table full

linux下gcc编译优化选项的大体操作是怎么样的?_linux gcc 性能优化 编译参数调优-程序员宅基地

文章浏览阅读1.3k次。起因: 目前项目使用nios IDE作为开发平台,其使用的编译器为gcc的交叉编译器。在设定编译条件时,在debug模式下生成的程序正常,但是在release模式下会出现LCD显示的开端显示不全,缺少一个字节或字的状况。为了了解具体为什么造成该问题,对两种模式下的配置做了对比,编译器皆为nios2-elf-gcc交叉编译器,debug模式编译器参数为:-DALT_DEBUG -O0 ..._linux gcc 性能优化 编译参数调优

各种免费好用的api,含天气查询、IP查询、物流查询等-程序员宅基地

文章浏览阅读706次,点赞7次,收藏9次。各种免费好用的api,含天气查询、IP查询、物流查询等

Qt工作笔记-Windows上界面滑动效果_qt 界面滑动-程序员宅基地

文章浏览阅读5.5k次,点赞7次,收藏20次。运行截图如下:源码如下:widget.h#ifndef WIDGET_H#define WIDGET_H#include &lt;QWidget&gt;QT_BEGIN_NAMESPACEclass QMouseEvent;class QPropertyAnimation;QT_END_NAMESPACEnamespace Ui {class Widget;..._qt 界面滑动

Qt工作笔记-使用Qt中QProcess与iostream中system调用外部进程_qt system-程序员宅基地

文章浏览阅读5.8k次,点赞9次,收藏22次。目录 基础概念演示 基础概念Qt中的QProcess类可以调用外部程序!同时iostream有个函数system也可以调用外部程序!这个system也是Windows,Linux上通用的 演示如下代码!这是一个外部程序,是用vs2012写的#include &lt;fstream&gt;#include &lt;iostream&gt;usin..._qt system

python︱matplotlib使用(读入、显示、写出、opencv混用、格式转换...)_<matplotlib.image.axesimage object at 0x7f1d27d0c2-程序员宅基地

文章浏览阅读1.3w次,点赞2次,收藏18次。opencv和matplotlib是比较常用的图像分析模块。在Ipython里面,opencv不能直接显示出来,所以有些时候会借助matplotlib来显示。%matplotlib inline .1、matplotlib的读入与显示import matplotlib.pyplot as plt import numpy as np from PIL import ..._

随便推点

关于java的equals和hashcode方法_java使用equals肯定会先调用hashcode吗-程序员宅基地

文章浏览阅读993次。大牛的博客http://blog.csdn.net/lifetragedy, 里面有一系列面试题. 本篇就写关于equals和hashcode的.首先,我们提出问题.1.equals和hashcode有没有什么样的规范呢?2.Object类的equals和hashcode方法是什么样的?3.什么情况下需要重写equals和hashcode方法?重写equals的话必须重写hashc_java使用equals肯定会先调用hashcode吗

快排递归非递归python_用Python递归思考-程序员宅基地

文章浏览阅读611次。快排递归非递归pythonOf all ideas I have introduced to children, recursion stands out as the one idea that is particularly able to evoke an excited response.Seymour Papert, Mindstorms 在我介绍给孩子们的所有想法中,递归特别引人注目...

MATLAB函数句柄(@),feval与内连函数(inline)_f=inline(a)内联函数-程序员宅基地

文章浏览阅读5.3k次,点赞5次,收藏9次。转载地址:http://www.cnblogs.com/begtostudy/archive/2012/06/27/2565920.htmlMATLAB函数句柄函数句柄(Function handle)是MATLAB的一种数据类型。包含了函数的路径、函数名、类型以及可能存在的重载方法;引入函数句柄是为了使feval及借助于它的泛函指令工作更可靠;使“函数调用”像“变_f=inline(a)内联函数

Android资源代码 源码 整理 Github开源项目下载地址_com.github.chenglei1986:datepicker:-snapshot aar依赖-程序员宅基地

文章浏览阅读5.4k次。第一部分 个性化控件(View)主要介绍那些不错个性化的View,包括ListView、ActionBar、Menu、ViewPager、Gallery、GridView、ImageView、ProgressBar、TextView、ScrollView、TimeView、TipView、FlipView、ColorPickView、GraphView、UI Style等等。一、L_com.github.chenglei1986:datepicker:-snapshot aar依赖下载

解决Cannot find module ‘element-ui‘_cannot find module './plugins/element.js' at webpa-程序员宅基地

文章浏览阅读3.6k次。**报错:Cannot find module ‘element-ui’**解决办法:步骤1.npm install element-ui --save步骤2.在main.js中import ElementUI from 'element-ui' import 'element-ui/lib/theme-chalk/index.css' Vue.use(ElementUI);芜湖!!_cannot find module './plugins/element.js' at webpackmissingmodule

S5P4418开发板 A9四核开发板 推荐 最新产品_cortex a9开发板推荐-程序员宅基地

文章浏览阅读565次。推荐理由:1、三星中国S5P4418开发系统国内唯一旗舰商;2、全球绝对独家发布,独家支持3G/4G电话和短信功能;支持休眠唤醒;3、真正的A9四核高端开发板; 4、严格意义上的企业产品级开发板; 5、强悍、稳定的硬件设计和军工级别的质量;6、免费送WIFI蓝牙二合一模块和超级电池盒 2015年5月10号,经过润尔信息(原广州华天正)技术研发团队历时_cortex a9开发板推荐