利用tail内置插件来获取日志

tail插件相当于tail -f，它会不断的获取更新的日志，

    @type     tail    path      /log-dir/*-app.log    pos_file  /log-dir/app.log.pos    tagidaas    refresh_interval 10s    read_from_head true    path_key path    
    
                 @type json      #把日志格式直接解析为json格式    
      @type     tail  path      /log-dir/*-debug.log  pos_file  /log-dir/debug.log.pos  tagdebug  multiline_flush_interval 2s  read_from_head true  path_key path    
    
                 @type   multiline               #multiline 相当于logstash的multiline             format_firstline /^(?
     
      (INFO|WARN|ERROR)+)/            format1 /(?
      
       [a-zA-Z]+)\s*\[(?
       
        [0-9/\-: ,]+)\] (?
        
         [a-zA-Z\.]+):(?
         
          [\d\D\s]+)/ 
         
        
       
      
     
        @type     tail    path      /log-dir/*-requests.log    pos_file  /log-dir/request.log.pos    tagrequest    refresh_interval 10s    read_from_head true    path_key path    
    
             @type regexp        expression /(?
     
      .*)/    
     
    

第一个filter是为日志添加字段，tag和宿主机的名字，这个可能需要调docker，直接取只会取到docker的ID

    
         @type record_transformer                 
     
              tag ${tag}        hostname "#{Socket.gethostname}"    
     
        
    
         @type    grep                         #排除掉一些不需要的日志    
     
              key message        pattern /.*healthcheck.*|.*prometheusMetrics.*|.*(v1+\/)+(configurations)+(\/+versions).*/    
     
    
    
         @type parser    key_name message    reserve_data yes    
     
                 @type regexp        expression  /(?
      
       [^|]+)\|(?
       
        [^|]+)\|(?
        
         [^|]+)\|(?
         
          [^|]+)\|(?
          
           [^|]+)\|(?
           
            [^|]+)\|(?
            
             [^|]+)\|(?
             
              [^|]+)\|(?
              
               [^|]+)\|(?
               
                [^|]+)\|(?
                
                 [^|]+)\|(?
                 
                  [^|]+) \|.*\|(?
                  
                   [\w]+)/ 
                  
                 
                
               
              
             
            
           
          
         
        
       
      
     
    
    
         @type rewrite_tag_filter        #重写tag，匹配的重写tag为app.token,不匹配的重写标app.idaas    
     
              key     thread_name        pattern /token/        tag     app.token    
         
     
               key     thread_name         pattern /token/         tag     app.idaas         invert  true    
     
    

上面已经把idaas进行分流处理，这里我们把app.token进行一次过滤，然后和app.idaas一起输入到ES中

    
         @type parser    key_name thread_name    reserve_data yes    
     
              @type regexp        expression /(?
      
       [A-Za-z0-9\.\-_=/\? ]+\.)/    
      
     
    
    
         @type elasticsearch    host elasticsearchlog-lb.elasticsearch-log    index_name    s3-fluentd-request-%Y%m%d    type_name     s3-fluentd-request    include_timestamp true    ssl_verify    false    
     
              timekey          4s                  timekey_wait     1s     #这里的刷新是把buffer的数据及时发送到ES，保持数据的实时性    
     
    
    
         @type elasticsearch    host elasticsearchlog-lb.elasticsearch-log    index_name    s3-fluentd-debug-%Y%m%d    type_name     s3-fluentd-debug    include_timestamp true    ssl_verify    false    
     
              timekey          4s                  timekey_wait     1s       
     
    
    
         @type elasticsearch    host elasticsearchlog-lb.elasticsearch-log    index_name    s3-fluentd-idaas-%Y%m%d    type_name     s3-fluentd-idaas    include_timestamp true    ssl_verify    false    
     
              timekey          4s                  timekey_wait     1s       
     
    

fluentd日志处理-安装配置（一）