從flume到kafka，日誌收集

實時日誌分析：

本篇文章主要測試 從flume到kafka的日誌收集，storm日誌分析，學習中！

flume 配置檔案

#collector

collector.sources=cs
collector.sinks=ck hbasesink
collector.channels=cc
hbasechannel
collector.sources.cs.type =exec
collector.sources.cs.command = tail -f /data/hudonglogs/self/channel.log
collector.sources.cs.channels=cc
hbasechannel
collector.channels.
cc.type =memory
collector.channels.
cc.capacity = 1000
collector.channels.
cc.transactioncapacity = 100
collector.channels.hbasechannel.type =memory
collector.channels.hbasechannel.capacity = 1000
collector.channels.hbasechannel.transactioncapacity = 100
#sink kafka
collector.sinks.ck.type =org.apache.flume.sink.kafka.kafkasink
collector.sinks.ck.topic =logs
collector.sinks.ck.brokerlist = localhost:9092
collector.sinks.ck.requiredacks = 1
collector.sinks.ck.batchsize = 20
collector.sinks.ck.channel = cc
#hbase sink
collector.sinks.hbasesink.type =asynchbase
collector.sinks.hbasesink.channel =hbasechannel
collector.sinks.hbasesink.table =logs
collector.sinks.hbasesink.columnfamily =content
collector.sinks.hbasesink.batchsize = 5

kafka consumer

public

class kafkaconsumer extends
thread 
private
consumerconfig createconsumerconfig() 
@override
public
void
run() 
catch
(interruptedexception e) }}
}

hbase檢視：

以上所有環境都是單節點部署！

flume實時收集日誌到kafka

flume實時收集日誌 kafka版本0.8.2 1.版本apache flume 1.7.0 bin.tar.gz 解壓後conf 目錄下配置以.conf結尾的檔案如 flume properties.conf 2.配置檔案資訊 sources 資料來源每增加乙個新增即可 a1.sources r...

flume從kafka導資料到hdfs

flume是cloudera提供的乙個高可用的，高可靠的，分布式的海量日誌採集 聚合和傳輸的系統，flume支援在日誌系統中定製各類資料傳送方，用於收集資料 同時，flume提供對資料進行簡單處理，並寫到各種資料接受方 可定製 的能力.利用flume從kafka導資料到hdfs 配置檔案如下 flu...

flume日誌採集 到kafka ，使用斷點續傳

使用斷點續傳所需要的jar，可直接替換flume bin目錄下的flume taildir source 1.9.0.jar agent custominterceptor.sources r1 custominterceptor.channels c1 custominterceptor.sink...