1）在 elk-03 的/bd/flume-1.7/conf 目錄下建立 kafka-flume-hdfs.conf 檔案[hadoop@elk-03 conf]$ vim kafka-flume-hdfs.conf
2 ) 在檔案配置如下內容
## 元件定義
a1.sources=r1 r2
a1.channels=c1 c2
a1.sinks=k1 k2
## source1
## kafka start 主題源資料
a1.sources.r1.type = org.apache.flume.source.kafka.kafkasource
a1.sources.r1.batchsize = 5000
a1.sources.r1.batchdurationmillis = 2000
a1.sources.r1.kafka.bootstrap.servers = elk-01:9092,elk-02:9092,elk-03:9092
a1.sources.r1.kafka.zookeeperconnect = elk-01:2181,elk-02:2181,elk-03:2181
a1.sources.r1.kafka.topics=topic_start
## source2
## kafka event 主題源資料
a1.sources.r2.type = org.apache.flume.source.kafka.kafkasource
a1.sources.r2.batchsize = 5000
a1.sources.r2.batchdurationmillis = 2000
a1.sources.r2.kafka.bootstrap.servers = elk-01:9092,elk-02:9092,elk-03:9092
a1.sources.r2.kafka.zookeeperconnect = elk-01:2181,elk-02:2181,elk-03:2181
a1.sources.r2.kafka.topics=topic_event
## channel1
a1.channels.c1.type=memory
a1.channels.c1.capacity=100000
a1.channels.c1.transactioncapacity=10000
## channel2
a1.channels.c2.type=memory
a1.channels.c2.capacity=100000
a1.channels.c2.transactioncapacity=10000
## sink1
## 主題 start 資料輸出到 hdfs 的路徑
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /origin_data/gmall/log/topic_start/%y-%m-%d
a1.sinks.k1.hdfs.fileprefix = logstarta1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundvalue = 30
a1.sinks.k1.hdfs.roundunit = second
##sink2
## 主題 event 資料輸出到 hdfs 的路徑
a1.sinks.k2.type = hdfs
a1.sinks.k2.hdfs.path = /origin_data/gmall/log/topic_event/%y-%m-%d
a1.sinks.k2.hdfs.fileprefix = logeventa1.sinks.k2.hdfs.round = true
a1.sinks.k2.hdfs.roundvalue = 30
a1.sinks.k2.hdfs.roundunit = second
##(生成檔案大小設定) 不要產生大量小檔案
a1.sinks.k1.hdfs.rollinterval = 30
a1.sinks.k1.hdfs.rollsize = 0
a1.sinks.k1.hdfs.rollcount = 0
a1.sinks.k2.hdfs.rollinterval = 30
a1.sinks.k2.hdfs.rollsize = 0
a1.sinks.k2.hdfs.rollcount = 0
## 控制輸出檔案是原生檔案。
a1.sinks.k1.hdfs.filetype = compressedstream
a1.sinks.k2.hdfs.filetype = compressedstream
## 支援 lzo 壓縮
a1.sinks.k1.hdfs.codec = lzop
a1.sinks.k2.hdfs.codec = lzop
## 拼裝
a1.sources.r1.channels = c1
a1.sinks.k1.channel= c1
a1.sources.r2.channels = c2
a1.sinks.k2.channel= c2
flume配置檔案example
flume配置檔案example agent1表示 名稱 agent1.sources source1 agent1.sinks sink1 agent1.channels channel1 spooling directory是監控指定資料夾中新檔案的變化，一旦新檔案出現，就解析該檔案內容，然後寫...
Flume配置檔案1
flume監聽埠的配置檔案 smple.conf a single node flume configuration name the components on this agent 定義變數方便呼叫 加s可以有多個此角色 a1.sources r1 a1.sinks k1 a1.channels...
Flume配置檔案簡單整理
各配置檔案資訊如下 一 flume監控hive日誌資訊並上傳到hdfs name the components on this agent a2.sources r2 a2.sinks k2 a2.channels c2 describe configure the source a2.source...