kafka集群搭建

1.先安裝好jdk1.8和zookeeper3.4.9, 安裝zookeeper3.4.9 

2.並啟動zookeeper集群

3.準備三颱伺服器搭建kafka集群環境

1.把壓縮包上傳到三颱伺服器同一路徑下

2.修改配置檔案 kafka/config/server.properties

三颱伺服器建立logs目錄

第一台伺服器 ip01：

broker.id=1                             #節點編號，每個伺服器不一樣

num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/usr/kafka_2.11-0.10.0.0/logs #日誌目錄要建立出來
num.partitions=2 #分割槽數
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=ip01:2181,ip02:2181,ip03:2181 #zookeeper集群位址
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true #可以刪除topic
host.name=ip01 #ip位址

第二台伺服器 ip02：

broker.id=2

num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/usr/kafka_2.11-0.10.0.0/logs
num.partitions=2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=ip01:2181,ip02:2181,ip03:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true
host.name=ip02

第三台伺服器 ip03：

broker.id=3

num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/usr/kafka_2.11-0.10.0.0/logs
num.partitions=2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=ip01:2181,ip02:2181,ip03:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true
host.name=ip03

三颱服務後台啟動命令：

nohup bin/kafka-server-start.sh config/server.properties 2>&1 &

檢視是否啟動成功:jps

建立乙個topic：

bin/kafka-topics.sh --create --zookeeper ip01:2181 --replication-factor 2 -- partitions 3 --topic test

生產者生產資料：

bin/kafka-console-producer.sh --broker-list ip01:9092,ip02:9092,ip03:9092 --topic test

消費者消費資料：

bin/kafka-console-consumer.sh --from-beginning --topic test  --zookeeper ip01:2181,ip02:2181,ip03:2181

檢視topic：

bin/kafka-topics.sh --zookeeper 172.20.8.58:2181 --list

kafka集群搭建

安裝zookeeper kafka自帶了zookeeper，建議還是用外部的zk集群，搭建步驟如下 準備3臺機器，假設我們叫c1 c2 c3，ip分別為ip1 ip2 ip3 將附件zookeeper 3.4.6.tar.gz分別拷貝至3臺機器 c1機器上解壓zookeeper 3.4.6.tar....

kafka 集群搭建

1 前置條件 安裝jdk zookeeper 集群搭建 wget tar xzf kafka 2.11 0.10.2.1.tgz 3 修改 config server.properties 三個地方 1 broker.id 要對上 log.dirs 下 meta.properties broker....

kafka集群搭建

1 上傳安裝包 把安裝包上傳到伺服器 put r d kafka 2.11 0.8.2.0.tgz 2 解壓縮安裝包 3 配置kafka環境變數 vim etc profile export path path kafka home bin 重新整理策略 source etc profile 4 修...