K8S-ELK日志系统之三：卡夫卡集群
k8s部署卡夫卡
kafka: https://mirrors .金枪鱼。清华。edu。cn/Apache/Kafka/3。0 .0/卡夫卡_ 2.12-3。0 .0 .tgz
1、创建卡夫卡网络文件系统
10.0.8.131:/data.kafka
2、用户和权限与一和二一致
3、新增NFS置备程序
新增容器和卷
一类：部署
2个应用程序版本：/v1
3元数据：
四名称：客户端-置备程序
5命名空间： es
6标签:
7 app : NFS-客户端-置备程序
8 spec:
9副本： 1
10战略：
11类型：重新创建
12选择器：
13匹配标签:
14 app : NFS-客户端-置备程序
15模板：
16元数据：
17标签:
18 app : NFS-客户端-置备程序
19 spec:
20服务帐户：置备程序
21集装箱：
22 -名称：客户端-置备程序
23 #图像： quay.io/external_storage/nfs-client-provisioner:latest
24图像： easzlab/NFS-子资源-外部资源：v 4。0 .一
25卷安装3360
26 -名称：客户端-根
27个挂载路径：/持久卷
28 env:
29 -名称：置备程序_名称
30值：置备程序
31 - name: NFS_SERVER
32 value: 10.0.8.131
33 -名称： NFS _路径
34值：/数据/弹性搜索
35 -名称：动物园管理员-供应员
36 image : easzlab/NFS-子供应商-外部供应商：v 4。0 .一
37卷安装3360
38 -名称：动物园管理员-根
39个挂载路径：/持久卷
40 env:
41 -名称：置备程序_名称
42价值：动物园管理员-供应商
43 -名称： NFS _服务器
44 value: 10.0.8.131
45 -名称： NFS _路径
46 value: /data/zookeeper
47-名称： NFS-卡夫卡-供应商
48 image : easzlab/NFS-子供应商-外部供应商：v 4。0 .一
49卷安装3360
50 - name: nfs-kafka-root
51个挂载路径：/持久卷
52 env:
53 -名称：置备程序_名称
54价值： nfs-kafka-provisioner
55 -名称： NFS _服务器
56
value: 10.0.8.131
57 - name: NFS_PATH
58 value: /data/kafka
59 volumes:
60 - name: nfs-client-root
61 nfs:
62 server: 10.0.8.131
63 path: /data/elasticsearch
64 - name: nfs-zookeeper-root
65 nfs:
66 server: 10.0.8.131
67 path: /data/zookeeper
68 - name: nfs-kafka-root
69 nfs:
70 server: 10.0.8.131
71 path: /data/kafka
nfs-provisioner.yaml
3、新增storageclass
1 apiVersion: storage.k8s.io/v1 2 kind: StorageClass 3 metadata: 4 name: nfs-kafka-storage 5 provisioner: nfs-kafka-provisioner 6 parameters: 7 archiveOnDelete: "true"
storageclass.yaml
4、Dockerfile
1 FROM openjdk:11
2 ADD kafka_2.12-3.0.0.tgz /
3 COPY sources.list /etc/apt/
4 COPY server.properties /kafka_2.12-3.0.0/config/
5 COPY entrypoint.sh /
6 RUN apt-get update -y \
7 apt-get install vim inetutils-ping net-tools telnet -y \
8 chmod 777 /tmp \
9 chmod +x entrypoint.sh
10 ENV PATH="/kafka_2.12-3.0.0/bin:${PATH}" \
11 KAFKA_VERSION=3.0.0
12
13 ENTRYPOINT ["/entrypoint.sh"]
14 CMD ["kafka-server-start.sh","/kafka_2.12-3.0.0/config/server.properties"]
15 EXPOSE 9092
Dockerfile
1 #!/bin/bash
2
3 CONFIGFILE_PATH="/kafka_2.12-3.0.0/config/server.properties"
4 IFS=\,, read -a zkservers "${ZKSERVERS}"
5 zkconnect=''
6 for i in ${zkservers[@]};do
7 if [ "${zkconnect}x" == "x" ];then
8 zkconnect="$i:$PORT"
9
10 else
11 zkconnect="${zkconnect},$i:$PORT"
12 fi
13 done
14 echo "broker.id=`hostname|awk -F'-' '{print $2}'`" ${CONFIGFILE_PATH}
15 echo "zookeeper.connect=${zkconnect}" ${CONFIGFILE_PATH}
16 echo "log.dirs=${LOG_DIR}" ${CONFIGFILE_PATH}
17 echo "listeners=PLAINTEXT://`hostname -i`:9092" ${CONFIGFILE_PATH}
18
19 exec "$@"
entrypoint.sh
# 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释 deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye main contrib non-free # deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye main contrib non-free deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-updates main contrib non-free # deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-updates main contrib non-free deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-backports main contrib non-free # deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-backports main contrib non-free deb https://mirrors.tuna.tsinghua.edu.cn/debian-security bullseye-security main contrib non-free # deb-src https://mirrors.tuna.tsinghua.edu.cn/debian-security bullseye-security main contrib non-free
sources.list
num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0
server.properties
传递到本地仓库：
docker build -t harbor.myland.com/baseimages/kafka-jdk11:3.0.0-20211119 .
docker push harbor.myland.com/baseimages/kafka-jdk11:3.0.0-20211119
5、statefulset.yaml
包含注册到zookeeper
1 apiVersion: v1
2 kind: Service
3 metadata:
4 name: kafka
5 namespace: es
6 labels:
7 app: kafka
8 spec:
9 type: NodePort
10 ports:
11 - name: kafka
12 port: 9020
13 targetPort: 9020
14 nodePort: 30209
15 selector:
16 app: kafka
17 ---
18 apiVersion: apps/v1
19 kind: StatefulSet
20 metadata:
21 name: kafka
22 namespace: es
23 spec:
24 selector:
25 matchLabels:
26 app: kafka
27 serviceName: kafka
28 replicas: 3
29 template:
30 metadata:
31 labels:
32 app: kafka
33 spec:
34 containers:
35 - name: kafka
36 image: harbor.myland.com/baseimages/kafka-jdk11:3.0.0-20211119
37 imagePullPolicy: Always
38 env:
39 - name: LOG_DIR
40 value: "/data/kafka-logs"
41 - name: PORT
42 value: "2181"
43 - name: ZKSERVERS
44 value: "zookeeper-0.zk,zookeeper-1.zk,zookeeper-2.zk"
45
46 ports:
47 - name: kafka
48 containerPort: 9020
49 volumeMounts:
50 - name: data
51 mountPath: /data
52 resources:
53 limits:
54 cpu: 200m
55 memory: 600Mi
56 requests:
57 cpu: 100m
58 memory: 100Mi
59 readinessProbe:
60 exec:
61 command: ["netstat", "-ant","grep","9200"]
62 initialDelaySeconds: 30
63 periodSeconds: 10
64 timeoutSeconds: 5
65 affinity:
66 podAntiAffinity:
67 requiredDuringSchedulingIgnoredDuringExecution:
68 - labelSelector:
69 matchExpressions:
70 - {key: app ,operator: In, values: ["kafka"]}
71 topologyKey: kubernetes.io/hostname
72
73 volumeClaimTemplates:
74 - metadata:
75 name: data
76 annotations:
77 volume.beta.kubernetes.io/storage-class: "nfs-kafka-storage"
78 namespace: es
79 spec:
80 accessModes: [ "ReadWriteOnce" ]
81 storageClassName: nfs-kafka-storage
82 resources:
83 requests:
84 storage: 5Gi
kafka_stateful.yaml
6、验证
内容来源网络，如有侵权，联系删除，本文地址：https://www.230890.com/zhan/153040.html