etcd集群部署

1. 概述

这是一个序列总结文档。

第1节安装etcd 中，在CentOS7上面安装etcd。
第2节单节点运行etcd 在单节点上面运行etcd。
第3节三节点部署etcd集群在三节点上面部署etcd集群，并为etcd配置了一些快捷命令。
第4节 etcd TLS集群部署在三节点上面部署etcd集群，并开启TLS协议的加密通讯。
第5节 etcd角色权限控制在三节点上面部署etcd TLS集群的基础上，开启角色控制。详细可参考 https://etcd.io/docs/v3.5/demo/ 。
第6节 etcd证书过期处理，在etcd证书过期后，etcd相关命令行都操作不了，讲解如何处理这个问题。
第7节 etcd配置文件, 通过etcd配置文件来配置相关参数，然后启动etcd服务。
第8节 etcd可视化工具, 介绍一款好用的etcd可视化工具etcd-workbench。
第9节 etcd的关键配置initial-cluster-state, 介绍initial-cluster-state设置成new和existing的区别。
第10节集群ID和成员ID的不同显示方式, 介绍etcd集群ID和成员ID的不同显示方式，以及ID值如何在十六进制和十进制间进行转换。
第11节 etcd compaction压缩, 介绍etcd compaction压缩相关说明，如何修改压缩配置等。

1.1 VirtualBox虚拟机信息记录

学习etcd时，使用以下几个虚拟机:

序号	虚拟机	主机名	IP	CPU	内存	说明
1	ansible-master	ansible	192.168.56.120	2核	4G	Ansible控制节点
2	ansible-node1	etcd-node1	192.168.56.121	2核	2G	Ansible工作节点1
3	ansible-node2	etcd-node2	192.168.56.122	2核	2G	Ansible工作节点2
4	ansible-node3	etcd-node3	192.168.56.123	2核	2G	Ansible工作节点3

后面会编写使用ansible部署etcd集群的剧本。

操作系统说明：

1
2
3
4
5
[root@etcd-node1 ~]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
[root@etcd-node1 ~]# hostname -I
192.168.56.121 10.0.3.15
[root@etcd-node1 ~]#

2. 三服务器部署etcd集群

参考官方文档How to Set Up a Demo etcd Cluster

ectd端口中是 2379 用于客户端连接，提供HTTP API服务，供客户端交互。而 2380 用于伙伴通讯。

由于是在三个服务器上部署etcd集群，我们可以直接使用默认端口，我们先规划一下：

序号	节点名称	节点运行目录	IP	通讯端口	客户端端口
1	node1	/srv/etcd/node	192.168.56.121	2380	2379
2	node2	/srv/etcd/node	192.168.56.121	2380	2379
3	node2	/srv/etcd/node	192.168.56.121	2380	2379

参考单节点部署etcd服务，配置不同的启动脚本。

查看第1个启动脚本：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[root@etcd-node1 etcd]# cat node/start.sh
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=node1
NAME_2=node2
NAME_3=node3
HOST_1=192.168.56.121
HOST_2=192.168.56.122
HOST_3=192.168.56.123
PEER_PORT_1=2380
PEER_PORT_2=2380
PEER_PORT_3=2380
API_PORT_1=2379
API_PORT_2=2379
API_PORT_3=2379
START_PATH_1=/srv/etcd/node
START_PATH_2=/srv/etcd/node
START_PATH_3=/srv/etcd/node
CLUSTER=${NAME_1}=http://${HOST_1}:${PEER_PORT_1},${NAME_2}=http://${HOST_2}:${PEER_PORT_2},${NAME_3}=http://${HOST_3}:${PEER_PORT_3}
echo "CLUSTER:${CLUSTER}"

## 节点1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
cd ${START_PATH_1}
nohup etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:${PEER_PORT_1} --listen-peer-urls http://${THIS_IP}:${PEER_PORT_1} \
    --advertise-client-urls http://${THIS_IP}:${API_PORT_1} --listen-client-urls http://${THIS_IP}:${API_PORT_1} \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} &

[root@etcd-node1 etcd]#

查看第2个启动脚本：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[root@etcd-node2 etcd]# cat node/start.sh
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=node1
NAME_2=node2
NAME_3=node3
HOST_1=192.168.56.121
HOST_2=192.168.56.122
HOST_3=192.168.56.123
PEER_PORT_1=2380
PEER_PORT_2=2380
PEER_PORT_3=2380
API_PORT_1=2379
API_PORT_2=2379
API_PORT_3=2379
START_PATH_1=/srv/etcd/node
START_PATH_2=/srv/etcd/node
START_PATH_3=/srv/etcd/node
CLUSTER=${NAME_1}=http://${HOST_1}:${PEER_PORT_1},${NAME_2}=http://${HOST_2}:${PEER_PORT_2},${NAME_3}=http://${HOST_3}:${PEER_PORT_3}
echo "CLUSTER:${CLUSTER}"

## 节点2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
cd ${START_PATH_2}
nohup etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:${PEER_PORT_2} --listen-peer-urls http://${THIS_IP}:${PEER_PORT_2} \
    --advertise-client-urls http://${THIS_IP}:${API_PORT_2} --listen-client-urls http://${THIS_IP}:${API_PORT_2} \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} &

[root@etcd-node2 etcd]#

查看第3个启动脚本：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[root@etcd-node3 etcd]# cat node/start.sh
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=node1
NAME_2=node2
NAME_3=node3
HOST_1=192.168.56.121
HOST_2=192.168.56.122
HOST_3=192.168.56.123
PEER_PORT_1=2380
PEER_PORT_2=2380
PEER_PORT_3=2380
API_PORT_1=2379
API_PORT_2=2379
API_PORT_3=2379
START_PATH_1=/srv/etcd/node
START_PATH_2=/srv/etcd/node
START_PATH_3=/srv/etcd/node
CLUSTER=${NAME_1}=http://${HOST_1}:${PEER_PORT_1},${NAME_2}=http://${HOST_2}:${PEER_PORT_2},${NAME_3}=http://${HOST_3}:${PEER_PORT_3}
echo "CLUSTER:${CLUSTER}"

## 节点3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
cd ${START_PATH_3}
nohup etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:${PEER_PORT_3} --listen-peer-urls http://${THIS_IP}:${PEER_PORT_3} \
    --advertise-client-urls http://${THIS_IP}:${API_PORT_3} --listen-client-urls http://${THIS_IP}:${API_PORT_3} \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} &

[root@etcd-node3 etcd]#

三个节点分别执行启动脚本：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
[root@etcd-node1 ~]# cd /srv/etcd/node && ./start.sh 
CLUSTER:node1=http://192.168.56.121:23801,node2=http://192.168.56.121:23802,node3=http://192.168.56.121:23803
/root
[root@etcd-node1 ~]# nohup: appending output to ‘nohup.out’


[root@etcd-node2 ~]# cd /srv/etcd/node && ./start.sh 
CLUSTER:node1=http://192.168.56.121:23801,node2=http://192.168.56.121:23802,node3=http://192.168.56.121:23803
/root
[root@etcd-node2 ~]# nohup: appending output to ‘nohup.out’


[root@etcd-node3 ~]# cd /srv/etcd/node && ./start.sh 
CLUSTER:node1=http://192.168.56.121:23801,node2=http://192.168.56.121:23802,node3=http://192.168.56.121:23803
/root
[root@etcd-node3 ~]# nohup: appending output to ‘nohup.out’

查看etcd进程和端口监听：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
## 第1个节点
[root@etcd-node1 ~]# ps -ef|grep -v grep|grep etcd
root      1440     1  0 22:55 pts/0    00:00:03 etcd --data-dir=data.etcd --name node1 --initial-advertise-peer-urls http://192.168.56.121:2380 --listen-peer-urls http://192.168.56.121:2380 --advertise-client-urls http://192.168.56.121:2379 --listen-client-urls http://192.168.56.121:2379 --initial-cluster node1=http://192.168.56.121:2380,node2=http://192.168.56.122:2380,node3=http://192.168.56.123:2380 --initial-cluster-state new --initial-cluster-token token-01
[root@etcd-node1 ~]# netstat -tunlp|grep etcd
tcp        0      0 192.168.56.121:2379     0.0.0.0:*               LISTEN      1440/etcd
tcp        0      0 192.168.56.121:2380     0.0.0.0:*               LISTEN      1440/etcd
[root@etcd-node1 ~]#


## 第2个节点
[root@etcd-node2 ~]# ps -ef|grep -v grep|grep etcd
root      1458     1  0 22:57 pts/0    00:00:01 etcd --data-dir=data.etcd --name node2 --initial-advertise-peer-urls http://192.168.56.122:2380 --listen-peer-urls http://192.168.56.122:2380 --advertise-client-urls http://192.168.56.122:2379 --listen-client-urls http://192.168.56.122:2379 --initial-cluster node1=http://192.168.56.121:2380,node2=http://192.168.56.122:2380,node3=http://192.168.56.123:2380 --initial-cluster-state new --initial-cluster-token token-01
[root@etcd-node2 ~]# netstat -tunlp|grep etcd
tcp        0      0 192.168.56.122:2379     0.0.0.0:*               LISTEN      1458/etcd
tcp        0      0 192.168.56.122:2380     0.0.0.0:*               LISTEN      1458/etcd


## 第3个节点
[root@etcd-node3 ~]# ps -ef|grep -v grep|grep etcd
root      1459     1  0 22:57 pts/0    00:00:01 etcd --data-dir=data.etcd --name node3 --initial-advertise-peer-urls http://192.168.56.123:2380 --listen-peer-urls http://192.168.56.123:2380 --advertise-client-urls http://192.168.56.123:2379 --listen-client-urls http://192.168.56.123:2379 --initial-cluster node1=http://192.168.56.121:2380,node2=http://192.168.56.122:2380,node3=http://192.168.56.123:2380 --initial-cluster-state new --initial-cluster-token token-01
[root@etcd-node3 ~]# netstat -tunlp|grep etcd
tcp        0      0 192.168.56.123:2379     0.0.0.0:*               LISTEN      1459/etcd
tcp        0      0 192.168.56.123:2380     0.0.0.0:*               LISTEN      1459/etcd
[root@etcd-node3 ~]#

3. 查看集群信息

将以下信息写入到~/.bashrc配置文件中：

1
2
3
4
5
6
7
8
export ETCDCTL_API=3
export HOST_1=192.168.56.121
export HOST_2=192.168.56.122
export HOST_3=192.168.56.123
export API_PORT_1=2379
export API_PORT_2=2379
export API_PORT_3=2379
export ENDPOINTS=${HOST_1}:${API_PORT_1},${HOST_2}:${API_PORT_2},${HOST_3}:${API_PORT_3}

然后使用source ~/.bashrc使配置生效。

3.1 查看集群成员列表

生效后，查看集群成员列表信息：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
[root@etcd-node1 ~]# etcdctl --endpoints=$ENDPOINTS member list
b8d58f2bdbad1d62, started, node1, http://192.168.56.121:2380, http://192.168.56.121:2379, false
c5a01234fd04a6a4, started, node3, http://192.168.56.123:2380, http://192.168.56.123:2379, false
fd3f25a01705c082, started, node2, http://192.168.56.122:2380, http://192.168.56.122:2379, false
[root@etcd-node1 ~]#


[root@etcd-node2 ~]# etcdctl --endpoints=$ENDPOINTS member list
b8d58f2bdbad1d62, started, node1, http://192.168.56.121:2380, http://192.168.56.121:2379, false
c5a01234fd04a6a4, started, node3, http://192.168.56.123:2380, http://192.168.56.123:2379, false
fd3f25a01705c082, started, node2, http://192.168.56.122:2380, http://192.168.56.122:2379, false
[root@etcd-node2 ~]#


[root@etcd-node3 ~]# etcdctl --endpoints=$ENDPOINTS member list
b8d58f2bdbad1d62, started, node1, http://192.168.56.121:2380, http://192.168.56.121:2379, false
c5a01234fd04a6a4, started, node3, http://192.168.56.123:2380, http://192.168.56.123:2379, false
fd3f25a01705c082, started, node2, http://192.168.56.122:2380, http://192.168.56.122:2379, false
[root@etcd-node3 ~]#

3.2 查看集群状态信息

1
etcdctl --endpoints=$ENDPOINTS --write-out=table endpoint status

在三个节点查看到信息应该是一样的，类似如下所示：

1
2
3
4
5
6
7
8
9
[root@etcd-node1 ~]# etcdctl --endpoints=$ENDPOINTS --write-out=table endpoint status
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.56.121:2379 | b8d58f2bdbad1d62 |  3.5.18 |   20 kB |      true |      false |         4 |         16 |                 16 |        |
| 192.168.56.122:2379 | fd3f25a01705c082 |  3.5.18 |   20 kB |     false |      false |         4 |         16 |                 16 |        |
| 192.168.56.123:2379 | c5a01234fd04a6a4 |  3.5.18 |   20 kB |     false |      false |         4 |         16 |                 16 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@etcd-node1 ~]#

可以看到，节点node1是Leader。

3.3 查看集群健康状态

1
etcdctl --endpoints=$ENDPOINTS --write-out=table endpoint health

在三个节点查看到信息应该是一样的，类似如下所示：

1
2
3
4
5
6
7
8
9
[root@etcd-node1 ~]# etcdctl --endpoints=$ENDPOINTS --write-out=table endpoint health
+---------------------+--------+------------+-------+
|      ENDPOINT       | HEALTH |    TOOK    | ERROR |
+---------------------+--------+------------+-------+
| 192.168.56.121:2379 |   true | 2.240929ms |       |
| 192.168.56.122:2379 |   true | 2.163503ms |       |
| 192.168.56.123:2379 |   true | 9.061671ms |       |
+---------------------+--------+------------+-------+
[root@etcd-node1 ~]# 

可以看到，三个节点健康都是true正常的。

3.4 集群的数据读写

在节点1上写入数据：

1
2
3
[root@etcd-node1 ~]# etcdctl --endpoints=$ENDPOINTS put greeting "Hello, etcd"
OK
[root@etcd-node1 ~]#

然后在三个节点上获取数据：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
[root@etcd-node1 ~]# etcdctl --endpoints=$ENDPOINTS get greeting
greeting
Hello, etcd
[root@etcd-node1 ~]#


[root@etcd-node2 ~]# etcdctl --endpoints=$ENDPOINTS get greeting
greeting
Hello, etcd
[root@etcd-node2 ~]#



[root@etcd-node3 ~]# etcdctl --endpoints=$ENDPOINTS get greeting
greeting
Hello, etcd
[root@etcd-node3 ~]#

三个节点都正常获取到数据了。

3.5 设置快捷命令

每次都使用etcdctl --endpoints=$ENDPOINTS --write-out=table endpoint status之类的命令实在太麻烦了，下面使用alias配置快捷命令。

将以下信息写入到~/.bashrc配置文件中：

1
2
3
4
5
6
7
8
## ETCD相关快捷命令
alias etcdctl='etcdctl --endpoints=$ENDPOINTS'
## 查看etcd集群状态信息
alias ecs='etcdClusterStatus'
alias etcdClusterStatus='etcdctl --write-out=table endpoint status'
## 查看etcd集群健康状态
alias ech='etcdClusterHealth'
alias etcdClusterHealth='etcdctl --write-out=table endpoint health'

然后使用source ~/.bashrc使配置生效。

测试快捷命令：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
[root@etcd-node1 ~]# alias|grep etcd
alias ech='etcdClusterHealth'
alias ecs='etcdClusterStatus'
alias etcdClusterHealth='etcdctl --write-out=table endpoint health'
alias etcdClusterStatus='etcdctl --write-out=table endpoint status'
alias etcdHealth='etcdctl --write-out=table endpoint status'
alias etcdStatus='etcdctl --write-out=table endpoint status'
alias etcdctl='etcdctl --endpoints=$ENDPOINTS'
[root@etcd-node1 ~]# ech
+---------------------+--------+------------+-------+
|      ENDPOINT       | HEALTH |    TOOK    | ERROR |
+---------------------+--------+------------+-------+
| 192.168.56.121:2379 |   true | 2.522502ms |       |
| 192.168.56.122:2379 |   true | 2.649364ms |       |
| 192.168.56.123:2379 |   true | 2.849195ms |       |
+---------------------+--------+------------+-------+
[root@etcd-node1 ~]# etcdClusterHealth
+---------------------+--------+------------+-------+
|      ENDPOINT       | HEALTH |    TOOK    | ERROR |
+---------------------+--------+------------+-------+
| 192.168.56.121:2379 |   true | 2.389573ms |       |
| 192.168.56.123:2379 |   true |  2.46838ms |       |
| 192.168.56.122:2379 |   true | 2.415691ms |       |
+---------------------+--------+------------+-------+
[root@etcd-node1 ~]# ecs
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.56.121:2379 | b8d58f2bdbad1d62 |  3.5.18 |   20 kB |      true |      false |         7 |         38 |                 38 |        |
| 192.168.56.122:2379 | fd3f25a01705c082 |  3.5.18 |   20 kB |     false |      false |         7 |         38 |                 38 |        |
| 192.168.56.123:2379 | c5a01234fd04a6a4 |  3.5.18 |   20 kB |     false |      false |         7 |         38 |                 38 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@etcd-node1 ~]# etcdClusterStatus
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.56.121:2379 | b8d58f2bdbad1d62 |  3.5.18 |   20 kB |      true |      false |         7 |         38 |                 38 |        |
| 192.168.56.122:2379 | fd3f25a01705c082 |  3.5.18 |   20 kB |     false |      false |         7 |         38 |                 38 |        |
| 192.168.56.123:2379 | c5a01234fd04a6a4 |  3.5.18 |   20 kB |     false |      false |         7 |         38 |                 38 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@etcd-node1 ~]# etcdctl get greeting
greeting
Hello, etcd
[root@etcd-node1 ~]#

可以看到，快捷命令正常工作！