编程技术分享
搜索 K

外观

etcd证书过期处理 ​

1. 概述 ​

这是一个序列总结文档。

1.1 VirtualBox虚拟机信息记录 ​

学习etcd时，使用以下几个虚拟机:

序号虚拟机主机名IPCPU内存说明
1ansible-masteransible192.168.56.1202核4GAnsible控制节点
2ansible-node1etcd-node1192.168.56.1212核2GAnsible工作节点1
3ansible-node2etcd-node2192.168.56.1222核2GAnsible工作节点2
4ansible-node3etcd-node3192.168.56.1232核2GAnsible工作节点3

后面会编写使用ansible部署etcd集群的剧本。

操作系统说明：

sh
[root@etcd-node1 ~]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
[root@etcd-node1 ~]# hostname -I
192.168.56.121 10.0.3.15
[root@etcd-node1 ~]#

1.2 前情说明 ​

我们在第4节 etcd TLS集群部署 在三节点上面部署etcd集群，并开启TLS协议的加密通讯。并特意将etcd节点证书和客户端证书有效期设置成5天。

以节点node1为例，查看证书有效期：

sh
[root@etcd-node1 ~]# cd /etc/etcd/ssl
[root@etcd-node1 ssl]# openssl x509 -in ca.crt -noout -text | grep 'Not'
            Not Before: Apr  6 14:47:21 2025 GMT
            Not After : Apr  4 14:47:21 2035 GMT
[root@etcd-node1 ssl]# openssl x509 -in node1.crt -noout -text | grep 'Not'
            Not Before: Apr  6 15:04:52 2025 GMT
            Not After : Apr 11 15:04:52 2025 GMT
[root@etcd-node1 ssl]# openssl x509 -in client.crt -noout -text | grep 'Not'
            Not Before: Apr  6 15:20:12 2025 GMT
            Not After : Apr 11 15:20:12 2025 GMT
[root@etcd-node1 ssl]# date
Sat Apr 12 22:06:34 CST 2025
[root@etcd-node1 ssl]#

可以看到，当前时间是2025年4月12日，而node1.crt和client.crt证书有期期都是到2025年4月11日，也就是说有效期已经过了！

修复方法就是用CA重新签发节点证书和客户端证书。

2. 重新签发证书 ​

查看之前生成的相关证书文件：

sh
[root@etcd-node1 ~]# cd cacenter/
[root@etcd-node1 cacenter]# ll
total 72
-rw-r--r-- 1 root root 1090 Apr  6 22:47 ca.crt
-rw-r--r-- 1 root root 1675 Apr  6 22:46 ca.key
-rw-r--r-- 1 root root   17 Apr  6 23:20 ca.srl
-rw-r--r-- 1 root root  977 Apr  6 23:20 client.crt
-rw-r--r-- 1 root root  895 Apr  6 23:19 client.csr
-rw-r--r-- 1 root root 1675 Apr  6 23:19 client.key
-rw-r--r-- 1 root root  317 Apr  6 22:58 node1.cnf
-rw-r--r-- 1 root root 1038 Apr  6 23:04 node1.crt
-rw-r--r-- 1 root root  964 Apr  6 22:58 node1.csr
-rw-r--r-- 1 root root 1675 Apr  6 22:58 node1.key
-rw-r--r-- 1 root root  317 Apr  6 23:09 node2.cnf
-rw-r--r-- 1 root root 1038 Apr  6 23:13 node2.crt
-rw-r--r-- 1 root root  964 Apr  6 23:13 node2.csr
-rw-r--r-- 1 root root 1675 Apr  6 23:13 node2.key
-rw-r--r-- 1 root root  317 Apr  6 23:16 node3.cnf
-rw-r--r-- 1 root root 1038 Apr  6 23:17 node3.crt
-rw-r--r-- 1 root root  964 Apr  6 23:17 node3.csr
-rw-r--r-- 1 root root 1679 Apr  6 23:17 node3.key
[root@etcd-node1 cacenter]#

先备份一下：

sh
[root@etcd-node1 ~]# cp -rp ~/cacenter ~/cacenter.20250412.bak
[root@etcd-node1 ~]# ll ~/cacenter.20250412.bak/
total 72
-rw-r--r-- 1 root root 1090 Apr  6 22:47 ca.crt
-rw-r--r-- 1 root root 1675 Apr  6 22:46 ca.key
-rw-r--r-- 1 root root   17 Apr  6 23:20 ca.srl
-rw-r--r-- 1 root root  977 Apr  6 23:20 client.crt
-rw-r--r-- 1 root root  895 Apr  6 23:19 client.csr
-rw-r--r-- 1 root root 1675 Apr  6 23:19 client.key
-rw-r--r-- 1 root root  317 Apr  6 22:58 node1.cnf
-rw-r--r-- 1 root root 1038 Apr  6 23:04 node1.crt
-rw-r--r-- 1 root root  964 Apr  6 22:58 node1.csr
-rw-r--r-- 1 root root 1675 Apr  6 22:58 node1.key
-rw-r--r-- 1 root root  317 Apr  6 23:09 node2.cnf
-rw-r--r-- 1 root root 1038 Apr  6 23:13 node2.crt
-rw-r--r-- 1 root root  964 Apr  6 23:13 node2.csr
-rw-r--r-- 1 root root 1675 Apr  6 23:13 node2.key
-rw-r--r-- 1 root root  317 Apr  6 23:16 node3.cnf
-rw-r--r-- 1 root root 1038 Apr  6 23:17 node3.crt
-rw-r--r-- 1 root root  964 Apr  6 23:17 node3.csr
-rw-r--r-- 1 root root 1679 Apr  6 23:17 node3.key
[root@etcd-node1 ~]#

可以看到，证书文件都备份成功。

然后我们先cacenter目录重新签发证书：

sh
openssl x509 -req -in node1.csr  -CA ca.crt -CAkey ca.key -CAcreateserial -out node1.crt  -days 1825 -extensions v3_req -extfile node1.cnf
openssl x509 -req -in node2.csr  -CA ca.crt -CAkey ca.key -CAcreateserial -out node2.crt  -days 1825 -extensions v3_req -extfile node2.cnf
openssl x509 -req -in node3.csr  -CA ca.crt -CAkey ca.key -CAcreateserial -out node3.crt  -days 1825 -extensions v3_req -extfile node3.cnf
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 1825

直接执行以上命令：

sh
[root@etcd-node1 ~]# cd cacenter
[root@etcd-node1 cacenter]# openssl x509 -req -in node1.csr  -CA ca.crt -CAkey ca.key -CAcreateserial -out node1.crt  -days 1825 -extensions v3_req -extfile node1.cnf
openssl x509 -req -in node2.csr  -CA ca.crt -CAkey ca.key -CAcreateserial -out node2.crt  -days 1825 -extensions v3_req -extfile node2.cnf
Signature ok
subject=/CN=node1
Getting CA Private Key
[root@etcd-node1 cacenter]# openssl x509 -req -in node2.csr  -CA ca.crt -CAkey ca.key -CAcreateserial -out node2.crt  -days 1825 -extensions v3_req -extfile node2.cnf
openssl x509 -req -in node3.csr  -CA ca.crt -CAkey ca.key -CAcreateserial -out node3.crt  -days 1825 -extensions v3_req -extfile node3.cnf
Signature ok
subject=/CN=node2
Getting CA Private Key
[root@etcd-node1 cacenter]# openssl x509 -req -in node3.csr  -CA ca.crt -CAkey ca.key -CAcreateserial -out node3.crt  -days 1825 -extensions v3_req -extfile node3.cnf
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 1825Signature ok
subject=/CN=node3
Getting CA Private Key
[root@etcd-node1 cacenter]# openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 1825
Signature ok
subject=/CN=etcd-client
Getting CA Private Key
[root@etcd-node1 cacenter]# ll
total 72
-rw-r--r-- 1 root root 1090 Apr  6 22:47 ca.crt
-rw-r--r-- 1 root root 1675 Apr  6 22:46 ca.key
-rw-r--r-- 1 root root   17 Apr 12 22:19 ca.srl
-rw-r--r-- 1 root root  977 Apr 12 22:19 client.crt
-rw-r--r-- 1 root root  895 Apr  6 23:19 client.csr
-rw-r--r-- 1 root root 1675 Apr  6 23:19 client.key
-rw-r--r-- 1 root root  317 Apr  6 22:58 node1.cnf
-rw-r--r-- 1 root root 1038 Apr 12 22:19 node1.crt
-rw-r--r-- 1 root root  964 Apr  6 22:58 node1.csr
-rw-r--r-- 1 root root 1675 Apr  6 22:58 node1.key
-rw-r--r-- 1 root root  317 Apr  6 23:09 node2.cnf
-rw-r--r-- 1 root root 1038 Apr 12 22:19 node2.crt
-rw-r--r-- 1 root root  964 Apr  6 23:13 node2.csr
-rw-r--r-- 1 root root 1675 Apr  6 23:13 node2.key
-rw-r--r-- 1 root root  317 Apr  6 23:16 node3.cnf
-rw-r--r-- 1 root root 1038 Apr 12 22:19 node3.crt
-rw-r--r-- 1 root root  964 Apr  6 23:17 node3.csr
-rw-r--r-- 1 root root 1679 Apr  6 23:17 node3.key
[root@etcd-node1 cacenter]#

此时，可以看到node1.crt、node2.crt、node3.crt、client.crt的时间都发生了变化。

重新检查下证书有效期：

sh
[root@etcd-node1 ssl]# cd ~/cacenter
[root@etcd-node1 cacenter]# openssl x509 -in ca.crt -noout -text | grep 'Not'            Not Before: Apr  6 14:47:21 2025 GMT
            Not After : Apr  4 14:47:21 2035 GMT
[root@etcd-node1 cacenter]# openssl x509 -in node1.crt -noout -text | grep 'Not'
            Not Before: Apr 12 14:19:23 2025 GMT
            Not After : Apr 11 14:19:23 2030 GMT
[root@etcd-node1 cacenter]# openssl x509 -in node2.crt -noout -text | grep 'Not'
            Not Before: Apr 12 14:19:23 2025 GMT
            Not After : Apr 11 14:19:23 2030 GMT
[root@etcd-node1 cacenter]# openssl x509 -in node3.crt -noout -text | grep 'Not'
            Not Before: Apr 12 14:19:23 2025 GMT
            Not After : Apr 11 14:19:23 2030 GMT
[root@etcd-node1 cacenter]# openssl x509 -in client.crt -noout -text | grep 'Not'
            Not Before: Apr 12 14:19:24 2025 GMT
            Not After : Apr 11 14:19:24 2030 GMT
[root@etcd-node1 cacenter]#

可以看到，节点证书和客户端证书有效期都到2030年了！

3. 重新分发证书到各节点 ​

将更新后的证书分发到各节点：

sh
# 分发到node1节点
[root@etcd-node1 cacenter]# scp -p node1.crt client.crt /etc/etcd/ssl

# 分发到node2节点
[root@etcd-node1 cacenter]# scp -p node2.crt client.crt root@192.168.56.122:/etc/etcd/ssl
node2.crt                                                                                                                                                                                           100% 1038     3.3MB/s   00:00    
client.crt                                                                                                                                                                                          100%  977     2.3MB/s   00:00    

# 分发到node3节点
[root@etcd-node1 cacenter]# scp -p node3.crt client.crt root@192.168.56.123:/etc/etcd/ssl
node3.crt                                                                                                                                                                                           100% 1038     1.9MB/s   00:00    
client.crt                                                                                                                                                                                          100%  977     1.5MB/s   00:00    
[root@etcd-node1 cacenter]#

分发后，在三个节点上面查看证书文件：

Snipaste_2025-04-12_22-28-01.png

4. 重启etcd服务 ​

三个节点在/srv/etcd/node目录执行stop.sh脚本，停止etcd服务，然后执行start.sh启动etcd服务：

sh
[root@etcd-node1 cacenter]# cd /srv/etcd/node
[root@etcd-node1 node]# ./stop.sh
[root@etcd-node1 node]# ps -ef|grep etcd
root      1645  1393  0 22:42 pts/0    00:00:00 grep --color=always etcd
[root@etcd-node1 node]# ./start.sh
CLUSTER:node1=https://192.168.56.121:2380,node2=https://192.168.56.122:2380,node3=https://192.168.56.123:2380
[root@etcd-node1 node]# nohup: appending output to ‘nohup.out’

[root@etcd-node1 node]#

5. 检查etcd服务是否正常 ​

使用之前定义的命令行，查看etcd服务是否健康：

sh
[root@etcd-node1 ~]# alias|grep etcd
alias ech='etcdClusterHealth'
alias ecm='etcdClusterMember'
alias ecs='etcdClusterStatus'
alias etcdClusterHealth='etcdctl --write-out=table endpoint health'
alias etcdClusterMember='etcdctl --write-out=table member list'
alias etcdClusterStatus='etcdctl --write-out=table endpoint status'
alias etcdctl='etcdctl --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.crt --cert=/etc/etcd/ssl/client.crt --key=/etc/etcd/ssl/client.key'
[root@etcd-node1 ~]# ech
+-----------------------------+--------+------------+-------+
|          ENDPOINT           | HEALTH |    TOOK    | ERROR |
+-----------------------------+--------+------------+-------+
| https://192.168.56.122:2379 |   true | 8.613939ms |       |
| https://192.168.56.123:2379 |   true | 9.629342ms |       |
| https://192.168.56.121:2379 |   true | 8.297087ms |       |
+-----------------------------+--------+------------+-------+
[root@etcd-node1 ~]# ecm
+------------------+---------+-------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  | NAME  |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+-------+-----------------------------+-----------------------------+------------+
| a7d7b09bf04ad21b | started | node3 | https://192.168.56.123:2380 | https://192.168.56.123:2379 |      false |
| d553b4da699c7263 | started | node2 | https://192.168.56.122:2380 | https://192.168.56.122:2379 |      false |
| e14cb1abc9daea5b | started | node1 | https://192.168.56.121:2380 | https://192.168.56.121:2379 |      false |
+------------------+---------+-------+-----------------------------+-----------------------------+------------+
[root@etcd-node1 ~]# ecs
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.56.121:2379 | e14cb1abc9daea5b |  3.5.18 |   20 kB |     false |      false |         6 |         59 |                 59 |        |
| https://192.168.56.122:2379 | d553b4da699c7263 |  3.5.18 |   20 kB |      true |      false |         6 |         59 |                 59 |        |
| https://192.168.56.123:2379 | a7d7b09bf04ad21b |  3.5.18 |   20 kB |     false |      false |         6 |         59 |                 59 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@etcd-node1 ~]# etcdctl get greeting
greeting
Hello, etcd
[root@etcd-node1 ~]#

Snipaste_2025-04-12_22-47-06.png

此时，可以看到，命令行输出都恢复正常了！

本首页参考 https://notes.fe-mm.com/ 配置而成

Copyright © 2019-2024 阿梅的博客 京ICP备19051502号