上一篇文章中我们用AppRole方式成功滴使Kubernetes集群在Vault中验证身份，配合安全策略获得了Vault中的CA签发的数字证书。这种方法有个缺点，就是安全令牌（token）是有寿命的，过家家的时候我们可以设置为永不过期，但是生产环境里肯定不能这么搞。这玩意过期了就得有人生成新的，再传递给Kubernetes管理员，去更新信息。
Vault支持的验证方式有很多，Kubernetes验证就是其中之一。
C 实验环境
先列出动手实验的计算机信息:
我在一台运行Debian 12的虚拟机安装的Minikube：它采用默认方式启动容器，也就是Docker。
ℹ️ 再次强烈推荐macOS用户用OrbStack来代替Minikube，又好用又轻👍。
Minikube版本：
1
2
| minikube version: v1.33.1
commit: 5883c09216182566a63dff4c326a6fc9ed2982ff
|
清理（可选）：
1
2
3
| minikube stop
minikube delete
minikube start
|
Docker版本：
1
2
3
4
5
6
| docker version
Client: Docker Engine - Community
Version: 27.1.1
API version: 1.46
Go version: go1.21.12
...
|
Helm是用apt 工具安装的：
1
2
3
4
5
6
| curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
helm version
|
1
| helm versionversion.BuildInfo{Version:"v3.15.3", GitCommit:"3bb50bbbdd9c946ba9989fbe4fb4104766302a64", GitTreeState:"clean", GoVersion:"go1.22.5"}
|
引入Hashicorp官方源：
1
2
| helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
|
验证下：
1
2
3
4
| helm search repo hashicorp/vault
NAME CHART VERSION APP VERSION DESCRIPTION
hashicorp/vault 0.28.1 1.17.2 Official HashiCorp Vault Chart
hashicorp/vault-secrets-operator 0.9.0 0.9.0 Official Vault Secrets Operator Chart
|
安装Vault
在Minikube机器上准备一个文件夹：
1
| mkdir ~/vault-in-minikube && cd ~/vault-in-minikube
|
新安装一个Vault：
1
| helm install vault hashicorp/vault --set "injector.enabled=false"
|
看看状态，可以看到还没好：
1
2
3
| kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 0/1 ContainerCreating 0 14s
|
初始化一下，走个简易的手续就行，安全凭据存在文件里：
1
2
3
4
| kubectl exec vault-0 -- vault operator init \
-key-shares=1 \
-key-threshold=1 \
-format=json > vault-in-k8s-key.json
|
⚠️ 只生成单个解封key是不靠谱的，不要在生产环境里面这么搞，过家家无所谓。
读取解封的key：
1
| VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" vault-in-k8s-key.json)
|
注：没有jq的自己装一下。
解封：
1
| kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY
|
这回应该好了：
1
2
3
| kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 20m
|
读取新装Vault管理员（root）的token：
1
| VAULT_ROOT_TOKEN=$(jq -r ".root_token" vault-in-k8s-key.json)
|
登录咧：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| kubectl exec vault-0 -- vault login $VAULT_ROOT_TOKEN
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token hvs.tLsjfeZiePgde8f5fwAVsCtO
token_accessor OESG7tyqJ9jxm4gQp9TK3vKd
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
|
我们将在此Vault中建立中间CA，并由一个外部的根（root）CA来为它签发中间CA的证书。
建立CA：
1
2
| kubectl exec vault-0 -- vault secrets enable -path=int-in-k8s pki
Success! Enabled the pki secrets engine at: int-in-k8s/
|
随便微调下：
1
2
| kubectl exec vault-0 -- vault secrets tune -max-lease-ttl=8760h int-in-k8s
Success! Tuned the secrets engine at: int-in-k8s/
|
生成CSR：
1
2
3
4
| kubectl exec vault-0 -- vault write -format=json int-in-k8s/intermediate/generate/internal \
common_name="Vault in k8s Intermediate Authority" \
issuer_name="int-in-k8s" \
| jq -r '.data.csr' > int-in-k8s.csr
|
将这个CSR文件复制到根CA那里，我就用之前几篇文章的那个Vault里的根CA。那么就把CSR文件复制到能访问老Vault的计算机里。
1
2
| export VAULT_ADDR='https://vault-dev.miyunda.com' # 引号里面替换成自己的FQDN/IP地址和端口号
vault login
|
签发中间证书：
1
2
3
4
5
| vault write -format=json <secret engine path>/root/sign-intermediate \
issuer_ref="<issuer_name>" \
csr=@int-in-k8s.csr \
format=pem_bundle ttl="87600h" \
| jq -r '.data.certificate' > int-in-k8s.cert.pem
|
假如忘记了issuer_name：
1
2
| vault read <secret engine path>/issuer/$(vault list -format=json <secret engine path>/issuers/ | jq -r '.[]') \
| tail -n 11
|
假如连Secret Engine的路径也忘了:
好奇的话可以看看签发的中间CA证书：
1
| openssl x509 -in int-in-k8s.cert.pem -text
|
把它复制回Minikube计算机，用什么方法都行，前面的CSR和这个证书都不含敏感信息。
然后复制到容器里并导入：
1
| kubectl cp int-in-k8s.cert.pem vault-0:/tmp
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| kubectl exec vault-0 -- vault write int-in-k8s/intermediate/set-signed certificate=@/tmp/int-in-k8s.cert.pem
Key Value
--- -----
existing_issuers <nil>
existing_keys <nil>
imported_issuers [79ab0305-cd2d-cfa7-be78-dcde40a4eeb0 a463f347-7710-aeba-6ee3-cd0ececfb631]
imported_keys <nil>
mapping map[79ab0305-cd2d-cfa7-be78-dcde40a4eeb0:a6a18e58-859b-d058-a770-b0681974852b a463f347-7710-aeba-6ee3-cd0ececfb631:]
WARNING! The following warnings were returned from Vault:
* This mount hasn't configured any authority information access (AIA)
fields; this may make it harder for systems to find missing certificates
in the chain or to validate revocation status of certificates. Consider
updating /config/urls or the newly generated issuer with this information.
|
洁癖患者也许不喜欢容器里面有多余的东西：
1
| kubectl exec vault-0 -- rm /tmp/int-in-k8s.cert.pem
|
把URL设置下：
1
2
3
| kubectl exec vault-0 -- vault write int-in-k8s/config/urls \
issuing_certificates="http://vault.default:8200/v1/int-in-k8s/ca" \
crl_distribution_points="http://vault.default:8200/v1/int-in-k8s/crl"
|
在中间CA下面新建一个角色（role）：
1
2
3
| kubectl exec vault-0 -- vault write int-in-k8s/roles/miyunda-com \
allowed_domains=miyunda.com \
allow_subdomains=true max_ttl=72h
|
还得解决权限问题：
1
| kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh
|
1
2
3
4
5
| vault policy write int-in-k8s-acl - <<EOF
path "int-in-k8s*" { capabilities = ["read", "list"] }
path "int-in-k8s/sign/miyunda-com" { capabilities = ["create", "update"] }
path "int-in-k8s/issue/miyunda-com" { capabilities = ["create"] }
EOF
|
不想给老CA作中间CA的可以建个新的根CA，后面到Kubernetes上记得自己把相应的路径都改下
1
| kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh
|
1
2
3
| vault write ca-root-k8s/root/generate/internal \
common_name=miyunda.com \
ttl=8760h
|
1
2
3
| vault write ca-root-k8s/config/urls \
issuing_certificates="http://vault.default:8200/v1/ca-root-k8s/ca" \
crl_distribution_points="http://vault.default:8200/v1/ca-root-k8s/crl"
|
1
2
3
4
| vault write ca-root-k8s/roles/miyunda-com \
allowed_domains=miyunda.com \
allow_subdomains=true \
max_ttl=72h
|
1
2
3
4
5
| vault policy write ca-root-k8s-acl - <<EOF
path "ca-root-k8s*" { capabilities = ["read", "list"] }
path "ca-root-k8s/sign/miyunda-com" { capabilities = ["create", "update"] }
path "ca-root-k8s/issue/miyunda-com" { capabilities = ["create"] }
EOF
|
配置Kubernetes验证方式
不要退出容器的命令行，继续搞，开Kubernetes验证方式：
1
| vault auth enable kubernetes
|
下一步是告诉Vault去哪找Kubernetes集群的API，因为Vault就运行在Kubernetes集群内部，所以可以轻松地与集群API交流。
1
2
3
| vault write auth/kubernetes/config \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
Success! Data written to: auth/kubernetes/config
|
下一步是在此验证方式内创建一个“角色”（可以理解为一个给机器用的账号）并赋权。
1
2
3
4
5
| vault write auth/kubernetes/role/int-in-k8s-issuer \
bound_service_account_names=int-in-k8s-issuer \
bound_service_account_namespaces=default \
policies=int-in-k8s-acl \
ttl=24h
|
服务账号
退出容器的命令行，在Kubernetes这边建立服务账号：
1
| kubectl create serviceaccount int-in-k8s-issuer
|
1
2
3
4
5
6
7
8
9
| cat >> int-in-k8s-issuer-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
name: issuer-token-abcde
annotations:
kubernetes.io/service-account.name: int-in-k8s-issuer
type: kubernetes.io/service-account-token
EOF
|
下面的secret是给Kubernetes的服务账号用的，里面有个token，Vault会用这个token去访问Kubernetes API。
1
| kubectl apply -f int-in-k8s-issuer-secret.yaml
|
1
2
3
4
| kubectl get secrets
NAME TYPE DATA AGE
issuer-token-abcde kubernetes.io/service-account-token 3 13s
sh.helm.release.v1.vault.v1 helm.sh/release.v1 1 761 17h
|
证书签发
（退出容器的命令行）安装cert-manager，由于上一篇文章我们有过经验，就不解释了：
1
2
3
4
5
6
7
8
| helm repo add jetstack https://charts.jetstack.io --force-update
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.15.3 \
--set crds.enabled=true \
--set prometheus.enabled=false
|
1
| kubectl create namespace cert-manager
|
1
2
3
4
5
| kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-69d959bd45-vpgg8 1/1 Running 0 4m41s
cert-manager-cainjector-5d8798687c-6sh54 1/1 Running 0 4m41s
cert-manager-webhook-c77744d75-q47q4 1/1 Running 0 4m41s
|
1
| ISSUER_SECRET_REF=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("issuer-token-a")).name')
|
和之前几篇文章一样——得让cert-manager知道去哪里申请证书：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| cat > int-in-k8s-issuer.yaml <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: int-in-k8s-issuer
namespace: default
spec:
vault:
server: http://vault.default:8200
path: int-in-k8s/sign/miyunda-com
auth:
kubernetes:
mountPath: /v1/auth/kubernetes
role: int-in-k8s-issuer
secretRef:
name: $ISSUER_SECRET_REF
key: token
EOF
|
1
| kubectl apply -f int-in-k8s-issuer.yaml
|
1
2
3
| kubectl get issuers
NAME READY AGE
int-in-k8s-issuer True 4s
|
尝试手搓一个证书请求：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| cat > hellok8s-miyunda-com.yaml <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: hellok8s-miyunda-com
namespace: default
spec:
secretName: hellok8s-miyunda-com-tls
issuerRef:
name: int-in-k8s-issuer
commonName: hellok8s.miyunda.com
dnsNames:
- hellok8s.miyunda.com
EOF
|
1
| kubectl apply --filename hellok8s-miyunda-com.yaml
|
可以看到新证书已经就绪了：
1
2
3
| kubectl get certificate
NAME READY SECRET AGE
hellok8s-miyunda-com True hellok8s-miyunda-com-tls 6s
|
看看它的详细信息：
1
| kubectl describe certificate hellok8s-miyunda-com
|
也可以让ingress通过cert-manger获取证书，与上一篇文章类似，不多解释了：
1
| minikube addons enable ingress
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| # ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kuard
annotations:
cert-manager.io/issuer: "int-in-k8s-issuer"
cert-manager.io/common-name: "hellonerd.miyunda.com"
spec:
ingressClassName: nginx
tls:
- hosts:
- hellonerd.miyunda.com
secretName: hellonerd-int-in-k8s-tls
rules:
- host: hellonerd.miyunda.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kuard
port:
number: 80
|
1
| kubectl apply -f ingress.yaml
|
1
2
3
4
5
| kubectl get certificate
NAME READY SECRET AGE
hellok8s-miyunda-com True hellok8s-miyunda-com-tls 5m43s
hellonerd-int-in-k8s-tls True hellonerd-int-in-k8s-tls 9s
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
| kubectl describe certificate hellonerd-int-in-k8s-tls
Name: hellonerd-int-in-k8s-tls
Namespace: default
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2024-10-20T14:39:14Z
Generation: 1
Owner References:
API Version: networking.k8s.io/v1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: kuard
UID: 31432297-7621-4b16-bcec-0c81044bb3d5
Resource Version: 5562
UID: b4eb5088-d60f-4739-9393-e64d7e1f1856
Spec:
Common Name: hellonerd.miyunda.com
Dns Names:
hellonerd.miyunda.com
Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: int-in-k8s-issuer
Secret Name: hellonerd-int-in-k8s-tls
Usages:
digital signature
key encipherment
Status:
Conditions:
Last Transition Time: 2024-10-20T14:39:14Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2024-10-23T14:39:14Z
Not Before: 2024-10-20T14:38:44Z
Renewal Time: 2024-10-22T14:39:04Z
Revision: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 74s cert-manager-certificates-trigger Issuing certificate as Secret does not exist
Normal Generated 74s cert-manager-certificates-key-manager Stored new private key in temporary Secret resource "hellonerd-int-in-k8s-tls-ljkzh"
Normal Requested 74s cert-manager-certificates-request-manager Created new CertificateRequest resource "hellonerd-int-in-k8s-tls-1"
Normal Issuing 74s cert-manager-certificates-issuing The certificate has been successfully issued
|
尝试去访问ingress看看，里面的IP地址换成$(minikube ip)也可以：
1
2
3
4
5
| curl -o my-ca.crt https://vault-dev.miyunda.com/v1/ca-root-x1/ca/pem
curl -ivL \
--cacert my-ca.crt \
--resolve hellonerd.miyunda.com:443:192.168.49.2 \
https://hellonerd.miyunda.com
|
返回的结果符合预期：
1
2
3
4
5
6
7
8
9
10
11
| * TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=hellonerd.miyunda.com
* start date: Oct 20 14:38:44 2024 GMT
* expire date: Oct 23 14:39:14 2024 GMT
* subjectAltName: host "hellonerd.miyunda.com" matched cert's "hellonerd.miyunda.com"
* issuer: CN=Vault in k8s Intermediate Authority
* SSL certificate verify ok.
* using HTTP/2
|
以上记录了我在Kubernetes内部运行Vault并签发证书的过程，它不用手动更新安全凭据，减少了管理员的工作量，提高了安全性；但是这种方法的缺点也是明显的：它需要每个Kubernetes集群都安装一个Vault，集群多了就很烦。下一篇文章我们仍然让Vault使用相同的方式验证，但将Vault移出Kubernetes集群。
好了，看过了就等于会了。感谢观看！😽
