开发一个禁止删除 Namespace 的控制器

​大家好，个禁我是止删制器乔克。

昨天收到一个朋友的个禁信息，说不小心把集群的止删制器业务namespace干掉了，导致整个业务都停滞了，个禁问我有没有禁止删除namespace的止删制器方案。

在我的个禁记忆里，Kubernetes的止删制器准入里并没有这个控制器，所以我就给他说需要自己开发一个准入控制器来实现自己的个禁目标。

作为人，止删制器何为正确！个禁我不能只脱裤子，止删制器不放屁。个禁所以这里也整理了一下如何自定义Kubernetes的止删制器准入控制器。

理论介绍

准入控制器（Admission Controller）位于 API Server 中，个禁在对象被持久化之前，准入控制器拦截对 API Server 的请求，一般用来做身份验证和授权。其中包含两个特殊的控制器：MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook。

MutatingAdmissionWebhook ：用于变更请求对象，比如istio为每个Pod注入sidecar，就是通过它实现。ValidatingAdmissionWebhook：用于验证请求对象。服务器托管

整个准入控制器的流程如下：

当 API 请求进入时，mutating 和 validating 控制器使用配置中的外部 webhooks 列表并发调用，规则如下：

如果所有的 webhooks 批准请求，准入控制链继续流转。如果有任意一个 webhooks 阻止请求，那么准入控制请求终止，并返回第一个 webhook 阻止的原因。其中，多个 webhooks 阻止也只会返回第一个 webhook 阻止的原因。如果在调用 webhook 过程中发生错误，那么请求会被终止或者忽略 webhook。

准入控制器是在 API Server 的启动参数中配置的。一个准入控制器可能属于以上两者中的一种，也可能两者都属于。

我们在部署 Kubernetes 集群的时候都会默认开启一系列准入控制器，如果没有设置这些准入控制器的话可以说你的 Kubernetes 集群就是在裸奔，应该叫管理员为集群添加准入控制器。

代码实现

实现逻辑

在开发之前先大致了解一下准入控制器的Webhook的大致实现逻辑：

Webhook是源码下载一个标准的HTTP服务，接收HTTP请求。接收到的请求是一个AdmissionReview对象。然后我们自定义的Hook会处理这个AdmissionReview对象。处理完过后再返回一个AdmissionReview对象，这里面会包含处理结果。

AdmissionReview的结构体如下：

// AdmissionReview describes an admission review request/response.

type AdmissionReview struct {

metav1.TypeMeta `json:",inline"`

// Request describes the attributes for the admission request.

// +optional

Request *AdmissionRequest `json:"request,omitempty" protobuf:"bytes,1,opt,name=request"`

// Response describes the attributes for the admission response.

// +optional

Response *AdmissionResponse `json:"response,omitempty" protobuf:"bytes,2,opt,name=response"`

}

从代码的命名中可以很清晰的看出，在请求发送到 WebHook 时我们只需要关注内部的 AdmissionRequest(实际入参)，在我们编写的 WebHook 处理完成后只需要返回包含有 AdmissionResponse(实际返回体) 的 AdmissionReview 对象即可；总的来说 AdmissionReview 对象是个套壳，请求是里面的 AdmissionRequest，响应是里面的 AdmissionResponse。

具体实现

（1）首先创建一个HTTP Server，监听端口，接收请求。

package main

import (

"context"

"flag"

"github.com/joker-bai/validate-namespace/http"

log "k8s.io/klog/v2"

"os"

"os/signal"

"syscall"

)

var (

tlscert, tlskey, port string

)

func main() {

flag.StringVar(&tlscert, "tlscert", "/etc/certs/cert.pem", "Path to the TLS certificate")

flag.StringVar(&tlskey, "tlskey", "/etc/certs/key.pem", "Path to the TLS key")

flag.StringVar(&port, "port", "8443", "The port to listen")

flag.Parse()

server := http.NewServer(port)

go func() {

if err := server.ListenAndServeTLS(tlscert, tlskey); err != nil {

log.Errorf("Failed to listen and serve: %v", err)

}

}()

log.Infof("Server running in port: %s", port)

// listen shutdown signal

signalChan := make(chan os.Signal, 1)

signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)

<-signalChan

log.Info("Shutdown gracefully...")

if err := server.Shutdown(context.Background()); err != nil {

log.Error(err)

}

}

由于准入控制器和Webhook之间需要使用TLS进行通信，所以上面监听的端口是亿华云TLS端口，通过server.ListenAndServeTLS实现，后续在部署服务的时候需要把证书挂到相应的目录中。

（2）定义Handler，将请求分发到具体的处理方法。

package http

import (

"fmt"

"github.com/joker-bai/validate-namespace/namespace"

"net/http"

)

// NewServer creates and return a http.Server

func NewServer(port string) *http.Server {

// Instances hooks

nsValidation := namespace.NewValidationHook()

// Routers

ah := newAdmissionHandler()

mux := http.NewServeMux()

mux.Handle("/healthz", healthz())

mux.Handle("/validate/delete-namespace", ah.Serve(nsValidation))

return &http.Server{

Addr: fmt.Sprintf(":%s", port),

Handler: mux,

}

}

实现admissionHandler，主要作用是将http body的内容解析成AdmissionReview对象，然后调用具体的Hook处理，再将结果放到AdmissionReview中，返回给客户端。

package http

import (

"encoding/json"

"fmt"

"io"

"net/http"

"github.com/douglasmakey/admissioncontroller"

"k8s.io/api/admission/v1beta1"

admission "k8s.io/api/admission/v1beta1"

meta "k8s.io/apimachinery/pkg/apis/meta/v1"

"k8s.io/apimachinery/pkg/runtime"

"k8s.io/apimachinery/pkg/runtime/serializer"

log "k8s.io/klog/v2"

)

// admissionHandler represents the HTTP handler for an admission webhook

type admissionHandler struct {

decoder runtime.Decoder

}

// newAdmissionHandler returns an instance of AdmissionHandler

func newAdmissionHandler() *admissionHandler {

return &admissionHandler{

decoder: serializer.NewCodecFactory(runtime.NewScheme()).UniversalDeserializer(),

}

}

// Serve returns a http.HandlerFunc for an admission webhook

func (h *admissionHandler) Serve(hook admissioncontroller.Hook) http.HandlerFunc {

return func(w http.ResponseWriter, r *http.Request) {

w.Header().Set("Content-Type", "application/json")

if r.Method != http.MethodPost {

http.Error(w, fmt.Sprint("invalid method only POST requests are allowed"), http.StatusMethodNotAllowed)

return

}

if contentType := r.Header.Get("Content-Type"); contentType != "application/json" {

http.Error(w, fmt.Sprint("only content type application/json is supported"), http.StatusBadRequest)

return

}

body, err := io.ReadAll(r.Body)

if err != nil {

http.Error(w, fmt.Sprintf("could not read request body: %v", err), http.StatusBadRequest)

return

}

var review admission.AdmissionReview

if _, _, err := h.decoder.Decode(body, nil, &review); err != nil {

http.Error(w, fmt.Sprintf("could not deserialize request: %v", err), http.StatusBadRequest)

return

}

if review.Request == nil {

http.Error(w, "malformed admission review: request is nil", http.StatusBadRequest)

return

}

result, err := hook.Execute(review.Request)

if err != nil {

log.Error(err)

w.WriteHeader(http.StatusInternalServerError)

return

}

admissionResponse := v1beta1.AdmissionReview{

Response: &v1beta1.AdmissionResponse{

UID: review.Request.UID,

Allowed: result.Allowed,

Result: &meta.Status{ Message: result.Msg},

},

}

res, err := json.Marshal(admissionResponse)

if err != nil {

log.Error(err)

http.Error(w, fmt.Sprintf("could not marshal response: %v", err), http.StatusInternalServerError)

return

}

log.Infof("Webhook [%s - %s] - Allowed: %t", r.URL.Path, review.Request.Operation, result.Allowed)

w.WriteHeader(http.StatusOK)

w.Write(res)

}

}

func healthz() http.HandlerFunc {

return func(w http.ResponseWriter, r *http.Request) {

w.WriteHeader(http.StatusOK)

w.Write([]byte("ok"))

}

}

上面处理是通过hook.Execute来处理请求，这是admissionController内部实现的一个结构体，它为每个操作定义了一个方法，如下：

// AdmitFunc defines how to process an admission request

type AdmitFunc func(request *admission.AdmissionRequest) (*Result, error)

// Hook represents the set of functions for each operation in an admission webhook.

type Hook struct {

Create AdmitFunc

Delete AdmitFunc

Update AdmitFunc

Connect AdmitFunc

}

我们就需要实现具体的AdmitFunc，并注册。

（3）将自己实现的方法注册到Hook中。

package namespace

import (

"github.com/douglasmakey/admissioncontroller"

)

// NewValidationHook delete namespace validation hook

func NewValidationHook() admissioncontroller.Hook {

return admissioncontroller.Hook{

Delete: validateDelete(),

}

}

（4）实现具体的AdmitFunc。

package namespace

import (

"github.com/douglasmakey/admissioncontroller"

log "k8s.io/klog/v2"

"k8s.io/api/admission/v1beta1"

)

func validateDelete() admissioncontroller.AdmitFunc {

return func(r *v1beta1.AdmissionRequest) (*admissioncontroller.Result, error) {

if r.Kind.Kind == "Namespace" {

log.Info("You cannot delete namespace: ", r.Name)

return &admissioncontroller.Result{ Allowed: false}, nil

} else {

return &admissioncontroller.Result{ Allowed: true}, nil

}

}

}

这里实现很简单，如果Kind为Namespace，就拒绝操作。

部署测试

上面完成了业务逻辑开发，下面就把它部署到Kubernetes集群测试一番。

部署

（1）编写Dockerfile，将应用打包成镜像

FROM golang:1.17.5 AS build-env

ENV GOPROXY https://goproxy.cn

ADD . /go/src/app

WORKDIR /go/src/app

RUN go mod tidy

RUN cd cmd && GOOS=linux GOARCH=amd64 go build -v -a -ldflags -extldflags "-static" -o /go/src/app/app-server /go/src/app/cmd/main.go

FROM registry.cn-hangzhou.aliyuncs.com/coolops/ubuntu:22.04

ENV TZ=Asia/Shanghai

COPY --from=build-env /go/src/app/app-server /opt/app-server

WORKDIR /opt

EXPOSE 80

CMD [ "./app-server" ]

（2）创建TLS证书，使用脚本进行创建。

#!/bin/bash

set -e

usage() {

cat <

Generate certificate suitable for use with an sidecar-injector webhook service.

This script uses k8s CertificateSigningRequest API to a generate a

certificate signed by k8s CA suitable for use with sidecar-injector webhook

services. This requires permissions to create and approve CSR. See

https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for

detailed explantion and additional instructions.

The server key/cert k8s CA cert are stored in a k8s secret.

usage: ${ 0} [OPTIONS]

The following flags are required.

--service Service name of webhook.

--namespace Namespace where webhook service and secret reside.

--secret Secret name for CA certificate and server certificate/key pair.

EOF

exit 1

}

while [[ $# -gt 0 ]]; do

case ${ 1} in

--service)

service="$2"

shift

;;

--secret)

secret="$2"

shift

;;

--namespace)

namespace="$2"

shift

;;

*)

usage

;;

esac

shift

done

[ -z ${ service} ] && service=validate-delete-namespace

[ -z ${ secret} ] && secret=validate-delete-namespace-tls

[ -z ${ namespace} ] && namespace=default

if [ ! -x "$(command -v openssl)" ]; then

echo "openssl not found"

exit 1

fi

csrName=${ service}.${ namespace}

tmpdir=$(mktemp -d)

echo "creating certs in tmpdir ${ tmpdir} "

cat <> ${ tmpdir}/csr.conf

[req]

req_extensions = v3_req

distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]

basicConstraints = CA:FALSE

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

extendedKeyUsage = serverAuth

subjectAltName = @alt_names

[alt_names]

DNS.1 = ${ service}

DNS.2 = ${ service}.${ namespace}

DNS.3 = ${ service}.${ namespace}.svc

EOF

openssl genrsa -out ${ tmpdir}/server-key.pem 2048

openssl req -new -key ${ tmpdir}/server-key.pem -subj "/CN=${ service}.${ namespace}.svc" -out ${ tmpdir}/server.csr -config ${ tmpdir}/csr.conf

# clean-up any previously created CSR for our service. Ignore errors if not present.

kubectl delete csr ${ csrName} 2>/dev/null || true

# create server cert/key CSR and send to k8s API

cat <

apiVersion: certificates.k8s.io/v1beta1

kind: CertificateSigningRequest

metadata:

name: ${ csrName}

spec:

groups:

- system:authenticated

request: $(cat ${ tmpdir}/server.csr | base64 | tr -d \n)

usages:

- digital signature

- key encipherment

- server auth

EOF

# verify CSR has been created

while true; do

kubectl get csr ${ csrName}

if [ "$?" -eq 0 ]; then

break

fi

done

# approve and fetch the signed certificate

kubectl certificate approve ${ csrName}

# verify certificate has been signed

for x in $(seq 10); do

serverCert=$(kubectl get csr ${ csrName} -o jsonpath={ .status.certificate})

if [[ ${ serverCert} != ]]; then

break

fi

sleep 1

done

if [[ ${ serverCert} == ]]; then

echo "ERROR: After approving csr ${ csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2

exit 1

fi

echo ${ serverCert} | openssl base64 -d -A -out ${ tmpdir}/server-cert.pem

# create the secret with CA cert and server cert/key

kubectl create secret generic ${ secret} \

--from-file=key.pem=${ tmpdir}/server-key.pem \

--from-file=cert.pem=${ tmpdir}/server-cert.pem \

--dry-run -o yaml |

kubectl -n ${ namespace} apply -f -

（3）编写Deployment部署服务。

apiVersion: apps/v1

kind: Deployment

metadata:

name: validate-delete-namespace

labels:

app: validate-delete-namespace

spec:

replicas: 1

selector:

matchLabels:

app: validate-delete-namespace

template:

metadata:

labels:

app: validate-delete-namespace

spec:

containers:

- name: server

image: registry.cn-hangzhou.aliyuncs.com/coolops/validate-delete-namespace:latest

imagePullPolicy: Always

livenessProbe:

httpGet:

path: /healthz

port: 8443

scheme: HTTPS

ports:

- containerPort: 8443

volumeMounts:

- name: tls-certs

mountPath: /etc/certs

readOnly: true

volumes:

- name: tls-certs

secret:

secretName: validate-delete-namespace-tls

---

apiVersion: v1

kind: Service

metadata:

name: validate-delete-namespace

spec:

selector:

app: validate-delete-namespace

ports:

- port: 443

targetPort: 8443

（4）部署Webhook

apiVersion: admissionregistration.k8s.io/v1beta1

kind: ValidatingWebhookConfiguration

metadata:

name: validate-delete-namespace

webhooks:

- name: validate-delete-namespace.default.svc.cluster.local

clientConfig:

service:

namespace: default

name: validate-delete-namespace

path: "/validate/delete-namespace"

caBundle: "${ CA_BUNDLE}"

rules:

- operations:

- DELETE

apiGroups:

- ""

apiVersions:

- "v1"

resources:

- namespaces

failurePolicy: Ignore

这里有一个${ CA_BUNDLE}占位符，在创建Webhook的时候要将其替换掉，使用如下命令：

cat ./validate-delete-namespace.yaml | sh ./patch-webhook-ca.sh > ./webhook.yaml

然后创建webhook.yaml即可。

kubectl apply -f webhook.yaml

上面的所有文件都在代码库里，可以直接使用脚本进行部署。

# sh deploy.sh

creating certs in tmpdir /tmp/tmp.SvMHWcPI6x

Generating RSA private key, 2048 bit long modulus

..........................................+++

.............................................................+++

e is 65537 (0x10001)

certificatesigningrequest.certificates.k8s.io/validate-delete-namespace.default created

NAME AGE REQUESTOR CONDITION

validate-delete-namespace.default 0s kubernetes-admin Pending

certificatesigningrequest.certificates.k8s.io/validate-delete-namespace.default approved

secret/validate-delete-namespace-tls created

Creating k8s admission deployment

deployment.apps/validate-delete-namespace created

service/validate-delete-namespace created

validatingwebhookconfiguration.admissionregistration.k8s.io/validate-delete-namespace created

执行完成过后，可以查看具体的信息。

# kubectl get po

NAME READY STATUS RESTARTS AGE

validate-delete-namespace-74c9b8b7bd-5g9zv 1/1 Running 0 3s

# kubectl get secret

NAME TYPE DATA AGE

default-token-kx5wf kubernetes.io/service-account-token 3 72d

validate-delete-namespace-tls Opaque 2 53s

# kubectl get ValidatingWebhookConfiguration

NAME CREATED AT

validate-delete-namespace 2022-06-24T09:39:26Z

测试

（1）首先打开webhook的pod日志。

# kubectl logs validate-delete-namespace-74c9b8b7bd-5g9zv -f

I0624 17:39:27.858753 1 main.go:30] Server running in port: 8443

（2）创建一个namespace并删除。

# kubectl create ns joker

# kubectl get ns | grep joker

joker Active 4h5m

# kubectl delete ns joker

Error from server: admission webhook "validate-delete-namespace.default.svc.cluster.local" denied the request without explanation

# kubectl get ns | grep joker

joker Active 4h5m

可以发现我们的删除操作被拒绝了，并且查看namespace还存在。

我们也可以到日志中查看，如下：

# kubectl logs validate-delete-namespace-74c9b8b7bd-5g9zv -f

I0624 17:39:27.858753 1 main.go:30] Server running in port: 8443

2022/06/24 17:43:34 You cannot delete namespace: joker

I0624 17:43:34.664945 1 handler.go:94] Webhook [/validate/delete-namespace - DELETE] - Allowed: false

2022/06/24 17:43:34 You cannot delete namespace: joker

I0624 17:43:34.667043 1 handler.go:94] Webhook [/validate/delete-namespace - DELETE] - Allowed: false

上面就是简单的实现了一个准入控制器。

参考

https://www.qikqiak.com/post/k8s-admission-webhook

https://github.com/douglasmakey/admissioncontroller

https://mritd.com/2020/08/19/write-a-dynamic-admission-control-webhook/

很赞哦!（49528）