Skip to content

Commit ccc60b7

Browse files
author
dongjiang
committed
dongjiang, add custom limitrange
1 parent fce40cc commit ccc60b7

File tree

3,496 files changed

+1138693
-2
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

3,496 files changed

+1138693
-2
lines changed

‎README.md‎

Lines changed: 147 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,147 @@
1-
# custom-limit-range
2-
kubernetest pod bandwidth rate limiting, setting bandwidth quota & custom-limitrange
1+
# Custom Limit Range CRD 使用
2+
3+
## Pod网络限流
4+
5+
### 一、设计文档
6+
7+
设计文档地址: [https://kubeservice.cn/2022/08/10/k8s-pod-bandwidth-limit/](https://kubeservice.cn/2022/08/10/k8s-pod-bandwidth-limit/)
8+
9+
### 二、部署使用步骤
10+
11+
0. Check 底层 `kubelet` 是否配置 `cni` 模式. 目前 `1.21.5是默认开启`的
12+
13+
```bash
14+
$ /usr/bin/kubelet
15+
--cni-bin-dir=/usr/lib/cni/
16+
--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
17+
--kubeconfig=/etc/kubernetes/kubelet.conf
18+
--config=/var/lib/kubelet/config.yaml
19+
--cgroup-driver=cgroupfs
20+
--cni-bin-dir=/opt/cni/bin
21+
--cni-conf-dir=/etc/cni/net.d ## cni-conf-dir设置为cni 配
22+
--network-plugin=cni ## network-plugin设置为cni
23+
...
24+
25+
```
26+
27+
1. Check cni 配置. 目前 `1.21.5是默认开启`
28+
29+
查看`/etc/cni/net.d/`下的 calico 配置
30+
31+
```json
32+
{
33+
"name": "k8s-pod-network",
34+
"cniVersion": "0.4.0",
35+
"plugins": [
36+
{
37+
"type": "calico",
38+
"log_level": "info",
39+
"datastore_type": "kubernetes",
40+
"nodename": "127.0.0.1",
41+
"ipam": {
42+
"type": "host-local",
43+
"subnet": "usePodCidr"
44+
},
45+
"policy": {
46+
"type": "k8s"
47+
},
48+
"kubernetes": {
49+
"kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
50+
}
51+
},
52+
# 设置包含为以下部分
53+
{
54+
"type": "bandwidth",
55+
"capabilities": {"bandwidth": true}
56+
}
57+
]
58+
}
59+
```
60+
61+
2. 部署基础 cert-manager 管理 webhook ca证书。
62+
如果集群中有这部分了， 可以跳过这步骤
63+
64+
```bash
65+
$ kubectl apply -f hack/deployment/certs/cert-manager.yaml
66+
```
67+
68+
确保启动成功
69+
70+
```bash
71+
$ kubectl get pod -A
72+
NAMESPACE NAME READY STATUS RESTARTS AGE
73+
cert-manager cert-manager-6789f4858b-6975h 1/1 Running 5 (158m ago) 4d
74+
cert-manager cert-manager-cainjector-659cdf7887-ftqqw 1/1 Running 44 (157m ago) 4d
75+
cert-manager cert-manager-webhook-69b9966fb-r2hj7 1/1 Running 4 (158m ago) 4d
76+
```
77+
78+
3. 创建 custom limit range ca文件、https pem文件
79+
80+
```bash
81+
$ kubectl apply -f hack/deployment/certs/issuer.yaml
82+
$ kubectl apply -f hack/deployment/certs/certificate.yaml
83+
```
84+
85+
4. 创建CRD文件
86+
87+
```bash
88+
$ kubectl apply -f hack/deployment/crds/custom.cmss.com_customlimitranges.yaml
89+
```
90+
91+
验证CRD创建成功
92+
93+
```bash
94+
$ kubectl get crd
95+
NAME CREATED AT
96+
certificaterequests.cert-manager.io 2022-08-18T03:01:29Z
97+
certificates.cert-manager.io 2022-08-18T03:01:29Z
98+
challenges.acme.cert-manager.io 2022-08-18T03:01:29Z
99+
clusterissuers.cert-manager.io 2022-08-18T03:01:30Z
100+
customlimitranges.custom.cmss.com 2022-08-10T16:04:00Z
101+
issuers.cert-manager.io 2022-08-18T03:01:31Z
102+
orders.acme.cert-manager.io 2022-08-18T03:01:32Z
103+
$ kubectl get crd | grep "customlimitranges"
104+
customlimitranges.custom.cmss.com 2022-08-10T16:04:00Z
105+
```
106+
107+
5. 部署webhook
108+
109+
```bash
110+
$ kubectl apply -f hack/deployment/webhook/
111+
```
112+
113+
验证webhook部署成功：
114+
115+
```bash
116+
$ kubectl get service -n kube-system
117+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
118+
customlimitrange-webhook-service ClusterIP 10.106.25.21 <none> 443/TCP 4d4h
119+
120+
$ kubectl get pod -n kube-system
121+
NAME READY STATUS RESTARTS AGE
122+
customlimitrange-webhook-ffddbd9f9-74kgz 1/1 Running 1 (4h40m ago) 2d19h
123+
```
124+
125+
6. 验证`customlimitrange`生效
126+
在`hack/deployment/example/`中，包括各种case
127+
128+
### 三、 验证注意点
129+
分为`2`部分:
130+
131+
> 基于`Pod`和`Deployment`添加`annotations` 方式验证
132+
>
133+
> 通过namespace添加`CustomLimitRange`, 设置上下限Max/Min、添加默认Default 方式
134+
135+
验证细节：
136+
137+
> 1）ingress 生效；
138+
>
139+
> 2）egress 生效；
140+
141+
1. ingress: client -> server. server端入口流量限流成功
142+
2. egress: client -> server. client端出口流量限流成功
143+
144+
验证工具，设计方案中的验证部分：
145+
[https://kubeservice.cn/2022/08/10/k8s-pod-bandwidth-limit/](https://kubeservice.cn/2022/08/10/k8s-pod-bandwidth-limit/)
146+
147+

‎cmd/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
cmd
2+
certs/certs

‎cmd/certs/main.go‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
/*
2+
Copyright 2022 The KubeService-Stack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package main
18+
19+
import (
20+
"bytes"
21+
cryptorand "crypto/rand"
22+
"crypto/rsa"
23+
"crypto/x509"
24+
"crypto/x509/pkix"
25+
"encoding/pem"
26+
"fmt"
27+
"math/big"
28+
"os"
29+
"path/filepath"
30+
"time"
31+
32+
log "github.com/sirupsen/logrus"
33+
34+
"github.com/jessevdk/go-flags"
35+
)
36+
37+
var opts struct {
38+
Namespace string `long:"namespace" short:"n" env:"NAMESPACE" default:"kube-system" description:"The namespace where the customlimitrange webhook is deployed"`
39+
ServiceName string `long:"service-name" short:"s" env:"SERVICE_NAME" default:"customlimitrange-webhook-service" description:"Name of the service object for the customlimitrange webhook"`
40+
CertFile string `lond:"cert" short:"c" env:"CERT" default:"/etc/webhook/certs/tls.crt" description:"Path to the cert file"`
41+
KeyFile string `lond:"key" short:"k" env:"KEY" default:"/etc/webhook/certs/tls.key" description:"Path to the key file"`
42+
}
43+
44+
func main() {
45+
flags.Parse(&opts)
46+
47+
var caPEM, serverCertPEM, serverPrivKeyPEM *bytes.Buffer
48+
// CA config
49+
ca := &x509.Certificate{
50+
SerialNumber: big.NewInt(2020),
51+
Subject: pkix.Name{
52+
Organization: []string{"cmss.com"},
53+
},
54+
NotBefore: time.Now(),
55+
NotAfter: time.Now().AddDate(1, 0, 0),
56+
IsCA: true,
57+
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
58+
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
59+
BasicConstraintsValid: true,
60+
}
61+
62+
// CA private key
63+
caPrivKey, err := rsa.GenerateKey(cryptorand.Reader, 4096)
64+
if err != nil {
65+
log.Fatalf("Failed to generate CA private key: %s", err)
66+
}
67+
68+
// Self signed CA certificate
69+
caBytes, err := x509.CreateCertificate(cryptorand.Reader, ca, ca, &caPrivKey.PublicKey, caPrivKey)
70+
if err != nil {
71+
log.Fatalf("Failed to create self-signed CA cert: %s", err)
72+
}
73+
74+
// PEM encode CA cert
75+
caPEM = new(bytes.Buffer)
76+
_ = pem.Encode(caPEM, &pem.Block{
77+
Type: "CERTIFICATE",
78+
Bytes: caBytes,
79+
})
80+
81+
commonName := fmt.Sprintf("%s.%s.svc", opts.ServiceName, opts.Namespace)
82+
dnsNames := []string{
83+
opts.ServiceName,
84+
fmt.Sprintf("%s.%s", opts.ServiceName, opts.Namespace),
85+
commonName,
86+
fmt.Sprintf("%s.%s.svc.cluster.local", opts.ServiceName, opts.Namespace),
87+
}
88+
89+
// server cert config
90+
cert := &x509.Certificate{
91+
DNSNames: dnsNames,
92+
SerialNumber: big.NewInt(1658),
93+
Subject: pkix.Name{
94+
CommonName: commonName,
95+
Organization: []string{"cmss.com"},
96+
},
97+
NotBefore: time.Now(),
98+
NotAfter: time.Now().AddDate(1, 0, 0),
99+
SubjectKeyId: []byte{1, 2, 3, 4, 6},
100+
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
101+
KeyUsage: x509.KeyUsageDigitalSignature,
102+
}
103+
104+
// server private key
105+
serverPrivKey, err := rsa.GenerateKey(cryptorand.Reader, 4096)
106+
if err != nil {
107+
log.Fatalf("Failed to generate server private key: %s", err)
108+
}
109+
110+
// sign the server cert
111+
serverCertBytes, err := x509.CreateCertificate(cryptorand.Reader, cert, ca, &serverPrivKey.PublicKey, caPrivKey)
112+
if err != nil {
113+
log.Fatalf("Failed to create server cert: %s", err)
114+
}
115+
116+
// PEM encode the server cert and key
117+
serverCertPEM = new(bytes.Buffer)
118+
_ = pem.Encode(serverCertPEM, &pem.Block{
119+
Type: "CERTIFICATE",
120+
Bytes: serverCertBytes,
121+
})
122+
123+
serverPrivKeyPEM = new(bytes.Buffer)
124+
_ = pem.Encode(serverPrivKeyPEM, &pem.Block{
125+
Type: "RSA PRIVATE KEY",
126+
Bytes: x509.MarshalPKCS1PrivateKey(serverPrivKey),
127+
})
128+
129+
err = writeFile(opts.CertFile, serverCertPEM)
130+
if err != nil {
131+
log.Fatalf("Failed to write certificate: %s", err)
132+
}
133+
134+
err = writeFile(opts.KeyFile, serverPrivKeyPEM)
135+
if err != nil {
136+
log.Fatalf("Failed to write key: %s", err)
137+
}
138+
}
139+
140+
func writeFile(path string, sCert *bytes.Buffer) error {
141+
err := os.MkdirAll(filepath.Dir(path), 0777)
142+
if err != nil {
143+
return err
144+
}
145+
146+
f, err := os.Create(path)
147+
if err != nil {
148+
return err
149+
}
150+
defer f.Close()
151+
152+
_, err = f.Write(sCert.Bytes())
153+
if err != nil {
154+
return err
155+
}
156+
return nil
157+
}

0 commit comments

Comments
 (0)