雲和安全管理服務專家新鈦雲服 劉川川翻譯
概述
隨著我們對 Kubernetes 的逐步了解，可能就會發現 Kubernetes 中內置的對象定義，比如 Deployment、StatefulSet、ConfigMap，可能已經不能滿足我們的需求。我們希望在 Kubernetes 定義一些自己的對象，一是可以通過 kube-apiserver 提供統一的訪問入口，二是可以像其他內置對象一樣，通過 kubectl 命令管理這些自定義的對象。
Kubernetes 中提供了這種自定義對象的方式，其中之一就是 CRD。
CRD 介紹
CRD（CustomResourceDefinitions）在 v1.7 剛引入進來的時候，其實是 ThirdPartyResources（TPR）的升級版本，而 TPR 在 v1.8 的版本被剔除了。CRD 目前使用非常廣泛，各個周邊項目都在使用它，比如 Ingress、Rancher 等。
我們來看一下官方提供的一個例子，通過如下的 YAML 文件，我們可以創建一個 API：
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
# 名字須與下面的 spec 欄位匹配，並且格式為 '<名稱的複數形式>.<組名>'
name: crontabs.stable.example.com
spec: # 組名稱，用於 REST API: /apis/<組>/<版本>
group: stable.example.com
# 列舉此 CustomResourceDefinition 所支持的版本
versions:
- name: v1
# 每個版本都可以通過 served 標誌來獨立啟用或禁止
served: true
# 其中一個且只有一個版本必需被標記為存儲版本
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
image:
type: string
replicas:
type: integer
# 可以是 Namespaced 或 Cluster
scope: Namespaced
names:
# 名稱的複數形式，用於 URL：/apis/<組>/<版本>/<名稱的複數形式>
plural: CronTabs
# 名稱的單數形式，作為命令行使用時和顯示時的別名
singular: crontab
# kind 通常是單數形式的駝峰編碼（CamelCased）形式。我們的資源清單會使用這一形式。
kind: CronTab
# shortNames 允許我們在命令行使用較短的字符串來匹配資源
shortNames:
- ct這樣我們就可以像創建其他對象一樣，通過 kubectl create 命令創建。創建完成以後，一個類型為 CronTab 的對象就在 kube-apiserver 中註冊好了，我們可以通過如下的 REST 接口訪問，比如查看命名空間 ns1 下的 CronTab 對象，可以通過這個 URL 「/apis/stable.example.com/v1/namespaces/ns1/crontabs/」 訪問。這種接口跟 Kubernetes 內置的其他對象的接口風格是一模一樣的。
聲明好了 CronTab，我們就來看看如何創建一個 CronTab 類型的對象。下面依然是來自官方的一個例子：
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: new-cron-object
spec:
cronSpec: "* * * * */5"
image: awesome-cron-image通過 kubectl create 創建 new-cron-object 後，就可以通過 kubectl get 查看，並使用 kubectl 管理這個 CronTab 對象了。例如：
kubectl get crontab
NAME AGE
new-cron-object 6s這裡的資源名是大小寫不敏感的，我們在這裡可以使用縮寫 kubectl get ct，也可以使用 kubectl get crontabs。同時原生內置的 API 對象一樣，這些 CRD 不僅可以通過 kubectl 來創建、查看、修改，刪除等操作，還可以給其配置 RBAC 規則。
我們還可以開發自定義的控制器，來感知和操作這些自定義的 API。接下來我們就開始介紹。可以參考 定製資源 | Kubernetes（https://kubernetes.io/zh/docs/concepts/extend-kubernetes/api-extension/custom-resources/#我是否應該向我的-kubernetes-集群添加定製資源） 這份說明確定是否需要在 Kubernetes 中定義 API，還是讓我們的 API 獨立運行。
什麼是 Kubernetes Operator
我們可能對 Operator 這個名字比較陌生。這個名字最早由 CoreOS（https://coreos.com/operators/） 在 2016 年提出來，我們來看看他們給出的定義：
An operator is a method of packaging, deploying and managing a Kubernetes application. A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling.
To be able to make the most of Kubernetes, you need a set of cohensive APIs to extend in order to service and manage your applications that run on Kubernetes. You can think of Operators as the runtime that manages this type of application on Kubernetes.
簡單概括一下，所謂的 Kubernetes Operator 其實就是藉助 Kubernetes 的控制器模式，配合一些自定義的 API，完成對某一類應用的操作，比如資源創建、變更、刪除等操作。
這裡對 Kubernetes 的控制器模式做個簡要說明。Kubernetes 通過聲明式 API 來定義對象，各個控制器負責實時查看對應對象的狀態，確保達到定義的期望狀態。這就是 Kubernetes 的控制器模式。
kube-controller-manager 就是由這樣一組控制器組成的。我們以 StatefulSet 為例來簡單說明下控制器的具體邏輯。
假設我們聲明了一個 StatefulSet，並將其副本數設置為 3。kube-controller-manager 中以 goroutine 方式運行的 StatefulSet 控制器在觀察 kube-apiserver 的時候，發現了這個新創建的對象，它會先創建一個 index 為 0 的 Pod ，並實時觀察這個 Pod 的狀態，待其狀態變為 Running 後，再創建 index 為 1 的 Pod。後續該控制器會一直觀察並維護這些 Pod 的狀態，保證 StatefulSet 的有效副本數始終為 3。
所以我們在聲明完成 CRD 之後，也需要創建一個控制器，即 Operator，來完成對應的控制邏輯。在了解了 Operator 的概念和控制器模式後，我們來看看 Operator 是如何工作的。
Kubernetes Operator 是如何工作的
Operator 工作的時候採用上述的控制器模式，會持續地觀察 Kubernetes 中的自定義對象，即 CR（Custom Resource）。我們通過 CRD 來定義一個對象，CR 則是 CRD 實例化的對象。
Operator 會持續跟蹤這些 CR 的變化事件，比如 ADD、UPDATE、DELETE，然後採取一系列操作，使其達到期望的狀態。上述的流程其實還是有些複雜的，尤其是對運維同學有一定的門檻。好在社區提供了一些腳手架，可以方便我們快速地構建自己的 Operator。
構建一個自己的 Kubernetes Operator
目前社區有一些可以用於創建 Kubernetes Operator 的開源項目，例如：Operator SDK（https://github.com/operator-framework/operator-sdk）、Kubebuilder（https://github.com/kubernetes-sigs/kubebuilder）、KUDO（https://github.com/kudobuilder/kudo）。我們這裡以 Operator SDK 為例，接下來就安裝 Operator SDK。
二進位安裝 Operator SDK
前提條件
- curl(https://curl.haxx.se/)
- gpg(https://gnupg.org/) version 2.0+
- 版本信息請參考：kubernetes/client-go: Go client for Kubernetes（https://github.com/kubernetes/client-go#compatibility-matrix）. (github.com)
1、下載二進位文件
設置平台信息：
[root@blog ~]# export ARCH=$(case $(uname -m) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; *) echo -n $(uname -m) ;; esac)
[root@blog ~]# export OS=$(uname | awk '{print tolower($0)}')下載指定的文件：
[root@blog ~]# export OPERATOR_SDK_DL_URL=https://github.com/operator-framework/operator-sdk/releases/download/v1.16.0
[root@blog ~]# curl -LO ${OPERATOR_SDK_DL_URL}/operator-sdk_${OS}_${ARCH}2、驗證已下載的文件（可選）
從 keyserver.ubuntu.com 導入 operator-sdk 發行版的 GPG key :
[root@blog ~]# gpg --keyserver keyserver.ubuntu.com --recv-keys 052996E2A20B5C7E下載 checksums 文件及其簽名，然後驗證簽名：
[root@blog ~]# curl -LO ${OPERATOR_SDK_DL_URL}/checksums.txt
[root@blog ~]# curl -LO ${OPERATOR_SDK_DL_URL}/checksums.txt.asc
[root@blog ~]# gpg -u "Operator SDK (release) <cncf-operator-sdk@cncf.io>" --verify checksums.txt.asc我們會看到一些類似下面的一些輸出信息：
gpg: assuming signed data in 'checksums.txt'
gpg: Signature made Fri 30 Oct 2020 12:15:15 PM PDT
gpg: using RSA key ADE83605E945FA5A1BD8639C59E5B47624962185
gpg: Good signature from "Operator SDK (release) <cncf-operator-sdk@cncf.io>" [ultimate]確保 checksums 匹配：
[root@blog ~]# grep operator-sdk_${OS}_${ARCH} checksums.txt | sha256sum -c -
operator-sdk_linux_amd64: OK確保類似下面的輸出信息：
operator-sdk_linux_amd64: OK3、把二進位文件放到 PATH 下面
[root@blog ~]# chmod +x operator-sdk_${OS}_${ARCH} && sudo mv operator-sdk_${OS}_${ARCH} /usr/local/bin/operator-sdk源碼編譯安裝 Operator SDK
前提條件
- git（https://git-scm.com/downloads）
- go version 1.16+
- 確保 goPROXY 設置為 "https://goproxy.cn"
[root@blog ~]# export GO111MODULE=on
[root@blog ~]# export GOPROXY=https://goproxy.cn
[root@blog ~]# git clone https://github.com/operator-framework/operator-sdk
[root@blog ~]# cd operator-sdk
[root@blog operator-sdk]# make install驗證版本：
[root@blog operator-sdk]# operator-sdk version
operator-sdk version: "v1.16.0", commit: "560044140c4f3d88677e4ef2872931f5bb97f255", kubernetes version: "1.21", go version: "go1.16.13", GOOS: "linux", GOARCH: "amd64"
# 由上述命令的輸出來看，我們應該可以看出要使用的版本信息。
# 如我們使用的 operator-sdk 版本為：v1.16.0
# Go 的版本為：1.16.13
# Kubernetes 版本為：1.21
[root@blog operator-sdk]# go version
go version go1.16.13 linux/amd64通過上述任何一種形式，就可以完成基礎環境的搭建。接下來我們就創建一個 Operator。我們可以使用 Ansible、Helm 及 Go 結合 SDK 創建 Operator，使用 Ansible 及 Helm 的形式相對簡單些。本文將使用 Go 的形式及 Operator SDK 來進行演示。
使用 Go 創建 Operator
Operator SDK 提供以下工作流來開發一個新的 Operator：
- 使用 SDK 創建一個新的 Operator 項目
- 通過添加自定義資源（CRD）定義新的資源 API
- 指定使用 SDK API 來 watch 的資源
- 定義 Operator 的協調（reconcile）邏輯
- 使用 Operator SDK 構建並生成 Operator 部署清單文件
前提條件
- 參照前面的介紹進行安裝 operator-sdk。
- 要有 cluster-admin 權限。
- 一個可以訪問的 Operator 鏡像（例如 hub.docker.com、quay.io），並可以在命令行環境中登錄。
- example.com 本例中在 Dockers Hub 上的一個命名空間。如果我們使用其他 registry 或命名空間的話，請相應的替換掉即可。
- 如果 registry 是私有的，請準備好相關的認證或證書。
接下來我們就會按照下面的流程創建一個工程：
- 如果不存在 memcached Deployment 就創建一個
- 確保 Deployment 中的 size 與 Memcached CR 中的 size一致
- 使用帶有 CR pod 名稱的狀態寫入器更新 Memcached CR 的狀態
創建工程
接下來使用命令行工具創建一個名為 memcached-operator 的工程：
[root@blog operator-sdk]# mkdir /root/memcached-operator
[root@blog operator-sdk]# cd /root/memcached-operator
[root@blog memcached-operator]# export GO111MODULE=on && export GOPROXY=https://goproxy.cn
[root@blog memcached-operator]# operator-sdk init \
--domain example.com \--repo github.com/example/memcached-operator
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.10.0
Update dependencies:
$ go mod tidy
Next: define a resource with:
$ operator-sdk create api創建完成之後，我們看一下代碼的目錄結構：
[root@blog memcached-operator]# tree -L 2
.
├── config
│ ├── default
│ ├── manager
│ ├── manifests
│ ├── prometheus
│ ├── rbac
│ └── scorecard
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
├── main.go
├── Makefile
└── PROJECT
8 directories, 7 filesoperator-sdk init 生成了一個 go.mod 文件。當我們的工程不在 $GOPATH/src 下面，則 --repo=<path> 選項是必須的，因為腳手架需要一個有效的 module 路徑。
在使用 SDK 前，我們要確保開啟了模塊支持。需要設置：export GO111MODULE=on。為了加速下載 Go 的依賴，需要設置合適的代理。如：export GOPROXY=https://goproxy.cn
此時，我們可以使用 go build 命令構建：
[root@blog memcached-operator]# go build
[root@blog memcached-operator]# ll
total 44788
drwx------ 8 root root 100 Mar 30 21:01 config
-rw------- 1 root root 776 Mar 30 20:59 Dockerfile
-rw------- 1 root root 162 Mar 30 21:01 go.mod
-rw-r--r-- 1 root root 77000 Mar 30 21:01 go.sum
drwx------ 2 root root 32 Mar 30 20:59 hack
-rw------- 1 root root 2780 Mar 30 20:59 main.go
-rw------- 1 root root 9449 Mar 30 21:01 Makefile
-rwxr-xr-x 1 root root 45754092 Mar 30 21:02 memcached-operator
-rw------- 1 root root 235 Mar 30 21:01 PROJECT目錄結構中，還有一個 PROJECT 的文件，我們看看它裡面有什麼內容。
[root@blog memcached-operator]# cat PROJECT
domain: example.com
layout:
- go.kubebuilder.io/v3
plugins:
manifests.sdk.operatorframework.io/v2: {}
scorecard.sdk.operatorframework.io/v2: {}
projectName: memcached-operator
repo: github.com/example/memcached-operator
version: "3"它主要是一些我們工程的配置信息。
Manager（管理器）
Operator 的主代碼 main.go 主要是初始化並運行 Manager（https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#Manager）.有關管理器如何為自定義資源 API 定義註冊 Scheme 以及設置和運行的更多詳細信息，請參閱 Kubebuilder 入口文檔（https://book.kubebuilder.io/cronjob-tutorial/empty-main.html） 控制器和 webhook。Manager 可以限制所有控制器監視資源的命名空間：
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
HealthProbeBindAddress: probeAddr,
LeaderElection: enableLeaderElection,
LeaderElectionID: "86f835c3.my.domain",
})當然，我們也可以使用 MultiNamespacedCacheBuilder 來 watch 一組 namespace：
var namespaces []string // List of Namespaces
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
NewCache: cache.MultiNamespacedCacheBuilder(namespaces),
MetricsBindAddress: fmt.Sprintf("%s:%d", metricsHost, metricsPort),
Port: 9443,
HealthProbeBindAddress: probeAddr,
LeaderElection: enableLeaderElection,
LeaderElectionID: "86f835c3.my.domain",
})對於更新詳細的信息，我們可以閱讀 MultiNamespacedCacheBuilder （https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/cache?tab=doc#MultiNamespacedCacheBuilder）文檔。
創建 API 及 Controller（控制器）
接下來使用 group 名為 cache， 版本為 v1alpha1 和 Kind 為 Memcached 創建一個新的自定義資源定義 (CRD) API。
[root@blog memcached-operator]# operator-sdk create api \
--group cache \
--version v1alpha1 \
--kind Memcached \
--resource \
--controller
# 下面是上述命令的輸出
Writing scaffold for you to edit...
api/v1alpha1/memcached_types.go
controllers/memcached_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.7.0 # 下載了 controller-gen 文件
go: downloading sigs.k8s.io/controller-tools v0.7.0
go: downloading golang.org/x/tools v0.1.5
go: downloading k8s.io/apimachinery v0.22.2
go: downloading k8s.io/api v0.22.2
go: downloading k8s.io/apiextensions-apiserver v0.22.2
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading golang.org/x/sys v0.0.0-20210616094352-59db8d763f22
go: downloading k8s.io/utils v0.0.0-20210819203725-bdf08cb9a70a
go: downloading golang.org/x/mod v0.4.2
go get: added sigs.k8s.io/controller-tools v0.7.0
/root/memcached-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests再次查看 PROJECT 文件：
[root@blog memcached-operator]# cat PROJECT
domain: example.com
layout:
- go.kubebuilder.io/v3
plugins:
manifests.sdk.operatorframework.io/v2: {}
scorecard.sdk.operatorframework.io/v2: {}
projectName: memcached-operator
repo: github.com/example/memcached-operator
resources:
- api:
crdVersion: v1
namespaced: true
controller: true
domain: example.com
group: cache
kind: Memcached
path: github.com/example/memcached-operator/api/v1alpha1
version: v1alpha1
version: "3"上述的操作將會生成 Memcached resource API 文件，其文件位於 api/v1alpha1/memcached_types.go 及控制器文件位於 controllers/memcached_controller.go 文件中。
注意：本文只介紹了單組 API 的使用。如果我們想支持多組 API，請參考 Single Group to Multi-Group （https://book.kubebuilder.io/migration/multi-group.html）文檔。
這個時候我們在看一下目錄結構：
[root@blog memcached-operator]# tree -L 2
.
├── api
│ └── v1alpha1
├── bin
│ └── controller-gen
├── config
│ ├── crd
│ ├── default
│ ├── manager
│ ├── manifests
│ ├── prometheus
│ ├── rbac
│ ├── samples
│ └── scorecard
├── controllers
│ ├── memcached_controller.go
│ └── suite_test.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
├── main.go
├── Makefile
└── PROJECT
14 directories, 10 files理解 Kubernetes 的 APIs
有關 Kubernetes API 和 group-version-kind 模型的深入解讀，我們可以查看這些 kubebuilder docs （https://book.kubebuilder.io/cronjob-tutorial/gvks.html）文檔。一般來說，建議讓一個控制器負責管理工程的每個 API，以遵循 controller-runtime （https://github.com/kubernetes-sigs/controller-runtime）設定的設計目標。
定義 API
首先，我們將通過定義 「Memcached」 類型來表示我們的 API，該類型有一個 「MemcachedSpec.Size」 欄位來設置要部署的 memcached 實例（CR）的數量，以及一個 「MemcachedStatus.Nodes」 欄位來存儲 CR 的 Pod 名稱。
注意： 這裡 Node 欄位只是用於演示 Status 欄位。在實際情況下，建議使用 Conditions （https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties）.
接下來修改 api/v1alpha1/memcached_types.go 中的 Go 類型定義，為 Memcached 自定義資源（CR）定義 API，使其具有以下規格和狀態：
// MemcachedSpec defines the desired state of Memcachedtype MemcachedSpec struct {
//+kubebuilder:validation:Minimum=0
// Size is the size of the memcached deploymentSize int32 `json:"size"`
}
// MemcachedStatus defines the observed state of Memcachedtype MemcachedStatus struct {
// Nodes are the names of the memcached pods
Nodes []string `json:"nodes"`
}接下來添加 +kubebuilder:subresource:statusmarker（https://book.kubebuilder.io/reference/generating-crd.html#status） 以添加 status subresource （https://kubernetes.io/ docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#status-subresource）到 CRD 清單，以便控制器可以在不更改 CR 對象的其餘部分的情況下更新 CR 狀態：
// Memcached is the Schema for the memcacheds API
//+kubebuilder:subresource:status // 增加此行type Memcached struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec MemcachedSpec `json:"spec,omitempty"`
Status MemcachedStatus `json:"status,omitempty"`
}修改 *_types.go 文件後，記得要運行以下命令來為該資源類型生成代碼：
[root@blog memcached-operator]# make generate
/root/memcached-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."上面的 makefile 的 generate 目標將調用 controller-gen（https://sigs.k8s.io/controller-tools）實用程序來更新 api/v1alpha1/zz_generated.deepcopy.go 文件以確保我們 API 的 Go 類型定義實現所有 Kind 類型必須實現的 runtime.Object 接口。
生成 CRD 清單
一旦使用 spec/status 欄位和 CRD 驗證標記定義 API 後，可以使用以下命令生成和更新 CRD 清單：
[root@blog memcached-operator]# make manifests
/root/memcached-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases這個 makefile 的 manifests 目標將調用 controller-gen 在 config/crd/bases/cache.example.com_memcacheds.yaml 文件中生成 CRD 清單。
驗證 OpenAPI
CRD 中定義的 OpenAPI 驗證可確保 CR 基於一組聲明性規則進行驗證。所有 CRD 都應該有驗證。有關詳細信息，請參閱 OpenAPI 驗證(https://sdk.operatorframework.io/docs/building-operators/golang/references/openapi-validation) 文檔。
實現 Controller
對於此示例，將生成的控制器文件 controllers/memcached_controller.go 替換為示例 memcached_controller.go 文件。其代碼如下：
/*
Copyright 2022.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/package controllers
import (
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
"reflect"
"time"
"context"
"k8s.io/apimachinery/pkg/runtime"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
ctrllog "sigs.k8s.io/controller-runtime/pkg/log"
cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
)
// MemcachedReconciler reconciles a Memcached object
type MemcachedReconciler struct {
client.Client
Scheme *runtime.Scheme
}
//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch
// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Memcached object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.0/pkg/reconcile
func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
log := ctrllog.FromContext(ctx)
// Fetch the Memcached instance
memcached := &cachev1alpha1.Memcached{}
err := r.Get(ctx, req.NamespacedName, memcached)
if err != nil {
if errors.IsNotFound(err) {
// Request object not found, could have been deleted after reconcile request. // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers. // Return and don't requeue
log.Info("Memcached resource not found. Ignoring since object must be deleted")
return ctrl.Result{}, nil
}
// Error reading the object - requeue the request.
log.Error(err, "Failed to get Memcached")
return ctrl.Result{}, err
}
// Check if the deployment already exists, if not create a new one
found := &appsv1.Deployment{}
err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
if err != nil && errors.IsNotFound(err) {
// Define a new deployment
dep := r.deploymentForMemcached(memcached)
log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
err = r.Create(ctx, dep)
if err != nil {
log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
return ctrl.Result{}, err
}
// Deployment created successfully - return and requeue
return ctrl.Result{Requeue: true}, nil
} else if err != nil {
log.Error(err, "Failed to get Deployment")
return ctrl.Result{}, err
}
// Ensure the deployment size is the same as the spec
size := memcached.Spec.Size
if *found.Spec.Replicas != size {
found.Spec.Replicas = &size
err = r.Update(ctx, found)
if err != nil {
log.Error(err, "Failed to update Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)
return ctrl.Result{}, err
}
// Ask to requeue after 1 minute in order to give enough time for the
// pods be created on the cluster side and the operand be able
// to do the next update step accurately.
return ctrl.Result{RequeueAfter: time.Minute}, nil
}
// Update the Memcached status with the pod names
// List the pods for this memcached's deployment
podList := &corev1.PodList{}
listOpts := []client.ListOption{
client.InNamespace(memcached.Namespace),
client.MatchingLabels(labelsForMemcached(memcached.Name)),
}
if err = r.List(ctx, podList, listOpts...); err != nil {
log.Error(err, "Failed to list pods", "Memcached.Namespace", memcached.Namespace, "Memcached.Name", memcached.Name)
return ctrl.Result{}, err
}
podNames := getPodNames(podList.Items)
// Update status.Nodes if needed
if !reflect.DeepEqual(podNames, memcached.Status.Nodes) {
memcached.Status.Nodes = podNames
err := r.Status().Update(ctx, memcached)
if err != nil {
log.Error(err, "Failed to update Memcached status")
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}
// deploymentForMemcached returns a memcached Deployment object
func (r *MemcachedReconciler) deploymentForMemcached(m *cachev1alpha1.Memcached) *appsv1.Deployment {
ls := labelsForMemcached(m.Name)
replicas := m.Spec.Size
dep := &appsv1.Deployment{
ObjectMeta: metav1.ObjectMeta{
Name: m.Name,
Namespace: m.Namespace,
},
Spec: appsv1.DeploymentSpec{
Replicas: &replicas,
Selector: &metav1.LabelSelector{
MatchLabels: ls,
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: ls,
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{{
Image: "memcached:1.4.36-alpine",
Name: "memcached",
Command: []string{"memcached", "-m=64", "-o", "modern", "-v"},
Ports: []corev1.ContainerPort{{
ContainerPort: 11211,
Name: "memcached",
}},
}},
},
},
},
}
// Set Memcached instance as the owner and controller
ctrl.SetControllerReference(m, dep, r.Scheme)
return dep
}
// labelsForMemcached returns the labels for selecting the resources
// belonging to the given memcached CR name.
func labelsForMemcached(name string) map[string]string {
return map[string]string{"app": "memcached", "memcached_cr": name}
}
// getPodNames returns the pod names of the array of pods passed in
func getPodNames(pods []corev1.Pod) []string {
var podNames []string
for _, pod := range pods {
podNames = append(podNames, pod.Name)
}
return podNames
}
// SetupWithManager sets up the controller with the Manager.
func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&cachev1alpha1.Memcached{}).
Owns(&appsv1.Deployment{}).
Complete(r)
}注意: 接下來的兩個小節將解釋控制器如何 watch 資源以及如何觸發 reconcile 循環。如果想跳過此部分，可以查看此文檔的《運行 Operator》章節，查看如何運行此 operator。
Controller watch 的資源
controllers/memcached_controller.go 中的 SetupWithManager() 函數指定了如何構建控制器以監視 CR 和該控制器擁有和管理的其他資源。
import (
...
appsv1"k8s.io/api/apps/v1"
...
)
func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&cachev1alpha1.Memcached{}).
Owns(&appsv1.Deployment{}).
Complete(r)
}NewControllerManagedBy() 提供了一個控制器構建器，允許各種控制器的配置。
For(&cachev1alpha1.Memcached{}) 將 Memcached 類型指定為要監視的主要資源。對於每個 Memcached 類型的 Add/Update/Delete 事件，reconcile loop 將為該 Memcached 對象發送一個 reconcile Request（命名空間/key 名稱）。
Owns(&appsv1.Deployment{}) 將 Deployments 類型指定為要 watch 的輔助資源。對於每個 Deployment 類型的添加/更新/刪除事件，事件處理程序會將每個事件映射到部署所有者的 reconcile 「請求」。在這種情況下，是為其創建 Deployment 的 Memcached 對象。
Controller 配置
在初始化控制器時可以進行許多其他有用的配置。有關這些配置的更多詳細信息，可以查看上游 builder 和 controller 的幫助文檔。
- 通過 MaxConcurrentReconciles 選項設置控制器的最大並發 Reconciles 數。默認為 1。
- func (r *MemcachedReconciler)
SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&cachev1alpha1.Memcached{}).
Owns(&appsv1.Deployment{}).
WithOptions(controller.Options{MaxConcurrentReconciles: 2}).
Complete(r)
} - 使用 predicates（https://sdk.operatorframework.io/docs/building-operators/golang/references/event-filtering/） 過濾監視事件。
- 選擇 EventHandler （https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/handler#hdr-EventHandlers）的類型以更改監視事件將如何轉換為 reconcile 請求以進行 reconcile 循環。對於比主次資源更複雜的 operator 關係，可以使用 EnqueueRequestsFromMapFunc （https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/handler#EnqueueRequestsFromMapFunc）處理程序以將監視事件轉換為任意一組 reconcile 請求。
Reconcile loop
reconcile 函數負責在系統的實際狀態上執行所需的 CR 狀態。每次在監視的 CR 或資源上發生事件時，它都會運行，並將根據這些狀態是否匹配並返回一些值。
這樣，每個 Controller 都有一個 Reconciler 對象，該對象帶有一個 Reconcile() 方法，用於實現 reconcile 循環。reconcile 循環傳遞了 Request（https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Request）參數，該參數是用於查找緩存中的主要資源對象 Memcached：
import (
ctrl "sigs.k8s.io/controller-runtime"
cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
...
)
func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
// Lookup the Memcached instance for this reconcile request
memcached := &cachev1alpha1.Memcached{}
err := r.Get(ctx, req.NamespacedName, memcached)
...
}有關 Reconcilers、客戶端以及與資源事件交互的指南，可以參考 客戶端 API （https://sdk.operatorframework.io/docs/building-operators/golang/references/client/）文檔。以下是 Reconciler 的一些可能的返回選項：
- 發生錯誤時：
- return ctrl.Result{}, err
- 沒有錯誤時：
- return ctrl.Result{Requeue: true}, nil
- 否則, 需要停止 Reconcile，如下：
- return ctrl.Result{}, nil
- 在 X 時間之後，再次 Reconcile：
- return ctrl.Result{RequeueAfter: nextRun.Sub(r.Now())}, nil
想要獲取更多詳細信息，檢查 Reconcile 及其文檔 Reconcile godoc（https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile）。
指定權限及生成 RBAC 清單
controller 需要一定的 RBAC（https://kubernetes.io/docs/reference/access-authn-authz/rbac/） 權限與其管理的資源進行交互。這些是通過 RBAC 標記（https://book.kubebuilder.io/reference/markers/rbac.html） 指定的，如下代碼所示：
//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
...
}The ClusterRole manifest at config/rbac/role.yaml is generated from the above markers via controller-gen with the following command:
[root@blog memcached-operator]# go mod tidy # 如果不執行這一步，執行下面的命令會報錯
[root@blog memcached-operator]# make manifests
/root/memcached-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases配置 operator 鏡像
現在萬事俱備，只欠東風了。剩下的就是構建 operator 鏡像並將其推送到指定的鏡像倉庫上面。
在構建操 operator 鏡像之前，請確保生成的 Dockerfile 引用了我們想要的基礎鏡像。我們可以通過將其標籤替換為另一個標籤（例如 alpine:latest）並刪除 USER 65532:65532 指令來更改默認的 「runner」 鏡像 gcr.io/distroless/static:nonroot。我們沒有刪除這些指令，而是注釋了它們。修改完成之後如下：
# Build the manager binary
FROM golang:1.16 as builder
WORKDIR /workspace
# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much# and so that source changes don't invalidate our downloaded layer
# 修改了此行
RUN export GOPROXY=https://goproxy.cn && go mod download
# Copy the go source
COPY main.go main.go
COPY api/ api/
COPY controllers/ controllers/
# Build
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go
# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
# 注釋了此行，並新增了 FROm alpine:latest
# FROM gcr.io/distroless/static:nonroot
FROM alpine:latest
WORKDIR /
COPY --from=builder /workspace/manager .
# 注釋了此行
# USER 65532:65532
ENTRYPOINT ["/manager"]我們的 Makefile 由項目初始化時寫入的值或命令行中的值組成鏡像標籤。特別是，IMAGE_TAG_BASE 變量允許我們為所有鏡像標籤定義一個通用的鏡像倉庫、命名空間和部分名稱。如果當前值不正確，請將其更新到另一個鏡像倉庫或命名空間。之後，我們可以像下面這樣更新 「IMG」 變量定義：
# 大概在 Makefile 文件的第 32 行，做如下修改，根據實際情況進行修改。
# 這裡使用的是我自己在 Docker Hub 上的命名空間 lavenliu
# 大家可以根據實際情況進行修改成自己的
IMAGE_TAG_BASE ?= lavenliu/memcached-operator
# 大概在 Makefile 文件的第 40 行做如下操作
# IMG ?= controller:latest # 注釋此行，並修改成如下的設置
IMG ?= $(IMAGE_TAG_BASE):$(VERSION)經過上述設置，我們就不用在命令行指定 IMG 環境變量了。下面的命令將會構建名為 lavenliu/memcached-operator 的鏡像，標籤為 v0.0.1，並推送到指定的倉庫上面。
注意：在執行下面的命令之前，我們需要修改一下 Dockerfile，因為要在容器裡面構建 Go 代碼，所以我們需要設置以 Go 的代碼，不然有些代碼會拉取不成功。修改如下：
RUN go mod download 修改為 RUN export GOPROXY=https://goproxy.cn && go mod download
在執行下面的命令之前，確保我們已經登錄 Docker Hub：
[root@blog memcached-operator]# docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: lavenliu
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded登錄成功之後，在執行下面的命令：
[root@blog memcached-operator]# make docker-build docker-push
/home/lcc/memcached-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/lcc/memcached-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
KUBEBUILDER_ASSETS="/root/.local/share/kubebuilder-envtest/k8s/1.22.1-linux-amd64" go test ./... -coverprofile cover.out
? github.com/example/memcached-operator [no test files]
? github.com/example/memcached-operator/api/v1alpha1 [no test files]
ok github.com/example/memcached-operator/controllers 8.846s coverage: 0.0% of statements
docker build -t example.com/memcached-operator:0.0.1 .
Sending build context to Docker daemon 174.1kB
Step 1/13 : FROM golang:1.16 as builder
1.16: Pulling from library/golang
......
Status: Downloaded newer image for golang:1.16
---> 8ffb179c0658
Step 2/13 : WORKDIR /workspace
---> Running in f8bfa670f96f
Removing intermediate container f8bfa670f96f
---> 98c265863c39
Step 3/13 : COPY go.mod go.mod
---> ef86cd6e1e92
Step 4/13 : COPY go.sum go.sum
---> a43ba540fe13
Step 5/13 : RUN export GOPROXY=https://goproxy.cn && go mod download
......
Successfully built 5e9931cceaf9
Successfully tagged lavenliu/memcached-operator:0.0.1
docker push lavenliu/memcached-operator:0.0.1
The push refers to repository [docker.io/lavenliu/memcached-operator]
......查看本地鏡像及遠端鏡像是否存在：
查看 Docker Hub 上的鏡像是否存在：
如果執行成功，我們自定義的 Operator 鏡像會被推送到我們指定的地方。
運行 Operator
我們將以下面兩種形式運行 Operator：
- 以 Go 代碼的形式在集群之外運行
- 以 Deployment 的形式在 Kubernetes 集群中運行
1、在本地（集群之外）運行
以下步驟將展示如何在集群上部署 operator。但是，要在本地運行以用於開發目的並在集群外部運行，請使用 makefile 的 target 「make install run」。或者分開使用也行。如：先運行 make install，之後再運行 make run 也是可以的。先運行 make install：
[root@blog memcached-operator]# make install
/root/memcached-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/kustomize/kustomize/v3@v3.8.7 # 下載了 kustomize 文件
go: downloading sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
......
go: downloading golang.org/x/net v0.0.0-20200625001655-4c5254603344
......
go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
/root/memcached-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.example.com created接著再運行 make run：
[root@blog memcached-operator]# make run
/root/memcached-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/memcached-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
I0331 17:58:25.054157 4005942 request.go:665] Waited for 1.02972392s due to client-side throttling, not priority and fairness, request: GET:https://192.168.56.101:6443/apis/admissionregistration.k8s.io/v1?timeout=32s
2022-03-31T17:58:25.605+0800INFOcontroller-runtime.metricsmetrics server is starting to listen{"addr": ":8080"}
2022-03-31T17:58:25.605+0800INFOsetupstarting manager
2022-03-31T17:58:25.605+0800INFOstarting metrics server{"path": "/metrics"}
2022-03-31T17:58:25.605+0800INFOcontroller.memcachedStarting EventSource{"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "source": "kind source: /, Kind="}
2022-03-31T17:58:25.605+0800INFOcontroller.memcachedStarting EventSource{"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "source": "kind source: /, Kind="}
2022-03-31T17:58:25.605+0800INFOcontroller.memcachedStarting Controller{"reconciler group": "cache.example.com", "reconciler kind": "Memcached"}
2022-03-31T17:58:25.707+0800INFOcontroller.memcachedStarting workers{"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "worker count": 1}查看 CRD：
[root@blog memcached-operator]# kubectl get crd |grep mem
memcacheds.cache.example.com 2022-03-31T09:57:16Z創建一個 Memcached 測試 CR
更新 Memcached CR 的配置文件 config/samples/cache_v1alpha1_memcached.yaml 並定義如下配置：
# 之前的樣板文件
[root@blog memcached-operator]# cat config/samples/cache_v1alpha1_memcached.yaml
apiVersion: cache.my.domain/v1alpha1
kind: Memcached
metadata:
name: memcached-sample
spec:
# TODO(user): Add fields here# 修改之後的配置文件
[root@blog memcached-operator]# cat config/samples/cache_v1alpha1_memcached.yaml
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:name: memcached-sample
spec:
size: 3創建上述 CR：
# 沒有創建之前的 pod 情況
[root@blog memcached-operator]# kubectl -n liucc-test get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
hello-spring-svc-deployment 1/1 1 1 79d
json-spring-svc-deployment 1/1 1 1 72d
memcached-sample 3/3 3 3 25s
world-spring-svc-deployment 1/1 1 1 78d
[root@blog memcached-operator]# kubectl -n liucc-test apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created確保 memcached operator 是否創建了相應的 deployment 及相應的數量。再次執行檢查的命令：
[root@blog memcached-operator]# kubectl -n liucc-test get pods
NAME READY STATUS RESTARTS AGE
hello-spring-svc-deployment-694b8cb9d4-4sjl5 1/1 Running 0 78d
json-spring-svc-deployment-cf88f85c8-rsqdc 1/1 Running 0 71d
memcached-sample-6c765df685-dtmqg 1/1 Running 0 51s
memcached-sample-6c765df685-nl8ks 1/1 Running 0 51s
memcached-sample-6c765df685-vpmfq 1/1 Running 0 51s
world-spring-svc-deployment-84b78bc8d4-tvdz8 1/1 Running 0 78d
[root@blog memcached-operator]# kubectl -n liucc-test get memcached/memcached-sample -o yaml
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"cache.example.com/v1alpha1","kind":"Memcached","metadata":{"annotations":{},"name":"memcached-sample","namespace":"liucc-test"},"spec":{"size":3}}
creationTimestamp: "2022-03-31T10:26:00Z"
generation: 1
managedFields:
- apiVersion: cache.example.com/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:size: {}
manager: kubectl-client-side-apply
operation: Update
time: "2022-03-31T10:26:00Z"
- apiVersion: cache.example.com/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:nodes: {}
manager: main
operation: Update
time: "2022-03-31T10:26:01Z"
name: memcached-sample
namespace: liucc-test
resourceVersion: "59357587"
uid: ef2efd68-c31b-4a9e-8795-6d61975e48dd
spec:
size: 3
status:
nodes:
- memcached-sample-6c765df685-vpmfq
- memcached-sample-6c765df685-dtmqg
- memcached-sample-6c765df685-nl8ks更新 pod 數量
接下來更新 config/samples/cache_v1alpha1_memcached.yaml 文件中的 spec.size，由 3 改為 5，並進行驗證：
[root@blog memcached-operator]# kubectl -n liucc-test patch memcached memcached-sample -p '{"spec":{"size": 5}}' --type=merge
memcached.cache.example.com/memcached-sample patched
[root@blog memcached-operator]# kubectl -n liucc-test get po
NAME READY STATUS RESTARTS AGE
hello-spring-svc-deployment-694b8cb9d4-4sjl5 1/1 Running 0 78d
json-spring-svc-deployment-cf88f85c8-rsqdc 1/1 Running 0 71d
memcached-sample-6c765df685-65wc8 1/1 Running 0 90s
memcached-sample-6c765df685-7r45t 1/1 Running 0 90s
memcached-sample-6c765df685-llhz2 1/1 Running 0 90s
memcached-sample-6c765df685-n6wsk 1/1 Running 0 5s
memcached-sample-6c765df685-nwkjv 1/1 Running 0 5s
world-spring-svc-deployment-84b78bc8d4-tvdz8 1/1 Running 0 78d再次查看 deployment：
[root@blog memcached-operator]# kubectl -n liucc-test get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
hello-spring-svc-deployment 1/1 1 1 79d
json-spring-svc-deployment 1/1 1 1 72d
memcached-sample 5/5 5 5 2m27s
world-spring-svc-deployment 1/1 1 1 78d清理環境
我們可以運行下面的命令，清理已經部署的資源：
[root@blog memcached-operator]# kubectl -n liucc-test delete -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com "memcached-sample" deleted
# 驗證 deployment 是否被刪除
[root@blog memcached-operator]# kubectl -n liucc-test get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
hello-spring-svc-deployment 1/1 1 1 79d
json-spring-svc-deployment 1/1 1 1 72d
world-spring-svc-deployment 1/1 1 1 78d
# 驗證 pod 是否被刪除
[root@blog memcached-operator]# kubectl -n liucc-test get po
NAME READY STATUS RESTARTS AGE
hello-spring-svc-deployment-694b8cb9d4-4sjl5 1/1 Running 0 78d
json-spring-svc-deployment-cf88f85c8-rsqdc 1/1 Running 0 71d
memcached-sample-6c765df685-dtmqg 0/1 Terminating 0 3m5s
memcached-sample-6c765df685-vmp6q 0/1 Terminating 0 60s
world-spring-svc-deployment-84b78bc8d4-tvdz8 1/1 Running 0 78d2、在集群內部以 Deployment 運行
默認情況下，會在 Kubernetes 集群上創建一個名為 <project-name>-system 的新命名空間，例如：memcached-operator-system。運行以下命令部署 operator，它還會從 config/rbac 清單文件安裝 RBAC。
# 查看 make 幫助信息
[root@blog memcached-operator]# make help
Usage:
make <target>
General
help Display this help.
Development
manifests Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
generate Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
fmt Run go fmt against code.
vet Run go vet against code.
test Run tests.
Build
build Build manager binary.
run Run a controller from your host.
docker-build Build docker image with the manager.
docker-push Push docker image with the manager.
Deployment
install Install CRDs into the K8s cluster specified in ~/.kube/config.
uninstall Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
deploy Deploy controller to the K8s cluster specified in ~/.kube/config.
undeploy Undeploy controller from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
controller-gen Download controller-gen locally if necessary.
kustomize Download kustomize locally if necessary.
envtest Download envtest-setup locally if necessary.
bundle Generate bundle manifests and metadata, then validate generated files.
bundle-build Build the bundle image.
bundle-push Push the bundle image.
opm Download opm locally if necessary.
catalog-build Build a catalog image.
catalog-push Push a catalog image.在部署之前，我們需要修改一下 Makefile 文件，修改鏡像的地址：
# 大概在 33 行
IMAGE_TAG_BASE ?= <修改為一個有效的地址>/memcached-operator接著進行部署：
[root@blog memcached-operator]# make deploy
/root/memcached-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /root/memcached-operator/bin/kustomize edit set image controller=lavenliu/memcached-operator:0.0.1
/root/memcached-operator/bin/kustomize build config/default | kubectl apply -f -
namespace/memcached-operator-system created
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.example.com configured
serviceaccount/memcached-operator-controller-manager created
role.rbac.authorization.k8s.io/memcached-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/memcached-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/memcached-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/memcached-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/memcached-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-proxy-rolebinding created
configmap/memcached-operator-manager-config created
service/memcached-operator-controller-manager-metrics-service created
deployment.apps/memcached-operator-controller-manager created
[root@blog memcached-operator]# echo $?
0驗證 memcached-operator 是否啟動成功：
[root@blog memcached-operator]# kubectl -n memcached-operator-system get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
memcached-operator-controller-manager 0/1 1 0 50s
[root@blog memcached-operator]# kubectl -n memcached-operator-system get po
NAME READY STATUS RESTARTS AGE
memcached-operator-controller-manager-7bbc46698f-wsqvp 0/2 ContainerCreating 0 50s報錯了，主要原因是拉取鏡像失敗及運行容器時也失敗了。查看一下錯誤原因：
[root@blog memcached-operator]# kubectl -n memcached-operator-system describe po memcached-operator-controller-manager-7bbc46698f-wsqvp
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 59s default-scheduler Successfully assigned memcached-operator-system/memcached-operator-controller-manager-7bbc46698f-wsqvp to node03.lavenliu.cn
Normal Pulling <invalid> kubelet Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0"
Normal Pulled <invalid> kubelet Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0" in 15.593866868s
Normal Created <invalid> kubelet Created container kube-rbac-proxy
Normal Started <invalid> kubelet Started container kube-rbac-proxy
Normal Pulling <invalid> kubelet Pulling image "lavenliu/memcached-operator:0.0.1"
Normal Pulled <invalid> kubelet Successfully pulled image "lavenliu/memcached-operator:0.0.1" in 19.848521963s
Warning Failed <invalid> (x4 over <invalid>) kubelet Error: container has runAsNonRoot and image will run as root (pod: "memcached-operator-controller-manager-7bbc46698f-wsqvp_memcached-operator-system(20bd3c70-7400-4497-904e-325122b364db)", container: manager)
Normal Pulled <invalid> (x3 over <invalid>) kubelet Container image "lavenliu/memcached-operator:0.0.1" already present on machine文件做如下修改：
[root@blog memcached-operator]# vim config/default/manager_auth_proxy_patch.yaml
spec:
template:
spec:
securityContext: # 新增此行
runAsUser: 1000 # 新增此行
containers:
- name: kube-rbac-proxy
# image: gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0 # 鏡像拉取不成功，所以注釋了此行
# 在 Docker HUB 上面找到了如下的鏡像可以正常使用
image: rancher/kube-rbac-proxy:v0.5.0 # 新增此行修改完成之後，執行 make undeloy 命令：
[root@blog memcached-operator]# make undeploy然後再次執行 make deploy 命令：
[root@blog memcached-operator]# make deploy最後查看 deployment 及 pod 信息：
[root@blog memcached-operator]# kubectl -n memcached-operator-system get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
memcached-operator-controller-manager 1/1 1 1 4m
[root@blog memcached-operator]# kubectl -n memcached-operator-system get po
NAME READY STATUS RESTARTS AGE
memcached-operator-controller-manager-54548dbf4d-drnhp 2/2 Running 0 4m15s
[root@blog memcached-operator]# kubectl -n memcached-operator-system describe po memcached-operator-controller-manager-54548dbf4d-drnhp
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m26s default-scheduler Successfully assigned memcached-operator-system/memcached-operator-controller-manager-54548dbf4d-drnhp to cn-shanghai.10.10.11.13
Normal AllocIPSucceed 4m26s terway-daemon Alloc IP 10.10.108.242/32 for Pod
Normal Pulled 4m26s kubelet Container image "rancher/kube-rbac-proxy:v0.5.0" already present on machine
Normal Created 4m26s kubelet Created container kube-rbac-proxy
Normal Started 4m26s kubelet Started container kube-rbac-proxy
Normal Pulled 4m26s kubelet Container image "harbor.lavenliu.cn/library/memcached-operator:0.0.1" already present on machine
Normal Created 4m26s kubelet Created container manager
Normal Started 4m26s kubelet Started container manager控制器部署完成，接著就部署實例：
[root@blog memcached-operator]# kubectl -n liucc-test apply -f config/samples/cache_v1alpha1_memcached.yaml查看 pod 信息：
[root@blog memcached-operator]# kubectl -n liucc-test get po
NAME READY STATUS RESTARTS AGE
hello-spring-svc-deployment-694b8cb9d4-4sjl5 1/1 Running 0 78d
json-spring-svc-deployment-cf88f85c8-rsqdc 1/1 Running 0 71d
memcached-sample-6c765df685-7xrn2 1/1 Running 0 6m15s
memcached-sample-6c765df685-tdjz8 1/1 Running 0 6m15s
memcached-sample-6c765df685-xvpqk 1/1 Running 0 6m15s
world-spring-svc-deployment-84b78bc8d4-tvdz8 1/1 Running 0 78d修改 pod 的數量為 5 個：
[root@blog memcached-operator]# kubectl -n liucc-test patch memcached memcached-sample -p '{"spec":{"size": 5}}' --type=merge
memcached.cache.example.com/memcached-sample patched
# 再次查看
[root@blog memcached-operator]# kubectl -n liucc-test get po
NAME READY STATUS RESTARTS AGE
hello-spring-svc-deployment-694b8cb9d4-4sjl5 1/1 Running 0 78d
json-spring-svc-deployment-cf88f85c8-rsqdc 1/1 Running 0 71d
memcached-sample-6c765df685-7xrn2 1/1 Running 0 7m27s
memcached-sample-6c765df685-tdjz8 1/1 Running 0 7m27s
memcached-sample-6c765df685-xvpqk 1/1 Running 0 7m27s
memcached-sample-6c765df685-b79kc 1/1 Running 0 3s # 新起的 pod
memcached-sample-6c765df685-qkxvc 1/1 Running 0 3s # 新起的 pod
world-spring-svc-deployment-84b78bc8d4-tvdz8 1/1 Running 0 78d清除演示環境：
[root@blog memcached-operator]# make undeploy
/root/memcached-operator/bin/kustomize build config/default | kubectl delete --ignore-not-found=false -f -
namespace "memcached-operator-system" deleted
customresourcedefinition.apiextensions.k8s.io "memcacheds.cache.example.com" deleted
serviceaccount "memcached-operator-controller-manager" deleted
role.rbac.authorization.k8s.io "memcached-operator-leader-election-role" deleted
clusterrole.rbac.authorization.k8s.io "memcached-operator-manager-role" deleted
clusterrole.rbac.authorization.k8s.io "memcached-operator-metrics-reader" deleted
clusterrole.rbac.authorization.k8s.io "memcached-operator-proxy-role" deleted
rolebinding.rbac.authorization.k8s.io "memcached-operator-leader-election-rolebinding" deleted
clusterrolebinding.rbac.authorization.k8s.io "memcached-operator-manager-rolebinding" deleted
clusterrolebinding.rbac.authorization.k8s.io "memcached-operator-proxy-rolebinding" deleted
configmap "memcached-operator-manager-config" deleted
service "memcached-operator-controller-manager-metrics-service" deleted
deployment.apps "memcached-operator-controller-manager" deleted驗證 memcached 的 pod 是否還在：
[root@blog memcached-operator]# kubectl -n liucc-test get po
NAME READY STATUS RESTARTS AGE
hello-spring-svc-deployment-694b8cb9d4-4sjl5 1/1 Running 0 78d
json-spring-svc-deployment-cf88f85c8-rsqdc 1/1 Running 0 71d
world-spring-svc-deployment-84b78bc8d4-tvdz8 1/1 Running 0 78d3、使用 OLM 部署 Operator
首先我們需要安裝 OLM：
[root@blog memcached-operator]# operator-sdk olm install
I0403 20:06:37.667540 1957628 request.go:665] Waited for 1.028361355s due to client-side throttling, not priority and fairness, request: GET:https://192.168.56.101:6443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s
INFO[0001] Fetching CRDs for version "latest"
INFO[0001] Fetching resources for resolved version "latest"
I0403 20:07:02.327494 1957628 request.go:665] Waited for 1.04307318s due to client-side throttling, not priority and fairness, request: GET:https://192.168.56.101:6443/apis/log.alibabacloud.com/v1alpha1?timeout=32s
I0403 20:07:13.324530 1957628 request.go:665] Waited for 1.045662694s due to client-side throttling, not priority and fairness, request: GET:https://192.168.56.101:6443/apis/install.istio.io/v1alpha1?timeout=32s
INFO[0038] Creating CRDs and resources
INFO[0038] Creating CustomResourceDefinition "catalogsources.operators.coreos.com"
INFO[0038] Creating CustomResourceDefinition "clusterserviceversions.operators.coreos.com"
INFO[0038] Creating CustomResourceDefinition "installplans.operators.coreos.com"
INFO[0038] Creating CustomResourceDefinition "olmconfigs.operators.coreos.com"
INFO[0038] Creating CustomResourceDefinition "operatorconditions.operators.coreos.com"
INFO[0038] Creating CustomResourceDefinition "operatorgroups.operators.coreos.com"
INFO[0038] Creating CustomResourceDefinition "operators.operators.coreos.com"
INFO[0038] Creating CustomResourceDefinition "subscriptions.operators.coreos.com"
INFO[0038] Creating Namespace "olm"
INFO[0038] Creating Namespace "operators"
INFO[0039] Creating ServiceAccount "olm/olm-operator-serviceaccount"
INFO[0039] Creating ClusterRole "system:controller:operator-lifecycle-manager"
INFO[0039] Creating ClusterRoleBinding "olm-operator-binding-olm"
INFO[0039] Creating OLMConfig "cluster"
INFO[0042] Creating Deployment "olm/olm-operator"
INFO[0042] Creating Deployment "olm/catalog-operator"
INFO[0042] Creating ClusterRole "aggregate-olm-edit"
INFO[0042] Creating ClusterRole "aggregate-olm-view"
INFO[0042] Creating OperatorGroup "operators/global-operators"
INFO[0042] Creating OperatorGroup "olm/olm-operators"
INFO[0042] Creating ClusterServiceVersion "olm/packageserver"
INFO[0042] Creating CatalogSource "olm/operatorhubio-catalog"
INFO[0042] Waiting for deployment/olm-operator rollout to complete
INFO[0042] Waiting for Deployment "olm/olm-operator" to rollout: 0 of 1 updated replicas are available
INFO[0063] Deployment "olm/olm-operator" successfully rolled out
INFO[0063] Waiting for deployment/catalog-operator rollout to complete
INFO[0063] Deployment "olm/catalog-operator" successfully rolled out
INFO[0063] Waiting for deployment/packageserver rollout to complete
INFO[0063] Waiting for Deployment "olm/packageserver" to rollout: 0 of 2 updated replicas are available
INFO[0079] Deployment "olm/packageserver" successfully rolled out
INFO[0079] Successfully installed OLM version "latest"
NAME NAMESPACE KIND STATUS
catalogsources.operators.coreos.com CustomResourceDefinition Installed
clusterserviceversions.operators.coreos.com CustomResourceDefinition Installed
installplans.operators.coreos.com CustomResourceDefinition Installed
olmconfigs.operators.coreos.com CustomResourceDefinition Installed
operatorconditions.operators.coreos.com CustomResourceDefinition Installed
operatorgroups.operators.coreos.com CustomResourceDefinition Installed
operators.operators.coreos.com CustomResourceDefinition Installed
subscriptions.operators.coreos.com CustomResourceDefinition Installed
olm Namespace Installed
operators Namespace Installed
olm-operator-serviceaccount olm ServiceAccount Installed
system:controller:operator-lifecycle-manager ClusterRole Installed
olm-operator-binding-olm ClusterRoleBinding Installed
cluster OLMConfig Installed
olm-operator olm Deployment Installed
catalog-operator olm Deployment Installed
aggregate-olm-edit ClusterRole Installed
aggregate-olm-view ClusterRole Installed
global-operators operators OperatorGroup Installed
olm-operators olm OperatorGroup Installed
packageserver olm ClusterServiceVersion Installed
operatorhubio-catalog olm CatalogSource Installed如果安裝失敗，再次進行安裝時，需要先卸載，然後再安裝：operator-sdk olm uninstall。
安裝完成可以查看一下狀態：
[root@blog memcached-operator]# operator-sdk olm status
I0406 11:28:17.359874 30074 request.go:665] Waited for 1.041471802s due to client-side throttling, not priority and fairness, request: GET:https://192.168.56.101:6443/apis/admissionregistration.k8s.io/v1?timeout=32s
INFO[0002] Fetching CRDs for version "v0.20.0"
INFO[0002] Fetching resources for resolved version "v0.20.0"
INFO[0007] Successfully got OLM status for version "v0.20.0"
NAME NAMESPACE KIND STATUS
operatorgroups.operators.coreos.com CustomResourceDefinition Installed
operatorconditions.operators.coreos.com CustomResourceDefinition Installed
olmconfigs.operators.coreos.com CustomResourceDefinition Installed
installplans.operators.coreos.com CustomResourceDefinition Installed
clusterserviceversions.operators.coreos.com CustomResourceDefinition Installed
olm-operator-binding-olm ClusterRoleBinding Installed
operatorhubio-catalog olm CatalogSource Installed
olm-operators olm OperatorGroup Installed
aggregate-olm-view ClusterRole Installed
catalog-operator olm Deployment Installed
cluster OLMConfig Installed
operators.operators.coreos.com CustomResourceDefinition Installed
olm-operator olm Deployment Installed
subscriptions.operators.coreos.com CustomResourceDefinition Installed
aggregate-olm-edit ClusterRole Installed
olm Namespace Installed
global-operators operators OperatorGroup Installed
operators Namespace Installed
packageserver olm ClusterServiceVersion Installed
olm-operator-serviceaccount olm ServiceAccount Installed
catalogsources.operators.coreos.com CustomResourceDefinition Installed
system:controller:operator-lifecycle-manager ClusterRole Installed我們看一下中間生成了哪些命名空間及 POD：
[root@blog memcached-operator]# kubectl get ns
NAME STATUS AGE
default Active 185d
kube-node-lease Active 185d
kube-public Active 185d
kube-system Active 185d
liucc-test Active 7h43m
olm Active 103m
operators Active 103m
[root@blog memcached-operator]# kubectl -n olm get po
NAME READY STATUS RESTARTS AGE
catalog-operator-5c4997c789-tk5cq 1/1 Running 0 103m
olm-operator-6d46969488-rc8zf 1/1 Running 0 103m
operatorhubio-catalog-nt5vw 1/1 Running 0 103m
packageserver-848bdb76dd-6snj4 1/1 Running 0 103m
packageserver-848bdb76dd-grdx4 1/1 Running 0 103m接下來對我們的 Operator 進行打包，然後構建並推送包鏡像。bundle 目標在 bundle 目錄包含定義了我們的 operator 清單和元數據。bundle-build 和 bundle-push 兩個目標將會構建和推送由 bundle.Dockerfile 文件定義的包鏡像。
[root@blog memcached-operator]# make bundle
/root/memcached-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
operator-sdk generate kustomize manifests -q
Display name for the operator (required): # 需要填寫如下信息
> memcached-operator
Description for the operator (required): # 需要填寫如下信息
> memcached-operator
Provider's name for the operator (required): # 需要填寫如下信息
> lavenliu
Any relevant URL for the provider name (optional):
>
Comma-separated list of keywords for your operator (required): # 需要填寫如下信息
> memcached,operator
Comma-separated list of maintainers and their emails (e.g. 'name1:email1, name2:email2') (required): # 需要填寫如下信息
> lcc@163.com
cd config/manager && /root/memcached-operator/bin/kustomize edit set image controller=lavenliu/memcached-operator:0.0.1
/root/memcached-operator/bin/kustomize build config/manifests | operator-sdk generate bundle -q --overwrite --version 0.0.1
INFO[0000] Creating bundle/metadata/annotations.yaml
INFO[0000] Creating bundle.Dockerfile
INFO[0000] Bundle metadata generated suceessfully
operator-sdk bundle validate ./bundle
INFO[0000] All validation tests have completed successfully我們看看是否有新的文件或目錄產生：
[root@blog memcached-operator]# ll -t
total 136
drwxr-xr-x 5 root root 4096 Apr 3 20:11 bundle # 新產生的目錄-rw-r--r-- 1 root root 923 Apr 3 20:11 bundle.Dockerfile # 新產生的文件
drwx------ 2 root root 4096 Apr 3 19:15 hack
drwxr-xr-x 2 root root 4096 Apr 3 15:41 bin
drwx------ 2 root root 4096 Apr 3 15:37 controllers
-rw------- 1 root root 9560 Mar 31 19:03 Makefile
-rw-r--r-- 1 root root 2361 Mar 31 16:49 cover.out
-rw------- 1 root root 780 Mar 31 16:46 Dockerfile
-rw------- 1 root root 246 Mar 31 16:20 go.mod
-rw------- 1 root root 3192 Mar 31 14:39 main.go
-rw------- 1 root root 448 Mar 31 14:39 PROJECT
drwx------ 3 root root 4096 Mar 31 14:39 api
drwx------ 10 root root 4096 Mar 31 14:39 config
-rw-r--r-- 1 root root 77000 Mar 31 14:37 go.sum接著運行 make bundle-build 目標：
[root@blog memcached-operator]# make bundle-build
docker build -f bundle.Dockerfile -t lavenliu/memcached-operator-bundle:v0.0.1 .
Sending build context to Docker daemon 196.6kB
Step 1/14 : FROM scratch
--->
Step 2/14 : LABEL operators.operatorframework.io.bundle.mediatype.v1=registry+v1
---> Running in acbee848ff30
Removing intermediate container acbee848ff30
---> 7840efc1acc7
Step 3/14 : LABEL operators.operatorframework.io.bundle.manifests.v1=manifests/
---> Running in 795bbd68aa0b
Removing intermediate container 795bbd68aa0b
---> 58c64b2c5bed
Step 4/14 : LABEL operators.operatorframework.io.bundle.metadata.v1=metadata/
---> Running in 3ba74dd41232
Removing intermediate container 3ba74dd41232
---> 4921148bcd53
Step 5/14 : LABEL operators.operatorframework.io.bundle.package.v1=memcached-operator
---> Running in c8ae15420ea2
Removing intermediate container c8ae15420ea2
---> 0c417435ceff
Step 6/14 : LABEL operators.operatorframework.io.bundle.channels.v1=alpha
---> Running in a4c7d20e793b
Removing intermediate container a4c7d20e793b
---> b549d7f0aa94
Step 7/14 : LABEL operators.operatorframework.io.metrics.builder=operator-sdk-v1.15.0+git
---> Running in 3dd418069c6b
Removing intermediate container 3dd418069c6b
---> a0ead127d313
Step 8/14 : LABEL operators.operatorframework.io.metrics.mediatype.v1=metrics+v1
---> Running in 513a0223cafb
Removing intermediate container 513a0223cafb
---> 97c961869eb3
Step 9/14 : LABEL operators.operatorframework.io.metrics.project_layout=go.kubebuilder.io/v3
---> Running in 24c6fbf30e77
Removing intermediate container 24c6fbf30e77
---> e523f8b86a47
Step 10/14 : LABEL operators.operatorframework.io.test.mediatype.v1=scorecard+v1
---> Running in 795730b93b89
Removing intermediate container 795730b93b89
---> 5f186fccf6fb
Step 11/14 : LABEL operators.operatorframework.io.test.config.v1=tests/scorecard/
---> Running in d29664ae092a
Removing intermediate container d29664ae092a
---> 7776ff18f767
Step 12/14 : COPY bundle/manifests /manifests/
---> cde9d176b798
Step 13/14 : COPY bundle/metadata /metadata/
---> 38d589cdd086
Step 14/14 : COPY bundle/tests/scorecard /tests/scorecard/
---> 976c6344511b
Successfully built 976c6344511b
Successfully tagged lavenliu/memcached-operator-bundle:v0.0.1再運行 make bundle-push 目標：
[root@blog memcached-operator]# make bundle-pushmake docker-push IMG=lavenliu/memcached-operator-bundle:v0.0.1
make[1]: Entering directory `/root/memcached-operator'
docker push lavenliu/memcached-operator-bundle:v0.0.1
The push refers to repository [docker.io/lavenliu/memcached-operator-bundle]
36632daec064: Layer already exists
ca08711083d4: Layer already exists
8b7611a97ff6: Layer already exists
v0.0.1: digest: sha256:9adbb5b9e2aede9108f9bba509dc8ca9aa0ed4aad0de6ad37cc8cb4eaa3b6c79 size: 939
make[1]: Leaving directory `/root/memcached-operator'最後，運行我們的包。如果我們的包鏡像託管在私有鏡像倉庫中或具有自定義 CA，則可以參考這些 配置步驟 （https://sdk.operatorframework.io/docs/olm-integration/cli-overview#private-bundle- and-catalog-image-registries）。
[root@blog memcached-operator]# operator-sdk run bundle lavenliu/memcached-operator-bundle:v0.0.1查看 docs(https://sdk.operatorframework.io/docs/olm-integration/tutorial-bundle) 深入了解 operator-sdk 的 OLM 集成。
總結
經過前面幾章的 「折騰」，我們終於完成了一個 Operator 的 tutorial。雖然是按照官方文檔進行一步一步的操作，但是中間過程還是挺曲折的。希望本文可以幫助到大家，對編寫 Operator 起到參考的作用。
附錄
推薦閱讀
- Introduction - The Kubebuilder Book（ https://book.kubebuilder.io/introduction.html）
- Operator SDK (operatorframework.io)（https://sdk.operatorframework.io/）
- 《Kubernetes Operators》，這本書使用的 operator-sdk 版本比較舊，但是裡面的講解還是非常不錯的
- CoreOS 關於 Operator 的介紹
- https://kubernetes.io/zh/docs/concepts/extend-kubernetes/operator/
- 在 OperatorHub.io(https://operatorhub.io/) 上找到現成的、適合你的 Operator