|
| 1 | + |
| 2 | + |
| 3 | +# Create a Windows workload cluster |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +CAPOCI enables users to create and manage Windows workload clusters in Oracle Cloud Infrastructure (OCI). |
| 8 | +This means that the [Kubernetes Control Plane][kubernetes-control-plane] will be Linux and the nodes will be Windows. |
| 9 | +First, users build the [Windows image using image-builder][image-builder], then use the Windows flavor |
| 10 | +template from the [latest release][capoci-latest-release]. Finally, install the [Calico CNI Provider |
| 11 | +and OCI Cloud Controller Manager](#install-calico-cni-provider-and-oci-cloud-controller-manager). |
| 12 | + |
| 13 | +## Known Limitations |
| 14 | + |
| 15 | +The Windows workload cluster has known limitations: |
| 16 | + |
| 17 | +- Limited to [Standard Bare Metal (BM) shapes][bm-shapes] |
| 18 | +- Limited to OCI provided platform images. See [image-build documentation][image-builder] for more details |
| 19 | +- Custom image MUST be built using the same shape of Bare Metal the worker nodes will run |
| 20 | +- CNI provider support is [Calico in VXLAN mode][calico-windows] |
| 21 | +- [Block volumes][block-volume] are not currently supported |
| 22 | +- Bring Your Own License (BYOL) is not supported |
| 23 | +- See [Calico windows docs][calico-limitations] for their limitations |
| 24 | + |
| 25 | +## Licensing |
| 26 | + |
| 27 | +BYOL is currently not supported using CAPOCI. For more info on Windows Licensing |
| 28 | +see the [Compute FAQ documentation][compute-windows-faq]. |
| 29 | + |
| 30 | +## Build Windows image |
| 31 | + |
| 32 | +> NOTE: It is recommended to [check shape availability](#check-shape-availability) before building image(s) |
| 33 | +
|
| 34 | +In order to launch Windows instances for the cluster a Windows image, [using image-builder][image-builder], |
| 35 | +will need to be built. It is **important** to make sure the same shape is used to build and launch the instance. |
| 36 | + |
| 37 | +Example: If a `BM.Standard2.52` is used to build then the `OCI_NODE_MACHINE_TYPE` MUST |
| 38 | +be `BM.Standard2.52` |
| 39 | + |
| 40 | + |
| 41 | +## Check shape availability |
| 42 | + |
| 43 | +Make sure the [OCI CLI][install-oci-cli] is installed. Then set the AD information if using |
| 44 | +muti-AD regions. |
| 45 | + |
| 46 | +> NOTE: Use the [OCI Regions and Availability Domains][regions] page to figure out which |
| 47 | +regions have multiple ADs. |
| 48 | + |
| 49 | +```bash |
| 50 | +oci iam availability-domain list --compartment-id=<your compartment> --region=<region> |
| 51 | +``` |
| 52 | + |
| 53 | +Using the AD `name` from the output above start searching for BM shape availability. |
| 54 | + |
| 55 | +```bash |
| 56 | +oci compute shape list --compartment-id=<your compartment> --profile=DEFAULT --region=us-ashburn-1 --availability-domain=<your AD ID> | grep BM |
| 57 | + |
| 58 | +"shape": "BM.Standard.E3.128" |
| 59 | +"shape-name": "BM.Standard2.52" |
| 60 | +"shape-name": "BM.Standard.E3.128" |
| 61 | +"shape": "BM.Standard.E2.64" |
| 62 | +"shape-name": "BM.Standard2.52" |
| 63 | +"shape-name": "BM.Standard3.64" |
| 64 | +"shape": "BM.Standard1.36" |
| 65 | +``` |
| 66 | + |
| 67 | +> NOTE: If the output is empty then the compartment for that region/AD doesn't have BM shapes. |
| 68 | +If you are unable to locate any shapes you may need to submit a |
| 69 | +[service limit increase request][compute-service-limit] |
| 70 | + |
| 71 | + |
| 72 | +## Create a new Windows workload cluster |
| 73 | + |
| 74 | +It is recommended to have the following guides handy: |
| 75 | + |
| 76 | +- [Windows Cluster Debugging][windows-cluster-debug] |
| 77 | +- [Windows Container Debugging][windows-containers-debug] |
| 78 | + |
| 79 | +When using `clusterctl` to generate the cluster use the `windows-calico` example flavor. |
| 80 | + |
| 81 | +The following command uses the `OCI_CONTROL_PLANE_MACHINE_TYPE` and `OCI_NODE_MACHINE_TYPE` |
| 82 | +parameters to specify bare metal shapes instead of using CAPOCI's default virtual |
| 83 | +instance shape. The `OCI_CONTROL_PLANE_PV_TRANSIT_ENCRYPTION` and `OCI_NODE_PV_TRANSIT_ENCRYPTION` |
| 84 | +parameters disable encryption of data in flight between the bare metal instance and the block storage resources. |
| 85 | + |
| 86 | +> NOTE: The `OCI_NODE_MACHINE_TYPE_OCPUS` must match the OPCU count of the BM shape. |
| 87 | +See the [Compute Shapes][bm-shapes] page to get the OCPU count for the specific shape. |
| 88 | + |
| 89 | +```bash |
| 90 | +OCI_COMPARTMENT_ID=<compartment-id> \ |
| 91 | +OCI_CONTROL_PLANE_IMAGE_ID=<linux-custom-image-id> \ |
| 92 | +OCI_NODE_IMAGE_ID=<windows-custom-image-id> \ |
| 93 | +OCI_SSH_KEY=<ssh-key> \ |
| 94 | +NODE_MACHINE_COUNT=1 \ |
| 95 | +OCI_NODE_MACHINE_TYPE=BM.Standard.E4.128 \ |
| 96 | +OCI_NODE_MACHINE_TYPE_OCPUS=128 \ |
| 97 | +OCI_NODE_PV_TRANSIT_ENCRYPTION=false \ |
| 98 | +OCI_CONTROL_PLANE_MACHINE_TYPE_OCPUS=3 \ |
| 99 | +OCI_CONTROL_PLANE_MACHINE_TYPE=VM.Standard3.Flex \ |
| 100 | +CONTROL_PLANE_MACHINE_COUNT=3 \ |
| 101 | +OCI_SHAPE_MEMORY_IN_GBS= \ |
| 102 | +KUBERNETES_VERSION=<k8s-version> \ |
| 103 | +clusterctl generate cluster <cluster-name> \ |
| 104 | +--target-namespace default \ |
| 105 | +--flavor windows-calico | kubectl apply -f - |
| 106 | +``` |
| 107 | + |
| 108 | +### Access workload cluster Kubeconfig |
| 109 | + |
| 110 | +Execute the following command to list all the workload clusters present: |
| 111 | + |
| 112 | +```bash |
| 113 | +kubectl get clusters -A |
| 114 | +``` |
| 115 | + |
| 116 | +Execute the following command to access the kubeconfig of a workload cluster: |
| 117 | + |
| 118 | +```bash |
| 119 | +clusterctl get kubeconfig <cluster-name> -n default > <cluster-name>.kubeconfig |
| 120 | +``` |
| 121 | + |
| 122 | +### Install Calico CNI Provider and OCI Cloud Controller Manager |
| 123 | + |
| 124 | +#### Install Calico |
| 125 | + |
| 126 | +The [Calico for Windows][calico-windows] getting started guide should be read for better understand of the CNI on Windows. |
| 127 | +It is recommended to have the following guides handy: |
| 128 | + |
| 129 | +- [Windows Calico Troubleshooting][calico-windows-debug] |
| 130 | + |
| 131 | +##### The steps to follow: |
| 132 | + |
| 133 | +**On the management cluster** |
| 134 | + |
| 135 | +1. Run |
| 136 | + ``` |
| 137 | + kubectl get OCICluster <cluster-name> -o jsonpath='{.spec.controlPlaneEndpoint.host}' |
| 138 | + ``` |
| 139 | + to get the `KUBERNETES_SERVICE_HOST` info that will be used in later steps |
| 140 | + |
| 141 | +**On the workload cluster** |
| 142 | + |
| 143 | +1. Download the [v3.24.5 calico release][calico-release] |
| 144 | + ``` |
| 145 | + curl -L https://github.com/projectcalico/calico/releases/download/v3.24.5/release-v3.24.5.tgz -o calico-v3.24.5.tgz |
| 146 | + ``` |
| 147 | +1. Uncompress the downloaded file and locate the `calico-vxlan.yaml`, `calico-windows-vxlan.yaml` and `windows-kube-proxy.yaml` |
| 148 | +files in the `manifests` dir |
| 149 | +1. Edit the `calico-vxlan.yaml` and modify the follow variables to allow Calico running on the nodes use VXLAN |
| 150 | + - `CALICO_IPV4POOL_IPIP` - set to `"Never"` |
| 151 | + - `CALICO_IPV4POOL_VXLAN` - set to `"Always"` |
| 152 | +1. ``` |
| 153 | + kubectl apply -f calico-vxlan.yaml |
| 154 | + ``` |
| 155 | +1. Wait for the IPAMConfig to be loaded |
| 156 | +1. ``` |
| 157 | + kubectl patch IPAMConfig default --type merge --patch='{"spec": {"strictAffinity": true}}' |
| 158 | + ``` |
| 159 | +1. Edit the `calico-windows-vxlan.yaml` and modify the follow variables to allow Calico running on the nodes to talk |
| 160 | +to the Kubernetes control plane |
| 161 | + - `KUBERNETES_SERVICE_HOST` - the IP address from the management cluster step |
| 162 | + - `KUBERNETES_SERVICE_PORT`- the port from step the management cluster step |
| 163 | + - `K8S_SERVICE_CIDR` - The service CIDR set in the cluster template |
| 164 | + - `DNS_NAME_SERVERS` - the IP address from dns service |
| 165 | + ``` |
| 166 | + kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}' |
| 167 | + ``` |
| 168 | + - Change the namespace from `calico-system` to `kube-system` |
| 169 | + - add the following `env` to the container named `node` |
| 170 | + ```yaml |
| 171 | + - name: VXLAN_ADAPTER |
| 172 | + value: "Ethernet 2" |
| 173 | + ``` |
| 174 | +1. ``` |
| 175 | + kubectl apply -f calico-windows-vxlan.yaml |
| 176 | + ``` |
| 177 | + (it takes a bit for this to pass livenessprobe) |
| 178 | +1. Edit the `windows-kube-proxy.yaml` |
| 179 | + - update the `kube-proxy` container environment variable `K8S_VERSION` to the version of kubernetes you are deploying |
| 180 | + - update the `image` version for the container named `kube-proxy` and make sure to set the |
| 181 | + correct [windows nanoserver version][docker-hub-nanoserver] example: `ltsc2019` |
| 182 | +1. ``` |
| 183 | + kubectl apply -f windows-kube-proxy.yaml |
| 184 | + ``` |
| 185 | + |
| 186 | +#### Install OCI Cloud Controller Manager |
| 187 | + |
| 188 | +By default, the [OCI Cloud Controller Manager (CCM)][oci-ccm] is not installed into a workload cluster. To install the OCI CCM, follow [these instructions][install-oci-ccm]. |
| 189 | + |
| 190 | +### Scheduling Windows containers |
| 191 | + |
| 192 | +With the cluster in a ready state and CCM installed, an [example deployment][win-webserver-deployment] |
| 193 | +can be used to test that pods are scheduled. Accessing the deployed pods and using `nslookup` you can |
| 194 | +[test that the cluster DNS is working][win-webserver-dns]. |
| 195 | + |
| 196 | + |
| 197 | +[block-volume]: https://docs.oracle.com/en-us/iaas/Content/GSG/Tasks/addingstorage.htm |
| 198 | +[bm-shapes]: https://docs.oracle.com/en-us/iaas/Content/Compute/References/computeshapes.htm#bm-standard |
| 199 | +[calico-limitations]: https://docs.tigera.io/calico/3.24/getting-started/kubernetes/windows-calico/limitations |
| 200 | +[calico-release]: https://github.com/projectcalico/calico/releases/download/v3.24.5/release-v3.24.5.tgz |
| 201 | +[calico-windows]: https://docs.tigera.io/calico/3.24/getting-started/kubernetes/windows-calico/ |
| 202 | +[calico-windows-debug]: https://docs.tigera.io/calico/3.24/getting-started/kubernetes/windows-calico/troubleshoot |
| 203 | +[capoci-latest-release]: https://github.com/oracle/cluster-api-provider-oci/releases/latest |
| 204 | +[compute-service-limit]: https://docs.oracle.com/en-us/iaas/Content/General/Concepts/servicelimits.htm#computelimits |
| 205 | +[compute-windows-faq]: https://www.oracle.com/cloud/compute/faq/#category-windows |
| 206 | +[docker-hub-nanoserver]: https://hub.docker.com/_/microsoft-windows-nanoserver |
| 207 | +[image-builder]: https://image-builder.sigs.k8s.io/capi/providers/oci.html#building-a-windows-image |
| 208 | +[install-ccm]: ../gs/create-workload-cluster.md#install-oci-cloud-controller-manager-and-csi-in-a-self-provisioned-cluster |
| 209 | +[install-oci-ccm]: ./install-oci-ccm.md |
| 210 | +[install-oci-cli]: https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm |
| 211 | +[kubernetes-control-plane]: https://kubernetes.io/docs/concepts/overview/components/#control-plane-components |
| 212 | +[oci-ccm]: https://github.com/oracle/oci-cloud-controller-manager |
| 213 | +[regions]: https://docs.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm |
| 214 | +[win-webserver-deployment]: https://kubernetes.io/docs/concepts/windows/user-guide/ |
| 215 | +[win-webserver-dns]: https://kubernetes.io/docs/tutorials/services/connect-applications-service/#dns |
| 216 | +[windows-cluster-debug]: https://kubernetes.io/docs/tasks/debug/debug-cluster/windows/ |
| 217 | +[windows-containers-debug]: https://learn.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/common-problems |
| 218 | + |
| 219 | + |
0 commit comments