Merge pull request jenkins-x#235 from sergiogiuffrida/gcp-artifact

Gcp artifact

Author: jenkins-x-bot

CONTRIBUTING.md

@@ -32,14 +32,15 @@ The CI build will fail on lint issues.  To format and run locally execute `make
 
 Helpful tip if using an IDE like intelij you can enable file watchers and auto format terraform files.
 
-## Generating terraform docs for the readme
+## Generating terraform docs and table of content in README.md
 
-### Prerequisite
+### Prerequisites
 
 - terraform-docs - for OSX `brew install terraform-docs` (v0.12+)
+- markdown-toc - `npm install --save markdown-toc`
 
-If you add or remove any terraform input or outputs you will need to regenerate the docs. Note that running the following will automatically update the README.md sections
+If you add or remove any terraform input or outputs or change headings in README.md you will need to regenerate the docs. Note that running the following will automatically update the README.md sections
 
 ```
-make markdown-table
+make README.md
 ```

Makefile

@@ -43,7 +43,7 @@ fmt: ## Reformats Terraform files accoring to standard
 	terraform fmt -diff -recursive
 
 .PHONY: fmt-check
-fmt: ## Checks if Terraform files are formatted accoring to standard
+fmt-check: ## Checks if Terraform files are formatted accoring to standard
 	terraform fmt -check -diff -recursive
 
 .PHONY: test
@@ -60,6 +60,11 @@ clean: ## Deletes temporary files
 	@rm -f jx-requirements.yml
 
 .PHONY: markdown-table
-markdown-table: ## Creates markdown tables for in- and output of this module
+markdown-table: ## Creates markdown tables for in- and output of this module using https://terraform-docs.io/
 	terraform-docs .
-			
+
+.PHONY: markdown-toc
+markdown-toc: ## Creates table of content using https://github.com/jonschlinkert/markdown-toc
+	markdown-toc -i README.md  --bullets "-"
+
+README.md: markdown-table markdown-toc

README.md

@@ -9,27 +9,32 @@ compatibility issues to be aware of are around provider requirements.  For more
 
 This repo contains a [Terraform](https://www.terraform.io/) module for provisioning a Kubernetes cluster for [Jenkins X](https://jenkins-x.io/) on [Google Cloud](https://cloud.google.com/).
 
-<!-- TOC -->
-
-- [Jenkins X GKE Module](#jenkins-x-gke-module)
-  - [What is a Terraform module](#what-is-a-terraform-module)
-  - [How do you use this module](#how-do-you-use-this-module)
-    - [Prerequisites](#prerequisites)
-    - [Cluster provisioning](#cluster-provisioning)
-      - [Inputs](#inputs)
-      - [Outputs](#outputs)
-    - [Running `jx boot`](#running-jx-boot)
-    - [Using a custom domain](#using-a-custom-domain)
-    - [Production cluster considerations](#production-cluster-considerations)
-  - [FAQ](#faq)
-    - [How do I get the latest version of the terraform-google-jx module](#how-do-i-get-the-latest-version-of-the-terraform-google-jx-module)
-    - [How to I specify a specific google provider version](#how-to-i-specify-a-specific-google-provider-version)
-    - [Why do I need Application Default Credentials](#why-do-i-need-application-default-credentials)
-  - [Development](#development)
-    - [Releasing](#releasing)
-  - [How do I contribute](#how-do-i-contribute)
-
-<!-- /TOC -->
+<!-- toc -->
+
+- [What is a Terraform module](#what-is-a-terraform-module)
+- [How do you use this module](#how-do-you-use-this-module)
+  - [Prerequisites](#prerequisites)
+  - [Cluster provisioning](#cluster-provisioning)
+    - [Inputs](#inputs)
+    - [Outputs](#outputs)
+  - [Artifact Registry in setup with multiple Jenkins X clusters](#artifact-registry-in-setup-with-multiple-jenkins-x-clusters)
+  - [Migration from Container to Artifact Registry](#migration-from-container-to-artifact-registry)
+    - [Configuration Note](#configuration-note)
+    - [Migration Options](#migration-options)
+      - [Don't Migrate Existing Images](#dont-migrate-existing-images)
+      - [Migrate Existing Images](#migrate-existing-images)
+  - [Using a custom domain](#using-a-custom-domain)
+  - [Production cluster considerations](#production-cluster-considerations)
+  - [Configuring a Terraform backend](#configuring-a-terraform-backend)
+- [FAQ](#faq)
+  - [How do I get the latest version of the terraform-google-jx module](#how-do-i-get-the-latest-version-of-the-terraform-google-jx-module)
+  - [How to I specify a specific google provider version](#how-to-i-specify-a-specific-google-provider-version)
+  - [Why do I need Application Default Credentials](#why-do-i-need-application-default-credentials)
+- [Development](#development)
+  - [Releasing](#releasing)
+- [How do I contribute](#how-do-i-contribute)
+
+<!-- tocstop -->
 
 ## What is a Terraform module
 
@@ -114,6 +119,10 @@ The following two paragraphs provide the full list of configuration and output v
 | <a name="input_apex_domain"></a> [apex\_domain](#input\_apex\_domain) | The parent / apex domain to be used for the cluster | `string` | `""` | no |
 | <a name="input_apex_domain_gcp_project"></a> [apex\_domain\_gcp\_project](#input\_apex\_domain\_gcp\_project) | The GCP project the apex domain is managed by, used to write recordsets for a subdomain if set.  Defaults to current project. | `string` | `""` | no |
 | <a name="input_apex_domain_integration_enabled"></a> [apex\_domain\_integration\_enabled](#input\_apex\_domain\_integration\_enabled) | Flag that when set attempts to create delegation records in apex domain to point to domain created by this module | `bool` | `true` | no |
+| <a name="input_artifact_description"></a> [artifact\_description](#input\_artifact\_description) | artifact registry repository Description | `string` | `"jenkins-x Docker Repository"` | no |
+| <a name="input_artifact_enable"></a> [artifact\_enable](#input\_artifact\_enable) | Create artifact registry repository | `bool` | `true` | no |
+| <a name="input_artifact_location"></a> [artifact\_location](#input\_artifact\_location) | artifact registry repository Location | `string` | `"us-central1"` | no |
+| <a name="input_artifact_repository_id"></a> [artifact\_repository\_id](#input\_artifact\_repository\_id) | artifact registry repository Name | `string` | `"oci"` | no |
 | <a name="input_autoscaler_location_policy"></a> [autoscaler\_location\_policy](#input\_autoscaler\_location\_policy) | location policy for primary node pool | `string` | `"ANY"` | no |
 | <a name="input_autoscaler_max_node_count"></a> [autoscaler\_max\_node\_count](#input\_autoscaler\_max\_node\_count) | primary node pool max nodes | `number` | `5` | no |
 | <a name="input_autoscaler_min_node_count"></a> [autoscaler\_min\_node\_count](#input\_autoscaler\_min\_node\_count) | primary node pool min nodes | `number` | `3` | no |
@@ -188,33 +197,52 @@ The following two paragraphs provide the full list of configuration and output v
 | <a name="output_vault_bucket_url"></a> [vault\_bucket\_url](#output\_vault\_bucket\_url) | The URL to the bucket for secret storage |
 <!-- END_TF_DOCS -->
 
-### Running `jx boot`
 
-A terraform output (_jx\_requirements_) is available after applying this Terraform module.
+### Artifact Registry in setup with multiple Jenkins X clusters
+In a multi cluster setup, you should leave the value of `artifact_enable` as `true` **only in a
+development cluster** and set `artifact_enable = false` for other clusters. A development cluster is
+one where application build pipelines are executed. If you have multiple development clusters you
+can set `artifact_repository_id` to different values for them. Alternatively you can have
+`artifact_enable = true` in one and manually copy the values of `cluster.registry` and
+`cluster.dockerRegistryOrg` from `jx-requirements.yml` from that cluster repository to the other
+cdevelopment cluster repositories.
 
-```sh
-terraform output jx_requirements
-```
+If you leave `artifact_enable` as `true` for multiple clusters and don't override
+`artifact_repository_id` terraform will fail since it can't create an already existing repository.
 
-This `jx_requirements` output can be used as input to [Jenkins X Boot](https://jenkins-x.io/docs/getting-started/setup/boot/) which is responsible for installing all the required Jenkins X components into the cluster created by this module.
+### Migration from Container to Artifact Registry
 
-![Jenkins X Installation/Update Flow](./images/terraform_google_jx.png)
+Google has deprecated `gcr.io` and now recommends the use of Artifact Registry. The default of this module is now to create and use a repository in Artifact Registry for container images.
 
-:warning: **Note**: The generated _jx-requirements_ is only used for the first run of `jx boot`.
-During this first run of `jx boot` a git repository containing the source code for Jenkins X Boot is created.
-This (_new_) repository contains a _jx-requirements.yml_ (_which is now ahead of the jx-requirements output from terraform_) used by successive runs of `jx boot`.
+Google GKE clusters automatically have permissions to download from the Artifact Registry. For multi cluster setups across different projects, additional permission configurations may be necessary.
 
-Execute:
+#### Configuration Note
+The `jx-requirements.yml` will be automatically updated by the Jenkins X boot job when triggered by a push to the main branch of the cluster repository.
 
-```sh
-terraform output jx_requirements > <some_empty_dir>/jx-requirements.yml
-# jenkins-x creates the environment repository directory localy before pushing to the Git server of choice
-cd <some_empty_dir>
-jx boot --requirements jx-requirements.yml
+#### Migration Options
+Here are two strategies for transitioning container images from `gcr.io` to the Artifact Registry:
+
+##### Don't Migrate Existing Images
+- Continue developing applications as usual. New images, upon their release, will be pushed to the Artifact Registry.
+- **Important**: Ensure that all builds are triggered and applications are promoted before Google completely shuts down the Container Registry. This step is critical to avoid disruptions in service.
+To identify which images from you container registry are currently used in your cluster, you can use the following command line (replace `project_id` with your actual GCP project id):
+```bash
+kubectl get pods --all-namespaces -o jsonpath="{range .items[*].spec['initContainers', 'containers'][*]}{.image}{'\n'}{end}" | fgrep gcr.io/project_id | sort -u
 ```
 
-You are prompted for any further required configuration.
-The number of prompts depends on how much you have [pre-configured](#inputs) via your Terraform variables.
+##### Migrate Existing Images
+If you have a large number of applications running that are unlikely to be released in the coming
+year, migration of images to artifact registry while retaining the image names (in the domain
+`gcr.io`) could be considered. This means that existing helm charts will continue to work.
+
+This process is not supported by this terraform module, instead you need to follow the steps outlined in the guide 
+[Set up repositories with gcr.io domain support](https://cloud.google.com/artifact-registry/docs/transition/setup-gcr-repo).
+These steps include create the a repository in Artifact Registry, migrate images to it from
+container registry and enable redirection of gcr.io traffic.
+
+If you keep the default settings for this module it will create another artifact repository that
+will be used for new images. If you want to use `gcr.io` artifact repository for new images you
+should set `artifact_enable = false`.
 
 ### Using a custom domain
 

main.tf

@@ -129,6 +129,12 @@ resource "google_project_service" "container_api" {
   disable_on_destroy = false
 }
 
+resource "google_project_service" "artifactregistry" {
+  provider           = google
+  project            = var.gcp_project
+  service            = "artifactregistry.googleapis.com"
+  disable_on_destroy = false
+}
 // ----------------------------------------------------------------------------
 // Create Kubernetes cluster
 // ----------------------------------------------------------------------------
@@ -148,6 +154,9 @@ module "cluster" {
   ip_range_services          = var.ip_range_services
   max_pods_per_node          = var.max_pods_per_node
   bucket_location            = var.bucket_location
+  artifact_enable            = var.artifact_enable
+  artifact_location          = var.artifact_location 
+  artifact_repository_id     = var.artifact_repository_id
   jenkins_x_namespace        = var.jenkins_x_namespace
   force_destroy              = var.force_destroy
   enable_primary_node_pool = var.enable_primary_node_pool
@@ -267,6 +276,10 @@ locals {
     git_owner_requirement_repos = var.git_owner_requirement_repos
     dev_env_approvers           = var.dev_env_approvers
     lets_encrypt_production     = var.lets_encrypt_production
+    // GCP Artifact
+    enable_artifact        = var.artifact_enable
+    registry               = module.cluster.artifact_registry_repository
+    docker_registry_org    = module.cluster.artifact_registry_repository_name
     // Storage buckets
     log_storage_url        = module.cluster.log_storage_url
     report_storage_url     = module.cluster.report_storage_url

modules/cluster/outputs.tf

@@ -45,3 +45,11 @@ output "tekton_sa_email" {
 output "tekton_sa_name" {
   value = google_service_account.tekton_sa.name
 }
+
+output "artifact_registry_repository" {
+  value = local.artifact_registry_repository
+}
+
+output "artifact_registry_repository_name" {
+  value = local.artifact_repositoryid
+}

modules/cluster/serviceaccount.tf

@@ -237,3 +237,18 @@ resource "google_service_account_iam_member" "boot_sa_workload_identity_user" {
   role               = "roles/iam.workloadIdentityUser"
   member             = "serviceAccount:${var.gcp_project}.svc.id.goog[jx-git-operator/jx-boot-job]"
 }
+
+// Artifact Registry
+resource "google_artifact_registry_repository_iam_member" "writers" {
+  count      = var.artifact_enable ? 1 : 0
+  project    = google_artifact_registry_repository.repo[count.index].project
+  location   = google_artifact_registry_repository.repo[count.index].location
+  repository = google_artifact_registry_repository.repo[count.index].name
+
+  role   = "roles/artifactregistry.writer"
+  member   = "serviceAccount:${google_service_account.tekton_sa.email}"
+
+  depends_on = [
+    google_artifact_registry_repository.repo
+  ]
+}

modules/cluster/storage.tf

@@ -33,3 +33,19 @@ resource "google_storage_bucket" "repository_bucket" {
 
   force_destroy = var.force_destroy
 }
+
+// Artifact Registry
+locals {
+   artifact_repositoryid = var.artifact_repository_id == "" ? var.cluster_name : var.artifact_repository_id
+   artifact_registry_repository = "${var.artifact_location}-docker.pkg.dev/${var.gcp_project}" 
+}
+
+resource "google_artifact_registry_repository" "repo" {
+      count                  = var.artifact_enable ? 1 : 0
+      format                 = "DOCKER"
+      location               = var.artifact_location
+      mode                   = "STANDARD_REPOSITORY"
+      project                = var.gcp_project
+      repository_id          = local.artifact_repositoryid
+      description            = var.artifact_description
+}

modules/cluster/variables.tf

@@ -517,4 +517,26 @@ variable "delete_protect" {
   type        = bool
   default     = true
 }
+// Artifact Registry
+variable "artifact_location" {
+  description = "artifact registry repository Location"
+  type        = string
+  default     = "us-central1" 
+}
 
+variable "artifact_repository_id" {
+  description = "artifact registry repository Name, Defaul Cluster Name"
+  type        = string
+  default     = "oci"
+}
+variable "artifact_description" {
+  description = "artifact registry repository Description"
+  type        = string
+  default     = "jenkins-x Docker Repository" 
+}
+
+variable "artifact_enable" {
+  description = "Create artifact registry repository"
+  type        = bool
+  default     = true
+}

modules/jx-requirements-v3.yml.tpl

@@ -6,6 +6,10 @@ spec:
     project: "${gcp_project}"
     provider: gke
     zone: "${zone}"
+%{ if enable_artifact }
+    dockerRegistryOrg: "${docker_registry_org}"
+    registry: "${registry}"
+%{ endif }
   ingress:
 %{ if subdomain != "" }
     domain: "${subdomain}.${apex_domain}"

variables.tf

@@ -358,3 +358,25 @@ variable "delete_protect" {
   default     = true
 }
 
+// GCP Artifact
+variable "artifact_location" {
+  description = "artifact registry repository Location"
+  type        = string
+  default     = "us-central1" 
+}
+variable "artifact_repository_id" {
+  description = "artifact registry repository Name"
+  type        = string
+  default     = "oci"
+}
+variable "artifact_description" {
+  description = "artifact registry repository Description"
+  type        = string
+  default     = "jenkins-x Docker Repository" 
+}
+
+variable "artifact_enable" {
+  description = "Create artifact registry repository"
+  type        = bool
+  default     = true
+}