feat: Switch to native Terraform resources for hub registration and ACM (#947)

Author: morgante

docs/upgrading_to_v21.0.md

@@ -1,5 +1,4 @@
 # Upgrading to v21.0
-
 The v21.0 release of *kubernetes-engine* is a backwards incompatible
 release.
 
@@ -14,3 +13,135 @@ The [Terraform Kubernetes Engine Module](https://github.com/terraform-google-mod
 ### Kubernetes Provider upgrade
 The Terraform Kubernetes Engine module now requires version 2.10 or higher of
 the Kubernetes Provider.
+
+### Hub module rewrite
+The old [Hub submodule](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/v20.0.0/modules/hub)
+has been renamed to `hub-legacy` and deprecated. It is replaced with a new [fleet membership](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/master/modules/fleet-membership)
+module to handle registering GKE clusters to [fleets](https://cloud.google.com/anthos/multicluster-management/fleets) using the native API.
+
+The new module relies exclusively on native Terraform resources and should therefore be more robust.
+
+### Migrating
+For GKE clusters, you should update your configuration as follows:
+
+```diff
+ module "register" {
+- source = "terraform-google-modules/kubernetes-engine/google//modules/hub"
+- version = "~> 20.0"
++ source = "terraform-google-modules/kubernetes-engine/google//modules/fleet-membership"
++ version = "~> 21.0"
+
+ project_id = "my-project-id"
+ cluster_name = "my-cluster-name"
+- gke_hub_membership_name = "gke-membership"
++ membership_name = "gke-hub-membership"
+ location = module.gke.location
+- cluster_endpoint = module.gke.endpoint
+- gke_hub_sa_name = "sa-for-kind-cluster-membership"
+- use_kubeconfig = true
+- labels = "testlabel=usekubecontext"
+- module_depends_on = [module.gke]
+}
+```
+
+You also need to follow these migration steps:
+
+1. Remove the old module from your state:
+
+ ```
+ terraform state rm module.register
+ ```
+
+2. Remove the cluster from the fleet:
+
+ ```
+ gcloud container fleet memberships delete gke-hub-membership-name
+ ```
+
+3. Apply the new configuration to re-register the cluster:
+
+ ```
+ terraform apply
+ ```
+
+#### Legacy module
+**The native API only supports registering GKE clusters**. Therefore, the old hub module is preserved as `hub-legacy`.
+
+You can continue using it by updating your configuration to point to the new location.
+
+```diff
+ module "register" {
+- source = "terraform-google-modules/kubernetes-engine/google//modules/hub"
+- version = "~> 20.0"
++ source = "terraform-google-modules/kubernetes-engine/google//modules/hub-legacy"
++ version = "~> 21.0"
+
+ project_id = "my-project-id"
+ cluster_name = "my-cluster-name"
+ location = module.gke.location
+ cluster_endpoint = module.gke.endpoint
+ }
+```
+
+### Anthos Config Management (ACM) and Config Sync Module Rewrite
+Together with the rewrite of the Hub module, the ACM module also has been rewritten to use native resources.
+
+You will need to follow these migration steps:
+
+1. Update your configuration to use the new module:
+
+ ```diff
+ module "acm" {
+ source = "terraform-google-modules/kubernetes-engine/google//modules/acm"
+ - version = "~> 20.0"
+ + version = "~> 21.0"
+
+ project_id = "my-project-id"
+ cluster_name = "simple-zonal-cluster"
+ location = "us-central1-a"
+ - cluster_endpoint = module.auth.host
+
+ sync_repo = "git@github.com:GoogleCloudPlatform/csp-config-management.git"
+ sync_branch = "1.0.0"
+ policy_dir = "foo-corp"
+
+ secret_type = "ssh"
+ }
+ ```
+
+1. Make sure you have the `kubernetes` provider configured:
+
+ ```hcl
+ provider "kubernetes" {
+ cluster_ca_certificate = module.auth.cluster_ca_certificate
+ host = module.auth.host
+ token = module.auth.token
+ }
+ ```
+
+1. Remove the old module from your state:
+
+ ```
+ terraform state rm module.acm
+ ```
+
+2. Import the old `git-creds` secret into Terraform:
+
+ ```
+ terraform import 'module.acm.module.acm_operator.kubernetes_secret_v1.creds' 'config-management-system/git-creds'
+ ```
+
+3. Apply the new configuration to re-register ACM and confirm everything is working:
+
+ ```
+ terraform apply
+ ```
+
+#### Feature Activation
+
+Only the first cluster in a fleet should activate the ACM fleet feature.
+Other clusters should disable feature activation by setting `enable_fleet_feature = false`.
+
+#### Config Sync Module Removed
+The dedicated Config Sync submodule has been removed.
+To use Config Sync, just invoke the ACM module with `enable_policy_controller = false`.

examples/simple_zonal_with_acm/README.md

@@ -4,23 +4,38 @@ This example illustrates how to create a simple cluster and install [Anthos Conf
 
 It incorporates the standard cluster module and the [ACM install module](../../modules/acm).
 
+## Verifying Success
+
+After applying the Terraform configuration, you can run the following commands to verify that your cluster has synced correctly:
+
+1. Check ACM install status:
+
+ ```
+ gcloud config set project $(terraform output --raw project_id)
+ gcloud alpha container hub config-management status
+ ```
+
+2. Connect to the cluster:
+
+ ```
+ gcloud container clusters get-credentials $(terraform output --raw cluster_name) --zone=$(terraform output --raw location)
+ ```
+
+3. Confirm the `shipping-dev` namespace was created:
+
+ ```
+ kubectl describe ns shipping-dev
+ ```
+
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
-| acm\_policy\_dir | Subfolder containing configs in ACM Git repo | `string` | `"foo-corp"` | no |
-| acm\_sync\_branch | Anthos config management Git branch | `string` | `"1.0.0"` | no |
-| acm\_sync\_repo | Anthos config management Git repo | `string` | `"git@github.com:GoogleCloudPlatform/csp-config-management.git"` | no |
 | cluster\_name\_suffix | A suffix to append to the default cluster name | `string` | `""` | no |
-| ip\_range\_pods | The secondary ip range to use for pods | `any` | n/a | yes |
-| ip\_range\_services | The secondary ip range to use for services | `any` | n/a | yes |
-| network | The VPC network to host the cluster in | `any` | n/a | yes |
-| operator\_path | Path to the operator yaml config. If unset, will download from GCS releases. | `string` | `null` | no |
 | project\_id | The project ID to host the cluster in | `any` | n/a | yes |
-| region | The region to host the cluster in | `any` | n/a | yes |
-| subnetwork | The subnetwork to host the cluster in | `any` | n/a | yes |
-| zones | The zone to host the cluster in (required if is a zonal cluster) | `list(string)` | n/a | yes |
+| region | The region to host the cluster in | `string` | `"us-central1"` | no |
+| zone | The zone to host the cluster in | `string` | `"us-central1-a"` | no |
 
 ## Outputs
 
@@ -36,7 +51,7 @@ It incorporates the standard cluster module and the [ACM install module](../../m
 | location | n/a |
 | master\_kubernetes\_version | The master Kubernetes version |
 | network | n/a |
-| project\_id | n/a |
+| project\_id | Standard test outputs |
 | region | n/a |
 | service\_account | The default service account used for running nodes. |
 | subnetwork | n/a |

examples/simple_zonal_with_acm/acm.tf

@@ -15,13 +15,14 @@
  */
 
 module "acm" {
- source = "../../modules/acm"
- project_id = var.project_id
- location = module.gke.location
- cluster_name = module.gke.name
- sync_repo = var.acm_sync_repo
- sync_branch = var.acm_sync_branch
- policy_dir = var.acm_policy_dir
- cluster_endpoint = module.gke.endpoint
- operator_path = var.operator_path
+ source = "../../modules/acm"
+ project_id = var.project_id
+ location = module.gke.location
+ cluster_name = module.gke.name
+
+ sync_repo = "git@github.com:GoogleCloudPlatform/csp-config-management.git"
+ sync_branch = "1.0.0"
+ policy_dir = "foo-corp"
+
+ secret_type = "ssh"
 }

examples/simple_zonal_with_acm/main.tf

@@ -18,6 +18,10 @@ locals {
  cluster_type = "simple-zonal"
 }
 
+provider "google" {
+ region = var.region
+}
+
 data "google_client_config" "default" {}
 
 provider "kubernetes" {
@@ -27,17 +31,20 @@ provider "kubernetes" {
 }
 
 module "gke" {
- source = "../../"
- project_id = var.project_id
- name = "${local.cluster_type}-cluster${var.cluster_name_suffix}"
- regional = false
- region = var.region
- zones = var.zones
- network = var.network
- subnetwork = var.subnetwork
- ip_range_pods = var.ip_range_pods
- ip_range_services = var.ip_range_services
- service_account = "create"
+ source = "../../"
+ project_id = var.project_id
+ regional = false
+ region = var.region
+ zones = [var.zone]
+
+ name = "${local.cluster_type}-cluster${var.cluster_name_suffix}"
+
+ network = google_compute_network.main.name
+ subnetwork = google_compute_subnetwork.main.name
+ ip_range_pods = google_compute_subnetwork.main.secondary_ip_range[0].range_name
+ ip_range_services = google_compute_subnetwork.main.secondary_ip_range[1].range_name
+
+ service_account = "create"
  node_pools = [
  {
  name = "acm-node-pool"

test/fixtures/simple_zonal/network.tf renamed to examples/simple_zonal_with_acm/network.tf

@@ -1,5 +1,5 @@
 /**
- * Copyright 2018 Google LLC
+ * Copyright 2021 Google LLC
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -20,16 +20,14 @@ resource "random_string" "suffix" {
  upper = false
 }
 
-provider "google" {
- project = var.project_ids[1]
-}
-
 resource "google_compute_network" "main" {
+ project = var.project_id
  name = "cft-gke-test-${random_string.suffix.result}"
  auto_create_subnetworks = false
 }
 
 resource "google_compute_subnetwork" "main" {
+ project = var.project_id
  name = "cft-gke-test-${random_string.suffix.result}"
  ip_cidr_range = "10.0.0.0/17"
  region = var.region

examples/simple_zonal_with_acm/outputs.tf

@@ -25,7 +25,8 @@ output "client_token" {
 }
 
 output "ca_certificate" {
- value = module.gke.ca_certificate
+ value = module.gke.ca_certificate
+ sensitive = true
 }
 
 output "service_account" {
@@ -38,3 +39,48 @@ output "acm_git_creds_public" {
  value = module.acm.git_creds_public
 }
 
+# Standard test outputs
+output "project_id" {
+ value = var.project_id
+}
+
+output "region" {
+ value = module.gke.region
+}
+
+output "cluster_name" {
+ description = "Cluster name"
+ value = module.gke.name
+}
+
+output "network" {
+ value = google_compute_network.main.name
+}
+
+output "subnetwork" {
+ value = google_compute_subnetwork.main.name
+}
+
+output "location" {
+ value = module.gke.location
+}
+
+output "ip_range_pods" {
+ description = "The secondary IP range used for pods"
+ value = google_compute_subnetwork.main.secondary_ip_range[0].range_name
+}
+
+output "ip_range_services" {
+ description = "The secondary IP range used for services"
+ value = google_compute_subnetwork.main.secondary_ip_range[1].range_name
+}
+
+output "zones" {
+ description = "List of zones in which the cluster resides"
+ value = module.gke.zones
+}
+
+output "master_kubernetes_version" {
+ description = "The master Kubernetes version"
+ value = module.gke.master_version
+}