Terraform + Cloud-Init via extra_config & DataSourceVMware

Question:

Has anyone successfully used terraform + extra_config + Ubuntu cloud images to interface with Cloud Init, providing metadata/userdata? I'm hoping this will interact with DataSourceVMware, but can't be sure at this stage.

What I've been doing:

I am using Terraform to deploy Ubuntu cloud images on VMware vSphere 7. It's been easy enough to use vApp Properties:

... below code snipped from resource "vsphere_virtual_machine" "vm" { }

vapp {
    properties = {
        hostname = var.vm_Name_Lower
        instance-id = var.vm_Name_Lower
        user-data = base64encode(file("${path.module}/userdata.yml"))
    }
}

But any attempt to use extra_config has failed. I would like to be able to supply both guestinfo.metadata and guestinfo.userdata. But currently, my tests with metadata (seen below) have appeared to fail as the hostname is not set in the VM:

data "cloudinit_config" "metadata" {
    gzip = true
    base64_encode = true
    part {
        content_type = "text/cloud-config"
        content = <<-EOF
            local-hostname: testvm
            instance-id: testvm
        EOF
    }
}

... below code snipped from resource "vsphere_virtual_machine" "vm" { }

extra_config = {
    "guestinfo.metadata" = data.cloudinit_config.metadata.rendered
    "guestinfo.metadata.encoding" = "gzip+base64"
}

I can see the vSphere log entry to prove that the extra_config was sent:

config.extraConfig("guestinfo.metadata"): (key = "guestinfo.metadata", value = "H4sIAAAAAAAA/2SOTUvGMBCE74H8h/De11dPQsSDHz14qIK ... snipped

References:

• https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/cloudinit_config
• https://github.com/vmware-archive/cloud-init-vmware-guestinfo
• https://grantorchard.com/terraform-vsphere-cloud-init/
• https://github.com/rgl/terraform-vsphere-ubuntu-example/blob/master/main.tf

Version details:

Client system (on which terraform is run): Ubuntu 20.04.3 LTS
ESXi: 7.0.2 / Build: 18538813
vCenter Server: 7.0.2 / Build: 18455184
Cloud Image: https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-amd64.ova
Terraform v1.0.7
on linux_amd64
provider registry.terraform.io/hashicorp/template v2.2.0
provider registry.terraform.io/hashicorp/vsphere v1.24.3

The problem is that cloud-init has, by default, the OVF datasource provider invoked prior to the new VMware datasource (as of cloud-init 21.3). Terraform is providing data that the OVF datasource provider likes and therefore it processes the information. That explains why vApp Properties "user-data" accepts the cloud-config.

The solution is to remove the OVF datasource provider from cloud-init:

1. [Web Browser[ Download OVA: https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-amd64.ova
2. [VC UI] Deploy from OVF, accept defaults (except disk provisioning, use Thin Provisioning).
3. [VC UI] Edit Settings / VM Options / Boot Options / Boot Delay = 2000ms.
4. [VC UI] Open VM Console.
5. [VM Console] Power On VM.
6. [VM Console] Hold Shift on BIOS screen (to force GRUB to display menu).
7. [VM Console] Select Advanced Options for Ubuntu.
8. [VM Console] Select latest kernel version with "(recovery mode)" at the end.
9. [VM Console] Select "root / Drop to root shell prompt"
10. [VM Console] Press Enter for maintenance
11. [VM Console] # dpkg-reconfigure cloud-init
12. [VM Console] Deselect everything except VMware and None
13. [VM Console] # cloud-init clean
14. [VM Console] # shutdown -h now
15. [VC UI] Edit Settings / VM Options / Boot Options / Boot Delay = 0ms.
16. [VC UI] Convert to template

Doing the same with the Ubuntu cloud OVA's and I've found a little bit of a workaround on this. It's not ideal, as it involves a reboot, meaning you need a local-exec provisioner to detect when the instance is actually finished, but it works.

Say you have two yaml files that you're using for userdata - the first that you're passing with vApp Properties named vapp-userdata.yaml and then the second you're passing for the VMware datasource named guest-userdata.yaml

Something like this in Terraform

  vapp {
    properties = {
      user-data = base64encode(data.template_file.vapp_userdata[count.index].rendered)
    }
  }

  extra_config = {
    "guestinfo.metadata"          = base64encode(data.template_file.guest-metadata[count.index].rendered)
    "guestinfo.metadata.encoding" = "base64"
    "guestinfo.userdata"          = base64encode(data.template_file.userdata.rendered)
    "guestinfo.userdata.encoding" = "base64"
  }

In your vapp-userdata.yaml have a write_file action that overwrites the current datasource definitions and removes OVF as you've already reached that point.

write_files:
  - path: /etc/cloud/cloud.cfg.d/90_dpkg.cfg
    owner: root:root
    permissions: "0644"
    content: |
      datasource_list: [ VMware, None ]

Then at the bottom finish with a reboot

power_state:
  timeout: 600
  mode: reboot

When the VM reboots after that last command, it will read in the VMware datasource as defined in the new cloud-init config and will process your guest-userdata.yaml and metadata.yaml if you've defined that too.

As for detecting when it's finished, I'm still trying to figure out the best way of doing that. The easy way is you start nc -l 12345 at the end of guest-userdata.yaml and have a local provisioner that proceeds when it can connect to tcp/12345, but that leaves you with a netcat listener open on tcp/12345 which is less than ideal.

If you find a better way, reply :)

Edit

There's absolutely going to be a better way of doing this, but...

At the bottom of guest-userdata.yaml

runcmd:
  - mkdir -p /mnt/sharedfolder
  - sysctl -w vm.overcommit_memory=1
  - sysctl -w kernel.panic=10
  - sysctl -w kernel.panic_on_oops=1
  - curl https://releases.rancher.com/install-docker/${docker_version}.sh | sh
  - usermod -aG docker ubuntu
  - nc -l 1234 & ncpid=$!  #start nc and get PID
  - sleep 20
  - kill $ncpid #kill PID once Terraform has had time to connect

Then as a provisioner at the end of your .tf file

  provisioner "local-exec" {
    # Wait for cloud-init userdata cmds
    # Netcat: z (scan port only), w1 (wait 1 second)
    command = "count=0; until $(nc -zw1 ${self.default_ip_address} 1234); do sleep 1; count=`expr $count + 1`; done"
  }