Custom Certificate outside, VMware CA inside – Replacing vCenter 6.0’s SSL Certificate

A customer recently asked me “How do I replace the “external” SSL certificate of vCenter but still use VMCA in default mode?” Ever curious, I asked “Why?”. His security team required that any “externally” facing management web pages needed to have a custom certificate that chained up to the corporate PKI. But behind that, they were totally cool with using VMCA in default mode (with the self-generated root certificate) for things like ESXi servers and solution users.

Here’s a description of how this should look:

Lab setup

Windows Domain Controller (DC) – DC1.lab.local. This was built using my DC-Builder Powershell script I wrote for creating fully functional lab-based domain controllers. When this CA was set up, the issuer became “lab-DC1-CA” under “lab.local”. In the certificate this will be referenced as:

 CN = lab-DC1-CA, DC = lab, DC = local

VMware vCenter Service Appliance (VCSA) 6.0 – vcsa.lab.local. This is a fully embedded installation. VMCA will issue the certificates and they will be referenced as

 O = vcsa.lab.local, C = US, DC = local, DC = vsphere, CN = CA

Note: Although the two CA’s come from systems in the same “lab.local” domain name in this example, they are two entirely different CA’s with no shared chain between them. As such, the Windows CA could just as easily be in an entirely different domain and this would still work.

VMware ESXi server – esxi-a.lab.local. This is part of the cluster that is managed by vcsa.lab.local. The VMCA on vcsa.lab.local manages the certificates on esxi-a.lab.local.

Create a folder that you'll be able to download and upload files to during the exercise. On the VCSA I created a folder in the root account. "/root/Machine_SSL". I'll be connecting using WinSCP later. If you are using WinSCP to connect to VCSA, read VMware KB 2107727 and save yourself some frustration!

Generate the Certificate Signing Request:

The first thing we are going to do is generate the Certificate Signing Request (CSR) for the “Machine SSL” certificate. The Machine SSL certificate is the certificate you get when you open the vSphere Web Client in a web browser. It is used by the reverse proxy service on every management node, Platform Services Controller, and embedded deployment. You can replace the certificate on each node with a custom certificate.” To generate the CSR we’ll use the Certificate Manager utility. This is located in:

* Windows:

– C:Program Files\VMwarevCenter\Servervmcad\certificate-manager

* VCSA:

– /usr/lib/vmware-vmca/bin/certificate-manager

Run the utility and select Option 1:

Certificate Manager Utility

Now select Option 1 again to generate the CSR and provide the output directory path to write out the files created. In my example I used the “/root/machine_ssl” folder I previously mentioned:

Generate CSR

UPLOAD AND GENERATE THE NEW CERTIFICATE

Next, I logged into my Windows system (DC1.lab.local), fired up WinSCP and downloaded the Machine_SSL csr and key files:

WinSCP copy CSR from VCSA

Open the CSR file in your favorite text editor and copy the contents to the clipboard:

Copy CSR contents to Clipboard

Open the web page of the Microsoft Certificate Authority and select “advanced certificate request”. Paste the contents of CSR and select the previously created “vSphere 6.0” template. Submit the request. Note that in some cases, your CA Administrator may have an approval process in place that will require additional steps. These haven’t been implemented in this lab-focused environment:

Submit Certificate Request to MS CA

DOWNLOAD THE NEW CERTIFICATE AND ROOT CERTIFICATE

After submitting the CSR, you’ll be presented with a download page:

Certificate Download Page

On the download page, Select Base 64 encoded and click on Download Certificate. The downloaded file will be called “certnew.cer”. Rename this to “machine_ssl.cer”

Go back to the download web page and click on “Download certificate chain” (ensuring that “Base 64 encoded” is still selected). The downloaded file will be called “certnew.pb7”. Rename this to “cachain.pb7”

We are now going to export the CA Root certificate from the cachain.pb7 files. Right-click on the cachain.pb7 file and select “Open”:

Open cachain.pb7 file for export

Expand the list and click on the Certificates folder. Right-click on the CA root cert (lab-DC1-CA in this example), select All Tasks…Export:

Export CA cert

You’re now presented with the Certificate Export Wizard. Click Next and then select the format you want to use. Select “Base 64 encoded X.509 (CER)”:

Export cert wizard – select Base 64 encoded

In the next window, click on Browse… and provide a file location and filename. I used “root-64.cer” in the c:tempvcsa.lab.localMachine_SSL” folder I had previously created.

Save root-64.cer file

After successfully saving and exporting the root-64.cer file, it's time to upload it to vCenter. Here I'll use WinSCP again to copy the machine_ssl.cer and root-64.cer file to VCSA. 

Copy cer files to VCSA

REPLACE THE EXISTING CERTIFICATE WITH THE NEWLY GENERATED CERTIFICATE

Now that the files have been copied, open up the Certificate Manager Utility and select Option 1, Replace Machine SSL certificate with Custom Certificate. Provide the password to your administrator@vsphere.local account and select Option 2, “Import Custom Certificate(s) and key(s) to replace existing Machine SSL certificate” You will be prompted for following files:

* machine_ssl.cer

* machine_ssl.key

* root-64.cer

Import Custom Certificates via Certificate Manager Utility

Select “Y” to continue the operation. This may take a few minutes, depending on how your systems are configured:

70 percent completed

100 percent completed

Verification Time:

Now it’s time to check to see if all this work has paid off. One thing to remember before we start. Because the new Machine SSL cert has been issued by the CA on the domain controller, browsers that use the Windows certificate store will automatically recognize the vCenter web page. In my experience, Internet Explorer and Google Chrome will use the Windows certificate store. Mozilla Firefox does NOT use the Windows certificate store and as such you need to import the root certificate. Import the previously created root-64.cer into Firefox's Trusted Root Authority and then open the vCenter web page and you should be all set.

Import root-64.cer into Firefox                                                       Root cert imported into Firefox

Now open your vCenter login page and check the certificate being used to protect it:

vCenter web certificate in Firefox

You’ll see that the certificate has been verified by “lab-DC1-CA”. This is the CA running on the Windows domain controller. If I click on More Information and then View Certificate and scroll down to Issuer, you’ll see the hierarchy details of the certificate I referenced above in Lab Details.:

vCenter Certificate Hierarchy

So now that we’ve confirmed we’re using the Microsoft CA issued cert for vSphere Web Client logon, let’s see what’s going on after we log in.

VMCA in Default Mode:

In a previous blog post I went over the different modes you can run VMCA in. The “Default Mode” is when VMCA uses its own root certificate and issues certificates that change to the root. When you add an ESXi server to vCenter, VMCA will automatically issue it a certificate:

ESXi server certificate

You can see that the issuer matches the values from the Lab Setup portion of this blog:

 And the SSL thumbprint/fingerprint (SHA256) has a specific value ending in 1C:0D:E8 

ESXi Original SSL thumbprint

Now we’ll go back to vCenter and renew the ESXi certificate. Open vCenter, right click on the host and select:

Certificates-->Renew

Renew ESXi Certificate

Confirm by clicking on Yes:

Confirm ESXi renew certificate

Refresh your browser and repeat the steps to display the certificate properties. You’ll see in this example that the SHA256 thumbprint/fingerprint value has changed. The new ending values are now E8:E5:E1. VMCA has re-issued a completely new certificate:

ESXi Renewed SSL Thumbprint

Wrap up:

To wrap up, let’s review what we’ve done:

• Using the Certificate Manager utility and the SSL Certificate Template we created in a previous blog article, we generated a Certificate Signing Request for the externally-facing vSphere Web Client login page (The Machine SSL certificate).

• We submitted the CSR to the Microsoft Certificate Authority that was installed on our Domain Controller and downloaded the newly generated certificate and root CA certificate

• Using the Certificate Manager utility we replaced the Machine SSL certificate with the certificate generated by the Microsoft CA

• We verified that the vSphere Web Client login page is now using the Microsoft CA-issued certificate

• We then verified that the ESXi host was using a VMware CA issued certificate

• Finally, we renewed the VMCA issued certificate and verified the change.

Want to download this blog as a PDF document, click the link below:
https://drive.google.com/file/d/1zuxjOFfkD4cYTrKdUzT2y-Bki8xhKVhb/view





Abd El-Rahman Oreiby
Senior Data Center Engineer
Al Thuraya Security Egypt
www.abdelrahmanoreiby.weebly.com