Let’s talk about VCF a little bit.
A recent VCF implementation conversation inspired this post.
You received your hardware to deploy your VCF Management Domain or Workload Domain. Exciting times after a long wait!
While you are preparing the host by installing the devices you need and the supported firmware is possible that your physical NICs are installed on a specific PCI slot in your vSAN ready node motherboard. This could result from how your server’s architecture was planned or unplanned.
The PCI slot used for your network adapters will reflect the VMNIC number assigned by ESXi.
If you are interested in reading more about how device names are assigned, check this article.
https://kb.vmware.com/s/article/2091560
Let’s imagine you are in this situation. Your VCF was deployed using vmnic2 and vmnic3 in the deployment parameter workbook.
This file is downloaded from the Cloud Builder appliance. It is an XLS file you complete with the details to perform the VCF Bring Up and then upload back to Cloud Builder. Once the file is converted to JSON format, all entries are validated before running the VCF installation.
Read more in the VCF Deployment Guide.
Now back to our situation and the reason for this post to exist.
It is supported to have the VCF deployment parameter workbook using pNICs vmnic2 and vmnic3.
You completed the bring up, and everything looks great as it should.
Hours later (maybe a day later), you have all the configurations you need, and it is time to stretch the vSAN cluster. After running the stretch cluster API call, you receive the below witness appliance error but don’t fully understand why if you followed all the VCF documentation.
Witness host is not registered with the vCenter
Remediation Message: Please register the witness host abcd with the Center and retry
Reason. The VCF documentation can be, for some of us, confusing.
We need to deploy the vSAN Witness Appliance in a different failure domain. This can be another VCF instance, vSphere infrastructure managed by you, your MSP or Cloud Provider.
Read more in the vSphere Documentation.
This appliance is a VM running ESXi. Add this nested ESXi as part of the vCenter Server inventory managed by the SDDC Manager.
This VM running ESXi runs outside VCF.
The nested ESXi is managed by the vCenter Server that is part of VCF. Either the Management Domain or Workload Domain getting the stretch configuration.
With this sorted, the stretch cluster task can be executed again and the subtask ‘Validate witness registration with Center’ completes.
But another subtask fails.
Failed to add host
Message: Failed to add host abcd to DVS vds01
Cause:
Type: java.util.NoSuchElementException
Message: No available vmnic found on the host to DVSThe logfile /var/log/vmware/vcf/domainmanager.log shows the scripted task failed to use vmnic2 and vmnic3.
2022-02-11T06:28:09.003+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.v.c.f.p.a.i.AddHostsToDvsAction,dm-exec-3] vmnics from dvs vds01 spec are []
2022-02-11T06:28:09.003+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.v.c.f.p.a.i.AddHostsToDvsAction,dm-exec-3] vmnic spec is not defined
2022-02-11T06:28:09.004+0000 WARN [vcf_dm,d643ac99a5a84698,fb95] [c.v.v.c.f.p.a.i.AddHostsToDvsAction,dm-exec-3] Cannot add host esxiabcd11.nz to dvs vds01, as free vmnics are not available on the host2022-02-11T06:28:08.968+0000 INFO [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-3] Successfully logged in to https://vcenterabcd.nz:443/sdk
2022-02-11T06:28:08.968+0000 INFO [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Retrieving ManagedObjectReference for host esxiabcd11.nz
2022-02-11T06:28:08.968+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Finding hosts with fqdn esxiabcd11.nz
2022-02-11T06:28:08.981+0000 INFO [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] In total 1 vmnics are in-use: ["vmnic2"]
2022-02-11T06:28:08.981+0000 INFO [vcf_dm,d643ac99a5a84698,fb95] [c.v.v.c.f.p.a.i.AddHostsToDvsAction,dm-exec-3] In Use Vmnics on Host esxiabcd11.nz : [vmnic2]
2022-02-11T06:28:08.995+0000 INFO [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] In total 1 vmnics are in-use: ["vmnic2"]
2022-02-11T06:28:08.995+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Listing all vmnics...
2022-02-11T06:28:08.996+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Checking physical nic vmnic0
2022-02-11T06:28:08.996+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Checking physical nic vmnic1
2022-02-11T06:28:08.996+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Checking physical nic vmnic2
2022-02-11T06:28:08.996+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Checking physical nic vmnic3
2022-02-11T06:28:08.996+0000 INFO [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Physical nic vmnic3 is available (connected and not in-use), speed: 10000MB
2022-02-11T06:28:08.996+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Checking physical nic vmnic4
2022-02-11T06:28:08.997+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Checking physical nic vmnic5
2022-02-11T06:28:08.997+0000 INFO [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] In total 1 vmnics are available (connected and not in-use): ["vmnic3"]
2022-02-11T06:28:08.997+0000 INFO [vcf_dm,d643ac99a5a84698,fb95] [c.v.v.c.f.p.a.i.AddHostsToDvsAction,dm-exec-3] Available Vmnics on Host esxiabcd11.nz : [vmnic3]
2022-02-11T06:28:08.997+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.s.adapter.util.VdsUtils,dm-exec-3] Dvs to Vmnic map {vds01=[]}
2022-02-11T06:28:08.997+0000 INFO [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Checking host esxiabcd11.nz attached to vds vds01:
2022-02-11T06:28:08.997+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-3] Finding host with fqdn esxiabcd11.nz
2022-02-11T06:28:09.003+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.v.c.f.p.a.i.AddHostsToDvsAction,dm-exec-3] vmnics from dvs vds01 spec are []
2022-02-11T06:28:09.003+0000 DEBUG [vcf_dm,d643ac99a5a84698,fb95] [c.v.v.c.f.p.a.i.AddHostsToDvsAction,dm-exec-3] vmnic spec is not defined
2022-02-11T06:28:09.004+0000 WARN [vcf_dm,d643ac99a5a84698,fb95] [c.v.v.c.f.p.a.i.AddHostsToDvsAction,dm-exec-3] Cannot add host esxiabcd11.nz to dvs vds01, as free vmnics are not available on the hostWorkaround for this bug.
- Perform the “Remove Host” operation from VCF sddc-manager for the hosts that are used for the stretch cluster operation
- De-commission the above hosts from the sddc-manager
- Restore ESXi to a clean state
- Commission the hosts back to sddc-manager
- Prepare the input JSON to include the host network specifications. Check below example below and API Documentation
- Perform the stretch cluster API validation with the input JSON
- Perform the stretch cluster operation with the input JSON
JSON Example and API Documentation
{
"clusterStretchSpec": {
"hostSpecs": [ {
"id": "ee47aae3-f703-4437-a96e-8495d1aa2d31",
"licenseKey":" XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
"hostNetworkSpec": {
"vmNics": [
{
"id": "vmnic2",
"vdsName": "<<VDS Name>>"
},
{
"id": "vmnic3",
"vdsName": "<<VDS Name>>"
}
:
:
:
]
}
}, {
"id": "f46703a3-d552-42f4-ae87-ebe50d81a60c",
"licenseKey":" XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
"hostNetworkSpec": {
"vmNics": [
{
"id": "vmnic2",
"vdsName": "<<VDS Name>>"
},
{
"id": "vmnic3",
"vdsName": "<<VDS Name>>"
}
:
:
:
]
}
}, {
"id": "939ed36b-a340-4f46-a92f-33518ad7893b",
"licenseKey":" XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
"hostNetworkSpec": {
"vmNics": [
{
"id": "vmnic2",
"vdsName": "<<VDS Name>>"
},
{
"id": "vmnic3",
"vdsName": "<<VDS Name>>"
}
:
:
:
]
}
}, {
"id": "f5a8b7bc-b56b-4fdc-a9d5-366b6e9a1482",
"licenseKey":" XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
"hostNetworkSpec": {
"vmNics": [
{
"id": "vmnic2",
"vdsName": "<<VDS Name>>"
},
{
"id": "vmnic3",
"vdsName": "<<VDS Name>>"
}
:
:
:
]
}
} ],
"secondaryAzOverlayVlanId": 123,
"witnessSpec": {
"fqdn":"c",
"vsanCidr":"",
" "vsanIp":""
}
}
}We hope this post is helpful. Send us your comment using the button below if you want more information.
You can also reach out and connect with us on social media using here and here
Disclaimer
Last updated: May 17, 2018
The information contained on www.bakingclouds.com website (the “Service”) is for general information purposes only.
Baking Clouds assumes no responsibility for errors or omissions in the contents on the Service.
In no event shall Baking Clouds be liable for any special, direct, indirect, consequential, or incidental damages or any damages whatsoever, whether in an action of contract, negligence or another tort, arising out of or in connection with the use of the Service or the contents of the Service. Baking Clouds reserves the right to make additions, deletions, or modifications to the contents on the Service at any time without prior notice.