VMware NSX Lab on ESXi - VCP6-NV Part2

In the previous post, we were able to get the 3x ESXi Nested hosts installed on the ESXi core server. Now we need to get the NSX Manager and Controllers setup.

There are a few steps needed here (well covered elsewhere):

1. Create Cluster
2. Move 3x ESXi VMs to cluster
3. Setup Distributed Switch
4. Make ESXi VMs talk to the dSwitch as their primary network path
5. Configure NTP on the host ESXi and all the ESXi VMs

Once this is completed, the system is ready for the deployment of the NSX Manager. This is an OVA file, so deploy it as a usual VM (to the ESXi core server, not one of the ESXi VMs !). You will have to go through and answer some questions passwords for CLI admin end privilege mode, DNS name and IP (this needs a static IP), DNS settings and NTP server list. Match this to your network and let the install complete.

Once the VM is running, you connect to the server via HTTPS. You will use the password you setup in the config above, and then this will get you in to the web GUI for the NSX Manager.

The 2 sections you need to change here (and won't need to change again) are Manage Appliance settings - for the NTP details, and Manage vCentre Registration - to connect the server to vCentre.

Once both are configured, you can exit the web interface, and go back to vCentre.

To access NSX manager, you need to go to the Networking and Security icon/menu option. Note this may take 1-2 mins to appear as the NSX manager initially talks to the vCentre to setup.

You need to prepare the Cluster for NSX and VXLan (done under the Host Preparation option under Installation and Upgrade)

The next item on the list is the NSX controller install. This is where the second trick came about. You can easily deploy the controller, but as per the way the LAB has been setup so far, it will sit in the Deploying phase until the system times it out and deletes the controller. So it never completes - and after some review I found that this was because any VM within one of the ESXi VMs (as all the controllers will be) does NOT is not able to get network access. I tested this by building a CentOS VM on one of the ESXi VMs, and I could not get any DHCP. Why ? 

The answer is HERE - Promiscuous Mode & Forged Transmits are required to be enabled on the dSwitch ! Makes complete sense when it is detailed like William Lam does, but of course you don't think about that when just creating the lab.

So make the required changes to the dSwitch, deploy the controller, and it will succeed. It took a while on my machine to be in an CONNECTED state (up to 10m) as I think it is looking for DNS resolution of the vSphere hosts names (which I don't have on my network), but in the end the controllers come up fine (note you can only build one at a time).

 

There is a recommended sequence for booting up the devices listed from VMware HERE. So it goes:

1. ESXi Host w/ vCenter Server
2. NSX Manager
3. ESXi VMs with the controllers (autostart should be on for the controllers)
4. Anything else

A strange issue that I had was that after the reboot of the ESXi server, the NSX manager would not show up in the security console of the Web Client (it said "No NSX Managers" available). This turned out to be a session issue, where the previous NSX session was still present. So to fix this - you have to close all the existing sessions and logout/login to the web client as per HERE. See image below.

Another issue that happened after a reboot was that once all was online and working, the NSX cluster would show as Not Ready (with no notes under it as to why the issue), and clicking Resolve would not change the status.
Some searching found THIS article (scroll to last comment), which suggested a reboot of the vCenter appliance. Completed that, and all green ! See original error screen that was encountered below. Some other guides can be found HERE as well on the issue of the NSX Cluster being Not Ready.

Now we are ready to LAB NSX SDN ! Looking forward to this part.

EDIT Feb 2019 - Having passed VCP6-NV, I have continued to work with this lab in different configurations. A few more notes came to be useful !
1. ESXi 6.5 GA is NOT supported by NSX 6.3 - you need to upgrade to 6.5 U1 or 6.7
2. For the distributed switch you need 2 NICs on each ESXi host (basically one for standard and one for distributed). Lots of trial and error here as I was playing with moving form the standard switch to the distributed version.