• Note: “Starting with version 4.0, VMware NSX-T Data Center is now known as VMware NSX.”1

Nodes and Configurations

test
test
Host Requirement Management IP Address(/24)
Physical ESXi 32 vCPU, 64 GB RAM, 2TB HDD  
 Virtual ESXi 1 10 vCPU, 16 GB RAM, 500GB HDD 10.103.7.131
  NSX Manager 1 vCPU, 5GB RAM [4 vCPU, 16GB RAM] 10.103.7.135
 Virtual ESXi 2 10 vCPU, 16 GB RAM, 500GB HDD 10.103.7.132
  NSX Edge 1 1 vCPU, 0.5GB RAM [compact] 10.103.7.138
  NSX Edge 2 1 vCPU, 0.5GB RAM [compact] 10.103.7.139
 Virtual ESXi 3 10 vCPU, 16 GB RAM, 100GB HDD 10.103.7.133
  Server 1 1 vCPU, 1GB RAM [Alpine Linux]  
  Server 2 1 vCPU, 1GB RAM [Alpine Linux]  
  Server 3 1 vCPU, 1GB RAM [Alpine Linux]  
  Server 4 1 vCPU, 1GB RAM [Alpine Linux]  
 vCenter 2 vCPU, 14GB RAM, 500GB HDD [tiny] 10.103.7.130
 DNS Server / Jumphost 2 vCPU, 8GB RAM [Windows Server 2022] 10.103.7.200

Procedures

  • NSX Quick Start Guide (VMware NSX 4.0)2
  • 自宅ラボ NSX-T 3.1 の構築。Step-01: はじめに (in Japanese)3

Requirement

  • DNS server
  • NTP server (private or public)

Prepare on the physical ESXi Host

  • Create two Port-groups for nested NSX networking and management.
    • Port binding must be ephemeral for vCenter deployment.
    • Security settings (Promiscuous mode, MAC address changes, Forged transmits) should be Allowed. [see (in Japanese)]4
    • You can also have only one port-group that allows all VLAN for this lab. I used separate port-groups because of the setting on the physical router.
    • All VMs go into management port-group (VLAN 1037 in my case); Virtual ESXi hosts also go into nested NSX networking port-group (VLAN 0-4094 in my case).
  • Set MTU to 1600 for GENEVE packets (it cannot be fragmented, 5).

Deploy nested ESXi 7 VM

  • Create 3 virtual ESXi hosts in this case.
  • Allow CPU Virtualization on virtual ESXi hosts (check 'Expose hardware assisted virtualization to the guest OS') or This host does not support Intel VT-x error pops up later.
  • Assign IP Address, set DNS server, DNS suffix.

Deploy vCenter Server 7.0.3

  • Install VSCA Appliance

Notes in Stage 1

  • If you are installing vCenter Server 8.0 on VM, you need to edit OVF file beforehand and deploy. 6
  • You need to have a valid DNS server to deploy NSX components and configure clusters.
    • If you want to deploy only ESXis and vCenter, then it is okay to proceed with no DNS server: Set FQDN as the same as IPv4 address of vCenter itself [see (in Japanese)]7. DNS address can be global one, such as 8.8.8.8. Additional setting is required (see Notes in Stage 2).
  • Deployment will be fail if you have slow network connection. Use jumphost and try to install locally in your network.

Notes in Stage 2

  • If you have decided not to have a DNS server, for some reason, then:
    • Allow SSH, and then change dns configurations on vCenter Server via SSH before proceeding. [see (in Japanese)]7
      • /etc/hosts
        • make localhost and localhost.localdomain point your IPv4 address.
      • /etc/dnsmasq.conf
        • comment out no-hosts.
        • Add no-resolv and bogus-priv.
      • systemctl restart dnsmasq.service
      • nslookup localhost and nslookup <IPv4 Address> to see if dns resolution is successful.

Register and Setup Virtual ESXi in Nested vSphere

  • Set NTP server and sync.
  • Create DataStores.
  • [Follow Step 2]2: Create distributed switch, change the MTU value, and create a port group. Change VMs’ network config accordingly.

Deploy NSX 4 Manager

  • I used vSphere 7.0.3 to deploy NSX Manager 4 as an OVF template.
  • I set RAM reservation down to 0MB after OVF deployment.
  • NSX management appliance takes a while to be fully set and booted up. In the meantime, you might see Some appliance components are not functioning properly. Component health: MANAGER:UNKNOWN, SEARCH:UNKNOWN, NODE_MGMT:UP, UI:UP. Error code: 101 error.
  • log in with “admin” and a password.

Deploy NSX Controller

  • I initially deployed a nested ESXi 7 node with 12 vCPUs, 32 GB RAM and 500GB Datastore, since small controller node required 4 vCPU, 16 GB RAM and 300GB Storage at minimum.
    • This setting was only temporal; I reduced the resources down to 2 vCPU and 8 GB RAM after I deployed NSX controller and changed VM hardware configuration successfully.
  • If you have not set up a valid DNS server, then you will likely encounter error even though installation is successful on the VM.
  • This was very tricky, since we had vSphere 6.5 U3u as a main vSphere that was holding physical hosts. I was not able to use it as a Compute Manager on NSX 4.0 (because of incompatibility), therefore I had to make a nested ESXi 7 hosts dedicated to deploying NSX controller with my nested vSphere 7.0.3 (see NSX lab configuration image above).
  • I decided not to deploy the controller because of several compatibility issues in my lab.

vSwitch and DSwitch Configuration

Setup a Host Transport Node

  • ESXi for compute nodes (virtual ESXi 03 in my case) only. Double check CPU and Memory requirements (Configuration process may fail due to insufficient resource).

Deploy NSX Edge Nodes & configure Edge Cluster

  • Additional Setting on virtual ESXi host (dedicated to edge nodes) is required: featMask.vm.cpuid.pdpe1gb = Val:1 8. Also disable EVC mode in the virtual ESXi host cluster, if it is enabled.

Create an overlay vlan segment

Create Tier-0 Gateway

Create Tier-1 Gateway

Deploy VMs for testing & Test East-West Connection

  • Deploy VMs and install OS (Alpine linux in my case, 9).
  • Choose the overlay segment (e.g. LS1.1, LS1.2, …) for a network adapter.
  • At this point, you CANNOT ping to external router (10.103.5.1) unless you have configured routes to VMs on the router.
east-west-ping test

Create SNAT Rule & Test North-South Connection

  • Configure SNAT rule on T-0 gateway for North-South connection.
North-South Ping Test
  1. https://docs.vmware.com/en/VMware-NSX/index.html 

  2. https://docs.vmware.com/en/VMware-NSX/4.0/nsx_40_quick_start.pdf  2

  3. https://vm.gowatana.jp/entry/2021/01/25/010514 

  4. https://changineer.info/vmware/hypervisor/vmware_promiscuous_mode.html 

  5. https://docs.vmware.com/en/VMware-Validated-Design/5.0.1/com.vmware.vvd.sddc-nsxt-design.doc/GUID-3FF2471C-665B-4E84-8DE4-ED3F35A58DE8.html 

  6. https://williamlam.com/2022/10/how-to-deploy-the-vcenter-server-appliance-vcsa-8-0-to-vmware-fusion-or-workstation.html 

  7. https://changineer.info/vmware/hypervisor/vmware_vcsa_gui02.html  2

  8. https://my-sddc.net/upgrade-failed-nsx-t-edge-nodes-1g-hugepage-support-required/ 

  9. https://www.virten.net/2016/05/deploy-vmware-nsx-in-homelabs-with-limited-resources/ 

Updated:

Leave a comment