Understanding basics of VMware ESXi networking

In this blog post we will discuss some fundamental ESXi networking concepts and observe VM to VM communication through packet captures on ESXi.

Image courtesy: VMware Virtual Networking Concepts Information Guide

Virtual Switch connects virtual NICs to Physical NICs. A virtual switch (vswitch) has different ports or port-groups that can be used for VM networking and management services.

VM port group on a vSwitch connects VMs to the physical network or to other VMs. VMkernel port connects VMkernel services (vMotion, iSCSI, NFS, Fault Tolerance) to the physical network and service console port connects ESX to network or remote management services, such as the vCenter Server.

Reference: https://docs.vmware.com/en/VMware-Smart-Assurance/10.1.0/ip-manager-delopment-guide-101/GUID-C7E7752E-D122-44EA-83DE-33F690B22313.html

Here we can see two vswitches in ESXi host. vSwitch0 is the standard virtual switch and dvs-SA-Datacenter is distributed virtual switch.

[root@sa-esxi-04:~] esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 2560 3 128 1500 vmnic4

PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 0 vmnic4

DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
dvs-SA-Datacenter 2560 13 512 1500 vmnic3,vmnic1,vmnic2,vmnic0

DVPort ID In Use Client
42 1 vmnic0
43 1 vmnic1
44 1 vmnic2
45 1 vmnic3
79 1 vmk1
73 1 vmk0
83 1 vmk2
202 1 Ubuntu-01a.eth0

[Notice the DVPort ID. This is a distributed switch.]

You can view configured virtual switches from ESXi UI (in this blog post I have not shown examples from vcenter.)

While configuring virtual switch you can define uplink port for that switch. Notice I am only getting vmnic5 as option, that is because vmnic0 vmnic4 are already in use by the two vswitches already configured. A vswitch can have more than one uplink but the same uplink can not be shared by other vswitch. You can create multiple port-groups within a vswitch.

Vmnic is the physical adaptor on ESXi host. It corresponds to the actual physical NIC on your physical server. The following VMware KB article shows how vmnics are assigned,
https://kb.vmware.com/s/article/2091560

If a system has:
Two onboard network ports
One single-port NIC in slot #3
One dual-port NIC in slot#5
Then devices names should be assigned as:

Physical Port	Device Alias
Onboard port 1	vmnic0
Onboard port 2	vmnic1
Slot #3 port 1	vmnic2
Slot #5 port 1	vmnic3
Slot #5 port 2	vmnic4

This means a server having 6 physical ports will have 6 vmnics. You can check vmnics available on ESXi from CLI and vcenter or ESXi UI.

[root@localhost:~] net-stats -l
PortNum          Type SubType SwitchName       MACAddress         ClientName
33554434            4       0 vSwitch0         08:f1:ea:76:92:aa vmnic0
33554436            4       0 vSwitch0         08:f1:ea:76:92:bb vmnic1
33554438            3       0 vSwitch0         08:f1:ea:76:92:aa vmk0

Notice by default vmnic0 and vmk0 have same mac address.

You can also get this information using,
[root@sa-esxi-04:~] excli network nic list

To get detailed information about vmnics run the following command from ESXi CLI,
[root@sa-esxi-04:~] esxcfg-info -n

Here is a snippet of output (this output is from nested ESXi)-

This output is from ESXi UI,

Vmk or VMKernel NIC is a virtual device in the VMkernel. It is the software layer that manages most of the physical resources on the ESX Server host. The vmknic is used by the TCP/IP stack that services VMotion, NFS and software iSCSI clients that run at the VMkernel level, and remote console traffic.

So VMKernel interface provides network connectivity to hosts and handles system traffic of vSphere vMotion, IP storage, Fault Tolerance, vSAN, etc. In simplest terms to connect to ESXi we need a vmkernel interface since IP address is assigned to vmknic, not to vmnic.

To check vmk interfaces you can use the following commands,
[root@sa-esxi-04:~] esxcli network ip interface list
[root@localhost:~] esxcli network ip interface ipv4 get

[root@sa-esxi-04:~] esxcli network ip interface ipv4 get
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
—– ————- ————- ————— ———— ———— ——–
vmk0 172.20.10.54 255.255.255.0 172.20.10.255 STATIC 172.20.10.10 false
vmk1 172.20.11.203 255.255.255.0 172.20.11.255 DHCP 172.20.11.10 false
vmk2 172.20.12.203 255.255.255.0 172.20.12.255 DHCP 172.20.12.1 false
vmk10 172.20.11.152 255.255.255.0 172.20.11.255 STATIC 172.20.11.10 false
vmk50 169.254.1.1 255.255.0.0 169.254.255.255 STATIC 0.0.0.0 false

Vmk is on ESXi, not on any VM running on ESXi.

Here is how to check this from ESXi UI,

Virtual Ports
Virtual ports are ports on vswitch (virtual switch) which provide logical connection points among virtual devices and between virtual and physical devices. These are like ports on a physical switch where we connect PC/Server etc. Each virtual switch can have up to 1016 virtual ports, with a limit of 1096 ports on all virtual switches on a host.

Below is log snippet. Here Ubuntu-01a.eth0 is connected to Port number 50331661 on virtual switch.

[root@sa-esxi-04:~] net-stats -l
PortNum Type SubType SwitchName MACAddress ClientName
50331661 5 9 DvsPortset-0 00:50:56:88:1b:93 Ubuntu-01a.eth0
50331662 5 9 DvsPortset-0 00:50:56:88:31:d8 Ubuntu-02a.eth0

To check vmnic to port mapping type ‘esxtop’ and press ‘n’.

Here is log snippet

Uplinks ports

Uplink ports are ports associated with physical adapters, providing a connection between a virtual network and a physical network. Just like regular physical switch uplink ports are required when traffic has to exit to physical network.

[root@sa-esxi-04:~] esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 2560 3 128 1500 vmnic4

Here vmnic4 is uplink for vSwitch0.

You can view this from ESXi UI.

Port-groups are user defined objects that contain configuration information to provide network access for virtual Ethernet adapters.

You can define the following parameters when configuring port-group from ESXi UI.

The port-group appears as network for VM to connect to. Different port-groups can have same VLAN.

Let’s use this knowledge and observe packet flow between two VMs when they are on same ESXi host and in different ESXi host. These VMs are same network.

When both VMs are on same ESXi host

From net-stats we can see the PortNum for both the VMs.

50331661 5 9 DvsPortset-0 00:50:56:88:1b:93 Ubuntu-01a.eth0
50331662 5 9 DvsPortset-0 00:50:56:88:31:d8 Ubuntu-02a.eth0

Let’s use pktcap-uw to capture traffic between these VMs.

[root@sa-esxi-04:~] pktcap-uw –switchport 50331661 –ng -o – | tcpdump-uw -enr –
The switch port id is 0x0300000d.
The output file format is pcapng.
pktcap: The output file is -.
pktcap: No server port specifed, select 47080 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 47080.
pktcap: Accept…
pktcap: Vsock connection from port 1028 cid 2.
reading from file -, link-type EN10MB (Ethernet)
15:45:12.355816 00:50:56:88:1b:93 > 00:50:56:88:31:d8, ethertype IPv4 (0x0800), length 98: 172.16.40.11 > 172.16.40.12: ICMP echo reply, id 16300, seq 1, length 64
15:45:13.357634 00:50:56:88:1b:93 > 00:50:56:88:31:d8, ethertype IPv4 (0x0800), length 98: 172.16.40.11 > 172.16.40.12: ICMP echo reply, id 16300, seq 2, length 64
15:45:14.359385 00:50:56:88:1b:93 > 00:50:56:88:31:d8, ethertype IPv4 (0x0800), length 98: 172.16.40.11 > 172.16.40.12: ICMP echo reply, id 16300, seq 3, length 64
15:45:15.358435 00:50:56:88:1b:93 > 00:50:56:88:31:d8, ethertype IPv4 (0x0800), length 98: 172.16.40.11 > 172.16.40.12: ICMP echo reply, id 16300, seq 4, length 64
15:45:16.360636 00:50:56:88:1b:93 > 00:50:56:88:31:d8, ethertype IPv4 (0x0800), length 98: 172.16.40.11 > 172.16.40.12: ICMP echo reply, id 16300, seq 5, length 64
15:45:17.361858 00:50:56:88:1b:93 > 00:50:56:88:31:d8, ethertype IPv4 (0x0800), length 98: 172.16.40.11 > 172.16.40.12: ICMP echo reply, id 16300, seq 6, length 64
15:45:17.371192 00:50:56:88:1b:93 > 00:50:56:88:31:d8, ethertype ARP (0x0806), length 60: Request who-has 172.16.40.12 tell 172.16.40.11, length 46
15:45:18.361299 00:50:56:88:1b:93 > 00:50:56:88:31:d8, ethertype IPv4 (0x0800), length 98: 172.16.40.11 > 172.16.40.12: ICMP echo reply, id 16300, seq 7, length 64

Since these are in same esxi host traffic does not go to uplink port.

When VMs are on different ESXi hosts

Now let’s move Ubuntu-02a to esxi-05.

[root@sa-esxi-04:~] net-stats -l | grep -i ubuntu
50331661 5 9 DvsPortset-0 00:50:56:88:1b:93 Ubuntu-01a.eth0

[root@sa-esxi-05:~] net-stats -l | grep -i ubuntu
50331662 5 9 DvsPortset-0 00:50:56:88:31:d8 Ubuntu-02a.eth0

[root@sa-esxi-05:~] pktcap-uw –switchport 50331662 –ng -o – | tcpdump-uw -enr –
The switch port id is 0x0300000e.
The output file format is pcapng.
pktcap: The output file is -.
pktcap: No server port specifed, select 61136 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 61136.
pktcap: Accept…
pktcap: Vsock connection from port 1027 cid 2.
reading from file -, link-type EN10MB (Ethernet)
15:54:33.653655 00:50:56:88:31:d8 > 00:50:56:88:1b:93, ethertype IPv4 (0x0800), length 98: 172.16.40.12 > 172.16.40.11: ICMP echo request, id 16300, seq 635, length 64
15:54:34.655587 00:50:56:88:31:d8 > 00:50:56:88:1b:93, ethertype IPv4 (0x0800), length 98: 172.16.40.12 > 172.16.40.11: ICMP echo request, id 16300, seq 636, length 64

Since this traffic exits ESXi it will use uplink port. Let’s find out uplink port for this VM.

Type ‘esxtop’ and then press ‘n’.

50331661 2101024:Ubuntu-01a.eth0 vmnic2 DvsPortset-0 1.30 0.00 90.00 5.95 0.00 67.00 0.00 0.00

Here we can see bidirectional traffic on vmnic2.

[root@sa-esxi-04:~] pktcap-uw –capture UplinkSndKernel,UplinkRcvKernel –uplink vmnic2 –ng -o – | tcpdump-uw -enr –
The session capture point is UplinkSndKernel,UplinkRcvKernel.
The name of the uplink is vmnic2.

16:08:28.568413 00:50:56:88:31:d8 > 00:50:56:88:1b:93, ethertype IPv4 (0x0800), length 98: 172.16.40.12 > 172.16.40.11: ICMP echo request, id 16303, seq 216, length 64
16:08:28.568647 00:50:56:88:1b:93 > 00:50:56:88:31:d8, ethertype IPv4 (0x0800), length 98: 172.16.40.11 > 172.16.40.12: ICMP echo reply, id 16303, seq 216, length 64
16:08:28.632096 00:50:56:88:1b:93 > 00:50:56:88:31:d8, ethertype IPv4 (0x0800), length 98: 172.16.40.11 > 172.16.40.12: ICMP echo request, id 5292, seq 198, length 64
16:08:28.633075 00:50:56:88:31:d8 > 00:50:56:88:1b:93, ethertype IPv4 (0x0800), length 98: 172.16.40.12 > 172.16.40.11: ICMP echo reply, id 5292, seq 198, length 64

In this blog post our VMs were in same port-group, in next post we will discuss traffic flow between port-groups and VMware vSphere Distributed Switch (vDS).