If you are new to VMware ESXi or NSX environment and find yourself in a situation where you have to troubleshoot connectivity issues, having list of commands that can provide information about switchports/uplinks and help you get packet captures would save a lot of time. In this blog post we will talk about 10 commands that are helpful while troubleshooting connectivity in ESXi environment.
1. Esxtop
This is a version of linux ‘top’ command. If we press ‘n’ after running the command it will show information about VMs and their switchports and uplink ports.
Here is an output from my environment –
Remember to press ‘n’ key to get port related stats.
[ root@sa-esxi:~] esxtop
PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PSZTX PKTRX/s MbRX/s PSZRX %DRPTX %DRPRX
67108870 vdr-vdrPort vmnic4 DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
67108871 2101191:Ubuntu-01a.eth0 vmnic4 DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00We can easily figure out a lot of parameters for these VMs.
2. Packet capture tool pktcap-uw
The ability to take packet capture is of immense help during troubleshooting connectivity issues. pktcap-uw allows us to capture packets from both switchports and uplink ports.
- For switchports
To save pcap to file –
pktcap-uw –switchport switchport-id –dir 2 -o /path/test.pcap
To view live packets –
pktcap-uw –-switchport switchport-id –dir 2 –ng -o – | tcpdump-uw -enr –
- For uplink ports
To save pcap to file –
pktcap-uw –uplink vmnicX –dir 2 -o /path/test.pcap
To view live packets –
pktcap-uw –uplink vmnicX –dir 2 -ng -o – | tcpdump-uw -enr –
You can also run the capture simultaneously on two different ports. Example below –
[root@esxi:~] pktcap-uw --switchport 33554440 --dir 2 -o /tmp/test1.pcap && pktcap-uw --switchport 83886031 --dir 2 -o /tmp/test2.pcap && pktcap-uw --uplink vmnic1 --dir 2 -o /tmp/test3.pcapYou can also refer to this blog post for more information about packet flow.
3. Check vswitches and their ports – esxcfg-vswitch -l
The following command lists all vswitches, vmk and vmnics associated with that switch.
[root@sa-esxi:~] esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 2560 1 128 1500
PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 0
DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
dvs-SA-Datacenter 2560 12 512 1500 vmnic3,vmnic1,vmnic2,vmnic0
DVPort ID In Use Client
42 1 vmnic0
43 1 vmnic1
44 1 vmnic2
45 1 vmnic3
79 1 vmk1
73 1 vmk0
83 1 vmk2
Switch Name Num Ports Used Ports Uplinks
PROD-Overlay-NVDS 2560 10 vmnic4
Switch Name Num Ports Used Ports Uplinks
PROD-VLAN-NVDS 2560 4 vmnic54. Net-stats -l
This command will list VMs and the mac address of connected port.
[ root@sa-esxi:~] net-stats -l
PortNum Type SubType SwitchName MACAddress ClientName
50331650 4 0 DvsPortset-0 00:50:56:01:53:af vmnic0
50331652 4 0 DvsPortset-0 00:50:56:01:53:b1 vmnic2
50331654 4 0 DvsPortset-0 00:50:56:01:53:b0 vmnic1
50331656 4 0 DvsPortset-0 00:50:56:01:53:b2 vmnic3
50331658 3 0 DvsPortset-0 00:50:56:01:ca:c2 vmk0
50331659 3 0 DvsPortset-0 00:50:56:63:29:a3 vmk1
50331660 3 0 DvsPortset-0 00:50:56:64:d6:41 vmk2
67108866 4 0 DvsPortset-1 00:50:56:01:53:b3 vmnic4
67108868 3 0 DvsPortset-1 00:50:56:62:3a:e9 vmk10
67108869 3 0 DvsPortset-1 00:50:56:67:da:ba vmk50
67108871 5 9 DvsPortset-1 00:50:56:88:1b:93 Ubuntu-01a.eth0
67108872 5 9 DvsPortset-1 00:50:56:ae:1d:9d T1-Web-01.eth0
[ root@sa-esxi:~]You can use net-stats -V to get detailed information about VMs ports.
5. Net-dvs -l
This command lists information about dvs, including state of ports and parameters like MTU.
6. List of vmnics
root@sa-esxi:~] esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ -------- ------------ ----------- ----- ------ ----------------- ---- -----------------------------------------------
vmnic0 0000:03:00.0 nvmxnet3 Up Up 10000 Full 00:50:56:01:53:af 1500 VMware Inc. vmxnet3 Virtual Ethernet Controller
7. Stats from vmnics and switchports use the following commands
This command is very useful for getting packet stats for any switch port of vmmic.
[root@sa-esxi:~] esxcli network nic stats get -n vmnic1
NIC statistics for vmnic1
Packets received: 4678
Packets sent: 1417
Bytes received: 333248
Bytes sent: 99404
Receive packets dropped: 0
Transmit packets dropped: 0
Multicast packets received: 2508
Broadcast packets received: 2016
Multicast packets sent: 1253
Broadcast packets sent: 64
Total receive errors: 0
Receive length errors: 0
Receive over errors: 0
Receive CRC errors: 0
Receive frame errors: 0
Receive FIFO errors: 0
Receive missed errors: 0
Total transmit errors: 0
Transmit aborted errors: 0
Transmit carrier errors: 0
Transmit FIFO errors: 0
Transmit heartbeat errors: 0
Transmit window errors: 0
[root@sa-esxi:~] esxcli network port stats get -p 50331650
Packet statistics for port 50331650
Packets received: 514460382
Packets sent: 67898125
Bytes received: 44353505590
Bytes sent: 229321118
Broadcast packets received: 440599
Broadcast packets sent: 6472
Multicast packets received: 134770
Multicast packets sent: 1617
Unicast packets received: 513885013
Unicast packets sent: 67890036
Receive packets dropped: 0
Transmit packets dropped: 0
8. Dvfilter and ESXi firewall
This command will show esxi firewall rules.
[root@sa-esxi:~] esxcli network firewall ruleset list
This is an interesting command. ‘nic-2101191-eth0-vmware-sfw.2’ is the name of dvfilter and we can capture packet before and after this filter.
[root@sa-esxi:~] summarize-dvfilter
<output snippet>
world 2101191 vmm0:Ubuntu-01a vcUuid:'50 08 e5 70 94 26 92 a6-b6 55 f4 d7 64 2f 89 ac'
port 67108871 Ubuntu-01a.eth0
vNic slot 2
name: nic-2101191-eth0-vmware-sfw.2
agentName: vmware-sfw
state: IOChain Attached
vmState: Attached
failurePolicy: failClosed
serviceVMID: 1
filter source: Dynamic Filter Creation
Here is command syntax –
pktcap-uw --dvFilter dvfilter_name --capture PreDVFilter|PostDVFilter [filter_options] [--outfile pcap_file_path [--ng]] [--count number_of_packets]9. Vmkping and NetCat
Netcat is a linux tool that can help confirm if a certain port is open. ESXi does not support telnet command.
[root@sa-esxi:~] nc -z sa-esxi-01.vsphere.local 80
Connection to sa-esxi-01.vsphere.local 80 port [tcp/http] succeeded!VMkping helps in troubleshooting VMkernel connectivity. For example here it pinged sa-esxi-01.vsphere.local with size 1400
[root@sa-esxi:~] vmkping -I vmk0 sa-esxi-01.vclass.local -d -s 1400
PING sa-esxi-01.vclass.local (172.20.10.51): 1400 data bytes
1408 bytes from 172.20.10.51: icmp_seq=0 ttl=64 time=1.202 ms
1408 bytes from 172.20.10.51: icmp_seq=1 ttl=64 time=1.406 ms
1408 bytes from 172.20.10.51: icmp_seq=2 ttl=64 time=1.179 ms
--- sa-esxi-01.vclass.local ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.179/1.262/1.406 ms10. Get list of VMs and associated details
[root@sa-esxi:~] esxcli network vm list
World ID Name Num Ports Networks
-------- ----------- --------- ------------------------------------
2101191 Ubuntu-01a 1 5484c91b-9135-4086-ab54-3c3158e68a42
[root@sa-esxi:~] esxcli network vm port list -w 2101191
Port ID: 67108871
vSwitch: DvsPortset-1
Portgroup: 5484c91b-9135-4086-ab54-3c3158e68a42
DVPort ID:
MAC Address: 00:50:56:88:1b:93
IP Address: 0.0.0.0
Team Uplink: vmnic4
Uplink Port ID: 67108866
Active Filters: vmware-sfwI hope this post was helpful.