{"id":1300,"date":"2025-03-24T10:22:20","date_gmt":"2025-03-24T08:22:20","guid":{"rendered":"https:\/\/dety.net.ua\/?p=1300"},"modified":"2025-03-24T10:22:20","modified_gmt":"2025-03-24T08:22:20","slug":"large-packet-loss-in-the-guest-os-using-vmxnet3-in-esxi","status":"publish","type":"post","link":"https:\/\/dety.net.ua\/?p=1300","title":{"rendered":"Large packet loss in the guest OS using VMXNET3 in ESXi"},"content":{"rendered":"\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Source: <a href=\"https:\/\/knowledge.broadcom.com\/external\/article?legacyId=2039495\">https:\/\/knowledge.broadcom.com\/external\/article?legacyId=2039495<\/a><\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">DETERMINE IF VIRTUAL NIC BUFFERS COULD BE INCREASED:<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>1) For a given virtual machine, determine the ESXi host on which it is running, and log into the ESXi host via SSH or a KVM with root privileges.<\/li>\n\n\n\n<li>2) Assuming an example of &#8220;VMNAME&#8221;, run the following command:\u00a0 &#8220;net-stats -l | grep VMNAME&#8221;<\/li>\n\n\n\n<li>3) The output might look something like the following:<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>100663349 <\/strong>2189023:VMNAME.eth0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;all(1) DvsPortset-2 &nbsp; &nbsp; &nbsp; 3821.41 &nbsp;349.01 11970.00 &nbsp; &nbsp;5663.45 &nbsp; &nbsp;5.34 &nbsp;123.00 &nbsp; 0.00 &nbsp; 0.00<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>4) Run a command such as the following to check the 1st ring buffer stats:<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">vsish -e get \/net\/portsets\/DvsPortset-2\/ports\/100663349\/vmxnet3\/rxSummary | grep &#8220;1st ring&#8221;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>5) The output might look something like the following:<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp; &nbsp;1st ring size:512<br>&nbsp; &nbsp;# of times the 1st ring is full:276<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>6) Another command that will show insight would be:<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">vsish -e get \/net\/portsets\/DvsPortset-2\/ports\/100663349\/vmxnet3\/rxSummary | grep &#8220;running out of buffers&#8221;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>7) The output might look something like the following:<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp; &nbsp; running out of buffers:3198<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>8) If the &#8220;# of times the 1st ring is full&#8221; and\/or &#8220;running out of buffers&#8221; is zero, then changing the ring buffer settings will have no effect on your symptoms.\u00a0\u00a0<\/li>\n\n\n\n<li>9) However, if they are > zero, then we would suggest the steps below.<\/li>\n\n\n\n<li>10) Please note that the virtual NIC counters as seen by the vsish command are reset, and the Port Number changes (100663349 in the above example) when the VM is vMotioned to another ESXi host.\u00a0<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">To resolve the&nbsp;issue of significant packet loss during periods of very high traffic bursts with the VMXNET3 vNIC:<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Ensure that there is no traffic filtering occurring (for example, with a mail filter).<\/li>\n\n\n\n<li>Check the driver\/firmware of the ESXi&#8217;s physical NICs and update if needed. To check the driver\/firmware, see\u00a0<a href=\"https:\/\/knowledge.broadcom.com\/external\/article?articleNumber=323110\" target=\"_blank\" rel=\"noreferrer noopener\">Determining Network\/Storage firmware and driver version in ESXi<\/a>.<\/li>\n\n\n\n<li>After eliminating the above\u00a0possibilities, slowly increase the number of buffers in the guest operating system.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">To reduce burst traffic drops in Windows Buffer Settings:<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Click <strong>Start<\/strong> > <strong>Control Panel<\/strong> > <strong>Device Manager<\/strong>.<\/li>\n\n\n\n<li>Right-click <strong>vmxnet3<\/strong> and click <strong>Properties<\/strong>.<\/li>\n\n\n\n<li>Click the <strong>Advanced<\/strong> tab.<\/li>\n\n\n\n<li>Click <strong>Small Rx Buffers<\/strong> and increase the value\u00a0(The maximum value is 8192).<\/li>\n\n\n\n<li>Click <strong>Rx Ring #1 Size<\/strong> and increase the value\u00a0(The maximum value is 4096).<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Notes<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>No reboot is required for these changes to take effect. However, any application sensitive to TCP session disruption can fail and have to be restarted. This applies to RDP, so it is better to do this work in a console window.<\/li>\n\n\n\n<li>This issue is seen in the Windows guest operating system with a VMXNET3 vNIC.<\/li>\n\n\n\n<li>It is important to increase the value of Small Rx Buffers and Rx Ring #1 gradually to avoid drastically increasing the memory overhead on the host and possibly causing performance issues if resources are close to capacity.<\/li>\n\n\n\n<li>If this issue occurs on only 2-3 virtual machines, set the value of Small Rx Buffers and Rx Ring #1 to the maximum value. Monitor virtual machine performance to see if this resolves the issue.<\/li>\n\n\n\n<li>The Small Rx Buffers and Rx Ring #1 variables affect non-jumbo frame traffic only on the adapter.<\/li>\n\n\n\n<li>Windows Servers must be on NDIS 6.3 or higher\u00a0to change RX Ring #2 (See Internal Notes)<br><a href=\"https:\/\/docs.microsoft.com\/en-us\/windows-hardware\/drivers\/network\/overview-of-ndis-versions\" target=\"_blank\" rel=\"noreferrer noopener\">Overview of NDIS versions &#8211; Windows driver<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Source: https:\/\/knowledge.broadcom.com\/external\/article?legacyId=2039495 DETERMINE IF VIRTUAL NIC BUFFERS COULD BE INCREASED: 100663349 2189023:VMNAME.eth0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;all(1) DvsPortset-2 &nbsp; &nbsp; &nbsp; 3821.41 &nbsp;349.01 11970.00 &nbsp; &nbsp;5663.45 &nbsp; &nbsp;5.34 &nbsp;123.00 &nbsp; 0.00 &nbsp; 0.00 vsish -e get \/net\/portsets\/DvsPortset-2\/ports\/100663349\/vmxnet3\/rxSummary | grep &#8220;1st ring&#8221; &nbsp; &nbsp;1st ring size:512&nbsp; &nbsp;# of times the 1st ring is full:276 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13,11],"tags":[],"class_list":["post-1300","post","type-post","status-publish","format-standard","hentry","category-novosti","category-vmware"],"_links":{"self":[{"href":"https:\/\/dety.net.ua\/index.php?rest_route=\/wp\/v2\/posts\/1300","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dety.net.ua\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dety.net.ua\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dety.net.ua\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dety.net.ua\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1300"}],"version-history":[{"count":1,"href":"https:\/\/dety.net.ua\/index.php?rest_route=\/wp\/v2\/posts\/1300\/revisions"}],"predecessor-version":[{"id":1301,"href":"https:\/\/dety.net.ua\/index.php?rest_route=\/wp\/v2\/posts\/1300\/revisions\/1301"}],"wp:attachment":[{"href":"https:\/\/dety.net.ua\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1300"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dety.net.ua\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1300"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dety.net.ua\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1300"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}