I’m going to start this post off by saying that I feel incredibly spoiled to even be bringing up a problem like this given that I started out decades ago on a 33.6k dialup modem and today’s cable and DSL connections still have relatively low upstream bandwidth. But it’s still a technically interesting issue that I think is worth bringing up to a larger audience. The gist of what’s going on is: I’m lucky enough to live in an area where symmetrical gigabit fiber internet is available, affordable, and maintained by an awesome ISP. The problem is that although my downstream speed is fine, I’m not seeing my full upstream bandwidth during speed tests when I’m using Windows 10. I’m “only” getting around 300 to 400 megabits instead of the expected 940-ish I should see with a wired Ethernet test.
I already know what everyone’s going to say, so I’ll say it all for you. My router sucks and isn’t capable of handling gigabit traffic. I need to update my drivers. My network card sucks. I’ve clearly installed some kind of network bloatware on my machine that’s slowing down my traffic. My firewall is slowing down my traffic. My home network is garbage. I need to do a clean install of Windows. It’s Speedtest.net’s fault. My ISP’s backbone is saturated.
I’m sure the suggestions listed above are true in some cases, but my situation is more complicated than that. Here’s the kicker: my connection tests perfectly in Linux, macOS, and Windows 11. Below is a test taken from the same computer as the one above, but booted 15 minutes later into Linux instead of Windows 10. I ran the speed test inside of Chrome.
This computer isn’t officially supported by Windows 11, but I bypassed the requirements check using Rufus and installed it on a spare SSD anyway. Here’s what my speed test looks like when booted into a fresh, fully-updated install of Windows 11.
At this point I can just feel my readers ready to pounce on this. Obviously, I’m an idiot and I’ve installed some bloatware onto my Windows 10 partition that is interfering with my upload speeds. Right? Let’s find out. I installed a fresh copy of Windows 10 to my spare SSD and fully updated it. Here’s what my speed test looks like from a fresh copy of Windows 10.
So this only seems to affect Windows 10. But I haven’t ruled out driver problems. What if my network card has a driver bug that was fixed in Windows 11?
My answer for that is: I have messed with various driver versions and it didn’t change anything. I’ve also tested this process on multiple computers that have completely different Ethernet chipsets, both Intel and Realtek, and they’re all affected by this problem when they are running Windows 10. They always work fine in Linux or Windows 11.
Here’s the really crazy part: the computer that performed all the speed tests above has VMware Workstation. So I booted into my older existing Windows 10 install, spun up a Windows 11 VM inside of it, and ran a speed test inside the VM.
Wait, what? A Windows 11 guest VM running inside of a Windows 10 host has a better upload speed than the host by itself? I feel like that pretty much rules out driver issues. By the way, a Windows 10 guest VM running inside a Windows 10 host or a Linux host also has the upload speed problem.
I’ve also used iperf3 in Windows 10 to verify that I can get full download and upload speeds to another computer on my LAN, so this issue seems to be isolated to internet traffic only. That’s more evidence in my mind that it’s probably not a driver bug.
Another thing I’ve tried is to take my router out of the picture and connect my laptop directly to my fiber ONT. My ISP conveniently allows for two simultaneous DHCP leases, so I don’t have to fiddle with release/renew and spoofing MAC addresses in order to test that out. Anyway, even without the router, the slow upload problem still occurs in Windows 10. This isn’t too surprising; I have already demonstrated that the router is perfectly capable of saturating my link in both directions with other operating systems.
Here’s a summary of tests I’ve performed:
- A Mac works fine with full downstream/upstream speeds.
- Multiple PCs booted into Windows 10 have good downstream speeds, but slow upstream.
- The same computers booted into Linux have full downstream/upstream speeds.
- The same computers booted into Windows 11 have full downstream/upstream speeds.
- Windows 11 VM running on a Windows 10 computer has full downstream/upstream speeds.
- Linux VM running on a Windows 10 computer has full downstream/upstream speeds.
- Windows 10 VM running a Windows 10 computer has good downstream, slow upstream.
- Windows 10 VM running on a Linux computer has good downstream, slow upstream.
- Windows 11 VM running on a Linux computer has full downstream/upstream speeds.
The common link here is that when the operating system running the actual test is Windows 10, I get slow upstream speeds. I tested this across 5 computers. I tested with various older Windows 10 versions as well as the current 21H2 version.
I posted about this on my ISP’s subreddit to see if anyone else is running into this problem. I mean, I can’t be the only person in the world doing gigabit speed tests on Windows 10, right? The reaction I got was mostly “this is something wrong on your end”. A few people posted speed test results showing that they are able to get their full bandwidth in both directions with Windows 10, so it’s definitely not a universal problem that everyone experiences. The other people do have lower pings than me, so they’re closer to the speed test server and not following the exact same path that my connection is using. So it seems like it’s some kind of interaction between Windows 10 and my connection.
I’ve been doing some research on this, and it sounds like Dropbox might have actually already figured out this issue a year ago and solved it in coordination with Microsoft. If this is indeed the same thing I’m experiencing, it’s related to out-of-order packet delivery and how it interacts with TCP congestion control mechanisms. They fixed it locally on their own servers, but it also led Microsoft to fix the issue in their TCP stack. In fact, Dropbox may be largely responsible for me being able to get full speeds in Windows 11 today. Perhaps something along the way between my computer and the test server is causing packets to get reordered. Windows 10 doesn’t handle that situation as well as the other operating systems.
The Dropbox blog post above says that the fix was first introduced with Windows 10 build 21332, which came out in March of 2021. Wouldn’t you think that would mean that Windows 10 should already have this fix? It’s looking more and more like it doesn’t. I haven’t been following Microsoft’s build numbers closely, but it seems like Windows 10 is still in the 19000s. I suspect that build numbers in the 20000s are part of what eventually became Windows 11 instead.
There are a few threads about this kind of problem on Microsoft’s forums. It seems like this issue only affects a handful of people, or at least only a handful of people actually notice it. Gary Nebbett has been bringing this issue up in several of these threads. I don’t know enough about the details of how TCP works to confidently say that this is my problem, but when I look at a trace of my speed test in Wireshark, I see lots of “TCP Out-Of-Order” and packets being retransmitted with the same content. I don’t know if that’s normal or not, but it seems like common sense that the more retransmissions that happen, the less effective bandwidth I will have.
This has to be the problem I’m seeing, doesn’t it? If other users from my same ISP aren’t seeing the problem, their packets must not be getting reordered like mine.
At the end of the day, this doesn’t really seem like a huge problem. It’s solvable by upgrading to Windows 11. What complicates matters is the fact that most of my computers don’t officially support it. In my mind, that makes upgrading risky. If anyone from Microsoft is reading this, is there any chance you can backport the Windows 11 TCP fixes to Windows 10? Pretty please? Whatever you changed in Windows 11 is definitely an improvement.
Either way, this issue isn’t a huge deal for me. 300 or 400 megabits is still amazing to have as an upload speed. I thank my lucky stars every day that I have access to these kinds of speeds. I wanted to share my discovery because I think this is an interesting problem. I’m also curious how widespread it is. Clearly it doesn’t affect everybody. Is anyone else who stumbled upon this post also running into this same issue?
no comments