OP mentions using "Cat 7" cables - please don't buy these. Cat 7 isn't something that exists in TIA/EIA standards, only in ISO/IEC and it requires GG45 or TERA connectors. Cat 7 with RJ45 connectors isn't standardized, so you have no idea what you're actually getting. Stick with pure copper Cat 6A.
I am not surprised. My Windows 11 systems with modern and beefy hardware frequently runs very slow for reasons unknown. I did use https://github.com/Raphire/Win11Debloat recently and that seemed to have helped. Windows by default comes with a lot of crap that most of us do not use but it consumes resources anyway.
I have been considering a move back to Linux. It is only Microsoft Teams on Windows that I have to use daily that is holding me back.
It also buffers the downloaded data completely into memory last time I checked. So downloading a file bigger than the available RAM just doesn't work and you have to use WebClient instead.
Another fun one is Extract-Archive which is painfully slow while using the System.IO.Compression.ZipFile CLR type directly is reasonably fast. Powershell is really a head scratcher sometimes.
This is atrocious. I get it, some things are less trivial than they seem - but I would be ashamed for shipping something like this, and even more for not fixing it.
Just tried copying a 20GB file to my Windows desktop from a mounted Samba share through gigabit ethernet (nvme on both sides). Explorer, Copy-Item and robocopy all saturated the connection with no issues.
There's definitely something off about OP's setup, though I have no idea what it could be. I'd start by checking the latency between the machines. Might also be the network adapter or its drivers.
My first thought would be some kind of "security" software (maybe even as simple as windows defender) inspecting the files as they're coming in, which might be done for any process not on some allow-list. And maybe the allow-list is basically just "explorer.exe". And maybe it's faster at checking some processes than others.
rsync being that much slower makes no sense since back when I used windows I rsync was saturating 1 gig easily, this has to be running on a very slow pentium or something.
ugh, I don't know why copying files and basic I/O is so fucked on Windows. Recently I was trying to copy some large movie files between 2 folders on an NVME SSD formatted to ExFAT in a good USB-C enclosure connected over 20Gbps USB-C port and explorer would literally just freeze & crash doing that. I had to copy one file at a time to make it not crash, and then it would have this weird I/O pattern where the transfer would do almost nothing for 1-2 minutes, then the speed eventually picked up.
This isn't even going into WSL. I specifically stopped using WSL and moved to a separate linux devbox because of all the weirdness and slowness with filesystem access across the WSL boundary. Something like listing a lot of files would be very slow IIRC. Slightly tangentially, the whole situation around sharing files across OSes is pretty frustrating. The only one that works without 3rd party paid drivers on all 3 major OSes is ExFAT and that is limited in many other ways compared to ext4 or NTFS.
This feels like usb 3 super speed flakeyness. Did you do all the usual things of trying different ports, moving sources of interferrence, etc? Front ports at super speed are typically the most trouble.
> SFTP is an encrypted protocol, so maybe those CPU cycles add up to a lot of extra work over time or slowdown. That… shouldn’t feel convincing to anyone who gives it more than 15 seconds of thought, but we all live with our eyes wide shut at times.
FWIW, I previously spent some time trying to get the maximum possible throughput when copying files between a Windows host and a Linux VM, and the encryption used by most protocols did actually become a bottleneck eventually. I expect this isn't a big factor on 1gbps ethernet, but I've never measured it.
The bottleneck with SFTP / SCP / SSH is usually the server software - SSH can multiplex streams, so it implements its own TCP-style sliding windows for channel data. Unfortunately OpenSSH and similar server implementations suffer from the exact same problems that TCP did, where the windows don't scale up to modern connection speeds, so the maximum data in-flight quickly gets limited at higher BDPs.
HPN-SSH[1] resolves this but isn't widely deployed.
OP mentions using "Cat 7" cables - please don't buy these. Cat 7 isn't something that exists in TIA/EIA standards, only in ISO/IEC and it requires GG45 or TERA connectors. Cat 7 with RJ45 connectors isn't standardized, so you have no idea what you're actually getting. Stick with pure copper Cat 6A.
It's fortunately been years since I have used Windows, but it looks like the old staples are still ahead of the curve:
https://fastcopy.jp/
https://www.codesector.com/teracopy
(I have certainly forgotten at least one...)
I am not surprised. My Windows 11 systems with modern and beefy hardware frequently runs very slow for reasons unknown. I did use https://github.com/Raphire/Win11Debloat recently and that seemed to have helped. Windows by default comes with a lot of crap that most of us do not use but it consumes resources anyway.
I have been considering a move back to Linux. It is only Microsoft Teams on Windows that I have to use daily that is holding me back.
> I have been considering a move back to Linux. It is only Microsoft Teams on Windows that I have to use daily that is holding me back.
Me too. I've not tried this yet, but will soon: https://github.com/IsmaelMartinez/teams-for-linux
Invoke-WebRequest is also very slow if you forget to disable the progress bar with $ProgressPreference = 'SilentlyContinue'
PowerShell has some "interesting" design choices...
It also buffers the downloaded data completely into memory last time I checked. So downloading a file bigger than the available RAM just doesn't work and you have to use WebClient instead.
Another fun one is Extract-Archive which is painfully slow while using the System.IO.Compression.ZipFile CLR type directly is reasonably fast. Powershell is really a head scratcher sometimes.
Wasn't something like npm much slower as well when it showed a progress indicator by default?
This is atrocious. I get it, some things are less trivial than they seem - but I would be ashamed for shipping something like this, and even more for not fixing it.
The page is 404 now. It looks like something went wrong when the author was trying to push a small edit to the page. The content is viewable at https://github.com/hiAndrewQuinn/til/blob/main/copy-item-is-...
Works fine for me.
Never assume anything done in Powershell is fast.
Just tried copying a 20GB file to my Windows desktop from a mounted Samba share through gigabit ethernet (nvme on both sides). Explorer, Copy-Item and robocopy all saturated the connection with no issues.
There's definitely something off about OP's setup, though I have no idea what it could be. I'd start by checking the latency between the machines. Might also be the network adapter or its drivers.
My first thought would be some kind of "security" software (maybe even as simple as windows defender) inspecting the files as they're coming in, which might be done for any process not on some allow-list. And maybe the allow-list is basically just "explorer.exe". And maybe it's faster at checking some processes than others.
rsync being that much slower makes no sense since back when I used windows I rsync was saturating 1 gig easily, this has to be running on a very slow pentium or something.
ugh, I don't know why copying files and basic I/O is so fucked on Windows. Recently I was trying to copy some large movie files between 2 folders on an NVME SSD formatted to ExFAT in a good USB-C enclosure connected over 20Gbps USB-C port and explorer would literally just freeze & crash doing that. I had to copy one file at a time to make it not crash, and then it would have this weird I/O pattern where the transfer would do almost nothing for 1-2 minutes, then the speed eventually picked up.
This isn't even going into WSL. I specifically stopped using WSL and moved to a separate linux devbox because of all the weirdness and slowness with filesystem access across the WSL boundary. Something like listing a lot of files would be very slow IIRC. Slightly tangentially, the whole situation around sharing files across OSes is pretty frustrating. The only one that works without 3rd party paid drivers on all 3 major OSes is ExFAT and that is limited in many other ways compared to ext4 or NTFS.
This feels like usb 3 super speed flakeyness. Did you do all the usual things of trying different ports, moving sources of interferrence, etc? Front ports at super speed are typically the most trouble.
want to see rsync WSL 1 in that comparison
filesystem should be faster in WSL2 but not if the file resides in the windows path I think
> SFTP is an encrypted protocol, so maybe those CPU cycles add up to a lot of extra work over time or slowdown. That… shouldn’t feel convincing to anyone who gives it more than 15 seconds of thought, but we all live with our eyes wide shut at times.
FWIW, I previously spent some time trying to get the maximum possible throughput when copying files between a Windows host and a Linux VM, and the encryption used by most protocols did actually become a bottleneck eventually. I expect this isn't a big factor on 1gbps ethernet, but I've never measured it.
The bottleneck with SFTP / SCP / SSH is usually the server software - SSH can multiplex streams, so it implements its own TCP-style sliding windows for channel data. Unfortunately OpenSSH and similar server implementations suffer from the exact same problems that TCP did, where the windows don't scale up to modern connection speeds, so the maximum data in-flight quickly gets limited at higher BDPs.
HPN-SSH[1] resolves this but isn't widely deployed.
[1] https://www.psc.edu/hpn-ssh-home/