> This is a bit dependent on the type of DAC, and the vendor. Generally speaking, we are moving into an era as we move above 100GbE speeds where DACs will be limited to 5m runs and shorter runs.
This is generally good enough for intra-rack runs, especially if you put your top-of-rack (ToR) switch in the middle: you basically only have to go about 1m/3' in either direction (up/down).
I'm waiting for 3 DACs and a few other bits to arrive today to move closer to 10G networking at home. Moving house soon and the new place will have 2.5Gbps FTTP (both up and down) so I wanted to be prepared for that. Given my existing broadband is only 500/75Mbps FTTP I was fine with a 1GbE internal network and Wifi-6 meshing. I could have planned to move to 2.5GbE but it may have been a bottleneck at some point, so may as well push straight on to 10G.
I have a USW-Aggregation with 8 SFP+ ports arriving today too. Just have to install Intel X520-DA2 cards in two of my servers (Proxmox host and a general Linux server), and the NAS also has a 10G SFP+ port, and then connect it all up.
Most of it second hand from eBay for half the usual retail price.
The card is obviously 16-lane, but it also has two ports; 40Gb total. In a server that’s fine, but if you want 10G in a desktop you’ll have a problem.
I’m probably not telling you anything new. NICs using newer PCI generations are rare as hen’s teeth. It should be possible to do this with four lanes, but isn’t…
Unless you find a 25G dual-port card, in which case the single lane my secondary slots hand out does at least suffice for 10G one way.
While working for a broadcasting/networking integration gig, we designed, assembled and field-terminated thousands of copper and fibre runs from 1/2 ft to football field lengths.
The amount of time required to terminate a copper cable in the field is seconds, and felt a bit like art. Something about the way it reliably reacted was magical and felt "strong."
Terminating or splicing a fibre cable felt like wrestling a snake covered in melted crayons, and the failure rate was significantly higher across the board. And it wasn't just workmanship, but quality of product, terminating environment, available equipment, misuse by future operators etc.
That said, at a certain point, we as a firm learned that most purchasers would rather the low latency/small footprint of optical/fibre versus copper, maintenece/failure be damned. Though, maybe part of our willingness to push fibre came from knowing that most purchasers would in 1-2 years call us back in to replace the rack terminations with copper :)
There are 2 very different main types of "copper" in this context. Each can break down into more specific subcategories but they have a relatively common general behavior with their primary type.
First there is the BASE-T RJ45 stuff, which it sounds like you might have been working with. At 10G or higher speeds this get relatively power hungry and is not really an advantage over fiber unless you are also delivering PoE or are trying to reuse existing cabling.
This type (DAC) is a special type of pre-made cable assembly which eschews much of the advanced signalling/conversion logic. The upside is the power usage is low (often even lower than fiber) and the cost is dirt cheap. The downside is the lengths are much more limited and it's intended to be preterminated SFP-to-SFP (or the like) cable assemblies instead of modular patching/custom built.
There is indeed a latency difference, usually DAC < fiber < BASE-T, but they are all within a few microseconds (not milliseconds) of each other so you really have to be pressed to care about it (to the point you're looking at specialized low latency switches and paying extra to lay things out in a way which minimizes the number of L2 hops rather than the cost).
Dacs are awesome until you encounter a full rack with nothing but 2 and 4 meter dacs. They appeal very much to the lazy engineer, and a lazy engineer can turn them into a rats nest.
> This is a bit dependent on the type of DAC, and the vendor. Generally speaking, we are moving into an era as we move above 100GbE speeds where DACs will be limited to 5m runs and shorter runs.
This is generally good enough for intra-rack runs, especially if you put your top-of-rack (ToR) switch in the middle: you basically only have to go about 1m/3' in either direction (up/down).
I'm waiting for 3 DACs and a few other bits to arrive today to move closer to 10G networking at home. Moving house soon and the new place will have 2.5Gbps FTTP (both up and down) so I wanted to be prepared for that. Given my existing broadband is only 500/75Mbps FTTP I was fine with a 1GbE internal network and Wifi-6 meshing. I could have planned to move to 2.5GbE but it may have been a bottleneck at some point, so may as well push straight on to 10G.
I have a USW-Aggregation with 8 SFP+ ports arriving today too. Just have to install Intel X520-DA2 cards in two of my servers (Proxmox host and a general Linux server), and the NAS also has a 10G SFP+ port, and then connect it all up.
Most of it second hand from eBay for half the usual retail price.
Nice work, that agg switch is excelllwnt.
I went with some cheap eBay cards and slotted them into a synology and PC.
They work great and have for years.
https://www.ebay.com/itm/384094168784?_skw=connectx+mellanox...
How many PCIe lanes are you allocating?
The card is obviously 16-lane, but it also has two ports; 40Gb total. In a server that’s fine, but if you want 10G in a desktop you’ll have a problem.
I’m probably not telling you anything new. NICs using newer PCI generations are rare as hen’s teeth. It should be possible to do this with four lanes, but isn’t…
Unless you find a 25G dual-port card, in which case the single lane my secondary slots hand out does at least suffice for 10G one way.
While working for a broadcasting/networking integration gig, we designed, assembled and field-terminated thousands of copper and fibre runs from 1/2 ft to football field lengths.
The amount of time required to terminate a copper cable in the field is seconds, and felt a bit like art. Something about the way it reliably reacted was magical and felt "strong."
Terminating or splicing a fibre cable felt like wrestling a snake covered in melted crayons, and the failure rate was significantly higher across the board. And it wasn't just workmanship, but quality of product, terminating environment, available equipment, misuse by future operators etc.
That said, at a certain point, we as a firm learned that most purchasers would rather the low latency/small footprint of optical/fibre versus copper, maintenece/failure be damned. Though, maybe part of our willingness to push fibre came from knowing that most purchasers would in 1-2 years call us back in to replace the rack terminations with copper :)
There are 2 very different main types of "copper" in this context. Each can break down into more specific subcategories but they have a relatively common general behavior with their primary type.
First there is the BASE-T RJ45 stuff, which it sounds like you might have been working with. At 10G or higher speeds this get relatively power hungry and is not really an advantage over fiber unless you are also delivering PoE or are trying to reuse existing cabling.
This type (DAC) is a special type of pre-made cable assembly which eschews much of the advanced signalling/conversion logic. The upside is the power usage is low (often even lower than fiber) and the cost is dirt cheap. The downside is the lengths are much more limited and it's intended to be preterminated SFP-to-SFP (or the like) cable assemblies instead of modular patching/custom built.
There is indeed a latency difference, usually DAC < fiber < BASE-T, but they are all within a few microseconds (not milliseconds) of each other so you really have to be pressed to care about it (to the point you're looking at specialized low latency switches and paying extra to lay things out in a way which minimizes the number of L2 hops rather than the cost).
Dacs are awesome until you encounter a full rack with nothing but 2 and 4 meter dacs. They appeal very much to the lazy engineer, and a lazy engineer can turn them into a rats nest.
Dacs are great … they let you avoid having to convert to optical or having to splice your own cable.
Downside the cables are kind of heavy and sometimes folks dont like that.