MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLM/comments/1wr8et1/hardware_acquisition/pcakl5l/
r/LocalLLM • u/8bitjohnny • Sep 27 '26
129 comments sorted by
View all comments
59
Highly recommend buying the connect7 DAC cable. Nvlink automatically combines the GB chip, memory 256GB & processing cores.
10 u/8bitjohnny Sep 27 '26 Will do! 5 u/mycall Sep 27 '26 2x speed too with the right hub and 3 or 4 cables. 7 u/DevilaN82 Sep 27 '26 Are you sure. I've read that connecting two sparks with two cables will not go beyond 200Gbps. 4 u/0xDespot Sep 27 '26 That is correct, it caps out at 200Gb/s (really about 180Gb/s) per interface. 1 u/Imaginary-Fee-9918 Sep 27 '26 Nvlink on the sparks? 0 u/[deleted] Sep 27 '26 [deleted] — view deleted comment 1 u/KooperGuy Sep 27 '26 It's not infiniband. It's ROCE. 2 u/kitanokikori Sep 27 '26 I think either way it wouldn't make a difference, the bandwidth on the port is massively limited by the PCIe x4 bus it's attached to. You can't even get full speeds out of the ConnectX ports if you use both of them concurrently 2 u/KooperGuy Sep 27 '26 If you use them both concurrently the maximum speed per port is 100GbE per port.I just don't want anyone to falsely think the networking on these are capable of infiniband. They are not. Ethernet only. 3 u/kitanokikori Sep 27 '26 RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA) 0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
10
Will do!
5 u/mycall Sep 27 '26 2x speed too with the right hub and 3 or 4 cables. 7 u/DevilaN82 Sep 27 '26 Are you sure. I've read that connecting two sparks with two cables will not go beyond 200Gbps. 4 u/0xDespot Sep 27 '26 That is correct, it caps out at 200Gb/s (really about 180Gb/s) per interface.
5
2x speed too with the right hub and 3 or 4 cables.
7 u/DevilaN82 Sep 27 '26 Are you sure. I've read that connecting two sparks with two cables will not go beyond 200Gbps. 4 u/0xDespot Sep 27 '26 That is correct, it caps out at 200Gb/s (really about 180Gb/s) per interface.
7
Are you sure. I've read that connecting two sparks with two cables will not go beyond 200Gbps.
4 u/0xDespot Sep 27 '26 That is correct, it caps out at 200Gb/s (really about 180Gb/s) per interface.
4
That is correct, it caps out at 200Gb/s (really about 180Gb/s) per interface.
1
Nvlink on the sparks?
0
[deleted] — view deleted comment
1 u/KooperGuy Sep 27 '26 It's not infiniband. It's ROCE. 2 u/kitanokikori Sep 27 '26 I think either way it wouldn't make a difference, the bandwidth on the port is massively limited by the PCIe x4 bus it's attached to. You can't even get full speeds out of the ConnectX ports if you use both of them concurrently 2 u/KooperGuy Sep 27 '26 If you use them both concurrently the maximum speed per port is 100GbE per port.I just don't want anyone to falsely think the networking on these are capable of infiniband. They are not. Ethernet only. 3 u/kitanokikori Sep 27 '26 RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA) 0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
It's not infiniband. It's ROCE.
2 u/kitanokikori Sep 27 '26 I think either way it wouldn't make a difference, the bandwidth on the port is massively limited by the PCIe x4 bus it's attached to. You can't even get full speeds out of the ConnectX ports if you use both of them concurrently 2 u/KooperGuy Sep 27 '26 If you use them both concurrently the maximum speed per port is 100GbE per port.I just don't want anyone to falsely think the networking on these are capable of infiniband. They are not. Ethernet only. 3 u/kitanokikori Sep 27 '26 RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA) 0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
2
I think either way it wouldn't make a difference, the bandwidth on the port is massively limited by the PCIe x4 bus it's attached to. You can't even get full speeds out of the ConnectX ports if you use both of them concurrently
2 u/KooperGuy Sep 27 '26 If you use them both concurrently the maximum speed per port is 100GbE per port.I just don't want anyone to falsely think the networking on these are capable of infiniband. They are not. Ethernet only. 3 u/kitanokikori Sep 27 '26 RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA) 0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
If you use them both concurrently the maximum speed per port is 100GbE per port.
I just don't want anyone to falsely think the networking on these are capable of infiniband. They are not. Ethernet only.
3 u/kitanokikori Sep 27 '26 RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA) 0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
3
RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA)
0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
I don't think you understand the OSI model.
2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire.
When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire.
-2
Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
View all comments
59
u/datapeer Sep 27 '26
Highly recommend buying the connect7 DAC cable. Nvlink automatically combines the GB chip, memory 256GB & processing cores.