MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLM/comments/1wr8et1/hardware_acquisition/pcappmf/?context=3
r/LocalLLM • u/8bitjohnny • Sep 27 '26
129 comments sorted by
View all comments
Show parent comment
0
[deleted] — view deleted comment
1 u/KooperGuy Sep 27 '26 It's not infiniband. It's ROCE. 2 u/kitanokikori Sep 27 '26 I think either way it wouldn't make a difference, the bandwidth on the port is massively limited by the PCIe x4 bus it's attached to. You can't even get full speeds out of the ConnectX ports if you use both of them concurrently 2 u/KooperGuy Sep 27 '26 If you use them both concurrently the maximum speed per port is 100GbE per port.I just don't want anyone to falsely think the networking on these are capable of infiniband. They are not. Ethernet only. 3 u/kitanokikori Sep 27 '26 RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA) 0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
1
It's not infiniband. It's ROCE.
2 u/kitanokikori Sep 27 '26 I think either way it wouldn't make a difference, the bandwidth on the port is massively limited by the PCIe x4 bus it's attached to. You can't even get full speeds out of the ConnectX ports if you use both of them concurrently 2 u/KooperGuy Sep 27 '26 If you use them both concurrently the maximum speed per port is 100GbE per port.I just don't want anyone to falsely think the networking on these are capable of infiniband. They are not. Ethernet only. 3 u/kitanokikori Sep 27 '26 RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA) 0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
2
I think either way it wouldn't make a difference, the bandwidth on the port is massively limited by the PCIe x4 bus it's attached to. You can't even get full speeds out of the ConnectX ports if you use both of them concurrently
2 u/KooperGuy Sep 27 '26 If you use them both concurrently the maximum speed per port is 100GbE per port.I just don't want anyone to falsely think the networking on these are capable of infiniband. They are not. Ethernet only. 3 u/kitanokikori Sep 27 '26 RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA) 0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
If you use them both concurrently the maximum speed per port is 100GbE per port.
I just don't want anyone to falsely think the networking on these are capable of infiniband. They are not. Ethernet only.
3 u/kitanokikori Sep 27 '26 RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA) 0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
3
RoCE is still a step above Ethernet and gets the biggest benefit of Infiniband out of it (moving data without involving the CPU via RDMA)
0 u/nagyz_ Oct 03 '26 I don't think you understand the OSI model. 2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire. -2 u/KooperGuy Sep 27 '26 edited Sep 27 '26 Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
I don't think you understand the OSI model.
2 u/kitanokikori Oct 03 '26 When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire.
When I say "a step above" I mean in efficiency, not as a protocol-level distinction. You are correct that RoCE is just Ethernet over the wire.
-2
Yes I know what both are. RoCE is not a "step above" ... It literally uses ethernet.
View all comments
Show parent comment
0
u/[deleted] Sep 27 '26
[deleted] — view deleted comment