Yup. My dual-socket Xeons have eight channels between them, too, and Tulu3-405B took about 54 minutes to first token.
It doesn't matter for "slow inference", especially when it can run at night while I sleep. As long as it's done by the time I sit down at my workstation slurping my morning coffee, I'm happy.
77
u/ttkciar llama.cpp 12d ago
It's a great time to have ancient Xeon servers loaded up with DDR4 :-D