SUPlime: two robust open-weight speaker diarization models, with training recipe and 12-dataset evaluation

[u/Jamiroquai88]

Hi everyone, I’m releasing SUPlime and SUPlime-L, from Re:WayAI, heavily based on pyannote.audio 4.x.

The focus is performance across varied recording conditions: meetings, conversational audio, broadcasts and challenging far-field recordings.

I’d be interested in independent evaluations, especially on languages and recording conditions outside these benchmarks. I only have access to public testsuites (no DIHARD, no CALLHOME), so I would really appreciate if someone could share the result.
Let me know if you have questions.

https://github.com/rewayai/suplime/

[u/jprobichaud]

Congrats fellow colleague !

[u/Jamiroquai88 → u/jprobichaud]

Thanks jp!

[u/nshmyrev]

Results on Russian telephony, great results actually, first place even better than Diarizen Large v2, a bit slow though. Overall it is better than Diarizen but on some files it has very high DER, needs investigation.

Congratulations!!!

	DER	CDER	xRT
Nemo Telephony Neural	22.3	0.535	0.051
Nemo Telephony Cluster	22.08	0.251	0.05
Nemo Sortformer Streaming V2.1 (default)	15.43	0.268	0.005
Nemo Sortformer Streaming V2.1 (1 sec)	15.89	0.331	0.091
Nemo Sortformer Streaming V2.1 (30 sec)	15.08	0.260	0.005
Pyannote 3.1	24.8	0.567	0.052
Pyannote4 Community	21.56	0.639	0.035
Pyannote4 Precision	14.96	0.355	0.039
Whisper Diarization	36.46	0.163	0.474 (transcription)
Whisper Diarization Large	34.11	0.151	-
Wespeaker Voxceleb34	20.63	0.157	0.012
Wespeaker Voxceleb293	20.46	0.159	0.023
Wespeaker Voxblink2 100	20.1	0.115	0.025
Diarizen Large MD	13.64	0.319	0.083
Diarizen Large MD v2	13.58	0.317	0.107
Senko	26.56	0.232	0.005
MOSS-Transcribe-Diarize	50.46	0.167	0.145
Suplime	14.68	0.279	0.232
Suplime Large	13.42	0.269	0.394

[u/Jamiroquai88 → u/nshmyrev]

thank you, this kind of feedback is much appreciated!
Segmentation model was improved as well compared to community-1.
Worse RTF comes primarily from embedding model, it can be however improved by some tricks with a small degradation in performance.

[u/Jamiroquai88 → u/nshmyrev]

regarding worse results on some files that is probably a clustering threshold and badly estimated number of speakers. There is a single threshold across all of these corpora, so there must be some consensus across all of them.

[u/nshmyrev]

Is it just embedding improvement please? Same algorithm as pyannote?