speaker diarization

Introduction

The PyAnnote Speaker Diarization model is a part of the PyAnnote toolkit, focused on audio and speech processing tasks. This model is designed to perform speaker diarization, which involves determining "who spoke when" in an audio recording. It utilizes the PyAnnote.audio library, which offers a comprehensive suite of neural building blocks for speaker diarization tasks.

Architecture

The PyAnnote Speaker Diarization pipeline is built on version 2.1.1 of the PyAnnote.audio library. It employs a combination of neural network inference and clustering techniques to accurately segment and label different speakers within an audio file. The model's performance is optimized for tasks such as speaker change detection, voice activity detection, and overlapped speech detection.

Training

The model was trained using a variety of datasets, including AMI, DIHARD, and VoxConverse, among others. It is designed to automatically handle speaker diarization without the need for manual voice activity detection or specifying the number of speakers. The model is benchmarked against several datasets using a strict diarization error rate (DER) setup, which includes no forgiveness collar and evaluation of overlapped speech.

Guide: Running Locally

1. Installation: Ensure that PyAnnote.audio version 2.1.1 is installed. Follow the detailed installation instructions available on the GitHub repository.

2. Access Requirements:

   • Visit hf.co/pyannote/speaker-diarization and hf.co/pyannote/segmentation to accept user conditions.
   • Create an access token at hf.co/settings/tokens.

3. Running the Pipeline:

   from pyannote.audio import Pipeline
   pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization@2.1", use_auth_token="ACCESS_TOKEN_GOES_HERE")
   diarization = pipeline("audio.wav")
   with open("audio.rttm", "w") as rttm:
       diarization.write_rttm(rttm)
   

4. Advanced Usage: Specify the number of speakers if known using num_speakers or define bounds with min_speakers and max_speakers.

5. Hardware: For optimal performance, using a cloud GPU such as an Nvidia Tesla V100 is recommended. The real-time factor is approximately 2.5%, meaning it takes about 1.5 minutes to process one hour of audio on this hardware.

License

The PyAnnote Speaker Diarization model is open-source and licensed under the MIT License. This permits wide usage and modification, with the condition of attribution to the original authors.