Speech Emotion Recognition with Wav2Vec2

Summary

• Introduction

• Parameters

• Usage for developers

• Evaluation Results

• Reference

• License

Introduction

Speech Emotion Recognition with Wav2Vec2 is a speech emotion recognition model based on Wav2Vec2. The model is designed to classify speech recordings into different emotional categories, including Angry, Disgust, Fearful, Happy, Neutral, Sad, and Surprised.

• Model: facebook/wav2vec2-large-xlsr-53

• Model Type: Wav2Vec2 Large XLSR-53 fine-tuned for audio classification

• Framework: Hugging Face Transformers

• Task: Speech Emotion Recognition

• Output Classes: Angry, Disgust, Fearful, Happy, Neutral, Sad, Surprised

Dataset

The training and evaluation data are sourced from multiple speech emotion datasets:

• RAVDESS
• SAVEE
• TESS
• URDU

The dataset contains the following emotion distribution:

Emotion	Count
Sad	752
Happy	752
Angry	752
Neutral	716
Disgust	652
Fearful	652
Surprised	652
Calm	192

The calm emotion was excluded during training because of its underrepresentation.

Preprocessing

• Audio Loading: Audio files are loaded using Librosa and converted to NumPy arrays.
• Feature Extraction: The audio is processed using the Wav2Vec2 Feature Extractor, which standardizes and normalizes the audio features before they are passed to the model.

Intended Use

The model is intended for speech emotion recognition applications where an audio recording is classified into one of the supported emotional categories.

Capabilities

• Speech emotion recognition
• Audio classification
• Multi-class emotion classification
• Inference from speech recordings
• GPU and CPU inference

Parameters

Inputs

• input_audio - (string): Path to the input audio file.

Output

• predicted_emotion - (string): Emotion label.

Usage for developers

Requirements

pip install -r requirements.txt

Code based on AIOZ structure

import torch
import librosa
import numpy as np
from pathlib import Path
from typing import Any, Union, Literal
from transformers import AutoModelForAudioClassification, AutoFeatureExtractor
from aioz_ainode_adapter.schemas import InputObject, OutputObject

import os

class MyInput(InputObject):
    input_audio: str


class MyOutput(OutputObject):
    predicted_emotion: str


def preprocess_audio(audio_path, feature_extractor, max_duration=30.0):
    audio_array, sampling_rate = librosa.load(audio_path, sr=feature_extractor.sampling_rate)

    max_length = int(feature_extractor.sampling_rate * max_duration)
    if len(audio_array) > max_length:
        audio_array = audio_array[:max_length]
    else:
        audio_array = np.pad(audio_array, (0, max_length - len(audio_array)))

    inputs = feature_extractor(
        audio_array,
        sampling_rate=feature_extractor.sampling_rate,
        max_length=max_length,
        truncation=True,
        return_attention_mask=True,
        return_tensors="pt",
    )
    return inputs


def do_ai_task(
        input_audio: Union[str, str],
        model_storage_directory: Union[str, Path],
        device: Literal["cpu", "cuda", "gpu"] = "cpu",
        *args, **kwargs) -> Any:
    """Define AI task: load model, pre-process, post-process, etc ..."""
    # Define AI task workflow. Below is an example

    max_duration=30.0
    model_path = model_storage_directory
    model = AutoModelForAudioClassification.from_pretrained(model_path)
    feature_extractor = AutoFeatureExtractor.from_pretrained(model_path, do_normalize=True, return_attention_mask=True)
    id2label = model.config.id2label

    inputs = preprocess_audio(input_audio, feature_extractor, max_duration)
    model = model.to(device)
    inputs = {key: value.to(device) for key, value in inputs.items()}

    with torch.no_grad():
        outputs = model(**inputs)

    logits = outputs.logits
    predicted_id = torch.argmax(logits, dim=-1).item()
    predicted_label = id2label[predicted_id]

    return predicted_label

Evaluation Results

The reported evaluation metrics are:

Metric	Score
Loss	0.4989
Accuracy	0.9168
Precision	0.9209
Recall	0.9168
F1 Score	0.9166

These metrics are reported after training and are used to evaluate the model's speech emotion recognition performance.

The training process was monitored using Weights & Biases (WandB).

Training

The model was trained using the following configuration:

• Learning Rate: 5e-05
• Train Batch Size: 2
• Evaluation Batch Size: 2
• Random Seed: 42
• Gradient Accumulation Steps: 5
• Total Effective Batch Size: 10
• Optimizer: Adam
• Adam Betas: (0.9, 0.999)
• Adam Epsilon: 1e-08
• Learning Rate Scheduler: Linear
• Warmup Ratio: 0.1
• Number of Epochs: 25
• Mixed Precision: Native AMP (Automatic Mixed Precision)

Reference

This implementation is based on the Hugging Face facebook/wav2vec2-large-xlsr-53 model, which is fine-tuned for audio classification.

The original model and implementation are available at:

Speech Emotion Recognition with Facebook Wav2Vec2 Large XLSR-53

License

We respect and comply with the terms of the author's apache-2.0 license cited in the Reference section.