We present Extreme Bandwidth Extension Network (EBEN), a Generative Adversarial network (GAN) that enhances audio measured with body-conduction microphones. This type of capture equipment suppresses ambient noise at the expense of speech bandwidth, thereby requiring signal enhancement techniques to recover the wideband speech signal. EBEN leverages a multiband decomposition of the raw captured speech to decrease the data time-domain dimensions, and give better control over the full-band signal. This multiband representation is fed to a U-Net-like model, which adopts a combination of feature and adversarial losses to recover an enhanced audio signal. We also benefit from this original representation in the proposed discriminator architecture. Our approach can achieve state-of-the-art results with a lightweight generator and real-time compatible operation.
In this example, we trained a bandwidth extension model on a given language and inferred it on degraded audio in another language. In the obtained enhanced audio, you'll notice a strong "accent", based on the language the model was trained for. Speech quality metrics do not penalize this behavior and give similar scores to models trained in the same language as inference.
(by a model trained with english content)
(by a model trained with french content)
In order to evaluate the performance of the trained models, we performed two MUSHRA listening tests with 88 (resp. 82) participants on sound quality and ease of understanding. Each participant had to listen to 7 different audios enhanced with 5 different approaches (+reference +low anchor + original corrupted audio). For each experiment, the MUSHRA procedure consisted in giving a note between 0 and 100. The results from this study are found below.
|Approach||Inference mode latency for a single one-second sample on a NVIDIA GeForce RTX 2080 Ti|
|Approach||Inference mode maximum memory allocation for a single one-second sample on a NVIDIA GeForce RTX 2080 Ti|
|Audio U-net||1117.3 MB|
|Hifi-GAN v3||22.2 MB|
|Batch size||Temporal length||Sampling rate|
|16||2 sec||16 kHz|
|Bands number (=decimation)||Kernel size||Beta|