.png)
Machine learning (ML) and deep learning (DL) have been increasingly used for solving bioacoustics tasks, but typical research studies in bioacoustics focus on different types of species and ML/DL models, making it difficult to compare across different approaches in a standardized way. Inspired by the benchmark datasets in other ML domains such as ImageNet and GLUE, we propose the ESP Bioacoustic Benchmark, a collection of bioacoustics tasks and datasets specifically designed to measure and compare the performance of ML algorithms in a standardized way. We specifically focus on the two unique aspects of bioacoustics—diversity of species and the small data size, and design the benchmark so that it rewards ML models that address these two issues successfully. We also publish the preprocessed datasets, the library code, and non-DL and DL baseline implementations to make it easier to reproduce and build new models on top of the benchmark infrastructure.
Affordable recording and storage devices are widely used for collecting bioacoustics data. These devices enable animal welfare and wildlife conservation applications, such as passive acoustic monitoring (PAM), which offers tools to monitor animal populations in a noninvasive and unbiased manner (Sugai et al., 2018). However, they produce a large amount of bioacoustic data, and processing and analysis of such data have been a bottleneck (Tuia et al., 2022, Stowell, 2021).
Machine learning (ML) has been used for automating the processing and analysis of bioacoustics tasks. It has been successfully applied to a variety of tasks such as classification of species, individuals, and various other characteristics of calls (Morfi et al., 2021), detection and recognition of vocalizations in passive recordings (LeBien et al., 2020), and automatic discovery of vocalization units (Sainburg et al., 2020), to name a few. In recent years, so-called “deep learning” models based on artificial neural networks drastically lowered the error rate and are increasingly used in those tasks.
However, typical research studies focus only on a small number of species and/or specific types of methodologies. This narrow focus has led to a proliferation of ML models and algorithms that do well on the datasets and tasks in question but are not necessarily outside their scope (Shwartz-Ziv and Armon, 2021). In addition, the lack of standard, publicly available datasets in bioacoustics (Mclouphlin et al., 2019, Baker and Vincent, 2019) makes it difficult to reproduce and compare different approaches in a standardized way.
If we turn our attention to other artificial intelligence (AI) fields, recent progress has primarily been driven by standard “benchmarks.” A benchmark is a collection of tasks and datasets specifically designed to measure the performance of ML algorithms in a standardized way. For example, standardized datasets such as MNIST (Deng, 2012), CIFAR-10/100 (Krizhevsky, 2009), and ImageNet (Deng et al., 2009) have been used to measure the accuracy of image recognition algorithms in computer vision for decades. Other examples include GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) for language and SUPERB (Yang et al., 2021) for human speech processing. A leaderboard is a ranking of ML approaches based on standard benchmarks. These standardized benchmarks and “leaderboards” make it possible to objectively and quantitatively compare different approaches, identify the strengths and weaknesses of different methodologies, and measure how the field is advancing overall (Olson et al., 2017). Moreover, they are the major driving force behind some of the recent developments in AI, including BERT (Devlin et al., 2018) and GPT-3 (Brown et al., 2020).
We believe that the field of bioacoustics can also benefit from such standardized benchmarks—they enable systematic and standardized comparisons of ML methodologies and can further push forward the field. There are, however, a few aspects that make the field of bioacoustics unique and different from other ML fields—diversity and lack of data. Unlike human speech, which is fairly uniform in terms of its frequency range, duration, sampling rate, etc., animal vocalizations are diverse, ranging from elephant low-frequency rumbles to ultrasonic vocalizations of mice and bats. Also, mainly due to the high acquisition cost, a typical bioacoustic dataset is considerably smaller than its human counterpart, effectively making it a low-resource setting. For example, a large dataset of cetaceans, one of the relatively well-studied taxa in bioacoustics, is still several orders of magnitude smaller than the human language dataset used to train recent powerful language models (Andreas et al., 2021). We take those unique aspects of the bioacoustics field into consideration and design the benchmark to encourage the development of ML methods that do well for diverse and low-resource settings.
In this benchmark, we focus on two main bioacoustics tasks—classification and recognition, the two most common tasks considered in the literature (Stowell, 2021). Classification is the task where an ML algorithm assigns one of the predefined labels to given audio, and the performance is measured by accuracy. We include the following classification datasets in this benchmark.