Joint Feature Learning of Image Data with Embedded Metadata to Leverage Snake Species Classification

Automatic identification of snake species from non-standard photos is an important task to improve medical treatment of snakebites. To address this problem, the SnakeCLEF 2023 competition provides a large data set of photos and metadata information for 1,784 snake species. This paper describes the FHDO Biomedical Computer Science Group’s (BCSG) participation in this competition. Through a series of experiments investigating the effects of pre-trained feature extractors, image sizes, metadata integrations, class balance learning and multiple instance pooling methods, a proposed model architecture for joint feature learning of image data and embedded metadata is presented to improve classification of snake species. With this proposal, the best model achieved a macro F1-Score of 81.90 % and challenge-specific metrics of 90.09 % Track 1 and 1, 149 Track 2 on the challenge public test data set.