Identification and Management of Difficult Samples in a Large Product Dataset
Nieminen, Anni (2026)
2026
Tietotekniikan DI-ohjelma - Master's Programme in Information Technology
Informaatioteknologian ja viestinnän tiedekunta - Faculty of Information Technology and Communication Sciences
Hyväksymispäivämäärä
2026-02-05
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:tuni-202602042297
https://urn.fi/URN:NBN:fi:tuni-202602042297
Tiivistelmä
Shelf intelligence systems provide valuable insights to retail store associates, allowing them to act based on the current shelf state. These systems are complex software solutions that heavily rely on machine learning and computer vision algorithms to effectively model retail shelf state. Product recognition is a central component of shelf intelligence, providing end users with information about which products are missing from the shelf.
Large-scale retail product recognition often suffers from significant data quality issues due to the real-world conditions of retail store data acquisition. Common issues in the data include, for example, poor lighting, noisy images, turned products, and occlusions. These factors negatively impact product recognition performance and increase the need for costly human-in-the-loop intervention.
The goal of this thesis project was to investigate the impact of such visually challenging samples on content-based image retrieval-based retail product recognition, as well as to explore methods to identify and manage these samples.
Two methods for identifying challenging samples were developed in this thesis: a convolutional neural network-based binary classifier and another method based on visual question answering using vision-language models. Both classification frameworks were integrated into Scandit’s retail product recognition component. This enabled an effective comparison of the methods and further experiments to investigate the impact of challenging samples on retail product recognition performance.
Both methods were able to accurately identify visually challenging samples, with the vision-language model outperforming the convolutional neural network, as shown by a larger gap in product recognition performance between predicted challenging and non-challenging samples.
Further experiments investigated strategies for managing challenging samples in retail product recognition. While filtering challenging query samples resulted in a moderate product recognition performance increase, it also resulted in the loss of a significant number of correct predictions. Filtering challenging reference samples from the retrieval index resulted in decreased performance.
While it was shown that visually challenging samples are difficult to correctly recognize, it was also shown that sample “hardness” does not provide an effective confidence metric in content-based image retrieval-based retail product recognition. Moreover, while challenging reference samples are involved in incorrect product category predictions, a comprehensive retrieval index also containing challenging samples is often important for accurate product recognition. Overall, the results demonstrate that while visually challenging samples can be accurately identified, managing them involves important trade-offs that must be considered on an application basis.
Large-scale retail product recognition often suffers from significant data quality issues due to the real-world conditions of retail store data acquisition. Common issues in the data include, for example, poor lighting, noisy images, turned products, and occlusions. These factors negatively impact product recognition performance and increase the need for costly human-in-the-loop intervention.
The goal of this thesis project was to investigate the impact of such visually challenging samples on content-based image retrieval-based retail product recognition, as well as to explore methods to identify and manage these samples.
Two methods for identifying challenging samples were developed in this thesis: a convolutional neural network-based binary classifier and another method based on visual question answering using vision-language models. Both classification frameworks were integrated into Scandit’s retail product recognition component. This enabled an effective comparison of the methods and further experiments to investigate the impact of challenging samples on retail product recognition performance.
Both methods were able to accurately identify visually challenging samples, with the vision-language model outperforming the convolutional neural network, as shown by a larger gap in product recognition performance between predicted challenging and non-challenging samples.
Further experiments investigated strategies for managing challenging samples in retail product recognition. While filtering challenging query samples resulted in a moderate product recognition performance increase, it also resulted in the loss of a significant number of correct predictions. Filtering challenging reference samples from the retrieval index resulted in decreased performance.
While it was shown that visually challenging samples are difficult to correctly recognize, it was also shown that sample “hardness” does not provide an effective confidence metric in content-based image retrieval-based retail product recognition. Moreover, while challenging reference samples are involved in incorrect product category predictions, a comprehensive retrieval index also containing challenging samples is often important for accurate product recognition. Overall, the results demonstrate that while visually challenging samples can be accurately identified, managing them involves important trade-offs that must be considered on an application basis.