EVALUATING LARGE LANGUAGE MODELS USING CONTRAST SETS: AN EXPERIMENTAL APPROACH

Abstract

In the field of Natural Language Inference (NLI), particularly for multi-input text classification tasks, Cross-Entropy Loss is commonly used as a general error metric. Although effective as a training benchmark, this metric does not adequately assess a model’s understanding of language entailments. In this work, we propose a novel approach for creating a contrast set for the Stanford Natural Language Inference (SNLI) dataset. Our method involves automatically replacing verbs, adverbs, and adjectives with their synonyms, maintaining the original sentence’s meaning. This approach helps determine whether a model truly comprehends the language or merely identifies recurring patterns for making predictions. We utilized the ELECTRA-small model for our investigation. While the model exhibits an 89.9% accuracy on the standard SNLI dataset, its performance drops to 72.5% on our contrast set—a significant 17% decrease. This finding prompted an in-depth analysis to understand the underlying learning patterns of the model. Subsequently, we enhanced the model’s robustness by fine-tuning it with a contrast training dataset tailored for SNLI, resulting in an improved accuracy of 85.5% on contrast sets. These experiments underscore the necessity for more balanced datasets in NLI tasks that account for varied linguistic expressions. We anticipate that our findings will inspire further development of comprehensive datasets, fostering the advancement of more nuanced and effective NLI models.