MITIGATING ORDER SENSITIVITY IN LARGE LANGUAGE MODELS FOR MULTIPLE-CHOICE QUESTION TASKS

Abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency in understanding and generating human language, driving advancements in various natural language processing (NLP) tasks. Despite their success, LLMs face a persistent challenge when handling multiple-choice questions (MCQs), particularly with the issue of order dependency. This phenomenon occurs when LLMs exhibit inconsistent accuracy based on the order in which answer options are presented, often leading to performance fluctuations. This paper investigates the sensitivity of LLMs to MCQs by systematically altering the order of answer options across multiple datasets and observing model performance. The experiment spans 57 diverse domains using various LLMs, including Mistral, LLaMA, and GPT variations. Our findings reveal that LLMs, regardless of their architecture, are sensitive to the order in which answers are presented, demonstrating a clear susceptibility to selection and positional bias in MCQ scenarios. To address this challenge, we propose a combination of zero-shot and few-shot learning techniques, along with fine-tuning strategies on domain-specific datasets, aimed at mitigating the effects of selection bias and positional dependency. Additionally, we introduce custom prompting techniques, leveraging a question dissector logic designed to enhance the reasoning capabilities of LLMs. By incorporating these methods, we aim to minimize biases and improve the consistency of LLM responses across different permutations of answer choices.