Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context
Abhishek Kumar Singh, Shrey Nag, Sachita, Lipi Goel, Rajeshwar Singh Janwar
Why It Matters
What makes this one worth your time
This framework is crucial for developing and assessing LLMs that are fair, safe, and accurate in the diverse Indian context, which is often overlooked by existing benchmarks.
Inspect India Evals provides a culturally and linguistically relevant benchmarking framework for evaluating LLMs in India.
Summary
The paper introduces Inspect India Evals, an open-source benchmarking framework designed to evaluate large language models within the Indian linguistic and cultural context, addressing the limitations of existing Western-centric benchmarks.
Key contributions
- Development of a multilingual benchmarking framework tailored for the Indian context.
- Introduction of the BharatBBQ benchmark for assessing social bias in India.
- Evaluation of open-weight models on Indian cultural knowledge and safety compliance.
Notable insights
- The use of a composite India Fairness Index to evaluate models on cultural knowledge and safety compliance is a novel approach.
- The framework's adaptability to multiple Indian languages and cultural nuances highlights the importance of context-specific evaluations.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.25375v1 Announce Type: new Abstract: India is a vast nation of over 1.4 billion people, varied by hundreds of diverse and locally specific traditions and cultures and 22 officially recognized languages. Large language models (LLMs) are now being deployed on a massive scale throughout the mainland as well as in remote villages. However, the common benchmarks - MMLU, BIG-Bench, and TruthfulQA are almost exclusively English- and Western-centric. They do not identify those safety, fairness, and accuracy failures unique to the Indian context. That is the gap Inspect India Evals seeks to fill. It is an open-source framework built on top of UK AISI's Inspect AI platform. It has six benchmarks: Multilingual MMLU across sixteen Indian languages, BharatBBQ (our adaptation of BBQ for Indian social bias), a safety evaluation for Digital Public Infrastructure, a multilingual safety test using harmful prompts in Indian languages, a multi-turn jailbreak resistance test, and an Indian cultural knowledge benchmark scored using LLM-as-judge rubrics. In this study, we tested five open-weight models ranging from 8B to 32B parameters. Sarvam-M 24B and Gemma 2 27B came out on top, both scoring 80% on the composite India Fairness Index, with Sarvam-M even beating larger 32B models on Indian cultural knowledge and DPI safety compliance. All models scored 100% refusal on Multilingual Safety, whereas DPI safety varied from 20% to 100%. The framework is public. It's built to work with the UK AISI registry. Anyone can reproduce or extend this work.