IIT Madras’ AI4Bharat Introduces New Benchmark Test for Indian Languages

AI4Bharat, a research lab at the Indian Institute of Technology (IIT) Madras, has launched a significant initiative aimed at evaluating large language models (LLMs) for Indian languages, cultural context, and safety. The initiative, known as Indic LLM-Arena, is an open-source benchmark platform specifically designed to assess the performance of LLMs in a manner that is tailored to the linguistic and social realities of India. Unlike traditional benchmarks, which often focus solely on translation or text comprehension, Indic LLM-Arena aims to capture the nuanced ways Indians communicate, including code-switching phenomena such as Hinglish or Tanglish, where multiple languages may be mixed within a single sentence.

The benchmark was developed with support from Google Cloud and operates as a crowd-sourced platform. Thousands of anonymous users participate by voting on the performance of different LLMs, creating a “human-in-the-loop” leaderboard that ranks the models based on real user feedback. This approach is intended to make the evaluation more reflective of real-world use, rather than relying solely on automated scoring metrics. By leveraging the input of a diverse set of users, Indic LLM-Arena ensures that performance rankings are grounded in human judgment and are more aligned with actual user expectations.

Currently, the benchmark supports text-based inputs in multiple Indian languages as well as scenarios that involve code-mixed text. However, AI4Bharat has indicated plans to expand the platform to include omni models with vision and audio capabilities, as well as AI agents, thereby broadening the scope of evaluation to multimodal AI systems. All anonymized data, code, and evaluation pipelines will be released under an open-source license, making them available for community inspection, validation, and further development. This open-source model ensures transparency and encourages collaboration within the Indian AI research ecosystem.

The creation of Indic LLM-Arena addresses a significant gap in the Indian AI landscape: the lack of local benchmarks for evaluating Indic LLMs. Several Indian AI developers have repeatedly pointed out that existing global benchmarks are not fully adequate for testing the performance of AI models in Indian languages and contexts. Indic LLM-Arena, therefore, provides a much-needed tool for researchers and startups to accurately assess their models against region-specific use cases. The platform allows developers to see precisely how their models perform relative to others, facilitating both improvement and innovation in the development of AI systems for India.

This development comes in the context of other efforts to create Indian-centric AI benchmarks. For example, OpenAI recently launched IndQA, a benchmark designed to evaluate linguistic ability and cultural understanding across 12 languages and 10 cultural domains, using 2,278 questions compiled in partnership with 261 experts from India. Indic LLM-Arena complements such initiatives by offering a broader, crowd-sourced perspective, emphasizing real-world usability and practical performance.

AI4Bharat hopes that Indic LLM-Arena will serve multiple purposes: it will provide startups and researchers with actionable insights on model performance, allow businesses across sectors to make informed decisions about which AI models to deploy, and ultimately accelerate the adoption of AI technologies that are relevant, safe, and culturally aligned with Indian users. By offering a structured and transparent benchmarking system, Indic LLM-Arena is poised to strengthen the AI ecosystem in India and ensure that AI systems deployed locally are tested against realistic and contextually relevant criteria.