Data Scientist - Evaluations, Chanakya

About this job

Sarvam AI is creating India's full-stack sovereign AI platform, focusing on research, models, infrastructure, and applications to make artificial intelligence effective for the country. As a Data Scientist for Evaluations under the Chanakya project, you will anchor the evaluations team by designing, building, and maintaining evaluation frameworks to measure model and system quality for real-world operational contexts. Instead of relying on standard benchmarks, you will build domain-specific evaluation harnesses for high-stakes use cases. Your daily routine involves collaborating closely with MLOps engineers, product managers, and deployment teams to define quality metrics, run structured evaluation cycles, build specialized datasets, and operationalize automated evaluation pipelines. You will proactively identify failure modes, edge cases, and distribution shifts, ensuring that AI models meet high standards before and after deployment.

Responsibilities

• Design and build evaluation frameworks for domain-specific requirements like document comprehension and geospatial reasoning
• Define quality metrics with domain experts and clients to translate operational needs into measurable signals
• Run structured pre- and post-deployment evaluation cycles and create dashboards for production quality
• Identify failure modes, edge cases, and distribution shifts with a focus on uncovering issues
• Collaborate with MLOps engineers to operationalize automated, versioned, and reproducible evaluation pipelines
• Build and manage domain-specific datasets for benchmarking, fine-tuning, and human annotation workflows
• Publish internal research findings and quality reports to influence product and engineering roadmaps

Requirements

• 3 to 6 years of experience in data science, ML research, or applied AI
• At least 2 years of practical experience working with LLMs in production environments
• Strong fundamentals in statistics and probability regarding valid evaluation design
• Demonstrated experience designing evaluation frameworks from scratch with custom metrics
• Proficiency in Python along with tools like pandas, NumPy, and HuggingFace datasets
• Familiarity with LangSmith, RAGAS, EleutherAI Eval Harness, or equivalent evaluation tools
• Experience with prompt engineering, model fine-tuning, or RLHF in applied settings
• Ability to work effectively with unstructured domain data such as PDFs and transcripts

Good to have

• Prior work experience in high-stakes domains like healthcare, legal, defence, or finance
• Experience with red-teaming or conducting adversarial evaluations

Who should apply

This role suits experienced data science professionals who have a strong background in machine learning research, applied AI, and working with production-level LLMs. Candidates should possess robust statistical skills and a knack for identifying failure modes in complex artificial intelligence systems. The posting does not mention hiring freshers, requiring instead several years of specialized industry experience. If you are passionate about building sovereign AI infrastructure for India and can take complete ownership of model evaluation outcomes, you are an ideal fit for this position at Sarvam AI.

Frequently asked questions