Principal Data Scientist | Conversational AI

- Design and establish cutting-edge evaluation frameworks for conversational agents using LLM-as-a-judge and hybrid scoring methods. - Create high-precision prompts for evaluator models and thoroughly validate them against human assessments to ensure reliable inter-rater consistency. - Oversee the fine-tuning of smaller, cost-effective models functioning as scalable Judge models, optimizing accuracy, latency, and expense. - Curate Golden Set datasets from extensive conversation logs and formulate annotation guidelines that standardize the ground truth for subjective tasks. - Work in tandem with Engineering teams to embed quality indicators into CI/CD pipelines for automated regression testing and production oversight. - Perform in-depth analysis of agent errors such as hallucinations, tool misuse, and safety breaches, establishing actionable feedback for the modeling team. - Use evaluation data to identify systemic flaws and root causes, effectively influencing modeling teams and collaborators to prioritize targeted enhancements in performance. - Mentor senior data scientists, standardize best practices for evaluations across the organization, and uphold excellence through patents, publications, or presentations at conferences.