Data Scientist - AI and Business Intelligence

- Define evaluation strategies for LLM/RAG/agents, focusing on groundedness, helpfulness, safety, regression testing, and release criteria - Build and maintain benchmark datasets and scoring criteria for agent performance across critical use cases - Establish monitoring protocols for agent outcomes including quality, latency, cost, drift, user feedback, and business KPIs - Collaborate with the AI Applications Engineer on prompt strategies, retrieval patterns, tool usage, and safe fallbacks - Conduct red team/adversarial testing and implement mitigations for unsafe or low-quality behavior - Ensure privacy and security by design, including PII minimization, role-based access control, and auditable deletions - Define human-in-the-loop processes when necessary for sampling, review queues, and labeling guidelines - Manage the LLM release procedure including prompt/model/versioning, offline evaluations, staging, and rollbacks - Maintain documentation for production AI models and metrics, including evaluation reports and failure modes - Implement automated regression checks to avoid quality or safety regressions during deployments - Define incident response protocols for addressing agent-related issues including triage and corrective actions - Partner with cross-functional teams to turn inquiries into actionable plans, success metrics, and data-driven recommendations - Lead predictive modeling for BI priorities such as scoring, demand planning, and forecasting - Design and conduct experiments (A/B testing, holdouts) with appropriate power and instrumentation - Define business and model metrics; establish golden labels and decision thresholds with measurable ROI - Work on feature engineering through Salesforce, NetSuite, Five9, and other data sources, collaborating with Data Engineering for operationalization - Translate modeling results into operational procedures for scoring, routing, and dashboard alerts - Contribute to BI and analytics tasks as needed to ensure that outcomes are measurable and actionable - Enhance data clarity through metric definitions, data quality checks, and stakeholder engagement - Undertake additional assigned duties as necessary