Senior Machine Learning Engineer - federal project

- Lead the design of machine learning solutions for operations and reliability applications, including alert noise reduction, alert clustering, anomaly detection, incident root cause analysis, and usage insights - Convert product specifications and reliability requirements into comprehensible machine learning challenges with defined metrics like false positive rates, alert reduction goals, and impacts on incident response times - Choose suitable model families for various use cases, covering both supervised and unsupervised classical models, deep learning models, and appropriate language model approaches - Collaborate with data engineering to establish and refine pipelines that process operational alerts, logs, metrics, and incident data - Create features capturing temporal patterns, relationships between services and infrastructure, and the criticality of systems and alerts - Enforce data validation protocols and quality checks, while working together on recognizing and addressing data drift and schema changes - Develop and maintain a robust machine learning operations workflow, including experiment tracking, model registry, automated training, and deployment - Create production-ready inference services, such as synchronous APIs, batch jobs, and streaming scoring integrated with backend solutions and user interfaces - Collaborate with on-site DevOps for deployment patterns in secure environments, including staging, canary releases, controlled rollouts, and rollback plans - Set retraining strategies and schedules for models impacted by changes in alert distributions and operational behaviors - Devise evaluation suites using historical alert and incident datasets to simulate realistic scenarios for alert suppression and recommendation quality - Construct dashboards to make model performance and impact visible to product stakeholders, operations teams, and technical leadership - Monitor model performance and drift in production, initiating corrective actions as necessary - Integrate feedback from operators and subject matter experts into continuous improvement initiatives and relevant active learning workflows - Operate within the parameters of secure and regulated deployments, ensuring adherence to access control, logging, and change management requirements - Ensure that training and experimentation setups utilizing sensitive data comply with security and regulatory standards for U.S. Federal workloads and FedRAMP environments - Document model inputs, outputs, assumptions, and controls for review by security and compliance teams - Coordinate on shared machine learning components across various products, such as embedding and semantic search services - Engage in architecture and design discussions to encourage the reuse of patterns and components across the AI and data platform - Provide mentorship and technical advice to junior engineers and data scientists as necessary - Primarily work from our Dallas office, collaborating closely with local engineering, product, and leadership teams - Participate in face-to-face design sessions, whiteboard discussions, and incident reviews requiring real-time interaction - Contribute to fostering a strong on-site engineering culture through knowledge sharing, design collaboration, and support for local colleagues