Senior Data Scientist (Databricks)

- Develop, train, assess, and optimize machine-learning models using tools like Spark MLlib, Python, and various cloud-based platforms - Conduct exploratory data analysis (EDA), statistical profiling, and feature engineering on extensive datasets within Databricks - Oversee and manage MLflow experiment tracking, model registry, version control, and reproducibility procedures - Contribute to model oversight, performance enhancements, drift detection, and continuous improvement initiatives - Create notebooks, jobs, and workflows in Databricks for data preparation, model training, and both batch and streaming inference - Employ Unity Catalog for secure and governed data access, lineage, and metadata management - Utilize Delta Lake for scalable feature pipelines that assist both training and production processes - Work collaboratively with Engineering to facilitate workload migration to Databricks and support optimizations for cost-effective computing - Design reusable, production-ready feature pipelines utilizing PySpark and SQL - Implement data validation, quality checks, and transformation logic compliant with enterprise standards - Engage in design discussions for data ingestion, medallion architecture workflows, and schema evolution - Collaborate with Data Engineering, Analytics, Product, and subject matter experts to convert business challenges into data-driven resolutions - Maintain thorough documentation of model assumptions, data transformations, evaluation metrics, and deployment strategies